Since you're doing a depth 1 "traversal" please use something like this
instead:
for ( Relationship rel : graphDb.getReferenceNode().getRelationships(
Relationships.ROUTE, Direction.OUTGOING ) )
{
Node node = rel.getEndNode();
// Do stuff
}
Since a traverser keeps more memory than a simple call to getRelationships.
Another thing, are you doing any write operation in that for-loop of yours?
Also do you shut down the batch inserter and start a new
EmbeddedGraphDatabase to traverse on, or how do you get a hold of the
graphDb?
2010/7/26 Tim Jones <bogol...@ymail.com>
> OK, I found out what's taking the time. It's iterating over the result set
> of a
> traverser:
>
> // visit each Route node, and add it to the array
> Traverser routes = graphDb.getReferenceNode().traverse(
> Traverser.Order.BREADTH_FIRST,
> StopEvaluator.DEPTH_ONE,
> ReturnableEvaluator.ALL_BUT_START_NODE,
> Relationships.ROUTE, Direction.OUTGOING);
>
> for (Node node : routes)
> {
> // do stuff
> }
>
>
> The 'for' loop takes ages. There are probably 2m nodes being returned by
> that
> traverser at the moment, and that's only a very small subset of the data I
> want
> to add to the database.
>
> is there any way to tinker with the neo4j properties or anything to improve
> performance here?
>
> Thanks
>
>
> ----- Original Message ----
> > From: Mattias Persson <matt...@neotechnology.com>
> > To: Neo4j user discussions <user@lists.neo4j.org>
> > Sent: Sat, July 24, 2010 10:23:02 PM
> > Subject: Re: [Neo4j] Batch inserter shutdown taking forever
> >
> > 2010/7/21 Tim Jones <bogol...@ymail.com>
> >
> > > Hi,
> > >
> > > I'm using a BatchInserter and a LuceneIndexBatchInserter to insert >5m
> > > nodes and
> > > >5m relationships into a graph in one go. The insertion seems to work,
> but
> > > shutting down takes forever - it's been 2 hours now.
> > >
> > > At first, the JVM gave me garbage collection exception, so I've set
> the
> > > heap to
> > > 2gb.
> > >
> > > 'top' tells me that the application is still running:
> > >
> > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > 9994 tim 17 0 2620m 2.3g 238m S 99.5 39.1 115:48.84 java
> > >
> > > but checking the filesystem by running 'ls -l' a few times doesn't
> indicate
> > > that
> > > files are being updated.
> > >
> > > Is this normal? Is there a way to improve performance?
> > >
> >
> > No, it sounds quite weird. Any chance to have a look at your code?
> >
> >
> > >
> > > I'm loading all my data in one go to ease creating the db - it's
> simpler to
> > > create it from scratch each time instead of updating an existing
> database
> -
> > > so
> > > ideally I don't want to break this job down into multiple smaller jobs
> > > (actually, this would be OK if performance was good, but I ran into
> > > problems
> > > inserting data and retrieving existing nodes).
> > >
> >
> > What kind of problems? could you supply code and description of your
> > problems?
>
> Problems doing something similar in relational dbs. Also, the API
> recommends to
> optimise the batch search index before using it for lookups. I just decided
> not
> to take this approach.
>
> >
> >
> > >
> > > Thanks,
> > > Tim
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Neo4j mailing list
> > > User@lists.neo4j.org
> > > https://lists.neo4j.org/mailman/listinfo/user
> > >
> >
> >
> >
> > --
> > Mattias Persson, [matt...@neotechnology.com]
> > Hacker, Neo Technology
> > www.neotechnology.com
> > _______________________________________________
> > Neo4j mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> >
>
>
>
>
> _______________________________________________
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
_______________________________________________
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
