I think this part of your query: (m {node:'%s'}), (n {node:'%s'}) might be
spending a lot of time building up a cartesian product of all your nodes. Try
inlining them into the path expression.
Don’t give the JVM more heap memory than you have RAM, with a GB or two to
spare for the operating system. If heap memory gets swapped out, then the GC
pauses can get very, very long.
--
Chris Vest
System Engineer, Neo Technology
[ skype: mr.chrisvest, twitter: chvest ]
> On 20 Nov 2014, at 19:09, Erika Arnold <[email protected]> wrote:
>
> tldr
>
> My project involves Wikipedia's pagelinks dataset. When imported into in
> Neo4j, this results in a large directed graph with ~11m nodes and ~172m
> relationships. I want to efficiently find the shortest path between any two
> nodes in the graph. With my current query--and after tweaking with Java's
> memory settings--the query takes ~60 seconds to return a path. I would like
> feedback to decrease this response time.
>
> details
>
> My setup is a MacBook Air (1.3 GHz Intel Core i5, 4G 1600 MHz DDR3, OS X
> 10.9.5) with Neo4j (v. 2.1.5) and Java (v. 1.7.0_71) installed.
>
> Here's my github repo <https://github.com/erabug/wikigraph> (the readme
> contains more details for the following methods).
>
> I successfully batch imported my nodes.csv and rels.csv files into Neo4j. As
> I mentioned above, this produces a graph with ~11m nodes and ~172m
> relationships.
>
> The data model is simple: All Wikipedia pages are nodes with an id and title
> ('node', 'name') as well as a label for its category (all nodes are 'Pages',
> some have specific categories also, e.g. 'OfficeHolder'). There is only one
> relationship type, 'LINKS_TO', that describes which pages the node links to.
>
> graph structure: (Page) -[:LINKS_TO]-> (Page)
>
> Here is the query I use, via py2neo (v. 1.6.4) CypherQuery object:
>
> query = neo4j.CypherQuery(
> graph_db,
> """MATCH (m {node:'%s'}), (n {node:'%s'}),
> p = shortestPath((m)-[*..20]->(n)) RETURN p""" % (node1, node2)
> )
> path = query.execute_one()
> Auto-indexing (on 'node', e.g. id number) is turned on. Increases in
> java.initmemory and java.maxmemory had a dramatic effect on response time. At
> default settings for both (512MB), the shortest path was returned in ~27
> minutes. At any setting higher than 4G (currently using 8192MB), the path is
> returned in ~60 seconds. I also tweaked settings in neo4j.properties, but saw
> no noticeable decreases.
>
> logs
>
> messages.log <https://gist.github.com/erabug/e2e683fbeae124804370>
> what I've found from googling
>
> Neo4j Cypher path finding slow in undirected graph
> <http://stackoverflow.com/questions/15456345/neo4j-cypher-path-finding-slow-in-undirected-graph>,
> Tuning neo4j for performance
> <http://stackoverflow.com/questions/17661902/tuning-neo4j-for-performance>,
> and Neo4j's Performance Guide
> <http://neo4j.com/docs/stable/performance-guide.html>. However, I'm not sure
> I know enough Java to try some of the suggestions on my own. If that's what
> is required to increase response time, I'm happy to learn, but I wanted to
> make sure it was the right approach first.
>
> server experiment
>
> I also deployed to an Amazon EC2 instance (t2.micro, 1G memory, 1vCU,
> ubuntu), just to experiment. I tried to change the same neo4j java settings
> there as I had on my local machine, but I could not run the neo4j server with
> anything other than the defaults. As a result, the query there takes ~22 mins.
>
> feedback
>
> I would love advice about the query, my settings, and things to try on the
> server (where I will ultimately want to house my project). Please let me know
> if I can provide any further information or clarification.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected]
> <mailto:[email protected]>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
