Hey guys,
So I have around 70 million spatial records that i want to add to the
spatial layer (I've tested with a small set and everything is smoothly,
queries returning the same results as postgis and the layer operation seems
fine)
But when i try to add all the spatial records to the database, the
performance degrades rapidly to the point that it gets really slow at
around 5 million (around 2h running time) records and hangs at ~7.7 million
(8 hours lapsed).
Since the spatial index is an Rtree that uses the graph structure to
construct itself, i am wondering why is it degrading when the number os
records increase.
Rtree insertions are O(n) if im not mistaken and thats why im concerned it
might be something between the rearranging of bounding boxes, nodes that
are not tree leaves that are causing the addToLayer process to get slower
over time.
Currently im adding nodes to the layer like that (lots of hardcoded stuff
since im trying to figure out the problem before patterns and code style):
Transaction tx = database.beginTx();
>
> try {
>
>
>> ResourceIterable<Node> layerNodes =
>> GlobalGraphOperations.at(database).getAllNodesWithLabel(label);
>
> long i = 0L;
>
> for (Node node : layerNodes) {
>
> Transaction tx2 = database.beginTx();
>
> try {
>
> layer.add(node);
>
> i++;
>
> if (i % commitInterval == 0) {
>
> log("indexing (" + i + " nodes added) ... time in seconds: "
>
> + (1.0 * (System.currentTimeMillis() - startTime) / 1000));
>
> }
>
> tx2.success();
>
> } finally {
>
> tx2.close();
>
> }
>
> }
>
> tx.success();
>
> } finally {
>
> tx.close();
>
> }
>
>
Any thoughts ? Any ideas of how performance could be increased ?
ps.: using java API
Neo4j 2.1.2, Spatial 0.13
Core i5 3570k @4.5Ghz, 32GB ram
dedicated 2TB 7200 hard drive to the database (no OS, no virtual memory
files, only the data itself)
ps2.: All geometries are LineStrings (if thats important :P) they represent
streets, roads, etc..
ps3.: the nodes are already in the database, i only need to add them to the
Layer so that i can perform spatial queries, bbox and wkb attributes are
OK, tested and working for a small set.
Thank you in advance
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
