Can you share the full output. What kind of system do you run this on? OS, RAM, DISK (type and speed). How did you configure heap / total RAM? I think the importer benefits more from having non-heap memory available, so it could be fine to limit your heap to 8G.
Can you share the full command-line that you start here? Also the size of the individual CSV files. Please also share which kinds of properties (counts and sizes) you add to nodes and relationships. And which kinds of id's you use to connect Nodes via relationships I think it is rather an RAM issue that the importer has available as well as a disk performance issue. We tested the importer with datasets of this size and had no issues. Without the information requested above it will be really hard to help you. The "index" that is build is really just an in-memory structure. Michael > Am 14.11.2015 um 18:18 schrieb Neha Agarwal <[email protected]>: > > Hi, > > I am importing data into Neo4j using the batch importer CLI. It has about a > billion nodes and roughly 2B relationships. The nodes seemed to have loaded > in an hour (if I am reading the output correctly) but now it is doing a node > index and has been doing so for > 12 hours. > > Nodes > [>:23.39 MB/s---|PROPERTIE|NODE:|LAB|*v:37.18 > MB/s---------------------------------------------] 1B > Done in 1h 7m 18s 54ms > Prepare node index > [*SORT:11.52 > GB--------------------------------------------------------------------------------]881M > > Any idea why it is so slow? I don't think it is meant to be? I can think of a > few things to speed up (which I am trying in another instance) but want to > get feedback from folks: > 1. Split up the creation of the nodes and relationships into 2 separate > commands > 2. Create indexes after the creation of the nodes > 3. Do match/merge to get rid of duplicates > 4. Run the import for relationships. > > Do you think this will help? If not, what am I doing wrong? > Thanks > Neha. > > PS: My heap size is large(ish) -- around 25G. Is that maybe an issue? > > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
