It finished. 32GB memory in the VM and 28GB of JVM heap. In total the import created: Created 4595170 relationships, returned 0 rows in 390028 ms
Chris On Wed, Aug 27, 2014 at 5:14 PM, Michael Hunger < [email protected]> wrote: > Chris, > > your cypher query seems to be wrong: > > 1. split it up into node creation and relationship creation > 2. use bigger transaciton sizes > 3. you forgot a colon before :GraphPart so it doesn't use an index for > that one > 4. you don't have do use the path and foreach a simple match is good enough > > USING PERIODIC COMMIT 10000 > > LOAD CSV WITH HEADERS FROM "file://localhost/home/deployer/tblMfr.csv" AS > csvLine > FIELDTERMINATOR '\t' > CREATE (vendor:GraphVendor { vendor_code_id: toInt(csvLine.Mfr_Code_ID), > vendor_id: toInt(csvLine.Mfr_ID), vendor_name: csvLine.Mfr_Name, > vendor_abbreviation: csvLine.Mfr_Abbr, vendor_status: csvLine.Mfr_Status }); > > > create index on :GraphVendor(vendor_id); > > USING PERIODIC COMMIT 10000 > > LOAD CSV WITH HEADERS FROM "file://localhost/home/deployer/tblMfr.csv" AS > csvLine > FIELDTERMINATOR '\t' > > WITH toInt(csvLine.Mfr_ID) as vendor_id > > MATCH (vendor:GraphVendor { vendor_id: vendor_id}) > MATCH (part:GraphPart {mfr_id: vendor_id}) > MERGE (part)-[:MANUFACTURED_BY]->(vendor); > > > > Am 26.08.2014 um 23:33 schrieb Chris G <[email protected]>: > > Group, I'm trying to wrap me head around the memory configuration for > Neo4j. > > I've got ~4 million parts that I have loaded and indexed via cypher and > have these indexes: > > Indexes > ON :GraphPart(mfr_id) ONLINE > ON :GraphPart(part_id) ONLINE (for uniqueness constraint) > > Constraints > ON (graphpart:GraphPart) ASSERT graphpart.part_id IS UNIQUE > > > > Now I want to import my vendors via this cypher: > > USING PERIODIC COMMIT 1 > LOAD CSV WITH HEADERS FROM "file://localhost/home/deployer/tblMfr.csv" AS > csvLine > FIELDTERMINATOR '\t' > CREATE (vendor:GraphVendor { vendor_code_id: toInt(csvLine.Mfr_Code_ID), > vendor_id: toInt(csvLine.Mfr_ID), vendor_name: csvLine.Mfr_Name, > vendor_abbreviation: csvLine.Mfr_Abbr, vendor_status: csvLine.Mfr_Status }) > WITH vendor > MATCH p = (GraphPart {mfr_id: vendor.vendor_id}) > FOREACH (n IN nodes(p) | MERGE (n)-[r:MANUFACTURED_BY]->(vendor)) > > > I have configured the conf files: > > neo4j.properties: > neostore.nodestore.db.mapped_memory=50M > neostore.relationshipstore.db.mapped_memory=500M > neostore.propertystore.db.mapped_memory=100M > neostore.propertystore.db.strings.mapped_memory=130M > neostore.propertystore.db.arrays.mapped_memory=0M > > neo4j-wrapper.conf: > > wrapper.java.initmemory=4096 > wrapper.java.maxmemory=12288 > > > even with 12G heap and PERIODIC COMMIT *1 *messages.log looks like this: > 2014-08-26 21:14:08.936+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 719ms [total block > time: 16.227s] > 2014-08-26 21:14:10.874+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 1630ms [total block > time: 17.857s] > 2014-08-26 21:14:12.377+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 673ms [total block > time: 18.53s] > 2014-08-26 21:14:13.715+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 719ms [total block > time: 19.249s] > 2014-08-26 21:14:15.424+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 1400ms [total block > time: 20.649s] > 2014-08-26 21:14:16.924+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 754ms [total block > time: 21.403s] > 2014-08-26 21:14:18.146+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 908ms [total block > time: 22.311s] > 2014-08-26 21:14:19.881+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 1207ms [total block > time: 23.518s] > 2014-08-26 21:14:21.551+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 1033ms [total block > time: 24.551s] > 2014-08-26 21:14:22.801+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 827ms [total block > time: 25.378s] > 2014-08-26 21:14:49.154+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 26040ms [total block > time: 51.418s] > 2014-08-26 21:14:49.524+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 270ms [total block > time: 51.688s] > 2014-08-26 21:15:24.662+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 32772ms [total block > time: 84.46s] > 2014-08-26 21:15:51.122+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 26039ms [total block > time: 110.499s] > 2014-08-26 21:16:24.233+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 32902ms [total block > time: 143.401s] > 2014-08-26 21:16:50.232+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 25898ms [total block > time: 169.299s] > 2014-08-26 21:17:20.085+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 29753ms [total block > time: 199.052s] > 2014-08-26 21:17:46.225+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 26040ms [total block > time: 225.092s] > 2014-08-26 21:21:04.960+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC > Monitor: Application threads blocked for an additional 29433ms [total block > time: 254.525s] > > > Could anyone suggest what I can try next, or some alternative memory > settings? > > I'm trying to get proof of concept up and running so I can present this to > my bosses. > > I hope I am missing something simple, if not I think it's time for Neo4j > to invest in some canonical documentation on how to configure neo4j memory > usage, There are sparse mentions in the user guide, but most of what I find > related to performance comes from blog posts, stack overflow questions, and > mailing list posts (most of which Michael Hunger is answering). I also hope > once I get past these initial memory settings the rest of neo4j will just > work. > > Thanks for reading, > > Chris > > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Neo4j" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/neo4j/rOr8tL1r-R8/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- CR -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
