Chris,
your cypher query seems to be wrong:
1. split it up into node creation and relationship creation
2. use bigger transaciton sizes
3. you forgot a colon before :GraphPart so it doesn't use an index for that one
4. you don't have do use the path and foreach a simple match is good enough
> USING PERIODIC COMMIT 10000
> LOAD CSV WITH HEADERS FROM "file://localhost/home/deployer/tblMfr.csv" AS
> csvLine
> FIELDTERMINATOR '\t'
> CREATE (vendor:GraphVendor { vendor_code_id: toInt(csvLine.Mfr_Code_ID),
> vendor_id: toInt(csvLine.Mfr_ID), vendor_name: csvLine.Mfr_Name,
> vendor_abbreviation: csvLine.Mfr_Abbr, vendor_status: csvLine.Mfr_Status });
create index on :GraphVendor(vendor_id);
> USING PERIODIC COMMIT 10000
> LOAD CSV WITH HEADERS FROM "file://localhost/home/deployer/tblMfr.csv" AS
> csvLine
> FIELDTERMINATOR '\t'
WITH toInt(csvLine.Mfr_ID) as vendor_id
>
> MATCH (vendor:GraphVendor { vendor_id: vendor_id})
> MATCH (part:GraphPart {mfr_id: vendor_id})
> MERGE (part)-[:MANUFACTURED_BY]->(vendor);
Am 26.08.2014 um 23:33 schrieb Chris G <[email protected]>:
> Group, I'm trying to wrap me head around the memory configuration for Neo4j.
>
> I've got ~4 million parts that I have loaded and indexed via cypher and have
> these indexes:
>
> Indexes
> ON :GraphPart(mfr_id) ONLINE
> ON :GraphPart(part_id) ONLINE (for uniqueness constraint)
>
> Constraints
> ON (graphpart:GraphPart) ASSERT graphpart.part_id IS UNIQUE
>
>
> Now I want to import my vendors via this cypher:
>
> USING PERIODIC COMMIT 1
> LOAD CSV WITH HEADERS FROM "file://localhost/home/deployer/tblMfr.csv" AS
> csvLine
> FIELDTERMINATOR '\t'
> CREATE (vendor:GraphVendor { vendor_code_id: toInt(csvLine.Mfr_Code_ID),
> vendor_id: toInt(csvLine.Mfr_ID), vendor_name: csvLine.Mfr_Name,
> vendor_abbreviation: csvLine.Mfr_Abbr, vendor_status: csvLine.Mfr_Status })
> WITH vendor
> MATCH p = (GraphPart {mfr_id: vendor.vendor_id})
> FOREACH (n IN nodes(p) | MERGE (n)-[r:MANUFACTURED_BY]->(vendor))
>
>
> I have configured the conf files:
>
> neo4j.properties:
> neostore.nodestore.db.mapped_memory=50M
> neostore.relationshipstore.db.mapped_memory=500M
> neostore.propertystore.db.mapped_memory=100M
> neostore.propertystore.db.strings.mapped_memory=130M
> neostore.propertystore.db.arrays.mapped_memory=0M
>
> neo4j-wrapper.conf:
>
> wrapper.java.initmemory=4096
> wrapper.java.maxmemory=12288
>
>
> even with 12G heap and PERIODIC COMMIT 1 messages.log looks like this:
> 2014-08-26 21:14:08.936+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 719ms [total block time:
> 16.227s]
> 2014-08-26 21:14:10.874+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 1630ms [total block time:
> 17.857s]
> 2014-08-26 21:14:12.377+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 673ms [total block time: 18.53s]
> 2014-08-26 21:14:13.715+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 719ms [total block time:
> 19.249s]
> 2014-08-26 21:14:15.424+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 1400ms [total block time:
> 20.649s]
> 2014-08-26 21:14:16.924+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 754ms [total block time:
> 21.403s]
> 2014-08-26 21:14:18.146+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 908ms [total block time:
> 22.311s]
> 2014-08-26 21:14:19.881+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 1207ms [total block time:
> 23.518s]
> 2014-08-26 21:14:21.551+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 1033ms [total block time:
> 24.551s]
> 2014-08-26 21:14:22.801+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 827ms [total block time:
> 25.378s]
> 2014-08-26 21:14:49.154+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 26040ms [total block time:
> 51.418s]
> 2014-08-26 21:14:49.524+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 270ms [total block time:
> 51.688s]
> 2014-08-26 21:15:24.662+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 32772ms [total block time:
> 84.46s]
> 2014-08-26 21:15:51.122+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 26039ms [total block time:
> 110.499s]
> 2014-08-26 21:16:24.233+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 32902ms [total block time:
> 143.401s]
> 2014-08-26 21:16:50.232+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 25898ms [total block time:
> 169.299s]
> 2014-08-26 21:17:20.085+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 29753ms [total block time:
> 199.052s]
> 2014-08-26 21:17:46.225+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 26040ms [total block time:
> 225.092s]
> 2014-08-26 21:21:04.960+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC Monitor:
> Application threads blocked for an additional 29433ms [total block time:
> 254.525s]
>
>
> Could anyone suggest what I can try next, or some alternative memory
> settings?
>
> I'm trying to get proof of concept up and running so I can present this to my
> bosses.
>
> I hope I am missing something simple, if not I think it's time for Neo4j to
> invest in some canonical documentation on how to configure neo4j memory
> usage, There are sparse mentions in the user guide, but most of what I find
> related to performance comes from blog posts, stack overflow questions, and
> mailing list posts (most of which Michael Hunger is answering). I also hope
> once I get past these initial memory settings the rest of neo4j will just
> work.
>
> Thanks for reading,
>
> Chris
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
