I'm going to use the Talend connector for my initial load, then the rest
api to keep my graph in sync.
On Aug 26, 2014 5:35 PM, "Chris G" <[email protected]> wrote:
> I want to add that I am using 2.1.3 Enterprise, running on Ubuntu 12.04, 8
> cpu's, 16GB memory, virtual machine.
>
> On Tuesday, August 26, 2014 5:33:22 PM UTC-4, Chris G wrote:
>>
>> Group, I'm trying to wrap me head around the memory configuration for
>> Neo4j.
>>
>> I've got ~4 million parts that I have loaded and indexed via cypher and
>> have these indexes:
>>
>> Indexes
>> ON :GraphPart(mfr_id) ONLINE
>> ON :GraphPart(part_id) ONLINE (for uniqueness constraint)
>>
>> Constraints
>> ON (graphpart:GraphPart) ASSERT graphpart.part_id IS UNIQUE
>>
>>
>>
>> Now I want to import my vendors via this cypher:
>>
>> USING PERIODIC COMMIT 1
>> LOAD CSV WITH HEADERS FROM "file://localhost/home/deployer/tblMfr.csv"
>> AS csvLine
>> FIELDTERMINATOR '\t'
>> CREATE (vendor:GraphVendor { vendor_code_id: toInt(csvLine.Mfr_Code_ID),
>> vendor_id: toInt(csvLine.Mfr_ID), vendor_name: csvLine.Mfr_Name,
>> vendor_abbreviation: csvLine.Mfr_Abbr, vendor_status: csvLine.Mfr_Status })
>> WITH vendor
>> MATCH p = (GraphPart {mfr_id: vendor.vendor_id})
>> FOREACH (n IN nodes(p) | MERGE (n)-[r:MANUFACTURED_BY]->(vendor))
>>
>>
>> I have configured the conf files:
>>
>> neo4j.properties:
>> neostore.nodestore.db.mapped_memory=50M
>> neostore.relationshipstore.db.mapped_memory=500M
>> neostore.propertystore.db.mapped_memory=100M
>> neostore.propertystore.db.strings.mapped_memory=130M
>> neostore.propertystore.db.arrays.mapped_memory=0M
>>
>> neo4j-wrapper.conf:
>>
>> wrapper.java.initmemory=4096
>> wrapper.java.maxmemory=12288
>>
>>
>> even with 12G heap and PERIODIC COMMIT *1 *messages.log looks like this:
>> 2014-08-26 21:14:08.936+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 719ms [total block
>> time: 16.227s]
>> 2014-08-26 21:14:10.874+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 1630ms [total block
>> time: 17.857s]
>> 2014-08-26 21:14:12.377+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 673ms [total block
>> time: 18.53s]
>> 2014-08-26 21:14:13.715+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 719ms [total block
>> time: 19.249s]
>> 2014-08-26 21:14:15.424+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 1400ms [total block
>> time: 20.649s]
>> 2014-08-26 21:14:16.924+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 754ms [total block
>> time: 21.403s]
>> 2014-08-26 21:14:18.146+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 908ms [total block
>> time: 22.311s]
>> 2014-08-26 21:14:19.881+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 1207ms [total block
>> time: 23.518s]
>> 2014-08-26 21:14:21.551+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 1033ms [total block
>> time: 24.551s]
>> 2014-08-26 21:14:22.801+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 827ms [total block
>> time: 25.378s]
>> 2014-08-26 21:14:49.154+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 26040ms [total block
>> time: 51.418s]
>> 2014-08-26 21:14:49.524+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 270ms [total block
>> time: 51.688s]
>> 2014-08-26 21:15:24.662+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 32772ms [total block
>> time: 84.46s]
>> 2014-08-26 21:15:51.122+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 26039ms [total block
>> time: 110.499s]
>> 2014-08-26 21:16:24.233+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 32902ms [total block
>> time: 143.401s]
>> 2014-08-26 21:16:50.232+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 25898ms [total block
>> time: 169.299s]
>> 2014-08-26 21:17:20.085+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 29753ms [total block
>> time: 199.052s]
>> 2014-08-26 21:17:46.225+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 26040ms [total block
>> time: 225.092s]
>> 2014-08-26 21:21:04.960+0000 WARN [o.n.k.EmbeddedGraphDatabase]: GC
>> Monitor: Application threads blocked for an additional 29433ms [total block
>> time: 254.525s]
>>
>>
>> Could anyone suggest what I can try next, or some alternative memory
>> settings?
>>
>> I'm trying to get proof of concept up and running so I can present this
>> to my bosses.
>>
>> I hope I am missing something simple, if not I think it's time for Neo4j
>> to invest in some canonical documentation on how to configure neo4j memory
>> usage, There are sparse mentions in the user guide, but most of what I find
>> related to performance comes from blog posts, stack overflow questions, and
>> mailing list posts (most of which Michael Hunger is answering). I also hope
>> once I get past these initial memory settings the rest of neo4j will just
>> work.
>>
>> Thanks for reading,
>>
>> Chris
>>
>>
>>
>>
>>
>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Neo4j" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/neo4j/rOr8tL1r-R8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
