Can you also check messages.log and share the load csv statement The memory should not grow to exceed the heap, esp. with periodic commit
M Von meinem iPhone gesendet > Am 23.03.2015 um 22:22 schrieb cui r <[email protected]>: > > Yes, I did send them to the master node. Since I am monitoring mem from "top" > so I can see the cpu jumps when the req is coming in. > > I can see the mem steadily going up, once it reaches the 24G ceiling I set, > it throws out the error. That's when the 7th or 8th file loading starts. > > Once the error is thrown, neo4j master somehow notices that its data store is > corrupted and stops being the master, then copies the data from another node. > Here is the relevant printout: > > 2015-03-23 21:04:03.238+0000 INFO [Cluster] The store is inconsistent. Will > treat it as branched and fetch a new one from the master > 2015-03-23 21:04:03.246+0000 INFO [Cluster] Instance 822760924 (this server) > is unavailable as backup > 2015-03-23 21:04:08.365+0000 INFO [Cluster] ServerId 822760924, moving to > slave for master ha://10.49.220.86:6001?serverId=822761180 > 2015-03-23 21:04:08.426+0000 INFO [Cluster] Copying store from master > 2015-03-23 21:04:08.938+0000 INFO [Cluster] Copying index.db > 2015-03-23 21:04:08.938+0000 INFO [Cluster] Copied index.db 239.00 B > > If you need more info, let me know. > > Appreciate it. > Rick > >> On Monday, March 23, 2015 at 4:00:13 PM UTC-4, Michael Hunger wrote: >> Just send it to the master or single machine only >> >> Don't send writes to slaves >> >> Von meinem iPhone gesendet >> >>> Am 23.03.2015 um 20:45 schrieb cui r <[email protected]>: >>> >>> Hi Michael, >>> >>> I was using M04. I just tried RC01 as you suggested, it worked. >>> >>> However, it still does not solve my real problem. (Sorry for asking more, >>> :-)). >>> >>> I have 5.3 million records that I need to load into Neo4j (a cluster of 3 >>> in dev and 6 in prod). With one single file, it's too much. So I chunk it >>> into half-million-record files, 11 files. >>> >>> Cluster node has 24G memory. >>> >>> With or without periodic commit, load csv uses total memory quickly and >>> breaks down near 7th or 8th files. >>> >>> >>> If I reboot the cluster after 6th file, the rest works fine. (On startup, >>> neo4j uses around 3G mem) >>> >>> At first sight, looks like there is a memory leak? >>> >>> Also, once it goes back, the db is kind of corrupted - after reboot, if I >>> do another load csv, the error is: >>> >>> Unable to begin transaction. >>> Neo.DatabaseError.Statement.ExecutionFailure >>> >>> From the log, it seems that the server thinks the data is out of sync and >>> then does a copying from the master. >>> >>> Any suggestions? >>> >>> Thanks in advance. >>> Rick >>> >>> >>>> On Friday, March 20, 2015 at 8:43:19 AM UTC-4, Michael Hunger wrote: >>>> Hey Rick, which version did you use for this? >>>> >>>> I think it was fixed after that version you have. >>>> >>>> Michael >>>> >>>>> Am 19.03.2015 um 15:46 schrieb cui r <[email protected]>: >>>>> >>>>> Hi, >>>>> >>>>> I am trying to use USING PERIODIC COMMIT to load a csv with 0.5 million >>>>> rows, but never get it work. The error is: >>>>> >>>>> USING PERIODIC COMMIT 1000 load csv with headers from >>>>> "file:///.../data.csv" as csvline >>>>> merge (h:Holding{holdingId:csvline.holdingId}) set ... >>>>> >>>>> The statement has been closed. >>>>> Neo.DatabaseError.Statement.ExecutionFailure >>>>> >>>>> I couldn't find any error in the logs (message.log, std err/out, or >>>>> http). I do see this statement in the http log with return code 200. >>>>> >>>>> Without using periodic commit, it takes about 50 seconds to load the >>>>> file. With the periodic commit, the error returns within 10 seconds. >>>>> >>>>> Thanks. >>>>> Rick >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Neo4j" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
