Yes, I did send them to the master node. Since I am monitoring mem from "top" so I can see the cpu jumps when the req is coming in.
I can see the mem steadily going up, once it reaches the 24G ceiling I set, it throws out the error. That's when the 7th or 8th file loading starts. Once the error is thrown, neo4j master somehow notices that its data store is corrupted and stops being the master, then copies the data from another node. Here is the relevant printout: 2015-03-23 21:04:03.238+0000 INFO [Cluster] The store is inconsistent. Will treat it as branched and fetch a new one from the master 2015-03-23 21:04:03.246+0000 INFO [Cluster] Instance 822760924 (this server) is unavailable as backup 2015-03-23 21:04:08.365+0000 INFO [Cluster] ServerId 822760924, moving to slave for master ha://10.49.220.86:6001?serverId=822761180 2015-03-23 21:04:08.426+0000 INFO [Cluster] Copying store from master 2015-03-23 21:04:08.938+0000 INFO [Cluster] Copying index.db 2015-03-23 21:04:08.938+0000 INFO [Cluster] Copied index.db 239.00 B If you need more info, let me know. Appreciate it. Rick On Monday, March 23, 2015 at 4:00:13 PM UTC-4, Michael Hunger wrote: > > Just send it to the master or single machine only > > Don't send writes to slaves > > Von meinem iPhone gesendet > > Am 23.03.2015 um 20:45 schrieb cui r <[email protected] <javascript:>>: > > Hi Michael, > > I was using M04. I just tried RC01 as you suggested, it worked. > > However, it still does not solve my real problem. (Sorry for asking more, > :-)). > > I have 5.3 million records that I need to load into Neo4j (a cluster of 3 > in dev and 6 in prod). With one single file, it's too much. So I chunk it > into half-million-record files, 11 files. > > Cluster node has 24G memory. > > With or without periodic commit, load csv uses total memory quickly and > breaks down near 7th or 8th files. > > > If I reboot the cluster after 6th file, the rest works fine. (On startup, > neo4j uses around 3G mem) > > At first sight, looks like there is a memory leak? > > Also, once it goes back, the db is kind of corrupted - after reboot, if I > do another load csv, the error is: > > Unable to begin transaction. > > Neo.DatabaseError.Statement.ExecutionFailure > > > From the log, it seems that the server thinks the data is out of sync and > then does a copying from the master. > > > Any suggestions? > > Thanks in advance. > Rick > > > On Friday, March 20, 2015 at 8:43:19 AM UTC-4, Michael Hunger wrote: >> >> Hey Rick, which version did you use for this? >> >> I think it was fixed after that version you have. >> >> Michael >> >> Am 19.03.2015 um 15:46 schrieb cui r <[email protected]>: >> >> Hi, >> >> I am trying to use USING PERIODIC COMMIT to load a csv with 0.5 million >> rows, but never get it work. The error is: >> >> USING PERIODIC COMMIT 1000 load csv with headers from " >> file:///.../data.csv" as csvline >> merge (h:Holding{holdingId:csvline.holdingId}) set ... >> >> The statement has been closed. >> >> Neo.DatabaseError.Statement.ExecutionFailure >> >> >> I couldn't find any error in the logs (message.log, std err/out, or http). I >> do see this statement in the http log with return code 200. >> >> >> Without using periodic commit, it takes about 50 seconds to load the file. >> With the periodic commit, the error returns within 10 seconds. >> >> >> Thanks. >> >> Rick >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
