I could see the memory increase, with or without periodic commit, starts
with 4G, eventually reach 24G.
The jvm setting is:
-server
-XX:NewSize=2g -XX:MaxNewSize=2g
-Xms12g -Xmx24g
-XX:PermSize=512M -XX:MaxPermSize=512M
-verbose:gc -Xloggc:${NEO4J_LOG}/gc.log
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
-XX:CMSInitiatingOccupancyFraction=70
The load csv is:
USING PERIODIC COMMIT load csv with headers from
"file:///home/rcui/plexus/plexus-holdings8.csv" as csvline
merge (h:Holding{holdingId:csvline.holdingId}) set
h.reportDate=csvline.reportDate,h.fundId=csvline.fundId,h.fundName=csvline.fundName,h.firmName=csvline.companyName,h.cusip=csvline.cusip,h.secGroup=csvline.secGroup,h.positionDate=csvline.positionDate,h.marketValue=toFloat(csvline.marketValue),h.netChange=toFloat(csvline.marketValueChange),h.originalFace=toFloat(csvline.par)
where the number 8 is changed from 1 to 11.
The message log is clean up to number 7, then error is printed:
2015-03-23 20:58:48.220+0000 INFO [o.n.k.i.s.StoreFactory]: Successfully
rotated counts store at transaction 51881 to
[/local/kozo/neo4j/data/graph.db/neostore.counts.db.b], from
[/local/kozo/neo4j/data/graph.db/neostore.counts.db.a].
2015-03-23 20:58:48.749+0000 INFO [o.n.k.LogRotationImpl]: Log Rotation
[59]: Preparing new log file...
2015-03-23 20:58:48.749+0000 INFO [o.n.k.NeoStoreDataSource]: Opened
logical log [/local/kozo/neo4j/data/graph.db/neostore.transaction.db.60]
version=60, lastTxId=51881 (clean)
2015-03-23 20:58:48.750+0000 INFO [o.n.k.NeoStoreDataSource]: Finished
rotating log version:59
2015-03-23 20:58:48.750+0000 INFO [o.n.k.i.t.l.p.LogPruning]: Log Rotation
[59]: [84:qtp1788679549-84] Starting log pruning.
2015-03-23 20:58:48.751+0000 INFO [o.n.k.i.t.l.p.LogPruning]: Log Rotation
[59]: [84:qtp1788679549-84] Log pruning complete.
2015-03-23 20:58:54.573+0000 WARN [o.n.k.h.HighlyAvailableGraphDatabase]:
GC Monitor: Application threads blocked for 1418ms.
2015-03-23 20:58:59.654+0000 WARN [o.n.k.h.HighlyAvailableGraphDatabase]:
GC Monitor: Application threads blocked for 576ms.
2015-03-23 20:59:09.621+0000 WARN [o.n.k.h.HighlyAvailableGraphDatabase]:
GC Monitor: Application threads blocked for 458ms.
2015-03-23 20:59:12.236+0000 DEBUG [o.n.k.h.c.m.SlaveClient]: Thread[154,
Neo4j HighlyAvailableGraphDatabase[/local/kozo/neo4j/data/graph.db]-3]
Opened a new channel to /10.49.220.86:6001
2015-03-23 20:59:12.236+0000 DEBUG [o.n.k.h.c.m.SlaveClient]: Thread[153,
Neo4j HighlyAvailableGraphDatabase[/local/kozo/neo4j/data/graph.db]-2]
Opened a new channel to /10.49.220.87:6001
2015-03-23 20:59:12.236+0000 DEBUG [o.n.k.h.c.m.SlaveClient]: ResourcePool
create resource ChannelContext{channel=[id: 0x216997a3, /10.49.220.85:54146
=> /10.49.220.86:6001], output=DynamicChannelBuffer(ridx=0, widx=0,
cap=256), input=java.nio.HeapByteBuffer[pos=0 lim=1048576 cap=1048576]}
2015-03-23 20:59:12.236+0000 DEBUG [o.n.k.h.c.m.SlaveClient]: ResourcePool
create resource ChannelContext{channel=[id: 0x3424a938, /10.49.220.85:46193
=> /10.49.220.87:6001], output=DynamicChannelBuffer(ridx=0, widx=0,
cap=256), input=java.nio.HeapByteBuffer[pos=0 lim=1048576 cap=1048576]}
2015-03-23 20:59:15.494+0000 WARN [o.n.k.h.HighlyAvailableGraphDatabase]:
GC Monitor: Application threads blocked for 2370ms.
2015-03-23 20:59:21.293+0000 WARN [o.n.k.h.HighlyAvailableGraphDatabase]:
GC Monitor: Application threads blocked for 2569ms.
2015-03-23 20:59:22.493+0000 INFO [o.n.k.i.a.i.s.OnlineIndexSamplingJob]:
Sampled index :Holding(firmName) with 6108 unique values in sample of avg
size 6108 taken from index containing 6108 entries
2015-03-23 20:59:29.135+0000 WARN [o.n.k.h.HighlyAvailableGraphDatabase]:
GC Monitor: Application threads blocked for 335ms.
2015-03-23 20:59:30.527+0000 INFO [o.n.k.i.a.i.s.OnlineIndexSamplingJob]:
Sampled index :Holding(fundName) with 81048 unique values in sample of avg
size 81048 taken from index containing 81048 entries
2015-03-23 20:59:32.238+0000 ERROR [o.n.k.h.HighlyAvailableGraphDatabase]:
Slave 822761180: Replication commit threw communication exception:
org.neo4j.com.ComException:
org.jboss.netty.handler.queue.BlockingReadTimeoutException
at
org.neo4j.com.DechunkingChannelBuffer.readNext(DechunkingChannelBuffer.java:75)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at
org.neo4j.com.DechunkingChannelBuffer.readNextChunk(DechunkingChannelBuffer.java:93)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at
org.neo4j.com.DechunkingChannelBuffer.<init>(DechunkingChannelBuffer.java:59)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at org.neo4j.com.Protocol.deserializeResponse(Protocol.java:234)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at org.neo4j.com.Client.sendRequest(Client.java:225)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at org.neo4j.com.Client.sendRequest(Client.java:207)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at
org.neo4j.kernel.ha.com.master.SlaveClient.pullUpdates(SlaveClient.java:69)
~[neo4j-ha-2.2.0-RC01.jar:2.2.0-RC01]
at
org.neo4j.kernel.ha.transaction.CommitPusher.askSlaveToPullUpdates(CommitPusher.java:192)
~[neo4j-ha-2.2.0-RC01.jar:2.2.0-RC01]
at
org.neo4j.kernel.ha.transaction.CommitPusher.access$000(CommitPusher.java:39)
~[neo4j-ha-2.2.0-RC01.jar:2.2.0-RC01]
at
org.neo4j.kernel.ha.transaction.CommitPusher$1.run(CommitPusher.java:160)
~[neo4j-ha-2.2.0-RC01.jar:2.2.0-RC01]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_25]
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
~[na:1.7.0_25]
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
~[na:1.7.0_25]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_25]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
Caused by: org.jboss.netty.handler.queue.BlockingReadTimeoutException: null
at
org.jboss.netty.handler.queue.BlockingReadHandler.readEvent(BlockingReadHandler.java:232)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.handler.queue.BlockingReadHandler.read(BlockingReadHandler.java:162)
~[netty-3.6.3.Final.jar:na]
at
org.neo4j.com.DechunkingChannelBuffer.readNext(DechunkingChannelBuffer.java:66)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
... 16 common frames omitted
2015-03-23 20:59:32.238+0000 DEBUG [o.n.k.h.HighlyAvailableGraphDatabase]:
Transaction 51882 couldn't commit on enough slaves, desired 2, but could
only commit at 0
This is from master's message log. There is no printout in the slave logs.
But I do see the same line:
[/local/kozo/neo4j/data/graph.db/neostore.transaction.db.60] version=60,
lastTxId=51881 (clean)
there too.
In the slave logs, a few minutes later, I saw this:
2015-03-23 21:03:02.725+0000 INFO [o.n.c.p.h.HeartbeatContext]:
822761436(me) is now suspecting 822760924
2015-03-23 21:03:02.725+0000 INFO [o.n.c.p.h.HeartbeatContext]: 822761180
is now suspecting 822760924
2015-03-23 21:03:02.726+0000 WARN [o.n.c.p.e.HeartbeatReelectionListener]:
instance 822760924 is being demoted since it failed
2015-03-23 21:03:02.726+0000 INFO [o.n.k.h.HighAvailabilityConsoleLogger]:
Instance 822760924 has failed
2015-03-23 21:03:02.728+0000 DEBUG
[o.n.k.h.c.HighAvailabilityMemberStateMachine]: Got
memberIsFailed(822760924)
2015-03-23 21:03:02.729+0000 INFO [o.n.c.p.c.ClusterConfiguration]:
Removed role coordinator from instance 822760924
Master log prints:
2015-03-23 21:04:03.072+0000 INFO [o.n.c.p.h.HeartbeatContext]:
822760924(me) is now suspecting 822761436
2015-03-23 21:04:03.076+0000 INFO [o.n.c.p.h.HeartbeatContext]:
822760924(me) is now suspecting 822761180
2015-03-23 21:04:03.077+0000 WARN [o.n.k.h.c.m.MasterServer]: Exception
from Netty
java.nio.channels.ClosedChannelException: null
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:409)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:349)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:81)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:54)
~[netty-3.6.3.Final.jar:na]
at org.jboss.netty.channel.Channels.close(Channels.java:812)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:197)
~[netty-3.6.3.Final.jar:na]
at
org.neo4j.com.ChunkingChannelBuffer.operationComplete(ChunkingChannelBuffer.java:590)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:427)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:413)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:380)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:245)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromSelectorLoop(AbstractNioWorker.java:157)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:113)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
~[netty-3.6.3.Final.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_25]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
at
org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:99)
~[neo4j-kernel-2.2.0-RC01.jar:2.2.0-RC01]
2015-03-23 21:04:03.084+0000 WARN [o.n.k.h.c.m.MasterServer]: Exception
from Netty
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
~[na:1.7.0_25]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
~[na:1.7.0_25]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:94)
~[na:1.7.0_25]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_25]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
~[na:1.7.0_25]
at
org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:198)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromSelectorLoop(AbstractNioWorker.java:157)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:113)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
~[netty-3.6.3.Final.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_25]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
at
org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:99)
~[neo4j-kernel-2.2.0-RC01.jar:2.2.0-RC01]
2015-03-23 21:04:03.084+0000 WARN [o.n.k.h.c.m.MasterServer]: Exception
from Netty
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
~[na:1.7.0_25]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
~[na:1.7.0_25]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:94)
~[na:1.7.0_25]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_25]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
~[na:1.7.0_25]
at
org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:198)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromSelectorLoop(AbstractNioWorker.java:157)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:113)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
~[netty-3.6.3.Final.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_25]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
at
org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:99)
~[neo4j-kernel-2.2.0-RC01.jar:2.2.0-RC01]
2015-03-23 21:04:03.085+0000 ERROR [o.n.k.h.HighlyAvailableGraphDatabase]:
Slave 822761180: Replication commit threw communication exception:
org.neo4j.com.ComException:
org.jboss.netty.handler.queue.BlockingReadTimeoutException
at
org.neo4j.com.DechunkingChannelBuffer.readNext(DechunkingChannelBuffer.java:75)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at
org.neo4j.com.DechunkingChannelBuffer.readNextChunk(DechunkingChannelBuffer.java:93)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at
org.neo4j.com.DechunkingChannelBuffer.<init>(DechunkingChannelBuffer.java:59)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at org.neo4j.com.Protocol.deserializeResponse(Protocol.java:234)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at org.neo4j.com.Client.sendRequest(Client.java:225)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at org.neo4j.com.Client.sendRequest(Client.java:207)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
at
org.neo4j.kernel.ha.com.master.SlaveClient.pullUpdates(SlaveClient.java:69)
~[neo4j-ha-2.2.0-RC01.jar:2.2.0-RC01]
at
org.neo4j.kernel.ha.transaction.CommitPusher.askSlaveToPullUpdates(CommitPusher.java:192)
~[neo4j-ha-2.2.0-RC01.jar:2.2.0-RC01]
at
org.neo4j.kernel.ha.transaction.CommitPusher.access$000(CommitPusher.java:39)
~[neo4j-ha-2.2.0-RC01.jar:2.2.0-RC01]
at
org.neo4j.kernel.ha.transaction.CommitPusher$1.run(CommitPusher.java:160)
~[neo4j-ha-2.2.0-RC01.jar:2.2.0-RC01]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_25]
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
~[na:1.7.0_25]
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
~[na:1.7.0_25]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_25]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
at
org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:99)
~[neo4j-kernel-2.2.0-RC01.jar:2.2.0-RC01]
Caused by: org.jboss.netty.handler.queue.BlockingReadTimeoutException: null
at
org.jboss.netty.handler.queue.BlockingReadHandler.readEvent(BlockingReadHandler.java:232)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.handler.queue.BlockingReadHandler.read(BlockingReadHandler.java:162)
~[netty-3.6.3.Final.jar:na]
at
org.neo4j.com.DechunkingChannelBuffer.readNext(DechunkingChannelBuffer.java:66)
~[neo4j-com-2.2.0-RC01.jar:2.2.0-RC01]
... 16 common frames omitted
2015-03-23 21:04:03.085+0000 DEBUG [o.n.k.h.HighlyAvailableGraphDatabase]:
Transaction 51883 couldn't commit on enough slaves, desired 2, but could
only commit at 0
Thanks for the help. If you need more info, let me know.
Rick
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
