Hi,
I am trying following workload from ycsb with 2 node ignite cluster with 1
backup enabled. writesync=prim_sync.
recordcount=4000000
operationcount=4000000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
insertproportion=0.23
updateproportion=0.46
readproportion=0.31
requestdistribution=zipfian
threadcount=180
target=180000
But in few seconds one of the ignite node stops with following error:
[19:07:36,283][SEVERE][grid-nio-worker-tcp-comm-3-#28][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=3,
bytesRcvd=2843314842, bytesSent=487323705, bytesRcvd0=0, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-3,
igniteInstanceName=null, finished=false, hashCode=170320615,
interrupted=false, runner=grid-nio-worker-tcp-comm-3-#28]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=2885472, resendCnt=0,
rcvCnt=2767636, sentCnt=2885548, reserved=true, lastAck=2767616,
nodeLeft=false, node=TcpDiscoveryNode
[id=cbc18e72-429a-44ef-84eb-df2e8a805866, addrs=[172.16.0.17],
sockAddrs=[/172.16.0.17:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1543950039896, loc=false, ver=2.6.0#20180710-sha1:669feacc,
isClient=false], connected=false, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=2885472, resendCnt=0,
rcvCnt=2767636, sentCnt=2885548, reserved=true, lastAck=2767616,
nodeLeft=false, node=TcpDiscoveryNode
[id=cbc18e72-429a-44ef-84eb-df2e8a805866, addrs=[172.16.0.17],
sockAddrs=[/172.16.0.17:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1543950039896, loc=false, ver=2.6.0#20180710-sha1:669feacc,
isClient=false], connected=false, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/172.16.0.4:41328, rmtAddr=/172.16.0.17:47100,
createTime=1543950040183, closeTime=0, bytesSent=3480594647,
bytesRcvd=3339280186, bytesSent0=0, bytesRcvd0=0,
sndSchedTime=1543950040183, lastSndTime=1543950373693,
lastRcvTime=1543950373693, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5c6f7548, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]]]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1649)
at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1306)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2342)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
[19:07:36,872][SEVERE][grid-nio-worker-tcp-comm-0-#25][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0,
bytesRcvd=3228052543, bytesSent=3293205484, bytesRcvd0=0, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0,
igniteInstanceName=null, finished=false, hashCode=1267030710,
interrupted=false, runner=grid-nio-worker-tcp-comm-0-#25]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=6163 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=1672992, resendCnt=0,
rcvCnt=1673192, sentCnt=1673110, reserved=true, lastAck=1673184,
nodeLeft=false, node=TcpDiscoveryNode
[id=839c980a-5d38-416c-9b82-32ce9dfefc4c, addrs=[0:0:0:0:0:0:0:1%lo,
10.62.18.32, 127.0.0.1, 172.16.0.11], sockAddrs=[/0:0:0:0:0:0:0:1%lo:0,
/127.0.0.1:0, /172.16.0.11:0, /10.62.18.32:0], discPort=0, order=7,
intOrder=6, lastExchangeTime=1543950304173, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true], connected=true,
connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=1672992, resendCnt=0,
rcvCnt=1673192, sentCnt=1673110, reserved=true, lastAck=1673184,
nodeLeft=false, node=TcpDiscoveryNode
[id=839c980a-5d38-416c-9b82-32ce9dfefc4c, addrs=[0:0:0:0:0:0:0:1%lo,
10.62.18.32, 127.0.0.1, 172.16.0.11], sockAddrs=[/0:0:0:0:0:0:0:1%lo:0,
/127.0.0.1:0, /172.16.0.11:0, /10.62.18.32:0], discPort=0, order=7,
intOrder=6, lastExchangeTime=1543950304173, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true], connected=true,
connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false],
super=GridNioSessionImpl [locAddr=/172.16.0.4:47100,
rmtAddr=/172.16.0.11:34656, createTime=1543950304452, closeTime=0,
bytesSent=608945515, bytesRcvd=686391430, bytesSent0=0, bytesRcvd0=0,
sndSchedTime=1543950304452, lastSndTime=1543950400759,
lastRcvTime=1543950405400, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5c6f7548, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]]
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1649)
at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1306)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2342)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
[19:07:41,332][SEVERE][tcp-disco-msg-worker-#2][TcpDiscoverySpi]
TcpDiscoverSpi's message worker thread failed abnormally. Stopping the node
in order to prevent cluster wide instability.
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.socketStream(TcpDiscoverySpi.java:1424)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.sendMessageAcrossRing(ServerImpl.java:2962)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMetricsUpdateMessage(ServerImpl.java:5154)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2756)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2536)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6775)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2621)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
^[[A^[[A^[[5~ at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1649)
at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1306)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2342)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
can i know what could be the issue in ignite side?
Also is my proportion looks correct compared to no of operations?
recordcount=4000000
operationcount=4000000 --> 4 million operations
insertproportion=0.23 - ---> 23% insert
updateproportion=0.46 - ---> 46% update
readproportion=0.31 - ---> 31% read
threadcount=180
target=180000
Please clarify.
Thanks,
...summa
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
