Hello! > java.lang.OutOfMemoryError
Have you tried increasing heap size of your process? Or trying to figure out why you end up using so much heap. Regards, -- Ilya Kasnacheev ср, 5 дек. 2018 г. в 05:40, summasumma <[email protected]>: > Hi, > I am trying following workload from ycsb with 2 node ignite cluster with 1 > backup enabled. writesync=prim_sync. > > recordcount=4000000 > operationcount=4000000 > workload=com.yahoo.ycsb.workloads.CoreWorkload > readallfields=true > > insertproportion=0.23 > updateproportion=0.46 > readproportion=0.31 > > requestdistribution=zipfian > threadcount=180 > target=180000 > > But in few seconds one of the ignite node stops with following error: > [19:07:36,283][SEVERE][grid-nio-worker-tcp-comm-3-#28][TcpCommunicationSpi] > Failed to process selector key [ses=GridSelectorNioSessionImpl > [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=3, > bytesRcvd=2843314842, bytesSent=487323705, bytesRcvd0=0, bytesSent0=0, > select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-3, > igniteInstanceName=null, finished=false, hashCode=170320615, > interrupted=false, runner=grid-nio-worker-tcp-comm-3-#28]]], > writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], > readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], > inRecovery=GridNioRecoveryDescriptor [acked=2885472, resendCnt=0, > rcvCnt=2767636, sentCnt=2885548, reserved=true, lastAck=2767616, > nodeLeft=false, node=TcpDiscoveryNode > [id=cbc18e72-429a-44ef-84eb-df2e8a805866, addrs=[172.16.0.17], > sockAddrs=[/172.16.0.17:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1543950039896, loc=false, > ver=2.6.0#20180710-sha1:669feacc, > isClient=false], connected=false, connectCnt=1, queueLimit=4096, > reserveCnt=1, pairedConnections=false], > outRecovery=GridNioRecoveryDescriptor [acked=2885472, resendCnt=0, > rcvCnt=2767636, sentCnt=2885548, reserved=true, lastAck=2767616, > nodeLeft=false, node=TcpDiscoveryNode > [id=cbc18e72-429a-44ef-84eb-df2e8a805866, addrs=[172.16.0.17], > sockAddrs=[/172.16.0.17:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1543950039896, loc=false, > ver=2.6.0#20180710-sha1:669feacc, > isClient=false], connected=false, connectCnt=1, queueLimit=4096, > reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl > [locAddr=/172.16.0.4:41328, rmtAddr=/172.16.0.17:47100, > createTime=1543950040183, closeTime=0, bytesSent=3480594647, > bytesRcvd=3339280186, bytesSent0=0, bytesRcvd0=0, > sndSchedTime=1543950040183, lastSndTime=1543950373693, > lastRcvTime=1543950373693, readsPaused=false, > filterChain=FilterChain[filters=[GridNioCodecFilter > [parser=o.a.i.i.util.nio.GridDirectParser@5c6f7548, directMode=true], > GridConnectionBytesVerifyFilter], accepted=false]]] > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > at sun.nio.ch.IOUtil.write(IOUtil.java:51) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > at > > org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1649) > at > > org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1306) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2342) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > [19:07:36,872][SEVERE][grid-nio-worker-tcp-comm-0-#25][TcpCommunicationSpi] > Failed to process selector key [ses=GridSelectorNioSessionImpl > [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, > bytesRcvd=3228052543, bytesSent=3293205484, bytesRcvd0=0, bytesSent0=0, > select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, > igniteInstanceName=null, finished=false, hashCode=1267030710, > interrupted=false, runner=grid-nio-worker-tcp-comm-0-#25]]], > writeBuf=java.nio.DirectByteBuffer[pos=0 lim=6163 cap=32768], > readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], > inRecovery=GridNioRecoveryDescriptor [acked=1672992, resendCnt=0, > rcvCnt=1673192, sentCnt=1673110, reserved=true, lastAck=1673184, > nodeLeft=false, node=TcpDiscoveryNode > [id=839c980a-5d38-416c-9b82-32ce9dfefc4c, addrs=[0:0:0:0:0:0:0:1%lo, > 10.62.18.32, 127.0.0.1, 172.16.0.11], sockAddrs=[/0:0:0:0:0:0:0:1%lo:0, > /127.0.0.1:0, /172.16.0.11:0, /10.62.18.32:0], discPort=0, order=7, > intOrder=6, lastExchangeTime=1543950304173, loc=false, > ver=2.6.0#20180710-sha1:669feacc, isClient=true], connected=true, > connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], > outRecovery=GridNioRecoveryDescriptor [acked=1672992, resendCnt=0, > rcvCnt=1673192, sentCnt=1673110, reserved=true, lastAck=1673184, > nodeLeft=false, node=TcpDiscoveryNode > [id=839c980a-5d38-416c-9b82-32ce9dfefc4c, addrs=[0:0:0:0:0:0:0:1%lo, > 10.62.18.32, 127.0.0.1, 172.16.0.11], sockAddrs=[/0:0:0:0:0:0:0:1%lo:0, > /127.0.0.1:0, /172.16.0.11:0, /10.62.18.32:0], discPort=0, order=7, > intOrder=6, lastExchangeTime=1543950304173, loc=false, > ver=2.6.0#20180710-sha1:669feacc, isClient=true], connected=true, > connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], > super=GridNioSessionImpl [locAddr=/172.16.0.4:47100, > rmtAddr=/172.16.0.11:34656, createTime=1543950304452, closeTime=0, > bytesSent=608945515, bytesRcvd=686391430, bytesSent0=0, bytesRcvd0=0, > sndSchedTime=1543950304452, lastSndTime=1543950400759, > lastRcvTime=1543950405400, readsPaused=false, > filterChain=FilterChain[filters=[GridNioCodecFilter > [parser=o.a.i.i.util.nio.GridDirectParser@5c6f7548, directMode=true], > GridConnectionBytesVerifyFilter], accepted=true]]] > java.io.IOException: Broken pipe > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > at sun.nio.ch.IOUtil.write(IOUtil.java:51) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > at > > org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1649) > at > > org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1306) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2342) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > [19:07:41,332][SEVERE][tcp-disco-msg-worker-#2][TcpDiscoverySpi] > TcpDiscoverSpi's message worker thread failed abnormally. Stopping the node > in order to prevent cluster wide instability. > java.lang.OutOfMemoryError: Java heap space > at > java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76) > at > > org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.socketStream(TcpDiscoverySpi.java:1424) > at > > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.sendMessageAcrossRing(ServerImpl.java:2962) > at > > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMetricsUpdateMessage(ServerImpl.java:5154) > at > > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2756) > at > > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2536) > at > > org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6775) > at > > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2621) > at > org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > ^[[A^[[A^[[5~ at > > org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1649) > at > > org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1306) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2342) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > > > can i know what could be the issue in ignite side? > Also is my proportion looks correct compared to no of operations? > recordcount=4000000 > operationcount=4000000 --> 4 million operations > insertproportion=0.23 - ---> 23% insert > updateproportion=0.46 - ---> 46% update > readproportion=0.31 - ---> 31% read > threadcount=180 > target=180000 > > Please clarify. > > Thanks, > ...summa > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
