Hello! Looks like some network problems, such as forced network timeout in case of inactivity.
Regards, -- Ilya Kasnacheev чт, 6 сент. 2018 г. в 5:14, Jeff Jiao <[email protected]>: > Hi guys, > > we are using Ignite 2.3.0 > we have a Ignite cluster in production which has 4 server nodes, recently > we > found that Ignite throws "Connection reset by peer" occasionally after some > complex query, we know it is caused by connection closed while socket > reading/writing, but why it closed? > > can you observe any abnormal info from our log below? > Ignite server and client almost threw the exception at the same time, and > this exception occurs only 3 seconds after we issued the query. i see > bytesSent and Rcvd is very big, don't know whether it is related.. > > Thanks. > > > > Server log > > 2018-09-06_00:28:13.996 [ERROR] > [grid-nio-worker-tcp-comm-23-#192%PROD_IDEA_default_SZ_NewCluster%] > [o.a.i.s.c.tcp.TcpCommunicationSpi] Failed to process selec > tor key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker > [super=AbstractNioClientWorker [idx=23, bytesRcvd=10166800, > bytesSent=27235902413, bytesR > cvd0=730, bytesSent0=119379534, select=true, super=GridWorker > [name=grid-nio-worker-tcp-comm-23, > igniteInstanceName=PROD_IDEA_default_SZ_NewCluster, finished=false, > hashCode=1200501937, interrupted=false, > > runner=grid-nio-worker-tcp-comm-23-#192%PROD_IDEA_default_SZ_NewCluster%]]], > writeBuf=java.nio.DirectByteBuffer[pos=4786 lim=32768 cap=32768], > readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], > inRecovery=GridNioRecoveryDescriptor [acked=2784, resendCnt=0, rcvCnt=2261, > sentCnt=2805, reserved=true, lastAck=2240, nodeLeft=false, > node=TcpDiscoveryNode [id=9770d3e3-83e8-498c-a10b-c0bb991cbd60, > addrs=[10.42.223.207, 127.0.0.1], sockAddrs=[/10.42.223.207:0, > /127.0.0.1:0], discPort=0, order=335, intOrder=179, > lastExchangeTime=1536094829004, loc=false, > ver=2.3.0#20171028-sha1:8add7fd5, > isClient=true], connected=true, connectCnt=0, queueLimit=4096, > reserveCnt=1, > pairedConnections=false], outRecovery=GridNioRecoveryDescriptor > [acked=2784, > resendCnt=0, rcvCnt=2261, sentCnt=2805, reserved=true, lastAck=2240, > nodeLeft=false, node=TcpDiscoveryNode > [id=9770d3e3-83e8-498c-a10b-c0bb991cbd60, addrs=[10.42.223.207, 127.0.0.1], > sockAddrs=[/10.42.223.207:0, /127.0.0.1:0], discPort=0, order=335, > intOrder=179, lastExchangeTime=1536094829004, loc=false, > ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], connected=true, > connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], > super=GridNioSessionImpl [locAddr=/26.2.17.163:47100, > rmtAddr=/26.2.17.15:37836, createTime=1536094833590, closeTime=0, > bytesSent=5894311985, bytesRcvd=2541962, bytesSent0=119379534, > bytesRcvd0=730, sndSchedTime=1536163292987, lastSndTime=1536164892990, > lastRcvTime=1536164892980, readsPaused=false, > filterChain=FilterChain[filters=[GridNioCodecFilter > [parser=o.a.i.i.util.nio.GridDirectParser@40ad221b, directMode=true], > GridConnectionBytesVerifyFilter], accepted=true]]] > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:192) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) > at > > org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1233) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2272) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2048) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > 2018-09-06_00:28:13.997 [WARN ] > [grid-nio-worker-tcp-comm-23-#192%PROD_IDEA_default_SZ_NewCluster%] > [o.a.i.s.c.tcp.TcpCommunicationSpi] Closing NIO session bec > ause of unhandled exception [cls=class o.a.i.i.util.nio.GridNioException, > msg=Connection reset by peer] > > > > Client log > > 2018-09-06_00:28:13.995 [ERROR] > [grid-nio-worker-tcp-comm-5-#181%PROD_IDEA_default_SZ_NewCluster%] > [o.a.i.s.c.tcp.TcpCommunicationSpi] Failed to process select > or key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker > [super=AbstractNioClientWorker [idx=5, bytesRcvd=5830373027, > bytesSent=2485475, bytesRcvd0 > =233844279, bytesSent0=9333, select=true, super=GridWorker > [name=grid-nio-worker-tcp-comm-5, > igniteInstanceName=PROD_IDEA_default_SZ_NewCluster, finished=false > , hashCode=918657525, interrupted=false, > runner=grid-nio-worker-tcp-comm-5-#181%PROD_IDEA_default_SZ_NewCluster%]]], > writeBuf=java.nio.DirectByteBuffer[pos=0 l > im=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 > cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=2240, resendCnt=0, > rcvCnt=2804, > sentCnt=2261, reserved=true, lastAck=2784, nodeLeft=false, > node=TcpDiscoveryNode [id=ad141a60-e830-4e48-850a-ece6a8a45eb1, > addrs=[26.2.17.163], sockAddrs=[stsz > 030013/26.2.17.163:47500], discPort=47500, order=4, intOrder=4, > lastExchangeTime=1536094829041, loc=false, > ver=2.3.0#20171028-sha1:8add7fd5, > isClient=false], c > onnected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, > pairedConnections=false], outRecovery=GridNioRecoveryDescriptor > [acked=2240, > resendCnt=0, rcvCnt=2 > 804, sentCnt=2261, reserved=true, lastAck=2784, nodeLeft=false, > node=TcpDiscoveryNode [id=ad141a60-e830-4e48-850a-ece6a8a45eb1, > addrs=[26.2.17.163], sockAddrs= > [stsz030013/26.2.17.163:47500], discPort=47500, order=4, intOrder=4, > lastExchangeTime=1536094829041, loc=false, > ver=2.3.0#20171028-sha1:8add7fd5, > isClient=fals > e], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, > pairedConnections=false], super=GridNioSessionImpl > [locAddr=/10.42.223.207:37836, rmtAddr=sts > z030013/26.2.17.163:47100, createTime=1536094833584, closeTime=0, > bytesSent=2541920, bytesRcvd=5894286307, bytesSent0=9333, > bytesRcvd0=233844279, sndSchedTime= > 1536163834744, lastSndTime=1536164892989, lastRcvTime=1536164892989, > readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter > [parser=o.a.i.i.uti > l.nio.GridDirectParser@5df3dd31, directMode=true], > GridConnectionBytesVerifyFilter], accepted=false]]] > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:192) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) > at > > org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1233) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2272) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2048) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > 2018-09-06_00:28:13.996 [WARN ] > [grid-nio-worker-tcp-comm-5-#181%PROD_IDEA_default_SZ_NewCluster%] > [o.a.i.s.c.tcp.TcpCommunicationSpi] Closing NIO session beca > use of unhandled exception [cls=class o.a.i.i.util.nio.GridNioException, > msg=Connection reset by peer] > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
