Hi, I had submitted a spark job for ignite-yarn application on last 12th March, 2020 with 40 executors and 2 cores (running containers 41 and allocated CPU 81 on spark web UI page) and after running it for 7 days, Today on 19th March I have seen on spark web UI page that resources were went up by "running containers 247 and allocated CPU 493" and it's keep on increasing every one hour and so. How can I maintain the ignite-yarn application not to increase the resources of its own? Otherwise, it will a concern for the cluster if it keep on increasing. Below is the part of yarn log for my ignite-yarn job.
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. [08:58:25,564][SEVERE][grid-timeout-worker-#103%titan-resourceUP%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=tcp-comm-worker, blockedFor=17s] [08:58:25,565][SEVERE][grid-timeout-worker-#103%titan-resourceUP%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003488512]]] class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003488512] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:58:34,698][SEVERE][grid-timeout-worker-#103%titan-resourceUP%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=tcp-comm-worker, blockedFor=26s] [08:58:34,699][SEVERE][grid-timeout-worker-#103%titan-resourceUP%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003488512]]] class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003488512] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:58:43,805][SEVERE][nio-acceptor-tcp-comm-#128%titan-resourceUP%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=tcp-comm-worker, blockedFor=35s] [08:58:43,805][SEVERE][nio-acceptor-tcp-comm-#128%titan-resourceUP%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003488512]]] class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003488512] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2938) at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:58:56,406][SEVERE][ttl-cleanup-worker-#171%titan-resourceUP%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=tcp-comm-worker, blockedFor=47s] [08:58:56,407][SEVERE][ttl-cleanup-worker-#171%titan-resourceUP%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003488512]]] class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003488512] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:151) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:05,506][SEVERE][nio-acceptor-tcp-comm-#128%titan-resourceUP%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=tcp-comm-worker, blockedFor=56s] [08:59:05,506][SEVERE][nio-acceptor-tcp-comm-#128%titan-resourceUP%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003488512]]] class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003488512] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2938) at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,591][SEVERE][grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=9, bytesRcvd=4475652, bytesSent=1731122, bytesRcvd0=18, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-9, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003549587, hashCode=344426864, interrupted=false, runner=grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=166400 cap=166400], readBuf=java.nio.DirectByteBuffer[pos=18 lim=18 cap=117552], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/10.204.138.138:39970, rmtAddr=c250xmv.intqa.bigdata.int.thomsonreuters.com/10.204.138.66:45400, createTime=1584003549587, closeTime=0, bytesSent=0, bytesRcvd=18, bytesSent0=0, bytesRcvd0=18, sndSchedTime=1584003549587, lastSndTime=1584003549587, lastRcvTime=1584003549587, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=false, markedForClose=false]]] java.lang.NullPointerException at org.apache.ignite.internal.util.nio.GridNioServer.cancelConnect(GridNioServer.java:898) at org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.cancel(TcpCommunicationConnectionCheckFuture.java:338) at org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture.cancelFutures(TcpCommunicationConnectionCheckFuture.java:475) at org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture.receivedAddressStatus(TcpCommunicationConnectionCheckFuture.java:494) at org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture$1.onStatusReceived(TcpCommunicationConnectionCheckFuture.java:433) at org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.finish(TcpCommunicationConnectionCheckFuture.java:362) at org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.onConnected(TcpCommunicationConnectionCheckFuture.java:348) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:779) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:389) at org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279) at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:117) at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) at org.apache.ignite.internal.util.nio.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:88) at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3553) at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1303) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,591][SEVERE][grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%][TcpCommunicationSpi] Closing NIO session because of unhandled exception. class org.apache.ignite.internal.util.nio.GridNioException: null at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2415) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at org.apache.ignite.internal.util.nio.GridNioServer.cancelConnect(GridNioServer.java:898) at org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.cancel(TcpCommunicationConnectionCheckFuture.java:338) at org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture.cancelFutures(TcpCommunicationConnectionCheckFuture.java:475) at org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture.receivedAddressStatus(TcpCommunicationConnectionCheckFuture.java:494) at org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$MultipleAddressesConnectFuture$1.onStatusReceived(TcpCommunicationConnectionCheckFuture.java:433) at org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.finish(TcpCommunicationConnectionCheckFuture.java:362) at org.apache.ignite.spi.communication.tcp.internal.TcpCommunicationConnectionCheckFuture$SingleAddressConnectFuture.onConnected(TcpCommunicationConnectionCheckFuture.java:348) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:779) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:389) at org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279) at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:117) at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) at org.apache.ignite.internal.util.nio.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:88) at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3553) at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1303) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) ... 4 more [08:59:10,593][SEVERE][grid-nio-worker-tcp-comm-19-#123%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=19, bytesRcvd=4112614, bytesSent=5001528, bytesRcvd0=0, bytesSent0=18, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-19, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003549587, hashCode=1777632028, interrupted=false, runner=grid-nio-worker-tcp-comm-19-#123%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40456, createTime=1584003549587, closeTime=0, bytesSent=18, bytesRcvd=0, bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549587, lastSndTime=1584003549587, lastRcvTime=1584003549587, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,597][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=2, bytesRcvd=6049119, bytesSent=6778991, bytesRcvd0=18, bytesSent0=36, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-2, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003549587, hashCode=813216397, interrupted=false, runner=grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40512, createTime=1584003549587, closeTime=0, bytesSent=18, bytesRcvd=0, bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549587, lastSndTime=1584003549587, lastRcvTime=1584003549587, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,597][SEVERE][grid-nio-worker-tcp-comm-7-#111%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,599][SEVERE][grid-nio-worker-tcp-comm-6-#110%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=6, bytesRcvd=5589344, bytesSent=3069900, bytesRcvd0=54, bytesSent0=18, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-6, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003549597, hashCode=293379764, interrupted=false, runner=grid-nio-worker-tcp-comm-6-#110%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40540, createTime=1584003549597, closeTime=0, bytesSent=18, bytesRcvd=0, bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549597, lastSndTime=1584003549597, lastRcvTime=1584003549597, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-21-#125%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-3-#107%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-18-#122%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=18, bytesRcvd=3987619, bytesSent=1696258, bytesRcvd0=36, bytesSent0=54, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-18, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003549597, hashCode=1083458718, interrupted=false, runner=grid-nio-worker-tcp-comm-18-#122%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40552, createTime=1584003549597, closeTime=0, bytesSent=18, bytesRcvd=0, bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549597, lastSndTime=1584003549597, lastRcvTime=1584003549597, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-11-#115%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,600][SEVERE][grid-nio-worker-tcp-comm-13-#117%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,601][SEVERE][grid-nio-worker-tcp-comm-15-#119%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,601][SEVERE][grid-nio-worker-tcp-comm-17-#121%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,601][SEVERE][grid-nio-worker-tcp-comm-23-#127%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,602][SEVERE][grid-nio-worker-tcp-comm-1-#105%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,602][SEVERE][grid-nio-worker-tcp-comm-4-#108%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,602][SEVERE][grid-nio-worker-tcp-comm-8-#112%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,603][SEVERE][grid-nio-worker-tcp-comm-20-#124%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,608][SEVERE][grid-nio-worker-tcp-comm-0-#104%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:10,609][SEVERE][grid-nio-worker-tcp-comm-10-#114%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,592][SEVERE][grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,595][SEVERE][grid-nio-worker-tcp-comm-19-#123%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,598][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,598][SEVERE][grid-nio-worker-tcp-comm-7-#111%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,601][SEVERE][grid-nio-worker-tcp-comm-14-#118%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=14, bytesRcvd=3350068, bytesSent=8196, bytesRcvd0=72, bytesSent0=126, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-14, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003549617, hashCode=907545167, interrupted=false, runner=grid-nio-worker-tcp-comm-14-#118%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40716, createTime=1584003549607, closeTime=0, bytesSent=18, bytesRcvd=0, bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549607, lastSndTime=1584003549607, lastRcvTime=1584003549607, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,601][SEVERE][grid-nio-worker-tcp-comm-12-#116%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=12, bytesRcvd=3969821, bytesSent=3288859, bytesRcvd0=108, bytesSent0=108, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-12, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003549617, hashCode=1020912454, interrupted=false, runner=grid-nio-worker-tcp-comm-12-#116%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40748, createTime=1584003549607, closeTime=0, bytesSent=18, bytesRcvd=0, bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549607, lastSndTime=1584003549607, lastRcvTime=1584003549607, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,602][SEVERE][grid-nio-worker-tcp-comm-17-#121%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,602][SEVERE][grid-nio-worker-tcp-comm-11-#115%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,603][SEVERE][grid-nio-worker-tcp-comm-8-#112%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,604][SEVERE][grid-nio-worker-tcp-comm-5-#109%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=5, bytesRcvd=4011044, bytesSent=7399489, bytesRcvd0=72, bytesSent0=108, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-5, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003549607, hashCode=606372690, interrupted=false, runner=grid-nio-worker-tcp-comm-5-#109%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40648, createTime=1584003549597, closeTime=0, bytesSent=18, bytesRcvd=0, bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549597, lastSndTime=1584003549597, lastRcvTime=1584003549597, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,609][SEVERE][grid-nio-worker-tcp-comm-1-#105%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=1, bytesRcvd=2726130, bytesSent=6003920, bytesRcvd0=90, bytesSent0=72, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003550603, hashCode=1883663272, interrupted=false, runner=grid-nio-worker-tcp-comm-1-#105%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40664, createTime=1584003550593, closeTime=0, bytesSent=18, bytesRcvd=0, bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003550593, lastSndTime=1584003550593, lastRcvTime=1584003550593, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,609][SEVERE][grid-nio-worker-tcp-comm-0-#104%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=4890192, bytesSent=12608476, bytesRcvd0=54, bytesSent0=595, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003550603, hashCode=1994247289, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#104%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40640, createTime=1584003549597, closeTime=0, bytesSent=18, bytesRcvd=0, bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549597, lastSndTime=1584003549597, lastRcvTime=1584003549597, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:11,610][SEVERE][grid-nio-worker-tcp-comm-10-#114%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=10, bytesRcvd=4013857, bytesSent=3362419, bytesRcvd0=90, bytesSent0=72, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-10, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003550603, hashCode=1302629280, interrupted=false, runner=grid-nio-worker-tcp-comm-10-#114%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40496, createTime=1584003549587, closeTime=0, bytesSent=18, bytesRcvd=0, bytesSent0=18, bytesRcvd0=0, sndSchedTime=1584003549587, lastSndTime=1584003549587, lastRcvTime=1584003549587, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:12,593][SEVERE][grid-nio-worker-tcp-comm-9-#113%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:12,598][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:12,599][SEVERE][grid-nio-worker-tcp-comm-7-#111%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:12,602][SEVERE][grid-nio-worker-tcp-comm-15-#119%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=15, bytesRcvd=5571714, bytesSent=1717776, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-15, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003550623, hashCode=1293929915, interrupted=false, runner=grid-nio-worker-tcp-comm-15-#119%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40684, createTime=1584003550593, closeTime=0, bytesSent=18, bytesRcvd=0, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1584003550593, lastSndTime=1584003550593, lastRcvTime=1584003550593, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:12,602][SEVERE][grid-nio-worker-tcp-comm-18-#122%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=18, bytesRcvd=3987655, bytesSent=1696348, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-18, igniteInstanceName=titan-resourceUP, finished=false, heartbeatTs=1584003551589, hashCode=1083458718, interrupted=false, runner=grid-nio-worker-tcp-comm-18-#122%titan-resourceUP%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45400, rmtAddr=/127.0.0.1:40720, createTime=1584003550593, closeTime=0, bytesSent=18, bytesRcvd=0, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1584003550593, lastSndTime=1584003550593, lastRcvTime=1584003550593, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1be59704, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:13,598][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi] Failed to process selector key [ses=null] java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processConnect(GridNioServer.java:2764) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2383) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [08:59:41,932][SEVERE][grid-nio-worker-tcp-comm-2-#106%titan-resourceUP%][TcpCommunicationSpi] Failed to read data from remote connection (will wait for 2000ms). class org.apache.ignite.IgniteCheckedException: Failed to select events on selector. at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2185) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Caused by: java.nio.channels.ClosedChannelException at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:197) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2000) ... 3 more [13:49:50,352][SEVERE][grid-nio-worker-tcp-comm-0-#104%titan-resourceUP%][TcpCommunicationSpi] Failed to read data from remote connection (will wait for 2000ms). class org.apache.ignite.IgniteCheckedException: Failed to select events on selector. at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2185) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Caused by: java.nio.channels.ClosedChannelException at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:197) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2000) ... 3 more [14:45:35,550][SEVERE][grid-nio-worker-tcp-comm-1-#105%titan-resourceUP%][TcpCommunicationSpi] Failed to read data from remote connection (will wait for 2000ms). class org.apache.ignite.IgniteCheckedException: Failed to select events on selector. at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2185) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Caused by: java.nio.channels.ClosedChannelException at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:197) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2000) ... 3 more LogType:stdout Log Upload Time:Sat Mar 14 08:29:54 +0000 2020 LogLength:19390 Log Contents: [08:57:21] __________ ________________ [08:57:21] / _/ ___/ |/ / _/_ __/ __/ [08:57:21] _/ // (7 7 // / / / / _/ [08:57:21] /___/\___/_/|_/___/ /_/ /___/ [08:57:21] [08:57:21] ver. 2.7.0#20181130-sha1:256ae401 [08:57:21] 2018 Copyright(C) Apache Software Foundation [08:57:21] [08:57:21] Ignite documentation: http://ignite.apache.org [08:57:21] [08:57:21] Quiet mode. [08:57:21] ^-- Logging to file '/data/12/yarn/nm/usercache/bigdata-app-ecpdevbermuda-uncomparator/appcache/application_1564355610025_561993/container_e192_1564355610025_561993_01_000044/ignite/apache-ignite-2.7.0-bin/work/log/ignite-2ee49457.0.log' [08:57:21] ^-- Logging by 'JavaLogger [quiet=true, config=null]' [08:57:21] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat} [08:57:21] [08:57:21] OS: Linux 3.10.0-1062.9.1.el7.x86_64 amd64 [08:57:21] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [08:57:21] Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments. [08:57:21] Configured plugins: [08:57:21] ^-- None [08:57:21] [08:57:21] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]]] [08:57:21] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [08:57:22] Security status [authentication=off, tls/ssl=off] [08:57:22] Topology snapshot [ver=12, locNode=2ee49457, servers=12, clients=0, state=ACTIVE, CPUs=576, offheap=910.0GB, heap=180.0GB] [08:57:22] Topology snapshot [ver=13, locNode=2ee49457, servers=13, clients=0, state=ACTIVE, CPUs=624, offheap=980.0GB, heap=190.0GB] [08:57:22] Topology snapshot [ver=14, locNode=2ee49457, servers=14, clients=0, state=ACTIVE, CPUs=672, offheap=1100.0GB, heap=210.0GB] [08:57:23] Performance suggestions for grid 'titan-resourceUP' (fix if possible) [08:57:23] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true [08:57:23] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options) [08:57:23] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options) [08:57:23] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) [08:57:23] ^-- Speed up flushing of dirty pages by OS (alter vm.dirty_expire_centisecs parameter by setting to 500) [08:57:23] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [08:57:23] [08:57:23] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [08:57:23] Data Regions Configured: [08:57:23] ^-- default [initSize=256.0 MiB, maxSize=75.5 GiB, persistence=false] [08:57:23] [08:57:23] Ignite node started OK (id=2ee49457, instance name=titan-resourceUP) [08:57:23] Topology snapshot [ver=11, locNode=2ee49457, servers=11, clients=0, state=ACTIVE, CPUs=528, offheap=830.0GB, heap=160.0GB] [08:57:23] Topology snapshot [ver=15, locNode=2ee49457, servers=15, clients=0, state=ACTIVE, CPUs=712, offheap=1100.0GB, heap=220.0GB] [08:57:23] Topology snapshot [ver=16, locNode=2ee49457, servers=16, clients=0, state=ACTIVE, CPUs=752, offheap=1200.0GB, heap=230.0GB] [08:57:23] Topology snapshot [ver=17, locNode=2ee49457, servers=17, clients=0, state=ACTIVE, CPUs=792, offheap=1200.0GB, heap=250.0GB] [08:57:23] Topology snapshot [ver=18, locNode=2ee49457, servers=18, clients=0, state=ACTIVE, CPUs=832, offheap=1300.0GB, heap=260.0GB] [08:57:23] Topology snapshot [ver=19, locNode=2ee49457, servers=19, clients=0, state=ACTIVE, CPUs=872, offheap=1300.0GB, heap=280.0GB] [08:57:23] Topology snapshot [ver=20, locNode=2ee49457, servers=20, clients=0, state=ACTIVE, CPUs=912, offheap=1400.0GB, heap=290.0GB] [08:57:23] Topology snapshot [ver=21, locNode=2ee49457, servers=21, clients=0, state=ACTIVE, CPUs=952, offheap=1400.0GB, heap=310.0GB] [08:57:23] Topology snapshot [ver=22, locNode=2ee49457, servers=22, clients=0, state=ACTIVE, CPUs=992, offheap=1500.0GB, heap=320.0GB] [08:57:23] Topology snapshot [ver=23, locNode=2ee49457, servers=23, clients=0, state=ACTIVE, CPUs=1032, offheap=1500.0GB, heap=340.0GB] [08:57:23] Topology snapshot [ver=24, locNode=2ee49457, servers=24, clients=0, state=ACTIVE, CPUs=1072, offheap=1600.0GB, heap=350.0GB] [08:57:23] Topology snapshot [ver=25, locNode=2ee49457, servers=25, clients=0, state=ACTIVE, CPUs=1112, offheap=1600.0GB, heap=370.0GB] [08:57:24] Topology snapshot [ver=26, locNode=2ee49457, servers=26, clients=0, state=ACTIVE, CPUs=1152, offheap=1700.0GB, heap=380.0GB] [08:57:29] Topology snapshot [ver=27, locNode=2ee49457, servers=27, clients=0, state=ACTIVE, CPUs=1200, offheap=1700.0GB, heap=400.0GB] [08:57:30] Topology snapshot [ver=28, locNode=2ee49457, servers=28, clients=0, state=ACTIVE, CPUs=1248, offheap=1800.0GB, heap=410.0GB] [08:57:31] Topology snapshot [ver=29, locNode=2ee49457, servers=29, clients=0, state=ACTIVE, CPUs=1296, offheap=1900.0GB, heap=420.0GB] [08:57:31] Topology snapshot [ver=30, locNode=2ee49457, servers=29, clients=1, state=ACTIVE, CPUs=1296, offheap=1900.0GB, heap=460.0GB] [08:57:36] Topology snapshot [ver=31, locNode=2ee49457, servers=29, clients=2, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=480.0GB] [08:57:37] Topology snapshot [ver=32, locNode=2ee49457, servers=29, clients=3, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=490.0GB] [08:57:37] Topology snapshot [ver=33, locNode=2ee49457, servers=29, clients=4, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=510.0GB] [08:57:37] Topology snapshot [ver=34, locNode=2ee49457, servers=29, clients=5, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=520.0GB] [08:57:37] Topology snapshot [ver=35, locNode=2ee49457, servers=29, clients=6, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=540.0GB] [08:57:37] Topology snapshot [ver=36, locNode=2ee49457, servers=29, clients=7, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=550.0GB] [08:57:37] Topology snapshot [ver=37, locNode=2ee49457, servers=29, clients=8, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=570.0GB] [08:57:37] Topology snapshot [ver=38, locNode=2ee49457, servers=29, clients=9, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=580.0GB] [08:57:37] Topology snapshot [ver=39, locNode=2ee49457, servers=29, clients=10, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=600.0GB] [08:57:37] Topology snapshot [ver=40, locNode=2ee49457, servers=29, clients=11, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=610.0GB] [08:57:37] Topology snapshot [ver=41, locNode=2ee49457, servers=29, clients=12, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=630.0GB] [08:57:37] Topology snapshot [ver=42, locNode=2ee49457, servers=29, clients=13, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=640.0GB] [08:57:37] Topology snapshot [ver=43, locNode=2ee49457, servers=29, clients=14, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=660.0GB] [08:57:37] Topology snapshot [ver=44, locNode=2ee49457, servers=29, clients=15, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=670.0GB] [08:57:37] Topology snapshot [ver=45, locNode=2ee49457, servers=29, clients=16, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=690.0GB] [08:57:37] Topology snapshot [ver=46, locNode=2ee49457, servers=29, clients=17, state=ACTIVE, CPUs=1344, offheap=1900.0GB, heap=700.0GB] [08:57:37] Topology snapshot [ver=47, locNode=2ee49457, servers=29, clients=18, state=ACTIVE, CPUs=1384, offheap=1900.0GB, heap=720.0GB] [08:57:37] Topology snapshot [ver=48, locNode=2ee49457, servers=29, clients=19, state=ACTIVE, CPUs=1384, offheap=1900.0GB, heap=730.0GB] [08:57:37] Topology snapshot [ver=49, locNode=2ee49457, servers=29, clients=20, state=ACTIVE, CPUs=1424, offheap=1900.0GB, heap=750.0GB] [08:57:37] Topology snapshot [ver=50, locNode=2ee49457, servers=29, clients=21, state=ACTIVE, CPUs=1424, offheap=1900.0GB, heap=760.0GB] [08:57:37] Topology snapshot [ver=51, locNode=2ee49457, servers=29, clients=22, state=ACTIVE, CPUs=1464, offheap=1900.0GB, heap=780.0GB] [08:57:37] Topology snapshot [ver=52, locNode=2ee49457, servers=29, clients=23, state=ACTIVE, CPUs=1464, offheap=1900.0GB, heap=790.0GB] [08:57:37] Topology snapshot [ver=53, locNode=2ee49457, servers=29, clients=24, state=ACTIVE, CPUs=1504, offheap=1900.0GB, heap=810.0GB] [08:57:37] Topology snapshot [ver=54, locNode=2ee49457, servers=29, clients=25, state=ACTIVE, CPUs=1504, offheap=1900.0GB, heap=820.0GB] [08:57:37] Topology snapshot [ver=55, locNode=2ee49457, servers=29, clients=26, state=ACTIVE, CPUs=1544, offheap=1900.0GB, heap=840.0GB] [08:57:37] Topology snapshot [ver=56, locNode=2ee49457, servers=29, clients=27, state=ACTIVE, CPUs=1584, offheap=1900.0GB, heap=850.0GB] [08:57:37] Topology snapshot [ver=57, locNode=2ee49457, servers=29, clients=28, state=ACTIVE, CPUs=1584, offheap=1900.0GB, heap=870.0GB] [08:57:38] Topology snapshot [ver=58, locNode=2ee49457, servers=29, clients=29, state=ACTIVE, CPUs=1584, offheap=1900.0GB, heap=880.0GB] [08:57:38] Topology snapshot [ver=59, locNode=2ee49457, servers=29, clients=30, state=ACTIVE, CPUs=1584, offheap=1900.0GB, heap=900.0GB] [08:57:38] Topology snapshot [ver=60, locNode=2ee49457, servers=29, clients=31, state=ACTIVE, CPUs=1616, offheap=1900.0GB, heap=910.0GB] [08:57:38] Topology snapshot [ver=61, locNode=2ee49457, servers=29, clients=32, state=ACTIVE, CPUs=1656, offheap=1900.0GB, heap=930.0GB] [08:57:38] Topology snapshot [ver=62, locNode=2ee49457, servers=29, clients=33, state=ACTIVE, CPUs=1696, offheap=1900.0GB, heap=940.0GB] [08:57:38] Topology snapshot [ver=63, locNode=2ee49457, servers=29, clients=34, state=ACTIVE, CPUs=1696, offheap=1900.0GB, heap=960.0GB] [08:57:38] Topology snapshot [ver=64, locNode=2ee49457, servers=29, clients=35, state=ACTIVE, CPUs=1696, offheap=1900.0GB, heap=970.0GB] [08:57:38] Topology snapshot [ver=65, locNode=2ee49457, servers=29, clients=36, state=ACTIVE, CPUs=1696, offheap=1900.0GB, heap=990.0GB] [08:57:38] Topology snapshot [ver=66, locNode=2ee49457, servers=29, clients=37, state=ACTIVE, CPUs=1736, offheap=1900.0GB, heap=1000.0GB] [08:57:38] Topology snapshot [ver=67, locNode=2ee49457, servers=29, clients=38, state=ACTIVE, CPUs=1736, offheap=1900.0GB, heap=1000.0GB] [08:57:38] Topology snapshot [ver=68, locNode=2ee49457, servers=29, clients=39, state=ACTIVE, CPUs=1776, offheap=1900.0GB, heap=1000.0GB] [08:57:38] Topology snapshot [ver=69, locNode=2ee49457, servers=29, clients=40, state=ACTIVE, CPUs=1808, offheap=1900.0GB, heap=1000.0GB] [08:57:38] Topology snapshot [ver=70, locNode=2ee49457, servers=29, clients=41, state=ACTIVE, CPUs=1840, offheap=1900.0GB, heap=1100.0GB] [08:58:01] Topology snapshot [ver=71, locNode=2ee49457, servers=30, clients=41, state=ACTIVE, CPUs=1840, offheap=1900.0GB, heap=1100.0GB] [08:58:02] Topology snapshot [ver=72, locNode=2ee49457, servers=31, clients=41, state=ACTIVE, CPUs=1840, offheap=2000.0GB, heap=1100.0GB] [08:58:02] Topology snapshot [ver=73, locNode=2ee49457, servers=32, clients=41, state=ACTIVE, CPUs=1840, offheap=2000.0GB, heap=1100.0GB] [08:58:02] Topology snapshot [ver=74, locNode=2ee49457, servers=33, clients=41, state=ACTIVE, CPUs=1840, offheap=2100.0GB, heap=1100.0GB] [08:58:02] Topology snapshot [ver=75, locNode=2ee49457, servers=34, clients=41, state=ACTIVE, CPUs=1840, offheap=2100.0GB, heap=1100.0GB] [08:58:02] Topology snapshot [ver=76, locNode=2ee49457, servers=35, clients=41, state=ACTIVE, CPUs=1880, offheap=2200.0GB, heap=1100.0GB] [08:58:02] Topology snapshot [ver=77, locNode=2ee49457, servers=36, clients=41, state=ACTIVE, CPUs=1880, offheap=2200.0GB, heap=1200.0GB] [08:58:02] Topology snapshot [ver=78, locNode=2ee49457, servers=37, clients=41, state=ACTIVE, CPUs=1920, offheap=2300.0GB, heap=1200.0GB] [08:58:03] Topology snapshot [ver=79, locNode=2ee49457, servers=38, clients=41, state=ACTIVE, CPUs=1960, offheap=2300.0GB, heap=1200.0GB] [08:58:05] Topology snapshot [ver=80, locNode=2ee49457, servers=39, clients=41, state=ACTIVE, CPUs=2000, offheap=2400.0GB, heap=1200.0GB] [08:59:10] Topology snapshot [ver=81, locNode=2ee49457, servers=38, clients=41, state=ACTIVE, CPUs=1960, offheap=2300.0GB, heap=1200.0GB] [08:59:16] Topology snapshot [ver=82, locNode=2ee49457, servers=38, clients=40, state=ACTIVE, CPUs=1960, offheap=2300.0GB, heap=1200.0GB] [08:59:17] Topology snapshot [ver=83, locNode=2ee49457, servers=38, clients=39, state=ACTIVE, CPUs=1960, offheap=2300.0GB, heap=1100.0GB] [08:59:17] Topology snapshot [ver=84, locNode=2ee49457, servers=38, clients=38, state=ACTIVE, CPUs=1912, offheap=2300.0GB, heap=1100.0GB] [08:59:17] Topology snapshot [ver=85, locNode=2ee49457, servers=38, clients=37, state=ACTIVE, CPUs=1912, offheap=2300.0GB, heap=1100.0GB] [08:59:17] Topology snapshot [ver=86, locNode=2ee49457, servers=38, clients=36, state=ACTIVE, CPUs=1912, offheap=2300.0GB, heap=1100.0GB] [08:59:17] Topology snapshot [ver=87, locNode=2ee49457, servers=38, clients=35, state=ACTIVE, CPUs=1872, offheap=2300.0GB, heap=1100.0GB] [08:59:17] Topology snapshot [ver=88, locNode=2ee49457, servers=38, clients=34, state=ACTIVE, CPUs=1832, offheap=2300.0GB, heap=1100.0GB] [08:59:17] Topology snapshot [ver=89, locNode=2ee49457, servers=38, clients=33, state=ACTIVE, CPUs=1832, offheap=2300.0GB, heap=1100.0GB] [08:59:17] Topology snapshot [ver=90, locNode=2ee49457, servers=38, clients=32, state=ACTIVE, CPUs=1800, offheap=2300.0GB, heap=1000.0GB] [08:59:17] Topology snapshot [ver=91, locNode=2ee49457, servers=38, clients=31, state=ACTIVE, CPUs=1800, offheap=2300.0GB, heap=1000.0GB] [08:59:17] Topology snapshot [ver=92, locNode=2ee49457, servers=38, clients=30, state=ACTIVE, CPUs=1800, offheap=2300.0GB, heap=1000.0GB] [08:59:17] Topology snapshot [ver=93, locNode=2ee49457, servers=38, clients=29, state=ACTIVE, CPUs=1800, offheap=2300.0GB, heap=990.0GB] [08:59:17] Topology snapshot [ver=94, locNode=2ee49457, servers=38, clients=28, state=ACTIVE, CPUs=1760, offheap=2300.0GB, heap=980.0GB] [08:59:17] Topology snapshot [ver=95, locNode=2ee49457, servers=38, clients=27, state=ACTIVE, CPUs=1760, offheap=2300.0GB, heap=960.0GB] [08:59:17] Topology snapshot [ver=96, locNode=2ee49457, servers=38, clients=26, state=ACTIVE, CPUs=1760, offheap=2300.0GB, heap=950.0GB] [08:59:18] Topology snapshot [ver=97, locNode=2ee49457, servers=38, clients=25, state=ACTIVE, CPUs=1760, offheap=2300.0GB, heap=930.0GB] [08:59:18] Topology snapshot [ver=98, locNode=2ee49457, servers=38, clients=24, state=ACTIVE, CPUs=1720, offheap=2300.0GB, heap=920.0GB] [08:59:18] Topology snapshot [ver=99, locNode=2ee49457, servers=38, clients=23, state=ACTIVE, CPUs=1720, offheap=2300.0GB, heap=900.0GB] [08:59:18] Topology snapshot [ver=100, locNode=2ee49457, servers=38, clients=22, state=ACTIVE, CPUs=1720, offheap=2300.0GB, heap=890.0GB] [08:59:18] Topology snapshot [ver=101, locNode=2ee49457, servers=38, clients=21, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=870.0GB] [08:59:18] Topology snapshot [ver=102, locNode=2ee49457, servers=38, clients=20, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=860.0GB] [08:59:18] Topology snapshot [ver=103, locNode=2ee49457, servers=38, clients=19, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=840.0GB] [08:59:18] Topology snapshot [ver=104, locNode=2ee49457, servers=38, clients=18, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=830.0GB] [08:59:18] Topology snapshot [ver=105, locNode=2ee49457, servers=38, clients=17, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=810.0GB] [08:59:18] Topology snapshot [ver=106, locNode=2ee49457, servers=38, clients=16, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=800.0GB] [08:59:18] Topology snapshot [ver=107, locNode=2ee49457, servers=38, clients=15, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=780.0GB] [08:59:18] Topology snapshot [ver=108, locNode=2ee49457, servers=38, clients=14, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=770.0GB] [08:59:18] Topology snapshot [ver=109, locNode=2ee49457, servers=38, clients=13, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=750.0GB] [08:59:18] Topology snapshot [ver=110, locNode=2ee49457, servers=38, clients=12, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=740.0GB] [08:59:18] Topology snapshot [ver=111, locNode=2ee49457, servers=38, clients=11, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=720.0GB] [08:59:18] Topology snapshot [ver=112, locNode=2ee49457, servers=38, clients=10, state=ACTIVE, CPUs=1688, offheap=2300.0GB, heap=710.0GB] [08:59:18] Topology snapshot [ver=113, locNode=2ee49457, servers=38, clients=9, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=690.0GB] [08:59:18] Topology snapshot [ver=114, locNode=2ee49457, servers=38, clients=8, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=680.0GB] [08:59:18] Topology snapshot [ver=115, locNode=2ee49457, servers=38, clients=7, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=660.0GB] [08:59:18] Topology snapshot [ver=116, locNode=2ee49457, servers=38, clients=6, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=650.0GB] [08:59:18] Topology snapshot [ver=117, locNode=2ee49457, servers=38, clients=5, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=630.0GB] [08:59:18] Topology snapshot [ver=118, locNode=2ee49457, servers=38, clients=4, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=620.0GB] [08:59:18] Topology snapshot [ver=119, locNode=2ee49457, servers=38, clients=3, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=600.0GB] [08:59:18] Topology snapshot [ver=120, locNode=2ee49457, servers=38, clients=2, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=590.0GB] [08:59:18] Topology snapshot [ver=121, locNode=2ee49457, servers=38, clients=1, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=570.0GB] [08:59:18] Topology snapshot [ver=122, locNode=2ee49457, servers=38, clients=0, state=ACTIVE, CPUs=1656, offheap=2300.0GB, heap=560.0GB] [08:59:19] Topology snapshot [ver=123, locNode=2ee49457, servers=37, clients=0, state=ACTIVE, CPUs=1616, offheap=2300.0GB, heap=540.0GB] [08:59:20] Topology snapshot [ver=124, locNode=2ee49457, servers=38, clients=0, state=ACTIVE, CPUs=1616, offheap=2300.0GB, heap=560.0GB] [08:59:35] Topology snapshot [ver=125, locNode=2ee49457, servers=39, clients=0, state=ACTIVE, CPUs=1664, offheap=2400.0GB, heap=570.0GB] [08:59:52] Topology snapshot [ver=126, locNode=2ee49457, servers=40, clients=0, state=ACTIVE, CPUs=1704, offheap=2500.0GB, heap=590.0GB] [13:32:21] Topology snapshot [ver=127, locNode=2ee49457, servers=39, clients=0, state=ACTIVE, CPUs=1656, offheap=2400.0GB, heap=570.0GB] [13:32:25] Topology snapshot [ver=128, locNode=2ee49457, servers=40, clients=0, state=ACTIVE, CPUs=1656, offheap=2400.0GB, heap=590.0GB] [13:49:48] Topology snapshot [ver=129, locNode=2ee49457, servers=39, clients=0, state=ACTIVE, CPUs=1616, offheap=2400.0GB, heap=570.0GB] [13:50:03] Topology snapshot [ver=130, locNode=2ee49457, servers=40, clients=0, state=ACTIVE, CPUs=1664, offheap=2500.0GB, heap=590.0GB] [14:45:29] Topology snapshot [ver=131, locNode=2ee49457, servers=39, clients=0, state=ACTIVE, CPUs=1664, offheap=2400.0GB, heap=570.0GB] [14:45:46] Topology snapshot [ver=132, locNode=2ee49457, servers=40, clients=0, state=ACTIVE, CPUs=1704, offheap=2500.0GB, heap=590.0GB] Container: container_e192_1564355610025_561993_01_008768 on c010gst.intqa.bigdata.int.thomsonreuters.com_8041 =============================================================================================================== LogType:stderr Log Upload Time:Thu Mar 19 13:32:36 +0000 2020 LogLength:11894 Log Contents: log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. [08:32:34,822][SEVERE][sys-#148%titan-resourceUP%][TcpCommunicationSpi] Failed to send message to remote node [node=ZookeeperClusterNode [id=00d6c796-5bce-404e-867f-3dd67e7c88f6, addrs=[10.204.138.136, 127.0.0.1], order=258, loc=false, client=false], msg=GridIoMessage [plc=2, topic=T6 [topic=TOPIC_CACHE, id1=83e8ca36-2305-3266-8e65-1463be879baa, id2=0], topicOrd=-1, ordered=true, timeout=10000, skipOnTimeout=false, msg=GridDhtPartitionDemandMessage [rebalanceId=41922, parts=IgniteDhtDemandedPartitionsMap [historical=null], timeout=10000, workerId=-1, topVer=AffinityTopologyVersion [topVer=323, minorTopVer=0], partCnt=0, super=GridCacheGroupIdMessage [grpId=-1569773056]]]] class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to send message (node left topology): ZookeeperClusterNode [id=00d6c796-5bce-404e-867f-3dd67e7c88f6, addrs=[10.204.138.136, 127.0.0.1], order=258, loc=false, client=false] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3270) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2713) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2672) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1656) at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1766) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendOrderedMessage(GridCacheIoManager.java:1231) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.handleSupplyMessage(GridDhtPartitionDemander.java:845) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.handleSupplyMessage(GridDhtPreloader.java:387) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:418) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:408) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$700(GridCacheIoManager.java:101) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1613) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4100(GridIoManager.java:127) at org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2768) at org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1529) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4400(GridIoManager.java:127) at org.apache.ignite.internal.managers.communication.GridIoManager$10.run(GridIoManager.java:1498) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [08:32:35,003][SEVERE][exchange-worker-#123%titan-resourceUP%][TcpCommunicationSpi] Failed to send message to remote node [node=ZookeeperClusterNode [id=f8377bf6-7e5e-4efd-bd31-95af11edc52a, addrs=[10.204.138.219, 127.0.0.1], order=130, loc=false, client=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtPartitionsSingleMessage [parts={-2100569601=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=326, minorTopVer=0], updateSeq=440, size=100], -2085876587=GridDhtPartitionMap [moving=5, top=AffinityTopologyVersion [topVer=326, minorTopVer=0], updateSeq=495, size=75], -1569773056=GridDhtPartitionMap [moving=4, top=AffinityTopologyVersion [topVer=326, minorTopVer=0], updateSeq=501, size=75]}, partCntrs={-2100569601=CachePartitionPartialCountersMap {}, -2085876587=CachePartitionPartialCountersMap {7=(0,5825), 53=(0,5709), 60=(0,5911), 79=(0,5806), 96=(0,5754), 100=(0,5678), 102=(0,5803), 114=(0,5856), 127=(0,5695), 167=(0,5766), 179=(0,5800), 204=(0,5869), 205=(0,5761), 208=(0,5792), 212=(0,5763), 221=(0,5778), 228=(0,5797), 235=(0,5818), 239=(0,5728), 245=(0,5726), 249=(0,5754), 250=(0,5819), 253=(0,5745), 262=(0,5719), 286=(0,5728), 287=(0,5783), 294=(0,5713), 307=(0,5760), 312=(0,5702), 318=(0,5738), 323=(0,5782), 335=(0,5824), 351=(0,5826), 353=(0,5725), 367=(0,5725), 379=(0,5856), 385=(0,5801), 391=(0,5805), 402=(0,5886), 411=(0,5770), 417=(0,5764), 432=(0,5792), 442=(0,5749), 449=(0,5679), 452=(0,5722), 456=(0,5828), 470=(0,5689), 485=(0,5732), 494=(0,5837), 504=(0,5809), 505=(0,5679), 516=(0,5786), 528=(0,5793), 530=(0,5699), 557=(0,5746), 580=(0,5680), 594=(0,5830), 603=(0,5758), 608=(0,5876), 624=(0,5715), 626=(0,5743), 636=(0,5795), 674=(0,5751), 676=(0,5875), 725=(0,5780), 729=(0,5795), 741=(0,5782), 759=(0,5765), 762=(0,5728), 767=(0,5697), 777=(0,5762), 798=(0,5774), 829=(0,5705), 849=(0,5876), 853=(0,5854), 854=(0,5834), 857=(0,5660), 863=(0,5742), 864=(0,5824), 867=(0,5757), 877=(0,5680), 883=(0,5758), 905=(0,5911), 917=(0,5630), 920=(0,5832), 924=(0,5580), 939=(0,5805), 940=(0,5644), 951=(0,5726), 955=(0,5750), 963=(0,5699), 965=(0,5731), 971=(0,5707), 985=(0,5735), 1017=(0,5729)}, -1569773056=CachePartitionPartialCountersMap {7=(0,6915), 53=(0,6912), 60=(0,6825), 79=(0,6962), 96=(0,7046), 100=(0,6926), 102=(0,6813), 114=(0,6850), 127=(0,6926), 167=(0,6816), 179=(0,6795), 204=(0,6894), 205=(0,6950), 208=(0,6742), 212=(0,6939), 221=(0,6933), 228=(0,6932), 235=(0,6903), 239=(0,6816), 245=(0,6928), 249=(0,6895), 250=(0,6892), 253=(0,6934), 262=(0,6927), 286=(0,6906), 287=(0,7015), 294=(0,6781), 307=(0,6899), 312=(0,6773), 318=(0,6944), 323=(0,6999), 335=(0,6925), 351=(0,6885), 353=(0,6797), 367=(0,7074), 379=(0,7041), 385=(0,6921), 391=(0,7101), 402=(0,6858), 411=(0,7057), 417=(0,6862), 432=(0,6752), 442=(0,6911), 449=(0,6939), 452=(0,6834), 456=(0,6792), 470=(0,6936), 485=(0,6903), 494=(0,6955), 504=(0,6943), 505=(0,6893), 516=(0,6838), 528=(0,6922), 530=(0,6893), 557=(0,6849), 580=(0,6822), 594=(0,6994), 603=(0,6872), 608=(0,6862), 624=(0,6987), 626=(0,6913), 636=(0,7033), 674=(0,6921), 676=(0,6967), 725=(0,6831), 729=(0,6854), 741=(0,6853), 759=(0,6871), 762=(0,6873), 767=(0,6700), 777=(0,6765), 798=(0,6712), 829=(0,6933), 849=(0,6936), 853=(0,6967), 854=(0,6995), 857=(0,6908), 863=(0,6987), 864=(0,6872), 867=(0,6847), 877=(0,7032), 883=(0,6830), 905=(0,6790), 917=(0,6953), 920=(0,6941), 924=(0,6819), 939=(0,6859), 940=(0,6814), 951=(0,6879), 955=(0,6915), 963=(0,7002), 965=(0,6858), 971=(0,6919), 985=(0,6746), 1017=(0,6700)}}, partsSizes={-2085876587={385=5801, 262=5719, 7=5825, 391=5805, 777=5762, 905=5911, 528=5793, 402=5886, 530=5699, 920=5832, 286=5728, 798=5774, 287=5783, 294=5713, 940=5644, 557=5746, 432=5792, 179=5800, 53=5709, 312=5702, 442=5749, 955=5750, 60=5911, 318=5738, 449=5679, 323=5782, 580=5680, 456=5828, 971=5707, 205=5761, 79=5806, 335=5824, 208=5792, 594=5830, 725=5780, 853=5854, 729=5795, 857=5660, 985=5735, 603=5758, 221=5778, 351=5826, 863=5742, 96=5754, 864=5824, 353=5725, 867=5757, 100=5678, 228=5797, 485=5732, 102=5803, 235=5818, 877=5680, 494=5837, 367=5725, 624=5715, 883=5758, 245=5726, 759=5765, 504=5809, 249=5754, 505=5679, 379=5856, 636=5795, 253=5745, 127=5695, 767=5697}, -1569773056={385=6921, 262=6927, 7=6915, 391=7101, 777=6765, 905=6790, 528=6922, 402=6858, 530=6893, 920=6941, 286=6906, 798=6712, 287=7015, 294=6781, 167=6816, 940=6814, 557=6849, 432=6752, 179=6795, 53=6912, 312=6773, 442=6911, 955=6915, 60=6825, 318=6944, 449=6939, 323=6999, 580=6822, 456=6792, 971=6919, 205=6950, 79=6962, 335=6925, 208=6742, 594=6994, 725=6831, 853=6967, 854=5329, 729=6854, 857=6908, 985=6746, 603=6872, 221=6933, 351=6885, 863=6987, 96=7046, 864=6872, 353=6797, 867=6847, 100=6926, 228=6932, 485=6903, 102=6813, 235=6903, 877=7032, 494=6955, 367=7074, 624=6987, 883=6830, 245=6928, 759=6871, 504=6943, 249=6895, 505=6893, 379=7041, 636=7033, 253=6934, 127=6926, 767=6700}}, partHistCntrs=null, err=null, client=false, compress=true, finishMsg=null, activeQryTrackers=null, super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=327, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode [id=00d6c796-5bce-404e-867f-3dd67e7c88f6, addrs=[10.204.138.136, 127.0.0.1], order=258, loc=false, client=false], topVer=327, nodeId8=654c8cd2, msg=Node failed: ZookeeperClusterNode [id=00d6c796-5bce-404e-867f-3dd67e7c88f6, addrs=[10.204.138.136, 127.0.0.1], order=258, loc=false, client=false], type=NODE_FAILED, tstamp=1584624754811], nodeId=00d6c796, evt=NODE_FAILED], lastVer=GridCacheVersion [topVer=195483760, order=1584599672955, nodeOrder=274], super=GridCacheMessage [msgId=1510, depInfo=null, err=null, skipPrepare=false]]]]] class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to send message (node left topology): ZookeeperClusterNode [id=f8377bf6-7e5e-4efd-bd31-95af11edc52a, addrs=[10.204.138.219, 127.0.0.1], order=130, loc=false, client=false] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3270) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2713) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2672) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1656) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1731) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1170) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendLocalPartitions(GridDhtPartitionsExchangeFuture.java:1880) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendPartitions(GridDhtPartitionsExchangeFuture.java:2011) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1501) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:806) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2667) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2539) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) How to keep the ignite-yarn application steady without increasing the resources of its own? Thanks, Chandan -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
