Hi, We have been running ignite v2.5.0 snapshot (2018/5/11) fine for about two weeks. But suddenly, the grid stopped responding with massive infinite log spams, / 2018/05/24 18:56:14.909 [INFO ][grid-nio-worker-tcp-comm-5-#46][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/10.0.0.38:47102, rmtAddr=/10.0.0.29:53819] 2018/05/24 18:56:14.910 [WARN ][grid-nio-worker-tcp-comm-5-#46][TcpCommunicationSpi] Close incoming connection, unknown node [nodeId=2f4ed0a7-cf1f-4ad1-a6eb-e4171985eb97, ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=5, bytesRcvd=39218524780, bytesSent=34572936118, bytesRcvd0=159844, bytesSent0=273948, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-5, igniteInstanceName=null, finished=false, hashCode=1499496401, interrupted=false, runner=grid-nio-worker-tcp-comm-5-#46]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=42 lim=42 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/10.0.0.38:47102, rmtAddr=/10.0.0.29:53819, createTime=1527188174903, closeTime=0, bytesSent=18, bytesRcvd=42, bytesSent0=18, bytesRcvd0=42, sndSchedTime=1527188174903, lastSndTime=1527188174903, lastRcvTime=1527188174903, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@60af162b, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]]] 2018/05/24 18:56:15.006 [WARN ][sys-stripe-14-#15][GridDhtPartitionTopologyImpl] Requested topology version does not match calculated diff, will require full iteration tocalculate mapping [grp=xxxx, topVer=AffinityTopologyVersion [topVer=121, minorTopVer=0], diffVer=AffinityTopologyVersion [topVer=138, minorTopVer=0]]/
It appears that ignite server in node04 was causing it. So after killed the specific ignite server instance, the grid stopped spamming errors. Please see attached ignite grid configuration and logs. Can you please provide any tips or information to track down what could trigger this problem and fix it ? *Jvm options* "-Duser.timezone=UTC", "-DIGNITE_QUIET=false", "-Djava.net.preferIPv4Stack=true", "-Djava.awt.headless=true", "-Xms10g", "-Xmx10g", "-XX:+AlwaysPreTouch", "-XX:+UseG1GC", "-XX:+ScavengeBeforeFullGC", "-XX:+DisableExplicitGC" *Ignite configs* Topology snapshot [ver=120, servers=40, clients=80, CPUs=640, offheap=480.0GB, heap=560.0GB] Default_Region [initSize=2.0 GiB, maxSize=4.0 GiB, persistenceEnabled=false] ignite-config.xml <http://apache-ignite-users.70518.x6.nabble.com/file/t1784/ignite-config.xml> ignite-cache-config.txt <http://apache-ignite-users.70518.x6.nabble.com/file/t1784/ignite-cache-config.txt> *Logs* ignite-startup-logs.txt <http://apache-ignite-users.70518.x6.nabble.com/file/t1784/ignite-startup-logs.txt> ignite-node04.log <http://apache-ignite-users.70518.x6.nabble.com/file/t1784/ignite-node04.log> ignite-node18.log <http://apache-ignite-users.70518.x6.nabble.com/file/t1784/ignite-node18.log> Thanks Sam -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
