Ilya Suntsov created IGNITE-7212: ------------------------------------ Summary: Load stoped after server node kill Key: IGNITE-7212 URL: https://issues.apache.org/jira/browse/IGNITE-7212 Project: Ignite Issue Type: Bug Components: general Affects Versions: 2.4 Reporter: Ilya Suntsov Priority: Critical
Scenario: * Start 4 servers * Start load tasks on 5 clients * Kill 1 server * Waiting for rebalancing * Kill 1 server Result: After the kill of second servers node load stoped. In servers logs I see messages like this: {noformat} [2017-12-15 11:23:50][DEBUG][grid-nio-worker-tcp-comm-0-#41] Remote client closed connection: GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=130952565, bytesSent=131203245, bytesRcvd0=3069200, bytesSent0=3068083, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=null, finished=false, hashCode=1748650517, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#41]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=1024, resendCnt=0, rcvCnt=1026, sentCnt=1029, reserved=true, lastAck=1024, nodeLeft=false, node=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=1024, resendCnt=0, rcvCnt=1026, sentCnt=1029, reserved=true, lastAck=1024, nodeLeft=false, node=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/172.31.23.220:41732, rmtAddr=ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47100, createTime=1513335774008, closeTime=0, bytesSent=131203245, bytesRcvd=130952565, bytesSent0=3068083, bytesRcvd0=3069200, sndSchedTime=1513335774008, lastSndTime=1513337029027, lastRcvTime=1513337029027, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@11ae7d3b, directMode=true], GridConnectionBytesVerifyFilter], accepted=false]] [2017-12-15 11:23:50][WARN ][tcp-disco-msg-worker-#2] Failed to send message to next node [msg=TcpDiscoveryConnectionCheckMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=6c7f6d95061-c3cf9fe4-ab13-4d95-ace3-84a54cd73e08, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false], errMsg=Failed to send message to next node [msg=TcpDiscoveryConnectionCheckMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=6c7f6d95061-c3cf9fe4-ab13-4d95-ace3-84a54cd73e08, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=ClusterNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, order=1, addr=[127.0.0.1, 172.31.20.3], daemon=false]]] [2017-12-15 11:23:50][DEBUG][grid-nio-worker-tcp-comm-0-#41] Session was closed but there are unacknowledged messages, will try to reconnect [rmtNode=b7cfaa4e-b3b7-4485-a421-c731d9ed869d] [2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Recovery reconnect [rmtNode=b7cfaa4e-b3b7-4485-a421-c731d9ed869d] [2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Creating NIO client to node: TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false] [2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Addresses resolved from attributes [rmtNode=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47100, /127.0.0.1:47100], isRmtAddrsExist=true] [2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Client creation failed [addr=ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47100, err=java.net.ConnectException: Connection refused] [2017-12-15 11:23:50][WARN ][tcp-comm-worker-#1] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47100, failureDetectionTimeout=10000] [2017-12-15 11:23:50][WARN ][disco-event-worker-#61] Node FAILED: TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false] [2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Skipping local address [addr=/127.0.0.1:47100, locAddrs=[172.31.20.3, 127.0.0.1], node=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false]] [2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Skipping local address [addr=/127.0.0.1:47100, locAddrs=[172.31.20.3, 127.0.0.1], node=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false]] {noformat} Logs and configs was attached to this ticket. -- This message was sent by Atlassian JIRA (v6.4.14#64029)