Hello! Please provide complete log files from both nodes. These errors may be a sign of node leaving some time ago.
Regards, -- Ilya Kasnacheev чт, 17 дек. 2020 г. в 19:43, rakshita04 <[email protected]>: > Hi Teams, > > I am using apache-ignite 2.8.0 version for our C++ applications. > We are using "TcpDiscoveryVmIpFinder" for connecting our 2 ignite nodes, > running on 2 separate machines. > But while running the ignite node are getting below error on one of the > nodes and application and node is stopping- > "Failed to send message to remote node" > Below are the full logs for node 1- > [14:36:32,572][SEVERE][exchange-worker-#46][TcpCommunicationSpi] Failed to > send message to remote node [node=TcpDiscoveryNode > [id=7f683a9a-afcc-4d20-95f4-855147d759fd, consistentId=DSU_A, > addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.119, 127.0.0.1, > 192.168.10.201, 192.168.100.100], sockAddrs=HashSet [/ > 192.168.100.100:47500, > /0:0:0:0:0:0:0:1%lo:47500, /192.168.10.201:47500, /127.0.0.1:47500, > /10.100.126.119:47500], discPort=47500, order=4, intOrder=3, > lastExchangeTime=1608212160446, loc=false, > ver=2.8.0#20200226-sha1:341b01df, > isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, > msg=GridDhtPartitionsSingleMessage [parts=HashMap > {-2100569601=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion > [topVer=-1, minorTopVer=0], updateSeq=2, size=0], > -510489548=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion > [topVer=-1, minorTopVer=0], updateSeq=2, size=0]}, partCntrs=HashMap > {-2100569601=CachePartitionPartialCountersMap {}, > -510489548=CachePartitionPartialCountersMap {}}, partsSizes=null, > partHistCntrs=null, err=null, client=false, exchangeStartTime=92638170010, > finishMsg=null, super=GridDhtPartitionsAbstractMessage > [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion > [topVer=6, minorTopVer=0], discoEvt=DiscoveryEvent > [evtNode=TcpDiscoveryNode > [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, > addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, > 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/ > 192.168.100.101:47500, > /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, > /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, > lastExchangeTime=1608212191947, loc=true, ver=2.8.0#20200226-sha1:341b01df, > isClient=false], topVer=6, nodeId8=0c98991e, msg=null, type=NODE_JOINED, > tstamp=1608212161928], nodeId=0c98991e, evt=NODE_JOINED], > lastVer=GridCacheVersion [topVer=0, order=1608212143633, nodeOrder=0], > super=GridCacheMessage [msgId=1, depInfo=null, > lastAffChangedTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], > err=null, skipPrepare=false]]]]] > class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: > Failed to send message (node left topology): TcpDiscoveryNode > [id=7f683a9a-afcc-4d20-95f4-855147d759fd, consistentId=DSU_A, > addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.119, 127.0.0.1, > 192.168.10.201, 192.168.100.100], sockAddrs=HashSet [/ > 192.168.100.100:47500, > /0:0:0:0:0:0:0:1%lo:47500, /192.168.10.201:47500, /127.0.0.1:47500, > /10.100.126.119:47500], discPort=47500, order=4, intOrder=3, > lastExchangeTime=1608212160446, loc=false, > ver=2.8.0#20200226-sha1:341b01df, > isClient=false] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3521) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128) > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1257) > at > > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendLocalPartitions(GridDhtPartitionsExchangeFuture.java:2014) > at > > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendPartitions(GridDhtPartitionsExchangeFuture.java:2149) > at > > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1614) > at > > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:891) > at > > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3172) > at > > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3021) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at java.base/java.lang.Thread.run(Thread.java:834) > > > Below are the logs for node 2- > [14:36:31,026][SEVERE][exchange-worker-#46][TcpCommunicationSpi] Failed to > send message to remote node [node=TcpDiscoveryNode > [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, > addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, > 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/ > 192.168.100.101:47500, > /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, > /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, > lastExchangeTime=1608212157863, loc=false, > ver=2.8.0#20200226-sha1:341b01df, > isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_INTERNAL_DIAGNOSTIC, > topicOrd=27, ordered=false, timeout=0, skipOnTimeout=false, > msg=IgniteDiagnosticMessage [flags=1, futId=0]]] > class org.apache.ignite.IgniteCheckedException: Failed to connect to node > (is node still alive?). Make sure that each ComputeTask and cache > Transaction has a timeout set in order to prevent parties from waiting > forever in case of network issues > [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/ > 192.168.100.101:47100, > /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, > /0:0:0:0:0:0:0:1%lo:47100]] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at > > org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829) > at > > org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497) > at > > org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453) > at > > org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055) > at > > org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019) > at > > org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982) > at > > org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544) > at > > org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115) > at > > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55) > Suppressed: class org.apache.ignite.IgniteCheckedException: Failed > to > connect to node (is node still alive?). Make sure that each ComputeTask and > cache Transaction has a timeout set in order to prevent parties from > waiting > forever in case of network issues > [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/ > 192.168.100.101:47100, > /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, > /0:0:0:0:0:0:0:1%lo:47100]] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) > ... 20 more > Caused by: class > org.apache.ignite.spi.IgniteSpiOperationTimeoutException: > Failed to perform handshake due to timeout (consider increasing > 'connectionTimeout' configuration property). > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) > ... 20 more > Suppressed: class org.apache.ignite.IgniteCheckedException: Failed > to > connect to node (is node still alive?). Make sure that each ComputeTask and > cache Transaction has a timeout set in order to prevent parties from > waiting > forever in case of network issues > [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/ > 192.168.100.101:47100, > /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, > /0:0:0:0:0:0:0:1%lo:47100]] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) > ... 20 more > Caused by: class > org.apache.ignite.spi.IgniteSpiOperationTimeoutException: > Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy > [maxTimeout=600000, totalTimeout=10000, startNanos=518549051061, > currTimeout=600000]] > at > > org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) > at > org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564) > ... 20 more > Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: > Failed to perform handshake due to timeout (consider increasing > 'connectionTimeout' configuration property). > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) > ... 20 more > [14:36:31,082][SEVERE][Thread-9][TcpCommunicationSpi] Failed to send > message > to remote node [node=TcpDiscoveryNode > [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, > addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, > 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/ > 192.168.100.101:47500, > /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, > /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, > lastExchangeTime=1608212157863, loc=false, > ver=2.8.0#20200226-sha1:341b01df, > isClient=false], msg=GridIoMessage [plc=3, topic=TOPIC_JOB, topicOrd=0, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridJobExecuteRequest > [subjId=7f683a9a-afcc-4d20-95f4-855147d759fd, > sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, > jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, > startTaskTime=1608212160962, timeout=9223372036854775807, > > taskName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask, > userVer=0, > > taskClsName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask, > ldrParticipants=null, cpSpi=null, createTime=1608212160982, > clsLdrId=f7c73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, depMode=SHARED, > dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true, > topPred=null, part=-1, topVer=null, execName=null]]] > class org.apache.ignite.IgniteCheckedException: Failed to connect to node > (is node still alive?). Make sure that each ComputeTask and cache > Transaction has a timeout set in order to prevent parties from waiting > forever in case of network issues > [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/ > 192.168.100.101:47100, > /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, > /0:0:0:0:0:0:0:1%lo:47100]] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at > > org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829) > at > > org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497) > at > > org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453) > at > > org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055) > at > > org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019) > at > > org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982) > at > > org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544) > at > > org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115) > at > > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55) > Suppressed: class org.apache.ignite.IgniteCheckedException: Failed > to > connect to node (is node still alive?). Make sure that each ComputeTask and > cache Transaction has a timeout set in order to prevent parties from > waiting > forever in case of network issues > [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/ > 192.168.100.101:47100, > /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, > /0:0:0:0:0:0:0:1%lo:47100]] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) > ... 20 more > Caused by: class > org.apache.ignite.spi.IgniteSpiOperationTimeoutException: > Failed to perform handshake due to timeout (consider increasing > 'connectionTimeout' configuration property). > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) > ... 20 more > Suppressed: class org.apache.ignite.IgniteCheckedException: Failed > to > connect to node (is node still alive?). Make sure that each ComputeTask and > cache Transaction has a timeout set in order to prevent parties from > waiting > forever in case of network issues > [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/ > 192.168.100.101:47100, > /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, > /0:0:0:0:0:0:0:1%lo:47100]] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) > ... 20 more > Caused by: class > org.apache.ignite.spi.IgniteSpiOperationTimeoutException: > Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy > [maxTimeout=600000, totalTimeout=10000, startNanos=518549051061, > currTimeout=600000]] > at > > org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) > at > org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564) > ... 20 more > Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: > Failed to perform handshake due to timeout (consider increasing > 'connectionTimeout' configuration property). > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) > ... 20 more > [14:36:31,084][SEVERE][exchange-worker-#46][diagnostic] Failed to send > diagnostic message: class o.a.i.IgniteCheckedException: Failed to send > message (node may have left the grid or TCP connection cannot be > established > due to firewall issues) [node=TcpDiscoveryNode > [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, > addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, > 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/ > 192.168.100.101:47500, > /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, > /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, > lastExchangeTime=1608212157863, loc=false, > ver=2.8.0#20200226-sha1:341b01df, > isClient=false], topic=TOPIC_INTERNAL_DIAGNOSTIC, > msg=IgniteDiagnosticMessage [flags=1, futId=0], policy=2] > [14:36:31,314][SEVERE][Thread-9][GridTaskWorker] Failed to send job > request: > GridJobExecuteRequest [subjId=7f683a9a-afcc-4d20-95f4-855147d759fd, > sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, > jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, > startTaskTime=1608212160962, timeout=9223372036854775807, > taskName=o.a.i.i.processors.cache.GridCacheAdapter$SizeTask, userVer=0, > taskClsName=o.a.i.i.processors.cache.GridCacheAdapter$SizeTask, > ldrParticipants=null, cpSpi=null, createTime=1608212160982, > clsLdrId=f7c73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, depMode=SHARED, > dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true, > topPred=null, part=-1, topVer=null, execName=null] > class org.apache.ignite.IgniteCheckedException: Failed to send message > (node > may have left the grid or TCP connection cannot be established due to > firewall issues) [node=TcpDiscoveryNode > [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, > addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, > 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/ > 192.168.100.101:47500, > /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, > /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, > lastExchangeTime=1608212157863, loc=false, > ver=2.8.0#20200226-sha1:341b01df, > isClient=false], topic=TOPIC_JOB, msg=GridJobExecuteRequest > [subjId=7f683a9a-afcc-4d20-95f4-855147d759fd, > sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, > jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, > startTaskTime=1608212160962, timeout=9223372036854775807, > > taskName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask, > userVer=0, > > taskClsName=org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask, > ldrParticipants=null, cpSpi=null, createTime=1608212160982, > clsLdrId=f7c73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, depMode=SHARED, > dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true, > topPred=null, part=-1, topVer=null, execName=null], policy=3] > at > > org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2042) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at > > org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829) > at > > org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497) > at > > org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453) > at > > org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055) > at > > org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019) > at > > org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982) > at > > org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544) > at > > org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115) > at > > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55) > Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send > message to remote node: TcpDiscoveryNode > [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, > addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, > 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/ > 192.168.100.101:47500, > /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, > /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, > lastExchangeTime=1608212157863, loc=false, > ver=2.8.0#20200226-sha1:341b01df, > isClient=false] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2943) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031) > ... 14 more > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to > connect > to node (is node still alive?). Make sure that each ComputeTask and cache > Transaction has a timeout set in order to prevent parties from waiting > forever in case of network issues > [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/ > 192.168.100.101:47100, > /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, > /0:0:0:0:0:0:0:1%lo:47100]] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906) > ... 16 more > Suppressed: class org.apache.ignite.IgniteCheckedException: Failed > to > connect to node (is node still alive?). Make sure that each ComputeTask and > cache Transaction has a timeout set in order to prevent parties from > waiting > forever in case of network issues > [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/ > 192.168.100.101:47100, > /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, > /0:0:0:0:0:0:0:1%lo:47100]] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) > ... 20 more > Caused by: class > org.apache.ignite.spi.IgniteSpiOperationTimeoutException: > Failed to perform handshake due to timeout (consider increasing > 'connectionTimeout' configuration property). > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) > ... 20 more > Suppressed: class org.apache.ignite.IgniteCheckedException: Failed > to > connect to node (is node still alive?). Make sure that each ComputeTask and > cache Transaction has a timeout set in order to prevent parties from > waiting > forever in case of network issues > [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/ > 192.168.100.101:47100, > /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, > /0:0:0:0:0:0:0:1%lo:47100]] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) > ... 20 more > Caused by: class > org.apache.ignite.spi.IgniteSpiOperationTimeoutException: > Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy > [maxTimeout=600000, totalTimeout=10000, startNanos=518549051061, > currTimeout=600000]] > at > > org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) > at > org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564) > ... 20 more > Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: > Failed to perform handshake due to timeout (consider increasing > 'connectionTimeout' configuration property). > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) > ... 20 more > [14:36:31,387][SEVERE][Thread-9][GridTaskWorker] Failed to obtain remote > job > result policy for result from ComputeTask.result(..) method (will fail the > whole task): GridJobResultImpl [job=SizeJob [], sib=GridJobSiblingImpl > [sesId=1cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, > jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, > nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, isJobDone=false], > jobCtx=GridJobContextImpl > [jobId=3cf73e07671-7f683a9a-afcc-4d20-95f4-855147d759fd, timeoutObj=null, > attrs=HashMap {}], node=TcpDiscoveryNode > [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, > addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, > 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/ > 192.168.100.101:47500, > /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, > /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, > lastExchangeTime=1608212157863, loc=false, > ver=2.8.0#20200226-sha1:341b01df, > isClient=false], ex=class o.a.i.spi.IgniteSpiException: Failed to send > message to remote node: TcpDiscoveryNode > [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, > addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, > 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/ > 192.168.100.101:47500, > /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, > /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, > lastExchangeTime=1608212157863, loc=false, > ver=2.8.0#20200226-sha1:341b01df, > isClient=false], hasRes=true, isCancelled=false, isOccupied=true] > class org.apache.ignite.IgniteException: Remote job threw exception. > at > > org.apache.ignite.internal.processors.cache.GridCacheAdapter$SizeTask.result(GridCacheAdapter.java:6811) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1062) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1055) > at > > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7037) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1055) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:862) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.processDelayedResponses(GridTaskWorker.java:711) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:542) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at > > org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:829) > at > > org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:497) > at > > org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:453) > at > > org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:4055) > at > > org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:4019) > at > > org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:982) > at > > org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:544) > at > > org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInLongOutLong(PlatformCache.java:1115) > at > > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inLongOutLong(PlatformTargetProxyImpl.java:55) > Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send > message to remote node: TcpDiscoveryNode > [id=0c98991e-b271-4cae-b1c2-06036bd666e9, consistentId=DSU_B, > addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.100.126.120, 127.0.0.1, > 192.168.100.101, 192.168.20.201], sockAddrs=HashSet [/ > 192.168.100.101:47500, > /10.100.126.120:47500, /0:0:0:0:0:0:0:1%lo:47500, /192.168.20.201:47500, > /127.0.0.1:47500], discPort=47500, order=6, intOrder=4, > lastExchangeTime=1608212157863, loc=false, > ver=2.8.0#20200226-sha1:341b01df, > isClient=false] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2943) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2865) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1445) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:664) > at > > org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:536) > ... 10 more > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to > connect > to node (is node still alive?). Make sure that each ComputeTask and cache > Transaction has a timeout set in order to prevent parties from waiting > forever in case of network issues > [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/ > 192.168.100.101:47100, > /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, > /0:0:0:0:0:0:0:1%lo:47100]] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3680) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3443) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3183) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3066) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2906) > ... 16 more > Suppressed: class org.apache.ignite.IgniteCheckedException: Failed > to > connect to node (is node still alive?). Make sure that each ComputeTask and > cache Transaction has a timeout set in order to prevent parties from > waiting > forever in case of network issues > [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/ > 192.168.100.101:47100, > /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, > /0:0:0:0:0:0:0:1%lo:47100]] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) > ... 20 more > Caused by: class > org.apache.ignite.spi.IgniteSpiOperationTimeoutException: > Failed to perform handshake due to timeout (consider increasing > 'connectionTimeout' configuration property). > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) > ... 20 more > Suppressed: class org.apache.ignite.IgniteCheckedException: Failed > to > connect to node (is node still alive?). Make sure that each ComputeTask and > cache Transaction has a timeout set in order to prevent parties from > waiting > forever in case of network issues > [nodeId=0c98991e-b271-4cae-b1c2-06036bd666e9, addrs=[/ > 192.168.100.101:47100, > /10.100.126.120:47100, /192.168.20.201:47100, /127.0.0.1:47100, > /0:0:0:0:0:0:0:1%lo:47100]] > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3682) > ... 20 more > Caused by: class > org.apache.ignite.spi.IgniteSpiOperationTimeoutException: > Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy > [maxTimeout=600000, totalTimeout=10000, startNanos=518549051061, > currTimeout=600000]] > at > > org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) > at > org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3564) > ... 20 more > Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException: > Failed to perform handshake due to timeout (consider increasing > 'connectionTimeout' configuration property). > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.handshakeTimeoutException(TcpCommunicationSpi.java:3839) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeTcpHandshake(TcpCommunicationSpi.java:4083) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3589) > ... 20 more > terminate called after throwing an instance of 'ignite::IgniteError' > what(): class org.apache.ignite.IgniteCheckedException: Remote job threw > exception. > > > Please suggest how to resolve this issue > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
