We're observing something *very similar* in our setup. Setup:
- k8s cluster running on AWS - Ignite deployment (official Ignite 2.6.0 docker image), server mode, stores key/values in memory (12 pods) - Java service deployment (embeds Ignite 2.6.0), client mode, talks to Ignite deployment Several times a week (sometimes several times a day) we observe Ignite cluster segmentation, it crashes Ignite servers and hangs connections on Java service side. It can happen without any activity thus is very unlikely related to long GC pauses. In fact I was able to reproduce it in isolated setup where only health checks from k8s were touching the setup (in a health check I try to read a value out of Ignite cache). The worst part about that is that this cascadely brings whole cluster down and in fact client nodes become split as well and I'm often times observing at least two Ignite clusters instead of one (I expose /metrics endpoint which looks up into Ignite topology). Here is typical picture: ``` kubectl get pods -n somenamespace NAME READY STATUS RESTARTS AGE java-service-5494f775c4-fxb7j 1/1 Running 1 13m java-service-5494f775c4-lrfks 1/1 Running 3 13m ignite-78c4598c97-28fb6 1/1 Running 0 15m ignite-78c4598c97-58dk4 1/1 Running 1 15m ignite-78c4598c97-6nqpv 1/1 Running 0 15m ignite-78c4598c97-c8sfb 1/1 Running 1 15m ignite-78c4598c97-j5nlx 1/1 Running 0 15m ignite-78c4598c97-jr2s8 1/1 Running 0 15m ignite-78c4598c97-k6xh9 1/1 Running 0 15m ignite-78c4598c97-rnmf8 1/1 Running 0 15m ignite-78c4598c97-sqf6j 1/1 Running 0 15m ignite-78c4598c97-v5b6b 1/1 Running 0 15m ignite-78c4598c97-wqr49 1/1 Running 1 15m ignite-78c4598c97-zxm24 1/1 Running 0 15m ``` You can see part of Ignite deployment pods being restarted (because default behavior is to restart on critical failure) and clients are restarted because Ignite operations just timeout and cause k8s health check to fail. The only way for us to bring it up (temporary, until it splits again) is to scale whole deployment down to 0 replicas losing all data and scale it up, otherwise we only make things worse bringing more pods. We run on AWS, same region between pods, no Ignite locks used in our code, just key/value reads/writes, standard Ignite's TcpDiscoveryKubernetesIpFinder is used for autodiscovery. I'm happy to provide more info if needed, there is nothing super secret about our service or use of Ignite :) Below is full a log of Ignite pod restarted without production load, and only one client running, just after few minutes of deployment run-time: ``` Ignite Command Line Startup, ver. 2.6.0#20180710-sha1:669feacc 2018 Copyright(C) Apache Software Foundation [21:53:43,075][INFO][main][IgniteKernal] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.6.0#20180710-sha1:669feacc >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [21:53:43,076][INFO][main][IgniteKernal] Config URL: file:/config/ignite-server-config.xml [21:53:43,090][INFO][main][IgniteKernal] IgniteConfiguration [igniteInstanceName=null, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=8, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/opt/ignite/apache-ignite-fabric, igniteWorkDir=/opt/ignite/apache-ignite-fabric/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94fa3e, nodeId=c0377e70-4c1f-4bce-a1e9-cf8cdad84fc8, marsh=org.apache.ignite.internal.binary.BinaryMarshaller@3eb25e1a, marshLocJobs=false, daemon=false, p2pEnabled=false, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=null, enableForcibleNodeKill=false, enableTroubleshootingLog=false, srvLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2@3eb738bb, locAddr=null, locHost=null, locPort=47100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=1024, slowClientQueueLimit=512, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, lsnr=null, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@5bda8e08[Count = 1], stopping=false, metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@1e800aaa], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@185a6e9, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [lsnr=null], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@3cd3e762, addrRslvr=null, clientMode=false, rebalanceThreadPoolSize=1, txCfg=org.apache.ignite.configuration.TransactionConfiguration@1fa121e2, cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@7eac9008, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysCacheMaxSize=104857600, pageSize=8192, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=Default_Region, maxSize=25769803776, initSize=25769803776, swapPath=null, pageEvictionMode=RANDOM_2_LRU, evictionThreshold=0.9, emptyPagesPoolSize=25600, metricsEnabled=true, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@62ddbd7e, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], authEnabled=false, failureHnd=null, commFailureRslvr=null] [21:53:43,090][INFO][main][IgniteKernal] Daemon mode: off [21:53:43,090][INFO][main][IgniteKernal] OS: Linux 4.17.17-200.fc28.x86_64 amd64 [21:53:43,091][INFO][main][IgniteKernal] OS user: root [21:53:43,091][INFO][main][IgniteKernal] PID: 97 [21:53:43,091][INFO][main][IgniteKernal] Language runtime: Java Platform API Specification ver. 1.8 [21:53:43,091][INFO][main][IgniteKernal] VM information: OpenJDK Runtime Environment 1.8.0_171-b11 Oracle Corporation OpenJDK 64-Bit Server VM 25.171-b11 [21:53:43,092][INFO][main][IgniteKernal] VM total memory: 3.0GB [21:53:43,092][INFO][main][IgniteKernal] Remote Management [restart: on, REST: on, JMX (remote: on, port: 49112, auth: off, ssl: off)] [21:53:43,093][INFO][main][IgniteKernal] Logger: JavaLogger [quiet=false, config=null] [21:53:43,093][INFO][main][IgniteKernal] IGNITE_HOME=/opt/ignite/apache-ignite-fabric [21:53:43,093][INFO][main][IgniteKernal] VM arguments: [-Djava.net.preferIPv4Stack=true, -XX:+UnlockExperimentalVMOptions, -XX:+UseCGroupMemoryLimitForHeap, -XX:MaxDirectMemorySize=25g, -Xms3g, -Xmx3g, -XX:+UseG1GC, -XX:+AlwaysPreTouch, -XX:+ScavengeBeforeFullGC, -XX:+DisableExplicitGC, -DIGNITE_QUIET=false, -DIGNITE_SUCCESS_FILE=/opt/ignite/apache-ignite-fabric/work/ignite_success_09f322c2-7864-444b-96fa-3dfbb5eef466, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=49112, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -DIGNITE_HOME=/opt/ignite/apache-ignite-fabric, -DIGNITE_PROG_NAME=/opt/ignite/apache-ignite-fabric/bin/ignite.sh] [21:53:43,094][INFO][main][IgniteKernal] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemCacheMemorySize property to change the setting. [21:53:43,094][INFO][main][IgniteKernal] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache'], in 'Default_Region' dataRegion: ['buildcache']] [21:53:43,097][INFO][main][IgniteKernal] 3-rd party licenses can be found at: /opt/ignite/apache-ignite-fabric/libs/licenses [21:53:43,161][INFO][main][IgnitePluginProcessor] Configured plugins: [21:53:43,161][INFO][main][IgnitePluginProcessor] ^-- None [21:53:43,161][INFO][main][IgnitePluginProcessor] [21:53:43,162][INFO][main][FailureProcessor] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0]] [21:53:43,199][INFO][main][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0, selectorsCnt=4, selectorSpins=0, pairedConn=false] [21:53:43,215][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation) [21:53:43,236][WARNING][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [21:53:43,236][INFO][main][IgniteKernal] Security status [authentication=off, tls/ssl=off] [21:53:43,497][INFO][main][ClientListenerProcessor] Client connector processor has started on TCP port 10800 [21:53:43,534][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211] [21:53:43,565][INFO][main][IgniteKernal] Non-loopback local IPs: 10.42.69.5 [21:53:43,565][INFO][main][IgniteKernal] Enabled local MACs: 0AF321998484 [21:53:43,606][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port [port=47500, localHost=0.0.0.0/0.0.0.0, locNodeId=c0377e70-4c1f-4bce-a1e9-cf8cdad84fc8] [21:53:44,011][WARNING][main][IgniteKernal] Nodes started on local machine require more than 20% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=27748MB, available=31366MB] [21:53:44,059][INFO][exchange-worker-#42][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], crd=true, evt=NODE_JOINED, evtNode=c0377e70-4c1f-4bce-a1e9-cf8cdad84fc8, customEvt=null, allowMerge=false] [21:53:44,399][INFO][exchange-worker-#42][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, memoryPolicyName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647] [21:53:44,402][WARNING][exchange-worker-#42][GridCacheProcessor] Cache write synchronization mode is set to FULL_ASYNC. All single-key 'put' and 'remove' operations will return 'null', all 'putx' and 'removex' operations will return 'true' [cacheName=buildcache] [21:53:44,466][INFO][exchange-worker-#42][GridCacheProcessor] Started cache [name=buildcache, id=-467288620, memoryPolicyName=Default_Region, mode=PARTITIONED, atomicity=ATOMIC, backups=0] [21:53:44,503][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], waitTime=0ms, futInfo=NA] [21:53:44,505][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], waitTime=0ms, futInfo=NA] [21:53:44,683][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]] [21:53:44,705][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], err=null] [21:53:44,714][INFO][exchange-worker-#42][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], crd=true] [21:53:44,719][INFO][exchange-worker-#42][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=1, minorTopVer=0], evt=NODE_JOINED, node=c0377e70-4c1f-4bce-a1e9-cf8cdad84fc8] [21:53:44,805][INFO][main][IgniteKernal] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [21:53:44,805][INFO][main][IgniteKernal] [21:53:44,805][INFO][main][IgniteKernal] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [21:53:44,805][INFO][main][IgniteKernal] [21:53:44,806][INFO][main][IgniteKernal] >>> +----------------------------------------------------------------------+ >>> Ignite ver. 2.6.0#20180710-sha1:669feacc5d3a4e60efcdd300dc8de99780f38eed >>> +----------------------------------------------------------------------+ >>> OS name: Linux 4.17.17-200.fc28.x86_64 amd64 >>> CPU(s): 8 >>> Heap: 3.0GB >>> VM name: 97@ignite-6cc588fd6f-dw78n >>> Local node [ID=C0377E70-4C1F-4BCE-A1E9-CF8CDAD84FC8, order=1, >>> clientMode=false] >>> Local node addresses: [ignite-6cc588fd6f-dw78n/10.42.69.5, /127.0.0.1] >>> Local ports: TCP:10800 TCP:11211 TCP:47100 TCP:47500 [21:53:44,807][INFO][main][GridDiscoveryManager] Topology snapshot [ver=1, servers=1, clients=0, CPUs=8, offheap=24.0GB, heap=3.0GB] [21:53:44,807][INFO][main][GridDiscoveryManager] ^-- Node [id=C0377E70-4C1F-4BCE-A1E9-CF8CDAD84FC8, clusterState=ACTIVE] [21:53:44,807][INFO][main][GridDiscoveryManager] Data Regions Configured: [21:53:44,808][INFO][main][GridDiscoveryManager] ^-- Default_Region [initSize=24.0 GiB, maxSize=24.0 GiB, persistenceEnabled=false] [21:53:54,018][INFO][ignite-update-notifier-timer][GridUpdateNotifier] Your version is up to date. [21:54:01,813][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.41.210, rmtPort=50641] [21:54:01,821][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.41.210, rmtPort=50641] [21:54:01,821][INFO][tcp-disco-sock-reader-#5][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.41.210:50641, rmtPort=50641] [21:54:01,823][INFO][tcp-disco-sock-reader-#5][TcpDiscoverySpi] Received ping request from the remote node [rmtNodeId=abb0bd1b-4a2b-4cf6-90fc-ea223169d1e0, rmtAddr=/10.42.41.210:50641, rmtPort=50641] [21:54:01,825][INFO][tcp-disco-sock-reader-#5][TcpDiscoverySpi] Finished writing ping response [rmtNodeId=abb0bd1b-4a2b-4cf6-90fc-ea223169d1e0, rmtAddr=/10.42.41.210:50641, rmtPort=50641] [21:54:01,825][INFO][tcp-disco-sock-reader-#5][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.41.210:50641, rmtPort=50641 [21:54:03,049][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.58.43, rmtPort=39907] [21:54:03,049][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.58.43, rmtPort=39907] [21:54:03,050][INFO][tcp-disco-sock-reader-#6][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.58.43:39907, rmtPort=39907] [21:54:03,068][INFO][tcp-disco-sock-reader-#6][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.58.43:39907, rmtPort=39907 [21:54:03,156][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.58.43, rmtPort=56747] [21:54:03,157][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.58.43, rmtPort=56747] [21:54:03,157][INFO][tcp-disco-sock-reader-#7][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.58.43:56747, rmtPort=56747] [21:54:03,162][INFO][disco-event-worker-#41][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=2e4b4b61-4d24-447a-ad6d-d9d560c65eae, addrs=[10.42.58.43, 127.0.0.1], sockAddrs=[/10.42.58.43:47500, /127.0.0.1:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1537653243051, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false] [21:54:03,162][INFO][disco-event-worker-#41][GridDiscoveryManager] Topology snapshot [ver=2, servers=2, clients=0, CPUs=16, offheap=48.0GB, heap=6.0GB] [21:54:03,162][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Node [id=C0377E70-4C1F-4BCE-A1E9-CF8CDAD84FC8, clusterState=ACTIVE] [21:54:03,162][INFO][disco-event-worker-#41][GridDiscoveryManager] Data Regions Configured: [21:54:03,163][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Default_Region [initSize=24.0 GiB, maxSize=24.0 GiB, persistenceEnabled=false] [21:54:03,164][INFO][exchange-worker-#42][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], crd=true, evt=NODE_JOINED, evtNode=2e4b4b61-4d24-447a-ad6d-d9d560c65eae, customEvt=null, allowMerge=true] [21:54:03,164][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], waitTime=0ms, futInfo=NA] [21:54:03,617][INFO][grid-nio-worker-tcp-comm-0-#25][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/10.42.69.5:47100, rmtAddr=/10.42.58.43:47398] [21:54:03,646][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=[], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]]] [21:54:03,647][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], waitTime=0ms, futInfo=NA] [21:54:03,647][INFO][exchange-worker-#42][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], crd=true] [21:54:03,675][INFO][sys-#47][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=2, minorTopVer=0], node=2e4b4b61-4d24-447a-ad6d-d9d560c65eae, allReceived=true] [21:54:03,675][INFO][sys-#47][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=2, minorTopVer=0]] [21:54:03,675][INFO][sys-#47][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]] [21:54:03,691][INFO][sys-#47][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], err=null] [21:54:03,695][INFO][exchange-worker-#42][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=2, minorTopVer=0], evt=NODE_JOINED, node=2e4b4b61-4d24-447a-ad6d-d9d560c65eae] [21:54:05,170][INFO][tcp-disco-sock-reader-#7][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.58.43:56747, rmtPort=56747 [21:54:05,251][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.33.231, rmtPort=41995] [21:54:05,251][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.33.231, rmtPort=41995] [21:54:05,251][INFO][tcp-disco-sock-reader-#8][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.33.231:41995, rmtPort=41995] [21:54:05,255][INFO][disco-event-worker-#41][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=dfda47e8-9b48-42e1-b2f2-81e660ded9ce, addrs=[10.42.33.231, 127.0.0.1], sockAddrs=[/10.42.33.231:47500, /127.0.0.1:47500], discPort=47500, order=3, intOrder=3, lastExchangeTime=1537653245095, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false] [21:54:05,256][INFO][disco-event-worker-#41][GridDiscoveryManager] Topology snapshot [ver=3, servers=3, clients=0, CPUs=24, offheap=72.0GB, heap=9.0GB] [21:54:05,256][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Node [id=C0377E70-4C1F-4BCE-A1E9-CF8CDAD84FC8, clusterState=ACTIVE] [21:54:05,256][INFO][disco-event-worker-#41][GridDiscoveryManager] Data Regions Configured: [21:54:05,256][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Default_Region [initSize=24.0 GiB, maxSize=24.0 GiB, persistenceEnabled=false] [21:54:05,257][INFO][exchange-worker-#42][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], crd=true, evt=NODE_JOINED, evtNode=dfda47e8-9b48-42e1-b2f2-81e660ded9ce, customEvt=null, allowMerge=true] [21:54:05,257][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], waitTime=0ms, futInfo=NA] [21:54:05,735][INFO][grid-nio-worker-tcp-comm-1-#26][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/10.42.69.5:47100, rmtAddr=/10.42.33.231:60236] [21:54:05,752][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=[], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]]] [21:54:05,752][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], waitTime=0ms, futInfo=NA] [21:54:05,752][INFO][exchange-worker-#42][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], crd=true] [21:54:05,757][INFO][sys-#45][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=3, minorTopVer=0], node=2e4b4b61-4d24-447a-ad6d-d9d560c65eae, allReceived=false] [21:54:05,758][INFO][sys-#46][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=3, minorTopVer=0], node=dfda47e8-9b48-42e1-b2f2-81e660ded9ce, allReceived=true] [21:54:05,761][INFO][sys-#46][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=3, minorTopVer=0]] [21:54:05,761][INFO][sys-#46][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]] [21:54:05,773][INFO][sys-#46][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], err=null] [21:54:05,776][INFO][exchange-worker-#42][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=3, minorTopVer=0], evt=NODE_JOINED, node=dfda47e8-9b48-42e1-b2f2-81e660ded9ce] [21:54:06,166][INFO][tcp-disco-sock-reader-#8][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.33.231:41995, rmtPort=41995 [21:54:06,217][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.36.168, rmtPort=40153] [21:54:06,217][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.36.168, rmtPort=40153] [21:54:06,218][INFO][tcp-disco-sock-reader-#9][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.36.168:40153, rmtPort=40153] [21:54:06,221][INFO][disco-event-worker-#41][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=c562fb32-5ed6-4ad1-a8f8-6a95203dd519, addrs=[10.42.36.168, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, /10.42.36.168:47500], discPort=47500, order=4, intOrder=4, lastExchangeTime=1537653246143, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false] [21:54:06,222][INFO][disco-event-worker-#41][GridDiscoveryManager] Topology snapshot [ver=4, servers=4, clients=0, CPUs=32, offheap=96.0GB, heap=12.0GB] [21:54:06,222][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Node [id=C0377E70-4C1F-4BCE-A1E9-CF8CDAD84FC8, clusterState=ACTIVE] [21:54:06,222][INFO][disco-event-worker-#41][GridDiscoveryManager] Data Regions Configured: [21:54:06,222][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Default_Region [initSize=24.0 GiB, maxSize=24.0 GiB, persistenceEnabled=false] [21:54:06,222][INFO][exchange-worker-#42][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], crd=true, evt=NODE_JOINED, evtNode=c562fb32-5ed6-4ad1-a8f8-6a95203dd519, customEvt=null, allowMerge=true] [21:54:06,223][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], waitTime=0ms, futInfo=NA] [21:54:06,281][INFO][tcp-disco-sock-reader-#9][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.36.168:40153, rmtPort=40153 [21:54:06,344][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.33.158, rmtPort=53895] [21:54:06,344][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.33.158, rmtPort=53895] [21:54:06,344][INFO][tcp-disco-sock-reader-#10][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.33.158:53895, rmtPort=53895] [21:54:06,348][INFO][disco-event-worker-#41][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=23c48f2c-571e-4b92-8b76-75e4658d7d77, addrs=[10.42.33.158, 127.0.0.1], sockAddrs=[/10.42.33.158:47500, /127.0.0.1:47500], discPort=47500, order=5, intOrder=5, lastExchangeTime=1537653246254, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false] [21:54:06,348][INFO][disco-event-worker-#41][GridDiscoveryManager] Topology snapshot [ver=5, servers=5, clients=0, CPUs=40, offheap=120.0GB, heap=15.0GB] [21:54:06,348][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Node [id=C0377E70-4C1F-4BCE-A1E9-CF8CDAD84FC8, clusterState=ACTIVE] [21:54:06,348][INFO][disco-event-worker-#41][GridDiscoveryManager] Data Regions Configured: [21:54:06,348][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Default_Region [initSize=24.0 GiB, maxSize=24.0 GiB, persistenceEnabled=false] [21:54:06,691][INFO][grid-nio-worker-tcp-comm-2-#27][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/10.42.69.5:47100, rmtAddr=/10.42.36.168:36522] [21:54:06,705][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=[], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]]] [21:54:06,705][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], waitTime=0ms, futInfo=NA] [21:54:06,705][INFO][exchange-worker-#42][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], crd=true] [21:54:06,709][INFO][sys-#52][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=4, minorTopVer=0], node=2e4b4b61-4d24-447a-ad6d-d9d560c65eae, allReceived=false] [21:54:06,710][INFO][sys-#45][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=4, minorTopVer=0], node=c562fb32-5ed6-4ad1-a8f8-6a95203dd519, allReceived=false] [21:54:06,710][INFO][sys-#53][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=4, minorTopVer=0], node=dfda47e8-9b48-42e1-b2f2-81e660ded9ce, allReceived=true] [21:54:06,758][INFO][sys-#53][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=4, minorTopVer=0]] [21:54:06,758][INFO][sys-#53][GridCachePartitionExchangeManager] Merge exchange future [curFut=AffinityTopologyVersion [topVer=4, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_JOINED, evtNode=23c48f2c-571e-4b92-8b76-75e4658d7d77, evtNodeClient=false] [21:54:06,758][INFO][sys-#53][GridDhtPartitionsExchangeFuture] Merge server join exchange, wait for message [curFut=AffinityTopologyVersion [topVer=4, minorTopVer=0], node=23c48f2c-571e-4b92-8b76-75e4658d7d77] [21:54:06,832][INFO][grid-nio-worker-tcp-comm-3-#28][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/10.42.69.5:47100, rmtAddr=/10.42.33.158:41558] [21:54:06,854][INFO][sys-#47][GridDhtPartitionsExchangeFuture] Merge server join exchange, received message [curFut=AffinityTopologyVersion [topVer=4, minorTopVer=0], node=23c48f2c-571e-4b92-8b76-75e4658d7d77, msgVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], process=true, awaited=1] [21:54:06,854][INFO][sys-#47][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [21:54:06,868][INFO][sys-#47][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], err=null] [21:54:06,871][INFO][exchange-worker-#42][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_JOINED, node=c562fb32-5ed6-4ad1-a8f8-6a95203dd519] [21:54:08,316][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.60.115, rmtPort=49065] [21:54:08,316][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.60.115, rmtPort=49065] [21:54:08,317][INFO][tcp-disco-sock-reader-#11][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.60.115:49065, rmtPort=49065] [21:54:08,661][INFO][exchange-worker-#42][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=1], crd=true, evt=DISCOVERY_CUSTOM_EVT, evtNode=c0377e70-4c1f-4bce-a1e9-cf8cdad84fc8, customEvt=CacheAffinityChangeMessage [id=3ec26430661-55f9c16f-e3e4-4420-90dc-dc89e739996a, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], exchId=null, partsMsg=null, exchangeNeeded=true], allowMerge=false] [21:54:08,666][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=1], waitTime=0ms, futInfo=NA] [21:54:08,715][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=[], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=1]]] [21:54:08,716][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=1], waitTime=0ms, futInfo=NA] [21:54:08,717][INFO][exchange-worker-#42][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=1], crd=true] [21:54:08,719][INFO][sys-#53][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=5, minorTopVer=1], node=2e4b4b61-4d24-447a-ad6d-d9d560c65eae, allReceived=false] [21:54:08,721][INFO][sys-#46][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=5, minorTopVer=1], node=dfda47e8-9b48-42e1-b2f2-81e660ded9ce, allReceived=false] [21:54:08,721][INFO][sys-#47][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=5, minorTopVer=1], node=23c48f2c-571e-4b92-8b76-75e4658d7d77, allReceived=false] [21:54:08,721][INFO][sys-#49][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=5, minorTopVer=1], node=c562fb32-5ed6-4ad1-a8f8-6a95203dd519, allReceived=true] [21:54:08,722][INFO][sys-#49][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=1]] [21:54:08,729][INFO][sys-#49][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=1], err=null] [21:54:08,799][INFO][exchange-worker-#42][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=5, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, node=c0377e70-4c1f-4bce-a1e9-cf8cdad84fc8] [21:54:13,333][INFO][tcp-disco-sock-reader-#11][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.60.115:49065, rmtPort=49065 [21:54:19,353][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.16.120, rmtPort=55009] [21:54:19,354][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.16.120, rmtPort=55009] [21:54:19,354][INFO][tcp-disco-sock-reader-#12][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.16.120:55009, rmtPort=55009] [21:54:19,357][INFO][tcp-disco-sock-reader-#12][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.16.120:55009, rmtPort=55009 [21:54:19,370][INFO][tcp-disco-sock-reader-#10][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.33.158:53895, rmtPort=53895 [21:54:19,445][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.60.115, rmtPort=43839] [21:54:19,445][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.60.115, rmtPort=43839] [21:54:19,445][INFO][tcp-disco-sock-reader-#13][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.60.115:43839, rmtPort=43839] [21:54:19,448][INFO][disco-event-worker-#41][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=375ce9ce-36c8-46d2-a4f8-41674df021ad, addrs=[10.42.60.115, 127.0.0.1], sockAddrs=[/10.42.60.115:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=6, lastExchangeTime=1537653248324, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false] [21:54:19,449][INFO][disco-event-worker-#41][GridDiscoveryManager] Topology snapshot [ver=6, servers=6, clients=0, CPUs=48, offheap=140.0GB, heap=18.0GB] [21:54:19,449][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Node [id=C0377E70-4C1F-4BCE-A1E9-CF8CDAD84FC8, clusterState=ACTIVE] [21:54:19,449][INFO][disco-event-worker-#41][GridDiscoveryManager] Data Regions Configured: [21:54:19,449][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Default_Region [initSize=24.0 GiB, maxSize=24.0 GiB, persistenceEnabled=false] [21:54:19,450][INFO][exchange-worker-#42][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], crd=true, evt=NODE_JOINED, evtNode=375ce9ce-36c8-46d2-a4f8-41674df021ad, customEvt=null, allowMerge=true] [21:54:19,450][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], waitTime=0ms, futInfo=NA] [21:54:19,456][INFO][tcp-disco-sock-reader-#13][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.60.115:43839, rmtPort=43839 [21:54:19,529][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.16.120, rmtPort=43151] [21:54:19,529][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.16.120, rmtPort=43151] [21:54:19,530][INFO][tcp-disco-sock-reader-#14][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.16.120:43151, rmtPort=43151] [21:54:19,574][INFO][disco-event-worker-#41][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=edad0b96-c58d-43f4-8511-533ddfae1069, addrs=[10.42.16.120, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, /10.42.16.120:47500], discPort=47500, order=7, intOrder=7, lastExchangeTime=1537653249302, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false] [21:54:19,575][INFO][disco-event-worker-#41][GridDiscoveryManager] Topology snapshot [ver=7, servers=7, clients=0, CPUs=56, offheap=170.0GB, heap=21.0GB] [21:54:19,575][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Node [id=C0377E70-4C1F-4BCE-A1E9-CF8CDAD84FC8, clusterState=ACTIVE] [21:54:19,575][INFO][disco-event-worker-#41][GridDiscoveryManager] Data Regions Configured: [21:54:19,575][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Default_Region [initSize=24.0 GiB, maxSize=24.0 GiB, persistenceEnabled=false] [21:54:19,772][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.47.29, rmtPort=45773] [21:54:19,772][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.47.29, rmtPort=45773] [21:54:19,773][INFO][tcp-disco-sock-reader-#15][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.47.29:45773, rmtPort=45773] [21:54:19,775][INFO][tcp-disco-sock-reader-#15][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.47.29:45773, rmtPort=45773 [21:54:20,058][INFO][grid-nio-worker-tcp-comm-0-#25][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/10.42.69.5:47100, rmtAddr=/10.42.60.115:49612] [21:54:20,074][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=[], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]]] [21:54:20,074][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], waitTime=0ms, futInfo=NA] [21:54:20,074][INFO][exchange-worker-#42][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], crd=true] [21:54:20,157][INFO][sys-#53][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=6, minorTopVer=0], node=2e4b4b61-4d24-447a-ad6d-d9d560c65eae, allReceived=false] [21:54:20,158][INFO][sys-#47][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=6, minorTopVer=0], node=dfda47e8-9b48-42e1-b2f2-81e660ded9ce, allReceived=false] [21:54:20,158][INFO][sys-#46][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=6, minorTopVer=0], node=c562fb32-5ed6-4ad1-a8f8-6a95203dd519, allReceived=false] [21:54:20,159][INFO][sys-#50][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=6, minorTopVer=0], node=375ce9ce-36c8-46d2-a4f8-41674df021ad, allReceived=false] [21:54:20,159][INFO][sys-#49][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=6, minorTopVer=0], node=23c48f2c-571e-4b92-8b76-75e4658d7d77, allReceived=true] [21:54:20,159][INFO][sys-#49][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=6, minorTopVer=0]] [21:54:20,159][INFO][sys-#49][GridCachePartitionExchangeManager] Merge exchange future [curFut=AffinityTopologyVersion [topVer=6, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_JOINED, evtNode=edad0b96-c58d-43f4-8511-533ddfae1069, evtNodeClient=false] [21:54:20,160][INFO][sys-#49][GridDhtPartitionsExchangeFuture] Merge server join exchange, wait for message [curFut=AffinityTopologyVersion [topVer=6, minorTopVer=0], node=edad0b96-c58d-43f4-8511-533ddfae1069] [21:54:29,663][INFO][tcp-disco-sock-reader-#14][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.16.120:43151, rmtPort=43151 [21:54:29,810][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.47.29, rmtPort=40445] [21:54:29,810][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.47.29, rmtPort=40445] [21:54:29,810][INFO][tcp-disco-sock-reader-#16][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.47.29:40445, rmtPort=40445] [21:54:29,815][INFO][disco-event-worker-#41][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=f879f812-c1bc-43bf-a4f4-95af9aa6d647, addrs=[10.42.47.29, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, /10.42.47.29:47500], discPort=47500, order=8, intOrder=8, lastExchangeTime=1537653254313, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false] [21:54:29,815][INFO][disco-event-worker-#41][GridDiscoveryManager] Topology snapshot [ver=8, servers=8, clients=0, CPUs=64, offheap=190.0GB, heap=24.0GB] [21:54:29,815][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Node [id=C0377E70-4C1F-4BCE-A1E9-CF8CDAD84FC8, clusterState=ACTIVE] [21:54:29,815][INFO][disco-event-worker-#41][GridDiscoveryManager] Data Regions Configured: [21:54:29,815][INFO][disco-event-worker-#41][GridDiscoveryManager] ^-- Default_Region [initSize=24.0 GiB, maxSize=24.0 GiB, persistenceEnabled=false] [21:54:29,818][INFO][tcp-disco-sock-reader-#16][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.47.29:40445, rmtPort=40445 [21:54:30,075][WARNING][exchange-worker-#42][diagnostic] Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], node=c0377e70-4c1f-4bce-a1e9-cf8cdad84fc8]. Consider changing TransactionConfiguration.txTimeoutOnPartitionMapSynchronization to non default value to avoid this message. Dumping pending objects that might be the cause: [21:54:30,075][WARNING][exchange-worker-#42][diagnostic] Ready affinity version: AffinityTopologyVersion [topVer=5, minorTopVer=1] [21:54:30,077][WARNING][exchange-worker-#42][diagnostic] Last exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=375ce9ce-36c8-46d2-a4f8-41674df021ad, addrs=[10.42.60.115, 127.0.0.1], sockAddrs=[/10.42.60.115:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=6, lastExchangeTime=1537653248324, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], topVer=6, nodeId8=c0377e70, msg=Node joined: TcpDiscoveryNode [id=375ce9ce-36c8-46d2-a4f8-41674df021ad, addrs=[10.42.60.115, 127.0.0.1], sockAddrs=[/10.42.60.115:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=6, lastExchangeTime=1537653248324, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], type=NODE_JOINED, tstamp=1537653259449], crd=TcpDiscoveryNode [id=c0377e70-4c1f-4bce-a1e9-cf8cdad84fc8, addrs=[10.42.69.5, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, ignite-6cc588fd6f-dw78n/10.42.69.5:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1537653269848, loc=true, ver=2.6.0#20180710-sha1:669feacc, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=375ce9ce-36c8-46d2-a4f8-41674df021ad, addrs=[10.42.60.115, 127.0.0.1], sockAddrs=[/10.42.60.115:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=6, lastExchangeTime=1537653248324, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], topVer=6, nodeId8=c0377e70, msg=Node joined: TcpDiscoveryNode [id=375ce9ce-36c8-46d2-a4f8-41674df021ad, addrs=[10.42.60.115, 127.0.0.1], sockAddrs=[/10.42.60.115:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=6, lastExchangeTime=1537653248324, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], type=NODE_JOINED, tstamp=1537653259449], nodeId=375ce9ce, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1593307370], init=true, lastVer=GridCacheVersion [topVer=0, order=1537653247737, nodeOrder=0], partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], futures=[ExplicitLockReleaseFuture [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], futures=[]], AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], futures=[]], DataStreamerReleaseFuture [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], futures=[]], LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], futures=[]], AllTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], futures=[]]]]]], exchActions=null, affChangeMsg=null, initTs=1537653259449, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CRD, evtLatch=0, remaining=[], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=763033251]] [21:54:30,077][WARNING][exchange-worker-#42][GridCachePartitionExchangeManager] First 10 pending exchange futures [total=2] [21:54:30,078][WARNING][exchange-worker-#42][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=edad0b96-c58d-43f4-8511-533ddfae1069, addrs=[10.42.16.120, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, /10.42.16.120:47500], discPort=47500, order=7, intOrder=7, lastExchangeTime=1537653249302, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=false] [21:54:30,078][WARNING][exchange-worker-#42][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=f879f812-c1bc-43bf-a4f4-95af9aa6d647, addrs=[10.42.47.29, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, /10.42.47.29:47500], discPort=47500, order=8, intOrder=8, lastExchangeTime=1537653254313, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=false] [21:54:30,079][WARNING][exchange-worker-#42][diagnostic] Last 10 exchange futures (total: 9): [21:54:30,079][WARNING][exchange-worker-#42][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=f879f812-c1bc-43bf-a4f4-95af9aa6d647, addrs=[10.42.47.29, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, /10.42.47.29:47500], discPort=47500, order=8, intOrder=8, lastExchangeTime=1537653254313, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=false] [21:54:30,080][WARNING][exchange-worker-#42][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=edad0b96-c58d-43f4-8511-533ddfae1069, addrs=[10.42.16.120, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, /10.42.16.120:47500], discPort=47500, order=7, intOrder=7, lastExchangeTime=1537653249302, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=false] [21:54:30,080][WARNING][exchange-worker-#42][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=375ce9ce-36c8-46d2-a4f8-41674df021ad, addrs=[10.42.60.115, 127.0.0.1], sockAddrs=[/10.42.60.115:47500, /127.0.0.1:47500], discPort=47500, order=6, intOrder=6, lastExchangeTime=1537653248324, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=false] [21:54:30,080][WARNING][exchange-worker-#42][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=c0377e70-4c1f-4bce-a1e9-cf8cdad84fc8, addrs=[10.42.69.5, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, ignite-6cc588fd6f-dw78n/10.42.69.5:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1537653270070, loc=true, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=true] [21:54:30,080][WARNING][exchange-worker-#42][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=23c48f2c-571e-4b92-8b76-75e4658d7d77, addrs=[10.42.33.158, 127.0.0.1], sockAddrs=[/10.42.33.158:47500, /127.0.0.1:47500], discPort=47500, order=5, intOrder=5, lastExchangeTime=1537653246254, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=true] [21:54:30,081][WARNING][exchange-worker-#42][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=c562fb32-5ed6-4ad1-a8f8-6a95203dd519, addrs=[10.42.36.168, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, /10.42.36.168:47500], discPort=47500, order=4, intOrder=4, lastExchangeTime=1537653246143, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=true] [21:54:30,081][WARNING][exchange-worker-#42][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=dfda47e8-9b48-42e1-b2f2-81e660ded9ce, addrs=[10.42.33.231, 127.0.0.1], sockAddrs=[/10.42.33.231:47500, /127.0.0.1:47500], discPort=47500, order=3, intOrder=3, lastExchangeTime=1537653245095, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=true] [21:54:30,081][WARNING][exchange-worker-#42][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=2e4b4b61-4d24-447a-ad6d-d9d560c65eae, addrs=[10.42.58.43, 127.0.0.1], sockAddrs=[/10.42.58.43:47500, /127.0.0.1:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1537653243051, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=true] [21:54:30,082][WARNING][exchange-worker-#42][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=c0377e70-4c1f-4bce-a1e9-cf8cdad84fc8, addrs=[10.42.69.5, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, ignite-6cc588fd6f-dw78n/10.42.69.5:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1537653270070, loc=true, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=true] [21:54:30,082][WARNING][exchange-worker-#42][diagnostic] Latch manager state: ExchangeLatchManager [serverLatches={}, clientLatches={}] [21:54:30,082][WARNING][exchange-worker-#42][diagnostic] Pending transactions: [21:54:30,082][WARNING][exchange-worker-#42][diagnostic] Pending explicit locks: [21:54:30,082][WARNING][exchange-worker-#42][diagnostic] Pending cache futures: [21:54:30,083][WARNING][exchange-worker-#42][diagnostic] Pending atomic cache futures: [21:54:30,083][WARNING][exchange-worker-#42][diagnostic] Pending data streamer futures: [21:54:30,083][WARNING][exchange-worker-#42][diagnostic] Pending transaction deadlock detection futures: [21:54:30,217][INFO][grid-nio-worker-tcp-comm-1-#26][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/10.42.69.5:47100, rmtAddr=/10.42.16.120:35696] [21:54:30,235][INFO][sys-#49][GridDhtPartitionsExchangeFuture] Merge server join exchange, received message [curFut=AffinityTopologyVersion [topVer=6, minorTopVer=0], node=edad0b96-c58d-43f4-8511-533ddfae1069, msgVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], process=true, awaited=1] [21:54:30,235][INFO][sys-#49][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]] [21:54:30,244][INFO][sys-#49][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], err=null] [21:54:30,247][INFO][exchange-worker-#42][GridCachePartitionExchangeManager] Skipping rebalancing (obsolete exchange ID) [top=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_JOINED, node=375ce9ce-36c8-46d2-a4f8-41674df021ad] [21:54:30,247][INFO][exchange-worker-#42][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], crd=true, evt=NODE_JOINED, evtNode=f879f812-c1bc-43bf-a4f4-95af9aa6d647, customEvt=null, allowMerge=true] [21:54:30,247][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], waitTime=0ms, futInfo=NA] [21:54:30,571][INFO][grid-nio-worker-tcp-comm-2-#27][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/10.42.69.5:47100, rmtAddr=/10.42.47.29:36634] [21:54:30,585][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=[], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]]] [21:54:30,585][INFO][exchange-worker-#42][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], waitTime=0ms, futInfo=NA] [21:54:30,585][INFO][exchange-worker-#42][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], crd=true] [21:54:30,658][INFO][sys-#49][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0], node=23c48f2c-571e-4b92-8b76-75e4658d7d77, allReceived=false] [21:54:30,658][INFO][sys-#52][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0], node=dfda47e8-9b48-42e1-b2f2-81e660ded9ce, allReceived=false] [21:54:30,659][INFO][sys-#46][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0], node=edad0b96-c58d-43f4-8511-533ddfae1069, allReceived=false] [21:54:30,659][INFO][sys-#51][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0], node=2e4b4b61-4d24-447a-ad6d-d9d560c65eae, allReceived=false] [21:54:30,659][INFO][sys-#45][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0], node=c562fb32-5ed6-4ad1-a8f8-6a95203dd519, allReceived=false] [21:54:30,659][INFO][sys-#53][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0], node=375ce9ce-36c8-46d2-a4f8-41674df021ad, allReceived=false] [21:54:30,659][INFO][sys-#50][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0], node=f879f812-c1bc-43bf-a4f4-95af9aa6d647, allReceived=true] [21:54:30,660][INFO][sys-#50][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0]] [21:54:30,660][INFO][sys-#50][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]] [21:54:30,668][INFO][sys-#50][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], err=null] [21:54:30,671][INFO][exchange-worker-#42][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=8, minorTopVer=0], evt=NODE_JOINED, node=f879f812-c1bc-43bf-a4f4-95af9aa6d647] [21:54:40,438][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/10.42.58.43, rmtPort=55679] [21:54:40,438][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/10.42.58.43, rmtPort=55679] [21:54:40,438][INFO][tcp-disco-sock-reader-#17][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/10.42.58.43:55679, rmtPort=55679] [21:54:40,440][WARNING][tcp-disco-msg-worker-#2][TcpDiscoverySpi] Node is out of topology (probably, due to short-time network problems). [21:54:40,440][INFO][tcp-disco-sock-reader-#17][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/10.42.58.43:55679, rmtPort=55679 [21:54:40,441][WARNING][disco-event-worker-#41][GridDiscoveryManager] Local node SEGMENTED: TcpDiscoveryNode [id=c0377e70-4c1f-4bce-a1e9-cf8cdad84fc8, addrs=[10.42.69.5, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, ignite-6cc588fd6f-dw78n/10.42.69.5:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1537653280433, loc=true, ver=2.6.0#20180710-sha1:669feacc, isClient=false] [21:54:40,442][SEVERE][tcp-disco-srvr-#3][] Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Thread tcp-disco-srvr-#3 is terminated unexpectedly.]] java.lang.IllegalStateException: Thread tcp-disco-srvr-#3 is terminated unexpectedly. at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5686) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) [21:54:40,443][SEVERE][tcp-disco-srvr-#3][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Thread tcp-disco-srvr-#3 is terminated unexpectedly.]] ``` -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
