Yes. I have IGNITE_EXCHANGE_HISTORY_SIZE set to 10. should I set IGNITE_EXCHANGE_HISTORY_SIZE = 0 ???
will file a bug shortly ________________________________ From: Ilya Kasnacheev <[email protected]> Sent: Tuesday, November 3, 2020 8:57 PM To: [email protected] <[email protected]> Subject: Re: Failed to Resolve NodeTopology - ignite 2.8.1 Hello! You seemed to have a very old transaction which tried to access a topology version which was not in the history. Do you have IGNITE_EXCHANGE_HISTORY_SIZE set? I think this is a bug that it causes node failure. I would expect that transaction is killed, that's all. Can you please file a ticket about this issue against Apache Ignite JIRA? https://issues.apache.org/jira/projects/IGNITE Regards, -- Ilya Kasnacheev вт, 3 нояб. 2020 г. в 05:59, Mahesh Renduchintala <[email protected]<mailto:[email protected]>>: Hi, We saw all ignite nodes crash today morning. Below are the error logs? why would "Failed to Resolve Node Topology" occur? what would cause this? If there is a network disturbance, should I not get some sort of segmentation error? [14:30:10,095][INFO][grid-timeout-worker-#43][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=1ec1e3f9, uptime=1 day, 03:54:03.143] ^-- H/N/C [hosts=11, nodes=11, CPUs=46] ^-- CPU [cur=0.93%, avg=7.03%, GC=0%] ^-- PageMemory [pages=11582153] ^-- Heap [used=25392MB, free=48.34%, comm=49152MB] ^-- Off-heap [used=45772MB, free=30.47%, comm=65736MB] ^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB] ^-- default region [used=45771MB, free=30.16%, comm=65536MB] ^-- metastoreMemPlc region [used=0MB, free=99.03%, comm=0MB] ^-- TxLog region [used=0MB, free=100%, comm=100MB] ^-- Ignite persistence [used=53690MB] ^-- sysMemPlc region [used=0MB] ^-- default region [used=53689MB] ^-- metastoreMemPlc region [used=0MB] ^-- TxLog region [used=0MB] ^-- Outbound messages queue [size=0] ^-- Public thread pool [active=0, idle=2, qSize=0] ^-- System thread pool [active=0, idle=32, qSize=0] [14:30:10,976][SEVERE][sys-stripe-17-#18][GridCacheIoManager] Failed processing message [senderId=8f0d3c00-7b18-456c-9066-3852abca7254, msg=GridNearTxPrepareRequest [futId=196a4f48571-8d97cb24-fd81-45b2-a60f-60c26db22c90, miniId=1, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], subjId=8f0d3c00-7b18-456c-9066-3852abca7254, taskNameHash=0, txLbl=null, flags=[firstClientReq][implicitSingle], super=GridDistributedTxPrepareRequest [threadId=2651, concurrency=OPTIMISTIC, isolation=READ_COMMITTED, writeVer=GridCacheVersion [topVer=215702708, order=1604374451397, nodeOrder=21], timeout=420000, reads=null, writes=ArrayList [IgniteTxEntry [txKey=null, val=CacheObjectImpl [val=null, hasValBytes=true][op=CREATE, val=], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false, entry=null, prepared=0, locked=false, nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=2, txState=null, flags=onePhase|last, super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=215702708, order=1604374451397, nodeOrder=21], committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage [cacheId=0, super=GridCacheMessage [msgId=2767960, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], err=null, skipPrepare=false]]]]]] class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500<http://192.168.1.6:47500>], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]] at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472) at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367) at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:745) [14:30:10,976][SEVERE][sys-stripe-17-#18][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500<http://192.168.1.6:47500>], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]] class org.apache.ignite.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500<http://192.168.1.6:47500>], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]] at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472) at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367) at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:745) [14:30:11,078][WARNING][sys-stripe-17-#18][CacheDiagnosticManager] Page locks dump: Thread=[name=auth-#69, id=119], state=WAITING Locked pages = [] Locked pages log: name=auth-#69 time=(1604327410981, 2020-11-02 14:30:10.981) Thread=[name=checkpoint-runner-#113, id=167], state=WAITING Locked pages = [] Locked pages log: name=checkpoint-runner-#113 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=checkpoint-runner-#114, id=168], state=WAITING Locked pages = [] Locked pages log: name=checkpoint-runner-#114 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=checkpoint-runner-#115, id=169], state=WAITING Locked pages = [] Locked pages log: name=checkpoint-runner-#115 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=checkpoint-runner-#116, id=170], state=WAITING Locked pages = [] Locked pages log: name=checkpoint-runner-#116 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=client-connector-#1396, id=1594], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#1396 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=client-connector-#1397, id=1595], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#1397 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=client-connector-#239, id=370], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#239 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=client-connector-#240, id=371], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#240 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=client-connector-#241, id=372], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#241 time=(1604327410981, 2020-11-02 14:30:10.981) Thread=[name=client-connector-#242, id=374], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#242 time=(1604327410981, 2020-11-02 14:30:10.981) Thread=[name=client-connector-#435, id=620], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#435 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=client-connector-#437, id=622], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#437 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=client-connector-#439, id=624], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#439 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=client-connector-#440, id=625], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#440 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=client-connector-#441, id=626], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#441 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=client-connector-#442, id=627], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#442 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=client-connector-#443, id=628], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#443 time=(1604327410981, 2020-11-02 14:30:10.981) Thread=[name=client-connector-#444, id=629], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#444 time=(1604327410981, 2020-11-02 14:30:10.981) Thread=[name=client-connector-#445, id=630], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#445 time=(1604327410981, 2020-11-02 14:30:10.981) Thread=[name=client-connector-#446, id=631], state=WAITING Locked pages = [] Locked pages log: name=client-connector-#446 time=(1604327410981, 2020-11-02 14:30:10.981) Thread=[name=db-checkpoint-thread-#112, id=166], state=TIMED_WAITING Locked pages = [] Locked pages log: name=db-checkpoint-thread-#112 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=dms-writer-thread-#78, id=130], state=WAITING Locked pages = [] Locked pages log: name=dms-writer-thread-#78 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=exchange-worker-#70, id=122], state=TIMED_WAITING Locked pages = [] Locked pages log: name=exchange-worker-#70 time=(1604327410981, 2020-11-02 14:30:10.981) Thread=[name=main, id=1], state=WAITING Locked pages = [] Locked pages log: name=main time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-0-#1, id=20], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-0-#1 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-1-#2, id=21], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-1-#2 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-10-#11, id=30], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-10-#11 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-11-#12, id=31], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-11-#12 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-12-#13, id=32], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-12-#13 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-13-#14, id=33], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-13-#14 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-14-#15, id=34], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-14-#15 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-15-#16, id=35], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-15-#16 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-16-#17, id=36], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-16-#17 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-17-#18, id=37], state=RUNNABLE Locked pages = [] Locked pages log: name=sys-stripe-17-#18 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-18-#19, id=38], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-18-#19 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-19-#20, id=39], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-19-#20 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-2-#3, id=22], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-2-#3 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-20-#21, id=40], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-20-#21 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-21-#22, id=41], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-21-#22 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-22-#23, id=42], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-22-#23 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-23-#24, id=43], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-23-#24 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-24-#25, id=44], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-24-#25 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-25-#26, id=45], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-25-#26 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-26-#27, id=46], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-26-#27 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-27-#28, id=47], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-27-#28 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-28-#29, id=48], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-28-#29 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-29-#30, id=49], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-29-#30 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-3-#4, id=23], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-3-#4 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-30-#31, id=50], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-30-#31 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-31-#32, id=51], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-31-#32 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-4-#5, id=24], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-4-#5 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-5-#6, id=25], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-5-#6 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-6-#7, id=26], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-6-#7 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-7-#8, id=27], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-7-#8 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-8-#9, id=28], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-8-#9 time=(1604327410980, 2020-11-02 14:30:10.980) Thread=[name=sys-stripe-9-#10, id=29], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-9-#10 time=(1604327410980, 2020-11-02 14:30:10.980) [14:30:11,079][SEVERE][sys-stripe-17-#18][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteException: Failed to resolve nodes topology [cacheGrp=DataStructure_DisHashMap, topVer=AffinityTopologyVersion [topVer=158, minorTopVer=2], history=[AffinityTopologyVersion [topVer=218, minorTopVer=0], AffinityTopologyVersion [topVer=219, minorTopVer=0], AffinityTopologyVersion [topVer=220, minorTopVer=0], AffinityTopologyVersion [topVer=221, minorTopVer=0], AffinityTopologyVersion [topVer=222, minorTopVer=0], AffinityTopologyVersion [topVer=223, minorTopVer=0], AffinityTopologyVersion [topVer=224, minorTopVer=0], AffinityTopologyVersion [topVer=225, minorTopVer=0], AffinityTopologyVersion [topVer=226, minorTopVer=0], AffinityTopologyVersion [topVer=227, minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=227, minorTopVer=0]], locNode=TcpDiscoveryNode [id=1ec1e3f9-4c61-4316-9db3-5b35379570ab, consistentId=afc388a3-aa34-4553-9b62-1ea36657feb0, addrs=ArrayList [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500<http://192.168.1.6:47500>], discPort=47500, order=2, intOrder=2, lastExchangeTime=1604224232725, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]
