> some changes can be already made in the DB prior to the exception Ok. Then fixing the .NET error and the documentation would be nice.
From: Alex Plehanov <plehanov.a...@gmail.com> Sent: lundi 18 octobre 2021 14:55 To: user@ignite.apache.org Subject: Re: [EXT] Re: Crash of Ignite (B+Tree corrupted) on a large PutIfAbsent > Will you add a test in Ignite to avoid the crash I'm not quite sure, but perhaps a crash is the best we can do in this case. Throwing exception to the user might be not enough, since some changes can be already made in the DB prior to the exception and this can lead to data inconsistency. > give a clearer error message Nested exception already contains a root cause of the problem (For example: "Record is too long [capacity=67108864, size=92984514]", see ticket with the same problem reproduced with java [1]), but perhaps this nested exception is not correctly displayed by Ignite .NET [1]: https://issues.apache.org/jira/browse/IGNITE-13965<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FIGNITE-13965&data=04%7C01%7CVincent.Semeria%40finastra.com%7Cd54873a188604413936908d9923686c0%7C0b9b90da3fe1457ab340f1b67e1024fb%7C0%7C0%7C637701585167858681%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZXPgqf1vuz%2FzPJ95DbFxDxaKdjjHtrdcjFqUZ%2BeTqWE%3D&reserved=0> сб, 16 окт. 2021 г. в 15:02, Semeria, Vincent <vincent.seme...@finastra.com<mailto:vincent.seme...@finastra.com>>: Yes indeed, setting the WAL segment size to 150 megabytes accepts my PutIfAbsent. Thanks. Will you add a test in Ignite to avoid the crash, give a clearer error message and mention in the documentation that the WAL segment size should be higher than any single cache entry ? At the moment the doc just says this // // Summary: // Gets or sets the size of the WAL (Write Ahead Log) segment. For performance reasons, // the whole WAL is split into files of fixed length called segments. The limit should also be written in this page Ignite Persistence | Ignite Documentation (apache.org)<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fignite.apache.org%2Fdocs%2Flatest%2Fpersistence%2Fnative-persistence&data=04%7C01%7CVincent.Semeria%40finastra.com%7Cd54873a188604413936908d9923686c0%7C0b9b90da3fe1457ab340f1b67e1024fb%7C0%7C0%7C637701585167868630%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=esygpLK%2BBs9ZkH%2FO95%2F28J4E6ZgU2qvBw7dd7%2FjBF3o%3D&reserved=0> From: Alex Plehanov <plehanov.a...@gmail.com<mailto:plehanov.a...@gmail.com>> Sent: Friday, October 15, 2021 16:12 To: user@ignite.apache.org<mailto:user@ignite.apache.org> Subject: [EXT] Re: Crash of Ignite (B+Tree corrupted) on a large PutIfAbsent Hello, Perhaps you have too small WAL segment size (WAL segment should be large enough to fit the whole cache entry), try to change DataStorageConfiguration.WalSegmentSize property. чт, 14 окт. 2021 г. в 00:43, Semeria, Vincent <vincent.seme...@finastra.com<mailto:vincent.seme...@finastra.com>>: Hello Ignite, I currently use the C# API of Ignite 2.10 to store large objects of type V in an ICache<string, V>. Typically an object of V is around 100 megabytes. My data region is persisted on the hard drive. PutIfAbsent crashes Ignite with the complicated message below. As a workaround, I split type V into smaller types and used loops of smaller PutIfAbsent, which succeeded. Ultimately the data stored in the cache is the same, which shows that Ignite accepts my data (this is not a problem in the binary serializer). Is there a configuration of the data region that would accept a single PutIfAbsent of 100 megabytes ? Anyway Ignite should probably not crash when this limit is exceeded. Please send a clean error instead like "Insertion request exceeds limit XYZ" and keep Ignite alive in this case. Regards, Vincent Semeria error : PerformanceAttributionServer[PerformanceAttributionServer1]_Global@PARDH7JQHS2 (41660.6010) : [2021/10/12-10:21:48.215] : Apache.Ignite.NLog.IgniteNLogLogger::LoggerLog() : Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=241659666, val2=1127239936638982]], msg=Runtime failure on search row: SearchRow [key=KeyCacheObjectImpl [part=312, val=56ae72d3-a91a-4211-8279-0b0447881544, hasValBytes=true], hash=746501958, cacheId=0]]]] error : PerformanceAttributionServer[PerformanceAttributionServer1]_Global@PARDH7JQHS2 (41660.6010) : [2021/10/12-10:21:48.216] : org.apache.ignite.internal.processors.failure.FailureProcessor::LoggerLog() : A critical problem with persistence data structures was detected. Please make backup of persistence storage and WAL files for further analysis. Persistence storage path: WAL path: db/wal WAL archive path: db/wal/archive error : PerformanceAttributionServer[PerformanceAttributionServer1]_Global@PARDH7JQHS2 (41660.6010) : [2021/10/12-10:21:48.219] : org.apache.ignite.internal.processors.failure.FailureProcessor::LoggerLog() : No deadlocked threads detected. error : PerformanceAttributionServer[PerformanceAttributionServer1]_Global@PARDH7JQHS2 (41660.6010) : [2021/10/12-10:21:48.276] : org.apache.ignite.internal.processors.failure.FailureProcessor::LoggerLog() : Thread dump at 2021/10/12 10:21:48 CEST Thread [name="sys-#200", id=235, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@529e4706, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#199", id=234, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@529e4706, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#198", id=233, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@529e4706, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#197", id=232, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@529e4706, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#196", id=231, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@529e4706, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#195", id=230, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@529e4706, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#194", id=229, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@529e4706, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="Thread-18", id=204, state=RUNNABLE, blockCnt=0, waitCnt=0] Thread [name="Thread-17", id=176, state=RUNNABLE, blockCnt=0, waitCnt=2] Thread [name="Thread-16", id=168, state=RUNNABLE, blockCnt=0, waitCnt=0] Thread [name="Thread-15", id=117, state=RUNNABLE, blockCnt=0, waitCnt=0] Thread [name="Thread-14", id=116, state=RUNNABLE, blockCnt=0, waitCnt=0] Thread [name="Thread-13", id=115, state=RUNNABLE, blockCnt=0, waitCnt=0] Thread [name="Thread-12", id=111, state=RUNNABLE, blockCnt=0, waitCnt=82] at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454) at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1471) at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:192) - locked o.a.i.i.processors.failure.FailureProcessor@67349fda<mailto:o.a.i.i.processors.failure.FailureProcessor@67349fda> at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:155) at o.a.i.i.processors.cache.persistence.tree.BPlusTree.processFailure(BPlusTree.java:6152) at o.a.i.i.processors.cache.persistence.tree.BPlusTree.corruptedTreeException(BPlusTree.java:6139) at o.a.i.i.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1953) at o.a.i.i.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1758) at o.a.i.i.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1741) at o.a.i.i.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:2766) at o.a.i.i.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:439) at o.a.i.i.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2338) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2654) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:2114) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1931) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1724) at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:306) at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:486) at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:446) at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:255) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1169) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:634) at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2596) at o.a.i.i.processors.cache.GridCacheAdapter.putIfAbsent(GridCacheAdapter.java:3041) at o.a.i.i.processors.cache.IgniteCacheProxyImpl.putIfAbsent(IgniteCacheProxyImpl.java:1412) at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.putIfAbsent(GatewayProtectedCacheProxy.java:928) at o.a.i.i.processors.platform.cache.PlatformCache.processInStreamOutLong(PlatformCache.java:516) at o.a.i.i.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67) Locked synchronizers: java.util.concurrent.locks.ReentrantLock$NonfairSync@3de6a6db<mailto:java.util.concurrent.locks.ReentrantLock$NonfairSync@3de6a6db> Thread [name="Thread-11", id=102, state=RUNNABLE, blockCnt=0, waitCnt=0] Thread [name="ttl-cleanup-worker-#72", id=99, state=TIMED_WAITING, blockCnt=0, waitCnt=2297] at java.lang.Thread.sleep(Native Method) at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7984) at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:210) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="checkpoint-runner-IO-#71", id=98, state=WAITING, blockCnt=1, waitCnt=722] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@565aa90c, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="checkpoint-runner-IO-#70", id=97, state=WAITING, blockCnt=2, waitCnt=585] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@565aa90c, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="checkpoint-runner-IO-#69", id=96, state=WAITING, blockCnt=4, waitCnt=660] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@565aa90c, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="checkpoint-runner-IO-#68", id=95, state=WAITING, blockCnt=2, waitCnt=867] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@565aa90c, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="checkpoint-runner-cpu-#65", id=92, state=WAITING, blockCnt=54, waitCnt=478] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@76dc62b6, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="checkpoint-runner-cpu-#64", id=91, state=WAITING, blockCnt=38, waitCnt=494] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@76dc62b6, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="checkpoint-runner-cpu-#63", id=90, state=WAITING, blockCnt=75, waitCnt=505] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@76dc62b6, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="checkpoint-runner-cpu-#62", id=89, state=WAITING, blockCnt=17, waitCnt=545] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@76dc62b6, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="db-checkpoint-thread-#61", id=88, state=TIMED_WAITING, blockCnt=0, waitCnt=45] Lock [object=o.a.i.i.processors.cache.persistence.checkpoint.Checkpointer@35d5f9f8, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at o.a.i.i.processors.cache.persistence.checkpoint.Checkpointer.waitCheckpointEvent(Checkpointer.java:755) at o.a.i.i.processors.cache.persistence.checkpoint.Checkpointer.body(Checkpointer.java:246) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="dms-writer-thread-#55", id=82, state=WAITING, blockCnt=1, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d8d51e7, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at o.a.i.i.processors.metastorage.persistence.DmsDataWriterWorker.body(DmsDataWriterWorker.java:146) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="wal-segment-syncer-#54", id=81, state=TIMED_WAITING, blockCnt=0, waitCnt=4577] at java.lang.Thread.sleep(Native Method) at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7984) at o.a.i.i.processors.cache.persistence.wal.filehandle.FileHandleManagerImpl$WalSegmentSyncer.body(FileHandleManagerImpl.java:622) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="wal-file-cleaner%null-#53", id=80, state=WAITING, blockCnt=0, waitCnt=1] Lock [object=o.a.i.i.processors.cache.persistence.wal.aware.SegmentArchiveSizeStorage@3e9c9d93, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at o.a.i.i.processors.cache.persistence.wal.aware.SegmentArchiveSizeStorage.awaitExceedMaxSize(SegmentArchiveSizeStorage.java:79) at o.a.i.i.processors.cache.persistence.wal.aware.SegmentAware.awaitExceedMaxArchiveSize(SegmentAware.java:354) at o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileCleaner.body(FileWriteAheadLogManager.java:3171) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run( warning : PerformanceAttributionServer[PerformanceAttributionServer1]_Global@PARDH7JQHS2 (41660.6010) : [2021/10/12-10:21:48.296] : org.apache.ignite.internal.processors.cache.CacheDiagnosticManager::LoggerLog() : Page locks dump: Thread=[name=Thread-12, id=111], state=RUNNABLE Locked pages = [] Locked pages log: name=Thread-12 time=(1634026908291, 2021-10-12 10:21:48.291) Thread=[name=Thread-13, id=115], state=RUNNABLE Locked pages = [] Locked pages log: name=Thread-13 time=(1634026908291, 2021-10-12 10:21:48.291) Thread=[name=Thread-14, id=116], state=RUNNABLE Locked pages = [] Locked pages log: name=Thread-14 time=(1634026908291, 2021-10-12 10:21:48.291) Thread=[name=Thread-15, id=117], state=RUNNABLE Locked pages = [] Locked pages log: name=Thread-15 time=(1634026908291, 2021-10-12 10:21:48.291) Thread=[name=Thread-16, id=168], state=RUNNABLE Locked pages = [] Locked pages log: name=Thread-16 time=(1634026908291, 2021-10-12 10:21:48.291) Thread=[name=Thread-17, id=176], state=RUNNABLE Locked pages = [] Locked pages log: name=Thread-17 time=(1634026908291, 2021-10-12 10:21:48.291) Thread=[name=Thread-18, id=204], state=RUNNABLE Locked pages = [] Locked pages log: name=Thread-18 time=(1634026908291, 2021-10-12 10:21:48.291) Thread=[name=checkpoint-runner-cpu-#62, id=89], state=WAITING Locked pages = [] Locked pages log: name=checkpoint-runner-cpu-#62 time=(1634026908291, 2021-10-12 10:21:48.291) Thread=[name=checkpoint-runner-cpu-#63, id=90], state=WAITING Locked pages = [] Locked pages log: name=checkpoint-runner-cpu-#63 time=(1634026908291, 2021-10-12 10:21:48.291) Thread=[name=db-checkpoint-thread-#61, id=88], state=TIMED_WAITING Locked pages = [] Locked pages log: name=db-checkpoint-thread-#61 time=(1634026908291, 2021-10-12 10:21:48.291) Thread=[name=dms-writer-thread-#55, id=82], state=WAITING Locked pages = [] Locked pages log: name=dms-writer-thread-#55 time=(1634026908291, 2021-10-12 10:21:48.291) Thread=[name=exchange-worker-#50, id=76], state=TIMED_WAITING Locked pages = [] Locked pages log: name=exchange-worker-#50 time=(1634026908291, 2021-10-12 10:21:48.291) Thread=[name=main, id=1], state=RUNNABLE Locked pages = [] Locked pages log: name=main time=(1634026908291, 2021-10-12 10:21:48.291) Reached logical end of the segment for file D:\sophis_beta\tests\installs\Debug\servers\sophis\PerformanceAttribution\PerformanceAttributionServer\work\db\wal\node00-d410e934-f1a8-474b-b5c2-a5dc783b186c\0000000000000001.wal Reached logical end of the segment for file D:\sophis_beta\tests\installs\Debug\servers\sophis\PerformanceAttribution\PerformanceAttributionServer\work\db\wal\node00-d410e934-f1a8-474b-b5c2-a5dc783b186c\0000000000000002.wal Reached logical end of the segment for file D:\sophis_beta\tests\installs\Debug\servers\sophis\PerformanceAttribution\PerformanceAttributionServer\work\db\wal\node00-d410e934-f1a8-474b-b5c2-a5dc783b186c\0000000000000003.wal Reached logical end of the segment for file D:\sophis_beta\tests\installs\Debug\servers\sophis\PerformanceAttribution\PerformanceAttributionServer\work\db\wal\node00-d410e934-f1a8-474b-b5c2-a5dc783b186c\0000000000000004.wal Reached logical end of the segment for file D:\sophis_beta\tests\installs\Debug\servers\sophis\PerformanceAttribution\PerformanceAttributionServer\work\db\wal\node00-d410e934-f1a8-474b-b5c2-a5dc783b186c\0000000000000005.wal Reached logical end of the segment for file D:\sophis_beta\tests\installs\Debug\servers\sophis\PerformanceAttribution\PerformanceAttributionServer\work\db\wal\node00-d410e934-f1a8-474b-b5c2-a5dc783b186c\0000000000000006.wal Reached logical end of the segment for file D:\sophis_beta\tests\installs\Debug\servers\sophis\PerformanceAttribution\PerformanceAttributionServer\work\db\wal\node00-d410e934-f1a8-474b-b5c2-a5dc783b186c\0000000000000007.wal Reached logical end of the segment for file D:\sophis_beta\tests\installs\Debug\servers\sophis\PerformanceAttribution\PerformanceAttributionServer\work\db\wal\node00-d410e934-f1a8-474b-b5c2-a5dc783b186c\0000000000000008.wal Reached logical end of the segment for file D:\sophis_beta\tests\installs\Debug\servers\sophis\PerformanceAttribution\PerformanceAttributionServer\work\db\wal\node00-d410e934-f1a8-474b-b5c2-a5dc783b186c\0000000000000009.wal error : PerformanceAttributionServer[PerformanceAttributionServer1]_Global@PARDH7JQHS2 (41660.6010) : [2021/10/12-10:21:48.574] : Apache.Ignite.NLog.IgniteNLogLogger::LoggerLog() : JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=241659666, val2=1127239936638982]], msg=Runtime failure on search row: SearchRow [key=KeyCacheObjectImpl [part=312, val=56ae72d3-a91a-4211-8279-0b0447881544, hasValBytes=true], hash=746501958, cacheId=0]]]] Application is shutting down... "FINASTRA" is the trade name of the FINASTRA group of companies. This email and any attachments have been scanned for known viruses using multiple scanners. This email message is intended for the named recipient only. It may be privileged and/or confidential. If you are not the named recipient of this email please notify us immediately and do not copy it or use it for any purpose, nor disclose its contents to any other person. This email does not constitute the commencement of legal relations between you and FINASTRA. Please refer to the executed contract between you and the relevant member of the FINASTRA group for the identity of the contracting party with which you are dealing. "FINASTRA" is the trade name of the FINASTRA group of companies. This email and any attachments have been scanned for known viruses using multiple scanners. This email message is intended for the named recipient only. It may be privileged and/or confidential. If you are not the named recipient of this email please notify us immediately and do not copy it or use it for any purpose, nor disclose its contents to any other person. This email does not constitute the commencement of legal relations between you and FINASTRA. Please refer to the executed contract between you and the relevant member of the FINASTRA group for the identity of the contracting party with which you are dealing. "FINASTRA" is the trade name of the FINASTRA group of companies. This email and any attachments have been scanned for known viruses using multiple scanners. This email message is intended for the named recipient only. It may be privileged and/or confidential. If you are not the named recipient of this email please notify us immediately and do not copy it or use it for any purpose, nor disclose its contents to any other person. This email does not constitute the commencement of legal relations between you and FINASTRA. Please refer to the executed contract between you and the relevant member of the FINASTRA group for the identity of the contracting party with which you are dealing.
