Ivan Bessonov created IGNITE-16926:
--------------------------------------
Summary: Interrupted compute job may fail a node
Key: IGNITE-16926
URL: https://issues.apache.org/jira/browse/IGNITE-16926
Project: Ignite
Issue Type: Bug
Components: persistence
Reporter: Ivan Bessonov
{code:java}
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is
corrupted [groupId=1234619879, pageIds=[72902014673333513], cacheId=645096946,
cacheName=*, indexName=*, msg=Runtime failure on row: Row@79570772[ key:
1168930235, val: Data hidden due to IGNITE_SENSITIVE_DATA_LOGGING flag. ][ data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden, data hidden, data hidden, data
hidden, data hidden, data hidden, data hidden
]]]]","logger_name":"ROOT","thread_name":"pub-#1278%x%","level":"ERROR","level_value":40000,"stack_trace":"org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [groupId=1234619879, pageIds=[72902014673333513],
cacheId=645096946, cacheName=*, indexName=*, msg=Runtime failure on row:
Row@79570772[ key: 1168930235, val: Data hidden due to
IGNITE_SENSITIVE_DATA_LOGGING flag. ][ data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden, data hidden, data hidden, data hidden, data hidden, data hidden,
data hidden ]] at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.corruptedTreeException(H2Tree.java:1003)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2492)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2432)
at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:500)
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.addToIndex(GridH2Table.java:880)
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:794)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:411)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:2546)
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:418)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:2815)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1871)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1842)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:3067)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:471)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2262)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2539)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1999)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1830)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1685)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:319)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.map(GridNearAtomicUpdateFuture.java:825)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapOnTopology(GridNearAtomicUpdateFuture.java:673)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:267)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAll0(GridDhtAtomicCache.java:1118)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAll0(GridDhtAtomicCache.java:669)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAll(GridCacheAdapter.java:2564)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1382)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.putAll(GatewayProtectedCacheProxy.java:865)
at x.Job.execute(...) at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:609)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7435)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:603)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:528)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)\n Caused by:
org.apache.ignite.IgniteException: Failed to close WAL write handle [idx=13554]
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlockPage(PageMemoryImpl.java:1713)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:525)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:517)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writeUnlock(PageHandler.java:401)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:376)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:347)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$11400(BPlusTree.java:100)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.tryReplace(BPlusTree.java:3932)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2863)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2843)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2843)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2843)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2452)\n
\t... 35 common frames omitted\n Caused by:
org.apache.ignite.internal.processors.cache.persistence.StorageException:
Failed to close WAL write handle [idx=13554] at
org.apache.ignite.internal.processors.cache.persistence.wal.filehandle.FileWriteHandleImpl.close(FileWriteHandleImpl.java:554)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.rollOver(FileWriteAheadLogManager.java:1300)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.log(FileWriteAheadLogManager.java:906)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.log(FileWriteAheadLogManager.java:827)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.beforeReleaseWrite(PageMemoryImpl.java:1879)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlockPage(PageMemoryImpl.java:1710)\n
\t... 47 common frames omitted\n Caused by:
java.nio.channels.ClosedByInterruptException: null at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:392) at
org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIO.force(RandomAccessFileIO.java:122)
at
org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIO.force(RandomAccessFileIO.java:147)
at
org.apache.ignite.internal.processors.cache.persistence.file.FileIODecorator.force(FileIODecorator.java:100)
at
org.apache.ignite.internal.processors.cache.persistence.wal.filehandle.FileWriteHandleImpl.close(FileWriteHandleImpl.java:533)\n
\t... 52 common frames omitted\n
{code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
