Hi,
There is some issue related to B+tree corruption that would be included
in next release 2.13: https://issues.apache.org/jira/browse/IGNITE-15990
Does this scenario looks similar to your case?
If you have a working reproducer feel free to submit a new Jira ticket.
15.04.2022 22:04, [email protected] пишет:
Hi,
In one of our Ignite cluster’s node, we have got *B+Tree* exception
due to which the node got stopped/disconnected from the cluster.
*Ignite Version :* 2.11.0.
Is this a product bug? If Yes then what is the workaround. Also has
this been fixed in any version?
*Logs:*
*[2022-04-14T14:50:17,989][INFO
][wal-file-archiver%EDIFMEDIA_PROD-#235%EDIFMEDIA_PROD%][FileWriteAheadLogManager]
Copied file
[src=/datastore2/wal/node00-060224e3-53ac-4555-876e-166a7480c142/0000000000000000.wal,
dst=/datastore2/archive/node00-060224e3-53ac-4555-876e-166a7480c142/0000000000211270.wal]*
*[2022-04-14T14:50:21,441][ERROR][sys-stripe-17-#18%EDIFMEDIA_PROD%][]
Critical system error detected. Will be handled accordingly to
configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false,
timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple
[val1=-291739550, val2=844420635435161]], cacheId=-291739550,
cacheName=MapCustOTTCache, indexName=_key_PK, msg=Runtime failure on
row: Row@5eaffcf8[ key: com.media.digitalapi.edif.model.MapCustOTTKey
[idHash=1519412213, hash=2147463230, serviceType=Z0114, partyId=p123],
val: com.media.digitalapi.edif.model.MapCustOTT [idHash=529725123,
hash=-1059328655, serviceType=null, updatedBy=TB00001KGQEC,
updateddatetime=2022-04-14 14:50:21.437, partyId=null,
subscriptionIdList=22414145021425514379098800228113357] ][ ]]]]*
*org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[**IgniteBiTuple
[val1=-291739550, val2=844420635435161]], cacheId=-291739550,
cacheName=MapCustOTTCache, indexName=_key_PK, msg=Runtime failure on
row: Row@5eaffcf8[ key: com.media.digitalapi.edif.model.MapCustOTTKey
[idHash=1519412213, hash=2147463230, serviceType=Z0114, partyId=p123],
val: com.media.digitalapi.edif.model.MapCustOTT [idHash=529725123,
hash=-1059328655, serviceType=null, updatedBy=TB00001KGQEC,
updateddatetime=2022-04-14 14:50:21.437, partyId=null,
subscriptionIdList=22414145021425514379098800228113357] ][ ]]*
* at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.corruptedTreeException(InlineIndexTree.java:581)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2464)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2411)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexImpl.putx(InlineIndexImpl.java:269)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexImpl.onUpdate(InlineIndexImpl.java:251)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.cache.query.index.IndexProcessor.updateIndex(IndexProcessor.java:419)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.cache.query.index.IndexProcessor.updateIndexes(IndexProcessor.java:291)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.cache.query.index.IndexProcessor.store(IndexProcessor.java:142)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:2512)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:2697)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1773)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1748)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:2794)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:441)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2342)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3338)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:143)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:322)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:317)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1151)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:592)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:393)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:319)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:110)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:309)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1908)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1529)
[ignite-core-2.11.0.jar:2.11.0]*
*at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:242)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1422)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569)
[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[ignite-core-2.11.0.jar:2.11.0]*
* at java.lang.Thread.run(Thread.java:748) [?:1.8.0_311]*
*Caused by:
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException:
java.lang.IllegalStateException: Item not found: 20***
* at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:345)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:165)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:136)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.createIndexRow(InlineIndexTree.java:377)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.cache.query.index.sorted.inline.io.AbstractInlineLeafIO.getLookupRow(AbstractInlineLeafIO.java:127)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.cache.query.index.sorted.inline.io.AbstractInlineLeafIO.getLookupRow(AbstractInlineLeafIO.java:37)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.getRow(InlineIndexTree.java:420)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.getRow(InlineIndexTree.java:70)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:5423)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:326)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:70)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:5410)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:5330)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1100(BPlusTree.java:100)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:310)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5921)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:290)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5907)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:174)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:397)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read(BPlusTree.java:6108)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2796)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2808)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2808)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2808)
~[ignite-core-2.11.0.jar:2.11.0]*
* at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2431)
[ignite-core-2.11.0.jar:2.11.0]*
* 663
*
[2022-04-14T14:50:23,192][WARN
][sys-stripe-17-#18%EDIFMEDIA_PROD%][DiagnosticProcessor]
CorruptedTreeException has occurred. To diagnose it, make a backup of
the following directories:
[/datastore2/wal/node00-060224e3-53ac-4555-876e-166a7480c142,
/datastore2/archive/node00-060224e3-53ac-4555-876e-166a7480c142].
Then, run the following command: java -cp <classpath>
org.apache.ignite.development.utils.IgniteWalConverter
walDir=/datastore2/wal/node00-060224e3-53ac-4555-876e-166a7480c142
walArchiveDir=/datastore2/archive/node00-060224e3-53ac-4555-876e-166a7480c142
pages=/datastore1/metadataWorkDir/diagnostic/corruptedPages_2022-04-14_14-50-23_184.txt
[2022-04-14T14:50:23,192][ERROR][sys-stripe-17-#18%EDIFMEDIA_PROD%][]
JVM will be halted immediately due to the failure:
[failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple
[val1=-291739550, val2=844420635435161]], cacheId=-291739550,
cacheName=MapCustOTTCache, indexName=_key_PK, msg=Runtime failure on
row: Row@5eaffcf8[ key: com.media.digitalapi.edif.model.MapCustOTTKey
[idHash=1519412213, hash=2147463230, serviceType=Z0114, partyId=p123],
val: com.media.digitalapi.edif.model.MapCustOTT [idHash=529725123,
hash=-1059328655, serviceType=null, updatedBy=TB00001KGQEC,
updateddatetime=2022-04-14 14:50:21.437, partyId=null,
subscriptionIdList=22414145021425514379098800228113357] ][ ]]]]
Thanks and Regards,
"*Confidentiality Warning*: This message and any attachments are
intended only for the use of the intended recipient(s), are
confidential and may be privileged. If you are not the intended
recipient, you are hereby notified that any review, re-transmission,
conversion to hard copy, copying, circulation or other use of this
message and any attachments is strictly prohibited. If you are not the
intended recipient, please notify the sender immediately by return
email and delete this message and any attachments from your system.
*Virus Warning:* Although the company has taken reasonable precautions
to ensure no viruses are present in this email. The company cannot
accept responsibility for any loss or damage arising from the use of
this email or attachment."
