Ivan Bessonov created IGNITE-13808:
--------------------------------------
Summary: Control.sh validate_indexes throws CorruptedTreeException
and fails server node during check
Key: IGNITE-13808
URL: https://issues.apache.org/jira/browse/IGNITE-13808
Project: Ignite
Issue Type: Bug
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov
CorruptedTreeException during validate index command calls Failure handler and
stops server node:
{code:java}
[21:44:26,257][WARNING][pool-5-thread-2][ValidateIndexesClosure] Current
progress of ValidateIndexesClosure: checked integrity of 1 index partitions of
14 cache groups
[21:44:26,852][SEVERE][pool-5-thread-16][] Critical system error detected. Will
be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is
corrupted [pages(groupId, pageId)=[], msg=Runtime failure on bounds:
[lower=null, upper=null]]]]
class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[], msg=Runtime failure on bounds:
[lower=null, upper=null]]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.corruptedTreeException(BPlusTree.java:5126)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1029)
at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.find(H2TreeIndex.java:243)
at
org.apache.ignite.internal.visor.verify.ValidateIndexesClosure.processIndex(ValidateIndexesClosure.java:651)
at
org.apache.ignite.internal.visor.verify.ValidateIndexesClosure.access$200(ValidateIndexesClosure.java:93)
at
org.apache.ignite.internal.visor.verify.ValidateIndexesClosure$4.call(ValidateIndexesClosure.java:631)
at
org.apache.ignite.internal.visor.verify.ValidateIndexesClosure$4.call(ValidateIndexesClosure.java:629)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException:
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException:
java.lang.IllegalStateException: Item not found: 11
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:987)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1014)
... 9 more
Caused by:
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException:
java.lang.IllegalStateException: Item not found: 11
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:203)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:104)
at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:246)
at
org.apache.ignite.internal.processors.query.h2.database.io.H2ExtrasLeafIO.getLookupRow(H2ExtrasLeafIO.java:126)
at
org.apache.ignite.internal.processors.query.h2.database.io.H2ExtrasLeafIO.getLookupRow(H2ExtrasLeafIO.java:36)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:264)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:56)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4808)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4710)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5000(BPlusTree.java:4646)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:976)
... 10 more
Caused by: java.lang.IllegalStateException: Item not found: 11
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:153)
... 21 more
[21:44:26,853][SEVERE][pool-5-thread-16][FailureProcessor] An critical problem
with persistence data structures has detected. Please make backup for
persistence storage and WAL files for further analysis. Persistence storage
path: null WAL path: db/wal WAL archive path: db/wal/archive
[21:44:26,854][WARNING][pool-5-thread-16][FailureProcessor] No deadlocked
threads detected.
[21:44:26,975][WARNING][pool-5-thread-16][FailureProcessor] Thread dump at
2019/09/15 21:44:26 MSK
Thread [name="pool-5-thread-16", id=240, state=RUNNABLE, blockCnt=14, waitCnt=0]
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1335)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
