Re: Item not found, B+Tree is corrupted and critical error detected after add nodes to baseline topology

2020-09-07 Thread Ilya Kasnacheev 
Hello!

This looks like a PDS corruption: an entry was expected to present in data
page, but was not found.

I recommend removing PDS files of that node and re-adding it to baseline
topology, if you can survive the data loss.

Regards,
-- 
Ilya Kasnacheev


ср, 26 авг. 2020 г. в 05:27, Steven Zheng :

> Hi community,
> Currently I have 25 nodes in my ignite cluster and all of them were added
> into the baseline, and I was trying to add another 5 nodes into it. The
> data in my cluster is about 8TB and the persistence is enabled.
> At first I start all the 5 nodes; then execute in the command line:
> ```
> bin/control.sh --baseline add ${my_node_id}
> ```
> Meanwhile, there is still read/write workloads on the cluster.
>
> After a few minutes, a critical error detected , several nodes crashed and
> emits the logs like this(json format):
> ```
> {
>   "message": "Critical system error detected. Will be handled accordingly
> to configured handler [hnd=StopNodeFailureHandler
> [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
> o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is
> corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570,
> val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE,
> indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[
> key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
> [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
> [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
> ][ 25323111322, , 0, 1596023962011 ",
>   "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
>   "thrown": {
> "localizedMessage": "B+Tree is corrupted [pages(groupId,
> pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]],
> cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY,
> msg=Runtime failure on search row: Row@d00bc1[ key:
> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
> [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
> [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
> ][ 25323111322, , 0, 1596023962011 ]]",
> "message": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple
> [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086,
> cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on
> search row: Row@d00bc1[ key:
> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
> [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
> [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
> ][ 25323111322, , 0, 1596023962011 ]]",
> "commonElementCount": 0,
> "cause": {
>   "localizedMessage": "java.lang.IllegalStateException: Item not
> found: 27",
>   "message": "java.lang.IllegalStateException: Item not found: 27",
>   "commonElementCount": 21,
>   "cause": {
> "localizedMessage": "Item not found: 27",
> "message": "Item not found: 27",
> "commonElementCount": 21,
> "name": "java.lang.IllegalStateException",
> "extendedStackTrace": [{
> "line": 351,
> "method": "findIndirectItemIndex",
> "exact": false,
> "file": "AbstractDataPageIO.java",
> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
> "class":
> "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
> "version": "2.8.0-SNAPSHOT"
>   },
>   {
> "line": 459,
> "method": "getDataOffset",
> "exact": false,
> "file": "AbstractDataPageIO.java",
> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
> "class":
> "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
> "version": "2.8.0-SNAPSHOT"
>   },
>   {
> "line": 501,
> "method": "readPayload",
> "exact": false,
> "file": "AbstractDataPageIO.java",
> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
> "class":
> "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
> "version": "2.8.0-SNAPSHOT"
>   },
>   {
> "line": 325,
> "method": "readIncomplete",
> "exact": false,
> "file": "CacheDataRowAdapter.java",
> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
> "class":
> "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
>

Re: Item not found, B+Tree is corrupted and critical error detected after add nodes to baseline topology

2020-08-31 Thread Evgenii Zhuravlev 
Can you export logs from your system in a different format? I don't really
understand how it can be analyzed by a person in this format.

Evgenii

пн, 31 авг. 2020 г. в 00:36, Steven Zheng :

> Thanks evgenii, but origin log is not been saved and I can only provide
> json format logs.
> Best Regards,
> -
> Steven Zheng
> E-mail: closee...@gmail.com
>
>
> Evgenii Zhuravlev  于2020年8月28日周五 上午6:45写道：
>
>> Hi,
>>
>> Can you attach logs in normal format? It's really hard to read it. Also,
>> please attach full logs from nodes, not only the stacktrace.
>>
>> Thanks,
>> Evgenii
>>
>> вт, 25 авг. 2020 г. в 19:27, Steven Zheng :
>>
>>> Hi community,
>>> Currently I have 25 nodes in my ignite cluster and all of them were
>>> added into the baseline, and I was trying to add another 5 nodes into it.
>>> The data in my cluster is about 8TB and the persistence is enabled.
>>> At first I start all the 5 nodes; then execute in the command line:
>>> ```
>>> bin/control.sh --baseline add ${my_node_id}
>>> ```
>>> Meanwhile, there is still read/write workloads on the cluster.
>>>
>>> After a few minutes, a critical error detected , several nodes crashed
>>> and emits the logs like this(json format):
>>> ```
>>> {
>>>   "message": "Critical system error detected. Will be handled
>>> accordingly to configured handler [hnd=StopNodeFailureHandler
>>> [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
>>> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
>>> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
>>> o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is
>>> corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570,
>>> val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE,
>>> indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[
>>> key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
>>> [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
>>> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
>>> [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
>>> ][ 25323111322, , 0, 1596023962011 ",
>>>   "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
>>>   "thrown": {
>>> "localizedMessage": "B+Tree is corrupted [pages(groupId,
>>> pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]],
>>> cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY,
>>> msg=Runtime failure on search row: Row@d00bc1[ key:
>>> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
>>> [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
>>> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
>>> [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
>>> ][ 25323111322, , 0, 1596023962011 ]]",
>>> "message": "B+Tree is corrupted [pages(groupId,
>>> pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]],
>>> cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY,
>>> msg=Runtime failure on search row: Row@d00bc1[ key:
>>> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
>>> [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
>>> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
>>> [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
>>> ][ 25323111322, , 0, 1596023962011 ]]",
>>> "commonElementCount": 0,
>>> "cause": {
>>>   "localizedMessage": "java.lang.IllegalStateException: Item not
>>> found: 27",
>>>   "message": "java.lang.IllegalStateException: Item not found: 27",
>>>   "commonElementCount": 21,
>>>   "cause": {
>>> "localizedMessage": "Item not found: 27",
>>> "message": "Item not found: 27",
>>> "commonElementCount": 21,
>>> "name": "java.lang.IllegalStateException",
>>> "extendedStackTrace": [{
>>> "line": 351,
>>> "method": "findIndirectItemIndex",
>>> "exact": false,
>>> "file": "AbstractDataPageIO.java",
>>> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
>>> "class":
>>> "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
>>> "version": "2.8.0-SNAPSHOT"
>>>   },
>>>   {
>>> "line": 459,
>>> "method": "getDataOffset",
>>> "exact": false,
>>> "file": "AbstractDataPageIO.java",
>>> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
>>> "class":
>>> "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
>>> "version": "2.8.0-SNAPSHOT"
>>>   },
>>>   {
>>> "line": 501,
>>> "method": "readPayload",
>>> "exact": false,
>>> "file": "AbstractDataPageIO.java",
>>>

Re: Item not found, B+Tree is corrupted and critical error detected after add nodes to baseline topology

2020-08-31 Thread Steven Zheng 
Thanks evgenii, but origin log is not been saved and I can only provide
json format logs.
Best Regards,
-
Steven Zheng
E-mail: closee...@gmail.com


Evgenii Zhuravlev  于2020年8月28日周五 上午6:45写道：

> Hi,
>
> Can you attach logs in normal format? It's really hard to read it. Also,
> please attach full logs from nodes, not only the stacktrace.
>
> Thanks,
> Evgenii
>
> вт, 25 авг. 2020 г. в 19:27, Steven Zheng :
>
>> Hi community,
>> Currently I have 25 nodes in my ignite cluster and all of them were added
>> into the baseline, and I was trying to add another 5 nodes into it. The
>> data in my cluster is about 8TB and the persistence is enabled.
>> At first I start all the 5 nodes; then execute in the command line:
>> ```
>> bin/control.sh --baseline add ${my_node_id}
>> ```
>> Meanwhile, there is still read/write workloads on the cluster.
>>
>> After a few minutes, a critical error detected , several nodes crashed
>> and emits the logs like this(json format):
>> ```
>> {
>>   "message": "Critical system error detected. Will be handled accordingly
>> to configured handler [hnd=StopNodeFailureHandler
>> [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
>> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
>> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
>> o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is
>> corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570,
>> val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE,
>> indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[
>> key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
>> [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
>> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
>> [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
>> ][ 25323111322, , 0, 1596023962011 ",
>>   "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
>>   "thrown": {
>> "localizedMessage": "B+Tree is corrupted [pages(groupId,
>> pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]],
>> cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY,
>> msg=Runtime failure on search row: Row@d00bc1[ key:
>> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
>> [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
>> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
>> [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
>> ][ 25323111322, , 0, 1596023962011 ]]",
>> "message": "B+Tree is corrupted [pages(groupId,
>> pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]],
>> cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY,
>> msg=Runtime failure on search row: Row@d00bc1[ key:
>> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
>> [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
>> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
>> [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
>> ][ 25323111322, , 0, 1596023962011 ]]",
>> "commonElementCount": 0,
>> "cause": {
>>   "localizedMessage": "java.lang.IllegalStateException: Item not
>> found: 27",
>>   "message": "java.lang.IllegalStateException: Item not found: 27",
>>   "commonElementCount": 21,
>>   "cause": {
>> "localizedMessage": "Item not found: 27",
>> "message": "Item not found: 27",
>> "commonElementCount": 21,
>> "name": "java.lang.IllegalStateException",
>> "extendedStackTrace": [{
>> "line": 351,
>> "method": "findIndirectItemIndex",
>> "exact": false,
>> "file": "AbstractDataPageIO.java",
>> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
>> "class":
>> "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
>> "version": "2.8.0-SNAPSHOT"
>>   },
>>   {
>> "line": 459,
>> "method": "getDataOffset",
>> "exact": false,
>> "file": "AbstractDataPageIO.java",
>> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
>> "class":
>> "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
>> "version": "2.8.0-SNAPSHOT"
>>   },
>>   {
>> "line": 501,
>> "method": "readPayload",
>> "exact": false,
>> "file": "AbstractDataPageIO.java",
>> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
>> "class":
>> "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
>> "version": "2.8.0-SNAPSHOT"
>>   },
>>   {
>> "line": 325,
>> "method": "readIncomplete",

Re: Item not found, B+Tree is corrupted and critical error detected after add nodes to baseline topology

2020-08-27 Thread Evgenii Zhuravlev 
Hi,

Can you attach logs in normal format? It's really hard to read it. Also,
please attach full logs from nodes, not only the stacktrace.

Thanks,
Evgenii

вт, 25 авг. 2020 г. в 19:27, Steven Zheng :

> Hi community,
> Currently I have 25 nodes in my ignite cluster and all of them were added
> into the baseline, and I was trying to add another 5 nodes into it. The
> data in my cluster is about 8TB and the persistence is enabled.
> At first I start all the 5 nodes; then execute in the command line:
> ```
> bin/control.sh --baseline add ${my_node_id}
> ```
> Meanwhile, there is still read/write workloads on the cluster.
>
> After a few minutes, a critical error detected , several nodes crashed and
> emits the logs like this(json format):
> ```
> {
>   "message": "Critical system error detected. Will be handled accordingly
> to configured handler [hnd=StopNodeFailureHandler
> [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
> o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is
> corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570,
> val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE,
> indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[
> key: SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
> [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
> [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
> ][ 25323111322, , 0, 1596023962011 ",
>   "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
>   "thrown": {
> "localizedMessage": "B+Tree is corrupted [pages(groupId,
> pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]],
> cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY,
> msg=Runtime failure on search row: Row@d00bc1[ key:
> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
> [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
> [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
> ][ 25323111322, , 0, 1596023962011 ]]",
> "message": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple
> [val1=-1526563570, val2=844420635165881]], cacheId=-2021712086,
> cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on
> search row: Row@d00bc1[ key:
> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
> [idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
> SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
> [idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
> ][ 25323111322, , 0, 1596023962011 ]]",
> "commonElementCount": 0,
> "cause": {
>   "localizedMessage": "java.lang.IllegalStateException: Item not
> found: 27",
>   "message": "java.lang.IllegalStateException: Item not found: 27",
>   "commonElementCount": 21,
>   "cause": {
> "localizedMessage": "Item not found: 27",
> "message": "Item not found: 27",
> "commonElementCount": 21,
> "name": "java.lang.IllegalStateException",
> "extendedStackTrace": [{
> "line": 351,
> "method": "findIndirectItemIndex",
> "exact": false,
> "file": "AbstractDataPageIO.java",
> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
> "class":
> "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
> "version": "2.8.0-SNAPSHOT"
>   },
>   {
> "line": 459,
> "method": "getDataOffset",
> "exact": false,
> "file": "AbstractDataPageIO.java",
> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
> "class":
> "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
> "version": "2.8.0-SNAPSHOT"
>   },
>   {
> "line": 501,
> "method": "readPayload",
> "exact": false,
> "file": "AbstractDataPageIO.java",
> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
> "class":
> "org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
> "version": "2.8.0-SNAPSHOT"
>   },
>   {
> "line": 325,
> "method": "readIncomplete",
> "exact": false,
> "file": "CacheDataRowAdapter.java",
> "location": "ignite-core-2.8.0-SNAPSHOT.jar",
> "class":
> "org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
> "version": "2.8.0-SNAPSHOT"
>   },
>   {
> "line": 261,
> "

Item not found, B+Tree is corrupted and critical error detected after add nodes to baseline topology

2020-08-25 Thread Steven Zheng 
Hi community,
Currently I have 25 nodes in my ignite cluster and all of them were added
into the baseline, and I was trying to add another 5 nodes into it. The
data in my cluster is about 8TB and the persistence is enabled.
At first I start all the 5 nodes; then execute in the command line:
```
bin/control.sh --baseline add ${my_node_id}
```
Meanwhile, there is still read/write workloads on the cluster.

After a few minutes, a critical error detected , several nodes crashed and
emits the logs like this(json format):
```
{
  "message": "Critical system error detected. Will be handled accordingly
to configured handler [hnd=StopNodeFailureHandler
[super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is
corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1526563570,
val2=844420635165881]], cacheId=-2021712086, cacheName=MY_SQL_TABLE,
indexName=AFFINITY_KEY, msg=Runtime failure on search row: Row@d00bc1[ key:
SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
[idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
[idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
][ 25323111322, , 0, 1596023962011 ",
  "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
  "thrown": {
"localizedMessage": "B+Tree is corrupted [pages(groupId,
pageId)=[IgniteBiTuple [val1=-1526563570, val2=844420635165881]],
cacheId=-2021712086, cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY,
msg=Runtime failure on search row: Row@d00bc1[ key:
SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
[idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
[idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
][ 25323111322, , 0, 1596023962011 ]]",
"message": "B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple
[val1=-1526563570, val2=844420635165881]], cacheId=-2021712086,
cacheName=MY_SQL_TABLE, indexName=AFFINITY_KEY, msg=Runtime failure on
search row: Row@d00bc1[ key:
SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf_KEY
[idHash=2015873411, hash=-2113698004, CARDNUMBER=25323111322, IP=], val:
SQL_FEAT_MY_SQL_TABLE_7f8539ef_08a1_421e_a0b8_60f80cdbbbdf
[idHash=1729397221, hash=2090272279, MY_BIZ_UID=0, ETL_TIME=1596023962011]
][ 25323111322, , 0, 1596023962011 ]]",
"commonElementCount": 0,
"cause": {
  "localizedMessage": "java.lang.IllegalStateException: Item not found:
27",
  "message": "java.lang.IllegalStateException: Item not found: 27",
  "commonElementCount": 21,
  "cause": {
"localizedMessage": "Item not found: 27",
"message": "Item not found: 27",
"commonElementCount": 21,
"name": "java.lang.IllegalStateException",
"extendedStackTrace": [{
"line": 351,
"method": "findIndirectItemIndex",
"exact": false,
"file": "AbstractDataPageIO.java",
"location": "ignite-core-2.8.0-SNAPSHOT.jar",
"class":
"org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
"version": "2.8.0-SNAPSHOT"
  },
  {
"line": 459,
"method": "getDataOffset",
"exact": false,
"file": "AbstractDataPageIO.java",
"location": "ignite-core-2.8.0-SNAPSHOT.jar",
"class":
"org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
"version": "2.8.0-SNAPSHOT"
  },
  {
"line": 501,
"method": "readPayload",
"exact": false,
"file": "AbstractDataPageIO.java",
"location": "ignite-core-2.8.0-SNAPSHOT.jar",
"class":
"org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO",
"version": "2.8.0-SNAPSHOT"
  },
  {
"line": 325,
"method": "readIncomplete",
"exact": false,
"file": "CacheDataRowAdapter.java",
"location": "ignite-core-2.8.0-SNAPSHOT.jar",
"class":
"org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
"version": "2.8.0-SNAPSHOT"
  },
  {
"line": 261,
"method": "doInitFromLink",
"exact": false,
"file": "CacheDataRowAdapter.java",
"location": "ignite-core-2.8.0-SNAPSHOT.jar",
"class":
"org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter",
"version": "2.8.0-SNAPSHOT"
  },
  {
"line": 160,
"method": "initFromLink",
"exa