Hi all,
I have deployed 3 node Ignite cluster with native persistence on Kubernetes
and one of the node crashed with below error message,
*org.h2.message.DbException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on search row: Row@8cfe967[ key: epro_model_abcdKey
[idHash=822184780, hash=737706081, NE_ID=, NAME=], val: epro_model_abcd
[idHash=60444003, hash=1539928610, epro_ID=51, LONGITUDE=null,
DELETE_TIME=null, VENDOR=null, CREATE_TIME=2019-09-19T20:38:32.361929Z,
UPDATE_TIME=2019-09-19T20:40:05.821447Z, ADDITIONAL_INFO=null,
VALID_UNTIL=2019-11-18T20:38:32.362036Z, TYPE=null, LATITUDE=null], ver:
GridCacheVersion [topVer=180326822, order=1568925345552, nodeOrder=6] ][
51, 2019-09-19T20:38:32.361929Z, 2019-09-19T20:40:05.821447Z, null,
2019-11-18T20:38:32.362036Z, , , null, null, null, null, null ]"
[50000-197]|*
Please find attached file [index_corruption.txt] for complete backtrace.
It looks like the Index got corrupted, I am not sure what exactly caused
the index to corrupt. Any knows issues related to this?
In my cluster, many applications write into many tables simultaneously and
some queries run on many tables simultaneously and frequently application
deletes unwanted rows[old data] in the tables using *delete from table* SQL
operation.
Failed to reinitialize local partitions (rebalancing will be stopped):
GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=21,
minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=f3d7fb8c-0cda-42d0-a171-0155a171405b, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1,
192.168.*.*], sockAddrs=[/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500,
ignite-cluster-ignite-epro-0.ignite-service.default.svc.cluster.local/192.168.*.*:47500],
discPort=47500, order=21, intOrder=12, lastExchangeTime=1568926175782,
loc=true, ver=2.7.0#19700101-sha1:00000000, isClient=false], topVer=21,
nodeId8=f3d7fb8c, msg=null, type=NODE_JOINED, tstamp=1568926160054],
nodeId=f3d7fb8c, evt=NODE_JOINED] org.h2.message.DbException: General error:
"class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on search row: Row@8cfe967[ key: epro_model_abcdKey
[idHash=822184780, hash=737706081, NE_ID=, NAME=], val: epro_model_abcd
[idHash=60444003, hash=1539928610, epro_ID=51, LONGITUDE=null,
DELETE_TIME=null, VENDOR=null, CREATE_TIME=2019-09-19T20:38:32.361929Z,
UPDATE_TIME=2019-09-19T20:40:05.821447Z, ADDITIONAL_INFO=null,
VALID_UNTIL=2019-11-18T20:38:32.362036Z, TYPE=null, LATITUDE=null], ver:
GridCacheVersion [topVer=180326822, order=1568925345552, nodeOrder=6] ][ 51,
2019-09-19T20:38:32.361929Z, 2019-09-19T20:40:05.821447Z, null,
2019-11-18T20:38:32.362036Z, , , null, null, null, null, null ]" [50000-197]|
at org.h2.message.DbException.get(DbException.java:168)| at
org.h2.message.DbException.convert(DbException.java:307)| at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:348)|
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.addToIndex(GridH2Table.java:550)|
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:479)|
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:768)|
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1905)|
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:404)|
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:2633)|
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(IgniteCacheOffheapManagerImpl.java:2524)|
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.update(GridCacheOffheapManager.java:1759)|
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.java:443)|
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyUpdate(GridCacheDatabaseSharedManager.java:2653)|
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:2339)|
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:1628)|
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:1302)|
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1453)|
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:806)|
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2667)|
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2539)|
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)| at
java.lang.Thread.run(Thread.java:748)|Caused by: org.h2.jdbc.JdbcSQLException:
General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on search row: Row@8cfe967[ key: epro_model_abcdKey
[idHash=822184780, hash=737706081, NE_ID=, NAME=], val: epro_model_abcd
[idHash=60444003, hash=1539928610, epro_ID=51, LONGITUDE=null,
DELETE_TIME=null, VENDOR=null, CREATE_TIME=2019-09-19T20:38:32.361929Z,
UPDATE_TIME=2019-09-19T20:40:05.821447Z, ADDITIONAL_INFO=null,
VALID_UNTIL=2019-11-18T20:38:32.362036Z, TYPE=null, LATITUDE=null], ver:
GridCacheVersion [topVer=180326822, order=1568925345552, nodeOrder=6] ][ 51,
2019-09-19T20:38:32.361929Z, 2019-09-19T20:40:05.821447Z, null,
2019-11-18T20:38:32.362036Z, , , null, null, null, null, null ]" [50000-197]|
at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)|
... 22 more|Caused by: class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on search row: Row@8cfe967[ key: epro_model_abcdKey
[idHash=822184780, hash=737706081, NE_ID=, NAME=], val: epro_model_abcd
[idHash=60444003, hash=1539928610, epro_ID=51, LONGITUDE=null,
DELETE_TIME=null, VENDOR=null, CREATE_TIME=2019-09-19T20:38:32.361929Z,
UPDATE_TIME=2019-09-19T20:40:05.821447Z, ADDITIONAL_INFO=null,
VALID_UNTIL=2019-11-18T20:38:32.362036Z, TYPE=null, LATITUDE=null], ver:
GridCacheVersion [topVer=180326822, order=1568925345552, nodeOrder=6] ][ 51,
2019-09-19T20:38:32.361929Z, 2019-09-19T20:40:05.821447Z, null,
2019-11-18T20:38:32.362036Z, , , null, null, null, null, null ]| at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1969)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removex(BPlusTree.java:1764)|
at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:345)|
... 19 more|Caused by: java.lang.IllegalStateException: Item not found: 4|
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:342)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:451)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:493)|
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:156)|
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:108)|
at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62)|
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:217)|
at
org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO.getLookupRow(AbstractH2ExtrasLeafIO.java:154)|
at
org.apache.ignite.internal.processors.query.h2.database.io.AbstractH2ExtrasLeafIO.getLookupRow(AbstractH2ExtrasLeafIO.java:36)|
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:248)|
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:55)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:5118)|
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:327)|
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:55)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:5105)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:5025)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5615)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5600)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:159)|
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:2004)|
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1934)|
... 21 more"}
