Sergey Kozlov created IGNITE-10052: -------------------------------------- Summary: Restart node during TX causes vacuum error Key: IGNITE-10052 URL: https://issues.apache.org/jira/browse/IGNITE-10052 Project: Ignite Issue Type: Bug Components: mvcc Affects Versions: 2.7 Reporter: Sergey Kozlov Fix For: 2.7
1. Start 2 nodes with PDS, activate , start {{sqlline}} 2. Create table {noformat} 0: jdbc:ignite:thin://127.0.0.1/> create table t1 (a int, b int, primary key(a)) with "atomicity=TRANSACTIONAL_SNAPSHOT, backups=1"; No rows affected (0,294 seconds) {noformat} 3. Open TX: {noformat} 0: jdbc:ignite:thin://127.0.0.1/> begin; No rows affected (0,007 seconds) 0: jdbc:ignite:thin://127.0.0.1/> insert into t1 values (1,1); 1 row affected (0,112 seconds) {noformat} 4. Stop and then start 2nd node 5. Rollback TX and check the table data (no rows added): {noformat} 0: jdbc:ignite:thin://127.0.0.1/> rollback; No rows affected (0,011 seconds) 0: jdbc:ignite:thin://127.0.0.1/> select * from t1; +--------------------------------+--------------------------------+ | A | B | +--------------------------------+--------------------------------+ +--------------------------------+--------------------------------+ No rows selected (0,067 seconds) {noformat} 6. Start second TX that throws the excpetion: {noformat} 0: jdbc:ignite:thin://127.0.0.1/> begin; No rows affected (0,001 seconds) 0: jdbc:ignite:thin://127.0.0.1/> insert into t1 values (1,1); Error: Failed to run update. Runtime failure on bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchRow []] (state=50 000,code=1) java.sql.SQLException: Failed to run update. Runtime failure on bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchR ow []] at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:750) at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:212) at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute(JdbcThinStatement.java:475) at sqlline.Commands.execute(Commands.java:823) at sqlline.Commands.sql(Commands.java:733) at sqlline.SqlLine.dispatch(SqlLine.java:795) at sqlline.SqlLine.begin(SqlLine.java:668) at sqlline.SqlLine.start(SqlLine.java:373) at sqlline.SqlLine.main(SqlLine.java:265) 0: jdbc:ignite:thin://127.0.0.1/> {noformat} The server nodes print out: {noformat} [18:46:32,136][SEVERE][jdbc-request-handler-worker-#62][DmlStatementsProcessor] Error during update [localNodeId=319a2fda-1315-4bdf-8647-7ddee2d3342e] class org.apache.ignite.IgniteCheckedException: Runtime failure on bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchRow []] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1061) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:1919) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.mvccUpdate(GridCacheOffheapManager.java:1840) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:530) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.mvccSet(GridCacheMapEntry.java:1104) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.continueLoop(GridDhtTxAbstractEnlistFuture.java:460) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.init(GridDhtTxAbstractEnlistFuture.java:368) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxQueryResultsEnlistRequest(GridDhtTransactionalCacheAdapter.java:2000) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$700(GridDhtTransactionalCacheAdapter.java:112) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$12.apply(GridDhtTransactionalCacheAdapter.java:215) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$12.apply(GridDhtTransactionalCacheAdapter.java:213) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Caused by: class org.apache.ignite.internal.transactions.IgniteTxMvccVersionCheckedException: Unexpected state: [state=0, rowVer=1540827914710:6:1, txVer=1540827914710:21:1, localNodeId=4d66a9d5-4352-495c-9c64-405ccdf8a079] at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:310) at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:297) at org.apache.ignite.internal.processors.cache.tree.mvcc.data.MvccUpdateDataRow.visit(MvccUpdateDataRow.java:413) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3021) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.init(BPlusTree.java:2990) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.notFound(BPlusTree.java:2946) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:306) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5615) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5600) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:159) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2585) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doVisit(BPlusTree.java:2552) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8200(BPlusTree.java:90) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3148) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.access$5100(BPlusTree.java:2908) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1058) ... 23 more [18:46:32,138][SEVERE][jdbc-request-handler-worker-#62][JdbcRequestHandler] Failed to execute SQL query [reqId=0, req=JdbcQueryExecuteRequest [schemaName=PUBLIC, pageSize=1024, maxRows=0, sqlQry=insert into t1 values (1,1), args=Object[] [], stmtType=ANY_STATEMENT_TYPE, autoCommit=true]] class org.apache.ignite.internal.processors.query.IgniteSQLException: Failed to run update. Runtime failure on bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchRow []] at org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.executeUpdateStatement(DmlStatementsProcessor.java:593) at org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.updateSqlFields(DmlStatementsProcessor.java:185) at org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.updateSqlFieldsDistributed(DmlStatementsProcessor.java:358) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.doRunPrepared(IgniteH2Indexing.java:2239) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.querySqlFields(IgniteH2Indexing.java:2158) at org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2141) at org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2136) at org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36) at org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2713) at org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2150) at org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.executeQuery(JdbcRequestHandler.java:511) at org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.doHandle(JdbcRequestHandler.java:245) at org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandlerWorker.body(JdbcRequestHandlerWorker.java:90) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchRow []] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1061) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:1919) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.mvccUpdate(GridCacheOffheapManager.java:1840) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:530) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.mvccSet(GridCacheMapEntry.java:1104) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.continueLoop(GridDhtTxAbstractEnlistFuture.java:460) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.init(GridDhtTxAbstractEnlistFuture.java:368) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxQueryResultsEnlistRequest(GridDhtTransactionalCacheAdapter.java:2000) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$700(GridDhtTransactionalCacheAdapter.java:112) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$12.apply(GridDhtTransactionalCacheAdapter.java:215) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$12.apply(GridDhtTransactionalCacheAdapter.java:213) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505) ... 2 more Caused by: class org.apache.ignite.internal.transactions.IgniteTxMvccVersionCheckedException: Unexpected state: [state=0, rowVer=1540827914710:6:1, txVer=1540827914710:21:1, localNodeId=4d66a9d5-4352-495c-9c64-405ccdf8a079] at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:310) at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:297) at org.apache.ignite.internal.processors.cache.tree.mvcc.data.MvccUpdateDataRow.visit(MvccUpdateDataRow.java:413) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3021) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.init(BPlusTree.java:2990) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.notFound(BPlusTree.java:2946) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:306) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5615) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5600) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:159) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2585) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doVisit(BPlusTree.java:2552) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8200(BPlusTree.java:90) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3148) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.access$5100(BPlusTree.java:2908) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1058) ... 23 more {noformat} {noformat} [18:46:20,439][SEVERE][vacuum-scheduler-#47][MvccProcessorImpl] Vacuum error. class org.apache.ignite.internal.transactions.IgniteTxMvccVersionCheckedException: Unexpected state: [state=0, rowVer=1540827914710:6:536870913, localNodeId=4d66a9d5-4352-495c-9c64-405ccdf8a079] at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:310) at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:281) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$MvccUpdateTxStateHintHandler.run(IgniteCacheOffheapManagerImpl.java:3206) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$MvccUpdateTxStateHintHandler.run(IgniteCacheOffheapManagerImpl.java:3180) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:279) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:256) at org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.updateDataRow(AbstractFreeList.java:555) at org.apache.ignite.internal.processors.cache.persistence.RowStore.updateDataRow(RowStore.java:143) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.updateTxState(IgniteCacheOffheapManagerImpl.java:2449) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.updateTxState(GridCacheOffheapManager.java:1917) at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.cleanup(MvccProcessorImpl.java:2415) at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.processPartition(MvccProcessorImpl.java:2300) at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.body(MvccProcessorImpl.java:2198) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [18:46:20,582][INFO][db-checkpoint-thread-#51][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointLockWait=0ms, checkpointLockHoldTime=12ms, reason='timeout'] [18:46:20,988][INFO][exchange-worker-#43][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], mvccCrd=MvccCoordinator [nodeId=319a2fda-1315-4bdf-8647-7ddee2d3342e, crdVer=1540827914710, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=DISCOVERY_CUSTOM_EVT, evtNode=319a2fda-1315-4bdf-8647-7ddee2d3342e, customEvt=CacheAffinityChangeMessage [id=f4aff70c661-30eb588f-bc2f-4094-86c4-e7070c6fcb1e, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], exchId=null, partsMsg=null, exchangeNeeded=true], allowMerge=false] [18:46:20,998][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [18:46:21,004][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=319a2fda-1315-4bdf-8647-7ddee2d3342e, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1540827969985, loc=false, ver=2.7.1#20181029-sha1:83c27088, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]] [18:46:21,004][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=LOCAL] [18:46:21,005][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Logical recovery performed in 0 ms. [18:46:21,015][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 3 ms. [18:46:21,016][INFO][exchange-worker-#43][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], crd=false] [18:46:21,035][INFO][sys-#64][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=319a2fda-1315-4bdf-8647-7ddee2d3342e, resVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]] [18:46:21,039][INFO][sys-#64][GridDhtPartitionsExchangeFuture] Affinity changes applied in 0 ms. [18:46:21,044][INFO][sys-#64][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 2 ms. [18:46:21,044][INFO][sys-#64][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], err=null] [18:46:21,050][INFO][sys-#64][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=4d66a9d5-4352-495c-9c64-405ccdf8a079, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=319a2fda-1315-4bdf-8647-7ddee2d3342e, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1540827969985, loc=false, ver=2.7.1#20181029-sha1:83c27088, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], durationFromInit=66] [18:46:21,052][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] Skipping rebalancing (no affinity changes) [top=AffinityTopologyVersion [topVer=4, minorTopVer=1], rebTopVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], evt=DISCOVERY_CUSTOM_EVT, evtNode=319a2fda-1315-4bdf-8647-7ddee2d3342e, client=false] [18:46:21,586][INFO][db-checkpoint-thread-#51][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointLockWait=0ms, checkpointLockHoldTime=10ms, reason='timeout'] [18:46:22,595][INFO][db-checkpoint-thread-#51][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointLockWait=0ms, checkpointLockHoldTime=9ms, reason='timeout'] [18:46:24,602][INFO][db-checkpoint-thread-#51][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointLockWait=0ms, checkpointLockHoldTime=7ms, reason='timeout'] [18:46:25,406][SEVERE][vacuum-scheduler-#47][MvccProcessorImpl] Vacuum error. class org.apache.ignite.internal.transactions.IgniteTxMvccVersionCheckedException: Unexpected state: [state=0, rowVer=1540827914710:6:536870913, localNodeId=4d66a9d5-4352-495c-9c64-405ccdf8a079] at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:310) at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:281) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$MvccUpdateTxStateHintHandler.run(IgniteCacheOffheapManagerImpl.java:3206) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$MvccUpdateTxStateHintHandler.run(IgniteCacheOffheapManagerImpl.java:3180) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:279) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:256) at org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.updateDataRow(AbstractFreeList.java:555) at org.apache.ignite.internal.processors.cache.persistence.RowStore.updateDataRow(RowStore.java:143) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.updateTxState(IgniteCacheOffheapManagerImpl.java:2449) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.updateTxState(GridCacheOffheapManager.java:1917) at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.cleanup(MvccProcessorImpl.java:2415) at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.processPartition(MvccProcessorImpl.java:2300) at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.body(MvccProcessorImpl.java:2198) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [18:46:26,612][INFO][db-checkpoint-thread-#51][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointLockWait=0ms, checkpointLockHoldTime=5ms, reason='timeout'] [18:46:29,632][INFO][db-checkpoint-thread-#51][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointLockWait=0ms, checkpointLockHoldTime=6ms, reason='timeout'] [18:46:30,406][SEVERE][vacuum-scheduler-#47][MvccProcessorImpl] Vacuum error. class org.apache.ignite.internal.transactions.IgniteTxMvccVersionCheckedException: Unexpected state: [state=0, rowVer=1540827914710:6:536870913, localNodeId=4d66a9d5-4352-495c-9c64-405ccdf8a079] at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:310) at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:281) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$MvccUpdateTxStateHintHandler.run(IgniteCacheOffheapManagerImpl.java:3206) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$MvccUpdateTxStateHintHandler.run(IgniteCacheOffheapManagerImpl.java:3180) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:279) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:256) at org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.updateDataRow(AbstractFreeList.java:555) at org.apache.ignite.internal.processors.cache.persistence.RowStore.updateDataRow(RowStore.java:143) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.updateTxState(IgniteCacheOffheapManagerImpl.java:2449) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.updateTxState(GridCacheOffheapManager.java:1917) at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.cleanup(MvccProcessorImpl.java:2415) at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.processPartition(MvccProcessorImpl.java:2300) at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.body(MvccProcessorImpl.java:2198) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [18:46:35,406][SEVERE][vacuum-scheduler-#47][MvccProcessorImpl] Vacuum error. class org.apache.ignite.internal.transactions.IgniteTxMvccVersionCheckedException: Unexpected state: [state=0, rowVer=1540827914710:6:536870913, localNodeId=4d66a9d5-4352-495c-9c64-405ccdf8a079] at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:310) at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:281) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$MvccUpdateTxStateHintHandler.run(IgniteCacheOffheapManagerImpl.java:3206) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$MvccUpdateTxStateHintHandler.run(IgniteCacheOffheapManagerImpl.java:3180) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:279) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:256) at org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.updateDataRow(AbstractFreeList.java:555) at org.apache.ignite.internal.processors.cache.persistence.RowStore.updateDataRow(RowStore.java:143) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.updateTxState(IgniteCacheOffheapManagerImpl.java:2449) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.updateTxState(GridCacheOffheapManager.java:1917) at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.cleanup(MvccProcessorImpl.java:2415) at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.processPartition(MvccProcessorImpl.java:2300) at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.body(MvccProcessorImpl.java:2198) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [18:46:38,674][INFO][db-checkpoint-thread-#51][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointLockWait=0ms, checkpointLockHoldTime=4ms, reason='timeout'] {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)