[ https://issues.apache.org/jira/browse/IGNITE-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16865448#comment-16865448 ]
Andrew Mashenkov commented on IGNITE-11623: ------------------------------------------- Single-threaded tests (like testRebalancingDuringLoad_**_1_1 ) looks ok. But multi-threaded tests (like testRebalancingDuringLoad_**_8_16) are broken and fails on master. Possible reasons: * Race b/w transactions. New transaction starts before previous one update it's state in TxLog. * Race in Mvcc coordinator. Somehow, vacuum get cleanup version too early. * Missed TxLog WAL record. * Vacuum didn't clear rolled back version. > MVCC: testRebalancingDuringLoad_N tests are flacky in master > ------------------------------------------------------------ > > Key: IGNITE-11623 > URL: https://issues.apache.org/jira/browse/IGNITE-11623 > Project: Ignite > Issue Type: Bug > Reporter: Amelchev Nikita > Priority: Major > Labels: MakeTeamcityGreenAgain > > The following tests are flacky in master: > IgnitePdsMvccTestSuite3: > IgnitePdsContinuousRestartTest.testRebalancingDuringLoad_10_10_1_1 ( [Test > history|https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-5373044394008664141&branch=%3Cdefault%3E&tab=testDetails] > ) > testRebalancingDuringLoad_1000_20000_8_16 ( [Test > history|https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&buildTypeId=&tab=testDetails&testNameId=8776582252343254845&order=TEST_STATUS_DESC&branch_IgniteTests24Java8=%3Cdefault%3E&itemsCount=50]) > testRebalancingDuringLoad_1000_500_8_1 ([Test > history|https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=4172157337542230771&branch=%3Cdefault%3E&tab=testDetails]) > IgniteClusterActivateDeactivateTestWithPersistence.testDeactivateDuringEvictionAndRebalance > ([Test > history|https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=4619901385813836807&branch=%3Cdefault%3E&tab=testDetails]) > According to the logs, the cause of the fall: > *IgniteTxUnexpectedStateCheckedException: Unexpected state* > {noformat} > java.lang.AssertionError: Unexpected exception: javax.cache.CacheException: > class org.apache.ignite.transactions.TransactionRollbackException: > Transaction has been rolled back: > be9b08fa961-00000000-09d4-4a7d-0000-000000000001 > at > org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1318) > at > org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:2064) > at > org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1371) > at > org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.putAll(GatewayProtectedCacheProxy.java:866) > at > org.apache.ignite.internal.processors.cache.persistence.IgnitePdsContinuousRestartTest$1.call(IgnitePdsContinuousRestartTest.java:280) > at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) > Caused by: class org.apache.ignite.transactions.TransactionRollbackException: > Transaction has been rolled back: > be9b08fa961-00000000-09d4-4a7d-0000-000000000001 > at org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:935) > at org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:933) > ... 6 more > Caused by: class > org.apache.ignite.internal.transactions.IgniteTxRollbackCheckedException: > Transaction has been rolled back: > be9b08fa961-00000000-09d4-4a7d-0000-000000000001 > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4297) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAll0(GridCacheAdapter.java:3000) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAll(GridCacheAdapter.java:2989) > at > org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1368) > ... 3 more > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to update > backup node: [localNodeId=44bb5db4-5317-4ab1-8379-16f3b9200003, > remoteNodeId=08a44c02-ebe8-46e8-a7ca-890c62500001] > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.onResult(GridDhtTxAbstractEnlistFuture.java:999) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistResponse(GridDhtTransactionalCacheAdapter.java:2350) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$1200(GridDhtTransactionalCacheAdapter.java:112) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$18.apply(GridDhtTransactionalCacheAdapter.java:257) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$18.apply(GridDhtTransactionalCacheAdapter.java:255) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1126) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1561) > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1189) > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) > at > org.apache.ignite.internal.managers.communication.GridIoManager$8.run(GridIoManager.java:1086) > at > org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:550) > at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on > bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchRow []] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1069) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:1938) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.mvccUpdate(GridCacheOffheapManager.java:2027) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:523) > at > org.apache.ignite.internal.processors.cache.GridCacheMapEntry.mvccSet(GridCacheMapEntry.java:1115) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.mvccEnlistBatch(IgniteTxHandler.java:1937) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistRequest(GridDhtTransactionalCacheAdapter.java:2299) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$1100(GridDhtTransactionalCacheAdapter.java:112) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$17.apply(GridDhtTransactionalCacheAdapter.java:250) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$17.apply(GridDhtTransactionalCacheAdapter.java:248) > ... 13 more > Caused by: class > org.apache.ignite.internal.transactions.IgniteTxUnexpectedStateCheckedException: > Unexpected state: [state=0, rowVer=1553427560308:27927:1, > txVer=1553427560308:29840:1, localNodeId=08a44c02-ebe8-46e8-a7ca-890c62500001] > at > org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:328) > at > org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:315) > at > org.apache.ignite.internal.processors.cache.tree.mvcc.data.MvccUpdateDataRow.visit(MvccUpdateDataRow.java:416) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3060) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.init(BPlusTree.java:3030) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.notFound(BPlusTree.java:2986) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:308) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5643) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:273) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5629) > at > org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:168) > at > org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:349) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read(BPlusTree.java:5830) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2626) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2635) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doVisit(BPlusTree.java:2593) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$7800(BPlusTree.java:92) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3187) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.access$4900(BPlusTree.java:2948) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1066) > ... 22 more > java.lang.AssertionError: Unexpected exception: javax.cache.CacheException: > class org.apache.ignite.transactions.TransactionRollbackException: > Transaction has been rolled back: > 2dbd8eba961-00000000-09d3-5d15-0000-000000000001 > at > org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1318) > at > org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:2064) > at > org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1371) > at > org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.putAll(GatewayProtectedCacheProxy.java:866) > at > org.apache.ignite.internal.processors.cache.persistence.IgnitePdsContinuousRestartTest$1.call(IgnitePdsContinuousRestartTest.java:280) > at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) > Caused by: class org.apache.ignite.transactions.TransactionRollbackException: > Transaction has been rolled back: > 2dbd8eba961-00000000-09d3-5d15-0000-000000000001 > at org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:935) > at org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:933) > ... 6 more > Caused by: class > org.apache.ignite.internal.transactions.IgniteTxRollbackCheckedException: > Transaction has been rolled back: > 2dbd8eba961-00000000-09d3-5d15-0000-000000000001 > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4283) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAll0(GridCacheAdapter.java:2986) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAll(GridCacheAdapter.java:2975) > at > org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1368) > ... 3 more > Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on > bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchRow []] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1069) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:1958) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.mvccUpdate(GridCacheOffheapManager.java:2026) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:536) > at > org.apache.ignite.internal.processors.cache.GridCacheMapEntry.mvccSet(GridCacheMapEntry.java:1117) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.continueLoop(GridDhtTxAbstractEnlistFuture.java:461) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.init(GridDhtTxAbstractEnlistFuture.java:363) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxEnlistRequest(GridDhtTransactionalCacheAdapter.java:2070) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:112) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:229) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:227) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1126) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1$2$1.run(GridCacheIoManager.java:288) > at > org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:550) > at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Caused by: class > org.apache.ignite.internal.transactions.IgniteTxUnexpectedStateCheckedException: > Unexpected state: [state=0, rowVer=1553366856829:189821:1, > txVer=1553366856829:189838:1, > localNodeId=aa9e5187-4b3e-456c-8eaa-7ccf6a500001] > at > org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:328) > at > org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:315) > at > org.apache.ignite.internal.processors.cache.tree.mvcc.data.MvccUpdateDataRow.visit(MvccUpdateDataRow.java:416) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3060) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.init(BPlusTree.java:3030) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.notFound(BPlusTree.java:2986) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:308) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5643) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:273) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5629) > at > org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:168) > at > org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:349) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read(BPlusTree.java:5830) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2626) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2635) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doVisit(BPlusTree.java:2593) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$7800(BPlusTree.java:92) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3187) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.access$4900(BPlusTree.java:2948) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1066) > ... 19 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)