[
https://issues.apache.org/jira/browse/IGNITE-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Pavlukhin updated IGNITE-9292:
-----------------------------------
Description:
Unexpected state exception is observed during concurrent execution insert and
fast update for same key.
One of observed with concurrent update and insert for the same key from the
same node serializes as follows:
1. insert on primary
2. insert on backup
3. update on primary waits on lock
4. insert is prepared on backup
5. insert is prepared on primary
6. insert is committed on primary (before committing on backup)
7. update waiting on lock wakes up
8. update is executed on primary
9. update fails on backup because sees inserted row in prepared state
It is one possible erroneous scenario. There might be others. TX log update
procedure should be thoroughly reviewed.
{noformat}
[2018-08-16
16:32:25,452][ERROR][ForkJoinPool.commonPool-worker-2][DmlStatementsProcessor]
Error during update [localNodeId=6709af72-2fe9-4eab-9ee5-be83eab00000]
class org.apache.ignite.IgniteCheckedException: Failed to update backup node:
[localNodeId=6709af72-2fe9-4eab-9ee5-be83eab00000,
remoteNodeId=4ca5b185-c5d1-4756-8977-e57aec300001]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.onResult(GridDhtTxAbstractEnlistFuture.java:931)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistResponse(GridDhtTransactionalCacheAdapter.java:2245)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$1000(GridDhtTransactionalCacheAdapter.java:106)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$16.apply(GridDhtTransactionalCacheAdapter.java:237)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$16.apply(GridDhtTransactionalCacheAdapter.java:235)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchRow []]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1047)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:1887)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:523)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.mvccSet(GridCacheMapEntry.java:1080)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxRemote.mvccEnlistBatch(GridDhtTxRemote.java:451)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistRequest(GridDhtTransactionalCacheAdapter.java:2209)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:106)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$15.apply(GridDhtTransactionalCacheAdapter.java:230)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$15.apply(GridDhtTransactionalCacheAdapter.java:228)
... 13 more
Caused by: class
org.apache.ignite.internal.transactions.IgniteTxMvccVersionCheckedException:
Unexpected state: [state=1, rowVer=1534426342972:4:1, txVer=1534426342972:5:1,
localNodeId=4ca5b185-c5d1-4756-8977-e57aec300001]
at
org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:292)
at
org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:279)
at
org.apache.ignite.internal.processors.cache.tree.mvcc.data.MvccUpdateDataRow.visit(MvccUpdateDataRow.java:329)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3015)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.init(BPlusTree.java:2984)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.notFound(BPlusTree.java:2940)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:300)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5594)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5579)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2586)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doVisit(BPlusTree.java:2553)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8500(BPlusTree.java:87)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3142)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.access$5400(BPlusTree.java:2902)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1044)
... 21 more{noformat}
was:
Unexpected state exception is observed during concurrent execution insert and
fast update for same key. Most likely it is caused by premature updating
transaction state when near node is backup for some key.
{noformat}
[2018-08-16
16:32:25,452][ERROR][ForkJoinPool.commonPool-worker-2][DmlStatementsProcessor]
Error during update [localNodeId=6709af72-2fe9-4eab-9ee5-be83eab00000]
class org.apache.ignite.IgniteCheckedException: Failed to update backup node:
[localNodeId=6709af72-2fe9-4eab-9ee5-be83eab00000,
remoteNodeId=4ca5b185-c5d1-4756-8977-e57aec300001]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.onResult(GridDhtTxAbstractEnlistFuture.java:931)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistResponse(GridDhtTransactionalCacheAdapter.java:2245)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$1000(GridDhtTransactionalCacheAdapter.java:106)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$16.apply(GridDhtTransactionalCacheAdapter.java:237)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$16.apply(GridDhtTransactionalCacheAdapter.java:235)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchRow []]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1047)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:1887)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:523)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.mvccSet(GridCacheMapEntry.java:1080)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxRemote.mvccEnlistBatch(GridDhtTxRemote.java:451)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistRequest(GridDhtTransactionalCacheAdapter.java:2209)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:106)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$15.apply(GridDhtTransactionalCacheAdapter.java:230)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$15.apply(GridDhtTransactionalCacheAdapter.java:228)
... 13 more
Caused by: class
org.apache.ignite.internal.transactions.IgniteTxMvccVersionCheckedException:
Unexpected state: [state=1, rowVer=1534426342972:4:1, txVer=1534426342972:5:1,
localNodeId=4ca5b185-c5d1-4756-8977-e57aec300001]
at
org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:292)
at
org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:279)
at
org.apache.ignite.internal.processors.cache.tree.mvcc.data.MvccUpdateDataRow.visit(MvccUpdateDataRow.java:329)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3015)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.init(BPlusTree.java:2984)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.notFound(BPlusTree.java:2940)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:300)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5594)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5579)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2586)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doVisit(BPlusTree.java:2553)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8500(BPlusTree.java:87)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3142)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.access$5400(BPlusTree.java:2902)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1044)
... 21 more{noformat}
> MVCC SQL: Unexpected state exception when updating backup
> ---------------------------------------------------------
>
> Key: IGNITE-9292
> URL: https://issues.apache.org/jira/browse/IGNITE-9292
> Project: Ignite
> Issue Type: Bug
> Components: mvcc, sql
> Reporter: Ivan Pavlukhin
> Assignee: Ivan Pavlukhin
> Priority: Major
> Attachments: MvccInsertUpdateConcurrent.java
>
>
> Unexpected state exception is observed during concurrent execution insert and
> fast update for same key.
> One of observed with concurrent update and insert for the same key from the
> same node serializes as follows:
> 1. insert on primary
> 2. insert on backup
> 3. update on primary waits on lock
> 4. insert is prepared on backup
> 5. insert is prepared on primary
> 6. insert is committed on primary (before committing on backup)
> 7. update waiting on lock wakes up
> 8. update is executed on primary
> 9. update fails on backup because sees inserted row in prepared state
> It is one possible erroneous scenario. There might be others. TX log update
> procedure should be thoroughly reviewed.
> {noformat}
> [2018-08-16
> 16:32:25,452][ERROR][ForkJoinPool.commonPool-worker-2][DmlStatementsProcessor]
> Error during update [localNodeId=6709af72-2fe9-4eab-9ee5-be83eab00000]
> class org.apache.ignite.IgniteCheckedException: Failed to update backup node:
> [localNodeId=6709af72-2fe9-4eab-9ee5-be83eab00000,
> remoteNodeId=4ca5b185-c5d1-4756-8977-e57aec300001]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.onResult(GridDhtTxAbstractEnlistFuture.java:931)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistResponse(GridDhtTransactionalCacheAdapter.java:2245)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$1000(GridDhtTransactionalCacheAdapter.java:106)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$16.apply(GridDhtTransactionalCacheAdapter.java:237)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$16.apply(GridDhtTransactionalCacheAdapter.java:235)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1047)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:1887)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:523)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.mvccSet(GridCacheMapEntry.java:1080)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxRemote.mvccEnlistBatch(GridDhtTxRemote.java:451)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistRequest(GridDhtTransactionalCacheAdapter.java:2209)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:106)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$15.apply(GridDhtTransactionalCacheAdapter.java:230)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$15.apply(GridDhtTransactionalCacheAdapter.java:228)
> ... 13 more
> Caused by: class
> org.apache.ignite.internal.transactions.IgniteTxMvccVersionCheckedException:
> Unexpected state: [state=1, rowVer=1534426342972:4:1,
> txVer=1534426342972:5:1, localNodeId=4ca5b185-c5d1-4756-8977-e57aec300001]
> at
> org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:292)
> at
> org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:279)
> at
> org.apache.ignite.internal.processors.cache.tree.mvcc.data.MvccUpdateDataRow.visit(MvccUpdateDataRow.java:329)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3015)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.init(BPlusTree.java:2984)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.notFound(BPlusTree.java:2940)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:300)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5594)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5579)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
> at
> org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2586)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doVisit(BPlusTree.java:2553)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8500(BPlusTree.java:87)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3142)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.access$5400(BPlusTree.java:2902)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1044)
> ... 21 more{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
