Alexei Scherbakov created IGNITE-11937:
------------------------------------------
Summary: Fix MVCC PDS flaky suites timeout
Key: IGNITE-11937
URL: https://issues.apache.org/jira/browse/IGNITE-11937
Project: Ignite
Issue Type: Bug
Reporter: Alexei Scherbakov
Currently we have non-zero failure rate for some MVCC PDS suites in master.
Seems this is due to failure [1] in testRebalancingDuringLoad* tests group,
which leads to dumping WAL and lock states at the time proportional to current
WAL length increasing test duration for random time depending on WAL length.
Worse thing the test remains green despite throwing a critical exception.
[1] Stacktrace
{noformat}
[2019-06-19
15:56:53,386][ERROR][sys-stripe-6-#134%persistence.IgnitePdsContinuousRestartTestWithSharedGroupAndIndexes3%][IgniteTestResources]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailure
Handler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is
corrupted [pages(groupId, page
Id)=[IgniteBiTuple [val1=81227264, val2=844420635164676]], msg=Runtime failure
on search row: TxKey [major=1560948946388, minor=17286]]]]
class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=81227264,
val2=844420635164676]], msg=Runtime failure on search row: TxKey
[major=1560948946388, minor=17286]]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.corruptedTreeException(BPlusTree.java:5909)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1859)
at
org.apache.ignite.internal.processors.cache.mvcc.txlog.TxLog.put(TxLog.java:293)
at
org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl.updateState(MvccProcessorImpl.java:699)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.setMvccState(IgniteTxManager.java:2570)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:1228)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:1070)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.prepareRemoteTx(GridDistributedTxRemoteAdapter.java:421)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.startRemoteTx(IgniteTxHandler.java:1837)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:1198)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:118)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:224)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:222)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1141)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1558)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1186)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
org.apache.ignite.internal.managers.communication.GridIoManager$8.run(GridIoManager.java:1083)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:559)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Unexpected new transaction state.
[currState=2, newState=1, cntr=17286]
at
org.apache.ignite.internal.processors.cache.mvcc.txlog.TxLog$TxLogUpdateClosure.invalid(TxLog.java:629)
at
org.apache.ignite.internal.processors.cache.mvcc.txlog.TxLog$TxLogUpdateClosure.checkAborted(TxLog.java:605)
at
org.apache.ignite.internal.processors.cache.mvcc.txlog.TxLog$TxLogUpdateClosure.call(TxLog.java:499)
at
org.apache.ignite.internal.processors.cache.mvcc.txlog.TxLog$TxLogUpdateClosure.call(TxLog.java:440)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:3800)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.access$5700(BPlusTree.java:3694)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1948)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1826)
... 24 more
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
