[
https://issues.apache.org/jira/browse/IGNITE-28866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Plekhanov updated IGNITE-28866:
---------------------------------------
Attachment:
IGNITE-28866_Fix_deadlock_on_schema_change_with_concurrent_node_leave.patch
> Deadlock on schema change with concurrent node leave
> ----------------------------------------------------
>
> Key: IGNITE-28866
> URL: https://issues.apache.org/jira/browse/IGNITE-28866
> Project: Ignite
> Issue Type: Bug
> Reporter: Aleksey Plekhanov
> Assignee: Aleksey Plekhanov
> Priority: Major
> Labels: MakeTeamcityGreenAgain, ise
> Attachments:
> IGNITE-28866_Fix_deadlock_on_schema_change_with_concurrent_node_leave.patch
>
>
> Java level deadlock can happen due to wrong lock order betweenÂ
> {{SchemaOperationManager.mux}} and {{GridQueryProcessor.stateMux}} in the
> following case:
> 1. Schema change operation started (for example index creation).
> 2. Remote (not coordinator) node completes operation, sends status massage
> and stops for some reason.
> 3. Remote node failure detected on coordinator before status message was
> processed.
> 4. Coordinator node completes operation and enters
> {{SchemaOperationManager.onLocalNodeFinished}} by thread "schema-op-worker",
> holding {{SchemaOperationManager.mux}}.
> 5. Meanwhile status message from failed node is processed by
> {{GridQueryProcessor.processStatusMessage}} holding
> {{GridQueryProcessor.stateMux}} and waiting for
> {{SchemaOperationManager.mux}} in {{SchemaOperationManager.onNodeFinished}}.
> 6. Thread "schema-op-worker" riches
> {{GridQueryProcessor.onCoordinatorFinished}} and waits for
> {{SchemaOperationManager.mux}}, so threads are deadlocked.
> Stack trace for locked threads:
> {noformat}
> "schema-#10368%index.DynamicIndexReplicatedAtomicConcurrentSelfTest1%":
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onNodeFinished(SchemaOperationManager.java:149)
> - waiting to lock <0x00000000b4d795c8> (a java.lang.Object)
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.processStatusMessage(GridQueryProcessor.java:3936)
> - locked <0x00000000b4d79910> (a java.lang.Object)
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.lambda$new$0(GridQueryProcessor.java:347)
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor$$Lambda$652/0x00007fb3445f2c98.onMessage(Unknown
> Source)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1837)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1458)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1362)
> at
> org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
> at java.lang.Thread.run([email protected]/Thread.java:840)
> "schema-op-worker-2207b04d-b282-4c37-9b43-e7b06aa9c982-#12657%index.DynamicIndexReplicatedAtomicConcurrentSelfTest1%":
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.onCoordinatorFinished(GridQueryProcessor.java:1897)
> - waiting to lock <0x00000000b4d79910> (a java.lang.Object)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.checkFinished(SchemaOperationManager.java:242)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onNodeFinished(SchemaOperationManager.java:173)
> - locked <0x00000000b4d795c8> (a java.lang.Object)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onLocalNodeFinished(SchemaOperationManager.java:136)
> - locked <0x00000000b4d795c8> (a java.lang.Object)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationManager$1.apply(SchemaOperationManager.java:106)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationManager$1.apply(SchemaOperationManager.java:104)
> at
> org.apache.ignite.internal.thread.context.function.OperationContextAwareInClosure.apply(OperationContextAwareInClosure.java:38)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:477)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:353)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:341)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:588)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:568)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker$1.apply(SchemaOperationWorker.java:161)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker$1.apply(SchemaOperationWorker.java:147)
> at
> org.apache.ignite.internal.thread.context.function.OperationContextAwareInClosure.apply(OperationContextAwareInClosure.java:38)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:477)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:353)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:341)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:588)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:568)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:534)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker.body(SchemaOperationWorker.java:116)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
> at java.lang.Thread.run([email protected]/Thread.java:840)
> {noformat}
> Failure on TC: https://ci2.ignite.apache.org/viewLog.html?buildId=9185883
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
