Aleksey Plekhanov created IGNITE-28866:
------------------------------------------
Summary: Deadlock on schema change with concurrent node leave
Key: IGNITE-28866
URL: https://issues.apache.org/jira/browse/IGNITE-28866
Project: Ignite
Issue Type: Bug
Reporter: Aleksey Plekhanov
Assignee: Aleksey Plekhanov
Java level deadlock can happen due to wrong lock order betweenÂ
{{SchemaOperationManager.mux}} andÂ
{{GridQueryProcessor.stateMux}} in the following case:
1. Schema change operation started (for example index creation).
2. Remote (not coordinator) node completes operation, sends status massage and
stops for some reason.
3. Remote node failure detected on coordinator before status message was
processed.
4. Coordinator node completes operation and enters
{{SchemaOperationManager.onLocalNodeFinished}} by thread "schema-op-worker",
holding {{SchemaOperationManager.mux}}.
5. Meanwhile status message from failed node is processed by
{{GridQueryProcessor.processStatusMessage}} holding
{{GridQueryProcessor.stateMux}} and waiting for {{SchemaOperationManager.mux}}
in {{SchemaOperationManager.onNodeFinished}}.
6. Thread "schema-op-worker" riches
{{GridQueryProcessor.onCoordinatorFinished}} and waits for
{{SchemaOperationManager.mux}}, so threads are deadlocked.
Stack trace for locked threads:
{noformat}
"schema-#10368%index.DynamicIndexReplicatedAtomicConcurrentSelfTest1%":
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onNodeFinished(SchemaOperationManager.java:149)
- waiting to lock <0x00000000b4d795c8> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.processStatusMessage(GridQueryProcessor.java:3936)
- locked <0x00000000b4d79910> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.lambda$new$0(GridQueryProcessor.java:347)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor$$Lambda$652/0x00007fb3445f2c98.onMessage(Unknown
Source)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1837)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1458)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1362)
at
org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
at java.lang.Thread.run([email protected]/Thread.java:840)
"schema-op-worker-2207b04d-b282-4c37-9b43-e7b06aa9c982-#12657%index.DynamicIndexReplicatedAtomicConcurrentSelfTest1%":
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCoordinatorFinished(GridQueryProcessor.java:1897)
- waiting to lock <0x00000000b4d79910> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.checkFinished(SchemaOperationManager.java:242)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onNodeFinished(SchemaOperationManager.java:173)
- locked <0x00000000b4d795c8> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onLocalNodeFinished(SchemaOperationManager.java:136)
- locked <0x00000000b4d795c8> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager$1.apply(SchemaOperationManager.java:106)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager$1.apply(SchemaOperationManager.java:104)
at
org.apache.ignite.internal.thread.context.function.OperationContextAwareInClosure.apply(OperationContextAwareInClosure.java:38)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:477)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:353)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:341)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:588)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:568)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker$1.apply(SchemaOperationWorker.java:161)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker$1.apply(SchemaOperationWorker.java:147)
at
org.apache.ignite.internal.thread.context.function.OperationContextAwareInClosure.apply(OperationContextAwareInClosure.java:38)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:477)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:353)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:341)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:588)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:568)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:534)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker.body(SchemaOperationWorker.java:116)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
at java.lang.Thread.run([email protected]/Thread.java:840)
{noformat}
Failure on TC: https://ci2.ignite.apache.org/viewLog.html?buildId=9185883
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
