[
https://issues.apache.org/jira/browse/IGNITE-28866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Plekhanov updated IGNITE-28866:
---------------------------------------
Description:
Java level deadlock can happen due to wrong lock order between
{{SchemaOperationManager.mux}} and {{GridQueryProcessor.stateMux}} in the
following case:
1. Schema change operation started (for example index creation).
2. Remote (not coordinator) node completes operation, sends status massage and
stops for some reason.
3. Remote node failure detected on coordinator before status message was
processed.
4. Coordinator node completes operation and enters
{{SchemaOperationManager.onLocalNodeFinished}} by thread "schema-op-worker",
holding {{SchemaOperationManager.mux}}.
5. Meanwhile status message from failed node is processed by
{{GridQueryProcessor.processStatusMessage}} holding
{{GridQueryProcessor.stateMux}} and waiting for {{SchemaOperationManager.mux}}
in {{SchemaOperationManager.onNodeFinished}}.
6. Thread "schema-op-worker" riches
{{GridQueryProcessor.onCoordinatorFinished}} and waits for
{{SchemaOperationManager.mux}}, so threads are deadlocked.
Stack trace for locked threads:
{noformat}
"schema-#10368%index.DynamicIndexReplicatedAtomicConcurrentSelfTest1%":
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onNodeFinished(SchemaOperationManager.java:149)
- waiting to lock <0x00000000b4d795c8> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.processStatusMessage(GridQueryProcessor.java:3936)
- locked <0x00000000b4d79910> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.lambda$new$0(GridQueryProcessor.java:347)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor$$Lambda$652/0x00007fb3445f2c98.onMessage(Unknown
Source)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1837)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1458)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1362)
at
org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
at java.lang.Thread.run([email protected]/Thread.java:840)
"schema-op-worker-2207b04d-b282-4c37-9b43-e7b06aa9c982-#12657%index.DynamicIndexReplicatedAtomicConcurrentSelfTest1%":
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCoordinatorFinished(GridQueryProcessor.java:1897)
- waiting to lock <0x00000000b4d79910> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.checkFinished(SchemaOperationManager.java:242)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onNodeFinished(SchemaOperationManager.java:173)
- locked <0x00000000b4d795c8> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onLocalNodeFinished(SchemaOperationManager.java:136)
- locked <0x00000000b4d795c8> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager$1.apply(SchemaOperationManager.java:106)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager$1.apply(SchemaOperationManager.java:104)
at
org.apache.ignite.internal.thread.context.function.OperationContextAwareInClosure.apply(OperationContextAwareInClosure.java:38)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:477)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:353)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:341)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:588)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:568)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker$1.apply(SchemaOperationWorker.java:161)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker$1.apply(SchemaOperationWorker.java:147)
at
org.apache.ignite.internal.thread.context.function.OperationContextAwareInClosure.apply(OperationContextAwareInClosure.java:38)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:477)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:353)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:341)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:588)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:568)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:534)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker.body(SchemaOperationWorker.java:116)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
at java.lang.Thread.run([email protected]/Thread.java:840)
{noformat}
Failure on TC: https://ci2.ignite.apache.org/viewLog.html?buildId=9185883
was:
Java level deadlock can happen due to wrong lock order between
{{SchemaOperationManager.mux}} and
{{GridQueryProcessor.stateMux}} in the following case:
1. Schema change operation started (for example index creation).
2. Remote (not coordinator) node completes operation, sends status massage and
stops for some reason.
3. Remote node failure detected on coordinator before status message was
processed.
4. Coordinator node completes operation and enters
{{SchemaOperationManager.onLocalNodeFinished}} by thread "schema-op-worker",
holding {{SchemaOperationManager.mux}}.
5. Meanwhile status message from failed node is processed by
{{GridQueryProcessor.processStatusMessage}} holding
{{GridQueryProcessor.stateMux}} and waiting for {{SchemaOperationManager.mux}}
in {{SchemaOperationManager.onNodeFinished}}.
6. Thread "schema-op-worker" riches
{{GridQueryProcessor.onCoordinatorFinished}} and waits for
{{SchemaOperationManager.mux}}, so threads are deadlocked.
Stack trace for locked threads:
{noformat}
"schema-#10368%index.DynamicIndexReplicatedAtomicConcurrentSelfTest1%":
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onNodeFinished(SchemaOperationManager.java:149)
- waiting to lock <0x00000000b4d795c8> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.processStatusMessage(GridQueryProcessor.java:3936)
- locked <0x00000000b4d79910> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.lambda$new$0(GridQueryProcessor.java:347)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor$$Lambda$652/0x00007fb3445f2c98.onMessage(Unknown
Source)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1837)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1458)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1362)
at
org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
at java.lang.Thread.run([email protected]/Thread.java:840)
"schema-op-worker-2207b04d-b282-4c37-9b43-e7b06aa9c982-#12657%index.DynamicIndexReplicatedAtomicConcurrentSelfTest1%":
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCoordinatorFinished(GridQueryProcessor.java:1897)
- waiting to lock <0x00000000b4d79910> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.checkFinished(SchemaOperationManager.java:242)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onNodeFinished(SchemaOperationManager.java:173)
- locked <0x00000000b4d795c8> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onLocalNodeFinished(SchemaOperationManager.java:136)
- locked <0x00000000b4d795c8> (a java.lang.Object)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager$1.apply(SchemaOperationManager.java:106)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationManager$1.apply(SchemaOperationManager.java:104)
at
org.apache.ignite.internal.thread.context.function.OperationContextAwareInClosure.apply(OperationContextAwareInClosure.java:38)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:477)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:353)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:341)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:588)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:568)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker$1.apply(SchemaOperationWorker.java:161)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker$1.apply(SchemaOperationWorker.java:147)
at
org.apache.ignite.internal.thread.context.function.OperationContextAwareInClosure.apply(OperationContextAwareInClosure.java:38)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:477)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:353)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:341)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:588)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:568)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:534)
at
org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker.body(SchemaOperationWorker.java:116)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
at java.lang.Thread.run([email protected]/Thread.java:840)
{noformat}
Failure on TC: https://ci2.ignite.apache.org/viewLog.html?buildId=9185883
> Deadlock on schema change with concurrent node leave
> ----------------------------------------------------
>
> Key: IGNITE-28866
> URL: https://issues.apache.org/jira/browse/IGNITE-28866
> Project: Ignite
> Issue Type: Bug
> Reporter: Aleksey Plekhanov
> Assignee: Aleksey Plekhanov
> Priority: Major
> Labels: MakeTeamcityGreenAgain, ise
>
> Java level deadlock can happen due to wrong lock order between
> {{SchemaOperationManager.mux}} and {{GridQueryProcessor.stateMux}} in the
> following case:
> 1. Schema change operation started (for example index creation).
> 2. Remote (not coordinator) node completes operation, sends status massage
> and stops for some reason.
> 3. Remote node failure detected on coordinator before status message was
> processed.
> 4. Coordinator node completes operation and enters
> {{SchemaOperationManager.onLocalNodeFinished}} by thread "schema-op-worker",
> holding {{SchemaOperationManager.mux}}.
> 5. Meanwhile status message from failed node is processed by
> {{GridQueryProcessor.processStatusMessage}} holding
> {{GridQueryProcessor.stateMux}} and waiting for
> {{SchemaOperationManager.mux}} in {{SchemaOperationManager.onNodeFinished}}.
> 6. Thread "schema-op-worker" riches
> {{GridQueryProcessor.onCoordinatorFinished}} and waits for
> {{SchemaOperationManager.mux}}, so threads are deadlocked.
> Stack trace for locked threads:
> {noformat}
> "schema-#10368%index.DynamicIndexReplicatedAtomicConcurrentSelfTest1%":
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onNodeFinished(SchemaOperationManager.java:149)
> - waiting to lock <0x00000000b4d795c8> (a java.lang.Object)
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.processStatusMessage(GridQueryProcessor.java:3936)
> - locked <0x00000000b4d79910> (a java.lang.Object)
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.lambda$new$0(GridQueryProcessor.java:347)
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor$$Lambda$652/0x00007fb3445f2c98.onMessage(Unknown
> Source)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1837)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1458)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1362)
> at
> org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
> at java.lang.Thread.run([email protected]/Thread.java:840)
> "schema-op-worker-2207b04d-b282-4c37-9b43-e7b06aa9c982-#12657%index.DynamicIndexReplicatedAtomicConcurrentSelfTest1%":
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.onCoordinatorFinished(GridQueryProcessor.java:1897)
> - waiting to lock <0x00000000b4d79910> (a java.lang.Object)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.checkFinished(SchemaOperationManager.java:242)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onNodeFinished(SchemaOperationManager.java:173)
> - locked <0x00000000b4d795c8> (a java.lang.Object)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationManager.onLocalNodeFinished(SchemaOperationManager.java:136)
> - locked <0x00000000b4d795c8> (a java.lang.Object)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationManager$1.apply(SchemaOperationManager.java:106)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationManager$1.apply(SchemaOperationManager.java:104)
> at
> org.apache.ignite.internal.thread.context.function.OperationContextAwareInClosure.apply(OperationContextAwareInClosure.java:38)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:477)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:353)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:341)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:588)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:568)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker$1.apply(SchemaOperationWorker.java:161)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker$1.apply(SchemaOperationWorker.java:147)
> at
> org.apache.ignite.internal.thread.context.function.OperationContextAwareInClosure.apply(OperationContextAwareInClosure.java:38)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:477)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:353)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:341)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:588)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:568)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:534)
> at
> org.apache.ignite.internal.processors.query.schema.SchemaOperationWorker.body(SchemaOperationWorker.java:116)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
> at java.lang.Thread.run([email protected]/Thread.java:840)
> {noformat}
> Failure on TC: https://ci2.ignite.apache.org/viewLog.html?buildId=9185883
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
