[
https://issues.apache.org/jira/browse/IGNITE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Mashenkov updated IGNITE-18448:
--------------------------------------
Description:
Two threads fall into deadlock when trying to remove nodes from collection.
See stacktraces below.
1. ConcurrentHashMap.compute methods must not use blocking operations.
2. IgnitionImpl.doStart() adds a node to _readyForInitNodes_ collection under
race.
The call _nodeToStart.start(cfgContent)_ returns a future may fails before the
_nodeToStart_ object will be added to collection.
3. If the future (_nodeToStart.start(cfgContent)_) fails, it possible, some
components are started and hold resources, which will never be released. Seems,
in case of failure, _nodeToStart.stop()_ has to be called.
{noformat}
"%node1%Raft-Group-Client-12@21385" prio=5 tid=0x127e nid=NA waiting for
monitor entry
java.lang.Thread.State: BLOCKED
waiting for Test worker@1 to release lock on <0x59df> (a
java.util.concurrent.ConcurrentHashMap$Node)
at
java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1122)
at
java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1102)
at
org.apache.ignite.internal.app.IgnitionImpl.handleStartException(IgnitionImpl.java:235)
{noformat}
{noformat}
"Test worker@1" prio=5 tid=0x1 nid=NA sleeping
java.lang.Thread.State: TIMED_WAITING
blocks %node1%Raft-Group-Client-12@21385
at java.lang.Thread.sleep(Thread.java:-1)
at
org.apache.ignite.internal.util.IgniteSpinReadWriteLock.writeLock(IgniteSpinReadWriteLock.java:255)
at
org.apache.ignite.internal.util.IgniteSpinBusyLock.block(IgniteSpinBusyLock.java:68)
at
org.apache.ignite.internal.distributionzones.DistributionZoneManager.stop(DistributionZoneManager.java:288)
at
org.apache.ignite.internal.app.LifecycleManager.lambda$stopAllComponents$1(LifecycleManager.java:133)
at
org.apache.ignite.internal.app.LifecycleManager$$Lambda$3032.1586776480.accept(Unknown
Source:-1)
at java.util.Iterator.forEachRemaining(Iterator.java:133)
at
org.apache.ignite.internal.app.LifecycleManager.stopAllComponents(LifecycleManager.java:131)
- locked <0x59de> (a org.apache.ignite.internal.app.LifecycleManager)
at
org.apache.ignite.internal.app.LifecycleManager.stopNode(LifecycleManager.java:115)
at org.apache.ignite.internal.app.IgniteImpl.stop(IgniteImpl.java:642)
at
org.apache.ignite.internal.app.IgnitionImpl.lambda$stop$0(IgnitionImpl.java:145)
at
org.apache.ignite.internal.app.IgnitionImpl$$Lambda$3001.460355950.apply(Unknown
Source:-1)
at
java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1822)
- locked <0x59df> (a java.util.concurrent.ConcurrentHashMap$Node)
at
org.apache.ignite.internal.app.IgnitionImpl.stop(IgnitionImpl.java:144)
at org.apache.ignite.IgnitionManager.stop(IgnitionManager.java:116)
{noformat}
was:
Two threads fall into deadlock when trying to remove nodes from collection.
ConcurrentHashMap.compute methods must not use blocking operations.
{noformat}
"%node1%Raft-Group-Client-12@21385" prio=5 tid=0x127e nid=NA waiting for
monitor entry
java.lang.Thread.State: BLOCKED
waiting for Test worker@1 to release lock on <0x59df> (a
java.util.concurrent.ConcurrentHashMap$Node)
at
java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1122)
at
java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1102)
at
org.apache.ignite.internal.app.IgnitionImpl.handleStartException(IgnitionImpl.java:235)
{noformat}
{noformat}
"Test worker@1" prio=5 tid=0x1 nid=NA sleeping
java.lang.Thread.State: TIMED_WAITING
blocks %node1%Raft-Group-Client-12@21385
at java.lang.Thread.sleep(Thread.java:-1)
at
org.apache.ignite.internal.util.IgniteSpinReadWriteLock.writeLock(IgniteSpinReadWriteLock.java:255)
at
org.apache.ignite.internal.util.IgniteSpinBusyLock.block(IgniteSpinBusyLock.java:68)
at
org.apache.ignite.internal.distributionzones.DistributionZoneManager.stop(DistributionZoneManager.java:288)
at
org.apache.ignite.internal.app.LifecycleManager.lambda$stopAllComponents$1(LifecycleManager.java:133)
at
org.apache.ignite.internal.app.LifecycleManager$$Lambda$3032.1586776480.accept(Unknown
Source:-1)
at java.util.Iterator.forEachRemaining(Iterator.java:133)
at
org.apache.ignite.internal.app.LifecycleManager.stopAllComponents(LifecycleManager.java:131)
- locked <0x59de> (a org.apache.ignite.internal.app.LifecycleManager)
at
org.apache.ignite.internal.app.LifecycleManager.stopNode(LifecycleManager.java:115)
at org.apache.ignite.internal.app.IgniteImpl.stop(IgniteImpl.java:642)
at
org.apache.ignite.internal.app.IgnitionImpl.lambda$stop$0(IgnitionImpl.java:145)
at
org.apache.ignite.internal.app.IgnitionImpl$$Lambda$3001.460355950.apply(Unknown
Source:-1)
at
java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1822)
- locked <0x59df> (a java.util.concurrent.ConcurrentHashMap$Node)
at
org.apache.ignite.internal.app.IgnitionImpl.stop(IgnitionImpl.java:144)
at org.apache.ignite.IgnitionManager.stop(IgnitionManager.java:116)
{noformat}
> Deadlock on node stop.
> ----------------------
>
> Key: IGNITE-18448
> URL: https://issues.apache.org/jira/browse/IGNITE-18448
> Project: Ignite
> Issue Type: Bug
> Reporter: Andrey Mashenkov
> Priority: Major
> Labels: ignite-3
>
> Two threads fall into deadlock when trying to remove nodes from collection.
> See stacktraces below.
> 1. ConcurrentHashMap.compute methods must not use blocking operations.
> 2. IgnitionImpl.doStart() adds a node to _readyForInitNodes_ collection under
> race.
> The call _nodeToStart.start(cfgContent)_ returns a future may fails before
> the _nodeToStart_ object will be added to collection.
> 3. If the future (_nodeToStart.start(cfgContent)_) fails, it possible, some
> components are started and hold resources, which will never be released.
> Seems, in case of failure, _nodeToStart.stop()_ has to be called.
> {noformat}
> "%node1%Raft-Group-Client-12@21385" prio=5 tid=0x127e nid=NA waiting for
> monitor entry
> java.lang.Thread.State: BLOCKED
> waiting for Test worker@1 to release lock on <0x59df> (a
> java.util.concurrent.ConcurrentHashMap$Node)
> at
> java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1122)
> at
> java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1102)
> at
> org.apache.ignite.internal.app.IgnitionImpl.handleStartException(IgnitionImpl.java:235)
> {noformat}
> {noformat}
> "Test worker@1" prio=5 tid=0x1 nid=NA sleeping
> java.lang.Thread.State: TIMED_WAITING
> blocks %node1%Raft-Group-Client-12@21385
> at java.lang.Thread.sleep(Thread.java:-1)
> at
> org.apache.ignite.internal.util.IgniteSpinReadWriteLock.writeLock(IgniteSpinReadWriteLock.java:255)
> at
> org.apache.ignite.internal.util.IgniteSpinBusyLock.block(IgniteSpinBusyLock.java:68)
> at
> org.apache.ignite.internal.distributionzones.DistributionZoneManager.stop(DistributionZoneManager.java:288)
> at
> org.apache.ignite.internal.app.LifecycleManager.lambda$stopAllComponents$1(LifecycleManager.java:133)
> at
> org.apache.ignite.internal.app.LifecycleManager$$Lambda$3032.1586776480.accept(Unknown
> Source:-1)
> at java.util.Iterator.forEachRemaining(Iterator.java:133)
> at
> org.apache.ignite.internal.app.LifecycleManager.stopAllComponents(LifecycleManager.java:131)
> - locked <0x59de> (a org.apache.ignite.internal.app.LifecycleManager)
> at
> org.apache.ignite.internal.app.LifecycleManager.stopNode(LifecycleManager.java:115)
> at org.apache.ignite.internal.app.IgniteImpl.stop(IgniteImpl.java:642)
> at
> org.apache.ignite.internal.app.IgnitionImpl.lambda$stop$0(IgnitionImpl.java:145)
> at
> org.apache.ignite.internal.app.IgnitionImpl$$Lambda$3001.460355950.apply(Unknown
> Source:-1)
> at
> java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1822)
> - locked <0x59df> (a java.util.concurrent.ConcurrentHashMap$Node)
> at
> org.apache.ignite.internal.app.IgnitionImpl.stop(IgnitionImpl.java:144)
> at org.apache.ignite.IgnitionManager.stop(IgnitionManager.java:116)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
