[
https://issues.apache.org/jira/browse/GEODE-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892268#comment-16892268
]
Dan Smith commented on GEODE-7012:
----------------------------------
Here's the stack trace of the stuck StartupMessage
{noformat}
"vm_6_thr_16_persist5_host6_30767" #23 daemon prio=5 os_prio=0
tid=0x00007f1d3000e000 nid=0x78d8 waiting on condition [0x00007f1d7c2b8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000dcda0980> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:61)
at
org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:716)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForReplies(ReplyProcessor21.java:624)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForReplies(ReplyProcessor21.java:605)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForReplies(ReplyProcessor21.java:519)
at
org.apache.geode.distributed.internal.StartupOperation.sendStartupMessage(StartupOperation.java:90)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.sendStartupMessage(ClusterDistributionManager.java:2258)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:572)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:854)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:446)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:432)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.connectInternal(InternalDistributedSystem.java:257{noformat}
And here is one of the 100 stuck high priority threads
{noformat}
"Pooled High Priority Message Processor 101" #160 daemon prio=10 os_prio=0
tid=0x00007f1d44053800 nid=0x7a49 in Object.wait() [0x00007f1cfce8f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000dcf56cc0> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.waitUntilReadyToSendMsgs(ClusterDistributionManager.java:1212)
- locked <0x00000000dcf56cc0> (a java.lang.Object)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2832)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1528)
at
org.apache.geode.internal.cache.CreateRegionProcessor$CreateRegionMessage.process(CreateRegionProcessor.java:435)
at
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:369)
at
org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:433)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManager.java:957)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.doHighPriorityThread(ClusterDistributionManager.java:835)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$46/1997642736.invoke(Unknown
Source)
at
org.apache.geode.internal.logging.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:121)
at
org.apache.geode.internal.logging.LoggingThreadFactory$$Lambda$44/1280734436.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:748)
{noformat}
> Distributed deadlock with StartupMessages if executor pools get full
> --------------------------------------------------------------------
>
> Key: GEODE-7012
> URL: https://issues.apache.org/jira/browse/GEODE-7012
> Project: Geode
> Issue Type: Bug
> Reporter: Dan Smith
> Priority: Major
>
> We hit a distributed deadlock in one of our tests where two members are hung
> sending startup messages to each other.
> It turns out that until a member gets a response to a StartupMessage, it is
> in a state where it blocks all outgoing messages. At the same time, the
> member is receiving an attempting to respond to other messages, but those
> responses get blocked. If too many messages come in before the
> StartupResponseMessage, this ends up filling up the
> ClusterDistributionManager.highPriorityPool.
> If two members are trying to start up at the same time, and they both fill up
> the highPriorityPool, they both will fail to process each other's
> StartupMessage, because that message is executed in the same pool.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
