Hi I got reported that unresponsive servers in production very frequently and the symptom is that all http threads are hung on ZK activity. It looks InternalBlockUntilConnectedOrTimeOut called from InterProcessMutex recipes. The followings are stack trace snippets. Do you have any clue what's wrong here?
Thank you Best, Jae "http-0.0.0.0-7101-3" daemon prio=10 tid=0x00000000015c6000 nid=0x11ae waiting on condition [0x00007f91771ee000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000005ee0b7688> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:282) at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:296) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105) at org.apache.curator.framework.imps.CreateBuilderImpl.findProtectedNodeInForeground(CreateBuilderImpl.java:660) at org.apache.curator.framework.imps.CreateBuilderImpl.access$800(CreateBuilderImpl.java:42) at org.apache.curator.framework.imps.CreateBuilderImpl$10.call(CreateBuilderImpl.java:619) at org.apache.curator.framework.imps.CreateBuilderImpl$10.call(CreateBuilderImpl.java:610) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:606) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:429) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:409) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:42) at org.apache.curator.framework.recipes.locks.LockInternals.attemptLock(LockInternals.java:224) at org.apache.curator.framework.recipes.locks.InterProcessMutex.internalLock(InterProcessMutex.java:221) at org.apache.curator.framework.recipes.locks.InterProcessMutex.acquire(InterProcessMutex.java:96)
