[
https://issues.apache.org/jira/browse/HBASE-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160629#comment-14160629
]
Andrew Purtell commented on HBASE-12179:
----------------------------------------
This is a potentially serious issue related to the embedded ZooKeeper client.
Hung processes exhibit this stacktrace in the main thread:
{noformat}
"main" #1 prio=5 os_prio=0 tid=0x00007ffc64008000 nid=0x3302 in Object.wait()
[0x00007ffc6b87d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000dda32630> (a
org.apache.zookeeper.ClientCnxn$Packet)
at java.lang.Object.wait(Object.java:502)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
- locked <0x00000000dda32630> (a org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:781)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:510)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:489)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1241)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1219)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:474)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:137)
at
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:202)
at
org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:152)
at
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:213)
at
org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:912)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:876)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:793)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:764)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:733)
at
org.apache.hadoop.hbase.constraint.TestConstraint.setUpBeforeClass(TestConstraint.java:68)
{noformat}
The ZooKeeper thread name is very concerning:
{noformat}
"main-SendThread(252.127.0.0:63034)" #105 daemon prio=5 os_prio=0
tid=0x00007ffc654d6800 nid=0x338f
runnable [0x00007ffbec3f9000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000dd8b5068> (a sun.nio.ch.Util$2)
- locked <0x00000000dd8b5058> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000dd8b4f40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
{noformat}
What the hell is '252.127.0.0' ? Looks like a localhost IPv4 address shifted
right 8 bits with junk in the most significant byte.
Other ZooKeeper minicluster threads are waiting for work:
{noformat}
"SyncThread:0" #84 prio=5 os_prio=0 tid=0x00007ffc65023800 nid=0x336f waiting
on condition [0x00007ffbed6db000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d9febc98> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127)
"SessionTracker" #83 prio=5 os_prio=0 tid=0x00007ffc65054000 nid=0x336e in
Object.wait() [0x00007ffbed7dc000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000d9f457b8> (a
org.apache.zookeeper.server.SessionTrackerImpl)
at
org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
- locked <0x00000000d9f457b8> (a
org.apache.zookeeper.server.SessionTrackerImpl)
"NIOServerCxn.Factory:0.0.0.0/0.0.0.0:63034" #82 daemon prio=5 os_prio=0
tid=0x00007ffc65016800 nid=0x336d runnable [0x00007ffbedadf000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000d9ea88c0> (a sun.nio.ch.Util$2)
- locked <0x00000000d9ea88b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d9ea8798> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> TestConstraints intermittently hangs on Java 8
> ----------------------------------------------
>
> Key: HBASE-12179
> URL: https://issues.apache.org/jira/browse/HBASE-12179
> Project: HBase
> Issue Type: Bug
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.98.7
>
>
> TestConstraints intermittently hangs on Java 8. There are a few things this
> test could be doing better. Looking at it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
