[
https://issues.apache.org/jira/browse/HBASE-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988567#comment-14988567
]
stack commented on HBASE-14754:
-------------------------------
This fail looks same latch issue identified in earlier issues:
{code}
Failed
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException50Times
Failing for the past 4 builds (Since Failed#423 )
Took 38 sec.
add description
Error Message
org.apache.hadoop.hbase.exceptions.PreemptiveFastFailException: Exception
happened 4 times. tolocalhost,1234,987654321
Stacktrace
java.util.concurrent.ExecutionException:
org.apache.hadoop.hbase.exceptions.PreemptiveFastFailException: Exception
happened 4 times. tolocalhost,1234,987654321
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException(TestFastFailWithoutTestUtil.java:478)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException50Times(TestFastFailWithoutTestUtil.java:342)
Caused by: org.apache.hadoop.hbase.exceptions.PreemptiveFastFailException:
Exception happened 4 times. tolocalhost,1234,987654321
at
org.apache.hadoop.hbase.client.PreemptiveFastFailInterceptor.intercept(PreemptiveFastFailInterceptor.java:124)
at
org.apache.hadoop.hbase.client.PreemptiveFastFailInterceptor.intercept(PreemptiveFastFailInterceptor.java:373)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil$7.intercept(TestFastFailWithoutTestUtil.java:569)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:117)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil$8.callWithRetries(TestFastFailWithoutTestUtil.java:589)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil$8.callWithRetries(TestFastFailWithoutTestUtil.java:585)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil$5.call(TestFastFailWithoutTestUtil.java:481)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil$5.call(TestFastFailWithoutTestUtil.java:478)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Standard Output
Process Thread Dump: This should not hang but seems to sometimes...FIX! Here is
a thread dump!
8 active threads
Thread 13 (pool-5-thread-2):
State: TIMED_WAITING
Blocked count: 0
Waited count: 108
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Thread 12 (pool-5-thread-1):
State: WAITING
Blocked count: 0
Waited count: 246
Waiting on java.util.concurrent.CountDownLatch$Sync@75d52c4c
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil$7.updateFailureInfo(TestFastFailWithoutTestUtil.java:546)
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:154)
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil$8.callWithRetries(TestFastFailWithoutTestUtil.java:589)
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil$8.callWithRetries(TestFastFailWithoutTestUtil.java:585)
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil$3.call(TestFastFailWithoutTestUtil.java:424)
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil$3.call(TestFastFailWithoutTestUtil.java:419)
java.util.concurrent.FutureTask.run(FutureTask.java:262)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Thread 11 (Time-limited test):
State: RUNNABLE
Blocked count: 0
Waited count: 107
Stack:
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:168)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294)
org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:339)
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException(TestFastFailWithoutTestUtil.java:467)
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException50Times(TestFastFailWithoutTestUtil.java:342)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Thread 9 (process reaper):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Thread 4 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 1
Waited count: 2
Waiting on java.lang.ref.ReferenceQueue$Lock@9d84476
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.lang.ref.Reference$Lock@600ab1bf
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
Thread 1 (main):
State: TIMED_WAITING
Blocked count: 1
Waited count: 3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:422)
java.util.concurrent.FutureTask.get(FutureTask.java:199)
org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141)
org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127)
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
org.junit.runners.Suite.runChild(Suite.java:128)
org.junit.runners.Suite.runChild(Suite.java:27)
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
{code}
> TestFastFailWithoutTestUtil failing on trunk now in
> #testPreemptiveFastFailException50Times
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-14754
> URL: https://issues.apache.org/jira/browse/HBASE-14754
> Project: HBase
> Issue Type: Bug
> Components: flakey, test
> Reporter: stack
> Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
>
> I'm just going to remove this suite. Its caused loads of pain over last few
> months. See HBASE-14484 and HBASE-14421 with all their amendments.
> HBASE-14422 was filed a while ago to fix this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
