[ 
https://issues.apache.org/jira/browse/HBASE-19986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361585#comment-16361585
 ] 

stack commented on HBASE-19986:
-------------------------------

.003 does the @appy suggestion.... Here is what it looks like when it trips:

{code}
2018-02-12 15:08:31,261 INFO  [Time-limited test] hbase.ResourceChecker(148): 
before: TestTimeout#run1 Thread=8, OpenFileDescriptor=89, 
MaxFileDescriptor=10240, SystemLoadAverage=478, ProcessCount=372, 
AvailableMemoryMB=327
2018-02-12 15:08:31,444 INFO  [Time-limited test] hbase.ResourceChecker(172): 
after: TestTimeout#run1 Thread=8 (was 8), OpenFileDescriptor=89 (was 89), 
MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=478 (was 478), 
ProcessCount=372 (was 372), AvailableMemoryMB=327 (was 327)
2018-02-12 15:08:31,523 INFO  [Time-limited test] hbase.ResourceChecker(148): 
before: TestTimeout#infiniteLoop Thread=8, OpenFileDescriptor=89, 
MaxFileDescriptor=10240, SystemLoadAverage=478, ProcessCount=372, 
AvailableMemoryMB=325
====> TEST TIMED OUT. PRINTING THREAD DUMP. <====

Timestamp: 2018-02-12 03:08:41,125

"surefire-forkedjvm-command-thread" daemon prio=5 tid=10 runnable
java.lang.Thread.State: RUNNABLE
        at java.io.FileInputStream.readBytes(Native Method)
        at java.io.FileInputStream.read(FileInputStream.java:255)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at 
org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115)
        at 
org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:391)
        at java.lang.Thread.run(Thread.java:745)
"Reference Handler" daemon prio=10 tid=2 in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main"  prio=5 tid=1 runnable
java.lang.Thread.State: RUNNABLE
....
{code}

> If HBaseTestClassRule timesout a test, thread dump.
> ---------------------------------------------------
>
>                 Key: HBASE-19986
>                 URL: https://issues.apache.org/jira/browse/HBASE-19986
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Major
>         Attachments: HBASE-19986.branch-2.001.patch, 
> HBASE-19986.branch-2.002.patch, HBASE-19986.branch-2.003.patch
>
>
> We set look for stuck thread in our timeout rule but it is super conservative 
> in what it prints.. it looks for a RUNNABLE thread and prints first found 
> ONLY. Pretty useless for us.  If a test timesout, often the printing has 
> stopped in the stderr/stdout.
> I'm trying to debug TestAsyncRegionAdminApi. It says test timed out after 10 
> minutes but we've stopped printing to the logs and here is what junit prints:
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi
> -------------------------------------------------------------------------------
> Tests run: 25, Failures: 0, Errors: 2, Skipped: 2, Time elapsed: 572.508 s 
> <<< FAILURE! - in org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi
> org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi  Time elapsed: 14.642 
> s  <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 600 
> seconds
>   at 
> org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:363)
> org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi  Time elapsed: 14.642 
> s  <<< ERROR!
> java.lang.Exception: Appears to be stuck in thread Socket Reader #1 for port 
> 35917



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to