[
https://issues.apache.org/jira/browse/HBASE-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602885#comment-13602885
]
Ted Yu commented on HBASE-8116:
-------------------------------
>From build #3958, we can get more clue:
{code}
2013-03-14 07:05:49,273 INFO [pool-1-thread-1] hbase.ResourceChecker(171):
after:
client.TestSnapshotCloneIndependence#testOfflineSnapshotRegionOperationsIndependent
Thread=276 (was 272)
Potentially hanging thread:
RegionServer:0;janus.apache.org,57486,1363244621206-splits-1363244745097
java.lang.Object.wait(Native Method)
java.lang.Thread.join(Thread.java:1258)
java.lang.Thread.join(Thread.java:1332)
org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:89)
org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:378)
org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:475)
org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:68)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Potentially hanging thread:
janus.apache.org,57486,1363244621206-daughterOpener=72730ff3c3745f6be273de75948a2c30
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:4100)
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3988)
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.writeRegionInfoFileContent(HRegionFileSystem.java:401)
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.writeRegionInfoOnFilesystem(HRegionFileSystem.java:473)
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.checkRegionInfoOnFilesystem(HRegionFileSystem.java:435)
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:567)
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:546)
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4041)
org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:527)
org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:508)
java.lang.Thread.run(Thread.java:722)
Potentially hanging thread:
janus.apache.org,57486,1363244621206-daughterOpener=adff955f142634edcdec1a090f47645f
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
org.apache.hadoop.ipc.Client.call(Client.java:1093)
org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
$Proxy10.delete(Unknown Source)
sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:601)
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
$Proxy10.delete(Unknown Source)
sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:601)
org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:267)
$Proxy19.delete(Unknown Source)
org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:981)
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:245)
org.apache.hadoop.fs.FilterFileSystem.delete(FilterFileSystem.java:154)
org.apache.hadoop.hbase.util.FSUtils.deleteDirectory(FSUtils.java:166)
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.cleanupTempDir(HRegionFileSystem.java:119)
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:571)
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:546)
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4041)
org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:527)
org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:508)
{code}
Looks like there might be race between the two daughterOpener threads.
> TestSnapshotCloneIndependence fails in trunk builds intermittently
> ------------------------------------------------------------------
>
> Key: HBASE-8116
> URL: https://issues.apache.org/jira/browse/HBASE-8116
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
>
> I was looking at
> https://builds.apache.org/job/HBase-TRUNK/3959/testReport/org.apache.hadoop.hbase.client/TestSnapshotCloneIndependence/testOfflineSnapshotRegionOperationsIndependent/
> and found the following:
> {code}
> 2013-03-14 11:11:07,323 INFO [pool-1-thread-1] hbase.ResourceChecker(171):
> after:
> client.TestSnapshotCloneIndependence#testOfflineSnapshotRegionOperationsIndependent
> Thread=275 (was 273)
> Potentially hanging thread:
> janus.apache.org,53542,1363259346619-daughterOpener=34172719c055b187015add70302ab50b
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:503)
> org.apache.hadoop.ipc.Client.call(Client.java:1093)
> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> $Proxy10.rename(Unknown Source)
> sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:601)
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> $Proxy10.rename(Unknown Source)
> sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:601)
> org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:267)
> $Proxy19.rename(Unknown Source)
> org.apache.hadoop.hdfs.DFSClient.rename(DFSClient.java:955)
>
> org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:227)
> org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:144)
>
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.writeRegionInfoOnFilesystem(HRegionFileSystem.java:476)
>
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.checkRegionInfoOnFilesystem(HRegionFileSystem.java:435)
>
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:567)
>
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:546)
>
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4041)
>
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:527)
>
> org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:508)
> java.lang.Thread.run(Thread.java:722)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
