[
https://issues.apache.org/jira/browse/HBASE-7729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13568154#comment-13568154
]
Lars Hofhansl commented on HBASE-7729:
--------------------------------------
I see the following event:
{{2013-01-31 04:51:22,352 INFO [pool-1-thread-1] util.JVMClusterUtil(262):
Shutdown of 1 master(s) and 1 regionserver(s) complete}}
Followed by:
{code}
013-01-31 04:51:22,354 DEBUG [Master:0;hemera.apache.org,40481,1359607875904]
catalog.CatalogTracker(227): Stopping catalog tracker
org.apache.hadoop.hbase.catalog.CatalogTracker@13c66d8
2013-01-31 04:51:22,354 INFO
[hemera.apache.org,40481,1359607875904.timeoutMonitor] hbase.Chore(80):
hemera.apache.org,40481,1359607875904.timeoutMonitor exiting
2013-01-31 04:51:22,354 INFO
[hemera.apache.org,40481,1359607875904.timerUpdater] hbase.Chore(80):
hemera.apache.org,40481,1359607875904.timerUpdater exiting
2013-01-31 04:51:22,355 DEBUG
[Master:0;hemera.apache.org,40481,1359607875904-EventThread]
zookeeper.ZKUtil(419): hconnection-0x13c8ef1c6650002 /hbase/master does not
exist. Watcher is set.
2013-01-31 04:51:22,356 INFO [Master:0;hemera.apache.org,40481,1359607875904]
master.HMaster(410): HMaster main thread exiting
{code}
So JVMClusterUtil concluded that is done with shutting down the cluster, which
is strange, since JVMClusterUtil loops until Thread.isAlive() returns false for
all region server and master threads.
Maybe it's a logging anomaly. In other runs I've seen as much as 8ms between
cluster thinking it's down and the Master actually leaving its main loop (the
ZK threads live on much longer anyway).
> TestCatalogTrackerOnCluster.testbadOriginalRootLocation fails occasionally
> --------------------------------------------------------------------------
>
> Key: HBASE-7729
> URL: https://issues.apache.org/jira/browse/HBASE-7729
> Project: HBase
> Issue Type: Bug
> Reporter: Lars Hofhansl
>
> Failure:
> {code}
> java.io.IOException: Shutting down
> at
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:223)
> at
> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:86)
> at
> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:77)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:650)
> at
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.testBadOriginalRootLocation(TestCatalogTrackerOnCluster.java:68)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
> at org.junit.rules.RunRules.evaluate(RunRules.java:18)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: Master not initialized after 200
> seconds
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:206)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:420)
> at
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:216)
> ... 32 more
> {code}
> Likely caused by this:
> {code}
> 2013-01-31 04:52:23,064 FATAL
> [Master:0;hemera.apache.org,52696,1359607882775] master.HMaster(1493):
> Unhandled exception. Starting shutdown.
> org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is
> in the failed servers list: example.org/192.0.43.10:1234
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:425)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
> at $Proxy19.getProtocolVersion(Unknown Source)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1335)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1291)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1278)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:506)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getRootServerConnection(CatalogTracker.java:343)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRootServerConnection(CatalogTracker.java:327)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRootRegionLocation(CatalogTracker.java:599)
> at
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:659)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:560)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:376)
> at java.lang.Thread.run(Thread.java:662)
> 2013-01-31 04:52:23,064 INFO
> [Master:0;hemera.apache.org,52696,1359607882775] master.HMaster(1652):
> Aborting
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
