In my cluster , I found a issue that root was assigned after 30 minutes when
the cluster start up, because region server startup slowly.
Region server start up :
Step 1: report to master it has started up.
Step 2: init the service
Hmaster start up:
Step 1:Wait for the region server.
Step 2:Assign the root and meta.
Because the preceding section, if the region server initializes service slowly.
Master may throw exception "Server is not running yet".
The root can't be assigned and need wait for 30 minutes.
Is there any good way to modify this bug? Who can give me some suggestions?
The logs:
2011-08-26 07:53:26,065 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
No previous transition plan was found (or we are ignoring an existing plan) for
-ROOT-,,0.70236052 so generated a random one; hri=-ROOT-,,0.70236052, src=,
dest=linux-kqm6,20020,1314316432465; 1 (online=1, exclude=null) available
servers
2011-08-26 07:53:26,065 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Assigning region -ROOT-,,0.70236052 to linux-kqm6,20020,1314316432465
2011-08-26 07:53:26,065 DEBUG org.apache.hadoop.hbase.master.ServerManager: New
connection to linux-kqm6,20020,1314316432465
2011-08-26 07:53:33,251 WARN
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
RemoteException connecting to RS
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running yet
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
at
org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:620)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:555)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1043)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:858)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:838)
at
org.apache.hadoop.hbase.master.AssignmentManager.assignRoot(AssignmentManager.java:1304)
at
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:431)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
2011-08-26 07:53:33,254 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Failed assignment of -ROOT-,,0.70236052 to
serverName=linux-kqm6,20020,1314316432465, load=(requests=0, regions=0,
usedHeap=0, maxHeap=0), trying to assign elsewhere instead; retry=0
org.apache.hadoop.hbase.ipc.ServerNotRunningException:
org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running yet
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:977)
at
org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:620)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:555)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1043)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:858)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:838)
at
org.apache.hadoop.hbase.master.AssignmentManager.assignRoot(AssignmentManager.java:1304)
at
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:431)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
2011-08-26 07:53:33,254 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Unable to find a viable location to assign region -ROOT-,,0.70236052
2011-08-26 07:55:41,226 INFO org.apache.hadoop.hbase.master.ServerManager:
Registering server=linux-kqm5,20020,1314316540832, regionCount=0, userLoad=false
2011-08-26 07:56:13,037 INFO org.apache.hadoop.hbase.master.ServerManager:
Registering server=linux-kqm4,20020,1314316645788, regionCount=0, userLoad=false
2011-08-26 08:23:36,528 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out: -ROOT-,,0.70236052 state=OFFLINE,
ts=1314316413254
2011-08-26 08:23:36,529 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OFFLINE for too long, reassigning -ROOT-,,0.70236052 to a
random server
2011-08-26 08:23:36,529 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE; was=-ROOT-,,0.70236052 state=OFFLINE, ts=1314316413254
