Is that really from the master log? Can we get the full log in a pastebin? J-D
On Tue, Nov 23, 2010 at 7:40 AM, Ted Yu <[email protected]> wrote: > I backed up zookeeper dataDir to another location. > After clearing zookeeper dataDir, HMaster still couldn't start: > > 2010-11-23 15:30:56,095 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/master > got 10.202.50.100:60000 > 2010-11-23 15:30:56,119 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/root-region-server > 2010-11-23 15:30:56,120 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping > 5000ms, waiting for root region. > 2010-11-23 15:31:01,125 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/root-region-server > 2010-11-23 15:31:01,125 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping > 5000ms, waiting for root region. > > Disk isn't full: > /dev/md2 2786058952 186234928 2456017408 8% / > > Comment is appreciated. > > On Tue, Nov 23, 2010 at 5:34 AM, Ted Yu <[email protected]> wrote: > >> I tried to restart hbase. But the region server identified by ZNode >> /hbase/root-region-server declared that it is not serving root region: >> >> 2010-11-23 13:26:49,617 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: >> locateRegionInMeta attempt 1 of 3 failed; retrying after sleep of 5000 >> because: Timed out trying to locate root region because: >> org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0 >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2274) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:1711) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:998) >> >> 2010-11-23 13:26:54,622 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.202.50.111:60020 >> 2010-11-23 13:26:54,624 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> 2010-11-23 13:26:59,626 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> 2010-11-23 13:26:59,629 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.202.50.111:60020 >> 2010-11-23 13:26:59,630 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> 2010-11-23 13:27:04,632 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> 2010-11-23 13:27:04,635 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.202.50.111:60020 >> >> What should I do next ? >> >> Thanks >> >> >> On Tue, Nov 23, 2010 at 1:37 AM, Lars George <[email protected]>wrote: >> >>> Hi Ted, >>> >>> So one of the regions is not being released? Could you try and see >>> from .META. which is still deployed and use the shell's "close_region" >>> to close it while looking at the master and region server logs to see >>> what is going on? Maybe best if you switch the RS to DEBUG level >>> logging first to get some info? >>> >>> Lars >>> >>> On Tue, Nov 23, 2010 at 8:25 AM, Ted Yu <[email protected]> wrote: >>> > Hi >>> > We use 0.20.6 >>> > >>> > I tried to disable packageindex table. From master log: >>> > >>> > 2010-11-23 07:21:06,326 DEBUG >>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region >>> > packageindex,CC7E6FEA4CDCF19C6F4AC9BB51EF6A33,1290230596786 to >>> setClosing >>> > list for us01-ciqps1-grid10.carrieriq.com,60020,1290493641949 >>> > 2010-11-23 07:21:06,326 DEBUG >>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region >>> > packageindex,F2A18967F48C9FDA9C23BF9A8210ED17,1290230394345 to >>> setClosing >>> > list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228 >>> > 2010-11-23 07:21:06,326 DEBUG >>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region >>> > packageindex,E8FA713B2F030EF012E5AB0A641CB1DB,1290230356969 to >>> setClosing >>> > list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228 >>> > 2010-11-23 07:21:06,327 DEBUG >>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region >>> > packageindex,5B10CA26DCAEFBFF4A63DB7D0432D628,1290229869191 to >>> setClosing >>> > list for us01-ciqps1-grid12.carrieriq.com,60020,1290493641232 >>> > 2010-11-23 07:21:20,178 INFO >>> org.apache.hadoop.hbase.master.ServerManager: >>> > 15 region servers, 0 dead, average load 123.66666666666667 >>> > 2010-11-23 07:21:20,252 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> > RegionManager.rootScanner scanning meta region {server: >>> 10.202.50.111:60020, >>> > regionname: -ROOT-,,0, startKey: <>} >>> > 2010-11-23 07:21:20,257 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> > RegionManager.rootScanner scan of 1 row(s) of meta region {server: >>> > 10.202.50.111:60020, regionname: -ROOT-,,0, startKey: <>} complete >>> > 2010-11-23 07:21:22,838 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> > RegionManager.metaScanner scanning meta region {server: >>> 10.202.50.101:60020, >>> > regionname: .META.,,1, startKey: <>} >>> > 2010-11-23 07:21:24,731 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> > RegionManager.metaScanner scan of 2086 row(s) of meta region {server: >>> > 10.202.50.101:60020, regionname: .META.,,1, startKey: <>} complete >>> > 2010-11-23 07:21:24,731 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> All >>> > 1 .META. region(s) scanned >>> > >>> > But I always got: >>> > hbase(main):004:0> disable 'packageindex' >>> > NativeException: org.apache.hadoop.hbase.RegionException: Retries >>> exhausted, >>> > it took too long to wait for the table packageindex to be disabled. >>> > >>> > What should I do to disable the table ? >>> > >>> > Thanks >>> > >>> >> >> >
