http://pastebin.com/E86iPnK4
On Tue, Nov 23, 2010 at 10:23 AM, Jean-Daniel Cryans <jdcry...@apache.org>wrote:
> Is that really from the master log? Can we get the full log in a pastebin?
>
> J-D
>
> On Tue, Nov 23, 2010 at 7:40 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > I backed up zookeeper dataDir to another location.
> > After clearing zookeeper dataDir, HMaster still couldn't start:
> >
> > 2010-11-23 15:30:56,095 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/master
> > got 10.202.50.100:60000
> > 2010-11-23 15:30:56,119 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read:
> > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> > NoNode for /hbase/root-region-server
> > 2010-11-23 15:30:56,120 DEBUG
> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping
> > 5000ms, waiting for root region.
> > 2010-11-23 15:31:01,125 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read:
> > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> > NoNode for /hbase/root-region-server
> > 2010-11-23 15:31:01,125 DEBUG
> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping
> > 5000ms, waiting for root region.
> >
> > Disk isn't full:
> > /dev/md2 2786058952 186234928 2456017408 8% /
> >
> > Comment is appreciated.
> >
> > On Tue, Nov 23, 2010 at 5:34 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> >> I tried to restart hbase. But the region server identified by ZNode
> >> /hbase/root-region-server declared that it is not serving root region:
> >>
> >> 2010-11-23 13:26:49,617 DEBUG
> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> >> locateRegionInMeta attempt 1 of 3 failed; retrying after sleep of 5000
> >> because: Timed out trying to locate root region because:
> >> org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> >> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2274)
> >> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:1711)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >> at
> >> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> >> at
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:998)
> >>
> >> 2010-11-23 13:26:54,622 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> >> /hbase/root-region-server got 10.202.50.111:60020
> >> 2010-11-23 13:26:54,624 DEBUG
> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root
> region
> >> location changed. Sleeping.
> >> 2010-11-23 13:26:59,626 DEBUG
> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake.
> Retry
> >> finding root region.
> >> 2010-11-23 13:26:59,629 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> >> /hbase/root-region-server got 10.202.50.111:60020
> >> 2010-11-23 13:26:59,630 DEBUG
> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root
> region
> >> location changed. Sleeping.
> >> 2010-11-23 13:27:04,632 DEBUG
> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake.
> Retry
> >> finding root region.
> >> 2010-11-23 13:27:04,635 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> >> /hbase/root-region-server got 10.202.50.111:60020
> >>
> >> What should I do next ?
> >>
> >> Thanks
> >>
> >>
> >> On Tue, Nov 23, 2010 at 1:37 AM, Lars George <lars.geo...@gmail.com
> >wrote:
> >>
> >>> Hi Ted,
> >>>
> >>> So one of the regions is not being released? Could you try and see
> >>> from .META. which is still deployed and use the shell's "close_region"
> >>> to close it while looking at the master and region server logs to see
> >>> what is going on? Maybe best if you switch the RS to DEBUG level
> >>> logging first to get some info?
> >>>
> >>> Lars
> >>>
> >>> On Tue, Nov 23, 2010 at 8:25 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >>> > Hi
> >>> > We use 0.20.6
> >>> >
> >>> > I tried to disable packageindex table. From master log:
> >>> >
> >>> > 2010-11-23 07:21:06,326 DEBUG
> >>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> >>> > packageindex,CC7E6FEA4CDCF19C6F4AC9BB51EF6A33,1290230596786 to
> >>> setClosing
> >>> > list for us01-ciqps1-grid10.carrieriq.com,60020,1290493641949
> >>> > 2010-11-23 07:21:06,326 DEBUG
> >>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> >>> > packageindex,F2A18967F48C9FDA9C23BF9A8210ED17,1290230394345 to
> >>> setClosing
> >>> > list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228
> >>> > 2010-11-23 07:21:06,326 DEBUG
> >>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> >>> > packageindex,E8FA713B2F030EF012E5AB0A641CB1DB,1290230356969 to
> >>> setClosing
> >>> > list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228
> >>> > 2010-11-23 07:21:06,327 DEBUG
> >>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> >>> > packageindex,5B10CA26DCAEFBFF4A63DB7D0432D628,1290229869191 to
> >>> setClosing
> >>> > list for us01-ciqps1-grid12.carrieriq.com,60020,1290493641232
> >>> > 2010-11-23 07:21:20,178 INFO
> >>> org.apache.hadoop.hbase.master.ServerManager:
> >>> > 15 region servers, 0 dead, average load 123.66666666666667
> >>> > 2010-11-23 07:21:20,252 INFO
> org.apache.hadoop.hbase.master.BaseScanner:
> >>> > RegionManager.rootScanner scanning meta region {server:
> >>> 10.202.50.111:60020,
> >>> > regionname: -ROOT-,,0, startKey: <>}
> >>> > 2010-11-23 07:21:20,257 INFO
> org.apache.hadoop.hbase.master.BaseScanner:
> >>> > RegionManager.rootScanner scan of 1 row(s) of meta region {server:
> >>> > 10.202.50.111:60020, regionname: -ROOT-,,0, startKey: <>} complete
> >>> > 2010-11-23 07:21:22,838 INFO
> org.apache.hadoop.hbase.master.BaseScanner:
> >>> > RegionManager.metaScanner scanning meta region {server:
> >>> 10.202.50.101:60020,
> >>> > regionname: .META.,,1, startKey: <>}
> >>> > 2010-11-23 07:21:24,731 INFO
> org.apache.hadoop.hbase.master.BaseScanner:
> >>> > RegionManager.metaScanner scan of 2086 row(s) of meta region {server:
> >>> > 10.202.50.101:60020, regionname: .META.,,1, startKey: <>} complete
> >>> > 2010-11-23 07:21:24,731 INFO
> org.apache.hadoop.hbase.master.BaseScanner:
> >>> All
> >>> > 1 .META. region(s) scanned
> >>> >
> >>> > But I always got:
> >>> > hbase(main):004:0> disable 'packageindex'
> >>> > NativeException: org.apache.hadoop.hbase.RegionException: Retries
> >>> exhausted,
> >>> > it took too long to wait for the table packageindex to be disabled.
> >>> >
> >>> > What should I do to disable the table ?
> >>> >
> >>> > Thanks
> >>> >
> >>>
> >>
> >>
> >
>
