So before that first line... everything was fine? What happened there?
On Tue, Nov 23, 2010 at 10:48 AM, Ted Yu <yuzhih...@gmail.com> wrote: > Here is relevant portion of master log: > http://pastebin.com/HiembAxc > > On Tue, Nov 23, 2010 at 10:37 AM, Jean-Daniel Cryans > <jdcry...@apache.org>wrote: > >> Can you dig up to where it started doing that? >> >> On Tue, Nov 23, 2010 at 10:27 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> > http://pastebin.com/E86iPnK4 >> > >> > On Tue, Nov 23, 2010 at 10:23 AM, Jean-Daniel Cryans < >> jdcry...@apache.org>wrote: >> > >> >> Is that really from the master log? Can we get the full log in a >> pastebin? >> >> >> >> J-D >> >> >> >> On Tue, Nov 23, 2010 at 7:40 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> >> > I backed up zookeeper dataDir to another location. >> >> > After clearing zookeeper dataDir, HMaster still couldn't start: >> >> > >> >> > 2010-11-23 15:30:56,095 DEBUG >> >> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> >> /hbase/master >> >> > got 10.202.50.100:60000 >> >> > 2010-11-23 15:30:56,119 DEBUG >> >> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read: >> >> > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode >> = >> >> > NoNode for /hbase/root-region-server >> >> > 2010-11-23 15:30:56,120 DEBUG >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers: >> Sleeping >> >> > 5000ms, waiting for root region. >> >> > 2010-11-23 15:31:01,125 DEBUG >> >> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read: >> >> > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode >> = >> >> > NoNode for /hbase/root-region-server >> >> > 2010-11-23 15:31:01,125 DEBUG >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers: >> Sleeping >> >> > 5000ms, waiting for root region. >> >> > >> >> > Disk isn't full: >> >> > /dev/md2 2786058952 186234928 2456017408 8% / >> >> > >> >> > Comment is appreciated. >> >> > >> >> > On Tue, Nov 23, 2010 at 5:34 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> >> > >> >> >> I tried to restart hbase. But the region server identified by ZNode >> >> >> /hbase/root-region-server declared that it is not serving root >> region: >> >> >> >> >> >> 2010-11-23 13:26:49,617 DEBUG >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: >> >> >> locateRegionInMeta attempt 1 of 3 failed; retrying after sleep of >> 5000 >> >> >> because: Timed out trying to locate root region because: >> >> >> org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0 >> >> >> at >> >> >> >> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2274) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:1711) >> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> >> >> at >> >> >> >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> >> >> at >> >> >> >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> >> >> at java.lang.reflect.Method.invoke(Method.java:597) >> >> >> at >> >> >> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:998) >> >> >> >> >> >> 2010-11-23 13:26:54,622 DEBUG >> >> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> >> >> /hbase/root-region-server got 10.202.50.111:60020 >> >> >> 2010-11-23 13:26:54,624 DEBUG >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root >> >> region >> >> >> location changed. Sleeping. >> >> >> 2010-11-23 13:26:59,626 DEBUG >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. >> >> Retry >> >> >> finding root region. >> >> >> 2010-11-23 13:26:59,629 DEBUG >> >> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> >> >> /hbase/root-region-server got 10.202.50.111:60020 >> >> >> 2010-11-23 13:26:59,630 DEBUG >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root >> >> region >> >> >> location changed. Sleeping. >> >> >> 2010-11-23 13:27:04,632 DEBUG >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. >> >> Retry >> >> >> finding root region. >> >> >> 2010-11-23 13:27:04,635 DEBUG >> >> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> >> >> /hbase/root-region-server got 10.202.50.111:60020 >> >> >> >> >> >> What should I do next ? >> >> >> >> >> >> Thanks >> >> >> >> >> >> >> >> >> On Tue, Nov 23, 2010 at 1:37 AM, Lars George <lars.geo...@gmail.com >> >> >wrote: >> >> >> >> >> >>> Hi Ted, >> >> >>> >> >> >>> So one of the regions is not being released? Could you try and see >> >> >>> from .META. which is still deployed and use the shell's >> "close_region" >> >> >>> to close it while looking at the master and region server logs to >> see >> >> >>> what is going on? Maybe best if you switch the RS to DEBUG level >> >> >>> logging first to get some info? >> >> >>> >> >> >>> Lars >> >> >>> >> >> >>> On Tue, Nov 23, 2010 at 8:25 AM, Ted Yu <yuzhih...@gmail.com> >> wrote: >> >> >>> > Hi >> >> >>> > We use 0.20.6 >> >> >>> > >> >> >>> > I tried to disable packageindex table. From master log: >> >> >>> > >> >> >>> > 2010-11-23 07:21:06,326 DEBUG >> >> >>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region >> >> >>> > packageindex,CC7E6FEA4CDCF19C6F4AC9BB51EF6A33,1290230596786 to >> >> >>> setClosing >> >> >>> > list for us01-ciqps1-grid10.carrieriq.com,60020,1290493641949 >> >> >>> > 2010-11-23 07:21:06,326 DEBUG >> >> >>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region >> >> >>> > packageindex,F2A18967F48C9FDA9C23BF9A8210ED17,1290230394345 to >> >> >>> setClosing >> >> >>> > list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228 >> >> >>> > 2010-11-23 07:21:06,326 DEBUG >> >> >>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region >> >> >>> > packageindex,E8FA713B2F030EF012E5AB0A641CB1DB,1290230356969 to >> >> >>> setClosing >> >> >>> > list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228 >> >> >>> > 2010-11-23 07:21:06,327 DEBUG >> >> >>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region >> >> >>> > packageindex,5B10CA26DCAEFBFF4A63DB7D0432D628,1290229869191 to >> >> >>> setClosing >> >> >>> > list for us01-ciqps1-grid12.carrieriq.com,60020,1290493641232 >> >> >>> > 2010-11-23 07:21:20,178 INFO >> >> >>> org.apache.hadoop.hbase.master.ServerManager: >> >> >>> > 15 region servers, 0 dead, average load 123.66666666666667 >> >> >>> > 2010-11-23 07:21:20,252 INFO >> >> org.apache.hadoop.hbase.master.BaseScanner: >> >> >>> > RegionManager.rootScanner scanning meta region {server: >> >> >>> 10.202.50.111:60020, >> >> >>> > regionname: -ROOT-,,0, startKey: <>} >> >> >>> > 2010-11-23 07:21:20,257 INFO >> >> org.apache.hadoop.hbase.master.BaseScanner: >> >> >>> > RegionManager.rootScanner scan of 1 row(s) of meta region {server: >> >> >>> > 10.202.50.111:60020, regionname: -ROOT-,,0, startKey: <>} >> complete >> >> >>> > 2010-11-23 07:21:22,838 INFO >> >> org.apache.hadoop.hbase.master.BaseScanner: >> >> >>> > RegionManager.metaScanner scanning meta region {server: >> >> >>> 10.202.50.101:60020, >> >> >>> > regionname: .META.,,1, startKey: <>} >> >> >>> > 2010-11-23 07:21:24,731 INFO >> >> org.apache.hadoop.hbase.master.BaseScanner: >> >> >>> > RegionManager.metaScanner scan of 2086 row(s) of meta region >> {server: >> >> >>> > 10.202.50.101:60020, regionname: .META.,,1, startKey: <>} >> complete >> >> >>> > 2010-11-23 07:21:24,731 INFO >> >> org.apache.hadoop.hbase.master.BaseScanner: >> >> >>> All >> >> >>> > 1 .META. region(s) scanned >> >> >>> > >> >> >>> > But I always got: >> >> >>> > hbase(main):004:0> disable 'packageindex' >> >> >>> > NativeException: org.apache.hadoop.hbase.RegionException: Retries >> >> >>> exhausted, >> >> >>> > it took too long to wait for the table packageindex to be >> disabled. >> >> >>> > >> >> >>> > What should I do to disable the table ? >> >> >>> > >> >> >>> > Thanks >> >> >>> > >> >> >>> >> >> >> >> >> >> >> >> > >> >> >> > >> >
