Can you see your two regions in the UI? What if you restart hbase? Does this fix the issue?
Going by your master log, it seems happy enough, almost as though it were a different cluser altogether. You could try scanning '.META.' to see if you got your two regions back. You could try restarting the regionserver that is currently hosting .META.? St.Ack 2011/4/28 bijieshan <[email protected]>: > Thanks Stack! > > When I scan the -ROOT- table, it's not empty. So I delete all the records in > -ROOT- using as you told: > > Hbase > delateall '-ROOT-','.META.,,1' > > After that, I scan the -ROOT again , the records was indeed deleted. > > Hbase > scan '-ROOT-' > 0 row(s) in 0.0210 seconds. > > And then, I unassign the META region: > > Hbase > unassign ".META.,,1" > > After a long while, I executed list: > > hbase(main):002:0> list > TABLE > > ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to > contact region server null for region , row '', but failed after 7 attempts. > Exceptions: > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > > Here is some help for this command: > List all tables in hbase. Optional regular expression parameter could > be used to filter the output. Examples: > > hbase> list > hbase> list 'abc.*' > > In HMaster logs, there's no effect happened: > > 2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 > leastloaded=1 > 2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: > Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s) > 2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 > leastloaded=1 > 2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: > Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s) > 2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 > leastloaded=1 > 2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: > Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s) > 2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 > leastloaded=1 > 2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: > Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s) > 2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 > leastloaded=1 > > Any problem exist in my operations? > > Jieshan Bean > > > -----邮件原件----- > 发件人: [email protected] [mailto:[email protected]] 代表 Stack > 发送时间: 2011年4月29日 11:41 > 收件人: [email protected] > 抄送: Chenjian > 主题: Re: HRegionInfo was null or empty in -ROOT- > > In general, its usually best to let hbase manage all under the /hbase > dir in hdfs. > > What happens if you scan the -ROOT- table in shell? > > hbase> scan "-ROOT-" > > Is it empty there? > > If so, thats not the real issue. The real issue is why .META. is not > being deployed. > > Try removing the entry in -ROOT- > > hbase> deleteall ".META.,,1" > > Then, try doing something like unassigning .META. > > hbase> unassign .META. > > Tail the master log. See if the above has any effect. > > St.Ack > > 2011/4/28 bijieshan <[email protected]>: >> Thanks J-D. >> I think it's not this problem. I have checked the logs, there's no DNS >> exception. >> Some one deleted some hdfs files, use the following method: >> >> ./hadoop fs -rm >> /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239 >> >> I don't know whether these operations can bring any effects. >> >> You can check the log files from the attachments. >> >> Jieshan Bean >> >> -----邮件原件----- >> 发件人: [email protected] [mailto:[email protected]] 代表 Jean-Daniel Cryans >> 发送时间: 2011年4月29日 10:53 >> 收件人: [email protected] >> 抄送: Chenjian >> 主题: Re: HRegionInfo was null or empty in -ROOT- >> >> Last time I heard about this issue, it was because of a DNS problem. >> Look at your master and region server logs. >> >> J-D >> >> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <[email protected]> wrote: >>> Hi, >>> >>> When I execute command of 'list' in hbase shell, it told me that the >>> HRegioninfo was null or empty in -ROOT-. >>> Did the -ROOT- table value got corrupted or any other reasons cause this >>> problem? >>> >>> Here's the info from the 'list' and 'hbck' tool: >>> -------------------------------------------------------------------------------------------------------------------- >>> hbase(main):001:0> list >>> TABLE >>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString >>> as an integer but got as null.Hence setting to default value 1048575 >>> >>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to >>> contact region server null for region , row '', but failed after 7 attempts. >>> Exceptions: >>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>> -------------------------------------------------------------------------------------------------------------------- >>> HBase hbck tool: >>> -------------------------------------------------------------------------------------------------------------------- >>> 11/04/29 10:45:42 DEBUG >>> client.HConnectionManager$HConnectionImplementation: locateRegionInMeta >>> parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: >>> -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 >>> because: HRegionInfo was null or empty in -ROOT-, >>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>> 11/04/29 10:45:42 DEBUG >>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>> location, >>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>> hsa=Linux-1:60020 >>> 11/04/29 10:45:44 DEBUG >>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>> location, >>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>> hsa=Linux-1:60020 >>> 11/04/29 10:45:44 DEBUG >>> client.HConnectionManager$HConnectionImplementation: locateRegionInMeta >>> parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: >>> -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 >>> because: HRegionInfo was null or empty in -ROOT-, >>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>> 11/04/29 10:45:44 DEBUG >>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>> location, >>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>> hsa=Linux-1:60020 >>> 11/04/29 10:45:48 DEBUG >>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>> location, >>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>> hsa=Linux-1:60020 >>> 11/04/29 10:45:48 DEBUG >>> client.HConnectionManager$HConnectionImplementation: locateRegionInMeta >>> parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: >>> -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 >>> because: HRegionInfo was null or empty in -ROOT-, >>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>> 11/04/29 10:45:48 DEBUG >>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>> location, >>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>> hsa=Linux-1:60020 >>> 11/04/29 10:45:52 DEBUG >>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>> location, >>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>> hsa=Linux-1:60020 >>> 11/04/29 10:45:52 DEBUG >>> client.HConnectionManager$HConnectionImplementation: locateRegionInMeta >>> parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: >>> -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 >>> because: HRegionInfo was null or empty in -ROOT-, >>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>> 11/04/29 10:45:52 DEBUG >>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>> location, >>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>> hsa=Linux-1:60020 >>> 11/04/29 10:46:00 DEBUG >>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>> location, >>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>> hsa=Linux-1:60020 >>> 11/04/29 10:46:00 DEBUG >>> client.HConnectionManager$HConnectionImplementation: locateRegionInMeta >>> parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: >>> -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 >>> because: HRegionInfo was null or empty in -ROOT-, >>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>> 11/04/29 10:46:00 DEBUG >>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>> location, >>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>> hsa=Linux-1:60020 >>> -------------------------------------------------------------------------------------------------------------------- >>> >>> Thanks, >>> >>> Regards, >>> Jieshan bean >>> >>> >> >
