Thanks Stack. Sorry for replying you so late. Because I just have several days of rest. I will notice that I should not attach the logs directly. And I hope I can do the test according to your suggestion immediately. What I'm not sure is whether the env is still kept there :(
This cluster of hbase is 0.90.1. Thanks, Jieshan Bean -----邮件原件----- 发件人: [email protected] [mailto:[email protected]] 代表 Stack 发送时间: 2011年4月30日 0:05 收件人: [email protected] 主题: Re: HRegionInfo was null or empty in -ROOT- Your logs did not come across. This list does not allow attachments usually. I would like you to delete all in the -ROOT- table and then pastebin what is in your master log (Did you say what version of hbase this?). Thanks, St.Ack 2011/4/29 bijieshan <[email protected]>: > I could't see the two regions in the UI. > After I restarted hbase, this issue was still exist. I scaned the -ROOT- > table and .META. table, saw the same exception as "HRegionInfo was null or > empty in -ROOT-". > Before the exception occurred, someone else deleted some log files: > > /hbase/.logs/Linux-1,60020,1303815506931/Linux-1%3A60020.1303822717531 > /hbase/.logs/Linux-2,60020,1303815512433/Linux-2%3A60020.1303826323694 > /hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692 > /hbase/.logs/Linux-1,60020,1303822864554/Linux-1%3A60020.1303822864894 > /hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692 > /hbase/.logs/Linux-3,60020,1303842736457/Linux-3%3A60020.1303842860921 > /hbase/.logs/Linux-2,60020,1303842932959/Linux-2%3A60020.1303842943315 > /hbase/.logs/Linux-1,60020,1303842785259/Linux-1%3A60020.1303846459356 > /hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368 > /hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368 > /hbase/.logs/Linux-2,60020,1303896592421/Linux-2%3A60020.1303896602802 > /hbase/.logs/Linux-3,60020,1303896579647/Linux-3%3A60020.1303896590052 > > I don't know whether these operations are the direct causation. > > 【Error info from the page of master.jsp 】 > > Caused by: > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact > region server null for region , row '', but failed after 10 attempts. > Exceptions: > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > > 【Error info from hbase shell】 > hbase(main):001:0> list > TABLE > 11/04/29 15:24:59 INFO jute.BinaryInputArchive: Expecting maxBufferString as > an integer but got as null.Hence setting to default value 1048575 > > ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to > contact region server null for region , row '', but failed after 7 attempts. > Exceptions: > java.io.IOException: HRegionInfo was null or empty in -ROOT-, > row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, > .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8} > java.io.IOException: HRegionInfo was null or empty in -ROOT-, > row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13, > .META.,,1/info:serverstartcode/1304060872102/Put/vlen=8} > > > hbase(main):002:0> > hbase(main):003:0* scan '-ROOT-' > ROW COLUMN+CELL > .META.,,1 column=info:server, > timestamp=1304062076821, value=Linux-2:60020 > .META.,,1 column=info:serverstartcode, > timestamp=1304062076821, value=1304060518330 > 1 row(s) in 0.0830 seconds > > hbase(main):004:0> scan '.META.' > ROW COLUMN+CELL > > ERROR: java.io.IOException: HRegionInfo was null or empty in -ROOT-, > row=keyvalues={.META.,,1/info:server/1304062076821/Put/vlen=13, > .META.,,1/info:serverstartcode/1304062076821/Put/vlen=8} > > > All the logs I have attached in the email. > > Thanks! > > Jieshan Bean > > > > -----邮件原件----- > 发件人: [email protected] [mailto:[email protected]] 代表 Stack > 发送时间: 2011年4月29日 12:10 > 收件人: [email protected] > 抄送: Chenjian > 主题: Re: HRegionInfo was null or empty in -ROOT- > > Can you see your two regions in the UI? > > What if you restart hbase? Does this fix the issue? > > Going by your master log, it seems happy enough, almost as though it > were a different cluser altogether. > > You could try scanning '.META.' to see if you got your two regions back. > > You could try restarting the regionserver that is currently hosting .META.? > > St.Ack > > 2011/4/28 bijieshan <[email protected]>: >> Thanks Stack! >> >> When I scan the -ROOT- table, it's not empty. So I delete all the records in >> -ROOT- using as you told: >> >> Hbase > delateall '-ROOT-','.META.,,1' >> >> After that, I scan the -ROOT again , the records was indeed deleted. >> >> Hbase > scan '-ROOT-' >> 0 row(s) in 0.0210 seconds. >> >> And then, I unassign the META region: >> >> Hbase > unassign ".META.,,1" >> >> After a long while, I executed list: >> >> hbase(main):002:0> list >> TABLE >> >> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to >> contact region server null for region , row '', but failed after 7 attempts. >> Exceptions: >> org.apache.hadoop.hbase.TableNotFoundException: .META. >> org.apache.hadoop.hbase.TableNotFoundException: .META. >> org.apache.hadoop.hbase.TableNotFoundException: .META. >> org.apache.hadoop.hbase.TableNotFoundException: .META. >> org.apache.hadoop.hbase.TableNotFoundException: .META. >> org.apache.hadoop.hbase.TableNotFoundException: .META. >> org.apache.hadoop.hbase.TableNotFoundException: .META. >> >> Here is some help for this command: >> List all tables in hbase. Optional regular expression parameter could >> be used to filter the output. Examples: >> >> hbase> list >> hbase> list 'abc.*' >> >> In HMaster logs, there's no effect happened: >> >> 2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: >> Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 >> leastloaded=1 >> 2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: >> Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s) >> 2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: >> Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 >> leastloaded=1 >> 2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: >> Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s) >> 2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: >> Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 >> leastloaded=1 >> 2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: >> Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s) >> 2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: >> Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 >> leastloaded=1 >> 2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: >> Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s) >> 2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: >> Skipping load balancing. servers=3 regions=2 average=0.6666667 mostloaded=1 >> leastloaded=1 >> >> Any problem exist in my operations? >> >> Jieshan Bean >> >> >> -----邮件原件----- >> 发件人: [email protected] [mailto:[email protected]] 代表 Stack >> 发送时间: 2011年4月29日 11:41 >> 收件人: [email protected] >> 抄送: Chenjian >> 主题: Re: HRegionInfo was null or empty in -ROOT- >> >> In general, its usually best to let hbase manage all under the /hbase >> dir in hdfs. >> >> What happens if you scan the -ROOT- table in shell? >> >> hbase> scan "-ROOT-" >> >> Is it empty there? >> >> If so, thats not the real issue. The real issue is why .META. is not >> being deployed. >> >> Try removing the entry in -ROOT- >> >> hbase> deleteall ".META.,,1" >> >> Then, try doing something like unassigning .META. >> >> hbase> unassign .META. >> >> Tail the master log. See if the above has any effect. >> >> St.Ack >> >> 2011/4/28 bijieshan <[email protected]>: >>> Thanks J-D. >>> I think it's not this problem. I have checked the logs, there's no DNS >>> exception. >>> Some one deleted some hdfs files, use the following method: >>> >>> ./hadoop fs -rm >>> /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239 >>> >>> I don't know whether these operations can bring any effects. >>> >>> You can check the log files from the attachments. >>> >>> Jieshan Bean >>> >>> -----邮件原件----- >>> 发件人: [email protected] [mailto:[email protected]] 代表 Jean-Daniel Cryans >>> 发送时间: 2011年4月29日 10:53 >>> 收件人: [email protected] >>> 抄送: Chenjian >>> 主题: Re: HRegionInfo was null or empty in -ROOT- >>> >>> Last time I heard about this issue, it was because of a DNS problem. >>> Look at your master and region server logs. >>> >>> J-D >>> >>> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <[email protected]> wrote: >>>> Hi, >>>> >>>> When I execute command of 'list' in hbase shell, it told me that the >>>> HRegioninfo was null or empty in -ROOT-. >>>> Did the -ROOT- table value got corrupted or any other reasons cause this >>>> problem? >>>> >>>> Here's the info from the 'list' and 'hbck' tool: >>>> -------------------------------------------------------------------------------------------------------------------- >>>> hbase(main):001:0> list >>>> TABLE >>>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString >>>> as an integer but got as null.Hence setting to default value 1048575 >>>> >>>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to >>>> contact region server null for region , row '', but failed after 7 >>>> attempts. >>>> Exceptions: >>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, >>>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>>> -------------------------------------------------------------------------------------------------------------------- >>>> HBase hbck tool: >>>> -------------------------------------------------------------------------------------------------------------------- >>>> 11/04/29 10:45:42 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: locateRegionInMeta >>>> parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: >>>> -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 >>>> because: HRegionInfo was null or empty in -ROOT-, >>>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>>> 11/04/29 10:45:42 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>>> location, >>>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>>> hsa=Linux-1:60020 >>>> 11/04/29 10:45:44 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>>> location, >>>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>>> hsa=Linux-1:60020 >>>> 11/04/29 10:45:44 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: locateRegionInMeta >>>> parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: >>>> -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 >>>> because: HRegionInfo was null or empty in -ROOT-, >>>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>>> 11/04/29 10:45:44 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>>> location, >>>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>>> hsa=Linux-1:60020 >>>> 11/04/29 10:45:48 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>>> location, >>>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>>> hsa=Linux-1:60020 >>>> 11/04/29 10:45:48 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: locateRegionInMeta >>>> parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: >>>> -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 >>>> because: HRegionInfo was null or empty in -ROOT-, >>>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>>> 11/04/29 10:45:48 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>>> location, >>>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>>> hsa=Linux-1:60020 >>>> 11/04/29 10:45:52 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>>> location, >>>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>>> hsa=Linux-1:60020 >>>> 11/04/29 10:45:52 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: locateRegionInMeta >>>> parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: >>>> -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 >>>> because: HRegionInfo was null or empty in -ROOT-, >>>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>>> 11/04/29 10:45:52 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>>> location, >>>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>>> hsa=Linux-1:60020 >>>> 11/04/29 10:46:00 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>>> location, >>>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>>> hsa=Linux-1:60020 >>>> 11/04/29 10:46:00 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: locateRegionInMeta >>>> parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: >>>> -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 >>>> because: HRegionInfo was null or empty in -ROOT-, >>>> row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, >>>> .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8} >>>> 11/04/29 10:46:00 DEBUG >>>> client.HConnectionManager$HConnectionImplementation: Lookedup root region >>>> location, >>>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; >>>> hsa=Linux-1:60020 >>>> -------------------------------------------------------------------------------------------------------------------- >>>> >>>> Thanks, >>>> >>>> Regards, >>>> Jieshan bean >>>> >>>> >>> >> >
