You can also cat the .regioninfo file that is under the region directory to learn more about the region -- its HRegionInfo (The file format is serialized HRegionInfo followed by a String version so intelligible to non-machines). St.Ack
2011/4/12 Stack <[email protected]>: > These regions that are in hdfs but not in .META. and not on any region > server are probably harmless. Would be interesting to trace how they > got to this state. My guess is that they are old let go regions that > were not cleaned up. > > The region directory name is its encoded name which is the tail part > of the row name in the .META. table. > > St.Ack > > > 2011/4/11 茅旭峰 <[email protected]>: >> the results of ./bin/hbase hbck show lots of 'inconsistence status' like >> >> ==== >> ERROR: Region >> hdfs://cloud137:9000/hbase/table1/01c80f8b54523ad6c242c5f695544f16 on HDFS, >> but not listed in META or deployed on any region server. >> ERROR: Region >> hdfs://cloud137:9000/hbase/table1/01ce4e2f72baa0df51b7b2010c9d1ef0 on HDFS, >> but not listed in META or deployed on any region server. >> ERROR: Region >> hdfs://cloud137:9000/hbase/table1/01d0855af5e646823bdff82fddadd81e on HDFS, >> but not listed in META or deployed on any region server. >> ==== >> >> How can I match the names of the directories from the .META.? >> >> I checked the help in hbase shell, but looks like it only has close_region, >> no open_region. >> >> 2011/4/12 Stack <[email protected]> >>> >>> Can you open the region again? (See shell commands for opening regions). >>> >>> What does hbck say: ./bin/hbase hbck. >>> >>> Add the -details flag. >>> >>> It might tell you a story about an offlined region. >>> >>> 0.90.2 has some fixes for issues in and around here (CDH3 release, out >>> on the 14th, has most of them bundled). >>> >>> St.Ack >>> >>> 2011/4/11 茅旭峰 <[email protected]>: >>> > From the output of scan '.META.' I pasted before, we can see there are >>> > two >>> > key ranges >>> > which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They are >>> > >>> > #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A=' >>> > #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key' >>> > >>> > The output has less info about the second key range. And I also found >>> > some >>> > log like >>> > >>> > ==== >>> > >>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >>> > 09:48:36,953 DEBUG >>> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: >>> > Processing >>> > close of >>> > >>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >>> > >>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >>> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing >>> > >>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.: >>> > disabling compactions & flushes >>> > >>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >>> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates >>> > disabled for region >>> > >>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >>> > >>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >>> > 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed >>> > >>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >>> > >>> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >>> > 09:48:36,953 DEBUG >>> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed >>> > region >>> > >>> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >>> > ==== >>> > >>> > It looks like the second had already been closed for some reason. Is it >>> > possile that the info of that region >>> > is not clean clearly? How can I check this in detail? >>> > >>> > I'm guessing maybe 'getClosestRowBefore' returns the closed region in >>> > HConnectionManager.locateRegionInMeta, >>> > which is closed, so this leads to the problem. >>> > >>> > 2011/4/12 茅旭峰 <[email protected]> >>> > >>> >> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the >>> >> issue further, but my understanding is the meta info and the root >>> >> region are saved by zookeeper, right? Do I need to check them there? >>> >> >>> >> m9suns >>> >> >>> >> 在 2011-4-12，0:40，Jean-Daniel Cryans <[email protected]> 写道： >>> >> >>> >> > It's possible under some bugs, which HBase version are you using? >>> >> > >>> >> > J-D >>> >> > >>> >> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <[email protected]> wrote: >>> >> >> Hi, >>> >> >> >>> >> >> Is it possible that some table cannot cover the whole key space. >>> >> >> What I >>> >> saw >>> >> >> was like >>> >> >> >>> >> >> ==== >>> >> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123' >>> >> >> >>> >> >> 0 row(s) in 0.3030 seconds >>> >> >> >>> >> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=', >>> >> >> 'cfEStore:dasd', '123' >>> >> >> >>> >> >> ERROR: java.io.IOException: HRegionInfo was null or empty in .META., >>> >> >> >>> >> >>> >> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14, >>> >> >> >>> >> >>> >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8} >>> >> >> >>> >> >> Here is some help for this command: >>> >> >> Put a cell 'value' at specified table/row/column and optionally >>> >> >> timestamp coordinates. To put a cell value into table 't1' at >>> >> >> row 'r1' under column 'c1' marked with the time 'ts1', do: >>> >> >> >>> >> >> hbase> put 't1', 'r1', 'c1', 'value', ts1 >>> >> >> >>> >> >> ==== >>> >> >> >>> >> >> I guess this means our table has lost some key range for the >>> >> >> specific >>> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=', >>> >> >> is this reasonable? I wonder what might cause this issue? How can I >>> >> check >>> >> >> it? >>> >> >> >>> >> >> One weird thing, I can see such lines like >>> >> >> >>> >> >> ==== >>> >> >> >>> >> >>> >> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625. >>> >> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A= >>> >> >> ==== >>> >> >> >>> >> >> from the web UI, which I think means the table has holds the key >>> >> >> range >>> >> for >>> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=' >>> >> >> >>> >> >> I also have not seen any errors/warnings in the log file. Any >>> >> suggestions? >>> >> >> >>> >> >> Thanks in advance. >>> >> >> >>> >> >>> > >> >> >
