These regions that are in hdfs but not in .META. and not on any region server are probably harmless. Would be interesting to trace how they got to this state. My guess is that they are old let go regions that were not cleaned up.
The region directory name is its encoded name which is the tail part of the row name in the .META. table. St.Ack 2011/4/11 茅旭峰 <[email protected]>: > the results of ./bin/hbase hbck show lots of 'inconsistence status' like > > ==== > ERROR: Region > hdfs://cloud137:9000/hbase/table1/01c80f8b54523ad6c242c5f695544f16 on HDFS, > but not listed in META or deployed on any region server. > ERROR: Region > hdfs://cloud137:9000/hbase/table1/01ce4e2f72baa0df51b7b2010c9d1ef0 on HDFS, > but not listed in META or deployed on any region server. > ERROR: Region > hdfs://cloud137:9000/hbase/table1/01d0855af5e646823bdff82fddadd81e on HDFS, > but not listed in META or deployed on any region server. > ==== > > How can I match the names of the directories from the .META.? > > I checked the help in hbase shell, but looks like it only has close_region, > no open_region. > > 2011/4/12 Stack <[email protected]> >> >> Can you open the region again? (See shell commands for opening regions). >> >> What does hbck say: ./bin/hbase hbck. >> >> Add the -details flag. >> >> It might tell you a story about an offlined region. >> >> 0.90.2 has some fixes for issues in and around here (CDH3 release, out >> on the 14th, has most of them bundled). >> >> St.Ack >> >> 2011/4/11 茅旭峰 <[email protected]>: >> > From the output of scan '.META.' I pasted before, we can see there are >> > two >> > key ranges >> > which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They are >> > >> > #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A=' >> > #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key' >> > >> > The output has less info about the second key range. And I also found >> > some >> > log like >> > >> > ==== >> > >> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> > 09:48:36,953 DEBUG >> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: >> > Processing >> > close of >> > >> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >> > >> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing >> > >> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.: >> > disabling compactions & flushes >> > >> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates >> > disabled for region >> > >> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >> > >> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> > 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed >> > >> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >> > >> > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> > 09:48:36,953 DEBUG >> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed >> > region >> > >> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >> > ==== >> > >> > It looks like the second had already been closed for some reason. Is it >> > possile that the info of that region >> > is not clean clearly? How can I check this in detail? >> > >> > I'm guessing maybe 'getClosestRowBefore' returns the closed region in >> > HConnectionManager.locateRegionInMeta, >> > which is closed, so this leads to the problem. >> > >> > 2011/4/12 茅旭峰 <[email protected]> >> > >> >> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the >> >> issue further, but my understanding is the meta info and the root >> >> region are saved by zookeeper, right? Do I need to check them there? >> >> >> >> m9suns >> >> >> >> 在 2011-4-12，0:40，Jean-Daniel Cryans <[email protected]> 写道： >> >> >> >> > It's possible under some bugs, which HBase version are you using? >> >> > >> >> > J-D >> >> > >> >> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <[email protected]> wrote: >> >> >> Hi, >> >> >> >> >> >> Is it possible that some table cannot cover the whole key space. >> >> >> What I >> >> saw >> >> >> was like >> >> >> >> >> >> ==== >> >> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123' >> >> >> >> >> >> 0 row(s) in 0.3030 seconds >> >> >> >> >> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=', >> >> >> 'cfEStore:dasd', '123' >> >> >> >> >> >> ERROR: java.io.IOException: HRegionInfo was null or empty in .META., >> >> >> >> >> >> >> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14, >> >> >> >> >> >> >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8} >> >> >> >> >> >> Here is some help for this command: >> >> >> Put a cell 'value' at specified table/row/column and optionally >> >> >> timestamp coordinates. To put a cell value into table 't1' at >> >> >> row 'r1' under column 'c1' marked with the time 'ts1', do: >> >> >> >> >> >> hbase> put 't1', 'r1', 'c1', 'value', ts1 >> >> >> >> >> >> ==== >> >> >> >> >> >> I guess this means our table has lost some key range for the >> >> >> specific >> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=', >> >> >> is this reasonable? I wonder what might cause this issue? How can I >> >> check >> >> >> it? >> >> >> >> >> >> One weird thing, I can see such lines like >> >> >> >> >> >> ==== >> >> >> >> >> >> >> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625. >> >> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A= >> >> >> ==== >> >> >> >> >> >> from the web UI, which I think means the table has holds the key >> >> >> range >> >> for >> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=' >> >> >> >> >> >> I also have not seen any errors/warnings in the log file. Any >> >> suggestions? >> >> >> >> >> >> Thanks in advance. >> >> >> >> >> >> > > >
