the results of ./bin/hbase hbck show lots of 'inconsistence status' like ==== ERROR: Region hdfs://cloud137:9000/hbase/table1/01c80f8b54523ad6c242c5f695544f16 on HDFS, but not listed in META or deployed on any region server. ERROR: Region hdfs://cloud137:9000/hbase/table1/01ce4e2f72baa0df51b7b2010c9d1ef0 on HDFS, but not listed in META or deployed on any region server. ERROR: Region hdfs://cloud137:9000/hbase/table1/01d0855af5e646823bdff82fddadd81e on HDFS, but not listed in META or deployed on any region server. ====
How can I match the names of the directories from the .META.? I checked the help in hbase shell, but looks like it only has close_region, no open_region. 2011/4/12 Stack <[email protected]> > Can you open the region again? (See shell commands for opening regions). > > What does hbck say: ./bin/hbase hbck. > > Add the -details flag. > > It might tell you a story about an offlined region. > > 0.90.2 has some fixes for issues in and around here (CDH3 release, out > on the 14th, has most of them bundled). > > St.Ack > > 2011/4/11 茅旭峰 <[email protected]>: > > From the output of scan '.META.' I pasted before, we can see there are > two > > key ranges > > which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They are > > > > #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A=' > > #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key' > > > > The output has less info about the second key range. And I also found > some > > log like > > > > ==== > > > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 > > 09:48:36,953 DEBUG > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: > Processing > > close of > > > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. > > > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 > > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing > > > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.: > > disabling compactions & flushes > > > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 > > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates > > disabled for region > > > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. > > > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 > > 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed > > > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. > > > essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 > > 09:48:36,953 DEBUG > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed > > region > > > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. > > ==== > > > > It looks like the second had already been closed for some reason. Is it > > possile that the info of that region > > is not clean clearly? How can I check this in detail? > > > > I'm guessing maybe 'getClosestRowBefore' returns the closed region in > > HConnectionManager.locateRegionInMeta, > > which is closed, so this leads to the problem. > > > > 2011/4/12 茅旭峰 <[email protected]> > > > >> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the > >> issue further, but my understanding is the meta info and the root > >> region are saved by zookeeper, right? Do I need to check them there? > >> > >> m9suns > >> > >> 在 2011-4-12，0:40，Jean-Daniel Cryans <[email protected]> 写道： > >> > >> > It's possible under some bugs, which HBase version are you using? > >> > > >> > J-D > >> > > >> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <[email protected]> wrote: > >> >> Hi, > >> >> > >> >> Is it possible that some table cannot cover the whole key space. What > I > >> saw > >> >> was like > >> >> > >> >> ==== > >> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123' > >> >> > >> >> 0 row(s) in 0.3030 seconds > >> >> > >> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=', > >> >> 'cfEStore:dasd', '123' > >> >> > >> >> ERROR: java.io.IOException: HRegionInfo was null or empty in .META., > >> >> > >> > row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14, > >> >> > >> > table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8} > >> >> > >> >> Here is some help for this command: > >> >> Put a cell 'value' at specified table/row/column and optionally > >> >> timestamp coordinates. To put a cell value into table 't1' at > >> >> row 'r1' under column 'c1' marked with the time 'ts1', do: > >> >> > >> >> hbase> put 't1', 'r1', 'c1', 'value', ts1 > >> >> > >> >> ==== > >> >> > >> >> I guess this means our table has lost some key range for the specific > >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=', > >> >> is this reasonable? I wonder what might cause this issue? How can I > >> check > >> >> it? > >> >> > >> >> One weird thing, I can see such lines like > >> >> > >> >> ==== > >> >> > >> > table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625. > >> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A= > >> >> ==== > >> >> > >> >> from the web UI, which I think means the table has holds the key > range > >> for > >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=' > >> >> > >> >> I also have not seen any errors/warnings in the log file. Any > >> suggestions? > >> >> > >> >> Thanks in advance. > >> >> > >> > > >
