My guess was the second region is closed, but its directory named of the encoded name is still on the hdfs, this leads to the inconsistency.
I wonder what happened when the region is closed, it looks like the process is over, from the log. 2011/4/12 茅旭峰 <[email protected]> > Anyway, looks like the two regions below have some kind of overlap, does > this make sense? > > > 2011/4/12 茅旭峰 <[email protected]> > >> From the output of scan '.META.' I pasted before, we can see there are two >> key ranges >> which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They are >> >> #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A=' >> #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key' >> >> The output has less info about the second key range. And I also found some >> log like >> >> ==== >> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> 09:48:36,953 DEBUG >> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing >> close of >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.: >> disabling compactions & flushes >> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates >> disabled for region >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> 09:48:36,953 DEBUG >> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed >> region >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >> ==== >> >> It looks like the second had already been closed for some reason. Is it >> possile that the info of that region >> is not clean clearly? How can I check this in detail? >> >> I'm guessing maybe 'getClosestRowBefore' returns the closed region in >> HConnectionManager.locateRegionInMeta, >> which is closed, so this leads to the problem. >> >> >> 2011/4/12 茅旭峰 <[email protected]> >> >>> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the >>> issue further, but my understanding is the meta info and the root >>> region are saved by zookeeper, right? Do I need to check them there? >>> >>> m9suns >>> >>> 在 2011-4-12，0:40，Jean-Daniel Cryans <[email protected]> 写道： >>> >>> > It's possible under some bugs, which HBase version are you using? >>> > >>> > J-D >>> > >>> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <[email protected]> wrote: >>> >> Hi, >>> >> >>> >> Is it possible that some table cannot cover the whole key space. What >>> I saw >>> >> was like >>> >> >>> >> ==== >>> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123' >>> >> >>> >> 0 row(s) in 0.3030 seconds >>> >> >>> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=', >>> >> 'cfEStore:dasd', '123' >>> >> >>> >> ERROR: java.io.IOException: HRegionInfo was null or empty in .META., >>> >> >>> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14, >>> >> >>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8} >>> >> >>> >> Here is some help for this command: >>> >> Put a cell 'value' at specified table/row/column and optionally >>> >> timestamp coordinates. To put a cell value into table 't1' at >>> >> row 'r1' under column 'c1' marked with the time 'ts1', do: >>> >> >>> >> hbase> put 't1', 'r1', 'c1', 'value', ts1 >>> >> >>> >> ==== >>> >> >>> >> I guess this means our table has lost some key range for the specific >>> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=', >>> >> is this reasonable? I wonder what might cause this issue? How can I >>> check >>> >> it? >>> >> >>> >> One weird thing, I can see such lines like >>> >> >>> >> ==== >>> >> >>> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625. >>> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A= >>> >> ==== >>> >> >>> >> from the web UI, which I think means the table has holds the key range >>> for >>> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=' >>> >> >>> >> I also have not seen any errors/warnings in the log file. Any >>> suggestions? >>> >> >>> >> Thanks in advance. >>> >> >>> >> >> >
