The hbck output lists several empty REGIONINFO_QUALIFIER rows in .META. I was not able to insert keys around these regions. I removed those records in .META. Insertion still does not work. Then we have some holes in .META. I tried bin/hbase org.jruby.Main bin/check_meta.rb but it complained == 11/04/13 22:57:50 INFO zookeeper.ClientCnxn: Session establishment complete on server cloud135/10.241.67.22:2181, sessionid = 0x12edcaa37f9005f, negotiated timeout = 40000 11/04/13 22:57:50 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@50b2fb1e; hsa=cloud140:60020 11/04/13 22:57:50 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is cloud136:60020 11/04/13 22:57:50 DEBUG client.HTable$ClientScanner: Creating scanner over .META. starting at key '' 11/04/13 22:57:50 DEBUG client.HTable$ClientScanner: Advancing internal scanner to startKey at '' 11/04/13 22:57:50 DEBUG client.HConnectionManager$HConnectionImplementation: Cache hit for row <> in tableName .META.: location server cloud136:60020, location region name .META.,,1.1028785192 Writables.java:75:in `org.apache.hadoop.hbase.util.Writables.getWritable': java.lang.NullPointerException: null (NativeException) from Writables.java:119:in `org.apache.hadoop.hbase.util.Writables.getHRegionInfo' from null:-1:in `sun.reflect.GeneratedMethodAccessor6.invoke' from DelegatingMethodAccessorImpl.java:25:in `sun.reflect.DelegatingMethodAccessorImpl.invoke' from Method.java:597:in `java.lang.reflect.Method.invoke' from JavaMethod.java:196:in `org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling' from JavaMethod.java:182:in `org.jruby.javasupport.JavaMethod.invoke_static' from JavaClass.java:371:in `org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute' from SimpleCallbackMethod.java:81:in `org.jruby.internal.runtime.methods.SimpleCallbackMethod.call' ... 16 levels... from Main.java:183:in `org.jruby.Main.runInterpreter' from Main.java:120:in `org.jruby.Main.run' from Main.java:95:in `org.jruby.Main.main' Complete Java stackTrace java.lang.NullPointerException at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75) at org.apache.hadoop.hbase.util.Writables.getHRegionInfo(Writables.java:119) at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:196) at org.jruby.javasupport.JavaMethod.invoke_static(JavaMethod.java:182) at org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute(JavaClass.java:371) at org.jruby.internal.runtime.methods.SimpleCallbackMethod.call(SimpleCallbackMethod.java:81) at org.jruby.evaluator.EvaluationState.callNode(EvaluationState.java:571) at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:207) at org.jruby.evaluator.EvaluationState.localAsgnNode(EvaluationState.java:1254) at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:286) at org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533) at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:201) at org.jruby.evaluator.EvaluationState.whileNode(EvaluationState.java:1793) at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:387) at org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533) at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:201) at org.jruby.evaluator.EvaluationState.rootNode(EvaluationState.java:1628) at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:356) at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:164) at org.jruby.Ruby.eval(Ruby.java:278) at org.jruby.Ruby.compileOrFallbackAndRun(Ruby.java:306) at org.jruby.Main.runInterpreter(Main.java:238) at org.jruby.Main.runInterpreter(Main.java:183) at org.jruby.Main.run(Main.java:120) at org.jruby.Main.main(Main.java:95) ==
>From a previous thread, I saw some one insert some records into .META. manually, to fill the holes, it seems like we need to generate encoded region name with tablename, start_key, id and md5-hash, I do not know how to generate id and start_key. Any tips on fill those region holes? Thanks and regards, Mao Xu-Feng 2011/4/13 茅旭峰 <[email protected]> > Hi Stack, > > Sorry for bothering you in private. > > Regarding this issue, so far I still can not put specific key into the > hbase? > > I attached the output of './bin/hbase shell' with this message. > > No matter what causes this issue, is there any operation I can perform to > cover this scenario, at least we need chance to put those key-values. > > And then, I'm trying to find the root cause by debugging. > > Will keep you update. Thanks! > > > 2011/4/13 Stack <[email protected]> > >> You can also cat the .regioninfo file that is under the region >> directory to learn more about the region -- its HRegionInfo (The file >> format is serialized HRegionInfo followed by a String version so >> intelligible to non-machines). >> St.Ack >> >> 2011/4/12 Stack <[email protected]>: >> > These regions that are in hdfs but not in .META. and not on any region >> > server are probably harmless. Would be interesting to trace how they >> > got to this state. My guess is that they are old let go regions that >> > were not cleaned up. >> > >> > The region directory name is its encoded name which is the tail part >> > of the row name in the .META. table. >> > >> > St.Ack >> > >> > >> > 2011/4/11 茅旭峰 <[email protected]>: >> >> the results of ./bin/hbase hbck show lots of 'inconsistence status' >> like >> >> >> >> ==== >> >> ERROR: Region >> >> hdfs://cloud137:9000/hbase/table1/01c80f8b54523ad6c242c5f695544f16 on >> HDFS, >> >> but not listed in META or deployed on any region server. >> >> ERROR: Region >> >> hdfs://cloud137:9000/hbase/table1/01ce4e2f72baa0df51b7b2010c9d1ef0 on >> HDFS, >> >> but not listed in META or deployed on any region server. >> >> ERROR: Region >> >> hdfs://cloud137:9000/hbase/table1/01d0855af5e646823bdff82fddadd81e on >> HDFS, >> >> but not listed in META or deployed on any region server. >> >> ==== >> >> >> >> How can I match the names of the directories from the .META.? >> >> >> >> I checked the help in hbase shell, but looks like it only has >> close_region, >> >> no open_region. >> >> >> >> 2011/4/12 Stack <[email protected]> >> >>> >> >>> Can you open the region again? (See shell commands for opening >> regions). >> >>> >> >>> What does hbck say: ./bin/hbase hbck. >> >>> >> >>> Add the -details flag. >> >>> >> >>> It might tell you a story about an offlined region. >> >>> >> >>> 0.90.2 has some fixes for issues in and around here (CDH3 release, out >> >>> on the 14th, has most of them bundled). >> >>> >> >>> St.Ack >> >>> >> >>> 2011/4/11 茅旭峰 <[email protected]>: >> >>> > From the output of scan '.META.' I pasted before, we can see there >> are >> >>> > two >> >>> > key ranges >> >>> > which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They >> are >> >>> > >> >>> > #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A=' >> >>> > #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key' >> >>> > >> >>> > The output has less info about the second key range. And I also >> found >> >>> > some >> >>> > log like >> >>> > >> >>> > ==== >> >>> > >> >>> > >> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> >>> > 09:48:36,953 DEBUG >> >>> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: >> >>> > Processing >> >>> > close of >> >>> > >> >>> > >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >> >>> > >> >>> > >> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> >>> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: >> Closing >> >>> > >> >>> > >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.: >> >>> > disabling compactions & flushes >> >>> > >> >>> > >> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> >>> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: >> Updates >> >>> > disabled for region >> >>> > >> >>> > >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >> >>> > >> >>> > >> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> >>> > 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: >> Closed >> >>> > >> >>> > >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >> >>> > >> >>> > >> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21 >> >>> > 09:48:36,953 DEBUG >> >>> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: >> Closed >> >>> > region >> >>> > >> >>> > >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd. >> >>> > ==== >> >>> > >> >>> > It looks like the second had already been closed for some reason. Is >> it >> >>> > possile that the info of that region >> >>> > is not clean clearly? How can I check this in detail? >> >>> > >> >>> > I'm guessing maybe 'getClosestRowBefore' returns the closed region >> in >> >>> > HConnectionManager.locateRegionInMeta, >> >>> > which is closed, so this leads to the problem. >> >>> > >> >>> > 2011/4/12 茅旭峰 <[email protected]> >> >>> > >> >>> >> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the >> >>> >> issue further, but my understanding is the meta info and the root >> >>> >> region are saved by zookeeper, right? Do I need to check them >> there? >> >>> >> >> >>> >> m9suns >> >>> >> >> >>> >> 在 2011-4-12，0:40，Jean-Daniel Cryans <[email protected]> 写道： >> >>> >> >> >>> >> > It's possible under some bugs, which HBase version are you using? >> >>> >> > >> >>> >> > J-D >> >>> >> > >> >>> >> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <[email protected]> wrote: >> >>> >> >> Hi, >> >>> >> >> >> >>> >> >> Is it possible that some table cannot cover the whole key space. >> >>> >> >> What I >> >>> >> saw >> >>> >> >> was like >> >>> >> >> >> >>> >> >> ==== >> >>> >> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123' >> >>> >> >> >> >>> >> >> 0 row(s) in 0.3030 seconds >> >>> >> >> >> >>> >> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=', >> >>> >> >> 'cfEStore:dasd', '123' >> >>> >> >> >> >>> >> >> ERROR: java.io.IOException: HRegionInfo was null or empty in >> .META., >> >>> >> >> >> >>> >> >> >>> >> >> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14, >> >>> >> >> >> >>> >> >> >>> >> >> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8} >> >>> >> >> >> >>> >> >> Here is some help for this command: >> >>> >> >> Put a cell 'value' at specified table/row/column and optionally >> >>> >> >> timestamp coordinates. To put a cell value into table 't1' at >> >>> >> >> row 'r1' under column 'c1' marked with the time 'ts1', do: >> >>> >> >> >> >>> >> >> hbase> put 't1', 'r1', 'c1', 'value', ts1 >> >>> >> >> >> >>> >> >> ==== >> >>> >> >> >> >>> >> >> I guess this means our table has lost some key range for the >> >>> >> >> specific >> >>> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=', >> >>> >> >> is this reasonable? I wonder what might cause this issue? How >> can I >> >>> >> check >> >>> >> >> it? >> >>> >> >> >> >>> >> >> One weird thing, I can see such lines like >> >>> >> >> >> >>> >> >> ==== >> >>> >> >> >> >>> >> >> >>> >> >> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625. >> >>> >> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A= >> >>> >> >> ==== >> >>> >> >> >> >>> >> >> from the web UI, which I think means the table has holds the key >> >>> >> >> range >> >>> >> for >> >>> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=' >> >>> >> >> >> >>> >> >> I also have not seen any errors/warnings in the log file. Any >> >>> >> suggestions? >> >>> >> >> >> >>> >> >> Thanks in advance. >> >>> >> >> >> >>> >> >> >>> > >> >> >> >> >> > >> > >
