Hey Andrea, That is very weird, I hit this similar extremely strange one off issue that I could never reproduce. After dropping a table doing X amount of inserts, I would receive the same error and my META table would be corrupted. What I did to correct this issue was to sideline the META table and then use offlineMetaRepair to rebuild it, and then I never ran into it again. It was quite strange and I am sorry I don't have more to offer.
On Tue, Dec 10, 2013 at 8:34 AM, Jean-Marc Spaggiari < [email protected]> wrote: > Hi Andrea, > > Which version of HBase are you testing with? > > JM > > > 2013/12/10 Andrea <[email protected]> > > > Hi, I encounter this problem: > > > > 13/12/10 13:25:15 WARN > client.HConnectionManager$HConnectionImplementation: > > Encountered problems when prefetch META table: > > java.io.IOException: HRegionInfo was null or empty in Meta for usertable, > > row=usertable,user4300292263938371081,99999999999999 > > at > > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:170) > > at > > > org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:54) > > at > > > org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133) > > at > > > org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130) > > at > > > > > org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager > > .java:360) > > at > > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130) > > at > > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105) > > at > > > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. > > prefetchRegionCache(HConnectionManager.java:918) > > at > > > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. > > locateRegionInMeta(HConnectionManager.java:973) > > at > > > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. > > locateRegion(HConnectionManager.java:860) > > at > > > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. > > locateRegion(HConnectionManager.java:817) > > at > > > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. > > processBatchCallback(HConnectionManager.java:1507) > > at > > > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. > > processBatch(HConnectionManager.java:1392) > > at > > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:918) > > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:774) > > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:749) > > at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:328) > > at com.yahoo.ycsb.db.HBaseClient.insert(HBaseClient.java:357) > > at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148) > > at > > com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461) > > at com.yahoo.ycsb.ClientThread.run(Client.java:269) > > > > It will happen when, on a netowrk with one zookeeper, one master and one > > regionserver I try to load a lot of data. This is the number of > operations > > before the error: > > > > 200 sec: 888708 operations; 7767.62 current ops/sec; [UPDATE > > AverageLatency(us)=27089963] [INSERT AverageLatency(us)=9738.99] [CLEANUP > > AverageLatency(us)=27089976] > > 210 sec: 893742 operations; 502.7 current ops/sec; [UPDATE > > AverageLatency(us)=14887298.5] [INSERT AverageLatency(us)=6937.27] > [CLEANUP > > AverageLatency(us)=14887312.5] > > 221 sec: 928277 operations; 3226.97 current ops/sec; [UPDATE > > AverageLatency(us)=27006074] [INSERT AverageLatency(us)=8529.21] [CLEANUP > > AverageLatency(us)=27006089] > > 231 sec: 930079 operations; 180.18 current ops/sec; [INSERT > > AverageLatency(us)=457.08] > > 13/12/10 13:25:13 WARN > client.HConnectionManager$HConnectionImplementation: > > Encountered problems when prefetch META table: ... > > > > Does this depend on the number of thread in whereby I insert the data? > > > > > -- Kevin O'Dell Systems Engineer, Cloudera
