Thank you J-D for the reply. Some clarification: 1. Actually we use hadoop 0.20.2+append and the cluster has worked for about half year. 2. It is hard to do so. There were about 5000 regions in the cluster. And it seems .META. was problematic at scan. In shell, I specified STARTROW as CLOUD_201203 and LIMIT as 1. However, it printed the row as CLOUD_201201xxxxx. 3. I will watch the log when I am in office.
Thanks Weihua 2012/3/16 Jean-Daniel Cryans <[email protected]> > First, upgrade to a hadoop version that supports append. > > When you disable/drop the table, do you still see CLOUD_201203 entries > in .META. (using the shell)? It seem there's something odd in there. > > Also look for odd exceptions in the master log when you create the > table, and make sure it assigns the newly created region. > > J-D > > On Thu, Mar 15, 2012 at 8:21 AM, Weihua JIANG <[email protected]> > wrote: > > We are using hadoop 0.20.2 + hbase 0.90.3. > > The network switch of our cluster was down for some time. After restore, > > several data nodes and RSes (about half) were down. After restart them, > the > > cluster was online. > > However, our writers to hbase always reported following info: > > 2012-03-15 22:50:17,313 WARN > > > [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] > > - Encountered problems when prefetch META table: > > java.io.IOException: HRegionInfo was null or empty in Meta for > > CLOUD_201203, > > row=CLOUD_201203,14522035490150643060323210-9883,99999999999999 > > at > > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:142) > > at > > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:648) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:702) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:593) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1117) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1237) > > at > > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819) > > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675) > > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:665) > > at com.huawei.cloud.output.HBaseWriter.put(HBaseWriter.java:59) > > at com.huawei.cloud.output.HBaseWriter.run(HBaseWriter.java:102) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:619) > > > > hbck reports lots of missing of region infos for CLOUD_201203 in .META. > > table. > > > > So, I disabled and dropped table CLOUD_201203 in hbase shell. Then, > > re-create the table using hbase shell. > > > > Initially, it looked normal with only one exception: the web page for > this > > table showed NO region. But, hbck reported OK. So I started the writer > > programs to write data to this table. After importing about 500MB data, > > client again got above errors. > > > > So, it seems the table re-creation still did NOT fix the problem. > > > > What shall I do then? > > > > Thanks > > Weihua >
