We are using hadoop 0.20.2 + hbase 0.90.3.
The network switch of our cluster was down for some time. After restore,
several data nodes and RSes (about half) were down. After restart them, the
cluster was online.
However, our writers to hbase always reported following info:
2012-03-15 22:50:17,313 WARN
[org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation]
- Encountered problems when prefetch META table:
java.io.IOException: HRegionInfo was null or empty in Meta for
CLOUD_201203,
row=CLOUD_201203,14522035490150643060323210-9883,99999999999999
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:142)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:648)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:702)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:593)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1117)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1237)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:665)
at com.huawei.cloud.output.HBaseWriter.put(HBaseWriter.java:59)
at com.huawei.cloud.output.HBaseWriter.run(HBaseWriter.java:102)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
hbck reports lots of missing of region infos for CLOUD_201203 in .META.
table.
So, I disabled and dropped table CLOUD_201203 in hbase shell. Then,
re-create the table using hbase shell.
Initially, it looked normal with only one exception: the web page for this
table showed NO region. But, hbck reported OK. So I started the writer
programs to write data to this table. After importing about 500MB data,
client again got above errors.
So, it seems the table re-creation still did NOT fix the problem.
What shall I do then?
Thanks
Weihua
