These days, we are loading a vast bulk of data to HBase. I closed the WAL for the speed issue.And a region server crashed yesterday, then I restarted it. Unfortunately, I found a big problem this morning. The row key between pox2qq1qV8 and q32i6GEYEP cannot write to HBase any more.
I checked the .META. table , found there is a hole in this table. The following lines is copied from table view of HBase web manage ( http://xxx.shendusou.com:60010/table.jsp?name=content) We can find the hole between the second line and the third line. One or more region which cover pox2qq1qV8 to q32i6GEYEP are lost. content,pguACQ4oZE,1276757449799 d1.shendusou.com:60030 <http://dev1.shendusou.com:60030/> 481213793 pguACQ4oZE pku/KN/RcN content,pku/KN/RcN,1276757449799 c3.shendusou.com:60030 <http://content3.shendusou.com:60030/> 543945981 pku/KN/RcN pox2qq1qV8 content,q32i6GEYEP,1276757404613 c1.shendusou.com:60030 <http://content1.shendusou.com:60030/> 1759457900 q32i6GEYEP q75kqgwVsz content,q75kqgwVsz,1276757404613 c1.shendusou.com:60030 <http://content1.shendusou.com:60030/> 2118337301 q75kqgwVsz qB68nrJBY7 Lose some data is not a problem, Recover it is easy for us, however , we run into difficulties that we cannot write any data into the table which key is in the range [pox2qq1qV8 , q32i6GEYEP], It is very bad and blocks us. I found the issue HBase-2526 is working for this problem. https://issues.apache.org/jira/browse/HBASE-2526 , I do not think it would be fixed until 0.21.0. I closed the whole HBase cluster and restarted it many times today, the problem also exsit. Anybody can tell me how to fix the hole? Thanks WZeng
