Thanks for the feedback Jay. I helped someone who faced the same issue recently. Might deserve a fix... (and so a JIRA with details)
Also, I would recommend you to migrate to a sooner 0.94.x version. JM 2013/5/22 Jay Talreja <[email protected]> > *0.94.0* > > The issue (I think) was related to a region split that didn't happen > cleanly. As a result there were references to daughter region present in > HDFS. > I believe the -fixSplitParents would have taken care of this but it is not > available in 0.94.0 > > I manually deleted reference files from HDFS and was able to bring the > table back to consistent state. > > Thanks, > Jay > > > On 5/21/13 11:12 PM, Jean-Marc Spaggiari wrote: > >> Hi Jay, >> >> Which 0.94 version are you running? 0.94.0? Or 0.94.7? >> >> >> JM >> >> 2013/5/21 Jay Talreja<[email protected]**> >> >> One of our production clusters had several region server failures. As a >>> result one of the tables is in an inconsistent state as reported by hbck. >>> We have tried using hbck repair commands but none seem to work. There is >>> one region that is stuck in a forever pending open state. >>> The error reported in RS log is about a StoreFile not found. But what is >>> really strange is that the store file that is reported as missing does >>> not >>> even belong to the region being opened. >>> >>> We tried to manually create a directory in HDFS and copy the missing file >>> but it causes hbck to report about a region in HDFS but not in Meta. >>> >>> There 4 inconsistencies currently. >>> >>> ERROR: Region { meta => <tableName>,I.1521_D.**** >>> 1361689200_9,1369099149747. >>> ****2123fc70fac804cd8d48ea4494cc81****84., hdfs => >>> hdfs://host:8020/hbase/** >>> tableName/****2123fc70fac804cd8d48ea4494cc81****84, deployed => } not >>> >>> deployed on any region server. >>> ERROR: Region { meta => null, hdfs => hdfs://hostname:8020/hbase/** >>> tableName/****450ed30b410e9d6d54ac53099039cb****28, deployed => } on >>> HDFS, >>> >>> but not listed in META or deployed on any region server >>> 13/05/21 10:51:11 DEBUG util.HBaseFsck: There are 1769 region info >>> entries >>> ERROR: There is a hole in the region chain between I.1521_D.1361689200_9 >>> and I.1521_D.1362150000_8. You need to create a new .regioninfo and >>> region >>> dir in hdfs to plug the hole. >>> ERROR: There is a hole in the region chain between I.1_D.1368392400_9 and >>> I.2020_D.1338948000_2. You need to create a new .regioninfo and region >>> dir >>> in hdfs to plug the hole. >>> ERROR: Found inconsistency in table<tableName> >>> >>> We are running Hbase 0.94 (Apache) on Hadoop 1.0.3 >>> >>> At this stage, we are stuck and are looking for help ! The cluster is in >>> an unbalanced state and region servers frequently keep dying. >>> >>> Thanks, >>> Jay >>> >>> >>> >
