Hi, Jean What is the jira #?
Thanks Tian-Ying -----Original Message----- From: Jean-Marc Spaggiari [mailto:[email protected]] Sent: Wednesday, May 22, 2013 7:57 AM To: [email protected] Subject: Re: Inconsistent Table HBCK Thanks for the feedback Jay. I helped someone who faced the same issue recently. Might deserve a fix... (and so a JIRA with details) Also, I would recommend you to migrate to a sooner 0.94.x version. JM 2013/5/22 Jay Talreja <[email protected]> > *0.94.0* > > The issue (I think) was related to a region split that didn't happen > cleanly. As a result there were references to daughter region present > in HDFS. > I believe the -fixSplitParents would have taken care of this but it is > not available in 0.94.0 > > I manually deleted reference files from HDFS and was able to bring the > table back to consistent state. > > Thanks, > Jay > > > On 5/21/13 11:12 PM, Jean-Marc Spaggiari wrote: > >> Hi Jay, >> >> Which 0.94 version are you running? 0.94.0? Or 0.94.7? >> >> >> JM >> >> 2013/5/21 Jay Talreja<[email protected]**> >> >> One of our production clusters had several region server failures. >> As a >>> result one of the tables is in an inconsistent state as reported by hbck. >>> We have tried using hbck repair commands but none seem to work. >>> There is one region that is stuck in a forever pending open state. >>> The error reported in RS log is about a StoreFile not found. But >>> what is really strange is that the store file that is reported as >>> missing does not even belong to the region being opened. >>> >>> We tried to manually create a directory in HDFS and copy the missing >>> file but it causes hbck to report about a region in HDFS but not in Meta. >>> >>> There 4 inconsistencies currently. >>> >>> ERROR: Region { meta => <tableName>,I.1521_D.**** >>> 1361689200_9,1369099149747. >>> ****2123fc70fac804cd8d48ea4494cc81****84., hdfs => >>> hdfs://host:8020/hbase/** >>> tableName/****2123fc70fac804cd8d48ea4494cc81****84, deployed => } not >>> >>> deployed on any region server. >>> ERROR: Region { meta => null, hdfs => hdfs://hostname:8020/hbase/** >>> tableName/****450ed30b410e9d6d54ac53099039cb****28, deployed => } on >>> HDFS, >>> >>> but not listed in META or deployed on any region server >>> 13/05/21 10:51:11 DEBUG util.HBaseFsck: There are 1769 region info >>> entries >>> ERROR: There is a hole in the region chain between >>> I.1521_D.1361689200_9 and I.1521_D.1362150000_8. You need to create >>> a new .regioninfo and region dir in hdfs to plug the hole. >>> ERROR: There is a hole in the region chain between >>> I.1_D.1368392400_9 and I.2020_D.1338948000_2. You need to create a >>> new .regioninfo and region dir in hdfs to plug the hole. >>> ERROR: Found inconsistency in table<tableName> >>> >>> We are running Hbase 0.94 (Apache) on Hadoop 1.0.3 >>> >>> At this stage, we are stuck and are looking for help ! The cluster >>> is in an unbalanced state and region servers frequently keep dying. >>> >>> Thanks, >>> Jay >>> >>> >>> >
