In addition to "holes" in .META. we often see this problem too. Now that we've turned off splitting, we no longer have seen either duplicate rows nor holes....I suppose that gives a clue as to the culprit: splitting...as far as my experience can show me.
-geoff -----Original Message----- From: Jonathan Hsieh [mailto:[email protected]] Sent: Friday, September 09, 2011 1:03 PM To: [email protected]; Douglas Campbell Subject: Re: 2 rows in .META. with same startkey in hbase 90.3 I've recently fixed this situation manually, and can give a high level description of the steps. There are a handful of scripts that can help, and I'm working on improving some of them. First run hbck to get some diagnostics about overlapping regions. You should see something like this: ERROR: (regions table,A,<ts1>.<dirname1>. and table,A,<ts>.<dirname2>.) Two regions have the same startkey: A ERROR: (regions table,A,<ts1>.<dirname1>. and table,A,<ts2>.<dirname2>.) There is an overlap in the region chain. ERROR: (regions table,A,<ts1>.<dirname1>. and table,B,<ts3>.<dirname3>.) There is an overlap in the region chain. Check out the data in /hbase/table/<dirname1>. The data in there are HFiles. These can be merged by copying them all into the same dir. Scan meta for entries in '.META.' -- and look at the STARTKEY and ENDKEY parts of the regions. Figure out if regions "lineup" and are fully contained. Ex: if I see region 1 is A-C, region2 is A-B, and region3 is B-C), region2 and region3 is contained, and line up because the start and endkeys are the same boundaries as region1. The goal is now to copy the data from region2 and region3 into region1. In the shell close_region the regions question Copy the data from the smaller regions into the larger region excluding the .regioninfo file. Move the smaller regions data out of the /hbase/table/ dir to some where else. In the shell, deleteall the smaller region's entry in meta. In the shell, assign the bigger, remaining region. Run HBCK to verify. Jon. On Thu, Sep 8, 2011 at 11:41 AM, Douglas Campbell <[email protected]>wrote: > How can we safely remove one or both of the rows? How to fix. > > > This is causing our table scans and gets within the corresponding regions > to hang and or timeout. > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // [email protected]
