St.Ack - I will definitely save the meta (before and after) and the master log, next time this problem occurs. At the moment, I am running with splitting disabled hoping the problem doesn't recur. Already, I see that the master logs are "quiet", whereas before they were churning endlessly with split-related info.
You are correct, we are running 90.1 (CDH3U0). I will keep you posted on this issue. Thanks a lot for the help! -geoff -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Stack Sent: Sunday, September 04, 2011 2:46 PM To: [email protected] Subject: Re: prevent region splits? On Sun, Sep 4, 2011 at 12:08 PM, Geoff Hendrey <[email protected]> wrote: > great advice guys. appreciate it. Have made the changes to increase > storefile size. I'd also like to prevent rebalancing while I am running > my large M/R Put job. Any way to do that? > In shell you can disable the balancer. Be warned its an in-memory setting the master only (unless you set config in hbase-site.xml to have balancer run once an eternity). > At present, 50% of the time that I run my large M/R Put job, the table > is corrupted (hole in .META.) and we have to run our repair program to > fix the hole. Mind doing a dump of meta region content pre MR job, and then master log and dump of meta after with identified messed up section of meta? > It's very labor intensive. I am hoping that be turning off > splitting, and deferring balancing, that I can prevent whatever > condition leads to the creation of the hole in .META.. My hope is that > if we prevent splitting and rebalancing then there would be no action > that could cause a whole to occur. > Probably. From previous email, up your timeout monitor (you are not running 0.90.4 I take it). St.Ack
