Thanks Mikhail. That's very helpful information.
We followed some of the advice that I believe was posted to this list in the
past to schedule major compactions manually to run during off-peak times. That
has gone very well for us, except when failures happen it seems we end up not
major compacting some regions. We currently have hbase.hregion.majorcompaction
set to 0 to disable automated major compactions. I'm wondering if I manually
run major compactions every 'x' days if I set hbase.hregion.majorcompaction to
'x+m' where m is some number of days after the number of scheduled compactions
then ideally the automated major compaction would just catch any regions that
didn't get compacted normally. For example, if I want to major compact every
Saturday then I could set hbase.hregion.majorcompaction to 9 days (777600000
milliseconds ) so if for some reason the weekly Saturday major compaction
didn't succeed for a subset of the regions it would be caught by the automated
compaction 2 days later. I would assume that if the manually scheduled major
compactions completed successfully the automated major compactions wouldn't
kick in because not enough time had passed.
Curious if you think that is a reasonably way to manually schedule major
compactions but also have a catch-all in case of regionserver failures.
Thanks again for your input. Very helpful.
-----Original Message-----
From: Mikhail Bautin [mailto:[email protected]]
Sent: Sunday, December 11, 2011 3:10 AM
To: [email protected]
Subject: Re: major compaction and regionserver failure...
Hi Andy,
We don't persist manual compaction requests across server restarts, so if e.g.
a major compaction was requested manually and the regionserver crashed, the new
regionserver that picks up that region will not know about the major compaction
request. For automatic compactions, however, compaction criteria are the same
on the old and the new region server, so we would expect the new server to
figure out that a compaction is necessary and start it again from the beginning
if the old regionserver died in the middle of a compaction.
Thanks,
--Mikhail
On Sat, Dec 10, 2011 at 2:25 PM, Andy Sautins
<[email protected]>wrote:
>
> Mikhail,
>
> Thanks for the response. So to summarize compaction requests in the
> compaction queue of the failed regionserver are lost and not picked up
> when the regions are picked up by another regionserver. So if we lose
> a regionserver during a major compaction the regions that had not yet
> been major compacted on the failed regionserver will not be major
> compacted as part of that major compaction request, correct?
>
> Thanks for the insight.
>
> Andy
>
> -----Original Message-----
> From: Mikhail Bautin [mailto:[email protected]]
> Sent: Saturday, December 10, 2011 1:50 PM
> To: [email protected]
> Subject: Re: major compaction and regionserver failure...
>
> Hi Andy,
>
> Compaction queues are not persisted between regionserver restarts, and
> the results of an incomplete compaction are discarded. Compactions
> write into an HFile in a temporary location and only move it to the
> region/CF directory in case of success (at least, this is how it works in
> trunk).
>
> Thanks,
> --Mikhail
>
> On Saturday, December 10, 2011, Andy Sautins wrote:
>
> >
> > I was wondering if someone could tell me what happens if a
> > regionserver fails during major compaction. Are the entries that
> > were in the compaction queue for the failed regionserver migrated
> > along with the regions to another server or are those requests for
> > major compaction effectively lost for that run of major compaction?
> > I guess what I'm really asking is do I still get a full major
> > compaction of all regions or do I lose the major compactions of the
> > regions that were in the queue on the failed regionserver.
> >
> > Any insight would be much appreciated. We are currently running HBase
> > 0.90.3-cdh3u1.
> >
> >
> >
>
