Digging in the (0.98) code a bit I find this:
HRegionServer.postOpenDeployTasks(): Request a compaction either when we're
past the minimum of file or there is any reference file. Good, that will
trigger
RatioBasedComactionPolicy.selectCompaction(): turns any compaction into a major
one if there reference files involved in the set of files already selected.
Also cool, after a split all files of a daughter will be reference files.
But... I do not see any code where it would make sure at least one reference
file is selected. So in theory the initial compaction started by
postOpenDeployTasks could have failed for some reason.Now more data is written
and following compaction selections won't pick up any reference files, as there
are many small, new files written.
So the reference files could in theory just linger until a selection just
happens to come across one, all the while the daughters are (a) unsplittable
and (b) cannot migrate to another region server.
That is unless I missed something... Maybe somebody could have a look too?
Seems to me we should sort reference files first _always_, to compact them away
first and allow to be split further. Thoughts? File a jira?
-- Lars
From: lars hofhansl <[email protected]>
To: "[email protected]" <[email protected]>
Sent: Friday, December 5, 2014 1:59 PM
Subject: Re: Region is out of bounds
We've run into something like this as well (probably).
Will be looking at this as well over the next days/weeks. Under heavy load
HBase seems to just not able to get the necessary compactions in, and until
that happens it cannot further split a region.
I wonder whether HBASE-12411 would be help here (this optionally allow
compaction to use private readers), I doubt it, though.
The details are probably tricky, I thought HBase would compact split regions
with higher priority (placing those first in the compaction queue)... Need to
actually check the code.
-- Lars
From: Qiang Tian <[email protected]>
To: "[email protected]" <[email protected]>
Sent: Thursday, December 4, 2014 7:26 PM
Subject: Re: Region is out of bounds
----My attempt to add reference files forcefully to compaction list in
Store.requetsCompaction() when region exceeds recommended maximum size did
not work out well - some weird results in our test cases (but HBase tests
are OK: small, medium and large).
interesting...perhaps it was filtered out in RatioBasedCompactionPolicy#
selectCompaction?
On Fri, Dec 5, 2014 at 5:20 AM, Andrew Purtell <[email protected]> wrote:
> Most versions of 0.98 since 0.98.1, but I haven't run a punishing high
> scale bulk ingest for its own sake, high-ish rate ingest and a setting of
> blockingStoreFiles to 200 have been in service of getting data in for
> subsequent testing.
>
>
> On Thu, Dec 4, 2014 at 12:43 PM, Vladimir Rodionov <[email protected]
> >
> wrote:
>
> > Andrew,
> >
> > What HBase version have you run your test on?
> >
> > This issue probably does not exist anymore in a latest Apache releases,
> but
> > still exists in not so latest, but still actively used, versions of CDH,
> > HDP etc. We have discovered it during large data set loading ( 100s of
> GB)
> > in our cluster (4 nodes).
> >
> > -Vladimir
> >
> > On Thu, Dec 4, 2014 at 10:23 AM, Andrew Purtell <[email protected]>
> > wrote:
> >
> > > Actually I have set hbase.hstore.
> >
> > blockingStoreFiles to 200 in testing
> > > exactly :-), but must not have generated sufficient load to encounter
> the
> > > issue you are seeing. Maybe it would be possible to adapt one of the
> > ingest
> > > integration tests to trigger this problem? Set blockingStoreFiles to
> 200
> > or
> > > more. Tune down the region size to 128K or similar. If
> > > it's reproducible like that please open a JIRA.
> > >
> > > On Wed, Dec 3, 2014 at 9:07 AM, Vladimir Rodionov <
> > [email protected]>
> > > wrote:
> > >
> > > > Kevin,
> > > >
> > > > Thank you for your response. This is not a question on how to
> configure
> > > > correctly HBase cluster for write heavy workloads. This is internal
> > HBase
> > > > issue - something is wrong in a default logic of compaction selection
> > > > algorithm in 0.94-0.98. It seems that nobody has ever tested
> importing
> > > data
> > > > with very high hbase.hstore.blockingStoreFiles value (200 in our
> case).
> > > >
> > > > -Vladimir Rodionov
> > > >
> > > > On Wed, Dec 3, 2014 at 6:38 AM, Kevin O'dell <
> [email protected]
> > >
> > > > wrote:
> > > >
> > > > > Vladimir,
> > > > >
> > > > > I know you said, "do not ask me why", but I am going to have to
> ask
> > > you
> > > > > why. The fact you are doing this(this being blocking store files >
> > > 200)
> > > > > tells me there is something or multiple somethings wrong with your
> > > > cluster
> > > > > setup. A couple things come to mind:
> > > > >
> > > > > * During this heavy write period, could we use bulk loads? If so,
> > this
> > > > > should solve almost all of your problems
> > > > >
> > > > > * 1GB region size is WAY too small, and if you are pushing the
> volume
> > > of
> > > > > data you are talking about I would recommend 10 - 20GB region sizes
> > > this
> > > > > should help keep your region count smaller as well which will
> result
> > in
> > > > > more optimal writes
> > > > >
> > > > > * Your cluster may be undersized, if you are setting the blocking
> to
> > be
> > > > > that high, you may be pushing too much data for your cluster
> overall.
> > > > >
> > > > > Would you be so kind as to pass me a few pieces of information?
> > > > >
> > > > > 1.) Cluster size
> > > > > 2.) Average region count per RS
> > > > > 3.) Heap size, Memstore global settings, and block cache settings
> > > > > 4.) a RS log to pastebin and a time frame of "high writes"
> > > > >
> > > > > I can probably make some solid suggestions for you based on the
> above
> > > > data.
> > > > >
> > > > > On Wed, Dec 3, 2014 at 1:04 AM, Vladimir Rodionov <
> > > > [email protected]>
> > > > > wrote:
> > > > >
> > > > > > This is what we observed in our environment(s)
> > > > > >
> > > > > > The issue exists in CDH4.5, 5.1, HDP2.1, Mapr4
> > > > > >
> > > > > > If some one sets # of blocking stores way above default value,
> say
> > -
> > > > 200
> > > > > to
> > > > > > avoid write stalls during intensive data loading (do not ask me ,
> > why
> > > > we
> > > > > do
> > > > > > this), then
> > > > > > one of the regions grows indefinitely and takes more 99% of
> overall
> > > > > table.
> > > > > >
> > > > > > It can't be split because it still has orphaned reference files.
> > Some
> > > > of
> > > > > a
> > > > > > reference files are able to avoid compactions for a long time,
> > > > obviously.
> > > > > >
> > > > > > The split policy is IncreasingToUpperBound, max region size is
> 1G.
> > I
> > > do
> > > > > my
> > > > > > tests on CDH4.5 mostly but all other distros seem have the same
> > > issue.
> > > > > >
> > > > > > My attempt to add reference files forcefully to compaction list
> in
> > > > > > Store.requetsCompaction() when region exceeds recommended maximum
> > > size
> > > > > did
> > > > > > not work out well - some weird results in our test cases (but
> HBase
> > > > tests
> > > > > > are OK: small, medium and large).
> > > > > >
> > > > > > What is so special with these reference files? Any ideas, what
> can
> > be
> > > > > done
> > > > > > here to fix the issue?
> > > > > >
> > > > > > -Vladimir Rodionov
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Kevin O'Dell
> > > > > Systems Engineer, Cloudera
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
