[
https://issues.apache.org/jira/browse/HBASE-12657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238546#comment-14238546
]
Vladimir Rodionov commented on HBASE-12657:
-------------------------------------------
Region can't be split because, despite of its size, it still has reference
files and this prevents region from splitting. The question : what does prevent
reference files from being compacted?
Lowest sequence ID among all store files in a region is the reason that
reference files are constantly getting removed from compaction selections if
there are newer files in a compaction queue. This is what is happening under
high load when there are too many minor compaction requests in a queue,
reference files do not have a chance to be compacted. Interestingly, that
current 0.94 and 0.98 code have different issues here and require different
patches.
0.94 does not treat compaction request with reference files as major one, but
ignores *hbase.hstore.compaction.max* for major compactions, 0.98 consider
compaction of reference files as a major one, but consults
*hbase.hstore.compaction.max* and downgrades request when # of files exceeds
this limit.
Low (default) value for *hbase.hstore.compaction.max* and prolonged write
activity w/o throttling will get you into this region is out of bounds
situation. Because, when compaction can't keep up with writes, number of store
files eventually exceeds *hbase.hstore.compaction.max*.
Here what is happening in 0.94:
*0.94 - Does not enforce major compaction flag on selections with reference
files, applies selection algorithm and applies the limit by removing first K =
N - max files from candidate list. N - # of files in selection list, max - is
*hbase.hstore.compaction.max* value.
*0.98 - It marks selection as major but after that applies limit and removes
first K = N - max files from a candidate list.
In both cases, if # of references files > max, some reference files will be
excluded.
What happens later, depends on what we have in a compaction file list for this
Store (HStore). If pending files list has at least one non-reference file, all
reference files will be excluded from above selection. What I have to say here,
it seems that the only way to compact all reference files is to enforce major
compaction immediately after region split. If we fail to do this, with a very
high probability, reference files will be pushing out of compaction until the
write load decrease substantially and # of store files becomes less than
*hbase.hstore.compaction.max*.
> The Region is not being split and far exceeds the desired maximum size.
> -----------------------------------------------------------------------
>
> Key: HBASE-12657
> URL: https://issues.apache.org/jira/browse/HBASE-12657
> Project: HBase
> Issue Type: Bug
> Components: Compaction
> Affects Versions: 0.98.8, 0.94.25, 0.99.2
> Reporter: Vladimir Rodionov
> Assignee: Vladimir Rodionov
> Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.9
>
>
> We are seeing this behavior when creating indexes in one of our environment.
> When an index is being created, most of the "requests" go into a single
> region. The amount of time to create an index seems to take longer than
> usual and it can take days for the regions to compact and split after the
> index is created.
> Here is a du of the HBase index table:
> {code}
> -bash-4.1$ sudo -su hdfs hadoop fs -du /hbase/43681
> 705 /hbase/43681/.tableinfo.0000000001
> 0 /hbase/43681/.tmp
> 27981697293 /hbase/43681/0492e22092e21d35fca8e779b21ec797
> 539687093 /hbase/43681/832298c4e975fc47210feb6bac3d2f71
> 560660531 /hbase/43681/be9bdb3bdf9365afe5fe90db4247d82c
> 7081938297 /hbase/43681/cd440e524f96fbe0719b2fe969848560
> 6297860287 /hbase/43681/dc893a2d8daa08c689dc69e6bb2c5b50
> 7189607722 /hbase/43681/ffbceaea5e2f142dbe6cd4cbeacc00e8
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
