[
https://issues.apache.org/jira/browse/HBASE-26242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-26242:
-------------------------------
Description:
In the requestSplit() function (called by the MemstoreFlusher and
CompactionRunner) for a region, it will check the compaction priority of the
region. If compact priority < PRIORITY_USER , it will not split.
{code:java}
public synchronized boolean requestSplit(final Region r) {
// don't split regions that are blocking
HRegion hr = (HRegion)r;
try {
if (shouldSplitRegion() && hr.getCompactPriority() >= PRIORITY_USER) {
byte[] midKey = hr.checkSplit().orElse(null);
if (midKey != null) {
requestSplit(r, midKey);
return true;
}
}
....{code}
But the region's compact priority is the minimum of all the stores, when the
number of storefiles in a store is larger than the configed
`hbase.hstore.blockingStoreFiles`, the priority will be a negative number, but
the compared priority in requestSplit() is 1(PRIORITY_USER).
{code:java}
public int getStoreCompactionPriority() {
int priority = blockingFileCount - storefiles.size();
return (priority == HStore.PRIORITY_USER) ? priority + 1 : priority;
}
{code}
As a result, when a region should split, but its speed of reducing the number
of files through compaction is slower than the speed of generating new
files(e.g. compacting L0 files to stripes, bulk load, flush memstore), the
region will never split. While split can divide the compaction pressure(1
parent compaction + 2 children compaction can be reduced to 2 children
compaction).
The problem is obvious in StripeStoreEngine, though memstore flushing is
pending when store file count up to the blocking count, each L0 compaction may
generate the stripe count new files to each stripe. And in this scenario, since
the store always compact priority to split, the stripe count is larger and
larger, the new files generated by compact is more and more, no split in the
end...
was:
In the requestSplit() function (called by the MemstoreFlusher and
CompactionRunner) for a region, it will check the compaction priority of the
region.
{code:java}
public synchronized boolean requestSplit(final Region r) {
// don't split regions that are blocking
HRegion hr = (HRegion)r;
try {
if (shouldSplitRegion() && hr.getCompactPriority() >= PRIORITY_USER) {
byte[] midKey = hr.checkSplit().orElse(null);
if (midKey != null) {
requestSplit(r, midKey);
return true;
}
}
....{code}
But the region's compact priority is the minimum of all the stores, when the
number of storefiles in one store is larger than the configed
`hbase.hstore.blockingStoreFiles`, the priority will be a negative number, but
the compared PRIORITY_USER is 1.
{code:java}
public int getStoreCompactionPriority() {
int priority = blockingFileCount - storefiles.size();
return (priority == HStore.PRIORITY_USER) ? priority + 1 : priority;
}
{code}
As a result, when a region should split, but its speed of reducing the number
of files through compaction is slower than the speed of flush, the region will
never split.
While split can divide the compaction pressure(1 parent compaction + 2 child
compaction can be reduced to 2 child compaction).
> Region never split when store file count larger than the configed blocking
> file count
> -------------------------------------------------------------------------------------
>
> Key: HBASE-26242
> URL: https://issues.apache.org/jira/browse/HBASE-26242
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 3.0.0-alpha-1, 1.4.0, 2.0.0
> Reporter: Xiaolin Ha
> Priority: Major
>
> In the requestSplit() function (called by the MemstoreFlusher and
> CompactionRunner) for a region, it will check the compaction priority of the
> region. If compact priority < PRIORITY_USER , it will not split.
> {code:java}
> public synchronized boolean requestSplit(final Region r) {
> // don't split regions that are blocking
> HRegion hr = (HRegion)r;
> try {
> if (shouldSplitRegion() && hr.getCompactPriority() >= PRIORITY_USER) {
> byte[] midKey = hr.checkSplit().orElse(null);
> if (midKey != null) {
> requestSplit(r, midKey);
> return true;
> }
> }
> ....{code}
> But the region's compact priority is the minimum of all the stores, when the
> number of storefiles in a store is larger than the configed
> `hbase.hstore.blockingStoreFiles`, the priority will be a negative number,
> but the compared priority in requestSplit() is 1(PRIORITY_USER).
> {code:java}
> public int getStoreCompactionPriority() {
> int priority = blockingFileCount - storefiles.size();
> return (priority == HStore.PRIORITY_USER) ? priority + 1 : priority;
> }
> {code}
> As a result, when a region should split, but its speed of reducing the number
> of files through compaction is slower than the speed of generating new
> files(e.g. compacting L0 files to stripes, bulk load, flush memstore), the
> region will never split. While split can divide the compaction pressure(1
> parent compaction + 2 children compaction can be reduced to 2 children
> compaction).
> The problem is obvious in StripeStoreEngine, though memstore flushing is
> pending when store file count up to the blocking count, each L0 compaction
> may generate the stripe count new files to each stripe. And in this scenario,
> since the store always compact priority to split, the stripe count is larger
> and larger, the new files generated by compact is more and more, no split in
> the end...
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
