[
https://issues.apache.org/jira/browse/HBASE-29904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109701#comment-18109701
]
mazhengxuan commented on HBASE-29904:
-------------------------------------
Hi, I'd like to work on this issue.
I checked the current master and the related changes in HBASE-29039 and
HBASE-30036. I will first reproduce the problem with the current code, then
look into adding store-level delete-marker metrics and an optional threshold
for triggering major compaction, reusing the existing compaction controls.
If this approach sounds reasonable, I‘ll assign the issue to me.
Thanks！
> Tombstone-heavy regions are not prioritized for major compaction, causing
> severe scan latency after deletes
> -----------------------------------------------------------------------------------------------------------
>
> Key: HBASE-29904
> URL: https://issues.apache.org/jira/browse/HBASE-29904
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 2.5.10
> Reporter: Emil Kleszcz
> Priority: Major
>
> After large delete workloads, some regions accumulate very high numbers of
> tombstones. Until a major compaction runs, these regions suffer severe scan
> latency degradation from milliseconds to tens of seconds or even minutes.
> HBase currently:
> * does not expose tombstone density per region/store, and
> * does not prioritize major compactions based on tombstone impact.
> As a result, periodic major compactions may miss the worst regions for days
> or weeks, even though user-facing queries are already severely affected.
> Observed behavior:
> * Before deletes or after major compaction: ~50–200 ms
> * After deletes, before major compaction: 10–120 seconds
> * After manual major compaction on the affected region: back to milliseconds
> The slowdown is caused by scanning and filtering large volumes of delete
> markers (tombstones), not by storefile count or region size. Minor
> compactions do not resolve the issue. The issue is not present when _RAW_
> option is set to _true_.
> *Current limitations*
> * No metric to identify tombstone-heavy regions.
> * Compaction selection is driven by file count / size, or last compaction
> time, not delete density.
> * Operators must manually correlate slow queries with regions and trigger
> targeted major compactions.
> * This risks both over-scheduling majors and missing the worst regions.
> *Requested improvement*
> * Expose tombstone-related metrics (e.g. delete marker count or ratio) at
> region/store level.
> * Allow compaction scheduling to prioritize regions with high tombstone
> density or reclaimable space.
> * Provide controls to rate-limit and restrict such tombstone-driven major
> compactions (e.g. off-peak only).
> *Expected benefit*
> * Predictable scan latency after deletes.
> * Faster reclamation of deleted data.
> * Reduced need for operator-driven major compactions prone to over-scheduling.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
