Emil Kleszcz created HBASE-29904:
------------------------------------
Summary: Tombstone-heavy regions are not prioritized for major
compaction, causing severe scan latency after deletes
Key: HBASE-29904
URL: https://issues.apache.org/jira/browse/HBASE-29904
Project: HBase
Issue Type: Improvement
Affects Versions: 2.5.10
Reporter: Emil Kleszcz
After large delete workloads, some regions accumulate very high numbers of
tombstones. Until a major compaction runs, these regions suffer severe scan
latency degradation from milliseconds to tens of seconds or even minutes.
HBase currently:
* does not expose tombstone density per region/store, and
* does not prioritize major compactions based on tombstone impact.
As a result, periodic major compactions may miss the worst regions for days or
weeks, even though user-facing queries are already severely affected.
Observed behavior:
* Before deletes or after major compaction: ~50–200 ms
* After deletes, before major compaction: 10–120 seconds
* After manual major compaction on the affected region: back to milliseconds
The slowdown is caused by scanning and filtering large volumes of delete
markers (tombstones), not by storefile count or region size. Minor compactions
do not resolve the issue. The issue is not present when _RAW_ option is set to
_true_.
*Current limitations*
* No metric to identify tombstone-heavy regions.
* Compaction selection is driven by file count / size, or last compaction time,
not delete density.
* Operators must manually correlate slow queries with regions and trigger
targeted major compactions.
* This risks both over-scheduling majors and missing the worst regions.
*Requested improvement*
* Expose tombstone-related metrics (e.g. delete marker count or ratio) at
region/store level.
* Allow compaction scheduling to prioritize regions with high tombstone density
or reclaimable space.
* Provide controls to rate-limit and restrict such tombstone-driven major
compactions (e.g. off-peak only).
*Expected benefit*
* Predictable scan latency after deletes.
* Faster reclamation of deleted data.
* Reduced need for operator-driven major compactions prone to over-scheduling.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
