Ashwani Raina has uploaded this change for review. (
http://gerrit.cloudera.org:8080/24618
Change subject: [compaction] budget refinements and observability
......................................................................
[compaction] budget refinements and observability
KUDU-3734 added UNDO delta sizes to the knapsack budget used for rowset
compaction selection to prevent OOM. While effective, including UNDO
deltas at full weight can cause compaction to skip rowsets
unnecessarily, particularly when UNDO deltas are large relative to
base+redo data. Skipped rowsets accumulate, increase average rowset
height, and eventually slow down scans because each scan must open
iterators across more overlapping DiskRowSets.
This change addresses three concerns raised as follow-ups:
1. Budget refinements to reduce unnecessary skipping
a) --rowset_undo_deltas_budget_weight(double, default 1.0, runtime):
A fractional multiplier applied to UNDO delta sizes when computing
the budget weight of a rowset candidate. At the default of 1.0
there is no change in behaviour.
b) --rowset_deltas_size_include_undo is now tagged 'runtime':
Previously this flag required a server restart to toggle. Making
it runtime-changeable enables operators and future auto-disable
mechanisms to flip it live without disruption.
2. Metrics to detect slow scans caused by budget-induced compaction lag
a) compact_rs_rowsets_budget_skipped (counter, kDebug):
Incremented with the number of candidate DiskRowSets not included
in the winning compaction selection after each run of
BudgetedCompactionPolicy::PickRowSets(). A rising rate along with
a rising average_diskrowset_height, indicates that the compaction
budget is preventing useful work.
b) compact_rs_bytes_budget_skipped (counter, kDebug):
The total on-disk footprint (base + redo + weighted undo) of those
skipped candidates, to given an idea of data size being deferred.
c) rowsets_per_scan (histogram, max=1024, kDebug):
Tracks the distribution of DiskRowSet iterators opened per scan
request, recorded in Tablet::CaptureConsistentIterators(). The MRS
and any txn_memrowset iterators are excluded so the histogram
measures only the on-disk overlap a scan must resolve. A rising
P99 is the most direct observable consequence of insufficient
compaction: more overlapping DiskRowSets means more data sources
that must be merged to produce a single scan result.
3. Detection and operator notification when the system is degraded
a. compaction_budget_skip_health (gauge, uint32, kWarn):
Set to 1 (degraded) when at least one DiskRowSet candidate
individually exceeds --tablet_compaction_budget_mb and is therefore
permanently unselectable under the current settings. Set to 0
(healthy) otherwise.
b. Throttled WARNING log (at most once per 300 s):
When oversized rowsets are detected, a WARNING is emitted naming
the exact flags to adjust, so users know what to do from the
tserver log.
Change-Id: Id63db0006c77ad1d55eb2119d99ee64499435819
---
M src/kudu/tablet/compaction_policy.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet_metrics.cc
M src/kudu/tablet/tablet_metrics.h
5 files changed, 130 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/18/24618/1
--
To view, visit http://gerrit.cloudera.org:8080/24618
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id63db0006c77ad1d55eb2119d99ee64499435819
Gerrit-Change-Number: 24618
Gerrit-PatchSet: 1
Gerrit-Owner: Ashwani Raina <[email protected]>
