Hello Marton Greber, Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24618
to look at the new patch set (#5).
Change subject: KUDU-3734 Compaction budget refinements and observability
......................................................................
KUDU-3734 Compaction budget refinements and observability
KUDU-3734 added UNDO delta sizes to the knapsack budget used for rowset
compaction selection to prevent OOM. While effective, including UNDO
deltas at full weight can cause compaction to skip rowsets
unnecessarily, particularly when UNDO deltas are large relative to
base+redo data. Skipped rowsets accumulate, increase average rowset
height, and eventually slow down scans because each scan must open
iterators across more overlapping DiskRowSets.
This change addresses three concerns raised as follow-ups:
1. Budget refinements to reduce unnecessary skipping
a) --rowset_undo_deltas_budget_weight(double, default 1.0, runtime):
A fractional multiplier applied to UNDO delta sizes when computing
the budget weight of a rowset candidate. At the default of 1.0
there is no change in behaviour.
b) --rowset_deltas_size_include_undo is now tagged 'runtime':
Previously this flag required a server restart to toggle. Making
it runtime-changeable enables operators and future auto-disable
mechanisms to flip it live without disruption.
2. Metrics to detect slow scans caused by budget-induced compaction lag
a) compact_rs_rowsets_budget_skipped (tablet-level, gauge, kDebug):
Number of DiskRowSet candidates not included in the most recent
compaction selection. This gauge reflects the last PickRowSets()
call and resets to 0 each run. A value that is persistently large
alongside a rising 'average_diskrowset_height' suggests the budget
is preventing useful compaction work on certain rowsets.
b) compact_rs_bytes_budget_skipped (tablet-level, gauge, kDebug):
The total on-disk footprint (base + redo + weighted undo) of those
skipped candidates, to given an idea of data size being deferred.
c) rowsets_per_scan (tablet-level, histogram, max=1024, kDebug):
Tracks the distribution of DiskRowSet iterators opened per scan
request, recorded in Tablet::CaptureConsistentIterators(). The MRS
and any txn_memrowset iterators are excluded so the histogram
measures only the on-disk overlap a scan must resolve. A rising
P99 is the most direct observable consequence of insufficient
compaction: more overlapping DiskRowSets means more data sources
that must be merged to produce a single scan result.
3. Detection and operator notification when the system is degraded
a. compaction_budget_skip_health (tablet-level, gauge, uint32, kInfo):
Set to 1 (degraded) when at least one DiskRowSet candidate
individually exceeds --tablet_compaction_budget_mb and is therefore
permanently unselectable under the current settings. Set to 0
(healthy) otherwise.
b. tablets_num_compaction_budget_degraded
(tablet-server-level, gauge, int32, kWarn):
Number of tablet replicas on the tablet server whose
compaction_budget_skip_health gauge is 1 (degraded).
c. Throttled WARNING log (at most once per 300 s):
When oversized rowsets are detected, a WARNING is emitted naming
the exact flags to adjust, so users know what to do from the
tserver log.
Change-Id: Id63db0006c77ad1d55eb2119d99ee64499435819
---
M src/kudu/tablet/compaction_policy.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/diskrowset.h
M src/kudu/tablet/rowset.cc
M src/kudu/tablet/rowset.h
M src/kudu/tablet/rowset_info.cc
M src/kudu/tablet/rowset_info.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet_metrics.cc
M src/kudu/tablet/tablet_metrics.h
M src/kudu/tserver/ts_tablet_manager.cc
11 files changed, 266 insertions(+), 19 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/18/24618/5
--
To view, visit http://gerrit.cloudera.org:8080/24618
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id63db0006c77ad1d55eb2119d99ee64499435819
Gerrit-Change-Number: 24618
Gerrit-PatchSet: 5
Gerrit-Owner: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
