haosenchen opened a new pull request, #8648:
URL: https://github.com/apache/hbase/pull/8648
Backport of the registry-cleanup mechanism only. The per-table
MetricsTableLatenciesImpl.histogramsByTable and
MetricsTableQueryMeterImpl.metersByTable are populated on first access but
never cleaned up when a table leaves the RegionServer (drop / disable / move /
region close). For short-lived tables (e.g. hourly tables with a 7-day
retention window) this leaks both the maps and the per-table entries under the
sub=TableLatencies JMX bean without bound.
* MetricsTableLatencies / MetricsTableQueryMeter: add deleteTable(...) on
both interfaces and their impls.
* MetricsTableLatenciesImpl: switch histogramsByTable to ConcurrentHashMap +
computeIfAbsent (also closes a pre-existing TODO) and implement
deleteTable(String) which removes the entry AND unregisters the 12
MutableTimeHistogram / MutableSizeHistogram objects from the underlying
DynamicMetricsRegistry via removeMetric(baseName). removeMetric is used rather
than removeHistogramMetrics because the histogram is stored under baseName
itself and the 14 suffix metrics (_num_ops / _sum_ops / _min / _max / _mean /
*_percentile) are produced dynamically at snapshot time --
removeHistogramMetrics only tries to remove baseName + 8 fixed suffix keys that
never actually exist in the map, so it is a no-op for MutableHistogram entries.
* MetricsTableQueryMeterImpl: implement deleteTable(TableName) that removes
the entry and unregisters both read/write meters from the shared MetricRegistry.
* RegionServerTableMetrics: expose a single deleteTable(TableName) that
forwards to both cleanup paths.
* MetricsRegionServer: expose getRegionServerTableMetrics() so the cleanup
driver can reach the container.
* MetricsTableWrapperAggregateImpl.TableMetricsWrapperRunnable: compute
onlineTables at the top of run(), take the difference with a new lastSeenTables
snapshot, and invoke RegionServerTableMetrics.deleteTable() for every table
that has left the RegionServer since the previous scheduled run. The cleanup
path runs before the aggregate metrics collection body so it also fires for
users that only enabled per-table latency / query meters and not the aggregate
table metrics.
Tests
-----
* TestMetricsTableLatencies#testDeleteTableRemovesAllLatencyHistograms (new,
unit): verifies that all 12 histogram families of the dropped table disappear,
other tables are untouched, re-adding samples lazily re-registers the
histograms, and deleting an unknown table is a no-op.
* TestMetricsTableWrapperCleanup (new, MediumTests, end-to-end): wires a
mocked HRegionServer + real RegionServerTableMetrics + real
MetricsTableLatenciesImpl and hand-drives two rounds of
TableMetricsWrapperRunnable across an online-regions transition {keep_table,
drop_table} -> {keep_table}. Asserts that every latency histogram and both
query meters of drop_table have vanished from the metrics registry / JMX
snapshot after round 2 while keep_table is preserved.
Notes
-----
HBASE-27681 (the upstream fix for HBASE-27486 on branch-2.6+) is not
backported wholesale because it renames the JMX beans (sub=TableLatencies ->
sub=TableRequests_<table>) which is not backward-compatible for existing
monitoring. This change intentionally keeps the JMX bean names and metric key
layout stable and only backports the registry cleanup mechanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
