Andrey Yarovoy created HDDS-16288:
-------------------------------------
Summary: Convert OMPerformanceMetrics MutableRate latency counters
to lock-free ConcurrentMutableStat
Key: HDDS-16288
URL: https://issues.apache.org/jira/browse/HDDS-16288
Project: Apache Ozone
Issue Type: Improvement
Reporter: Andrey Yarovoy
h2. Problem
OMPerformanceMetrics declares ~44 latency counters as @Metric MutableRate fields
(OMPerformanceMetrics.java). MutableRate extends MutableStat, whose add(long) is
synchronized on the counter instance:
public synchronized void add(long) // hadoop-common MutableStat
These counters are updated on the OM read/write RPC hot path, and several are
hit
per request. A single getKeyInfo, for example, records into six or seven of
them in
sequence (OmMetadataReader.getKeyInfo, ~lines 160-215):
getKeyInfoResolveBucketLatencyNs, getKeyInfoAclCheckLatencyNs,
getKeyInfoReadKeyInfoLatencyNs, getKeyInfoRefreshLocationLatencyNs,
getKeyInfoGenerateBlockTokenLatencyNs, getKeyInfoSortDatanodesLatencyNs,
getKeyInfoLatencyNs
lookup, listKeys, createKey and the ACL/checkAccess paths have their own sets.
Each
counter is a distinct monitor, so contention is per-counter across threads:
when many
handlers execute the same read op concurrently, they serialize on the same
synchronized
add(). This is the same thundering-herd contention that HDDS-9377 (PR #11085)
removed
from OMLockMetrics and the PerformanceMetrics wrapper by replacing MutableStat
with the lock-free ConcurrentMutableStat — but the raw MutableRate counters in
OMPerformanceMetrics were not part of that change and still use the
synchronized path.
h2. Approach
Reuse the ConcurrentMutableStat introduced in HDDS-9377 (hadoop-hdds/common)
for the OMPerformanceMetrics latency counters instead of MutableRate,
preserving the emitted metric names, num/avg samples, and min/max/stddev
semantics so dashboards and JMX consumers are unaffected. Because these fields
are currently auto-created by the @Metric annotation factory (which produces
MutableRate), registration moves to explicit
construction against the MetricsRegistry (mirroring how OMLockMetrics registers
its
ConcurrentMutableStat), and the capture helpers (captureLatencyNs / add*
methods) are
retargeted to the new type.
h2. Scope
In scope:
- The MutableRate latency counters in OMPerformanceMetrics on OM request paths.
Out of scope:
- OMLockMetrics and PerformanceMetrics (already covered by HDDS-9377).
- Changing hadoop-common MutableRate/MutableStat.
- MutableGaugeLong service-latency gauges in OMPerformanceMetrics
(single-writer,
not contended).
h2. Testing
- Unit coverage that a converted counter records count/sum/min/max equivalently
to
MutableRate and that JMX/metrics-sink snapshot output is unchanged.
- Re-run the mixed read/write OM workload used for HDDS-9377 to confirm the
residual
per-counter contention on OMPerformanceMetrics is removed and read p99 under
concurrent load is unaffected or improved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
