yandrey321 opened a new pull request, #11085: URL: https://github.com/apache/ozone/pull/11085
## What changes were proposed in this pull request? ### Problem MutableStat.add(long) is synchronized. OMLockMetrics records four timing stats (read/write lock held and wait times) on every lock acquire and release. Under high read-lock concurrency — when many OM handler threads release a read-lock simultaneously and all rush to call add() — every thread serialises through the same mutex. This creates a thundering-herd contention point that limits OM throughput as thread counts grow. ### Approach Introduce ConcurrentMutableStat, a drop-in subclass of MutableStat that makes add() non-blocking: Each call accumulates in LongAdder (sum, count) and LongAccumulator (min, max) — all cell-striped via Striped64, so threads write to independent cells with no cross-thread synchronisation. setChanged() is deferred out of the hot-path add() to avoid concurrent volatile writes from all calling threads. Pending cells are drained into the parent's running state lazily, only on snapshot(), lastStat(), or toString() — paths that already hold or take the stat's own lock and are called far less frequently than add(). OMLockMetrics and PerformanceMetrics are updated to use ConcurrentMutableStat for their stat fields. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-9377 ## How was this patch tested? Unit tests (TestConcurrentMutableStat): single-threaded count/mean/min/max correctness, multi-threaded count and extreme-value accuracy, multiple drain cycles. Integration tests: TestFreon (OmBucketReadWriteFileOps, OmBucketReadWriteKeyOps) — both pass; verifyOMLockMetrics asserts sample counts > 0 for all four lock stats through the full cluster lock/unlock path. Benchmark x86 ``` Threads MutableStat µs/burst ConcurrentMutableStat µs/burst Speedup -------------------------------------------------------------------------------------------- 1 29.39 µs 27.46 µs 1.1x 10 744.88 µs 150.38 µs 5.0x 20 1.51 ms 267.03 µs 5.7x 40 2.91 ms 518.87 µs 5.6x 60 8.85 ms 768.37 µs 11.5x 80 7.40 ms 996.14 µs 7.4x === Steady-State (continuous load, 500 ms window) === Threads MutableStat ops/ms ConcurrentMutableStat ops/ms Speedup -------------------------------------------------------------------------------------------- 1 35.83 k ops/ms 60.30 k ops/ms 1.7x 10 4.40 k ops/ms 538.24 k ops/ms 122.4x 20 4.85 k ops/ms 1.06 G ops/ms 218.4x 40 4.71 k ops/ms 1.45 G ops/ms 308.5x 60 5.42 k ops/ms 2.32 G ops/ms 427.7x 80 5.73 k ops/ms 2.58 G ops/ms 450.9x ``` ARM64 ``` Threads MutableStat µs/burst ConcurrentMutableStat µs/burst Speedup -------------------------------------------------------------------------------------------- 1 15.16 µs 14.50 µs 1.0x 10 177.60 µs 235.61 µs 0.8x 20 167.04 µs 378.45 µs 0.4x 40 336.60 µs 1.28 ms 0.3x 60 517.25 µs 1.95 ms 0.3x 80 791.09 µs 2.40 ms 0.3x === Steady-State (continuous load, 500 ms window) === Threads MutableStat ops/ms ConcurrentMutableStat ops/ms Speedup -------------------------------------------------------------------------------------------- 1 292.66 k ops/ms 190.49 k ops/ms 0.7x 10 10.77 k ops/ms 17.97 k ops/ms 1.7x 20 9.96 k ops/ms 18.10 k ops/ms 1.8x 40 10.22 k ops/ms 16.72 k ops/ms 1.6x 60 12.40 k ops/ms 16.57 k ops/ms 1.3x 80 9.44 k ops/ms 18.16 k ops/ms 1.9x ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
