[
https://issues.apache.org/jira/browse/HDDS-16693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16693:
------------------------------
Attachment: HDDS-16693-CR-R13-nested-read-lock-metric.patch
> Nested snapshot read locks on two stripes corrupt the OM lock hold time
> metrics
> -------------------------------------------------------------------------------
>
> Key: HDDS-16693
> URL: https://issues.apache.org/jira/browse/HDDS-16693
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: HDDS-16693-CR-R13-nested-read-lock-metric.patch
>
>
> h3. Mechanism
> {{IOzoneManagerLock.ResourceManager}} keeps one read start time per resource
> per thread and removes the {{ThreadLocal}} on read
> ({{IOzoneManagerLock.java:157-205}}). With two nested read locks on different
> stripes of one resource, the second release computes elapsed time from
> {{LockUsageInfo}}'s -1 initial value ({{LockUsageInfo.java:25}}), and
> {{updateReadUnlockMetrics}} ({{OzoneManagerLock.java:526-540}}) feeds that
> into the statistic. The {{readHoldCount == 0}} guard covers reentrancy on one
> lock but not two stripes of one resource.
> h3. Impact
> Ordinary nested snapdiff read locks permanently poison
> {{readLockHeldTimeMsStat}} and {{getLongestReadLockHeldTimeMs}}. Operator
> visible metrics only, with no data or availability effect, but it degrades
> the very signal that would reveal the deadlocks reported separately for MC-4
> and MC-5 in this umbrella.
> h3. Reproduction
> {{repro/test_bugCR-R13_nested_read_lock_metric.sh}}. Evidence in
> {{repro/logs/ev-cr2.log}}.
> h3. Patch
> {{HDDS-XXXXX-CR-R13-nested-read-lock-metric.patch}} is attached. It applies
> to 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535 and carries the fix plus a
> regression test that was verified to fail without the fix and pass with it.
> {code:none}
> mvn test -pl hadoop-ozone/ozone-manager -Dtest=TestOzoneManagerLock
> {code}
> Found by code review of the OM snapshot cache under HDDS-15926, on commit
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated with Specula (Claude Opus
> 5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
