Siyao Meng created HDDS-16693:
---------------------------------
Summary: Nested snapshot read locks on two stripes corrupt the OM
lock hold time metrics
Key: HDDS-16693
URL: https://issues.apache.org/jira/browse/HDDS-16693
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: HDDS-16693-CR-R13-nested-read-lock-metric.patch
h3. Mechanism
{{IOzoneManagerLock.ResourceManager}} keeps one read start time per resource
per thread and removes the {{ThreadLocal}} on read
({{IOzoneManagerLock.java:157-205}}). With two nested read locks on different
stripes of one resource, the second release computes elapsed time from
{{LockUsageInfo}}'s -1 initial value ({{LockUsageInfo.java:25}}), and
{{updateReadUnlockMetrics}} ({{OzoneManagerLock.java:526-540}}) feeds that into
the statistic. The {{readHoldCount == 0}} guard covers reentrancy on one lock
but not two stripes of one resource.
h3. Impact
Ordinary nested snapdiff read locks permanently poison
{{readLockHeldTimeMsStat}} and {{getLongestReadLockHeldTimeMs}}. Operator
visible metrics only, with no data or availability effect, but it degrades the
very signal that would reveal the deadlocks reported separately for MC-4 and
MC-5 in this umbrella.
h3. Reproduction
{{repro/test_bugCR-R13_nested_read_lock_metric.sh}}. Evidence in
{{repro/logs/ev-cr2.log}}.
h3. Patch
{{HDDS-XXXXX-CR-R13-nested-read-lock-metric.patch}} is attached. It applies to
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535 and carries the fix plus a regression
test that was verified to fail without the fix and pass with it.
{code:none}
mvn test -pl hadoop-ozone/ozone-manager -Dtest=TestOzoneManagerLock
{code}
Found by code review of the OM snapshot cache under HDDS-15926, on commit
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated with Specula (Claude Opus
5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
