Siyao Meng created HDDS-16686:
---------------------------------
Summary: Snapshot cache eviction erases a concurrent enqueue,
leaking an open snapshot DB and breaking later checkpoints
Key: HDDS-16686
URL: https://issues.apache.org/jira/browse/HDDS-16686
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: HDDS-16686-MC-2-eviction-queue-erasure.patch
h3. Mechanism
{{SnapshotCache.cleanup(UUID, boolean)}} ({{SnapshotCache.java:379-417}}) reads
{{v.getTotalRefCount()}} at {{:400}} inside the {{dbMap.compute}} lambda. On
the keep branch it falls through to {{pendingEvictionQueue.remove(k)}} at
{{:413}}, which executes on every branch.
A releasing thread that drops the last reference between those two points
enqueues the key outside that bin lock, and the evictor then erases the
enqueue. The entry stays in {{dbMap}} with refcount 0 and is no longer queued,
so nothing will ever evict it.
h3. Impact
The snapshot DB stays open until OM restart, and {{SnapshotCache.lock()}}'s
drain check at {{:334-336}} begins failing on every later attempt, which breaks
admin checkpoint download and follower bootstrap. There is no automatic
recovery.
h3. Reproduction
{{repro/test_bugMC-2_eviction_queue_erasure.sh}} with
{{repro/java/SpeculaMc2EvictionQueueTest.java}}. Evidence in
{{repro/logs/ev-mc2.log}} and {{ev-mc2b.log}}.
Note on probability: the race did not trigger in 4000 unassisted iterations,
because the window between the refcount read at {{:400}} and the queue removal
at {{:413}} is a single debug log statement. It was reproduced by widening that
window with {{repro/level3-snapshotcache-keep-window.patch}}, which confirms
the ordering and demonstrates the consequence: final state {{inDbMap=true
refCount=0 inQueue=false}}, and the next {{snapshotCache.lock()}} throws
{{IllegalStateException: Failed to acquire lock as cleanup did not drain the
cache.}} The natural probability on an unpatched build is low; the fix is a one
line move of the queue removal onto the evict branch.
h3. Patch
{{HDDS-XXXXX-MC-2-eviction-queue-erasure.patch}} is attached. It applies to
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535 and carries the fix plus a regression
test that was verified to fail without the fix and pass with it.
{code:none}
mvn test -pl hadoop-ozone/ozone-manager -Dtest=TestSnapshotCache
{code}
Found by TLA+ model checking and code review of the OM snapshot cache under
HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated with
Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
