Siyao Meng created HDDS-17291:
---------------------------------
Summary: SST value pruner waits for the bootstrap lock while
holding the differ monitor, so OM bootstrap can deadlock with snapshot diff or
defrag
Key: HDDS-17291
URL: https://issues.apache.org/jira/browse/HDDS-17291
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-1-take-bootstrap-lock-before-differ-monitor-in-sst-pruner.patch,
TestBugMC1PrunerSnapshotDiffCheckpointDeadlock.java,
TestBugMC2DefragPrunerCheckpointDeadlock.java
h3. Mechanism
{{RocksDBCheckpointDiffer.pruneSstFileValues}} enters {{synchronized (this)}}
for each compaction log entry, rewrites the entry's backup SST files and, still
inside the monitor, takes the BOOTSTRAP read lock (key
{{rocksdbCheckpointDifferAOS}}, supplied by {{OmMetadataManagerImpl.loadDB}})
around the {{Files.move}} that replaces each file. Apart from
{{loadAllCompactionLogs}}, which runs once while the DB is opened and removes
orphaned backup files under the bootstrap read lock inside the monitor, every
other user of the two locks takes them in the opposite order:
{{pruneOlderSnapshotsWithCompactionHistory}} and {{SnapshotDefragService}} take
the bootstrap lock first, and a snapshot diff job holds SNAPSHOT_DB read locks
on both snapshots ({{SnapshotCache.get}}) when
{{RDBDifferComputer.computeDeltaFiles}} enters {{synchronized (differ)}}. The
differ monitor is not a tracked lock resource, so the DAG lock tracker does not
catch this.
The checkpoint servlet ({{OMDBCheckpointServletInodeBasedXfer}}) write-locks
every BOOTSTRAP stripe through {{OMDBCheckpointServlet.Lock}} and holds them
while it collects the files. A follower bootstrap always requests snapshot
data, so in its last batch the servlet also write-locks every SNAPSHOT_DB
stripe through {{SnapshotCache.lock()}}. That gives two cycles:
* *Snapshot diff (default configuration).* The pruner holds the monitor and
waits for BOOTSTRAP read, which the servlet holds. The snapshot diff job holds
SNAPSHOT_DB read and waits for the monitor. The servlet holds BOOTSTRAP write
and waits for SNAPSHOT_DB write.
* *Defrag (non default stripe count).* {{SnapshotDefragService}} holds
BOOTSTRAP read with key {{SnapshotDefragService}} and waits for the monitor in
{{RDBDifferComputer.computeDeltaFiles}}. The pruner holds the monitor and waits
for BOOTSTRAP read on its own stripe. The servlet, which takes the stripes in
ascending order, has already taken the pruner's stripe (or is queued on it) and
waits for the defrag stripe. This needs the pruner's stripe at or below the
defrag stripe. Both keys are constants, so the outcome is fixed by
{{ozone.om.lock.stripes.bootstrap_lock}}: no cycle at the default 512 (or 32,
1024), a cycle at 1, 2, 4, 8, 16, 64, 128, 256 and 2048 to 32768 (computed from
the Guava stripe mapping; other values round up to the next power of two, so
100 behaves like 128). That key is not documented.
h3. Trigger
Both cycles need the rocks-tools native library, which Linux release builds
include (built with {{-Drocks_tools_native}}) and the OM loads by default;
without it the pruner is not scheduled. The 2.2.1 library does not load on RHEL
or Rocky Linux 9, including the {{apache/ozone:2.2.1}} image (HDDS-17153, fixed
in 2.3.0), so neither cycle forms on those hosts. Snapshots must exist and a
tracked column family must have been compacted since, so that the prune queue
is not empty.
For the snapshot diff cycle a follower must be bootstrapping from the leader,
because only a follower bootstrap requests snapshot data and so makes the
servlet take {{SnapshotCache.lock()}}. The pruner's periodic run (every 10
minutes by default) has to start while the servlet holds the bootstrap lock, or
be in its rewrite when the servlet arrives. Either way the pruner ends up
parked on the bootstrap lock while holding the monitor. A snapshot diff job
that reaches its delta file computation before the servlet reaches
{{SnapshotCache.lock()}} then closes the cycle. The servlet holds the bootstrap
lock for its whole collection phase, so this window is as long as copying the
snapshot directories (argued from the code; the reproduction orders the steps
with a latch).
For the defrag cycle, {{ozone.snapshot.defrag.service.interval}} must be
enabled (it is off by default) and the stripe count must be one of the values
above. Any request that takes the BOOTSTRAP write lock is enough, not only a
follower bootstrap: Recon's OM DB fetch (which does not request snapshot data)
and the v1 {{/dbCheckpoint}} servlet take the same lock.
h3. Impact
All waits are untimed lock or monitor waits, and the JVM's deadlock detection
does not report them because read locks have no owner. Only an OM restart
recovers. On the leader:
* the checkpoint request never returns, so the follower cannot finish its
bootstrap; a retry queues behind the same lock;
* every later user of the BOOTSTRAP read lock blocks behind the servlet: the
key, directory and snapshot deleting services when they submit purges,
{{SstFilteringService}}, {{SnapshotDefragService}} and the differ's pruning
tasks, so space is no longer reclaimed;
* later snapshot diff jobs block on the differ monitor. Snapshot reads and
diffs whose snapshots map to the SNAPSHOT_DB stripes the servlet has already
taken also block (argued, not run).
In the defrag cycle the servlet waits at the defrag stripe, so only bootstrap
readers on that stripe and the stripes below it block, and snapshot reads are
not affected. The requester (follower or Recon) never gets its checkpoint, and
defrag, the differ's pruning tasks (they share the pruner's thread) and later
snapshot diff jobs hang as above.
Released versions 2.2.0 and 2.2.1 have the same code (read from the tags).
2.1.x has the nesting, but its servlet does not take {{SnapshotCache.lock()}},
so the snapshot diff cycle does not form there.
h3. Reproduction
Two self contained classes for {{hadoop-ozone/ozone-manager}} (target path and
Maven command in the class comment). Both pass on unmodified source when the
bug is present. They use the real {{OzoneManagerLock}}, the OM DB and differ
built by {{OmMetadataManagerImpl.loadDB}} (so the production lock supplier), a
prune queue entry produced by a real compaction of keyTable with a
snapshotInfoTable row present, {{pruneSstFileValues}},
{{RDBDifferComputer.computeDeltaFiles}} and the servlet's
{{OMDBCheckpointServlet.Lock}}. The native SST reader is mocked as in
{{TestRocksDBCheckpointDiffer}}, so they run without the native library, and
the snapshot local data provider is mocked. The servlet request, the snapshot
diff job and the defrag task are replayed at their lock steps, not run end to
end through HTTP or the services.
* [^TestBugMC1PrunerSnapshotDiffCheckpointDeadlock.java]: default
configuration. A latch in the mocked SST scan keeps the pruner in its rewrite
until the snapshot diff thread is blocked on the monitor and the servlet thread
waits in {{SnapshotCache.lock()}}. After the latch opens none of the three
finishes; the pruner waits on the lock owned by the servlet thread, the
snapshot diff thread on the monitor owned by the pruner, and a later bootstrap
read (key {{KeyDeletingService}}) blocks too.
* [^TestBugMC2DefragPrunerCheckpointDeadlock.java]: defrag cycle with the
bootstrap lock of a real {{SnapshotDefragService}}, parameterized over 2, 256
and 2048 stripes (all three threads stuck, defrag blocked on the monitor owned
by the pruner) and a control at the default 512 (all three complete).
Not exercised: a real follower bootstrap over HTTP, a real snapshot diff or
defrag run, and the real native library.
h3. Patch
[^MC-1-take-bootstrap-lock-before-differ-monitor-in-sst-pruner.patch], against
e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master at
1ff29ee6b86.
{{pruneSstFileValues}} now takes the BOOTSTRAP read lock for each compaction
log entry before entering {{synchronized (this)}}, instead of around
{{Files.move}} inside it. The pruner then never waits for the bootstrap lock
while holding the monitor, which removes the edge both cycles need, at any
stripe count. The move is still done under the bootstrap read lock, so a
checkpoint that holds the write lock still cannot see a backup SST being
replaced (the protection added by HDDS-12501 and HDDS-13905). The rest of the
method is unchanged apart from indentation ({{git diff -w}} shows only the real
change).
The cost is a longer wait for a bootstrap. The lock now covers one compaction
log entry, whose input file list can be long after a large or manual
compaction, and the pruner holds it while it waits for the differ monitor, so a
bootstrap that arrives then also waits for any snapshot diff or defrag already
in the monitor. While the servlet waits on the pruner's stripe it holds the
stripes below it, so bootstrap readers on those stripes wait as well. The
servlet is not starved: the pruner releases the lock between entries, and with
the default nonfair lock its next read waits behind the queued writer.
After the patch {{loadAllCompactionLogs}} is the only path left that takes the
monitor and then the bootstrap lock (through {{cleanupOrphanedSstBackupFiles}},
added by HDDS-14742 after 2.2.1). It cannot deadlock: it runs once in the
{{RDBStore}} constructor, before the pruner's first run (scheduled one interval
after start) and before this OM serves checkpoint requests. The patch leaves it
unchanged.
New test
{{TestRocksDBCheckpointDiffer#testPruneSstFileValuesWaitsForBootstrapLockOutsideDifferMonitor}}:
while another thread holds the bootstrap write lock it checks that the backup
SST file is unchanged and that a snapshot diff style {{synchronized (differ)}}
can proceed, then that the file is pruned once the lock is released. It fails
without the change. With it the {{rocksdb-checkpoint-differ}} module tests
(121), the related {{ozone-manager}} unit tests (141, including
{{TestSnapshotDefragService}}, {{TestSnapshotDiffManager}},
{{TestRDBDifferComputer}}) and {{TestOMDbCheckpointServletInodeBasedXfer}} (15)
pass, and both attached reproductions no longer show the deadlock.
Found by TLA+ model checking and code review of the Ozone Manager lock ordering
paths under HDDS-15926, on commit e0de543cbe9d73f3bf5473a45e1c59cd6eae7173.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. Related but not duplicates: HDDS-13905 (resolved, made
background bootstrap locks read locks and kept this nesting), HDDS-9486
(resolved, an earlier pruner and checkpoint deadlock), HDDS-16685 (snapshot
diff takes SNAPSHOT_DB stripes out of order against the same servlet step) and
HDDS-17259 (the compaction listener synchronizes on the wrong object in the
same class). The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
