Siyao Meng created HDDS-17292:
---------------------------------
Summary: OMKeyRequest takes a second BUCKET lock while holding
one, out of stripe order, which can deadlock the OM apply thread
Key: HDDS-17292
URL: https://issues.apache.org/jira/browse/HDDS-17292
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-5-release-bucket-lock-before-resolving-source-bucket.patch
h3. Mechanism
{{OMKeyRequest.generateRequiredEncryptionInfo}} takes the BUCKET read lock of
the requested bucket and reads its {{OmBucketInfo}}. When that bucket points to
a source bucket, it calls {{OzoneManager.resolveBucketLink}} while it still
holds that lock, and the resolution takes the BUCKET read lock of the source
bucket in {{BucketManagerImpl.getBucketInfo}}.
{{OMKeyRequest.getFileEncryptionInfoForMpuKey}} does the same: it takes the
read lock of the requested bucket first and calls {{resolveBucketLink}} inside
it. Both run on the RPC handler thread, and both locks are stripes of the same
{{BUCKET_LOCK}} of {{OzoneManagerLock}}. So the handler holds one stripe and
waits for a second one, in whatever order the two stripes happen to be.
On the Ratis apply thread, {{OMKeyPurgeRequest.updateBucketSize}},
{{OMDirectoriesPurgeRequestWithFSO}} and
{{S3ExpiredMultipartUploadsAbortRequest}} write lock all buckets of one request
with {{IOzoneManagerLock.acquireWriteLocks}}, which takes the stripes in
ascending order ({{Striped.bulkGet}}). That order prevents a cycle between two
bulk holders, but not between a bulk holder and a thread that nests two single
acquisitions in the opposite order.
h3. Trigger
Two lock holders in opposite stripe order, with stripe(B) below stripe(A):
# An RPC handler, in one of the two methods above, holds the BUCKET read lock
stripe of bucket A and waits for the read lock stripe of B, the source of A.
# The apply thread, in a bulk acquire over a set of buckets with one bucket on
stripe(B) and one on stripe(A), holds stripe(B) for write and waits for
stripe(A) for write.
Neither wait has a timeout, and nothing else releases either lock.
h3. Impact
* The apply thread of the OM leader waits forever, so no later write is applied
on that OM and every write waits for its reply.
* Each such write keeps its RPC handler while it waits in
{{OzoneManagerRatisServer.submitRequest}} for the reply, which only comes after
the apply thread has applied it. Reads on the leader do not go through the
apply thread by default ({{ozone.om.ha.raft.server.read.option}} is
{{DEFAULT}}), but they share the handler pool ({{ozone.om.handler.count.key}}),
so once all handlers are held by waiting writes, reads wait in the call queue
too. Reads that need a bucket on one of the two stripes block on the lock
directly, since one stripe is write held and the other has a queued writer
(code reading).
* The followers keep applying the log, because the nested acquisition only runs
on the leader before submission. The leader stays leader, so the hang lasts
until that OM is restarted (argued, restart not run). No data is lost.
By code reading at the release tags, not run on older code: the nested
acquisition is present from 1.2.0 (HDDS-5530). Before 1.4.0 each bucket has its
own BUCKET lock (no striping), and the cycle would need the apply thread to
write lock the requested bucket itself, which does not appear in a purge. From
1.4.0 the BUCKET locks are striped (HDDS-8765) and
{{OMDirectoriesPurgeRequestWithFSO}} holds several bucket write locks taken one
at a time, and from 2.1.0 {{OMKeyPurgeRequest}} and
{{OMDirectoriesPurgeRequestWithFSO}} use the sorted bulk acquire.
h3. Reproduction
No standalone reproduction is attached.
The regression test in the patch sets up the conflicting holder through
internal calls.
{{TestOMKeyCreateRequest.preExecuteReleasesBucketLockBeforeResolvingSourceBucket}}
({{hadoop-ozone/ozone-manager}}, also in the inherited {{...WithFSO}}, for
both methods above) stubs {{resolveBucketLink}} so that, while the source
bucket is resolved, another thread write locks the requested bucket and its
source with {{acquireWriteLocks}}, as the apply thread does. That thread must
complete within 5 seconds. Without the change all four cases fail with
{{Expecting <CompletableFuture[Incomplete]> to be completed within 5S}},
because the resolving thread still holds the read lock of the requested bucket.
h3. Patch
[^MC-5-release-bucket-lock-before-resolving-source-bucket.patch], against
e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master at
1ff29ee6b86.
{{generateRequiredEncryptionInfo}} now releases the read lock of the requested
bucket before it calls {{resolveBucketLink}}, and then reads the resolved
bucket as before. {{getFileEncryptionInfoForMpuKey}} calls
{{resolveBucketLink}} before it takes the read lock. {{resolveBucketLink}}
takes and releases the lock of each bucket it reads on its own, so on these
paths the handler now holds at most one BUCKET lock at a time and the cycle
cannot form. The locks taken and the data read are otherwise the same: the
resolved bucket was not read under its own lock on this path before either, and
in both versions the lock is released before the request is submitted. Nothing
changes for a bucket that does not point to another bucket, and there is no
wire or protocol change.
With the patch the OM request unit suites that cover these two methods and the
purge requests pass (593 tests in {{hadoop-ozone/ozone-manager}}, 1 skipped as
before), as does {{TestOzoneAtRestEncryption}} on a mini cluster, and
checkstyle on {{hadoop-ozone/ozone-manager}} is clean.
Found by TLA+ model checking and code review of the Ozone Manager lock ordering
paths under HDDS-15926, on commit e0de543cbe9d73f3bf5473a45e1c59cd6eae7173.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. Related but not duplicates: HDDS-13729 (sorted bulk bucket locks
in the directory purge), HDDS-7755 (an earlier OM hang from a leaked lock,
HDDS-7925 is its duplicate). The attached patch is a proposal for review.
Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
