[
https://issues.apache.org/jira/browse/HDDS-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17292:
------------------------------
Attachment: MC-5-release-bucket-lock-before-resolving-source-bucket.patch
> OMKeyRequest takes a second BUCKET lock while holding one, out of stripe
> order, which can deadlock the OM apply thread
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17292
> URL: https://issues.apache.org/jira/browse/HDDS-17292
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Critical
> Attachments:
> MC-5-release-bucket-lock-before-resolving-source-bucket.patch
>
>
> h3. Mechanism
> {{OMKeyRequest.generateRequiredEncryptionInfo}} takes the BUCKET read lock of
> the requested bucket and reads its {{OmBucketInfo}}. When that bucket points
> to a source bucket, it calls {{OzoneManager.resolveBucketLink}} while it
> still holds that lock, and the resolution takes the BUCKET read lock of the
> source bucket in {{BucketManagerImpl.getBucketInfo}}.
> {{OMKeyRequest.getFileEncryptionInfoForMpuKey}} does the same: it takes the
> read lock of the requested bucket first and calls {{resolveBucketLink}}
> inside it. Both run on the RPC handler thread, and both locks are stripes of
> the same {{BUCKET_LOCK}} of {{OzoneManagerLock}}. So the handler holds one
> stripe and waits for a second one, in whatever order the two stripes happen
> to be.
> On the Ratis apply thread, {{OMKeyPurgeRequest.updateBucketSize}},
> {{OMDirectoriesPurgeRequestWithFSO}} and
> {{S3ExpiredMultipartUploadsAbortRequest}} write lock all buckets of one
> request with {{IOzoneManagerLock.acquireWriteLocks}}, which takes the stripes
> in ascending order ({{Striped.bulkGet}}). That order prevents a cycle between
> two bulk holders, but not between a bulk holder and a thread that nests two
> single acquisitions in the opposite order.
> h3. Trigger
> Two lock holders in opposite stripe order, with stripe(B) below stripe(A):
> # An RPC handler, in one of the two methods above, holds the BUCKET read lock
> stripe of bucket A and waits for the read lock stripe of B, the source of A.
> # The apply thread, in a bulk acquire over a set of buckets with one bucket
> on stripe(B) and one on stripe(A), holds stripe(B) for write and waits for
> stripe(A) for write.
> Neither wait has a timeout, and nothing else releases either lock.
> h3. Impact
> * The apply thread of the OM leader waits forever, so no later write is
> applied on that OM and every write waits for its reply.
> * Each such write keeps its RPC handler while it waits in
> {{OzoneManagerRatisServer.submitRequest}} for the reply, which only comes
> after the apply thread has applied it. Reads on the leader do not go through
> the apply thread by default ({{ozone.om.ha.raft.server.read.option}} is
> {{DEFAULT}}), but they share the handler pool
> ({{ozone.om.handler.count.key}}), so once all handlers are held by waiting
> writes, reads wait in the call queue too. Reads that need a bucket on one of
> the two stripes block on the lock directly, since one stripe is write held
> and the other has a queued writer (code reading).
> * The followers keep applying the log, because the nested acquisition only
> runs on the leader before submission. The leader stays leader, so the hang
> lasts until that OM is restarted (argued, restart not run). No data is lost.
> By code reading at the release tags, not run on older code: the nested
> acquisition is present from 1.2.0 (HDDS-5530). Before 1.4.0 each bucket has
> its own BUCKET lock (no striping), and the cycle would need the apply thread
> to write lock the requested bucket itself, which does not appear in a purge.
> From 1.4.0 the BUCKET locks are striped (HDDS-8765) and
> {{OMDirectoriesPurgeRequestWithFSO}} holds several bucket write locks taken
> one at a time, and from 2.1.0 {{OMKeyPurgeRequest}} and
> {{OMDirectoriesPurgeRequestWithFSO}} use the sorted bulk acquire.
> h3. Reproduction
> No standalone reproduction is attached.
> The regression test in the patch sets up the conflicting holder through
> internal calls.
> {{TestOMKeyCreateRequest.preExecuteReleasesBucketLockBeforeResolvingSourceBucket}}
> ({{hadoop-ozone/ozone-manager}}, also in the inherited {{...WithFSO}}, for
> both methods above) stubs {{resolveBucketLink}} so that, while the source
> bucket is resolved, another thread write locks the requested bucket and its
> source with {{acquireWriteLocks}}, as the apply thread does. That thread must
> complete within 5 seconds. Without the change all four cases fail with
> {{Expecting <CompletableFuture[Incomplete]> to be completed within 5S}},
> because the resolving thread still holds the read lock of the requested
> bucket.
> h3. Patch
> [^MC-5-release-bucket-lock-before-resolving-source-bucket.patch], against
> e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master at
> 1ff29ee6b86.
> {{generateRequiredEncryptionInfo}} now releases the read lock of the
> requested bucket before it calls {{resolveBucketLink}}, and then reads the
> resolved bucket as before. {{getFileEncryptionInfoForMpuKey}} calls
> {{resolveBucketLink}} before it takes the read lock. {{resolveBucketLink}}
> takes and releases the lock of each bucket it reads on its own, so on these
> paths the handler now holds at most one BUCKET lock at a time and the cycle
> cannot form. The locks taken and the data read are otherwise the same: the
> resolved bucket was not read under its own lock on this path before either,
> and in both versions the lock is released before the request is submitted.
> Nothing changes for a bucket that does not point to another bucket, and there
> is no wire or protocol change.
> With the patch the OM request unit suites that cover these two methods and
> the purge requests pass (593 tests in {{hadoop-ozone/ozone-manager}}, 1
> skipped as before), as does {{TestOzoneAtRestEncryption}} on a mini cluster,
> and checkstyle on {{hadoop-ozone/ozone-manager}} is clean.
> Found by TLA+ model checking and code review of the Ozone Manager lock
> ordering paths under HDDS-15926, on commit
> e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-13729 (sorted bulk bucket locks in the directory purge),
> HDDS-7755 (an earlier OM hang from a leaked lock, HDDS-7925 is its
> duplicate). The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
