[
https://issues.apache.org/jira/browse/HDDS-17419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17419:
------------------------------
Attachment: CR-8-reject-if-none-match-etag-before-om-finalization.patch
> OM accepts If-None-Match ETag on CreateKey and CompleteMultipartUpload before
> it is finalized for S3_IF_NONE_MATCH_ETAG
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17419
> URL: https://issues.apache.org/jira/browse/HDDS-17419
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-8-reject-if-none-match-etag-before-om-finalization.patch,
> TestBugCR8PreFinalizedIfNoneMatch.java
>
>
> h3. Mechanism
> {{OzoneManagerVersion.S3_IF_NONE_MATCH_ETAG}} (104) is gated in the OM only
> by {{OzoneManager.checkFeatureEnabled}}, called from
> {{OMKeyCreateRequest.preExecute}} and
> {{S3MultipartUploadCompleteRequest.preExecute}} when {{KeyArgs.excludedETag}}
> is set. That method reads {{ozone.om.features.disabled}} and never the
> apparent version. No request in main code uses {{@DisallowedUntilOmVersion}},
> and no request validator checks the field. So an OM that is not finalized for
> 104 evaluates the condition in {{validateAndUpdateCache}} like a finalized
> one. The ZDU design requires new features to be unavailable until the upgrade
> is finalized, with an error for a client that tries one.
> h3. Trigger
> # The OMs run the current software and are pre-finalized (apparent version
> ZDU or a pre-ZDU layout version), as during an upgrade before finalization.
> # A client that does not check the OM version first sends CreateKey or
> CompleteMultiPartUpload with {{excludedETag}} over the OM protocol. This
> could be a non-Java client, or {{OzoneManagerProtocol}} used directly.
> {{RpcClient}}, and S3 Gateway through it, refuses on its cached OM version,
> so the stock clients do not send the field in this state.
> h3. Impact
> The OM applies If-None-Match before finalization: {{KEY_ALREADY_EXISTS}} for
> a matching ETag, or an open key guarded by {{expectedDataGeneration}}
> otherwise. At this commit every ZDU capable OM runs the same software, so all
> replicas apply the request the same way, and the guard field predates ZDU, so
> nothing is persisted in a new format. I found no replica divergence or data
> loss reachable today. The gap matters for the pattern: a later OM version
> gated the same way would, during a rolling upgrade or downgrade between ZDU
> releases, let a replica on the older software ignore the field and apply the
> request differently (argued, not run).
> h3. Reproduction
> PASS on unmodified source. [^TestBugCR8PreFinalizedIfNoneMatch.java]
> ({{hadoop-ozone/ozone-manager}}) starts one real OM with apparent version ZDU
> (test only {{OMStorage.TESTING_INIT_APPARENT_VERSION_KEY}}, software version
> 104) and sends requests through {{OzoneManagerProtocol}}. As a control,
> {{RpcClient.createKeyIfNoneMatch}} refuses. Results:
> * CreateKey with the current ETag as If-None-Match returns
> {{KEY_ALREADY_EXISTS}}.
> * CreateKey with another ETag succeeds, and the open key carries
> {{expectedDataGeneration}}.
> * CompleteMultiPartUpload with the current ETag returns
> {{KEY_ALREADY_EXISTS}}.
> A passing test means the defect is present. An earlier run of the same
> scenario on a three OM mini cluster during the analysis showed all three
> replicas applying it; I did not repeat that run. No mixed software ring was
> run, as none can be built at this commit.
> h3. Suggested fix
> The patch below closes the gap for the one post-ZDU write feature that exists
> today. Versions 101 to 103 need no gate for replica consistency. 101 and 103
> only add a rejection to read requests. 102 is used on writes only by
> AbortMultipartUpload, where the receiving OM resolves the S3 volume and
> clears {{assumeS3Context}} before the request enters the Ratis log.
> h3. Patch
> [^CR-8-reject-if-none-match-etag-before-om-finalization.patch], against
> 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It applies to master at e03ba37c41d
> with {{git apply -3}}: an import that HDDS-17299 added to
> {{TestOMKeyCreateRequest}} shifts one context line, and the three way merge
> is clean. The two touched test classes and their FSO subclasses passed on the
> merged tree at master 3f1a4331ec2.
> The patch adds two {{CLUSTER_NEEDS_FINALIZATION}} pre-process request
> validators, {{OMKeyCreateRequest.disallowCreateKeyWithExcludedETag}} and
> {{S3MultipartUploadCompleteRequest.disallowCompleteMultiPartUploadWithExcludedETag}}.
> They follow the EC validators in the same classes and reject
> {{excludedETag}} with {{NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION}} while
> the OM is not finalized for 104. The check runs before the request becomes a
> Ratis entry, and only the leader can accept, so replicas never disagree on
> it. Finalized clusters are unaffected, and the config check stays.
> {{@DisallowedUntilOmVersion}} was not used because it would block the whole
> request type.
> The gate is only as correct as the leader's apparent version. An OM that
> wrongly starts finalized in a pre-finalized ring skips all
> {{CLUSTER_NEEDS_FINALIZATION}} validators, this one included, while it leads.
> Covered by {{testDisallowExcludedETagBeforeFinalization}} in the existing
> {{TestOMKeyCreateRequest}} and {{TestS3MultipartUploadCompleteRequest}}. Each
> test loads the real validator registry with a pre-finalized version manager
> that allows every version except {{S3_IF_NONE_MATCH_ETAG}}, and checks both
> the rejection and that a request without the field passes. Without the
> change, all four runs fail (including both FSO subclasses). With it, the key
> create and commit, MPU complete and validator suites pass (698 tests),
> {{TestS3SDK}} on a mini cluster passes (249 tests, including the
> If-None-Match ETag PutObject and CompleteMultipartUpload cases), and
> checkstyle is clean. With the patch, the reproduction fails because all three
> requests return {{NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION}}.
> Related, not duplicates: HDDS-17258 (new OM request versioning framework for
> ZDU), HDDS-16734 (conditional delete and conditional MPU complete are ungated
> on client and OM), HDDS-17140 (S3 Gateway does not pick up the new OM version
> after ZDU finalization).
> Found by code review of the zero downtime upgrade version gating in SCM, OM
> and the datanode write path, as part of the TLA+ verification effort under
> HDDS-15926, on commit 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked
> against HDDS issues and apache/ozone pull requests for duplicates before
> filing. The attached patch is a proposal for review. Generated with Specula
> (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
