Siyao Meng created HDDS-17419:
---------------------------------
Summary: OM accepts If-None-Match ETag on CreateKey and
CompleteMultipartUpload before it is finalized for S3_IF_NONE_MATCH_ETAG
Key: HDDS-17419
URL: https://issues.apache.org/jira/browse/HDDS-17419
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-8-reject-if-none-match-etag-before-om-finalization.patch,
TestBugCR8PreFinalizedIfNoneMatch.java
h3. Mechanism
{{OzoneManagerVersion.S3_IF_NONE_MATCH_ETAG}} (104) is gated in the OM only by
{{OzoneManager.checkFeatureEnabled}}, called from
{{OMKeyCreateRequest.preExecute}} and
{{S3MultipartUploadCompleteRequest.preExecute}} when {{KeyArgs.excludedETag}}
is set. That method reads {{ozone.om.features.disabled}} and never the apparent
version. No request in main code uses {{@DisallowedUntilOmVersion}}, and no
request validator checks the field. So an OM that is not finalized for 104
evaluates the condition in {{validateAndUpdateCache}} like a finalized one. The
ZDU design requires new features to be unavailable until the upgrade is
finalized, with an error for a client that tries one.
h3. Trigger
# The OMs run the current software and are pre-finalized (apparent version ZDU
or a pre-ZDU layout version), as during an upgrade before finalization.
# A client that does not check the OM version first sends CreateKey or
CompleteMultiPartUpload with {{excludedETag}} over the OM protocol. This could
be a non-Java client, or {{OzoneManagerProtocol}} used directly.
{{RpcClient}}, and S3 Gateway through it, refuses on its cached OM version, so
the stock clients do not send the field in this state.
h3. Impact
The OM applies If-None-Match before finalization: {{KEY_ALREADY_EXISTS}} for a
matching ETag, or an open key guarded by {{expectedDataGeneration}} otherwise.
At this commit every ZDU capable OM runs the same software, so all replicas
apply the request the same way, and the guard field predates ZDU, so nothing is
persisted in a new format. I found no replica divergence or data loss reachable
today. The gap matters for the pattern: a later OM version gated the same way
would, during a rolling upgrade or downgrade between ZDU releases, let a
replica on the older software ignore the field and apply the request
differently (argued, not run).
h3. Reproduction
PASS on unmodified source. [^TestBugCR8PreFinalizedIfNoneMatch.java]
({{hadoop-ozone/ozone-manager}}) starts one real OM with apparent version ZDU
(test only {{OMStorage.TESTING_INIT_APPARENT_VERSION_KEY}}, software version
104) and sends requests through {{OzoneManagerProtocol}}. As a control,
{{RpcClient.createKeyIfNoneMatch}} refuses. Results:
* CreateKey with the current ETag as If-None-Match returns
{{KEY_ALREADY_EXISTS}}.
* CreateKey with another ETag succeeds, and the open key carries
{{expectedDataGeneration}}.
* CompleteMultiPartUpload with the current ETag returns {{KEY_ALREADY_EXISTS}}.
A passing test means the defect is present. An earlier run of the same scenario
on a three OM mini cluster during the analysis showed all three replicas
applying it; I did not repeat that run. No mixed software ring was run, as none
can be built at this commit.
h3. Suggested fix
The patch below closes the gap for the one post-ZDU write feature that exists
today. Versions 101 to 103 need no gate for replica consistency. 101 and 103
only add a rejection to read requests. 102 is used on writes only by
AbortMultipartUpload, where the receiving OM resolves the S3 volume and clears
{{assumeS3Context}} before the request enters the Ratis log.
h3. Patch
[^CR-8-reject-if-none-match-etag-before-om-finalization.patch], against
5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It applies to master at e03ba37c41d
with {{git apply -3}}: an import that HDDS-17299 added to
{{TestOMKeyCreateRequest}} shifts one context line, and the three way merge is
clean. The two touched test classes and their FSO subclasses passed on the
merged tree at master 3f1a4331ec2.
The patch adds two {{CLUSTER_NEEDS_FINALIZATION}} pre-process request
validators, {{OMKeyCreateRequest.disallowCreateKeyWithExcludedETag}} and
{{S3MultipartUploadCompleteRequest.disallowCompleteMultiPartUploadWithExcludedETag}}.
They follow the EC validators in the same classes and reject {{excludedETag}}
with {{NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION}} while the OM is not
finalized for 104. The check runs before the request becomes a Ratis entry, and
only the leader can accept, so replicas never disagree on it. Finalized
clusters are unaffected, and the config check stays.
{{@DisallowedUntilOmVersion}} was not used because it would block the whole
request type.
The gate is only as correct as the leader's apparent version. An OM that
wrongly starts finalized in a pre-finalized ring skips all
{{CLUSTER_NEEDS_FINALIZATION}} validators, this one included, while it leads.
Covered by {{testDisallowExcludedETagBeforeFinalization}} in the existing
{{TestOMKeyCreateRequest}} and {{TestS3MultipartUploadCompleteRequest}}. Each
test loads the real validator registry with a pre-finalized version manager
that allows every version except {{S3_IF_NONE_MATCH_ETAG}}, and checks both the
rejection and that a request without the field passes. Without the change, all
four runs fail (including both FSO subclasses). With it, the key create and
commit, MPU complete and validator suites pass (698 tests), {{TestS3SDK}} on a
mini cluster passes (249 tests, including the If-None-Match ETag PutObject and
CompleteMultipartUpload cases), and checkstyle is clean. With the patch, the
reproduction fails because all three requests return
{{NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION}}.
Related, not duplicates: HDDS-17258 (new OM request versioning framework for
ZDU), HDDS-16734 (conditional delete and conditional MPU complete are ungated
on client and OM), HDDS-17140 (S3 Gateway does not pick up the new OM version
after ZDU finalization).
Found by code review of the zero downtime upgrade version gating in SCM, OM and
the datanode write path, as part of the TLA+ verification effort under
HDDS-15926, on commit 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked against
HDDS issues and apache/ozone pull requests for duplicates before filing. The
attached patch is a proposal for review. Generated with Specula (Claude Opus
5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
