Siyao Meng created HDDS-17420:
---------------------------------
Summary: S3 writes with an If-None-Match ETag return an unmapped
HTTP 500, retried by SDKs, while OM does not support S3_IF_NONE_MATCH_ETAG
Key: HDDS-17420
URL: https://issues.apache.org/jira/browse/HDDS-17420
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-10-return-not-implemented-for-if-none-match-etag-before-om-supports-it.patch,
TestBugCR10PreFinalizedIfNoneMatchS3Error.java
h3. Mechanism
When the OM version cached by {{RpcClient}} is below
{{OzoneManagerVersion.S3_IF_NONE_MATCH_ETAG}} (104),
{{RpcClient.checkIfNoneMatchETagSupport}} refuses {{createKeyIfNoneMatch}},
{{createStreamKeyIfNoneMatch}} and {{completeMultipartUpload}} with an excluded
ETag. It throws a plain {{IOException}}. S3 Gateway sends the condition from
{{ObjectEndpoint.openKeyForPut}},
{{ObjectEndpointStreaming.openStreamKeyForPut}} and
{{ObjectEndpoint.completeMultipartUpload}}, and these paths translate only
{{OMException}}. S3 Gateway has no exception mapper for {{IOException}}, so
Jersey passes it to Jetty, which answers HTTP 500 without an S3 error document.
Before HDDS-16622, S3 Gateway rejected a non {{*}} If-None-Match with 400
InvalidRequest. The ZDU design says S3 Gateway returns clean errors when a
feature is used before finalization.
h3. Trigger
# OM reports a version below 104 to clients, for example because it runs the
current software but is not finalized, as during an upgrade before finalization.
# An S3 client sends PutObject, CopyObject or CompleteMultipartUpload with
{{If-None-Match: "<etag>"}}. {{If-None-Match: *}} is not affected.
Because of HDDS-17140, S3 Gateway keeps the OM version it read first, so this
continues after finalization until S3 Gateway restarts (from code, not run).
h3. Impact
* The client gets HTTP 500 with no S3 error code. PutObject and CopyObject
return an empty body, and CompleteMultipartUpload returns a Jetty HTML page.
* The AWS SDK v2 treats it as retryable and sends each request 4 times with its
default retry policy.
* S3 Gateway logs a WARN with a stack trace from
{{HttpChannel.handleException}} on every attempt.
The write is refused in every case, as intended. No data is affected.
h3. Reproduction
PASS on unmodified source. [^TestBugCR10PreFinalizedIfNoneMatchS3Error.java]
({{hadoop-ozone/integration-test-s3}}) starts a mini cluster whose OM begins at
apparent version ZDU (test only
{{OMStorage.TESTING_INIT_APPARENT_VERSION_KEY}}), plus one S3 Gateway, and
sends requests with the AWS SDK v2. A plain PutObject succeeds. With an
If-None-Match ETag, PutObject, CopyObject and CompleteMultipartUpload each
report {{HTTP 500, S3 error code null, SDK attempts 4}}, and S3 Gateway counts
4 failed requests for each. A passing test means the defect is present. A
variant with datastream enabled ({{hdds.container.ratis.datastream.enabled}}
and {{ozone.fs.datastream.enabled}} set to true,
{{ozone.fs.datastream.auto.threshold}} 0MB), which sends PutObject and
CopyObject through {{ObjectEndpointStreaming}}, gave the same result.
h3. Patch
[^CR-10-return-not-implemented-for-if-none-match-etag-before-om-supports-it.patch],
against 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It also applies to master at
e03ba37c41d.
{{RpcClient.checkIfNoneMatchETagSupport}} now throws {{OMException}} with
{{NOT_SUPPORTED_OPERATION}} and the same message. {{OMException}} extends
{{IOException}}, so no signature changes. All S3 Gateway paths above already
translate it through {{S3ErrorTable}}, which maps {{NOT_SUPPORTED_OPERATION}}
to 501 NotImplemented. The result is an S3 error document that the SDK does not
retry. {{NOT_SUPPORTED_OPERATION}} is used instead of
{{NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION}} because the client cannot tell a
pre-finalized OM from older OM software. Finalized clusters are unaffected.
Other version checks in {{RpcClient}} also throw a plain {{IOException}}, but
they only trigger when OM runs older software than S3 Gateway, which the
upgrade order rules out. They are not changed here.
Covered by {{TestRpcClient.testIfNoneMatchETagReachesOMOrRejectsOldVersion}},
which now also asserts the exception type and result code for every OM version
below 104. Without the change, 21 of its 22 cases fail. With it:
* the client unit tests (339) and S3 Gateway unit tests (1689) pass;
* {{TestS3SDK}} on a mini cluster passes (249 tests, including the
If-None-Match ETag cases on a finalized cluster);
* checkstyle is clean.
With the patch, the reproduction and its datastream variant fail because each
request gets {{HTTP 501, S3 error code NotImplemented, SDK attempts 1}}.
Related, not duplicates: HDDS-16622 (added the check), HDDS-17140 (S3 Gateway
does not pick up the new OM version after ZDU finalization), HDDS-16994 (wrong
OM version check for atomic key creation).
Found by code review of the zero downtime upgrade version gating in SCM, OM and
the datanode write path, as part of the TLA+ verification effort under
HDDS-15926, on commit 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked against
HDDS issues and apache/ozone pull requests for duplicates before filing. The
attached patch is a proposal for review. Generated with Specula (Claude Opus
5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
