[
https://issues.apache.org/jira/browse/HDDS-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Kuan Hao updated HDDS-15739:
----------------------------------
Description:
ListObjectsV2 has two server-side bugs around the continuation token that make
S3 SDK clients fail:
1. Wrong response element name. The echoed request continuation token is
serialized as <continueToken> instead of AWS's <ContinuationToken>. botocore
therefore never populates response['ContinuationToken'], causing a KeyError in
clients that read it back.
2. Empty token is not treated as "no token". An empty ContinuationToken='' is
passed non-null into ContinueToken.decodeFromString(), which finds no separator
and throws InvalidArgument. AWS treats an empty continuation token as no token
— return all keys, IsTruncated=false, and echo '' back.
To align with AWS S3:
- Serialize the response continuation token as ContinuationToken.
- Treat a null/empty continuation-token query param as absent (list from the
start) rather than decoding it.
Failing Ceph s3-tests:
- test_bucket_listv2_continuationtoken
- test_bucket_listv2_continuationtoken_empty
- test_bucket_listv2_both_continuationtoken_startafter
Code pointers:
- hadoop-ozone/s3gateway/.../endpoint/ListObjectResponse.java (~L72–73:
@XmlElement(name = "continueToken") → ContinuationToken)
- hadoop-ozone/s3gateway/.../endpoint/BucketEndpoint.java (L108–131, 175 —
guard empty token before decode)
- hadoop-ozone/s3gateway/.../endpoint/ContinueToken.java (L84–88
decodeFromString)
Ref: https://ozone.s3.peterxcli.dev/
was:
The S3 Gateway accepts PutObject / CreateBucket when the Authorization header
is removed or empty, instead of rejecting with AccessDenied (403) /
InvalidRequest (400). Fix in AuthorizationFilter to reject absent/empty
credentials.
Failing tests: test_bucket_create_bad_authorization_none,
test_bucket_create_bad_authorization_empty,
test_object_create_bad_authorization_none,
test_object_create_bad_authorization_empty
Ref: [https://ozone.s3.peterxcli.dev/]
> Fix ListObjectsV2 continuation-token: wrong XML element name and empty-token
> handling
> -------------------------------------------------------------------------------------
>
> Key: HDDS-15739
> URL: https://issues.apache.org/jira/browse/HDDS-15739
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Huang Kuan Hao
> Assignee: Huang Kuan Hao
> Priority: Major
>
> ListObjectsV2 has two server-side bugs around the continuation token that
> make S3 SDK clients fail:
> 1. Wrong response element name. The echoed request continuation token is
> serialized as <continueToken> instead of AWS's <ContinuationToken>. botocore
> therefore never populates response['ContinuationToken'], causing a KeyError
> in clients that read it back.
> 2. Empty token is not treated as "no token". An empty ContinuationToken='' is
> passed non-null into ContinueToken.decodeFromString(), which finds no
> separator and throws InvalidArgument. AWS treats an empty continuation token
> as no token — return all keys, IsTruncated=false, and echo '' back.
> To align with AWS S3:
> - Serialize the response continuation token as ContinuationToken.
> - Treat a null/empty continuation-token query param as absent (list from the
> start) rather than decoding it.
> Failing Ceph s3-tests:
> - test_bucket_listv2_continuationtoken
> - test_bucket_listv2_continuationtoken_empty
> - test_bucket_listv2_both_continuationtoken_startafter
> Code pointers:
> - hadoop-ozone/s3gateway/.../endpoint/ListObjectResponse.java (~L72–73:
> @XmlElement(name = "continueToken") → ContinuationToken)
> - hadoop-ozone/s3gateway/.../endpoint/BucketEndpoint.java (L108–131, 175 —
> guard empty token before decode)
> - hadoop-ozone/s3gateway/.../endpoint/ContinueToken.java (L84–88
> decodeFromString)
> Ref: https://ozone.s3.peterxcli.dev/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
