[
https://issues.apache.org/jira/browse/HDDS-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Kuan Hao updated HDDS-17028:
----------------------------------
Description:
ListObjects V1 and ListObjectVersions can return the same CommonPrefixes entry
on consecutive pages when a page ends inside a prefix.
Reproduced on master at 10b5015f64d7 after HDDS-16739 / PR #11426 was merged.
h3. Reproduction
Create keys {{dir:a}}, {{dir:b}} and {{file}}. With the default S3 Gateway
configuration:
{noformat}
GET /bucket?versions&delimiter=:&max-keys=1
CommonPrefixes: dir:
NextKeyMarker: dir:a
NextVersionIdMarker: null
GET
/bucket?versions&delimiter=:&max-keys=1&key-marker=dir:a&version-id-marker=null
Actual: CommonPrefixes: dir:
Expected: Version: file, with no CommonPrefixes
{noformat}
The marker advances, so listing eventually completes, but paginators receive
duplicate prefixes and make unnecessary requests. The same filtering gap is
reproducible with keys {{dir/a}}, {{dir/b}}, {{file}}, {{delimiter=/}}, shallow
listing disabled, and {{key-marker=dir/a}}.
h3. Expected behavior
[The ListObjectVersions
API|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html]
excludes CommonPrefixes that are not lexicographically greater than
key-marker. Filter these prefixes before counting them against max-keys, and
cover a paginated listing that ends on a CommonPrefixes entry.
h3. Context
* Follow-up to [HDDS-16739 / PR
#11426|https://github.com/apache/ozone/pull/11426].
* ListObjects V1 shares the same loop and repeats the prefix the same way when
resuming with {{marker}} (e.g. {{delimiter=:}}, {{max-keys=1}},
{{marker=dir:a}}); the fix covers both.
* [HDDS-898|https://issues.apache.org/jira/browse/HDDS-898] fixed a related
continuation-token problem. ListObjectVersions resumes with key-marker and
version-id-marker instead.
* Fresh local JDK 21 run of TestBucketList: 41 tests, 2 failures, 0 errors. The
39 existing tests passed; both added probes failed because the response still
contained the earlier CommonPrefixes entry. These were endpoint unit tests, not
requests against AWS S3.
Generated-by: Codex (GPT-6)
was:
ListObjectVersions can return the same CommonPrefixes entry on consecutive
pages when a page ends inside a prefix.
Reproduced on master at 10b5015f64d7 after HDDS-16739 / PR #11426 was merged.
h3. Reproduction
Create keys {{dir:a}}, {{dir:b}} and {{file}}. With the default S3 Gateway
configuration:
{noformat}
GET /bucket?versions&delimiter=:&max-keys=1
CommonPrefixes: dir:
NextKeyMarker: dir:a
NextVersionIdMarker: null
GET
/bucket?versions&delimiter=:&max-keys=1&key-marker=dir:a&version-id-marker=null
Actual: CommonPrefixes: dir:
Expected: Version: file, with no CommonPrefixes
{noformat}
The marker advances, so listing eventually completes, but paginators receive
duplicate prefixes and make unnecessary requests. The same filtering gap is
reproducible with keys {{dir/a}}, {{dir/b}}, {{file}}, {{delimiter=/}}, shallow
listing disabled, and {{key-marker=dir/a}}.
h3. Expected behavior
[The ListObjectVersions
API|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html]
excludes CommonPrefixes that are not lexicographically greater than
key-marker. Filter these prefixes before counting them against max-keys, and
cover a paginated listing that ends on a CommonPrefixes entry.
h3. Context
* Follow-up to [HDDS-16739 / PR
#11426|https://github.com/apache/ozone/pull/11426].
* The shared ListObjects V1 loop also has this behavior; this report focuses on
ListObjectVersions.
* [HDDS-898|https://issues.apache.org/jira/browse/HDDS-898] fixed a related
continuation-token problem. ListObjectVersions resumes with key-marker and
version-id-marker instead.
* Fresh local JDK 21 run of TestBucketList: 41 tests, 2 failures, 0 errors. The
39 existing tests passed; both added probes failed because the response still
contained the earlier CommonPrefixes entry. These were endpoint unit tests, not
requests against AWS S3.
Generated-by: Codex (GPT-6)
Summary: ListObjects V1 and ListObjectVersions repeat CommonPrefixes
across pages (was: ListObjectVersions repeats CommonPrefixes across pages)
> ListObjects V1 and ListObjectVersions repeat CommonPrefixes across pages
> ------------------------------------------------------------------------
>
> Key: HDDS-17028
> URL: https://issues.apache.org/jira/browse/HDDS-17028
> Project: Apache Ozone
> Issue Type: Bug
> Components: s3gateway
> Reporter: Huang Kuan Hao
> Assignee: Huang Kuan Hao
> Priority: Major
> Labels: pull-request-available
>
> ListObjects V1 and ListObjectVersions can return the same CommonPrefixes
> entry on consecutive pages when a page ends inside a prefix.
> Reproduced on master at 10b5015f64d7 after HDDS-16739 / PR #11426 was merged.
> h3. Reproduction
> Create keys {{dir:a}}, {{dir:b}} and {{file}}. With the default S3 Gateway
> configuration:
> {noformat}
> GET /bucket?versions&delimiter=:&max-keys=1
> CommonPrefixes: dir:
> NextKeyMarker: dir:a
> NextVersionIdMarker: null
> GET
> /bucket?versions&delimiter=:&max-keys=1&key-marker=dir:a&version-id-marker=null
> Actual: CommonPrefixes: dir:
> Expected: Version: file, with no CommonPrefixes
> {noformat}
> The marker advances, so listing eventually completes, but paginators receive
> duplicate prefixes and make unnecessary requests. The same filtering gap is
> reproducible with keys {{dir/a}}, {{dir/b}}, {{file}}, {{delimiter=/}},
> shallow listing disabled, and {{key-marker=dir/a}}.
> h3. Expected behavior
> [The ListObjectVersions
> API|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html]
> excludes CommonPrefixes that are not lexicographically greater than
> key-marker. Filter these prefixes before counting them against max-keys, and
> cover a paginated listing that ends on a CommonPrefixes entry.
> h3. Context
> * Follow-up to [HDDS-16739 / PR
> #11426|https://github.com/apache/ozone/pull/11426].
> * ListObjects V1 shares the same loop and repeats the prefix the same way
> when resuming with {{marker}} (e.g. {{delimiter=:}}, {{max-keys=1}},
> {{marker=dir:a}}); the fix covers both.
> * [HDDS-898|https://issues.apache.org/jira/browse/HDDS-898] fixed a related
> continuation-token problem. ListObjectVersions resumes with key-marker and
> version-id-marker instead.
> * Fresh local JDK 21 run of TestBucketList: 41 tests, 2 failures, 0 errors.
> The 39 existing tests passed; both added probes failed because the response
> still contained the earlier CommonPrefixes entry. These were endpoint unit
> tests, not requests against AWS S3.
> Generated-by: Codex (GPT-6)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
