Kohei Sugihara created HDDS-7295:
------------------------------------
Summary: listKeys for large FSO-enabled bucket is timeout
Key: HDDS-7295
URL: https://issues.apache.org/jira/browse/HDDS-7295
Project: Apache Ozone
Issue Type: Bug
Components: S3
Affects Versions: 1.2.1
Environment: 1.3.0-SNAPSHOT
(https://github.com/apache/ozone/tree/843fac2fb646eecfc33103fdb16eaf77a66ca062)
Reporter: Kohei Sugihara
When we have a large bucket (e.g. # of keys > 1M; shows below) with
FSO-enabled, listKeys for top of bucket (ex. s3://bucket/) via S3G is timeout
even non-recursive request (also recursive request).
{{+ s3v/}}
{{ + bucket/}}
{{ + prefix1/}}
{{ - <1M keys>}}
{{ + prefix2/}}
{{{} - <1M keys>{}}}{{{}{}}}
{{ + prefix3/}}
{{ - <1M keys>}}
{{{} + ...{}}}{{{}{}}}{{{}{}}}{{{}{}}}{{{}{}}}
For S3G, I think BucketEndpoint.java [1] always queries all keys recursively
(both {{KeyIterator}} and {{{}KeyIteratorWithFSO{}}}).
[1]https://github.com/apache/ozone/blob/a8808d1c3781627c40e0ed25d0bb4ec1e74e3de2/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java#L134-L144
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
