[
https://issues.apache.org/jira/browse/HDDS-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duong updated HDDS-9843:
------------------------
Priority: Critical (was: Major)
> S3: Ozone S3 gateway memory utilization high on while writing small objects
> ---------------------------------------------------------------------------
>
> Key: HDDS-9843
> URL: https://issues.apache.org/jira/browse/HDDS-9843
> Project: Apache Ozone
> Issue Type: Improvement
> Components: S3, s3gateway
> Reporter: Duong
> Assignee: Duong
> Priority: Critical
> Attachments: image-2023-12-05-16-35-37-893.png,
> image-2023-12-05-16-36-17-726.png, s3_heap_metric.png
>
>
> S3G memory footprint is too high when reading/writing small objects.
> Ran a test to write thousand of small (400 bytes) objects per seconds and
> memory utilization and gc is too high.
> !image-2023-12-05-16-36-17-726.png|width=527,height=489!
>
> Root of probkem is S3G uses the same buffer size to copy streams (when from
> client connection to Ozone stream and vice-versus).
> For writes (put endpoint), the buffer size is 8kb and is not efficient when
> copying small objects because that unnecessarily create work for GC.
> For reads (get endpoint), the default buffer size is 8kb for full read and
> 4mb (?!!!) for range read.
> For small objects read/writes, a single S3G instance can handle thousands of
> requests. Stream buffer need to be dynamically calculated based on object
> size.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
