[
https://issues.apache.org/jira/browse/HDDS-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duong updated HDDS-9843:
------------------------
Description:
S3G memory footprint is too high when reading/writing small objects.
Ran a test to write thousand of small (400 bytes) objects per seconds and
memory utilization and gc is too high.
!image-2023-12-05-16-36-17-726.png|width=527,height=489!
was:
S3G memory footprint is too high when reading/writing small objects.
Ran a test to write thousand of small (400 bytes) objects per seconds and
memory utilization and gc is too high.
!image-2023-12-05-16-36-17-726.png|width=527,height=489!
Root of probkem is S3G uses the same buffer size to copy streams (when from
client connection to Ozone stream and vice-versus).
For writes (put endpoint), the buffer size is 8kb and is not efficient when
copying small objects because that unnecessarily create work for GC.
For reads (get endpoint), the default buffer size is 8kb for full read and 4mb
(?!!!) for range read.
For small objects read/writes, a single S3G instance can handle thousands of
requests. Stream buffer need to be dynamically calculated based on object size.
> S3: Ozone S3 gateway memory utilization high on while writing small objects
> ---------------------------------------------------------------------------
>
> Key: HDDS-9843
> URL: https://issues.apache.org/jira/browse/HDDS-9843
> Project: Apache Ozone
> Issue Type: Improvement
> Components: S3, s3gateway
> Reporter: Duong
> Assignee: Duong
> Priority: Critical
> Attachments: image-2023-12-05-16-35-37-893.png,
> image-2023-12-05-16-36-17-726.png, s3_heap_metric.png
>
>
> S3G memory footprint is too high when reading/writing small objects.
> Ran a test to write thousand of small (400 bytes) objects per seconds and
> memory utilization and gc is too high.
> !image-2023-12-05-16-36-17-726.png|width=527,height=489!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
