[
https://issues.apache.org/jira/browse/HDDS-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793919#comment-17793919
]
Duong edited comment on HDDS-9843 at 12/21/23 10:18 PM:
--------------------------------------------------------
This is because each `BlockOutputStream` has its own `BufferPool` which by
default allocates 1MB chunk buffers for WriteChnuk regardless the actual size
of the blocks. It results in 1Mb of garbage every block writing. This prevents
any client (including S3) to reach thousands of ops because of the allocation
cost and GC load.
!buffer_allocation.png|width=688,height=252!
This is not small key friendly and artificially inefficient. The chunk buffers
need to be either reused cross writing operations or sized appropriately based
no the actual key/block size.
was (Author: JIRAUSER290990):
This is because each `BlockOutputStream` has its own `BufferPool` which by
default allocates 4MB chunk buffers for WriteChnuk regardless the actual size
of the blocks. It results in 4Mb of garbage every block writing. This prevents
any client (including S3) to reach thousands of ops because of the allocation
cost and GC load.
!buffer_allocation.png|width=688,height=252!
This is not small key friendly and artificially inefficient. The chunk buffers
need to be either reused cross writing operations or sized appropriately based
no the actual key/block size.
> Ozone S3 gateway memory utilization high on while writing small objects
> -----------------------------------------------------------------------
>
> Key: HDDS-9843
> URL: https://issues.apache.org/jira/browse/HDDS-9843
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client, S3, s3gateway
> Reporter: Duong
> Priority: Major
> Attachments: buffer_allocation.png,
> image-2023-12-05-16-35-37-893.png, image-2023-12-05-16-36-17-726.png,
> s3_heap_metric.png
>
>
> S3G memory footprint is too high when reading/writing small objects.
> Ran a test to write thousand of small (400 bytes) objects per seconds and
> memory utilization and gc is too high.
> !image-2023-12-05-16-36-17-726.png|width=527,height=489!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
