ChenSammi commented on PR #7233: URL: https://github.com/apache/ozone/pull/7233#issuecomment-2370780389
> @ChenSammi Hi, based on your testing conclusions, we understand that changing the parameter ozone.s3g.client.buffer.size to 1M significantly improves performance. However, does this size have a relationship with the file sizes I need to read? Should I set this parameter to 1M, 2M, or even larger? How can I find the optimal value? Is there any guideline for setting this parameter that I can refer to? @weimingdiit , there is a previous patch #6331 which changes the ozone.client.bytes.per.checksum default from 1MB to 16KB. After that, it's found that the s3g read performance is degraded by 4X. This patch tries to fix the problem by increase the buffer size in s3g. If your cluster is still using 1MB ozone.client.bytes.per.checksum configuration, then you don't need to bother to apply this patch. Either 1MB ozone.client.bytes.per.checksum or 1MB s3g read buffer size will have the same read performance. For s3g, theoretically, bigger buffer size will have better performance. But the margin performance gain decreases with the buffer size increased. Here is some data of same "warp get" test, - 1MB s3g buffer, the result is "Average: 1002.26 MiB/s, 52.55 obj/s" - 128KB s3g buffer, the result is "Average: 757.84 MiB/s, 39.73 obj/s" - 64KB s3g buffer, the result is "Average: 550.13 MiB/s, 28.84 obj/s" - 16KB, the result is "Average: 224.64 MiB/s, 11.78 obj/s" If you want to find the optimal value, you can try test the different s3g buffer size values, for example, 2MB, 4MB etc. But bigger buffer size will cost more heap. Make sure there is enough heap configured for s3g. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
