ChenSammi opened a new pull request, #7233: URL: https://github.com/apache/ozone/pull/7233
## What changes were proposed in this pull request? 1. make s3g object read/write buffer configurable 2. change ozone.s3g.client.buffer.size default value from 4KB to 1MB ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-11483 ## How was this patch tested? wrap get benchmark test 1. before the patch, ozone.client.bytes.per.checksum = 16KB, s3 get buffer size 8KB [root@vc0616 systest]# warp get --warp-client=vc0616.halxg.cloudera.com --host=localhost:9879 --tls --autoterm --insecure --access-key=$AWS_ACCESS_KEY_ID --secret-key=$AWS_SECRET_ACCESS_KEY --obj.size=20mb --concurrent=20 --bucket=bucket-20-24 --objects 10000 --duration=5m warp: Running benchmark on all clients... warp: Benchmark data written to "warp-remote-2024-09-20[053918]-cTQe.csv.zst" ---------------------------------------- Operation: PUT. Concurrency: 20 * Average: 358.85 MiB/s, 18.81 obj/s Throughput, split into 104 x 5s: * Fastest: 452.0MiB/s, 23.70 obj/s * 50% Median: 360.3MiB/s, 18.89 obj/s * Slowest: 267.6MiB/s, 14.03 obj/s ---------------------------------------- Operation: GET. Concurrency: 20 * Average: 224.64 MiB/s, 11.78 obj/s Throughput, split into 294 x 1s: * Fastest: 262.4MiB/s, 13.76 obj/s * 50% Median: 230.3MiB/s, 12.08 obj/s * Slowest: 49.0MiB/s, 2.57 obj/s warp: Cleanup done. 2. with the patch, ozone.client.bytes.per.checksum = 16KB, s3 get buffer size 1MB [root@vc0616 systest]# warp get --warp-client=vc0616.halxg.cloudera.com --host=localhost:9879 --tls --autoterm --insecure --access-key=$AWS_ACCESS_KEY_ID --secret-key=$AWS_SECRET_ACCESS_KEY --obj.size=20mb --concurrent=20 --bucket=bucket-24-10-56 --objects 10000 --duration=5m warp: Running benchmark on all clients... warp: Benchmark data written to "warp-remote-2024-09-23[201117]-qHUD.csv.zst" ---------------------------------------- Operation: PUT. Concurrency: 20 * Average: 349.16 MiB/s, 18.31 obj/s Throughput, split into 107 x 5s: * Fastest: 441.5MiB/s, 23.15 obj/s * 50% Median: 351.3MiB/s, 18.42 obj/s * Slowest: 259.9MiB/s, 13.63 obj/s ---------------------------------------- Operation: GET. Concurrency: 20 * Average: 1002.26 MiB/s, 52.55 obj/s Throughput, split into 297 x 1s: * Fastest: 1064.7MiB/s, 55.82 obj/s * 50% Median: 1041.5MiB/s, 54.60 obj/s * Slowest: 59.2MiB/s, 3.10 obj/s warp: Cleanup done. 3. ozone.client.bytes.per.checksum = 1MB, s3 get buffer size 8KB [root@vc0616 systest]# warp get --warp-client=vc0616.halxg.cloudera.com --host=localhost:9879 --tls --autoterm --insecure --access-key=$AWS_ACCESS_KEY_ID --secret-key=$AWS_SECRET_ACCESS_KEY --obj.size=20mb --concurrent=20 --bucket=bucket-19-23 --objects 10000 --duration=5m warp: Client 10.17.212.26:7761: Requested stage prepare start... warp: Benchmark data written to "warp-remote-2024-09-20[044056]-999v.csv.zst" ---------------------------------------- Operation: PUT. Concurrency: 20 * Average: 371.36 MiB/s, 19.47 obj/s Throughput, split into 100 x 5s: * Fastest: 433.5MiB/s, 22.73 obj/s * 50% Median: 378.1MiB/s, 19.82 obj/s * Slowest: 255.2MiB/s, 13.38 obj/s ---------------------------------------- Operation: GET. Concurrency: 20 * Average: 980.89 MiB/s, 51.43 obj/s Throughput, split into 298 x 1s: * Fastest: 1093.4MiB/s, 57.33 obj/s * 50% Median: 1040.7MiB/s, 54.56 obj/s * Slowest: 31.8MiB/s, 1.67 obj/s warp: Cleanup done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
