[
https://issues.apache.org/jira/browse/FLINK-39811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-39811:
-----------------------------------
Labels: pull-request-available (was: )
> S5cmd effective concurrency capped by static part size for small downloads
> --------------------------------------------------------------------------
>
> Key: FLINK-39811
> URL: https://issues.apache.org/jira/browse/FLINK-39811
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Reporter: Efrat Levitan
> Priority: Major
> Labels: pull-request-available
>
> Unlike the upload path (recoverableWriter) which sets
> {{S3_MULTIPART_MIN_PART_SIZE}} of [5
> MiB|https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java#L93],
> we don't set any part size for s5cmd so downloads part size fall back to its
> default of
> [50MiB|https://github.com/peak/s5cmd/blob/54d6a8a955688f07e5acc40d61f9c42ceac6c33b/command/cp.go#L30].
> As a result, the effective concurrency is capped to batchSize / 50MiB,
> underutilizing the worker pool for downloaded batchSizes below
> (50*concurrency MiB)
> - s3 sdk source
> [ref|https://github.com/aws/aws-sdk-go/blob/070853e88d22854d2355c2543d0958a5f76ad407/service/s3/s3manager/download.go#L329-L338]
> N(=concurrency) channels are initialized but only (batchsize/partsize) of
> them are assigned a range to read.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
