[
https://issues.apache.org/jira/browse/FLINK-39811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-39811:
-----------------------------------
Release Note:
Fixed s5cmd effective concurrency that was capped by static part size. Now we
are using fileSize/concurrency to dynamically calculate downloads part size for
s5cmd.
Turn {{s3.s5cmd.adjust-part-size: false}} to restore the previous behavior (= a
static 50MiB part size)
was:
Fixed s5cmd effective concurrency is capped by static part size, using
fileSize/concurrency for s5cmd downloads part size
Turn {{s3.s5cmd.adjust-part-size: false}} to restore the previous behavior (= a
static 50MiB part size)
> S5cmd effective concurrency capped by static part size for small downloads
> --------------------------------------------------------------------------
>
> Key: FLINK-39811
> URL: https://issues.apache.org/jira/browse/FLINK-39811
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Reporter: Efrat Levitan
> Assignee: Efrat Levitan
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.4.0
>
>
> Unlike the upload path (recoverableWriter) which sets
> {{S3_MULTIPART_MIN_PART_SIZE}} of [5
> MiB|https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java#L93],
> we don't set any part size for s5cmd so downloads part size fall back to its
> default of
> [50MiB|https://github.com/peak/s5cmd/blob/54d6a8a955688f07e5acc40d61f9c42ceac6c33b/command/cp.go#L30].
> As a result, the effective concurrency is capped to batchSize / 50MiB,
> underutilizing the worker pool for downloaded batchSizes below
> (50*concurrency MiB)
> - s3 sdk source
> [ref|https://github.com/aws/aws-sdk-go/blob/070853e88d22854d2355c2543d0958a5f76ad407/service/s3/s3manager/download.go#L329-L338]
> N(=concurrency) channels are initialized but only (batchsize/partsize) of
> them are assigned a range to read.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
