[
https://issues.apache.org/jira/browse/HADOOP-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773741#comment-17773741
]
ASF GitHub Bot commented on HADOOP-18708:
-----------------------------------------
ahmarsuhail commented on code in PR #6164:
URL: https://github.com/apache/hadoop/pull/6164#discussion_r1352852582
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java:
##########
@@ -109,11 +144,17 @@ public S3AsyncClient createS3AsyncClient(
.thresholdInBytes(parameters.getMultiPartThreshold())
.build();
- return configureClientBuilder(S3AsyncClient.builder(), parameters, conf,
bucket)
- .httpClientBuilder(httpClientBuilder)
- .multipartConfiguration(multipartConfiguration)
- .multipartEnabled(parameters.isMultipartCopy())
- .build();
+ S3AsyncClientBuilder s3AsyncClientBuilder =
+ configureClientBuilder(S3AsyncClient.builder(), parameters, conf,
bucket).httpClientBuilder(
+ httpClientBuilder);
+
+ if (!parameters.isClientSideEncryptionEnabled()) {
Review Comment:
We need to do this because currently if you try to do a ranged GET with
multipart enabled, it fails. This is intentional, as the SDK team is working on
adding "automatic multipart download" for a GET to the async client, and this
capability is disabled till then.
When we build the S3EncryptionClient we configure and pass in a
S3AsyncClient. Under the hood, S3EC uses the S3Async client for all it's
encryption related operations (eg: GET). And so if we pass in a client with
multipartEnabled, all GETs will start failing.
The impact of this that if you're using CSE, there's no multipart mode, so
you copies become slower again.
A workaround can be to create a new async client with multipart disabled. So
we have two async clients, one with MPU enable, one with disabled. and share
underlying HTTP client b/w the two async clients, so it shouldn't have too much
of an impact.
I'll make that change if required, once I have more clarity from SDK team on
when ranged GETs become available in multipart mode.
> AWS SDK V2 - Implement CSE
> --------------------------
>
> Key: HADOOP-18708
> URL: https://issues.apache.org/jira/browse/HADOOP-18708
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Ahmar Suhail
> Priority: Major
> Labels: pull-request-available
>
> S3 Encryption client for SDK V2 is now available, so add client side
> encryption back in.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
