[
https://issues.apache.org/jira/browse/HADOOP-18853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17755901#comment-17755901
]
ASF GitHub Bot commented on HADOOP-18853:
-----------------------------------------
ahmarsuhail opened a new pull request, #5960:
URL: https://github.com/apache/hadoop/pull/5960
### Description of PR
* Upgrades SDK Version so it has the new Java Async client with MPU support.
* Configures java async client with MPU threshold and part size
* Removes CRT dependency
Since Multipart operations are in the java async client now, and not the TM,
we could consider removing the TM in a follow up PR. We'd lose out on the
transfer listener..but we don't really use any of the other additional
functionality the TM provides us.
### How was this patch tested?
Testing in progress in eu-west-1 with `mvn -Dparallel-tests
-DtestsThreadCount=16 clean verify`
Also ran checked output of
ITestS3HugeFileArrayBlocks.test_100_renameHugeFile(), time taken to rename a
256MB file is at par with V1, around ~1s from my m4.2xlarge EC2.
> AWS SDK V2 - Integrate new transfer manager
> -------------------------------------------
>
> Key: HADOOP-18853
> URL: https://issues.apache.org/jira/browse/HADOOP-18853
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Ahmar Suhail
> Priority: Major
>
> With 2.20.121, the TM has MPU functionality. Upgrading to to this version
> will also solve the issue with needing to include the CRT dependency.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
