[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17724890#comment-17724890
]
ASF GitHub Bot commented on HADOOP-18073:
-----------------------------------------
ahmarsuhail opened a new pull request, #5684:
URL: https://github.com/apache/hadoop/pull/5684
### Description of PR
JIRA: https://issues.apache.org/jira/browse/HADOOP-18073
This PR upgrades S3A to use AWS Java SDK V2. Changes made are detailed in
`aws_sdk_v2_changelog.md`.
Current known gaps/issues:
- No client side encryption. JIRA:
https://issues.apache.org/jira/browse/HADOOP-18708. The S3Encryption client
currently wraps some exceptions in `java.util.concurrent.CompletionException`
which impacts S3A's exception handling an translation. Ideally, the S3
encryption client should catch the completion exception, and instead throw the
underlying exception. This issue has been raised with the S3 encryption client
team.
- No SigV2 signing. SDK V2 does not support sigV2, however S3A supports
custom signers and so a sigV2 signer can still be configured. JIRA:
https://issues.apache.org/jira/browse/HADOOP-18747
- Regression in rename() performance. The transfer manager when used with
the Java async client currently does not implement multipart copy, which causes
a regression in rename() performance for files > multipart threshold. This has
been raised with the SDK team and a follow up PR to fix this issue will be
created once a patch becomes available.
### How was this patch tested?
Tested in `eu-west-1` by running `mvn -Dparallel-tests
-DtestsThreadCount=16` clean verify.
Known failures:
```
[ERROR] ITestS3ABlockOutputArray.testDiskBlockCreate:114 » IO File name
too long
[ERROR]
ITestS3ABlockOutputByteBuffer>ITestS3ABlockOutputArray.testDiskBlockCreate:114
» IO
[ERROR]
ITestS3ABlockOutputDisk>ITestS3ABlockOutputArray.testDiskBlockCreate:114 » IO
...
```
Jira created: https://issues.apache.org/jira/browse/HADOOP-18744
Other failures:
```
ITestS3AEndpointRegion.testWithoutRegionConfig:80 [Region is not
configured, region probe should have been made] expected:<[1]L> but was:<[0]L>
ITestS3SelectLandsat.testSelectSeekFullLandsat:419->AbstractS3SelectTest.seek:711
» AWSClientIO
```
Investigating the above and will open a PR to fix.
> Upgrade AWS SDK to v2
> ---------------------
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Task
> Components: auth, fs/s3
> Affects Versions: 3.3.1
> Reporter: xiaowei sun
> Assignee: Ahmar Suhail
> Priority: Major
> Labels: pull-request-available
> Attachments: Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
