ahmarsuhail opened a new pull request, #5684: URL: https://github.com/apache/hadoop/pull/5684
### Description of PR JIRA: https://issues.apache.org/jira/browse/HADOOP-18073 This PR upgrades S3A to use AWS Java SDK V2. Changes made are detailed in `aws_sdk_v2_changelog.md`. Current known gaps/issues: - No client side encryption. JIRA: https://issues.apache.org/jira/browse/HADOOP-18708. The S3Encryption client currently wraps some exceptions in `java.util.concurrent.CompletionException` which impacts S3A's exception handling an translation. Ideally, the S3 encryption client should catch the completion exception, and instead throw the underlying exception. This issue has been raised with the S3 encryption client team. - No SigV2 signing. SDK V2 does not support sigV2, however S3A supports custom signers and so a sigV2 signer can still be configured. JIRA: https://issues.apache.org/jira/browse/HADOOP-18747 - Regression in rename() performance. The transfer manager when used with the Java async client currently does not implement multipart copy, which causes a regression in rename() performance for files > multipart threshold. This has been raised with the SDK team and a follow up PR to fix this issue will be created once a patch becomes available. ### How was this patch tested? Tested in `eu-west-1` by running `mvn -Dparallel-tests -DtestsThreadCount=16` clean verify. Known failures: ``` [ERROR] ITestS3ABlockOutputArray.testDiskBlockCreate:114 » IO File name too long [ERROR] ITestS3ABlockOutputByteBuffer>ITestS3ABlockOutputArray.testDiskBlockCreate:114 » IO [ERROR] ITestS3ABlockOutputDisk>ITestS3ABlockOutputArray.testDiskBlockCreate:114 » IO ... ``` Jira created: https://issues.apache.org/jira/browse/HADOOP-18744 Other failures: ``` ITestS3AEndpointRegion.testWithoutRegionConfig:80 [Region is not configured, region probe should have been made] expected:<[1]L> but was:<[0]L> ITestS3SelectLandsat.testSelectSeekFullLandsat:419->AbstractS3SelectTest.seek:711 » AWSClientIO ``` Investigating the above and will open a PR to fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
