[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881122#comment-17881122
]
Ahnaf Khan edited comment on HADOOP-18073 at 9/11/24 8:54 PM:
--------------------------------------------------------------
[[email protected]] which version specifically did it change for? We are
trying to upgrade from hadoop 3.3.3 to hadoop 3.3.6 and it looks like the
defaults3clientfactory code still seems to be pulling from aws java sdk v1
instead of v2
([https://javadoc.io/doc/org.apache.hadoop/hadoop-aws/3.3.6/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.html]).
In other words, is Hadoop 3.3.6 compatible with aws java sdk v1 only or v2
only?
For context, we are trying to run our code on EMR 6.15, which uses hadoop 3.3.6
and both aws java sdk versions 1 and 2 as shown here:
(https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-6150-release.html),
but we are running into hadoop and aws related issues. We did not have any
issues with running on emr 6.14 btw, which uses hadoop 3.3.3 and aws java sdk
v1...
was (Author: JIRAUSER306988):
[[email protected]] which version specifically did it change for? For context,
we are trying to upgrade from hadoop 3.3.3 to hadoop 3.3.6 and it looks like
the defaults3clientfactory code still seems to be pulling from aws java sdk v1
instead of v2
([https://javadoc.io/doc/org.apache.hadoop/hadoop-aws/3.3.6/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.html]).
I guess in other words, is Hadoop 3.3.6 compatible with aws java sdk v1 only
or v2 only?
> S3A: Upgrade AWS SDK to V2
> --------------------------
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Task
> Components: auth, fs/s3
> Affects Versions: 3.3.1
> Reporter: xiaowei sun
> Assignee: Ahmar Suhail
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.7-aws
>
> Attachments: Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
