[
https://issues.apache.org/jira/browse/HADOOP-18382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583069#comment-17583069
]
ASF GitHub Bot commented on HADOOP-18382:
-----------------------------------------
steveloughran commented on PR #4698:
URL: https://github.com/apache/hadoop/pull/4698#issuecomment-1222593771
Looks good. Be aware that in CDP we will not backport this without the logs
disabled, unless/until we have our own code all happy, which will involve a
move to the v2 sdk elsewhere (STS interaction).
### getClient()
the s3a FS class has a getClient() call too, which is only used internally
or in external code which is doing things to get at it (the forTesting() one
came in later, see).
The internal stuff could make that private and instead pass it in to
WriteOperationsHelper.
really that class should
1. have the Select operation moved to its own interface and used exclusively
by SelectBinding
2. have an interface of all the ops it is allowed to call on the s3a fs.
None of our subsidiary classes should have direct refs to the owning fs; it
just complicates our life too much. seemed a good idea at the time though...
what does that mean for that patch? nothing, except by moving the (split)
callbacks into two interfaces/impls. that getClient call could be made private
and migration would be easier.
## datatypes
Does the migration also mean that all the datatypes, s3Object, multipart
input stream etc are all going to go too?
> Upgrade AWS SDK to V2 - Prerequisites
> --------------------------------------
>
> Key: HADOOP-18382
> URL: https://issues.apache.org/jira/browse/HADOOP-18382
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Ahmar Suhail
> Assignee: Ahmar Suhail
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> We want to update the AWS SDK to V2, before we do this we should warn on
> things that will no longer supported. The following changes should be made:
>
> *
> [getAmazonS3Client()|https://github.com/apache/hadoop/blob/221eb2d68d5b52e4394fd36cb30d5ee9ffeea7f0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1174]
> - Warn that this method will be removed
> *
> [initCustomSigners()|https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/SignerManager.java#L65]
> - Warn that the interface is changing, any custom signers will need to be
> updated
> *
> [bindAWSClient|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L840]
> - If DT is enabled, warn that credential providers interface is changing,
> any custom cred providers used in binding classes will need to be updated
> *
> [buildAWSProviderList|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L618]
> - if any SDK V1 cred providers are in this list, warn that these will be
> removed
> *
> [S3ClientFactory|https://github.com/apache/hadoop/blob/221eb2d68d5b52e4394fd36cb30d5ee9ffeea7f0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ClientFactory.java]
> - Update javadocs to say this interface will be replaced by a V2 client
> factory, mark interface deprecated?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
