[
https://issues.apache.org/jira/browse/HADOOP-18382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584748#comment-17584748
]
ASF GitHub Bot commented on HADOOP-18382:
-----------------------------------------
steveloughran commented on code in PR #4698:
URL: https://github.com/apache/hadoop/pull/4698#discussion_r954774809
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##########
@@ -1619,6 +1622,25 @@ public <T> CompletableFuture<T> submit(final
CallableRaisingIOE<T> operation) {
}
}
+ /**
+ * Callbacks for WriteOperationHelper.
+ */
+ private final class WriteOperationHelperCallbacksImpl
Review Comment:
i like this. yes, it makes for a bigger patch, but it was needed anyway
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/WriteOperationHelper.java:
##########
@@ -157,7 +162,8 @@ protected WriteOperationHelper(S3AFileSystem owner,
Configuration conf,
S3AStatisticsContext statisticsContext,
final AuditSpanSource auditSpanSource,
- final AuditSpan auditSpan) {
+ final AuditSpan auditSpan,
+ final WriteOperationHelperCallbacks writeOperationHelperCallbacks) {
Review Comment:
nit: needs a javadoc entry
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3AFileSystem.java:
##########
@@ -176,7 +177,8 @@ public void initialize(URI name, Configuration originalConf)
conf,
new EmptyS3AStatisticsContext(),
noopAuditor(conf),
- AuditTestSupport.NOOP_SPAN);
+ AuditTestSupport.NOOP_SPAN,
+ new MinimalWriteOperationHelperCallbacks());
Review Comment:
this is why i like the callback interface/impl design
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md:
##########
@@ -1989,3 +1990,51 @@ com.amazonaws.SdkClientException: Unable to execute HTTP
request:
When this happens, try to set `fs.s3a.connection.request.timeout` to a larger
value or disable it
completely by setting it to `0`.
+
+## <a name="upgrade_warnings"></a> SDK Upgrade Warnings
+
+S3A will soon be upgraded to [AWS's Java SDK
V2](https://github.com/aws/aws-sdk-java-v2).
+For more information on the upgrade and what's changing, see
+[Upcoming upgrade to AWS Java SDK V2](./aws_sdk_upgrade.html).
+
+S3A logs the following warnings for things that will be changing in the
upgrade. To disable these
+logs, comment out `log4j.logger.org.apache.hadoop.fs.s3a.SDKV2Upgrade` in
log4j.properties.
+
+### <a name="ProviderReferenced"></a> `Directly referencing AWS SDK V1
credential provider`
+
+This will be logged when an AWS credential provider is referenced directly in
+`fs.s3a.aws.credentials.provider`.
+For example, `com.amazonaws.auth.AWSSessionCredentialsProvider`
+
+To stop this warning, remove any AWS credential providers from
`fs.s3a.aws.credentials.provider`.
+Instead, use S3A's credential providers.
+
+### <a name="ClientRequested"></a> `getAmazonS3ClientForTesting() will be
removed`
+
+This will be logged when `getAmazonS3ClientForTesting()` is called to get the
S3 Client. With V2,
+the S3 client will change from type `com.amazonaws.services.s3.AmazonS3` to
+`software.amazon.awssdk.services.s3.S3Client`, and so this method will be
removed.
+
+### <a name="DelegationTokenProvider"></a>
+### `Custom credential providers used in delegation tokens binding classes
will need to be updated`
+
+This will be logged when delegation tokens are used.
+Delegation tokens allow the use of custom binding classes which can implement
custom credential
+providers.
+These credential providers will currently be implementing
+`com.amazonaws.auth.AWSCredentialsProvider` and will need to be updated to
implement
+`software.amazon.awssdk.auth.credentials.AwsCredentialsProvider`.
+
+### <a name="CustomSignerUsed"></a>
+### `The signer interface has changed in AWS SDK V2, custom signers will need
to be updated`
+
+This will be logged when a custom signer is used.
+Custom signers will currently be implementing `com.amazonaws.auth.Signer` and
will need to be
+updated to implement `software.amazon.awssdk.core.signer.Signer`.
+
+### <a name="GetObjectMetadataCalled"></a>
Review Comment:
afraid you need to strip the trailing spaces. we all do this on the markdowns
> Upgrade AWS SDK to V2 - Prerequisites
> --------------------------------------
>
> Key: HADOOP-18382
> URL: https://issues.apache.org/jira/browse/HADOOP-18382
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Ahmar Suhail
> Assignee: Ahmar Suhail
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> We want to update the AWS SDK to V2, before we do this we should warn on
> things that will no longer supported. The following changes should be made:
>
> *
> [getAmazonS3Client()|https://github.com/apache/hadoop/blob/221eb2d68d5b52e4394fd36cb30d5ee9ffeea7f0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1174]
> - Warn that this method will be removed
> *
> [initCustomSigners()|https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/SignerManager.java#L65]
> - Warn that the interface is changing, any custom signers will need to be
> updated
> *
> [bindAWSClient|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L840]
> - If DT is enabled, warn that credential providers interface is changing,
> any custom cred providers used in binding classes will need to be updated
> *
> [buildAWSProviderList|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L618]
> - if any SDK V1 cred providers are in this list, warn that these will be
> removed
> *
> [S3ClientFactory|https://github.com/apache/hadoop/blob/221eb2d68d5b52e4394fd36cb30d5ee9ffeea7f0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ClientFactory.java]
> - Update javadocs to say this interface will be replaced by a V2 client
> factory, mark interface deprecated?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
