[
https://issues.apache.org/jira/browse/HADOOP-19947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099116#comment-18099116
]
ASF GitHub Bot commented on HADOOP-19947:
-----------------------------------------
steveloughran commented on code in PR #8628:
URL: https://github.com/apache/hadoop/pull/8628#discussion_r3650148019
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/qualifying_a_new_sdk.md:
##########
@@ -0,0 +1,1899 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
+NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
+"OPTIONAL" in this document are to be interpreted as described in
+RFC 2119.
+
+# Qualifying an AWS SDK
+
+
+> The AWS SDKs and CLI are designed for usage with official AWS services.
+> We may introduce and enable new features by default, such as these new
default integrity protections,
+> prior to them being supported or otherwise handled by third-party service
implementations.
+
+That is a quote from an
+[announcement of a somewhat incompatible
change](https://github.com/aws/aws-sdk-java-v2/discussions/5802)
+which shipped in v2.30.0 of the AWS SDK.
+
+It highlights the SDK team's point of view: their job is to work on the SDK to
support AWS's own services.
+Compatibility with third-party services is not their problem, and they do not
test against such stores.
+
+This makes sense from their perspective: if someone implements their own S3
store, then it is
+their task to make it compatible with AWS S3, even as that is a moving target
with no public
+formal API specification.
+
+The S3A connector is one of the most popular of S3 connectors used to connect
+JVM-hosted big-data applications to AWS S3 *and to other S3-compatible stores*.
+We do not have the luxury of saying "third-party stores are not our problem",
so have to make
+sure that our release works with all stores.
+
+And because of that broad adoption, we need to make sure that it works in
+different deployment scenarios, with different configurations even within AWS.
+
+The task of qualifying an AWS SDK is a lot more than just incrementing a
number in a maven POM file:
+it is determining whether the SDK is safe to adopt, and, if safe,
+identifying and making any changes in our code that are needed to migrate.
+
+
+## Introduction
+
+The S3A connector is utterly dependent upon the AWS SDK; even a minor change
can have serious consequences.
+That is: changing a single number in a maven file can bring new features and
needed bug fixes.
+It can also cause a lot of damage, albeit unintentionally.
+
+Some example regressions encountered previously include:
+* The SDK printing a warning message telling developers off every time a
specific object in the SDK is instantiated
+ This breaks all tests which look for specific output strings and runs a risk
of generating support calls asking "why is my application telling me off?"
+* A change in the semantics of calling `abort()` on a stream.
+ This was a valid design decision — however, it was unexpected.
+ And again the warning message printed every time the stream was closed
prematurely flooded application logs.
+* Instabilities in the shading of third-party libraries (slf4j, etc.), with
consequences such as the inability to enable any form of logging.
+* The shaded library continuing to declare dependencies which are redundant
due to the shading.
Review Comment:
pretty much every upgrade was a pain point.
> S3A: Document the AWS SDK upgrade process
> -----------------------------------------
>
> Key: HADOOP-19947
> URL: https://issues.apache.org/jira/browse/HADOOP-19947
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 3.5.0, 3.4.3
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Document how to qualify the aws sdk.
> this is the work from
> https://github.com/steveloughran/engineering-proposals/blob/trunk/hardening-aws-sdk-qualification.md
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
