[
https://issues.apache.org/jira/browse/HADOOP-15553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519634#comment-16519634
]
Steve Loughran commented on HADOOP-15553:
-----------------------------------------
I agree, we can improve this.
The latest versions call out the key issue: don't try and mix AWS SDK and
hadoop-aws versions, or try to drop in a newhadoop-aws JAR. .This will only
[lead to stack
traces}https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md].
What we haven't done is ever do that big table of versioning,
[Mvnrepositoryhttp://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws]
is the best place to look there: it is guaranteed to be correct for all
versions.
Happy to take a patch to the hadoop-aws docs to point to it, and maybe pull out
the warning about version mixing to being on the core index page
> DOCO - not clear how to integrate hadoop-aws and aws-sdk jars with S3A
> ----------------------------------------------------------------------
>
> Key: HADOOP-15553
> URL: https://issues.apache.org/jira/browse/HADOOP-15553
> Project: Hadoop Common
> Issue Type: New Feature
> Components: documentation, fs/s3
> Affects Versions: 2.8.3
> Reporter: t oo
> Priority: Major
>
> Documentation update, a compatibility matrix would be nice, especially since
> upgrading to new aws java sdk does that work :
>
> On a hadoop common 2.7.5 cluster, I take AWS Java SDK
> [1.7.5|https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk/1.7.5]
> and Hadoop-aws
> [2.7.5|https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/2.7.5]
> and use them in my Spark app writing to S3a, this is fine.
> On another cluster (this time hadoop common 2.8.3 cluster) I take AWS Java
> SDK 1.10.6 and Hadoop-aws 2.8.3 and try to use them in my Spark app writing
> to S3a, but get errors about missing classes. Turns out this version of aws
> java sdk is tiny with basically nothing inside.
> Couldn't find any documentation helping users on this incompatibility.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
