t oo created HADOOP-15553:
-----------------------------
Summary: DOCO - not clear how to integrate hadoop-aws and aws-sdk
jars with S3A
Key: HADOOP-15553
URL: https://issues.apache.org/jira/browse/HADOOP-15553
Project: Hadoop Common
Issue Type: New Feature
Components: tools
Affects Versions: 2.8.3
Reporter: t oo
Documentation update, a compatibility matrix would be nice, especially since
upgrading to new aws java sdk does that work :
On a hadoop common 2.7.5 cluster, I take AWS Java SDK
[1.7.5|https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk/1.7.5]
and Hadoop-aws
[2.7.5|https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/2.7.5]
and use them in my Spark app writing to S3a, this is fine.
On another cluster (this time hadoop common 2.8.3 cluster) I take AWS Java SDK
1.10.6 and Hadoop-aws 2.8.3 and try to use them in my Spark app writing to
S3a, but get errors about missing classes. Turns out this version of aws java
sdk is tiny with basically nothing inside.
Couldn't find any documentation helping users on this incompatibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
