Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/8512#discussion_r47081031
--- Diff: core/pom.xml ---
@@ -40,6 +40,16 @@
<classifier>${avro.mapred.classifier}</classifier>
</dependency>
<dependency>
+ <groupId>com.amazonaws</groupId>
+ <artifactId>aws-java-sdk-s3</artifactId>
+ <version>${aws.java.sdk.version}</version>
--- End diff --
1. shouldn't version declaration go into /pom.xml, with /core/pom.xml just
declaring its use
1. Be aware that Hadoop branch-2 is on v. 1.10.6; HADOOP-12269. Between
1.7.4 and 1.10.6, Amazon did change the signature on one of the methods (int ->
long on multipart upload). This means that s3a is very brittle about libraries,
and some shading may be wise here. Or be in sync with the Hadoop 2.7 version
and say "Don't use S3a on Hadoop 2.6.x", which is something the Hadoop team
would concur with.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
