[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177623#comment-15177623 ]
Steve Loughran commented on SPARK-7481: --------------------------------------- Hadoop 2.6 added S3a, which we put into a new hadoop-tools/hadoop-aws JAR, along with a dependency on sun's {{aws-java}} SDK. Someone other than myself went and moved the existing S3n classes into the same JAR. If'd I'd seen that, i'd have -1'd it, but I didn't notice until 2.6 shipped. as stated, I wouldn't use S3a in Hadoop 2.6.x. HADOOP-11571 contains the reasons. It wasn't until Hadoop 2.7 that it became ready for serious use. Both come in hadoop-aws; s3a needs an amazon JAR, which must be matched precisely with the version used in the hadoop library. > Add Hadoop 2.6+ profile to pull in object store FS accessors > ------------------------------------------------------------ > > Key: SPARK-7481 > URL: https://issues.apache.org/jira/browse/SPARK-7481 > Project: Spark > Issue Type: Improvement > Components: Build > Affects Versions: 1.3.1 > Reporter: Steve Loughran > > To keep the s3n classpath right, to add s3a, swift & azure, the dependencies > of spark in a 2.6+ profile need to add the relevant object store packages > (hadoop-aws, hadoop-openstack, hadoop-azure) > this adds more stuff to the client bundle, but will mean a single spark > package can talk to all of the stores. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org