[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177918#comment-15177918 ]
Steve Loughran commented on SPARK-7481: --------------------------------------- Longer term, having spark_home /lib/*.jar is the best, general purpose solution. For now, {{hadoop-aws}} can be added to the existing 2.6 profile, explicitly excluding the full amazon SDK jar. This would give s3n back to the code. Jets3t is still in the spark-assembly JAR today. If built with 2.6.x, you'd get s3n and, if you added the full aws-SDK JAR with --addjars, S3a support If you built with 2.7.x (e.g {{-Dhadoop=version=2.7.2}}) you'd get, s3n, s3a and, implicitly, the (much smaller) {{amazon-s3-sdk}} JAR needed to talk with S3. Users wouldn't need to add the amazon-aws-sdk.jar to the submission (it would cause link problems if they tried). ..Or, to keep the assembly JAR small, {{amazon-s3-sdk}} could also be excluded. This would add the ASF classes, but you'd always need to add the right JAR for the hadoop version you compiled against (Amazon changed a parameter from an int to a long in a method, see) > Add Hadoop 2.6+ profile to pull in object store FS accessors > ------------------------------------------------------------ > > Key: SPARK-7481 > URL: https://issues.apache.org/jira/browse/SPARK-7481 > Project: Spark > Issue Type: Improvement > Components: Build > Affects Versions: 1.3.1 > Reporter: Steve Loughran > > To keep the s3n classpath right, to add s3a, swift & azure, the dependencies > of spark in a 2.6+ profile need to add the relevant object store packages > (hadoop-aws, hadoop-openstack, hadoop-azure) > this adds more stuff to the client bundle, but will mean a single spark > package can talk to all of the stores. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org