[
https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563033#comment-14563033
]
Thomas Demoor commented on SPARK-7481:
--------------------------------------
[~srowen] and [~ste...@apache.org], for s3a, things should improve in future
Hadoop versions. I have a first patch set up for [HADOOP-11684] that also bumps
the aws-sdk version to a recent version. From 1.9 onwards, you can pull in
individual components separately. For s3a, we only need s3 (and evidently the
core lib) which solves both the large size and the kinesis issue.
> Add Hadoop 2.6+ profile to pull in object store FS accessors
> ------------------------------------------------------------
>
> Key: SPARK-7481
> URL: https://issues.apache.org/jira/browse/SPARK-7481
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 1.3.1
> Reporter: Steve Loughran
>
> To keep the s3n classpath right, to add s3a, swift & azure, the dependencies
> of spark in a 2.6+ profile need to add the relevant object store packages
> (hadoop-aws, hadoop-openstack, hadoop-azure)
> this adds more stuff to the client bundle, but will mean a single spark
> package can talk to all of the stores.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
