[
https://issues.apache.org/jira/browse/SPARK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13975947#comment-13975947
]
Patrick Wendell commented on SPARK-1529:
----------------------------------------
[~liancheng] Hey Cheng, the tricky thing here is want to avoid _always_ going
through the HDFS filesystem interface when people are actually using local
files. We might need to add an intermediate abstraction to deal with this. We
already do this elsehwere in the code base, for instance the JobLogger will
load an output stream either directly form a file or from a hadoop file.
One thing to note is that the requirement here is really only for the shuffle
files, not for the other uses. But I realize we currently conflate these inside
of Spark so that might not buy us much. I'll look into this a bit more later.
> Support setting spark.local.dirs to a hadoop FileSystem
> --------------------------------------------------------
>
> Key: SPARK-1529
> URL: https://issues.apache.org/jira/browse/SPARK-1529
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: Patrick Wendell
> Assignee: Cheng Lian
> Fix For: 1.1.0
>
>
> In some environments, like with MapR, local volumes are accessed through the
> Hadoop filesystem interface. We should allow setting spark.local.dir to a
> Hadoop filesystem location.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
