[
https://issues.apache.org/jira/browse/SPARK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14263928#comment-14263928
]
Kannan Rajah commented on SPARK-1529:
-------------------------------------
In a MapR distribution, these files need to go into MapR volume and not local
disk. This MapR volume is local to the node though, but its part of the
distributed file system. This can be achieved in 2 ways:
1. Expose the MapR file system as a NFS mount point. Now, you can use the
normal java.io API and data will still get written to MapR volume instead of
local disk.
2. Use HDFS API (with underlying MapR implementation) instead of java.io to
make the IO go to MapR volume.
> Support setting spark.local.dirs to a hadoop FileSystem
> --------------------------------------------------------
>
> Key: SPARK-1529
> URL: https://issues.apache.org/jira/browse/SPARK-1529
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: Patrick Wendell
> Assignee: Cheng Lian
>
> In some environments, like with MapR, local volumes are accessed through the
> Hadoop filesystem interface. We should allow setting spark.local.dir to a
> Hadoop filesystem location.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
