[
https://issues.apache.org/jira/browse/SPARK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976038#comment-13976038
]
Patrick Wendell commented on SPARK-1529:
----------------------------------------
One idea proposed by [~adav] was to always use the Hadoop filesystem API, but
to potentially implement our own version of the local filesystem if we find the
Hadoop version has performance drawbacks.
Another issue is that we use FileChannel objects directly in the
`DiskBlockObjectWriter`. After looking through this a bit, the functionality
there to commit and rewind writes is not actually used anywhere, we could
probably just remove it.
[~liancheng] I think it would be worth it to look at a version where we just
take all of the File API's and replace them with Hadoop equivalents. I.e. your
proposal.
> Support setting spark.local.dirs to a hadoop FileSystem
> --------------------------------------------------------
>
> Key: SPARK-1529
> URL: https://issues.apache.org/jira/browse/SPARK-1529
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: Patrick Wendell
> Assignee: Cheng Lian
> Fix For: 1.1.0
>
>
> In some environments, like with MapR, local volumes are accessed through the
> Hadoop filesystem interface. We should allow setting spark.local.dir to a
> Hadoop filesystem location.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
