[
https://issues.apache.org/jira/browse/SPARK-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011942#comment-14011942
]
Matei Zaharia commented on SPARK-1518:
--------------------------------------
Sean, the model for linking to Hadoop has been that users also add a dependency
on hadoop-client if they want to access HDFS for the past few releases. See
http://spark.apache.org/docs/latest/scala-programming-guide.html#linking-with-spark
for example. This model is there because Hadoop itself has decided to create
the hadoop-client Maven artifact as a way to get apps to link to it. It works
for all the recent versions of Hadoop as far as I know -- users don't have to
link against a custom-built Spark for their distro.
Regarding binary builds on apache.org, we want users to be able to start using
Spark as conveniently as possible on any distribution. It is the goal of the
Apache project to have people use Apache Spark as easily as possible.
> Spark master doesn't compile against hadoop-common trunk
> --------------------------------------------------------
>
> Key: SPARK-1518
> URL: https://issues.apache.org/jira/browse/SPARK-1518
> Project: Spark
> Issue Type: Bug
> Reporter: Marcelo Vanzin
> Assignee: Colin Patrick McCabe
> Priority: Critical
>
> FSDataOutputStream::sync() has disappeared from trunk in Hadoop;
> FileLogger.scala is calling it.
> I've changed it locally to hsync() so I can compile the code, but haven't
> checked yet whether those are equivalent. hsync() seems to have been there
> forever, so it hopefully works with all versions Spark cares about.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
