[
https://issues.apache.org/jira/browse/SPARK-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011897#comment-14011897
]
Sean Owen commented on SPARK-1518:
----------------------------------
"they write their app against the Spark API's in Maven central (they can do
this no matter which cluster they want to run on)"
Yeah this is the issue. OK, if I compile against Spark artifacts as a runtime
dependency and submit an app to the cluster, it should be OK no matter what
build of Spark is running. The binding from Spark to Hadoop is hidden from the
app.
I am thinking of the case where I want to build an app that is a client of
Spark -- embedding it. Then I am including the client of Hadoop for example. I
have to match my cluster than and there is no Hadoop 2 Spark artifact.
Am I missing something big here? that's my premise about why there would ever
be a need for different artifacts. It's the same use case as in Sandy's blog:
http://blog.cloudera.com/blog/2014/04/how-to-run-a-simple-apache-spark-app-in-cdh-5/
> Spark master doesn't compile against hadoop-common trunk
> --------------------------------------------------------
>
> Key: SPARK-1518
> URL: https://issues.apache.org/jira/browse/SPARK-1518
> Project: Spark
> Issue Type: Bug
> Reporter: Marcelo Vanzin
> Assignee: Colin Patrick McCabe
> Priority: Critical
>
> FSDataOutputStream::sync() has disappeared from trunk in Hadoop;
> FileLogger.scala is calling it.
> I've changed it locally to hsync() so I can compile the code, but haven't
> checked yet whether those are equivalent. hsync() seems to have been there
> forever, so it hopefully works with all versions Spark cares about.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
