[
https://issues.apache.org/jira/browse/SPARK-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014910#comment-14014910
]
Colin Patrick McCabe commented on SPARK-1518:
---------------------------------------------
It seems reasonable to have a list of supported versions in the Maven build.
That wouldn't exclude people from building against other versions, of course,
but they might have to supply a maven definition via {{\-D}} or something.
bq. For example, do you consider 0.23 to be Hadoop 2? It's YARN-based and it
was (and still is AFAIK) widely used at Yahoo
0.23 is not Hadoop 2. It's a branch that Yahoo! uses internally. Everyone
else has moved on to branch-2 (Hortonworks, Cloudera, WANDisco, Intel, etc.
etc.) Yahoo! also has some clusters running on branch-2, and that is their
future too. More info here:
http://osdir.com/ml/general-hadoop-apache/2012-04/msg00000.html
bq. What about CDH4? Its version number is 2.0.0-something
Technically CDH4 is "Cloudera's distribution of Hadoop including Apache Hadoop
2.0.0." Its evolution didn't stop with 2.0.0, though. We still are going to
make another release in the cdh4 line where we backport some things. CDH5 is
where the focus is now, though.
bq. In any case we will be sensible about old versions, but my philosophy is
always to support the broadest range possible, and from everything I've seen
it's paid off – users appreciate when you do not force their hand to upgrade.
This is why Yahoo for example continues to be our biggest contributor on YARN
support, even though their YARN is pretty different.
Agree.
> Spark master doesn't compile against hadoop-common trunk
> --------------------------------------------------------
>
> Key: SPARK-1518
> URL: https://issues.apache.org/jira/browse/SPARK-1518
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: Marcelo Vanzin
> Assignee: Colin Patrick McCabe
> Priority: Critical
>
> FSDataOutputStream::sync() has disappeared from trunk in Hadoop;
> FileLogger.scala is calling it.
> I've changed it locally to hsync() so I can compile the code, but haven't
> checked yet whether those are equivalent. hsync() seems to have been there
> forever, so it hopefully works with all versions Spark cares about.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
