[
https://issues.apache.org/jira/browse/SPARK-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010395#comment-14010395
]
Patrick Wendell commented on SPARK-1518:
----------------------------------------
bq. Thanks, Patrick. This is useful info... I didn't realize there was still
interest in running Spark against CDH3. Certainly we'll never support it
directly, since CDH3 was end-of-lifed last year. So we don't really support
doing anything on CDH3... except upgrading it to CDH4 (or hopefully, 5 which
has Spark.
Yeah, the upstream project tries pretty hard to maintain compatibility with as
many Hadoop versions as possible since we see many people running even fairly
old ones in the wild. Of course, I'm sure Cloudera will commercially support
only the most recent ones.
bq. Re: versioning - I was just asking whether this changes the oldest version
we are compatible with. That's a question we should ask of all Hadoop patches.
I just tested Spark and it doesn't compile against 0.20.X, so this is a no-op
in terms of compatibility anyways.
It won't make the 1.0.0 window but we should get it into a 1.0.X release. My
concern is that once newer Hadoop versions come out, I want people to be able
to compile Spark against them. Again, since Spark is distributed independently
from HDFS, this is something that happens a lot, people try to compile older
Spark releases against newer Hadoop releases.
> Spark master doesn't compile against hadoop-common trunk
> --------------------------------------------------------
>
> Key: SPARK-1518
> URL: https://issues.apache.org/jira/browse/SPARK-1518
> Project: Spark
> Issue Type: Bug
> Reporter: Marcelo Vanzin
> Assignee: Colin Patrick McCabe
> Priority: Critical
>
> FSDataOutputStream::sync() has disappeared from trunk in Hadoop;
> FileLogger.scala is calling it.
> I've changed it locally to hsync() so I can compile the code, but haven't
> checked yet whether those are equivalent. hsync() seems to have been there
> forever, so it hopefully works with all versions Spark cares about.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
