[
https://issues.apache.org/jira/browse/SPARK-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010364#comment-14010364
]
Colin Patrick McCabe commented on SPARK-1518:
---------------------------------------------
bq. It would be good to list the oldest upstream Hadoop version we support.
Many people still run Spark against Hadoop 1.X/CDH3 variants. We get high
download rates for those pre-built packages. I think this is a little different
than e.g. CDH where people upgrade all components at the same time... people
download newer versions of Spark and run it with old filesystems very often.
Thanks, Patrick. This is useful info... I didn't realize there was still
interest in running Spark against CDH3. Certainly we'll never support it
directly, since CDH3 was end-of-lifed last year. So we don't really support
doing anything on CDH3... except upgrading it to CDH4 (or hopefully, 5 which
has Spark. :)
bq. Re: versioning - I was just asking whether this changes the oldest version
we are compatible with. That's a question we should ask of all Hadoop patches.
I just tested Spark and it doesn't compile against 0.20.X, so this is a no-op
in terms of compatibility anyways.
It sounds like we're good to go on replacing the {{flush}} with {{hsync}} then?
I notice you marked this as "critical" recently; do you think it's important
to 1.0?
> Spark master doesn't compile against hadoop-common trunk
> --------------------------------------------------------
>
> Key: SPARK-1518
> URL: https://issues.apache.org/jira/browse/SPARK-1518
> Project: Spark
> Issue Type: Bug
> Reporter: Marcelo Vanzin
> Assignee: Colin Patrick McCabe
> Priority: Critical
>
> FSDataOutputStream::sync() has disappeared from trunk in Hadoop;
> FileLogger.scala is calling it.
> I've changed it locally to hsync() so I can compile the code, but haven't
> checked yet whether those are equivalent. hsync() seems to have been there
> forever, so it hopefully works with all versions Spark cares about.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
