[
https://issues.apache.org/jira/browse/SPARK-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010305#comment-14010305
]
Marcelo Vanzin commented on SPARK-1518:
---------------------------------------
Hmm, may I suggest a different approach?
Andrew, who wrote the code, might have more info. But from my understanding,
the flushes were needed because the history server might read logs from
applications that were not yet finished. So the flush was a best-effort to
avoid having the HS read files that contained partial JSON objects (and fail to
parse them).
But since then the HS was changed to only read logs from finished applications.
I think it's safe to assume that finished applications are not writing to the
event log anymore, so the above scenario doesn't exist.
So could we just get rid of the explicit flush instead?
> Spark master doesn't compile against hadoop-common trunk
> --------------------------------------------------------
>
> Key: SPARK-1518
> URL: https://issues.apache.org/jira/browse/SPARK-1518
> Project: Spark
> Issue Type: Bug
> Reporter: Marcelo Vanzin
> Assignee: Colin Patrick McCabe
> Priority: Critical
>
> FSDataOutputStream::sync() has disappeared from trunk in Hadoop;
> FileLogger.scala is calling it.
> I've changed it locally to hsync() so I can compile the code, but haven't
> checked yet whether those are equivalent. hsync() seems to have been there
> forever, so it hopefully works with all versions Spark cares about.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
