[
https://issues.apache.org/jira/browse/SPARK-24787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579012#comment-16579012
]
Marcelo Vanzin commented on SPARK-24787:
----------------------------------------
Is the slowness really caused by the use of hsync vs. hflush? I'd expect the
flushing of the data, not the metadata update, to be the expensive part...
In any case, if you have any ideas, feel free to post a PR.
> Events being dropped at an alarming rate due to hsync being slow for
> eventLogging
> ---------------------------------------------------------------------------------
>
> Key: SPARK-24787
> URL: https://issues.apache.org/jira/browse/SPARK-24787
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Web UI
> Affects Versions: 2.3.0, 2.3.1
> Reporter: Sanket Reddy
> Priority: Minor
>
> [https://github.com/apache/spark/pull/16924/files] updates the length of the
> inprogress files allowing history server being responsive.
> Although we have a production job that has 60000 tasks per stage and due to
> hsync being slow it starts dropping events and the history server has wrong
> stats due to events being dropped.
> A viable solution is not to make it sync very frequently or make it
> configurable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
