[
https://issues.apache.org/jira/browse/SPARK-29105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932142#comment-16932142
]
Gabor Somogyi commented on SPARK-29105:
---------------------------------------
This is a known flakyness and I think it would be solved when we can remove
ZKUtils from Kafka tests. I've filed SPARK-28144 but Kafka feature is missing
at the moment (better to say it's there but not working). Trying to push this
area...
> SHS may delete driver log file of in progress application
> ---------------------------------------------------------
>
> Key: SPARK-29105
> URL: https://issues.apache.org/jira/browse/SPARK-29105
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Marcelo Vanzin
> Priority: Minor
>
> There's an issue with how the SHS cleans driver logs that is similar to the
> problem of event logs: because the file size is not updated when you write to
> it, the SHS fails to detect activity and thus may delete the file while it's
> still being written to.
> SPARK-24787 added a workaround in the SHS so that it can detect that
> situation for in-progress apps, replacing the previous solution which was too
> slow for event logs.
> But that doesn't work for driver logs because they do not follow the same
> pattern (different file names for in-progress files), and thus would require
> the SHS to open the driver log files on every scan, which is expensive.
> The old approach (using the {{hsync}} API) seems to be a good match for the
> driver logs, though, which don't slow down the listener bus like event logs
> do.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
