[
https://issues.apache.org/jira/browse/SPARK-28594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127776#comment-17127776
]
Jungtaek Lim commented on SPARK-28594:
--------------------------------------
Actually it has been an issue with almost all of Spark versions (streaming
support + event log), but it's unlikely that Spark community does the backport
of a new feature. I don't think it can land to the 2.4.x version line. I'm not
sure there's an interest of having 2.5 as well, so at least for now the only
way to get it is trying out Spark 3.0.0.
> Allow event logs for running streaming apps to be rolled over
> -------------------------------------------------------------
>
> Key: SPARK-28594
> URL: https://issues.apache.org/jira/browse/SPARK-28594
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Stephen Levett
> Assignee: Jungtaek Lim
> Priority: Major
> Labels: releasenotes
> Fix For: 3.0.0
>
>
> At all current Spark releases when event logging on spark streaming is
> enabled the event logs grow massively. The files continue to grow until the
> application is stopped or killed.
> The Spark history server then has difficulty processing the files.
> https://issues.apache.org/jira/browse/SPARK-8617
> Addresses .inprogress files but not event log files that are still running.
> Identify a mechanism to set a "max file" size so that the file is rolled over
> when it reaches this size?
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
