[
https://issues.apache.org/jira/browse/SPARK-28594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915383#comment-16915383
]
Jungtaek Lim commented on SPARK-28594:
--------------------------------------
To ensure creating smaller PRs (easier reviews) I would split this issue into
two sub-issues:
1) just roll event log files (no compaction)
2) compact old event log files
Note that even rolling event log files without compaction could help for some
extreme case, where the log file got really huge for running application so you
decide to drop some old logs bearing that it will lose the ability to replay
log file. Currently there's no way to do this - deleting event log file which
is open for writing would bring some unexpected issues and we would end up with
stopping application.
> Allow event logs for running streaming apps to be rolled over.
> --------------------------------------------------------------
>
> Key: SPARK-28594
> URL: https://issues.apache.org/jira/browse/SPARK-28594
> Project: Spark
> Issue Type: Improvement
> Components: Input/Output
> Affects Versions: 3.0.0
> Environment: This has been reported on 2.0.2.22 but affects all
> currently available versions.
> Reporter: Stephen Levett
> Priority: Major
>
> At all current Spark releases when event logging on spark streaming is
> enabled the event logs grow massively. The files continue to grow until the
> application is stopped or killed.
> The Spark history server then has difficulty processing the files.
> https://issues.apache.org/jira/browse/SPARK-8617
> Addresses .inprogress files but not event log files that are still running.
> Identify a mechanism to set a "max file" size so that the file is rolled over
> when it reaches this size?
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
