[
https://issues.apache.org/jira/browse/SPARK-60110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh resolved SPARK-60110.
---------------------------------
Fix Version/s: 4.2.2
4.1.5
4.0.6
4.3.1
4.4.0
Resolution: Fixed
Issue resolved by pull request 59328
[https://github.com/apache/spark/pull/59328]
> Skip event log compaction when replay skips lines over the line-length limit
> ----------------------------------------------------------------------------
>
> Key: SPARK-60110
> URL: https://issues.apache.org/jira/browse/SPARK-60110
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 4.3.0, 4.2.1, 4.0.5, 4.1.4, 3.5.10, 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.2, 4.1.5, 4.0.6, 4.3.1, 4.4.0
>
>
> SPARK-59407 bounded event log replay with
> spark.history.fs.eventLog.maxLineLength, but event log compaction only
> applies the limit in its first phase.
> EventLogFileCompactor.initializeBuilders replays through ReplayListenerBus,
> which skips over-long lines, while EventFilter.applyFilterToFile reads every
> line unbounded during rewrite.
> When a skipped line is a live job's SparkListenerJobStart (or a live
> SparkListenerSQLExecutionStart), the filter builders never learn the job is
> live, so the rewrite drops the job start (leaving its stage and task events
> without the job), and the original files are then deleted, losing it
> permanently. The rewrite phase also bypasses the memory bound.
> This only happens when compaction is enabled
> (spark.history.fs.eventLog.rolling.maxFilesToRetain) and a line exceeds the
> limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
