[
https://issues.apache.org/jira/browse/SPARK-60110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-60110:
--------------------------------
Description:
SPARK-59407 bounded event log replay with
spark.history.fs.eventLog.maxLineLength, but event log compaction only applies
the limit in its first phase. EventLogFileCompactor.initializeBuilders replays
through ReplayListenerBus, which skips over-long lines, while
EventFilter.applyFilterToFile reads every line unbounded during rewrite.
When a skipped line is a live job's SparkListenerJobStart (or a live
SparkListenerSQLExecutionStart), the filter builders never learn the job is
live, so the rewrite drops the job start (leaving its stage and task events
without the job), and the original files are then deleted, losing it
permanently. The rewrite phase also bypasses the memory bound.
This only happens when compaction is enabled
(spark.history.fs.eventLog.rolling.maxFilesToRetain) and a line exceeds the
limit.
was:
SPARK-59407 bounded event log replay with
spark.history.fs.eventLog.maxLineLength, but event log compaction only applies
the limit in its first phase. EventLogFileCompactor.initializeBuilders replays
through ReplayListenerBus, which skips over-long lines, while
EventFilter.applyFilterToFile reads every line unbounded during rewrite.
When a skipped line is a live job's SparkListenerJobStart (or a live
SparkListenerSQLExecutionStart), the filter builders never learn the job is
live, so the rewrite drops the job start and all of its stage and task events,
and the original files are then deleted, losing the data permanently. The
rewrite phase also bypasses the memory bound.
This only happens when compaction is enabled
(spark.history.fs.eventLog.rolling.maxFilesToRetain) and a line exceeds the
limit.
> Skip event log compaction when replay skips lines over the line-length limit
> ----------------------------------------------------------------------------
>
> Key: SPARK-60110
> URL: https://issues.apache.org/jira/browse/SPARK-60110
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 4.3.0, 4.2.1, 4.0.5, 4.1.4, 3.5.10, 4.4.0
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> SPARK-59407 bounded event log replay with
> spark.history.fs.eventLog.maxLineLength, but event log compaction only
> applies the limit in its first phase.
> EventLogFileCompactor.initializeBuilders replays through ReplayListenerBus,
> which skips over-long lines, while EventFilter.applyFilterToFile reads every
> line unbounded during rewrite.
> When a skipped line is a live job's SparkListenerJobStart (or a live
> SparkListenerSQLExecutionStart), the filter builders never learn the job is
> live, so the rewrite drops the job start (leaving its stage and task events
> without the job), and the original files are then deleted, losing it
> permanently. The rewrite phase also bypasses the memory bound.
> This only happens when compaction is enabled
> (spark.history.fs.eventLog.rolling.maxFilesToRetain) and a line exceeds the
> limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
