viirya opened a new pull request, #59328: URL: https://github.com/apache/spark/pull/59328
### What changes were proposed in this pull request? Event log compaction reads the files to compact twice: `EventLogFileCompactor.initializeBuilders` replays them through `ReplayListenerBus` to build the event filters, and `rewrite` applies the filters line by line. Since SPARK-59407, replay skips lines longer than `spark.history.fs.eventLog.maxLineLength`, but the filter builders are not aware of it. This PR makes `EventLogFileCompactor.compact` skip compaction, keeping the original files, when replay skips any line: - `ReplayListenerBus` counts the over-long lines it skips (`numSkippedLines`). - `initializeBuilders` stops replaying once a line is skipped and returns `None`. - `compact` then logs a warning and returns the new result code `CompactionResultCode.LINES_OVER_LENGTH_LIMIT` without rewriting or deleting any file. When compaction proceeds, every line of the files to compact was read within the limit, so the reads in `rewrite` stay within it as well. ### Why are the changes needed? The filter builders decide which jobs, stages, tasks and SQL executions are still live from the events they see. If a skipped line is the `SparkListenerJobStart` of a live job (for example, one with large job properties), the builders never learn that the job is live, so the rewrite drops the job start and all of its stage and task events. The same applies to a live `SparkListenerSQLExecutionStart` with a large plan. Compaction then deletes the original files, so these events are lost permanently, even if the limit is raised later. The rewrite phase also reads every line without the limit, bypassing the memory bound added by SPARK-59407. Other options were considered: - Applying the same limit in `rewrite`: both phases agree, but the dropped events (and the events of the live job that depend on them) are still lost permanently. - Copying over-long lines to the compacted file verbatim: the stage and task events of the live job are still dropped, and an over-long start of a finished job would be kept without its end event, so the job would appear running forever to a reader with a higher limit. Skipping compaction is the only option that does not lose data. The cost is that the event log of such an application is not compacted while the over-long line is within the range to compact; compaction is still evaluated once per new event log file, as for `LOW_SCORE_FOR_COMPACTION`. ### Does this PR introduce _any_ user-facing change? Yes, compared with unreleased branches that include SPARK-59407. When event log compaction is enabled (`spark.history.fs.eventLog.rolling.maxFilesToRetain`) and an event log line exceeds `spark.history.fs.eventLog.maxLineLength`, the History Server no longer compacts the event log files of that application (it logs a warning instead), rather than compacting them and losing the events of live jobs and SQL executions. No released version is affected. ### How was this patch tested? New test in `EventLogFileCompactorSuite` with a live job whose job start line exceeds the limit: compaction is skipped and all original files are kept; with the default limit the same files are compacted and the live job's events are kept. Also added an assertion on `numSkippedLines` in `ReplayListenerSuite`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
