DengYiping opened a new pull request, #59620:
URL: https://github.com/apache/spark/pull/59620
### What changes were proposed in this pull request?
Backport #58996 (SPARK-59747, master commit
`d9a35e971b22f2c663fdf4738442c53d1a33d020`) to `branch-4.0`, as requested by
@sarutak.
Catch `CharacterCodingException` in `FsHistoryProvider.compact()`, log a
WARN, and record the rolled event log index as evaluated.
`EventLogFileCompactor.initializeBuilders` also logs a WARN identifying the
malformed file before rethrowing.
The conflict resolution retains this branch's existing single
`fileCompactor`; it does not backport master's support for multiple event log
directories. The test aliases NIO `Files` as `JavaFiles` to preserve existing
Guava `Files` calls.
### Why are the changes needed?
Malformed UTF-8 in immutable rolled event log files causes compaction to
fail repeatedly as the current log grows. Retrying the same unchanged files
cannot succeed. Recording the evaluated index avoids repeated attempts until
another log file is rolled.
### Does this PR introduce _any_ user-facing change?
Yes. The History Server logs a WARN once per evaluated rolled log index
instead of repeatedly logging an ERROR. A separate WARN identifies the
malformed file. Application listing/replay behavior is unchanged, and malformed
logs remain uncompacted.
### How was this patch tested?
Passed locally with Java 21 on the `branch-4.0` backport:
```sh
build/sbt 'core/Compile/scalastyle' 'core/Test/scalastyle' \
'core/testOnly
org.apache.spark.deploy.history.RocksDBBackendFsHistoryProviderSuite
org.apache.spark.deploy.history.RocksDBBackendWithProtobufSerializerSuite -- -z
"SPARK-59747"' \
'core/testOnly org.apache.spark.deploy.history.EventLogFileCompactorSuite'
```
Both malformed-input regression variants passed (2 tests), including the
WARN naming the malformed file. All 12 compactor tests passed, including the
SPARK-60110 incomplete-replay coverage. Both style checks passed with zero
errors/warnings; `git diff --check` also passed. LevelDB-backed variants were
not run on Apple Silicon.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5) for the original patch; OpenAI
Codex (GPT-5) for the backport.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
