viirya opened a new pull request, #59327: URL: https://github.com/apache/spark/pull/59327
### What changes were proposed in this pull request? When building the application listing, `FsHistoryProvider` looks for the application end event by reopening the last event log file, skipping to `len - spark.history.fs.endEventReparseChunkSize` bytes and re-parsing the rest. This PR changes how that tail is read: - After the byte skip, the rest of the partial line is now discarded at the raw byte level (new `FsHistoryProvider.skipPartialLine`) before any UTF-8 decoding. `0x0A` never occurs inside a multi-byte UTF-8 sequence, so this is safe at any byte offset. Bytes are read one at a time from the stream returned by `openEventLog`, so the stream is left exactly at the start of the next line. - The tail is then replayed through `ReplayListenerBus.replay(InputStream, ...)` instead of the `Iterator[String]` overload fed by `Source.fromInputStream(in)(Codec.UTF8).getLines()`, so `spark.history.fs.eventLog.maxLineLength` and the strict decoder apply on this path as well. - On this path the source name passed to the replay is labeled with the offset that line numbers are counted from, e.g. `<path> (lines counted from uncompressed byte offset 123456)`. The offset is the exact start of line 1 (skip target plus the discarded partial line). This applies to all replay diagnostics that print a line number: JSON errors, `Malformed line #N`, the truncation warning and the over-long line warning. `maybeTruncated`, `eventsFilter` and `HaltReplayException` handling are unchanged, since both `replay` overloads share `replayEntries`. Without a skip (`target <= 0`), the source name is the plain path as before. ### Why are the changes needed? This is the tail-reparse follow-up discussed in #59074. With the defaults (completed apps, `endEventReparseChunkSize=1m`), every completed application takes this path, and because the target is computed from the compressed file length while the skip runs on the decompressed stream, most of a compressed log can be read there. 1. The `Iterator[String]` overload bypasses the bounded line reader (SPARK-59407 / SPARK-59804), so `maxLineLength` did not apply and an over-long line was fully materialized. 2. If the skip target landed inside a multi-byte UTF-8 character (e.g. CJK SQL text or job descriptions, which Jackson writes as raw UTF-8), `source.next()` threw `MalformedInputException` before replay started. `mergeApplicationListing` only logs it, and since the placeholder listing entry was already written with the same file size, the log is not retried, so the completed application stayed missing from the listing. The target is deterministic, so restarting the History Server failed the same way. 3. Line numbers in the replay diagnostics were relative to the skip point but printed next to the full file path. ### Does this PR introduce _any_ user-facing change? Yes, in the History Server: - A completed application whose end-event reparse offset lands inside a multi-byte UTF-8 character now appears in the application listing instead of being dropped. - `spark.history.fs.eventLog.maxLineLength` now also applies to the end-event reparse, so over-long lines there are skipped with a warning instead of being materialized. - Replay log messages on this path name the byte offset that their line numbers are relative to. - If the skip reaches the end of the stream, the replay now sees an empty input instead of failing with `NoSuchElementException` from `source.next()`. ### How was this patch tested? New tests in `FsHistoryProviderSuite`: - `end event reparse when the skip offset lands inside a multi-byte character`: the reparse offset is set to the second byte of a 3-byte character. Before this PR, the application is missing from the listing. - `end event reparse applies the line length limit and reports relative line numbers`: an over-long line in the tail is skipped under a 1k limit, and the warning names the offset and the relative line number. - `end event reparse labels JSON errors with the skip offset`: a malformed end event in the tail is reported with the offset and the relative line number. All three fail without the main change. `FsHistoryProviderSuite` (RocksDB backend), `ReplayListenerSuite`, `core/scalastyle` and `core/Test/scalastyle` pass on master. The patch cherry-picks cleanly to branch-4.x, branch-4.3, branch-4.2, branch-4.1 and branch-4.0. On branch-3.5 it needs a trivial import and `logInfo` conflict resolution; with that, the same suites and scalastyle pass there. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
