viirya opened a new pull request, #59074: URL: https://github.com/apache/spark/pull/59074
### What changes were proposed in this pull request? Fix two regressions introduced by SPARK-59407 (#58700) in `ReplayListenerBus`: - Measure `spark.history.fs.eventLog.maxLineLength` in UTF-8 content bytes, excluding line endings, and report the warning in bytes. - Preserve physical line indices when oversized lines are skipped so subsequent parsing diagnostics identify the correct file line. Clarify the configuration documentation and add regression coverage. The reader continues to drain oversized lines with bounded buffering and rejects malformed UTF-8 as before. The iterator-based replay API retains its existing behavior. JIRA: https://issues.apache.org/jira/browse/SPARK-59804 ### Why are the changes needed? The existing reader compares `StringBuilder.length() * 2` with a byte-valued setting. With an 8 KiB limit, a valid ASCII application-start event with a 6,000-character application name is incorrectly skipped. Multibyte UTF-8 characters also produce inconsistent boundaries, and CRLF can reject content exactly at the limit. The reader also drops oversized lines before `zipWithIndex`. A valid event on line 1, oversized lines 2 and 3, and an invalid event on line 4 therefore report `Malformed line #2` instead of `Malformed line #4`. ### Does this PR introduce _any_ user-facing change? Yes. Events at or below the configured UTF-8 byte limit are replayed, LF/CRLF endings do not consume the content limit, and parsing diagnostics retain physical file line numbers after skipped lines. These correct regressions introduced by SPARK-59407 on the development and maintenance branches. The setting remains a line-content limit rather than an exact JVM heap budget; memory use is bounded in proportion to the configured limit. ### How was this patch tested? Added tests for approximately 6 KiB of ASCII content under an 8 KiB limit; one- through four-byte UTF-8 characters; exact and over-limit boundaries; decoder-buffer crossings; LF, CRLF, and unterminated final lines; physical line numbers after skipped lines; filtered iterators; truncated logs; and malformed UTF-8 in oversized lines. Three new regression tests fail against the original implementation when compiled and run in isolation with the existing dependency classpath. Validation with SBT: ``` build/sbt 'core/testOnly *ReplayListenerSuite' build/sbt 'core/Compile/scalastyle' 'core/Test/scalastyle' ``` All 11 suite tests passed, including compressed-log and end-to-end replay coverage. Production and test scalastyle checks passed. After the localized non-ASCII test-data exemption was added, the affected UTF-8 boundary test and test scalastyle were rerun and passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
