[
https://issues.apache.org/jira/browse/SPARK-60114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-60114:
--------------------------------
Affects Version/s: 3.5.8
4.2.0
4.1.3
4.0.4
4.3.0
4.4.0
(was: 5.0.0)
> Tolerate an incomplete UTF-8 sequence at the end of a possibly truncated
> event log during replay
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-60114
> URL: https://issues.apache.org/jira/browse/SPARK-60114
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
> Reporter: L. C. Hsieh
> Priority: Major
>
> {{ReplayListenerBus.replay(InputStream, ...)}} decodes event logs with a
> strict UTF-8 decoder. When {{maybeTruncated}} is true (an in-progress log), a
> partial last line is meant to be tolerated: a {{JsonParseException}} on the
> last line is logged as a warning instead of failing the replay. However, if
> the writer has flushed only the first bytes of a multi-byte UTF-8 character
> at the end of the file (e.g. CJK SQL text or job descriptions, which Jackson
> writes as raw UTF-8), the decoder throws {{MalformedInputException}}. That is
> an {{IOException}}, which {{replayEntries}} rethrows regardless of
> {{maybeTruncated}}, so the replay fails instead of treating the incomplete
> last line as truncated.
> This affects both the main replay path (loading an in-progress application
> UI, and listing with fast in-progress parsing) and the History Server
> end-event reparse with
> {{spark.history.fs.inProgressOptimization.enabled=false}}. The previous
> {{Source.getLines()}}-based reader had the same behavior, so this is
> pre-existing.
> An incomplete UTF-8 sequence at EOF should be handled like a truncated last
> line when {{maybeTruncated}} is set, while malformed input elsewhere in the
> file should still fail.
> Raised in the review of https://github.com/apache/spark/pull/59327
> (SPARK-60109).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
