Eric Marnadi created SPARK-60227:
------------------------------------
Summary: Preserve self-union source slots in v2 offset logs
Key: SPARK-60227
URL: https://issues.apache.org/jira/browse/SPARK-60227
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Eric Marnadi
A streaming self-union (df.union(df)) can use one runtime source in two
positional slots.
When writing version 2 offset logs, reversing the source-ID mapping collapses
these slots
and persists only one offset. Query failure diagnostics use the same lossy
mapping.
Construct offsets by iterating the original source-ID mapping, and use that
conversion
for synchronous writes, asynchronous writes and query failure diagnostics.
Preserve
version 1 behavior and source-name validation.
Regression coverage includes fresh checkpoints, recovery from legacy single-slot
checkpoints without replay, repeated restarts, asynchronous writes and failure
diagnostics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
