ericm-db opened a new pull request, #59447:
URL: https://github.com/apache/spark/pull/59447
### What changes were proposed in this pull request?
Build v2 offsets from the source-ID mapping so every slot of a streaming
self-union is
preserved. Use the same conversion method for synchronous writes,
asynchronous writes and
failure diagnostics.
### Why are the changes needed?
Reversing the source-ID mapping collapses IDs that refer to the same runtime
source.
For `df.union(df)`, this drops one source slot from the offset log and from
query failure
diagnostics.
### Does this PR introduce _any_ user-facing change?
Yes. Version 2 offset logs retain both self-union slots, including after
checkpoint
recovery. Version 1 offsets and source-name validation are unchanged.
### How was this patch tested?
Added regressions for fresh and legacy single-slot checkpoints, repeated
restarts,
asynchronous writes, failure diagnostics, missing offsets and unknown source
IDs.
All six focused regression tests passed:
```sh
build/sbt \
'set ThisBuild / dependencyOverrides += "org.xerial.snappy" %
"snappy-java" % "1.1.10.8"' \
'sql/testOnly *OffsetSeqLogSuite -- -z "stream progress"' \
'sql/testOnly *MicroBatchExecutionSuite
*AsyncProgressTrackingMicroBatchExecutionSuite -- -z "self-union"'
```
Local validation used a temporary Snappy 1.1.10.8 override because the Maven
proxy rejected
the configured 1.1.10.12 artifact. The project dependency version is
unchanged.
`sql/scalastyle`, `sql/Test/scalastyle` and `git diff --check` also passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex CLI 0.156.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
