anew opened a new pull request, #57444:
URL: https://github.com/apache/spark/pull/57444
Note: This continues @AnishMahto's #56457, opened from my fork while the
original author is out of office. It carries his implement merge into aux and
target + self-review commits and adds review fixes and additional test coverage
on top. #56457 should be closed in favor of this PR to keep review in one place.
### What changes were proposed in this pull request?
This is the next step in the SCD2 AutoCDC batch-processor series (following
SPARK-57356, "Cleanup Delete Encoding Rows Post-Reconciliation"). It adds the
final merge stage that writes a reconciled microbatch into the auxiliary and
target tables. Three methods are added to Scd2BatchProcessor:
- identifyAndTagAuxRows — tags each post-reconciliation row with
shouldRouteToAuxTableColName, marking which rows belong in the auxiliary table
(tombstones and other delete-boundary encodings) versus the visible target
table.
- mergeRowsIntoAuxiliaryTable — merges the reconciled rows tagged for the
auxiliary table into it, handling tombstone insertion/advancement and
deletedByBatchId-scoped garbage collection.
- mergeRowsIntoTargetTable — merges the reconciled visible rows into the
target table (the user-facing SCD2 history).
Together with the reconciliation and cleanup steps already merged, this
completes the per-microbatch transformation pipeline; the foreachBatch callback
that drives it end-to-end lands in the follow-up (SPARK-57395).
### Why are the changes needed?
The SCD2 batch processor is being built incrementally across the SPARK-571xx
series. After a microbatch is preprocessed, decomposed, reconciled, and cleaned
up, its rows must actually be written to the auxiliary and target tables using
the correct MERGE semantics. This PR supplies that final merge step, which the
engine's SCD2 execution path depends on.
### Does this PR introduce any user-facing change?
No. These are internal (private[autocdc]) batch-processor methods; SCD2
AutoCDC is not yet reachable end-to-end (the driving foreachBatch handler and
the API/planner wiring land in separate changes).
### How was this patch tested?
New unit tests:
- Scd2BatchProcessorMergeSuite — covers the aux/target merge behavior
(tombstone insertion and advancement, GC of superseded aux rows, target-table
history writes).
- Additional cases in Scd2BatchProcessorSuite — identifyAndTagAuxRows
routing, per-key window isolation, dotted / identifier-sensitive column names,
and combined-branch / multi-column merges.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
