anew opened a new pull request, #57514:
URL: https://github.com/apache/spark/pull/57514
### What changes were proposed in this pull request?
Implement `buildScd2AuxiliaryTableSpecFor` in `AutoCdcAuxiliaryTable`,
which was previously an internal-error stub (`"SCD2 auxiliary table derivation
is not yet implemented."`).
Unlike the SCD1 auxiliary table -- which stores only the key columns plus
the CDC metadata column -- the SCD2 auxiliary table holds full hidden rows
(tombstones and coalesced no-op upserts that may later be promoted into the
visible target). Its schema is therefore the entire SCD2 target row schema plus
the aux-only `__spark_autocdc_deleted_by_batch_id` logical-delete marker (a
nullable `Long`). This matches what
`Scd2BatchProcessor.findAffectedRowsFromAuxiliaryTable` reads and what the
merges write ("`deletedByBatchId` in addition to all of the columns in the
target table"). The spec records the SCD2 type and the key column names for
drift detection, mirroring the SCD1 spec.
`DatasetManager` materialization/evolution is schema-driven and
SCD-type-agnostic, so it creates and evolves the SCD2 auxiliary table from this
spec with no further change.
Also widens `Scd2BatchProcessor.deletedByBatchIdColName` to
`private[pipelines]` so the graph package can name the marker column.
This is one sub-task of enabling SCD2 AutoCDC end to end (under
[SPARK-56249](https://issues.apache.org/jira/browse/SPARK-56249)), following
[SPARK-58319](https://issues.apache.org/jira/browse/SPARK-58319) (SCD2 target
schema derivation). It is dormant until the switch-flip sub-task
([SPARK-58321](https://issues.apache.org/jira/browse/SPARK-58321)) enables SCD2
flows: the flow planner and construction still reject SCD2 today, so no SCD2
auxiliary table is materialized yet.
### Why are the changes needed?
Deriving the SCD2 auxiliary-table spec is a prerequisite for SCD2 support.
The pipeline's dataset materialization builds and
schema-evolves the auxiliary table from this spec, and the SCD2
reconciliation engine (already landed in SPARK-57378) reads/writes exactly this
shape, so the auxiliary table cannot be created correctly until the spec
produces the right schema, properties, and drift metadata.
### Does this PR introduce _any_ user-facing change?
No. SCD2 AutoCDC flows are still not constructible/runnable (rejected by the
flow planner), so no SCD2 auxiliary table is created and there is no change to
released or user-facing behavior. This only makes the spec derivable
internally.
### How was this patch tested?
New `AutoCdcScd2AuxiliaryTableSpecSuite` resolves a real SCD2 AutoCDC graph
and asserts the derived auxiliary-table spec:
- the schema equals the full SCD2 target row schema plus the marker
column;
- the marker is a nullable `Long` named
`__spark_autocdc_deleted_by_batch_id`, appended last;
- the framework columns (`__START_AT`, `__END_AT`, `_cdc_metadata`) are
retained from the target row schema;
- the spec records `ScdType.Type2`, the key column names (property +
`expectedKeyFields`), and the correct identifiers;
- composite keys are all carried through.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
