anew commented on code in PR #57625:
URL: https://github.com/apache/spark/pull/57625#discussion_r3688689860
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/AutoCdcAuxiliaryTable.scala:
##########
@@ -227,13 +238,28 @@ object AutoCdcAuxiliaryTable {
StructField(Scd2BatchProcessor.deletedByBatchIdColName, LongType,
nullable = true)
val scd2AuxiliaryTableSchema = StructType(targetTableSchema.fields :+
deletedByBatchIdField)
+ // Resolve the effective track-history columns (an explicit TRACK HISTORY
selection, or the
+ // default of every eligible non-key/non-framework column) against the
target schema, using the
+ // same single source of truth the reconciler uses. A change in this set
reinterprets which
+ // transitions open a new historical record, so it is drift-checked.
+ val caseSensitive =
+ inputAutoCdcFlow.df.sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val trackHistoryColumnNames =
Scd2BatchProcessor.computeTrackedHistoryColumns(
+ schema = targetTableSchema,
+ changeArgs = inputAutoCdcFlow.changeArgs,
+ caseSensitive = caseSensitive
+ )
Review Comment:
Good catch, and you've pinned exactly the right tension. Two parts to the
response:
The divergence fix: I've moved the tracked-set resolution off the evolved
target schema and onto the flow's user-selected source schema, via a new
AutoCdcMergeFlow.trackHistoryColumnNames val (the shape you suggested in the
other comment) that the aux-spec builder now reads. So the recorded set follows
the flow's selection, not columns lingering in the sticky target — and the
implicit default/* EXCEPT cases are now actually detected.
The semantic decision: requiring a full refresh on any SCD2 tracked-set
change — including additive evolution under default/* EXCEPT — is intended, and
I've documented it as a deliberate divergence from SCD1 (on the new val) and
pinned it with end-to-end tests (add a column, drop a column, both →
TRACK_HISTORY_DRIFT).
The reason it can't just tolerate additive evolution: changing which columns
define an SCD2 run reinterprets where record boundaries fall, and that can't be
applied to already-reconciled history. We verified that letting it through
silently corrupts history when a late event drags pre-change records into the
reconciliation window (a widened set fabricates spurious splits; a narrowed set
destroys distinct records).
SCD1 has no run concept, so the same evolution is a no-op there.
We will evaluate whether we can support evolution of the tracked column set
in SPARK-58452, but for now, this PR prevents any drift in the tracked columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
