anew commented on code in PR #57625:
URL: https://github.com/apache/spark/pull/57625#discussion_r3688709867
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/AutoCdcAuxiliaryTable.scala:
##########
@@ -227,13 +238,28 @@ object AutoCdcAuxiliaryTable {
StructField(Scd2BatchProcessor.deletedByBatchIdColName, LongType,
nullable = true)
val scd2AuxiliaryTableSchema = StructType(targetTableSchema.fields :+
deletedByBatchIdField)
+ // Resolve the effective track-history columns (an explicit TRACK HISTORY
selection, or the
+ // default of every eligible non-key/non-framework column) against the
target schema, using the
+ // same single source of truth the reconciler uses. A change in this set
reinterprets which
+ // transitions open a new historical record, so it is drift-checked.
+ val caseSensitive =
+ inputAutoCdcFlow.df.sparkSession.sessionState.conf.caseSensitiveAnalysis
Review Comment:
Fixed both. validateNoTargetSequencingTypeDrift now takes expectedScdType
and looks the sequencing-typed inner field up by name —
Scd1BatchProcessor.cdcUpsertSequenceFieldName for SCD1,
Scd2BatchProcessor.recordStartAtFieldName for SCD2 — instead of fields.head, so
a metadata field added at position 0 later can't silently compare an unrelated
type. And the _cdc_metadata column and the inner field are both matched through
the session Resolver now, consistent with findFieldInTargetSchema, so a
case-differing hand-written target DDL no longer skips the check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
