anew commented on code in PR #57625:
URL: https://github.com/apache/spark/pull/57625#discussion_r3688694738
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/AutoCdcAuxiliaryTable.scala:
##########
@@ -227,13 +238,28 @@ object AutoCdcAuxiliaryTable {
StructField(Scd2BatchProcessor.deletedByBatchIdColName, LongType,
nullable = true)
val scd2AuxiliaryTableSchema = StructType(targetTableSchema.fields :+
deletedByBatchIdField)
+ // Resolve the effective track-history columns (an explicit TRACK HISTORY
selection, or the
+ // default of every eligible non-key/non-framework column) against the
target schema, using the
+ // same single source of truth the reconciler uses. A change in this set
reinterprets which
+ // transitions open a new historical record, so it is drift-checked.
+ val caseSensitive =
+ inputAutoCdcFlow.df.sparkSession.sessionState.conf.caseSensitiveAnalysis
Review Comment:
Done — took the val on AutoCdcMergeFlow you sketched, computed from
userSelectedSchema, mirroring the sequencingType line. The aux-spec builder
reads inputAutoCdcFlow.trackHistoryColumnNames, and it subsumes
requireTrackHistoryColumnsResolvableInSelectedSchema, which I removed. That
also resolves the recording-side session concern from your DatasetManager
comment: the set is now recorded once, by the flow, from a single session.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
