anew commented on code in PR #57625:
URL: https://github.com/apache/spark/pull/57625#discussion_r3688722225
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala:
##########
@@ -453,6 +468,12 @@ object DatasetManager extends Logging {
targetTableIdentifier = autoCdcSpec.targetTableIdentifier,
expectedScdType = autoCdcSpec.expectedScdType
)
+ AutoCdcAuxiliaryTable.validateNoTrackHistoryDrift(
+ existingAuxiliaryTable = existingAuxiliaryTable,
+ targetTableIdentifier = autoCdcSpec.targetTableIdentifier,
+ expectedTrackHistoryColumnNames =
autoCdcSpec.expectedTrackHistoryColumnNames,
+ resolver = context.spark.sessionState.conf.resolver
+ )
Review Comment:
The recording side now comes from the AutoCdcMergeFlow val (per the other
comment), and the sequencing-drift check's resolver is sourced from
context.spark at the call site, so record and validate are consistent. On the
case-only rename: I covered it as a direct unit test of
validateNoTargetSequencingTypeDrift (a _cdc_metadata/inner-field schema whose
casing differs from the constant, asserting the resolver still finds it and
detects drift) rather than end-to-end — the streaming write always persists the
canonical metadata casing, so a genuine case difference only reaches this
resolver via a hand-written schema, which the unit test constructs directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
