szehon-ho commented on code in PR #57488:
URL: https://github.com/apache/spark/pull/57488#discussion_r3648742822
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala:
##########
@@ -375,6 +377,52 @@ class AutoCdcMergeFlow(
}
}
+ /**
+ * Validate that the resolved source dataframe does not contain any column
whose name collides
+ * (by exact name, resolver-aware) with an SCD-type-specific reserved
framework column that is
+ * NOT covered by [[requireReservedPrefixAbsentInSourceColumns]].
+ *
+ * The prefix guard above only rejects names starting with
[[AutoCdcReservedNames.prefix]].
+ * SCD2 additionally persists the framework columns
[[Scd2BatchProcessor.startAtColName]] and
+ * [[Scd2BatchProcessor.endAtColName]], which do NOT carry that prefix, so a
colliding source
+ * column would otherwise be silently overwritten during preprocessing
(SPARK-57251). SCD1
Review Comment:
Nit; can we get rid of the SPARK JIRA in these comments (they show up quite
a few times and dont bring up much value.) its quite easy now to see which pr
did it.
And overall can we simplify the comments to reduce bloat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
