anew commented on code in PR #57488:
URL: https://github.com/apache/spark/pull/57488#discussion_r3648893438
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala:
##########
@@ -375,6 +377,52 @@ class AutoCdcMergeFlow(
}
}
+ /**
+ * Validate that the resolved source dataframe does not contain any column
whose name collides
+ * (by exact name, resolver-aware) with an SCD-type-specific reserved
framework column that is
+ * NOT covered by [[requireReservedPrefixAbsentInSourceColumns]].
+ *
+ * The prefix guard above only rejects names starting with
[[AutoCdcReservedNames.prefix]].
+ * SCD2 additionally persists the framework columns
[[Scd2BatchProcessor.startAtColName]] and
+ * [[Scd2BatchProcessor.endAtColName]], which do NOT carry that prefix, so a
colliding source
+ * column would otherwise be silently overwritten during preprocessing
(SPARK-57251). SCD1
+ * targets carry no such non-prefixed framework columns, so this guard is a
no-op for SCD1.
+ *
+ * Runs in the constructor before [[schema]] is forced, so it surfaces this
actionable error
+ * ahead of the (temporary) [[AUTOCDC_SCD2_NOT_SUPPORTED]] gate, and remains
correct once SCD2
+ * support lands.
+ */
+ private def requireReservedFrameworkColumnsAbsentInSourceColumns(): Unit = {
+ val resolver = spark.sessionState.conf.resolver
+ val reservedPrefix = AutoCdcReservedNames.prefix
+
+ // Only the non-prefixed reserved names need checking here; prefixed ones
are already rejected
+ // by [[requireReservedPrefixAbsentInSourceColumns]].
+ val reservedNames: Set[String] = changeArgs.storedAsScdType match {
+ case ScdType.Type2 =>
+
Scd2BatchProcessor.reservedFrameworkColNames.filterNot(_.startsWith(reservedPrefix))
+ case ScdType.Type1 =>
+ Set.empty
+ }
+
Review Comment:
Kept find as-is. You noted that keeping it "also seems reasonable" since it
matches the prefix guard. Changing only this guard to collect-all would make it
inconsistent with its sibling requireReservedPrefixAbsentInSourceColumns (which
also uses find), and the error message already lists all reserved names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
