anew commented on code in PR #57488:
URL: https://github.com/apache/spark/pull/57488#discussion_r3648896946
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala:
##########
@@ -375,6 +377,52 @@ class AutoCdcMergeFlow(
}
}
+ /**
+ * Validate that the resolved source dataframe does not contain any column
whose name collides
+ * (by exact name, resolver-aware) with an SCD-type-specific reserved
framework column that is
+ * NOT covered by [[requireReservedPrefixAbsentInSourceColumns]].
+ *
+ * The prefix guard above only rejects names starting with
[[AutoCdcReservedNames.prefix]].
+ * SCD2 additionally persists the framework columns
[[Scd2BatchProcessor.startAtColName]] and
+ * [[Scd2BatchProcessor.endAtColName]], which do NOT carry that prefix, so a
colliding source
+ * column would otherwise be silently overwritten during preprocessing
(SPARK-57251). SCD1
Review Comment:
done. This was Claude imitating other unit tests that also have the JIRA id,
but you are right, that convention is now not relly needed.
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala:
##########
@@ -375,6 +377,52 @@ class AutoCdcMergeFlow(
}
}
+ /**
+ * Validate that the resolved source dataframe does not contain any column
whose name collides
+ * (by exact name, resolver-aware) with an SCD-type-specific reserved
framework column that is
+ * NOT covered by [[requireReservedPrefixAbsentInSourceColumns]].
+ *
+ * The prefix guard above only rejects names starting with
[[AutoCdcReservedNames.prefix]].
+ * SCD2 additionally persists the framework columns
[[Scd2BatchProcessor.startAtColName]] and
+ * [[Scd2BatchProcessor.endAtColName]], which do NOT carry that prefix, so a
colliding source
+ * column would otherwise be silently overwritten during preprocessing
(SPARK-57251). SCD1
Review Comment:
done. This was Claude imitating other unit tests that also have the JIRA id,
but you are right, that convention is now not really needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
