anew commented on code in PR #57444:
URL: https://github.com/apache/spark/pull/57444#discussion_r3642374144
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala:
##########
@@ -1166,6 +1477,27 @@ object Scd2BatchProcessor {
*/
private val anchorSequenceColName: String =
s"${AutoCdcReservedNames.prefix}anchor_sequence"
+ /**
+ * Name of the temporary column projected by
[[Scd2BatchProcessor.identifyAndTagAuxRows]] to
+ * mark rows destined for the auxiliary table (tombstones and hidden no-op
upserts) rather than
+ * the target table.
+ *
+ * Temporary in that the column has no observable side effect or persistence
across microbatches.
+ */
+ private[autocdc] val shouldRouteToAuxTableColName: String =
+ s"${AutoCdcReservedNames.prefix}should_route_to_aux_table"
Review Comment:
Yes, this is true for all of the reserved metadata column names, already in
previous PRs. We should make sure at the time of graph registration (or flow
analysis) that no user columns collide with the autocdc metadata prefix. Do you
mind if we do that in a follow-up PR? I created
https://issues.apache.org/jira/browse/SPARK-58313
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
