AnishMahto commented on code in PR #56122:
URL: https://github.com/apache/spark/pull/56122#discussion_r3308504879
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala:
##########
@@ -303,6 +303,23 @@ object DatasetManager extends Logging {
context.spark.sql(s"TRUNCATE TABLE ${table.identifier.quotedString}")
}
+ if (isFullRefresh) {
+ // On full refresh, drop the AutoCDC auxiliary state associated with
this table (if any) so
+ // that stale delete-tracking data and table properties are not carried
forward into the new
+ // table generation. We unconditionally issue the DROP for every
fully-refreshed target; for
+ // non-AutoCDC tables this is a no-op because
[[AutoCdcAuxiliaryTable.identifier]] derives
+ // its name from [[AutoCdcReservedNames.prefix]], which is reserved by
AutoCDC and
+ // therefore cannot collide with a user-managed table.
Review Comment:
I went with (a) and softened the wording both here and in
`AutoCdcAuxiliaryTable.identifier` to make it clear that any table with the
same name as what we derive is always assumed to an auxiliary table, and
managed appropriately by the running pipeline.
I did not go with (b) because:
- adding validation to all tables in the dataflow graph incurs a non-trivial
cost, there can be thousands of tables
- the validation doesn't properly protect user tables anyway. There can be a
user table created outside of this dataflow graph (ex. by another pipeline or
just manually created) that shares the same name. If we introduced the concept
of pipeline<->table ownership, then we _could_ throw if there already exists a
table with the constructed name that does not belong to this pipeline, but that
concept does not exist today.
- we don't necessarily want to even block users from creating a table with
the same name as what we would deduce for AutoCDC flows; users may
intentionally want to manually tweak or prefill the aux table. The analogy here
to me is Spark streaming checkpoints; users are free to create their own
streaming checkpoint directories and hook it up to a new streaming query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
