anew opened a new pull request, #57488:
URL: https://github.com/apache/spark/pull/57488
### What changes were proposed in this pull request?
What changes were proposed in this pull request?
AutoCdcMergeFlow validates at construction time that a flow's source
change-data feed does not carry columns that collide with AutoCDC's internal
columns. Until now, that validation
(requireReservedPrefixAbsentInSourceColumns) only rejected column names
starting with the reserved prefix __spark_autocdc_.
SCD2, however, persists two framework columns to the target that do not
carry that prefix — __START_AT and __END_AT (see
Scd2BatchProcessor.reservedFrameworkColNames). A source column named __START_AT
or __END_AT therefore slipped past the guard and would be silently overwritten
during microbatch preprocessing.
This PR closes that gap (the TODO(SPARK-57251) in Scd2BatchProcessor):
- Adds requireReservedFrameworkColumnsAbsentInSourceColumns() to the
AutoCdcMergeFlow constructor. For SCD2 flows it rejects any source column
whose name collides (by exact name, resolver-aware so it respects
spark.sql.caseSensitive) with a non-prefixed reserved framework column. SCD1
targets carry no such columns, so the check is a no-op for SCD1.
- Runs the check before the flow's schema val is forced, so the actionable
reserved-name error surfaces ahead of the temporary AUTOCDC_SCD2_NOT_SUPPORTED
gate, and the check remains correct once SCD2 support lands.
- Adds a new error condition AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT
(SQLSTATE 42710), distinct from the existing prefix-based
AUTOCDC_RESERVED_COLUMN_NAME_PREFIX_CONFLICT, since this collision is by exact
name rather than by prefix.
- Widens the visibility of Scd2BatchProcessor.reservedFrameworkColNames to
private[pipelines] so the flow layer can validate against the single source of
truth.
### Why are the changes needed?
Without this check, a user whose CDC source happens to contain a __START_AT
or __END_AT column would have that data silently overwritten by AutoCDC's SCD2
framework columns, with no error and no diagnostic — a data-correctness
footgun. Failing fast at flow construction with a user-actionable error
("rename or remove the column") is the intended UX, consistent with the
existing reserved-prefix guard.
### Does this PR introduce _any_ user-facing change?
Yes. An AutoCDC SCD2 flow whose source change-data feed contains a column
named __START_AT or __END_AT (subject to case-sensitivity settings) now fails
at flow construction with AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT instead of
silently overwriting the column. There is no change for SCD1 flows, and no
change for SCD2 flows whose sources do not use these names. (Note: SCD2 AutoCDC
flows are not yet generally supported on master — they are still gated by
AUTOCDC_SCD2_NOT_SUPPORTED — so no released behavior changes.)
### How was this patch tested?
New unit tests in AutoCdcFlowSuite covering:
- an SCD2 flow with a __START_AT/__END_AT source column is rejected with
AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT;
- the reserved-name check fires before the AUTOCDC_SCD2_NOT_SUPPORTED
gate;
- an SCD1 flow with the same column name is allowed and the column
survives into the flow schema;
- case-sensitivity behavior (spark.sql.caseSensitive true/false);
- a guard test asserting the set of non-prefixed reserved names is exactly
{__START_AT, __END_AT}, so a future rename can't silently un-cover the
validation.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
