szehon-ho commented on PR #57644:
URL: https://github.com/apache/spark/pull/57644#issuecomment-5671578591
Thanks — preserving the declared field’s spelling and position fixes the
case where the user continues to declare it. The remaining concern is when an
existing user adopts this PR’s new behavior and removes that declaration.
For example, suppose a case-insensitive AUTO CDC target already contains:
```text
id, version, __SPARK_AUTOCDC_METADATA
```
The user then changes its declaration to the newly supported data-only
schema:
```sql
CREATE STREAMING TABLE target (
id INT,
version BIGINT
);
```
and has a case-sensitive downstream dataset:
```sql
CREATE MATERIALIZED VIEW copy AS
SELECT * FROM target;
```
`VirtualTableInput` plans `copy` using the canonical lowercase
`__spark_autocdc_metadata`, while incremental `evolveTable` preserves the
existing uppercase catalog field. During execution, the case-sensitive
downstream by-name write cannot match those names, even though the AUTO CDC
MERGE itself succeeds.
The fact that the analysis-time read path cannot currently see the catalog
spelling explains the mismatch, but it does not resolve this upgrade path.
Could we cover the omitted-declaration case in this PR? If reconciling the
catalog schema during analysis is not feasible, detecting and rejecting this
legacy mismatch early would still be preferable to a later downstream failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
