Andreas Neumann created SPARK-58118:
---------------------------------------
Summary: Allow a user-specified AUTO CDC streaming-table schema to
omit the reserved metadata column
Key: SPARK-58118
URL: https://issues.apache.org/jira/browse/SPARK-58118
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.2.0, 4.3.0
Reporter: Andreas Neumann
**Description:**
An AUTO CDC (SCD Type 1) flow appends a reserved metadata column named
`__spark_autocdc_metadata` (a struct of `deleteSequence`/`upsertSequence`) to
its output schema(`AutoCdcMergeFlow.schema`). Because
`validateUserSpecifiedSchemas` requires the user-declared schema to *exactly
equal* the inferred schema, a user who declares a schema on an AUTO CDC
streaming table is currently forced to also declare this internal reserved
column, e.g.:
```
CREATE STREAMING TABLE t (
id INT NOT NULL, name STRING, version BIGINT NOT NULL,
__spark_autocdc_metadata
STRUCT<deleteSequence:BIGINT,upsertSequence:BIGINT> NOT NULL
) FLOW AUTO CDC ...
```
That column is an engine implementation detail; users should not need to know
its name or shape. Declaring only the logical data columns should be accepted,
with the engine owning the metadata column.
Note: simply relaxing the validation is insufficient.
`DatasetManager.materializeTable` uses `table.specifiedSchema` verbatim as the
created-table schema, so if the user omits the metadata column the target table
is created without it and the SCD1 MERGE fails at runtime with
`UNRESOLVED_COLUMN ... __spark_autocdc_metadata` (verified empirically). The
fix must both:
1. Relax `validateUserSpecifiedSchemas` to permit the declared schema to differ
from the inferred schema *only* by the reserved AUTO CDC metadata column(s), and
2. Backfill the reserved metadata column into the effective output schema at
materialization when a user schema is specified for an AUTO CDC target, so the
created table actually contains it.
This is a deliberate semantics decision ("the user schema describes the logical
table; the engine owns reserved columns") and should be designed with the SDP
schema-model owners. Subsumes the ergonomic wart where the combined form forces
declaring the reserved column.
Note: this came up during the review of SPARK-57402
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
