Anish Mahto created SPARK-59268:
-----------------------------------
Summary: [SDP] Preserve desired column metadata during additive
schema evolution
Key: SPARK-59268
URL: https://issues.apache.org/jira/browse/SPARK-59268
Project: Spark
Issue Type: Bug
Components: Declarative Pipelines
Affects Versions: 4.2.0
Reporter: Anish Mahto
During incremental dataset evolution, SDP computes the target schema using:
```
SchemaMergingUtils.mergeSchemas(currentSchema, desiredSchema, caseSensitive)
```
Today, `StructType.merge` retains the left/current field metadata and comment
for matching columns. Consequently, changes made in the desired schema such as
modifying, adding, or dropping an existing column’s comment or current default
value, are discarded before `diffSchemas` runs to actually reconcile the
changing schema attributes.
Either `StructType.merge` should update the left field's metadata and comment
sub-fields the same way it already does for `dataType` and `nullable`, SDP
should implement its own more comprehensive merge, or SDP should explicitly
reject changes to an existing column's comment or default value on schema
evolution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
