[ 
https://issues.apache.org/jira/browse/SPARK-59268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-59268:
-----------------------------------
    Labels: pull-request-available  (was: )

> [SDP] Preserve desired column metadata during additive schema evolution
> -----------------------------------------------------------------------
>
>                 Key: SPARK-59268
>                 URL: https://issues.apache.org/jira/browse/SPARK-59268
>             Project: Spark
>          Issue Type: Bug
>          Components: Declarative Pipelines
>    Affects Versions: 4.2.0
>            Reporter: Anish Mahto
>            Priority: Minor
>              Labels: pull-request-available
>
> During incremental dataset evolution, SDP computes the target schema using:
> ```
> SchemaMergingUtils.mergeSchemas(currentSchema, desiredSchema, caseSensitive)
> ```
> Today, `StructType.merge` retains the left/current field metadata and comment 
> for matching columns. Consequently, changes made in the desired schema such 
> as modifying, adding, or dropping an existing column’s comment or current 
> default value, are discarded before `diffSchemas` runs to actually reconcile 
> the changing schema attributes.
> Either `StructType.merge` should update the left field's metadata and comment 
> sub-fields the same way it already does for `dataType` and `nullable`, SDP 
> should implement its own more comprehensive merge, or SDP should explicitly 
> reject changes to an existing column's comment or default value on schema 
> evolution.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to