[
https://issues.apache.org/jira/browse/SPARK-59164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59164:
-----------------------------------
Labels: pull-request-available (was: )
> Support nested column schema evolution
> --------------------------------------
>
> Key: SPARK-59164
> URL: https://issues.apache.org/jira/browse/SPARK-59164
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.4.0
> Reporter: Anish Mahto
> Priority: Major
> Labels: pull-request-available
>
> `SchemaInferenceUtils.diffSchemas` is used to deduce the `TableChanges` that
> need to be sent to the catalog in order to reflect the latest schema evolved
> state of datasets declared by a pipeline, during dataset materialization.
> Today however, the diff does not support nested column level evolution. If
> some nested column (ex. struct type) has one of its sub-fields changed
> between runs, but the column's top level name stays the same, it does not
> emit the correct (or any) column add or delete changes to the catalog.
> The fix is to recursively diff the schema, traversing every nested column as
> necessary. This is similar to the existing
> `ResolveSchemaEvolution.computeSchemaChanges`, except it is not just additive
> schema evolution - it supports dropping columns in the latest dataset's
> schema declaration.
> Two things intentionally not handled by this change:
> 1. Diff'ing changes to a nested columns `metadata`. That was not respected
> before, and continues to not be respected after these changes, because DSv2
> does not yet support `TableChanges` types for changes to column metadata
> 2. Respecting order that columns are defined in between the existing and new
> schemas. Ex. if the exact same column was previously declared as the first
> column in the schema, but now is declared as the second column in the schema.
> Instead, existing behavior is preserved; column equality is position
> invariant.
> This is a general improvement to SDP's schema evolution infrastructure, but
> is being tracked as part of the AutoCDC ignore-null implementation, which
> will start requiring AutoCDC's `_cdc_metadata` column to support schema
> evolution for a nested version map field.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
