Anish Mahto created SPARK-59269:
-----------------------------------
Summary: [SDP] Support `ColumnDefaultValue` changes in schema in
`SchemaInferenceUtils.diffSchemas`
Key: SPARK-59269
URL: https://issues.apache.org/jira/browse/SPARK-59269
Project: Spark
Issue Type: Bug
Components: Declarative Pipelines
Affects Versions: 4.2.0
Reporter: Anish Mahto
Today, `SchemaInferenceUtils.diffSchemas` recursively applies changes to a
schema field's data type, comment, and nullability.
We should expand it to also respect changes to a field's default value. Note
that `TableChange.addColumn` now supports specifying the default value, and
`TableChange.updateColumnDefaultValue` exists to update an existing column's
default value.
Care should be taken to respect both the current default value and existing
default values.
Note that for this to work E2E for a pipeline, this ticket is blocked by
https://issues.apache.org/jira/browse/SPARK-59268.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
