[
https://issues.apache.org/jira/browse/SPARK-59538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59538:
-----------------------------------
Labels: correctness pull-request-available schema-evolution (was:
correctness schema-evolution)
> [SQL] INSERT column list resolves unequal-arity nested structs by name with
> schema evolution
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-59538
> URL: https://issues.apache.org/jira/browse/SPARK-59538
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Environment: {{Apache Spark 4.2.0 / current master; DSv2 table
> implementing SupportsSchemaEvolution with AUTOMATIC_SCHEMA_EVOLUTION}}
> Reporter: Eames Trinh
> Priority: Major
> Labels: correctness, pull-request-available, schema-evolution
>
> [SPARK-58816](https://issues.apache.org/jira/browse/SPARK-58816) and
> [apache/spark#58086](https://github.com/apache/spark/pull/58086) fixed
> `INSERT INTO table (column_list)` so structs nested inside arrays and maps
> resolve positionally. However, `ResolveInsertionBase.renameFieldsInStruct`
> still skips positional renaming when the source and target structs have
> different numbers of fields.
> This produces incorrect results for column-list inserts into DSv2 tables
> supporting automatic schema evolution.
> ### Reproduction
> Given a DSv2 provider implementing `SupportsSchemaEvolution` and advertising
> `AUTOMATIC_SCHEMA_EVOLUTION`:
> ```sql
> CREATE TABLE t (s STRUCT<x: INT, y: INT>) USING provider;
> INSERT WITH SCHEMA EVOLUTION INTO t (s)
> SELECT named_struct('y', 1, 'x', 2, 'z', 3);
> ```
> ### Expected result
> The column list controls top-level reordering, while nested fields resolve
> positionally:
> ```text
> {x=1, y=2, z=3}
> ```
> ### Actual result
> ```text
> {x=2, y=1, z=3}
> ```
> ### Cause
> `renameFieldsInStruct` currently only renames fields when the source and
> target have equal arity:
> ```scala
> if (input.length == expected.length) {
> // Rename fields positionally.
> } else {
> input
> }
> ```
> Because the source has three fields and the target has two, Spark preserves
> the source field names. The column-list insert is subsequently represented as
> a by-name write, so the existing fields are resolved by name instead of
> position.
> ### Proposed fix
> Rename the overlapping source and target fields positionally while preserving
> unmatched source fields for schema evolution:
> ```scala
> val renamedFields = input.zip(expected).map { case (inputField,
> expectedField) =>
> inputField.copy(
> name = expectedField.name,
> dataType = renameFieldsInType(inputField.dataType,
> expectedField.dataType))
> }
> StructType(renamedFields ++ input.drop(expected.length))
> ```
> When schema evolution is disabled, the existing downstream validation will
> continue to reject extra fields.
> A non-Delta regression test can be added to `DataSourceV2SQLSuite` using
> Spark's in-memory DSv2 table with automatic schema evolution enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
