imonteroq opened a new pull request, #59299: URL: https://github.com/apache/spark/pull/59299
### What changes were proposed in this pull request? This PR adds a new configuration `spark.sql.schemaEvolution.preserveColumnOrder`. When enabled, fields added to a table by schema evolution keep the position they occupy in the source schema instead of being appended at the end. This applies to `MERGE INTO` and `INSERT INTO ... WITH SCHEMA EVOLUTION`, to top-level columns and to nested struct fields. When the flag is on, schema evolution emits `AddColumn` table changes carrying a `ColumnPosition`: `first()` when the new field is the first of its source siblings, otherwise `after()` the nearest preceding source sibling that exists in the target or that is itself added earlier in the same change list. Positioned adds are emitted for assignments that add missing columns directly (`SET new_col = s.new_col`, `INSERT (new_col) VALUES (s.new_col)`, star actions) and for fields added inside structs through whole-struct assignment, via `ResolveSchemaEvolution.computeSchemaChanges`, so the INSERT path behaves consistently. Two supporting fixes are included: - `ResolveSchemaEvolution.replaceWriteTarget` now maps rewritten attributes by name instead of position. With positioned adds the output ordinals of later columns shift, and positional mapping rewrote an attribute to the wrong counterpart, producing spurious type-change entries (e.g. a bogus `ALTER COLUMN dep TYPE STRING` or a `failedToMergeIncompatibleSchemas` error). - The in-memory test catalogs (`InMemoryBaseTable`, `InMemoryRowLevelOperationTable`) now extract partition keys using the stored row schema for positions and the evolved schema for value types, and read statistics columns by name. Previously they resolved a field's index in the evolved schema and read it from rows stored under the old schema, which broke once a partition column's stored index could shift; reading widened values with the evolved type also keeps partition keys consistent with the ordering reported by `SupportsReportPartitioning`. ### Why are the changes needed? Schema evolution appends every new column at the end of the target schema. For tables where column order carries meaning (aligned with a source table or an external contract), reordered output is awkward; the new option lets added columns keep their source positions. ### Does this PR introduce _any_ user-facing change? Yes. A new SQL configuration `spark.sql.schemaEvolution.preserveColumnOrder`, default `false`, which preserves the current behavior (new columns are appended). When set to `true`, columns added by schema evolution keep their source positions. ### How was this patch tested? New `MergeIntoSchemaEvolutionColumnOrderTests` run inside all four merge evolution suites (group/delta based, SQL and DataFrame API), covering middle/first/consecutive adds, explicit and star assignments, nested struct fields, case-insensitive anchors, and partitioned tables. New cases in `InsertIntoSchemaEvolutionTests` cover the `INSERT ... WITH SCHEMA EVOLUTION` path in `DataSourceV2SQLSuite`. The existing suites pass unchanged with the flag off. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Devin (Cognition) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
