dongjoon-hyun opened a new pull request, #59919: URL: https://github.com/apache/spark/pull/59919
### What changes were proposed in this pull request? This PR makes `ResolveSchemaEvolution` remap references to the metadata columns of the write target when it replaces the target after evolving the table schema. `ResolveSchemaEvolution.replaceWriteTarget` replaces the target with `relation.copy(table = newTable, output = newOutput)` and rewrites references to the old output with `rewriteAttrs`. However, `DataSourceV2Relation.metadataOutput` is a lazy val that is derived from the table, so the new relation exposes its metadata columns with new expression IDs, while the MERGE condition, clause conditions and assignments that were resolved before the evolution still refer to the metadata columns of the old relation. When `RewriteMergeIntoTable` reads the metadata columns from the new relation, those references are left dangling. The new `rewriteMetadataAttrs` rewrites them to the metadata columns of the new relation. Metadata columns are matched by their logical name (`MetadataAttributeWithLogicalName`), which stays the same even if a data column of the new table renames a conflicting metadata column. References to metadata columns that the new relation does not expose are left unchanged. This PR also adds `DeltaBasedMergeIntoSchemaEvolutionUpdateAsDeleteAndInsertSQLSuite`, which runs the MERGE schema evolution SQL tests for delta-based operations that split updates into deletes and inserts, as `DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite` does for the MERGE tests. ### Why are the changes needed? `MERGE WITH SCHEMA EVOLUTION` fails when the statement evolves the target schema and references a metadata column of the target table. For instance, with the in-memory row-level table of `RowLevelOperationSuiteBase` (`pk INT NOT NULL, salary INT, dep STRING`, partitioned by `dep`) and a source with an extra column `active`: ```sql MERGE WITH SCHEMA EVOLUTION INTO cat.ns1.test_table t USING source s ON t.pk = s.pk WHEN MATCHED AND t._partition = 'hr' THEN UPDATE SET * ``` ``` [MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_APPEAR_IN_OPERATION] Resolved attribute(s) "_partition" missing from "pk", "salary", "dep", "active", "_partition", "index", ... in operator MergeRows[...]. Attribute(s) with the same name appear in the operation: "_partition". ``` The same statement works without `WITH SCHEMA EVOLUTION`, and `MERGE WITH SCHEMA EVOLUTION` works when no schema change is needed. The failure happens with group-based and delta-based operations (including split updates and operations that require no metadata columns), with the SQL and DataFrame APIs, and wherever the metadata column is referenced: the ON condition, a WHEN MATCHED or WHEN NOT MATCHED BY SOURCE condition, or an assignment value. It also happens when the evolution only widens a column type. ### Does this PR introduce _any_ user-facing change? Yes. `MERGE WITH SCHEMA EVOLUTION` statements that reference metadata columns of the target table now succeed instead of failing with `MISSING_ATTRIBUTES`. By code reading, MERGE schema evolution has remapped only the output of the replaced target since it was added in Apache Spark [4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11). ### How was this patch tested? Added three tests to `MergeIntoSchemaEvolutionExtraSourceColumnTests` (an update condition with `UPDATE SET *`, the same with `INSERT *`, and the ON and WHEN NOT MATCHED BY SOURCE conditions with an explicit assignment to the new column). They run with and without the evolution clause in `GroupBasedMergeIntoSchemaEvolutionSQLSuite`, `GroupBasedMergeIntoSchemaEvolutionScalaSuite`, `DeltaBasedMergeIntoSchemaEvolutionSQLSuite`, `DeltaBasedMergeIntoSchemaEvolutionScalaSuite` and the new `DeltaBasedMergeIntoSchemaEvolutionUpdateAsDeleteAndInsertSQLSuite`. Without the fix, the tests with the evolution clause fail with `MISSING_ATTRIBUTES` in all five suites. ``` build/sbt "sql/testOnly *MergeIntoSchemaEvolution*Suite -- -z SPARK-60709" ``` Also ran the DSv2 DELETE, UPDATE and MERGE suites (including the NoMetadata, split update and NoCodegen variants), the MERGE and INSERT schema evolution suites, `MetadataColumnSuite` and `PlanResolutionSuite` locally. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
