szehon-ho opened a new pull request, #52866: URL: https://github.com/apache/spark/pull/52866
### What changes were proposed in this pull request? Change MERGE INTO schema evolution scope. Limit the scope of schema evolution to only add columns/nested fields that are referenced in the MERGE INTO query via UPDATE or INSERT statements. ### Why are the changes needed? https://github.com/apache/spark/pull/51698 added schema evolution support for MERGE INTO statements. However, it is a bit too broad. In some instances, source table may have many more fields than target tables. But user may only need a few new ones to be added to the target for the MERGE INTO statement. ### Does this PR introduce _any_ user-facing change? No, MERGE INTO schema evolution is not yet released in Spark 4.1. ### How was this patch tested? Added many unit tests in MergeIntoTableSuiteBase ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
