szehon-ho opened a new pull request, #53149: URL: https://github.com/apache/spark/pull/53149
### What changes were proposed in this pull request? Introduce a new flag spark.sql.merge.nested.type.assign.by.field that allows UPDATE SET * action in MERGE INTO to be shorthand to assign every nested struct (ie, UPDATE SET a.b.c = source.a.b.c). This will preserve existing struct field in the target table that has no source equivalent, when the corresponding source struct has less fields than target. Additional code is added to prevent null expansion in this case (ie, a null source struct expanding to a struct of nulls). ### Why are the changes needed? Following https://github.com/apache/spark/pull/52866, we now allow MERGE INTO to have a source table struct with less nested fields than target table struct. In this scenario, a user making a UPDATE SET * is the action may want to retain these nested fields in target. If we interpret UPDATE SET * as shorthand to assign every top-column level field, then the target struct is set to source struct object as is, with missing fields nullified. Instead, they may mean that UPDATE SET * is short-hand to assign every nested struct field (ie, UPDATE SET a.b.c = source.a.b.c), in which case the target struct fields missing in source are retained. ### Does this PR introduce _any_ user-facing change? No, the support to allow source structs to have less fields than target structs in MERGE INTO is unreleased yet, and in any case there is a flag. ### How was this patch tested? Unit tests, especially around cases where the source struct is null. ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
