szehon-ho opened a new pull request, #53360: URL: https://github.com/apache/spark/pull/53360
### What changes were proposed in this pull request? The 'struct coercion' feature for MERGE INTO (allowing it to pass if assigning a struct with less fields into a struct with more fields) is turned off in a flag in https://github.com/apache/spark/pull/53229, but was not removed because the community wanted to try it. We want to still keep it under a flag, but we make a choice about which behavior to support when the flag is on. In particular, we want UPDATE SET * to explode to all nested struct fields, so that in this scenario, existing nested struct fields are preserved. ### Why are the changes needed? @aokolnychyi tested the feature and thinks that even if it is behind the experimental flag, we should take the stance for now that UPDATE SET * should explode to all nested fields vs top level columns. The rationale being: * its always safer to not override user values with null * Spark in general tries to treat nested fields like columns * there's already a way for the user to override the whole struct with null, in particular by specifying the struct explicitly, ie UPDATE SET struct = source.struct ### Does this PR introduce _any_ user-facing change? No, the whole feature is new and hidden behind an experimental flag. ### How was this patch tested? Existing tests (some output changes to not be null) ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
