Szehon Ho created SPARK-54621:
---------------------------------
Summary: Merge Into Update Set * preserve nested fields if
coerceNestedTypes is enabled
Key: SPARK-54621
URL: https://issues.apache.org/jira/browse/SPARK-54621
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.1.0
Reporter: Szehon Ho
The 'struct coercion' feature for MERGE INTO (allowing it to pass if assigning
a struct with less fields into a struct with more fields) is turned off in a
flag in SPARK-54525, but was not removed because the community wanted to try it.
[~aokolnychyi] tested the feature and thinks that even if it is behind the
experimental flag, we should take the stance for now that UPDATE SET * should
explode to all nested fields vs top level columns. Ie in this case, missing
nested fields are preserved, and not overriden with null if we only explode it
top-level columns.
The rationale being:
* its safer to not override values
* Spark in general tries to treat nested fields like columns
* there's a way for the user to override with null, in particular by specifying
the struct explicitly, ie UPDATE SET struct = source.struct
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
