EamesTrinh commented on PR #58831:
URL: https://github.com/apache/spark/pull/58831#issuecomment-5920901497
> Maybe this behavior should be changed instead of introducing changes to
schema evolution?
Of the two options changing all non-schema evolution related resolution
orderings is certainly the much larger change.
Moreover, this behavior:
> This produces {x: 1, y: 2} because the nested fields are resolved
positionally.
is, and always has been, the correct behavior. The reason is that in a
column list insert, the target schema is matched to the column-list by name.
THe target schema for that column is then (recursively and) _positionally_
applied onto the source schema because the source schema is actually under no
obligation to have any matching names at all.
For example, take even a simple, non-nested valid insert operation like:
```
CREATE TABLE t (a INT, b INT) USING <v2-provider>;
INSERT INTO t (b, a)
SELECT x, y FROM source;
```
Notice the target columns a, b have to have by-name matches in the column
list (b, a), but there is no such requirement for the source table. It can be
anything. Therefore, for the insert to work, the (b, a) _must_ be positionally
applied onto the source schema (obviously, if we left it as x, y, the source
would insert nonsense (and would fail)).
Schema evolution inserts should respect these rules as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
