Eames Trinh created SPARK-58816:
-----------------------------------
Summary: INSERT with a column list resolves structs inside arrays
and maps by name instead of position
Key: SPARK-58816
URL: https://issues.apache.org/jira/browse/SPARK-58816
Project: Spark
Issue Type: Bug
Components: Bug
Affects Versions: 3.3.0
Reporter: Eames Trinh
h3. Problem
For {{{}INSERT INTO table (column_list){}}}, nested structs should be resolved
by position.
Spark currently behaves inconsistently:
* Direct structs are resolved by position.
* Structs inside arrays and maps are resolved by name.
This can silently write values into different fields depending only on where
the struct is nested.
h3. Reproduction
{code:java}
CREATE TABLE target (
s STRUCT<x: INT, y: INT>,
arr ARRAY<STRUCT<x: INT, y: INT>>,
m MAP<STRING, STRUCT<x: INT, y: INT>>
) USING parquet;
INSERT INTO target (s, arr, m)
SELECT
named_struct('y', 20, 'x', 10),
array(named_struct('y', 20, 'x', 10)),
map('k', named_struct('y', 20, 'x', 10));
SELECT
to_json(s) AS direct_struct,
to_json(arr[0]) AS array_struct,
to_json(m['k']) AS map_struct
FROM target; {code}
h3. Expected result
All three structs should be resolved by position: {{{"x":20,"y":10}}}
h3. Actual result
Only the direct struct is resolved by position. For the array and map structs
we get: {{{"x":10,"y":20}}}
h2. Root cause
{{ResolveInsertionBase.createProjectForByNameQuery}} adds a projection that
renames direct struct fields according to their positions. However,
{{renameFieldsInStruct}} only recurses through {{{}StructType{}}}; it does not
descend into {{ArrayType}} or {{{}MapType{}}}.
Consequently, field names inside arrays and maps remain unchanged, and
{{TableOutputResolver}} later resolves those nested structs by name.
h2. Proposed behavior
For {{{}INSERT INTO table (column_list){}}}, struct fields should be resolved
positionally at every nesting level, including:
* Array elements
* Map keys
* Map values
The projection’s field-renaming logic should recursively traverse arrays and
maps, or the resolver should otherwise preserve positional semantics throughout
the nested type.
h2. Impact
Queries can silently write values into the wrong nested fields when source and
target struct field names differ in order. The behavior is also inconsistent
with direct structs and with the positional semantics of {{INSERT INTO}} column
lists.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
