rajatrj20 commented on PR #45350:
URL: https://github.com/apache/spark/pull/45350#issuecomment-2089760654
@cloud-fan This change broke an existing behaviour. When a aliased generator
field A is referenced in some another field B in project list, it will create a
situation where the B will wait for A to resolve as it needs to know the field
identifier of A and A will wait for B to resolve since we have check in place
which needs all fields other than generator field to be resolved.
This is how you can reproduce this:
```
CREATE TABLE nestedTable1 (array_int_col array<int>, array_str_col
array<string>, map_str_col map<string,string>, map_int_col map<string,int>,
struct_col struct<col_3:string,col_1:int>, col_3 string, col_1 int) USING
iceberg
INSERT INTO nestedTable1 (array_int_col, array_str_col, map_str_col,
map_int_col, struct_col, col_3, col_1) VALUES (array(1, 2, 3, 4), array('a',
'b', 'c'), map('k1', 'v1', 'k2', 'v2'), map('k', 1), struct('a', 1), 'a', 1)
SELECT col_1, EXPLODE(MAP_KEYS(map_str_col)) AS key, map_str_col[key] AS
value FROM nestedTable1;
```
This results in following final analyzed plan:
```
'Project [col_1#192, 'EXPLODE(map_keys(map_str_col#188)) AS key#329,
map_str_col#188[lateralAliasReference(key)] AS value#330]
+- RelationV2[array_int_col#186, array_str_col#187, map_str_col#188,
map_int_col#189, struct_col#190, col_3#191, col_1#192]
spark_catalog.default.nestedtable1
```
Without this change the analyzed plan looks like:
```
'Project [col_1#192, key#331, map_str_col#188['key] AS value#330]
+- Generate explode(map_keys(map_str_col#188)), false, [key#331]
+- RelationV2[array_int_col#186, array_str_col#187, map_str_col#188,
map_int_col#189, struct_col#190, col_3#191, col_1#192]
spark_catalog.default.nestedtable1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
