Rui Wang created SPARK-44082:
--------------------------------
Summary: Generate operator does not update reference set properly
Key: SPARK-44082
URL: https://issues.apache.org/jira/browse/SPARK-44082
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.5.0
Reporter: Rui Wang
Assignee: Rui Wang
Before
```
== Optimized Logical Plan ==
Project [col1#2, col2#19]
+- Generate replicaterows(sum#17L, col1#2, col2#3), [2], false, [col1#2, col2#3]
+- Filter (isnotnull(sum#17L) AND (sum#17L > 0))
+- Aggregate [col1#2, col2#19], [col1#2, col2#19, sum(vcol#14L) AS
sum#17L]
+- Union false, false
:- Aggregate [col1#2], [1 AS vcol#14L, col1#2, first(col2#3, false)
AS col2#19]
: +- LogicalRDD [col1#2, col2#3], false
+- Project [-1 AS vcol#15L, col1#8, col2#9]
+- LogicalRDD [col1#8, col2#9], false
```
Couldn't find col2#3 in [col1#2,col2#19,sum#17L]
after
```
== Optimized Logical Plan ==
Project [col1#2, col2#19]
+- Generate replicaterows(sum#17L, col1#2, col2#19), [2], false, [col1#2,
col2#19]
+- Filter (isnotnull(sum#17L) AND (sum#17L > 0))
+- Aggregate [col1#2, col2#19], [col1#2, col2#19, sum(vcol#14L) AS
sum#17L]
+- Union false, false
:- Aggregate [col1#2], [1 AS vcol#14L, col1#2, first(col2#3, false)
AS col2#19]
: +- LogicalRDD [col1#2, col2#3], false
+- Project [-1 AS vcol#15L, col1#8, col2#9]
+- LogicalRDD [col1#8, col2#9], false
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
