Li Xian created SPARK-39429:
-------------------------------
Summary: Convert Inner Join With Aggregate To Semi Join
Key: SPARK-39429
URL: https://issues.apache.org/jira/browse/SPARK-39429
Project: Spark
Issue Type: Improvement
Components: Optimizer
Affects Versions: 3.2.1
Reporter: Li Xian
Inner Join with aggregation on one of its child it equivalent to a Semi Join
with the aggregation eliminated If the aggregation is just doing grouping and
all the grouping columns appears in the join condition.
For example, Inner Join like below
{code:java}
select * from table1 inner join (select key2 from table2 group by key2)
subquery on table1.key1 = subquery.key2{code}
can be transformed to a semi join and eliminate the aggregate in the subquery
{code:java}
select * from table1 semi join (select key2 from table2) subquery on
table1.key1 = subquery.key2
{code}
I think this can be added as a optimize rule to save the unnecessary
aggregation in the subquery.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
