Leanken.Lin created SPARK-32765:
-----------------------------------
Summary: EliminateJoinToEmptyRelation should respect exchange
behavior when canChangeNumPartitions == false
Key: SPARK-32765
URL: https://issues.apache.org/jira/browse/SPARK-32765
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.0
Environment: Currently, EliminateJoinToEmptyRelation Rule will convert
Join into EmptyRelation in some cases with AQE on. But if either sub plan of
Join contains a ShuffleQueryStage(canChangeNumPartitions == false), which means
the Exchange produced by repartition Or singlePartition, in this case, if we
were to convert it into an EmptyRelation, it will lost user specified number
partition information for downstream operator, it's not right.
So in the Patch, try not to do the conversion if either sub plan of Join
contains ShuffleQueryStage(canChangeNumPartitions == false)
Reporter: Leanken.Lin
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
