[ https://issues.apache.org/jira/browse/SPARK-32765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17188709#comment-17188709 ]
Apache Spark commented on SPARK-32765: -------------------------------------- User 'leanken' has created a pull request for this issue: https://github.com/apache/spark/pull/29614 > EliminateJoinToEmptyRelation should respect exchange behavior when > canChangeNumPartitions == false > -------------------------------------------------------------------------------------------------- > > Key: SPARK-32765 > URL: https://issues.apache.org/jira/browse/SPARK-32765 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.0.0 > Reporter: Leanken.Lin > Priority: Major > > Currently, EliminateJoinToEmptyRelation Rule will convert Join into > EmptyRelation in some cases with AQE on. But if either sub plan of Join > contains a ShuffleQueryStage(canChangeNumPartitions == false), which means > the Exchange produced by repartition Or singlePartition, in this case, if we > were to convert it into an EmptyRelation, it will lost user specified number > partition information for downstream operator, it's not right. > So in the Patch, try not to do the conversion if either sub plan of Join > contains ShuffleQueryStage(canChangeNumPartitions == false) -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org