[jira] [Commented] (SPARK-32765) EliminateJoinToEmptyRelation should respect exchange behavior when canChangeNumPartitions == false
[ https://issues.apache.org/jira/browse/SPARK-32765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188709#comment-17188709 ] Apache Spark commented on SPARK-32765: -- User 'leanken' has created a pull request for this issue: https://github.com/apache/spark/pull/29614 > EliminateJoinToEmptyRelation should respect exchange behavior when > canChangeNumPartitions == false > -- > > Key: SPARK-32765 > URL: https://issues.apache.org/jira/browse/SPARK-32765 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0 >Reporter: Leanken.Lin >Priority: Major > > Currently, EliminateJoinToEmptyRelation Rule will convert Join into > EmptyRelation in some cases with AQE on. But if either sub plan of Join > contains a ShuffleQueryStage(canChangeNumPartitions == false), which means > the Exchange produced by repartition Or singlePartition, in this case, if we > were to convert it into an EmptyRelation, it will lost user specified number > partition information for downstream operator, it's not right. > So in the Patch, try not to do the conversion if either sub plan of Join > contains ShuffleQueryStage(canChangeNumPartitions == false) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32765) EliminateJoinToEmptyRelation should respect exchange behavior when canChangeNumPartitions == false
[ https://issues.apache.org/jira/browse/SPARK-32765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188706#comment-17188706 ] Apache Spark commented on SPARK-32765: -- User 'leanken' has created a pull request for this issue: https://github.com/apache/spark/pull/29614 > EliminateJoinToEmptyRelation should respect exchange behavior when > canChangeNumPartitions == false > -- > > Key: SPARK-32765 > URL: https://issues.apache.org/jira/browse/SPARK-32765 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0 >Reporter: Leanken.Lin >Priority: Major > > Currently, EliminateJoinToEmptyRelation Rule will convert Join into > EmptyRelation in some cases with AQE on. But if either sub plan of Join > contains a ShuffleQueryStage(canChangeNumPartitions == false), which means > the Exchange produced by repartition Or singlePartition, in this case, if we > were to convert it into an EmptyRelation, it will lost user specified number > partition information for downstream operator, it's not right. > So in the Patch, try not to do the conversion if either sub plan of Join > contains ShuffleQueryStage(canChangeNumPartitions == false) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
