[
https://issues.apache.org/jira/browse/SPARK-29002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343740#comment-17343740
]
Penglei Shi edited comment on SPARK-29002 at 5/13/21, 6:58 AM:
---------------------------------------------------------------
When changing a SMJ to BHJ, there will be an additional BroadcastQueryStageExec
for build side, the new broadcast query stage contains a
CustomShuffleReaderExec after applying queryStageOptimizerRules, and the
CoalesceShufflePartitions in finalStageOptimizerRules will not work. Am i
correct? And on the other hand, i think it is not proper to coalesce partitions
for probe side in BHJ.
!image-2021-05-13-12-12-23-530.png|width=524,height=376!
was (Author: penglei shi):
When changing a SMJ to BHJ, there will be an additional BroadcastQueryStageExec
for build side, the new broadcast query stage contains a
CustomShuffleReaderExec after applying queryStageOptimizerRules, and the
CoalesceShufflePartitions in finalStageOptimizerRules will not work. Am i
correct?
!image-2021-05-13-12-12-23-530.png|width=524,height=376!
> Avoid changing SMJ to BHJ if the build side has a high ratio of empty
> partitions
> --------------------------------------------------------------------------------
>
> Key: SPARK-29002
> URL: https://issues.apache.org/jira/browse/SPARK-29002
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Wei Xue
> Assignee: Wei Xue
> Priority: Major
> Fix For: 3.0.0
>
> Attachments: image-2021-05-13-12-12-23-530.png
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
