leanken commented on a change in pull request #29455:
URL: https://github.com/apache/spark/pull/29455#discussion_r472649111
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
##########
@@ -235,8 +235,13 @@ abstract class SparkStrategies extends
QueryPlanner[SparkPlan] {
.getOrElse(createJoinWithoutHint())
case j @ ExtractSingleColumnNullAwareAntiJoin(leftKeys, rightKeys) =>
- Seq(joins.BroadcastHashJoinExec(leftKeys, rightKeys, LeftAnti,
BuildRight,
- None, planLater(j.left), planLater(j.right), isNullAwareAntiJoin =
true))
+ if (!canBroadcastBySize(j.right, conf) &&
conf.adaptiveExecutionEnabled) {
Review comment:
> Sorry I don't get why we don't worry about shuffled hash join OOM if
sub-query is way too big/skewed in certain keys. In general, shuffled hash join
OOM is a normal issue if people choose shuffled hash join blindly. Could we add
a check to make sure build size is not too big to be shuffled hash join?
>
> e.g.
>
> ```
> if (!canBroadcastBySize(j.right, conf) &&
canBuildLocalHashMapBySize(j.right, conf) && conf.adaptiveExecutionEnabled) {
> ...
> }
> ```
Do you have any contact that I can offline communicate with you maybe, maybe
you can reach me with my WeChat "18518298234"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
