cxzl25 opened a new pull request, #44982:
URL: https://github.com/apache/spark/pull/44982
### What changes were proposed in this pull request?
Introduce the configuration `spark.sql.shuffledHashJoinThreshold`, the
default is the maximum value of Long, which can limit the maximum value of the
size converted to SHJ.
### Why are the changes needed?
When we enable `spark.sql.join.preferSortMergeJoin=false`, we may get the
following error.
```java
org.apache.spark.SparkException: Can't acquire 1073741824 bytes memory to
build hash relation, got 478549889 bytes
at
org.apache.spark.sql.errors.QueryExecutionErrors$.cannotAcquireMemoryToBuildLongHashedRelationError(QueryExecutionErrors.scala:795)
at
org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.ensureAcquireMemory(HashedRelation.scala:581)
at
org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.grow(HashedRelation.scala:813)
at
org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.append(HashedRelation.scala:761)
at
org.apache.spark.sql.execution.joins.LongHashedRelation$.apply(HashedRelation.scala:1064)
at
org.apache.spark.sql.execution.joins.HashedRelation$.apply(HashedRelation.scala:153)
at
org.apache.spark.sql.execution.joins.ShuffledHashJoinExec.buildHashedRelation(ShuffledHashJoinExec.scala:75)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage11.init(Unknown
Source)
at
org.apache.spark.sql.execution.WholeStageCodegenExec.$anonfun$doExecute$6(WholeStageCodegenExec.scala:775)
at
org.apache.spark.sql.execution.WholeStageCodegenExec.$anonfun$doExecute$6$adapted(WholeStageCodegenExec.scala:771)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:915)
```
Because when converting SMJ to SHJ, it only determines whether the size of
the plan is smaller than `conf.autoBroadcastJoinThreshold *
conf.numShufflePartitions`.
When the configured `numShufflePartitions` is large enough, it is easy to
convert to SHJ. The executor build hash relation fails due to insufficient
memory.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Production environment verification
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
