ulysses-you commented on a change in pull request #32550:
URL: https://github.com/apache/spark/pull/32550#discussion_r639414658
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -598,6 +598,17 @@ object SQLConf {
.bytesConf(ByteUnit.BYTE)
.createOptional
+ val ADAPTIVE_MAX_SHUFFLE_HASH_JOIN_LOCAL_MAP_THRESHOLD =
+ buildConf("spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold")
+ .doc("Configures the maximum size in bytes per partition that can be
allowed to build " +
+ s"local hash map. If this value is bigger than
${ADVISORY_PARTITION_SIZE_IN_BYTES.key} " +
+ s"and all the partition size not larger than
${ADVISORY_PARTITION_SIZE_IN_BYTES.key}, " +
+ "join selection prefer to use shuffled hash join instead of sort merge
join regardless " +
+ s"of the value of ${PREFER_SORTMERGEJOIN.key}.")
+ .version("3.2.0")
+ .bytesConf(ByteUnit.BYTE)
+ .createOptional
Review comment:
If we do `fallbackConf`, many tests will fail since smj changes to shj.
Not sure we should enable this feature by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
