HyukjinKwon commented on a change in pull request #23670: [SPARK-26601][SQL]
Make broadcast-exchange thread pool configurable
URL: https://github.com/apache/spark/pull/23670#discussion_r283096710
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
##########
@@ -132,6 +132,19 @@ object StaticSQLConf {
.intConf
.createWithDefault(1000)
+ val BROADCAST_EXCHANGE_MAX_THREAD_THREASHOLD =
+ buildStaticConf("spark.sql.broadcastExchange.maxThreadThreshold")
+ .internal()
+ .doc("The maximum degree of parallelism to fetch and broadcast the
table. " +
+ "If we encounter memory issue like frequently full GC or OOM when
broadcast table " +
+ "we can decrease this number in order to reduce memory usage. " +
+ "Notice the number should be carefully chosen since decreasing
parallelism might " +
+ "cause longer waiting for other broadcasting. Also, increasing
parallelism may " +
+ "cause memory problem.")
+ .intConf
+ .checkValue(thres => thres > 0 && thres <= 128, "The threshold should be
positive.")
Review comment:
Yea let's improve this one before getting this in
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
