viirya commented on a change in pull request #23670: [SPARK-26601][SQL] Make
broadcast-exchange thread pool configurable
URL: https://github.com/apache/spark/pull/23670#discussion_r283082708
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
##########
@@ -132,6 +132,19 @@ object StaticSQLConf {
.intConf
.createWithDefault(1000)
+ val BROADCAST_EXCHANGE_MAX_THREAD_THREASHOLD =
+ buildStaticConf("spark.sql.broadcastExchange.maxThreadThreshold")
+ .internal()
+ .doc("The maximum degree of parallelism to fetch and broadcast the
table. " +
+ "If we encounter memory issue like frequently full GC or OOM when
broadcast table " +
+ "we can decrease this number in order to reduce memory usage. " +
+ "Notice the number should be carefully chosen since decreasing
parallelism might " +
+ "cause longer waiting for other broadcasting. Also, increasing
parallelism may " +
+ "cause memory problem.")
+ .intConf
+ .checkValue(thres => thres > 0 && thres <= 128, "The threshold should be
positive.")
Review comment:
There is a maximum limit (128) for this config. The error message doesn't
reflect that. Once an invalid value like 200 is set, it can't figure out what
goes wrong with it, if not reading this code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
