maropu commented on a change in pull request #23670: [SPARK-26601][SQL] Make
broadcast-exchange thread pool configurable
URL: https://github.com/apache/spark/pull/23670#discussion_r257507225
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
##########
@@ -132,4 +132,16 @@ object StaticSQLConf {
.intConf
.createWithDefault(1000)
+ val BROADCAST_EXCHANGE_MAX_THREAD_THREASHOLD =
+ buildStaticConf("spark.sql.broadcastExchange.maxThreadThreshold")
+ .doc("The maximum degree of parallelism to fetch and broadcast the
table." +
+ "If we encounter memory issue like frequently full GC or OOM when
broadcast table " +
+ "we can decrease this number in order to reduce memory usage." +
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
