gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE]
Allow to config different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#discussion_r350573375
##########
File path: core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala
##########
@@ -46,8 +46,18 @@ private class AsyncEventQueue(
// Cap the capacity of the queue so we get an explicit error (rather than an
OOM exception) if
// it's perpetually being added to more quickly than it's being drained.
- private val eventQueue = new LinkedBlockingQueue[SparkListenerEvent](
- conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY))
+ // The capacity can be configured by
spark.scheduler.listenerbus.eventqueue.${name}.capacity,
+ // if no such conf is specified, use the value specified in
+ // LISTENER_BUS_EVENT_QUEUE_CAPACITY
+ private[scheduler] def capacity: Int = {
+ val queuesize =
conf.getInt(s"spark.scheduler.listenerbus.eventqueue.${name}.capacity",
Review comment:
Instead of hard-coded here, can we define it in
core/src/main/scala/org/apache/spark/internal/config/package.scala ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
