[GitHub] [spark] gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues
gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues URL: https://github.com/apache/spark/pull/25307#discussion_r350573157 ## File path: core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala ## @@ -46,8 +46,18 @@ private class AsyncEventQueue( // Cap the capacity of the queue so we get an explicit error (rather than an OOM exception) if // it's perpetually being added to more quickly than it's being drained. - private val eventQueue = new LinkedBlockingQueue[SparkListenerEvent]( -conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY)) + // The capacity can be configured by spark.scheduler.listenerbus.eventqueue.${name}.capacity, + // if no such conf is specified, use the value specified in + // LISTENER_BUS_EVENT_QUEUE_CAPACITY Review comment: We need to update the conf description of LISTENER_BUS_EVENT_QUEUE_CAPACITY. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues
gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues URL: https://github.com/apache/spark/pull/25307#discussion_r350573375 ## File path: core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala ## @@ -46,8 +46,18 @@ private class AsyncEventQueue( // Cap the capacity of the queue so we get an explicit error (rather than an OOM exception) if // it's perpetually being added to more quickly than it's being drained. - private val eventQueue = new LinkedBlockingQueue[SparkListenerEvent]( -conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY)) + // The capacity can be configured by spark.scheduler.listenerbus.eventqueue.${name}.capacity, + // if no such conf is specified, use the value specified in + // LISTENER_BUS_EVENT_QUEUE_CAPACITY + private[scheduler] def capacity: Int = { +val queuesize = conf.getInt(s"spark.scheduler.listenerbus.eventqueue.${name}.capacity", Review comment: Instead of hard-coded here, can we define it in core/src/main/scala/org/apache/spark/internal/config/package.scala ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues
gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues URL: https://github.com/apache/spark/pull/25307#discussion_r350572738 ## File path: core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala ## @@ -46,8 +46,18 @@ private class AsyncEventQueue( // Cap the capacity of the queue so we get an explicit error (rather than an OOM exception) if // it's perpetually being added to more quickly than it's being drained. - private val eventQueue = new LinkedBlockingQueue[SparkListenerEvent]( -conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY)) + // The capacity can be configured by spark.scheduler.listenerbus.eventqueue.${name}.capacity, + // if no such conf is specified, use the value specified in + // LISTENER_BUS_EVENT_QUEUE_CAPACITY + private[scheduler] def capacity: Int = { +val queuesize = conf.getInt(s"spark.scheduler.listenerbus.eventqueue.${name}.capacity", +conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY)) Review comment: Nit: indent. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org