jiangxb1987 commented on a change in pull request #25702: [SPARK-29001][CORE]
Print events that take too long time to process
URL: https://github.com/apache/spark/pull/25702#discussion_r344400040
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -585,6 +585,21 @@ package object config {
.intConf
.createWithDefault(128)
+ private[spark] val LISTENER_BUS_LOG_SLOW_EVENT_ENABLED =
+ ConfigBuilder("spark.scheduler.listenerbus.logSlowEvent.enabled")
+ .doc("When enabled, log the event that takes too much time to process.
This helps us " +
+ "discover the event types that cause performance bottlenecks. The time
threshold is " +
+ "controlled by spark.scheduler.listenerbus.logSlowEvent.threshold.")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
Since this is really useful in tracking event lost related issues, and it's
light-weighted, I'd recommend turn it on by default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
