HeartSaVioR commented on a change in pull request #25702: [SPARK-29001][CORE]
Print events that take too long time to process
URL: https://github.com/apache/spark/pull/25702#discussion_r344407329
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -585,6 +585,21 @@ package object config {
.intConf
.createWithDefault(128)
+ private[spark] val LISTENER_BUS_LOG_SLOW_EVENT_ENABLED =
+ ConfigBuilder("spark.scheduler.listenerbus.logSlowEvent.enabled")
+ .doc("When enabled, log the event that takes too much time to process.
This helps us " +
+ "discover the event types that cause performance bottlenecks. The time
threshold is " +
+ "controlled by spark.scheduler.listenerbus.logSlowEvent.threshold.")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
+1 on true by default, since the issue might not be reproducible easily and
it is only adding the overhead of number comparison in happy case (overhead of
bad case shouldn't matter since if it occurs frequently it's likely having real
issue soon). Moreover, if I understand correctly, we can't change it from
outside of Spark while application is running - that means application should
be stopped and restarted due to turn on this config.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
