jiangxb1987 commented on a change in pull request #24303:
[SPARK-27394][WebUI]Flush LiveEntity if necessary when receiving
SparkListenerExecutorMetricsUpdate
URL: https://github.com/apache/spark/pull/24303#discussion_r272643471
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -831,6 +835,14 @@ private[spark] class AppStatusListener(
}
}
}
+ // Flush updates if necessary. Executor heartbeat is an event that happens
periodically. Flush
+ // here to ensure the staleness of Spark UI doesn't last more that the
executor heartbeat
+ // interval.
+ if (now - lastFlushTimeNs > liveUpdatePeriodNs) {
Review comment:
I'm also worried about the case when `flush()` takes a few milliseconds to
finish, and you end up with always dealing with updating all live entities for
each ExecutorMetricsUpdate event.
Is it possible to introduce a new config that specifies the live update
period for ExecutorMetricsUpdate only? The default value can be the same as
`liveUpdatePeriodNs`, while user can change it to a bigger value when the
`flush()` function become a issue in processing the event.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
