rezasafi commented on a change in pull request #23306: [SPARK-26357][Core]
Expose executors' procfs metrics to Metrics system
URL: https://github.com/apache/spark/pull/23306#discussion_r243055486
##########
File path:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala
##########
@@ -205,21 +210,44 @@ private[spark] class ProcfsMetricsGetter(procfsDir:
String = "/proc/") extends L
}
}
+ private[spark] def isCacheValid(): Boolean = {
+ val lastMetricComputation = System.currentTimeMillis() -
lastimeMetricsComputed
+ // ToDo: Should we make this configurable?
+ return Math.min(1000, HEARTBEAT_INTERVAL_MS) > lastMetricComputation
+ }
+
private[spark] def computeAllMetrics(): ProcfsMetrics = {
if (!isAvailable) {
+ lastimeMetricsComputed = System.currentTimeMillis
+ cachedAllMetric = ProcfsMetrics(0, 0, 0, 0, 0, 0)
return ProcfsMetrics(0, 0, 0, 0, 0, 0)
}
- val pids = computeProcessTree
- var allMetrics = ProcfsMetrics(0, 0, 0, 0, 0, 0)
- for (p <- pids) {
- allMetrics = addProcfsMetricsFromOneProcess(allMetrics, p)
- // if we had an error getting any of the metrics, we don't want to
report partial metrics, as
- // that would be misleading.
- if (!isAvailable) {
- return ProcfsMetrics(0, 0, 0, 0, 0, 0)
+
+ if (!isCacheValid) {
+ this.synchronized {
+ if (isCacheValid) {
Review comment:
This is a good point. I am worried that this whole caching thing with
synchronization and volatility can be an unnecessary overhead for the whole
system. Should we really care about having both metrics systems and UI metric
updater work together? Isn't it better to just enable one of them at the time
to avoid overhead. @squito I especially want to hear from you since you first
mentioned the idea of caching here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
