[
https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715491#comment-16715491
]
Edwina Lu commented on SPARK-26329:
-----------------------------------
More frequent polling (and more accurate metrics) sound good. Right now with
SPARK-23206, it will keep track of the peak values per stage. Would the stage
level information (peak values) still be kept with more frequent polling?
For the following example, with <time>: event/polled metric(s)
10: start stage 1; start task 1 stage 1
11: val = 25
12:start stage 2; start task 1 stage 2
13: val = 20
14: end stage 1; end task 1 stage 1
15: val = 15
16: heartbeat 1
17: val = 17
18: end stage 2; end task 1 stage 2
Stage 1 should have peak metric 25 (from time 11), and stage 2 should have peak
metric 20 ( from time 13, when both stage 1 and stage 2 are active). To
maintain stage level information, heartbeat 1 would need to send the peak
values for both stage 1 and stage 2.
> ExecutorMetrics should poll faster than heartbeats
> --------------------------------------------------
>
> Key: SPARK-26329
> URL: https://issues.apache.org/jira/browse/SPARK-26329
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, Web UI
> Affects Versions: 3.0.0
> Reporter: Imran Rashid
> Priority: Major
>
> We should allow faster polling of the executor memory metrics (SPARK-23429 /
> SPARK-23206) without requiring a faster heartbeat rate. We've seen the
> memory usage of executors pike over 1 GB in less than a second, but
> heartbeats are only every 10 seconds (by default). Spark needs to enable
> fast polling to capture these peaks, without causing too much strain on the
> system.
> In the current implementation, the metrics are polled along with the
> heartbeat, but this leads to a slow rate of polling metrics by default. If
> users were to increase the rate of the heartbeat, they risk overloading the
> driver on a large cluster, with too many messages and too much work to
> aggregate the metrics. But, the executor could poll the metrics more
> frequently, and still only send the *max* since the last heartbeat for each
> metric. This keeps the load on the driver the same, and only introduces a
> small overhead on the executor to grab the metrics and keep the max.
> The downside of this approach is that we still need to wait for the next
> heartbeat for the driver to be aware of the new peak. If the executor dies
> or is killed before then, then we won't find out. A potential future
> enhancement would be to send an update *anytime* there is an increase by some
> percentage, but we'll leave that out for now.
> Another possibility would be to change the metrics themselves to track peaks
> for us, so we don't have to fine-tune the polling rate. For example, some
> jvm metrics provide a usage threshold, and notification:
> https://docs.oracle.com/javase/7/docs/api/java/lang/management/MemoryPoolMXBean.html#UsageThreshold
> But, that is not available on all metrics. This proposal gives us a generic
> way to get a more accurate peak memory usage for *all* metrics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
