wypoon edited a comment on issue #23767: [SPARK-26329][CORE] Faster polling of executor memory metrics. URL: https://github.com/apache/spark/pull/23767#issuecomment-477877850 > Thanks for the work here. This enables user to pull metric faster at the expense of basically having a new heartbeat. I'm not sure why that is better. Why not just rely on the heartbeat and then add extra care for the corner cases of loosing an executor etc. There is no new heartbeat. There is still the same heartbeat as before, doing the same thing as before (including sending the executor metrics). Do you mean why not perform the faster polling in the heartbeater thread? @edwinalu suggested this at first but concluded that there are no advantages from doing this (you can find the discussion above). Also, remember that faster polling is optional. It is very useful for understanding memory usage in the executors, which can spike in a very short interval. But you turn it on if you need it. If you don't set spark.executor.metrics.pollingInterval, then we default to the existing behavior of polling on executor heartbeats. You don't pay for the overhead if you don't want it.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
