wypoon commented on issue #23767: [SPARK-26329][CORE] Faster polling of executor memory metrics. URL: https://github.com/apache/spark/pull/23767#issuecomment-477877850 > Thanks for the work here. This enables user to pull metric faster at the expense of basically having a new heartbeat. I'm not sure why that is better. Why not just rely on the heartbeat and then add extra care for the corner cases of loosing an executor etc. There is no new heartbeat. There is still the same heartbeat as before, doing the same thing as before (including sending the executor metrics). Do you mean why not perform the faster polling in the heartbeater thread? @edwinalu suggested this at first but concluded that there are no advantages from doing this (you can find the discussion above).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
