squito commented on issue #23767: [SPARK-26329][CORE] Faster polling of executor memory metrics. URL: https://github.com/apache/spark/pull/23767#issuecomment-478033373 > This change is using a heartbeat pattern. That's what I meant. It is suggesting to send a new heartbeat to executors for pulling metrics faster when it is needed. It is *not* sending a new heartbeat. It is changing the information sent in the existing heartbeat. There *is* more polling happening in the executor, but that is not the same as the heartbeat. A heartbeat is a message sent from the executor back to the driver. Sure, in general some performance analysis would help, but its well known that on large clusters you can't increase the rate of the heartbeats, that the driver is already struggling to keep up with the message it receives. We really don't need to waste effort analyzing that in detail again. That said, you could still ask what's the effect on the performance of the executors, and what's a reasonable polling frequency. Its impossible to answer that for all cases, as it'll depend on the workload, but still it is a good idea for us to have some measurement at least.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
