rezasafi commented on issue #23767: [SPARK-26329][CORE] Faster polling of executor memory metrics. URL: https://github.com/apache/spark/pull/23767#issuecomment-478039936 > It is not sending a new heartbeat. It is changing the information sent in the existing heartbeat. In Execuor.scala I can see this: `heartbeater.start() metricsPoller.start()` So I think it is introducing a new thread similar to heartbeater. It is just on the driver that the change is using the original heatbeater. > That said, you could still ask what's the effect on the performance of the executors, and what's a reasonable polling frequency. Its impossible to answer that for all cases, as it'll depend on the workload, but still it is a good idea for us to have some measurement at least. yeah, It is obvious that we can't test for every usecase/workload, but some analysis at least is good. I don't know you guys may have already done that. Unfortunately I wasn't involved in the whole process. I think based on the discussions that we had with @edwinalu, she did something like that for their pr at the time. For procfsbased metrics also some analysis was done and it was noticed that the overhead isn't significant. Saying that, in general it seems that community doesn't believe to do overhead/performance analysis for features and it is common practice to make the feature configurable and leave the analysis to the user. So with that in mind this change is ok as it is.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
