wypoon commented on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics and rename a configuration. URL: https://github.com/apache/spark/pull/27329#issuecomment-580397577 @LucaCanali thanks for your comments. At this point, I do not wish to change the behavior. Renaming a configuration property is one thing; it is a simple change. Changing the effect of a property is another thing, and requires testing. You can understand if I don't wish to take that on. On your last point, I don't know if the statement in question is inaccurate (as I haven't examined that type of metrics). What I can clarify is that the executor always polls memory metrics (either in the heartbeat thread or in a separate thread) and sends them in its heartbeat. In the `EventLoggingListener`, there is logic to aggregate these memory metrics and log per-stage per-executor peaks on stage completion, but only if `spark.eventLog.logStageExecutorMetrics.enabled` is true (otherwise executor metrics update events are ignored). Just to confirm, whenever the `ExecutorMetricsPoller` is asked to `poll`, then you update your `ExecutorMetricsSource` with the latest snapshot of the memory metrics, correct? In this case, this happens irrespective of `spark.eventLog.logStageExecutorMetrics.enabled`. If this is correct, that the statement you refer to in `docs/monitoring.md` is inaccurate and should be corrected.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
