wypoon commented on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics. URL: https://github.com/apache/spark/pull/27329#issuecomment-577975580 > @wypoon processtree metrics will also be written to eventlogs like other executor metrics and they will be gathered only if spark.eventLog.logStageExecutorMetrics.enabled is set to true. I think the current name which is "spark.eventLog.logStageExecutorProcessTreeMetrics.enabled" is self explanatory about this behavior. So If you want to change the name it makes more sense to change "spark.eventLog.logStageExecutorMetrics.enabled" as well. OtherwiseI think the current name is better to avoid user mis understanding. It is not self-evident that spark.eventLog.logStageExecutorProcessTreeMetrics.enabled being true is only effective if spark.eventLog.logStageExecutorMetrics.enabled is true as well. This is one reason why we need documentation and I document this fact in this PR. As I already pointed out, spark.eventLog.logStageExecutorMetrics.enabled is about whether to log metrics - the metrics (in general) are always collected - so that name is appropriate. spark.eventLog.logStageExecutorProcessTreeMetrics.enabled is about whether to collect a particular class of metrics in the first place. If they are not collected, they are logged as 0 (but still logged), if spark.eventLog.logStageExecutorMetrics.enabled is true. This is why I propose the rename.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
