rezasafi edited a comment on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics and modify a configuration. URL: https://github.com/apache/spark/pull/27329#issuecomment-580473118 The goal of SPARK-24958, which introduces procfs metrics, was to compute peak procefs metrics along with other memory metrics and log them in event logs. So because of that there was a need that spark.eventLog.logStageExecutorMetrics.enabled being true when you want procfs metrics. I wasn't aware of SPARK-27189 when I objected the rename. After SPARK-27189 this has been changed. The metric system doesn't deal with peak values, but it exposed the processtree metrics. So the rename make sense now to me. Still I think we need to document somewhere that if you need peak metrics for processtree metrics, you need to set both spark.eventLog.logStageExecutorMetrics.enabled and whatever name you decided for enabling processtree metrics.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
