wypoon commented on a change in pull request #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics and rename a configuration. URL: https://github.com/apache/spark/pull/27329#discussion_r373204613
########## File path: docs/configuration.md ########## @@ -1624,6 +1624,42 @@ Apart from these, the following properties are also available, and may be useful </tr> </table> +### Executor Metrics + +<table class="table"> +<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr> +<tr> + <td><code>spark.eventLog.logStageExecutorMetrics.enabled</code></td> + <td>false</td> + <td> + Whether to write per-stage peaks of executor metrics (for each executor) to the event log. + <br /> + <em>Note:</em> The metrics are polled (collected) and sent in the executor heartbeat, + and this is always done; this configuration is only to determine if aggregated metric peaks + are written to the event log. + </td> +</tr> + <td><code>spark.executor.processTreeMetrics.enabled</code></td> + <td>false</td> + <td> + Whether to collect process tree metrics (from the /proc filesystem) when collecting + executor metrics. + <br /> + <em>Note:</em> The process tree metrics are collected only if the /proc filesystem + exists and <code>spark.eventLog.logStageExecutorMetrics.enabled</code> is true as well. + </td> +<tr> + <td><code>spark.executor.metrics.pollingInterval</code></td> + <td>0</td> + <td> + How often to collect executor metrics. + <br /> + If 0, the polling is done on executor heartbeats (thus at the heartbeat interval). + If positive, the polling is done at this interval. Review comment: Added unit. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
