Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22504#discussion_r226466609
--- Diff: docs/configuration.md ---
@@ -266,6 +266,37 @@ of the most common options to set are:
Only has effect in Spark standalone mode or Mesos cluster deploy mode.
</td>
</tr>
+<tr>
+ <td><code>spark.driver.log.dfsDir</code></td>
+ <td>(none)</td>
+ <td>
+ Base directory in which Spark driver logs are synced, if
spark.driver.log.syncToDfs.enabled is true.
+ Within this base directory, Spark creates a sub-directory for each
application, and logs the driver logs
+ specific to the application in this directory. Users may want to set
this to a unified location like an
+ HDFS directory so driver log files can be persisted for later usage.
This directory should allow any spark
+ user to read/write files and the spark history server user to delete
files. Additionally, older logs from
+ this directory are cleaned by Spark History Server if
spark.history.fs.driverlog.cleaner.enabled is true.
--- End diff --
Put config names between `<code></code>`.
This is also wrong, since you do not have an explanation of this option in
the documentation. It will be true also if you just enable the normal SHS
cleaner. You should explain that option explicitly and say its default value is
the value of the other config.
The SHS options are in monitoring.md.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
