Repository: spark Updated Branches: refs/heads/master 5114db578 -> 71876633f
[SPARK-25583][DOC] Add history-server related configuration in the documentation. ## What changes were proposed in this pull request? Add history-server related configuration in the documentation. Some of the history server related configurations were missing in the documentation.Like, 'spark.history.store.maxDiskUsage', 'spark.ui.liveUpdate.period' etc. ## How was this patch tested?    Closes #22601 from shahidki31/historyConf. Authored-by: Shahid <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/71876633 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/71876633 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/71876633 Branch: refs/heads/master Commit: 71876633f3af706408355b5fb561b58dbc593360 Parents: 5114db5 Author: Shahid <[email protected]> Authored: Tue Oct 2 08:05:09 2018 -0700 Committer: Dongjoon Hyun <[email protected]> Committed: Tue Oct 2 08:05:09 2018 -0700 ---------------------------------------------------------------------- docs/configuration.md | 16 ++++++++++++++++ docs/monitoring.md | 25 +++++++++++++++++++++++++ 2 files changed, 41 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/71876633/docs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/configuration.md b/docs/configuration.md index 782ccff..5577393 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -794,6 +794,13 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.ui.dagGraph.retainedRootRDDs</code></td> + <td>Int.MaxValue</td> + <td> + How many DAG graph nodes the Spark UI and status APIs remember before garbage collecting. + </td> +</tr> +<tr> <td><code>spark.ui.enabled</code></td> <td>true</td> <td> @@ -808,6 +815,15 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.ui.liveUpdate.period</code></td> + <td>100ms</td> + <td> + How often to update live entities. -1 means "never update" when replaying applications, + meaning only the last write will happen. For live applications, this avoids a few + operations that we can live without when rapidly processing incoming task events. + </td> +</tr> +<tr> <td><code>spark.ui.port</code></td> <td>4040</td> <td> http://git-wip-us.apache.org/repos/asf/spark/blob/71876633/docs/monitoring.md ---------------------------------------------------------------------- diff --git a/docs/monitoring.md b/docs/monitoring.md index f6d52ef..69bf308 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -186,6 +186,23 @@ Security options for the Spark History Server are covered more detail in the </td> </tr> <tr> + <td>spark.history.fs.endEventReparseChunkSize</td> + <td>1m</td> + <td> + How many bytes to parse at the end of log files looking for the end event. + This is used to speed up generation of application listings by skipping unnecessary + parts of event log files. It can be disabled by setting this config to 0. + </td> + </tr> + <tr> + <td>spark.history.fs.inProgressOptimization.enabled</td> + <td>true</td> + <td> + Enable optimized handling of in-progress logs. This option may leave finished + applications that fail to rename their event logs listed as in-progress. + </td> + </tr> + <tr> <td>spark.history.fs.numReplayThreads</td> <td>25% of available cores</td> <td> @@ -193,6 +210,14 @@ Security options for the Spark History Server are covered more detail in the </td> </tr> <tr> + <td>spark.history.store.maxDiskUsage</td> + <td>10g</td> + <td> + Maximum disk usage for the local directory where the cache application history information + are stored. + </td> + </tr> + <tr> <td>spark.history.store.path</td> <td>(none)</td> <td> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
