This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 7bfb4f0a82a1 [SPARK-49176][K8S][DOCS] Fix
`spark.ui.custom.executor.log.url` docs by adding K8s
7bfb4f0a82a1 is described below
commit 7bfb4f0a82a1fa37f42000d3fd30cdf54f214d38
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Fri Aug 9 16:33:41 2024 +0900
[SPARK-49176][K8S][DOCS] Fix `spark.ui.custom.executor.log.url` docs by
adding K8s
### What changes were proposed in this pull request?
This PR aims to fix `spark.ui.custom.executor.log.url` configuration's
documentation by adding K8s on top of YARN.
```
- For now, only YARN mode supports this configuration
+ For now, only YARN and K8s cluster manager supports this configuration
```
In addition, K8s example is added to `running-on-kubernetes.md` docs to
help users refer the existing environment variables, `SPARK_APPLICATION_ID` and
`SPARK_EXECUTOR_ID`, more easily to enable this feature.
-
https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#using-environment-variables-inside-of-your-config
### Why are the changes needed?
To fix a wrong description.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual review. This is a doc-only PR.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47681 from dongjoon-hyun/SPARK-49176.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
docs/configuration.md | 2 +-
docs/running-on-kubernetes.md | 8 ++++++++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/docs/configuration.md b/docs/configuration.md
index 5457a77e4fc7..abd934572bd0 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1572,7 +1572,7 @@ Apart from these, the following properties are also
available, and may be useful
which will be also effective when accessing the application on history
server. The new log urls must be
permanent, otherwise you might have dead link for executor log urls.
<p/>
- For now, only YARN mode supports this configuration
+ For now, only YARN and K8s cluster manager supports this configuration
</td>
<td>3.0.0</td>
</tr>
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index a684e7caa1a0..cbddaaf90fca 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -430,6 +430,14 @@ The same logs can also be accessed through the
[Kubernetes
dashboard](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/)
if installed on
the cluster.
+When there exists a log collection system, you can expose it at Spark Driver
`Executors` tab UI. For example,
+
+```
+spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_APP_ID='$(SPARK_APPLICATION_ID)'
+spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_EXECUTOR_ID='$(SPARK_EXECUTOR_ID)'
+spark.ui.custom.executor.log.url='https://log-server/log?appId={{APP_ID}}&execId={{EXECUTOR_ID}}'
+```
+
### Accessing Driver UI
The UI associated with any application can be accessed locally using
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
