This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 614bfe646b6c [SPARK-49176][K8S][DOCS] Fix
`spark.ui.custom.executor.log.url` docs by adding K8s
614bfe646b6c is described below
commit 614bfe646b6ce1141c0585108132d05efdef8a42
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Fri Aug 9 16:33:41 2024 +0900
[SPARK-49176][K8S][DOCS] Fix `spark.ui.custom.executor.log.url` docs by
adding K8s
### What changes were proposed in this pull request?
This PR aims to fix `spark.ui.custom.executor.log.url` configuration's
documentation by adding K8s on top of YARN.
```
- For now, only YARN mode supports this configuration
+ For now, only YARN and K8s cluster manager supports this configuration
```
In addition, K8s example is added to `running-on-kubernetes.md` docs to
help users refer the existing environment variables, `SPARK_APPLICATION_ID` and
`SPARK_EXECUTOR_ID`, more easily to enable this feature.
-
https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#using-environment-variables-inside-of-your-config
### Why are the changes needed?
To fix a wrong description.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual review. This is a doc-only PR.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47681 from dongjoon-hyun/SPARK-49176.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
docs/configuration.md | 2 +-
docs/running-on-kubernetes.md | 8 ++++++++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/docs/configuration.md b/docs/configuration.md
index 9e2be30a8326..3ca78c1204a1 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1526,7 +1526,7 @@ Apart from these, the following properties are also
available, and may be useful
which will be also effective when accessing the application on history
server. The new log urls must be
permanent, otherwise you might have dead link for executor log urls.
<p/>
- For now, only YARN mode supports this configuration
+ For now, only YARN and K8s cluster manager supports this configuration
</td>
<td>3.0.0</td>
</tr>
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 7a35236afa0a..dd8e8239d08c 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -430,6 +430,14 @@ The same logs can also be accessed through the
[Kubernetes
dashboard](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/)
if installed on
the cluster.
+When there exists a log collection system, you can expose it at Spark Driver
`Executors` tab UI. For example,
+
+```
+spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_APP_ID='$(SPARK_APPLICATION_ID)'
+spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_EXECUTOR_ID='$(SPARK_EXECUTOR_ID)'
+spark.ui.custom.executor.log.url='https://log-server/log?appId={{APP_ID}}&execId={{EXECUTOR_ID}}'
+```
+
### Accessing Driver UI
The UI associated with any application can be accessed locally using
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
