GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/spark/pull/23260
[SPARK-26311][YARN] New feature: custom log URL for stdout/stderr
## What changes were proposed in this pull request?
This patch proposes adding a new configuration on YARN mode: custom log
URL. This will enable end users to point application logs to external log
service which enables to serve logs when NodeManager becomes unavailable.
Some pre-defined patterns are available for custom log URL to specify them
like path variables.
## How was this patch tested?
Manual test.
Below run changes executor log URLs in UI pages.
```
./bin/spark-submit --conf
spark.yarn.custom.log.url="{{HttpScheme}}{{NodeHttpAddress}}/test/cluster/{{ClusterId}}/container/{{ContainerId}}/user/{{User}}/filename/{{FileName}}"
--class org.apache.spark.examples.SparkPi
examples/jars/spark-examples_2.11-<spark_version>.jar
```
Example of stdout log url is below:
`http://node-address:node-port`/test/cluster/`workload1`/container/`container_e08_1542798098040_0012_01_000002`/user/`spark`/filename/`stdout`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/spark SPARK-26311
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23260.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23260
----
commit 65cc6a32729cccba340f66c766c7255be4d7f356
Author: Jungtaek Lim (HeartSaVioR) <kabhwan@...>
Date: 2018-12-08T06:32:46Z
[SPARK-26311][YARN] New feature: custom log URL for stdout/stderr
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
