GitHub user jzhuge opened a pull request:
https://github.com/apache/spark/pull/21709
[SPARK-5152][CORE] Read metrics config file from Hadoop file system
## What changes were proposed in this pull request?
Enable metrics config file to be read directly from Hadoop file system.
```
spark-submit --conf
spark.metrics.conf=hdfs://<nn>:8020/config/spark-metrics.properties
spark-submit --conf
spark.metrics.conf=s3a://<bucket>/config/spark-metrics.properties
```
There is no need to distribute the config file if either HDFS or S3A is
used.
If no URI scheme is specified, assume file://. This ensures backwards
compatibility.
## How was this patch tested?
Unit tests:
- MetricsSystemSuite
- MetricsConfigSuite
- TaskContextSuite
Manual tests:
- Local file without URI scheme
- Local file with file://
- HDFS file path
- S3 file path
Author: John Zhuge <[email protected]>
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jzhuge/spark SPARK-5152
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21709.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21709
----
commit 836ffdec985eac62396005726d5fb49a9e3455a5
Author: John Zhuge <jzhuge@...>
Date: 2018-06-12T05:27:56Z
[SPARK-5152][CORE] Read metrics config file from Hadoop file system
Enable metrics config file to be read directly from Hadoop file system.
```
spark-submit --conf
spark.metrics.conf=hdfs://<nn>:8020/config/spark-metrics.properties
spark-submit --conf
spark.metrics.conf=s3a://<bucket>/config/spark-metrics.properties
```
There is no need to distribute the config file if either HDFS or S3A is
used.
If no URI scheme is specified, assume file://. This ensures backwards
compatibility.
Unit tests:
- MetricsSystemSuite
- MetricsConfigSuite
- TaskContextSuite
Manual tests:
- Local file without URI scheme
- Local file with file://
- HDFS file path
- S3 file path
Author: John Zhuge <[email protected]>
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
