GitHub user markgrover opened a pull request:
https://github.com/apache/spark/pull/14270
[SPARK-5847][CORE] Allow for configuring MetricsSystem's use of app ID to
namespace all metrics
## What changes were proposed in this pull request?
Adding a new property to SparkConf called spark.metrics.namespace that
allows users to
set a custom namespace for executor and driver metrics in the metrics
systems.
By default, the root namespace used for driver or executor metrics is
the value of `spark.app.id`. However, often times, users want to be able to
track the metrics
across apps for driver and executor metrics, which is hard to do with
application ID
(i.e. `spark.app.id`) since it changes with every invocation of the app.
For such use cases,
users can set the `spark.metrics.namespace` property to another spark
configuration key like
`spark.app.name` which is then used to populate the root namespace of the
metrics system
(with the app name in our example). `spark.metrics.namespace` property can
be set to any
arbitrary spark property key, whose value would be used to set the root
namespace of the
metrics system. Non driver and executor metrics are never prefixed with
`spark.app.id`, nor
does the `spark.metrics.namespace` property have any such affect on such
metrics.
## How was this patch tested?
Added new unit tests, modified existing unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/markgrover/spark spark-5847
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14270.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14270
----
commit 7d023c820cae3b970e11b2278f6c2b3f3751cc3b
Author: Mark Grover <[email protected]>
Date: 2016-07-19T05:38:31Z
[SPARK-5847][CORE] Allow for configuring MetricsSystem's use of app ID to
namespace all metrics
Adding a new property to SparkConf called spark.metrics.namespace that
allows users to
set a custom namespace for executor and driver metrics in the metrics
systems.
By default, the root namespace used for driver or executor metrics is
the value of `spark.app.id`. However, often times, users want to be able to
track the metrics
across apps for driver and executor metrics, which is hard to do with
application ID
(i.e. `spark.app.id`) since it changes with every invocation of the app.
For such use cases,
users can set the `spark.metrics.namespace` property to another spark
configuration key like
`spark.app.name` which is then used to populate the root namespace of the
metrics system
(with the app name in our example). `spark.metrics.namespace` property can
be set to any
arbitrary spark property key, whose value would be used to set the root
namespace of the
metrics system. Non driver and executor metrics are never prefixed with
`spark.app.id`, nor
does the `spark.metrics.namespace` property have any such affect on such
metrics.
commit 605e690eefe905fa52979dde16f04bd208c8b219
Author: Mark Grover <[email protected]>
Date: 2016-07-19T19:44:55Z
Minor fixes based on self-reviewing
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
