Github user markgrover commented on the issue:
https://github.com/apache/spark/pull/14270
A few design choices I made along the way are:
1. Use a `SparkConf` property to control the namespace instead of a
`MetricsConfig` property. The reason is that metrics config properties mean
something particular and follow a particular pattern (see
[here](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala#L54),
for example). And, regardless of what we name this property, it doesn't follow
that paradigm - the first part of the property name before the first dot
doesn't represent an instance (like master, worker, executor, etc.) like it
does for legit metrics properties. Also, the properties defined in
`MetricsConfig` are obtained using the `getInstance()` call, where the instance
is passed and again, such a retrieval doesn't really apply to this
configuration property since it doesn't belong to an instance. I understand
it's one more property in `SparkConf` but I really feel its belongs there,
right by `spark.metrics.conf`.
2. Based on the previous point, this property shouldn't fall under the
`spark.metrics.conf.` prefix. This is because if it does, it's mistakenly
understood as a legit metrics property (see code
[here](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala#L54),
for example).
3. The proposed `spark.metrics.namespace` property allows one to specify an
arbitrary property to be used as namespace. I did consider whitespacing it so
users can use `spark.app.name` or `spark.app.id` only but then we'd have to
deal with magic strings(app.name|app.id), and I didn't really feel inclined to
do that. And, @ryan-williams who took a stab at the same issue #4632 made the
same call and I agree with him.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
