I think the main reason why that was not merged is that Spark itself doesn't have such requirement, and the metrics system is mainly used for spark itself. Most of the needs are from the custom sources/sinks, but Spark's MetricsSystem is not designed as a public API.
I think we could revisit or improve that PR if there's a solid reason about it. Thanks Saisai Sergey Zhemzhitsky <szh.s...@gmail.com> 于2019年5月7日周二 上午5:49写道： > Hi Saisai, > > Thanks a lot for the link! This is exactly what I need. > Just curious, why this PR has not been merged, as it seems to implement > rather natural requirement. > > There are a number or use cases which can benefit from this feature, e.g. > - collecting business metrics based on the data's attributes and reporting > them into the monitoring system as a side effect of the data processing > - visualizing technical metrics by means of alternative software (e.g. > grafana) - currently it's hardly possible to know the actual number of > jobs, stages, tasks and their names and IDs in advance to register all the > corresponding metrics statically. > > > Kind Regards, > Sergey > > > On Mon, May 6, 2019, 16:07 Saisai Shao <sai.sai.s...@gmail.com> wrote: > >> I remembered there was a PR about doing similar thing ( >> https://github.com/apache/spark/pull/18406). From my understanding, this >> seems like a quite specific requirement, it may requires code change to >> support your needs. >> >> Thanks >> Saisai >> >> Sergey Zhemzhitsky <szh.s...@gmail.com> 于2019年5月4日周六 下午4:44写道： >> >>> Hello Spark Users! >>> >>> Just wondering whether it is possible to register a metric source >>> without metrics known in advance and add the metrics themselves to this >>> source later on? >>> >>> It seems that currently MetricSystem puts all the metrics from the >>> source's MetricRegistry into a shared MetricRegistry of a MetricSystem >>> during metric source registration [1]. >>> >>> So in case there is a new metric with a new name added to the source's >>> registry after this source registration, then this new metric will not be >>> reported to the sinks. >>> >>> What I'd like to achieve is to be able to register new metrics with new >>> names dynamically using a single metric source. >>> Is it somehow possible? >>> >>> >>> [1] >>> https://github.com/apache/spark/blob/51de86baed0776304c6184f2c04b6303ef48df90/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala#L162 >>> >>
