Sergey Zhemzhitsky created SPARK-27641:
------------------------------------------
Summary: Unregistering a single Metrics Source with no metrics
leads to removing all the from other sources with the same name
Key: SPARK-27641
URL: https://issues.apache.org/jira/browse/SPARK-27641
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.2, 2.3.3, 2.2.3
Reporter: Sergey Zhemzhitsky
Currently Spark allows registering multiple Metric Sources with the same source
name like the following
{code:scala}
val acc1 = sc.longAccumulator
LongAccumulatorSource.register(sc, {"acc1" -> acc1})
val acc2 = sc.longAccumulator
LongAccumulatorSource.register(sc, {"acc2" -> acc2})
{code}
In that case there are two metric sources registered and both of these sources
have the same name -
[AccumulatorSource|https://github.com/apache/spark/blob/6ef45301a46c47c12fbc74bb9ceaffea685ed944/core/src/main/scala/org/apache/spark/metrics/source/AccumulatorSource.scala#L47]
If you try to unregister the source with no accumulators and metrics registered
like the following
{code:scala}
SparkEnv.get.metricsSystem.removeSource(new LongAccumulatorSource)
{code}
... then all the metrics for all the sources with the same name will be
unregistered because of the
[following|https://github.com/apache/spark/blob/6ef45301a46c47c12fbc74bb9ceaffea685ed944/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala#L171]
snippet which removes all matching records which start with the corresponding
prefix which includes the source name, but does not include metric name to be
removed.
{code:scala}
def removeSource(source: Source) {
sources -= source
val regName = buildRegistryName(source)
registry.removeMatching((name: String, _: Metric) => name.startsWith(regName))
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
