[
https://issues.apache.org/jira/browse/SPARK-36360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Neumann updated SPARK-36360:
-----------------------------------
Description:
The StreamingSource includes the appName in its sourceName. The appName should
not be handled by the StreamingSource. It is already handled by the
MetricsSystem. See all other MetricSources, e.g. ExecutorMetricsSource.
Why is this important? See this part from the
[documentation|https://spark.apache.org/docs/latest/monitoring.html#metrics]:
??Often times, users want to be able to track the metrics across apps for
driver and executors, which is hard to do with application ID (i.e.
{{spark.app.id}}) since it changes with every invocation of the app. For such
use cases, a custom namespace can be specified for metrics reporting using
{{spark.metrics.namespace}} configuration property. If, say, users wanted to
set the metrics namespace to the name of the application, they can set the
{{spark.metrics.namespace}} property to a value like {{${spark.app.name}}}.
This value is then expanded appropriately by Spark and is used as the root
namespace of the metrics system.??
This is only possible if the MetricsSystem handles the namespace which it does.
But the StreamingSource additionally adds the appName in its sourceName, thus
there is no way to configure a namespace that does not include the appName.
was:
The StreamingSource includes the appName in its sourceName. This is not desired
when using a custom namespace for metrics reporting using
{{spark.metrics.namespace}} configuration property as the {{spark.app.name}}
cannot be excluded in the name of the metric. Using a metrics namespace results
in a duplicated indicator for {{spark.app.name}}.
All other metrics do not include the appName in their sourceName, e.g.
ExecutorMetricsSource. If desired you can configure that with a metrics
namespace. The same should be done in StreamingSource.
> Delete appName from StreamingSource
> ------------------------------------
>
> Key: SPARK-36360
> URL: https://issues.apache.org/jira/browse/SPARK-36360
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.1.2
> Reporter: Marcel Neumann
> Priority: Blocker
>
> The StreamingSource includes the appName in its sourceName. The appName
> should not be handled by the StreamingSource. It is already handled by the
> MetricsSystem. See all other MetricSources, e.g. ExecutorMetricsSource.
> Why is this important? See this part from the
> [documentation|https://spark.apache.org/docs/latest/monitoring.html#metrics]:
> ??Often times, users want to be able to track the metrics across apps for
> driver and executors, which is hard to do with application ID (i.e.
> {{spark.app.id}}) since it changes with every invocation of the app. For such
> use cases, a custom namespace can be specified for metrics reporting using
> {{spark.metrics.namespace}} configuration property. If, say, users wanted to
> set the metrics namespace to the name of the application, they can set the
> {{spark.metrics.namespace}} property to a value like {{${spark.app.name}}}.
> This value is then expanded appropriately by Spark and is used as the root
> namespace of the metrics system.??
> This is only possible if the MetricsSystem handles the namespace which it
> does. But the StreamingSource additionally adds the appName in its
> sourceName, thus there is no way to configure a namespace that does not
> include the appName.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
