[
https://issues.apache.org/jira/browse/SPARK-36360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-36360:
---------------------------------
Component/s: (was: Spark Core)
DStreams
> Delete appName from StreamingSource
> ------------------------------------
>
> Key: SPARK-36360
> URL: https://issues.apache.org/jira/browse/SPARK-36360
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
> Affects Versions: 3.1.2
> Reporter: Marcel Neumann
> Priority: Major
>
> The StreamingSource includes the appName in its sourceName. However, the
> appName should not be handled by the StreamingSource. It is already handled
> by the MetricsSystem. See all other MetricSources, e.g. ExecutorMetricsSource.
> Why is this important? See this part from the
> [documentation|https://spark.apache.org/docs/latest/monitoring.html#metrics]:
> ??Often times, users want to be able to track the metrics across apps for
> driver and executors, which is hard to do with application ID (i.e.
> {{spark.app.id}}) since it changes with every invocation of the app. For such
> use cases, a custom namespace can be specified for metrics reporting using
> {{spark.metrics.namespace}} configuration property. If, say, users wanted to
> set the metrics namespace to the name of the application, they can set the
> {{spark.metrics.namespace}} property to a value like {{$\{spark.app.name}}}.
> This value is then expanded appropriately by Spark and is used as the root
> namespace of the metrics system.??
> This is only possible if the MetricsSystem handles the namespace which it
> does. But the StreamingSource additionally adds the appName in its
> sourceName, thus there is no way to configure a namespace that does not
> include the appName.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
