Github user LucaCanali commented on the issue:
https://github.com/apache/spark/pull/22279
Hi @jerryshao you can find here below an example of metrics currently
reported by applicationMaster, illustrating the issue reported here. You can
find there the list of AM metrics reported (with the application ID as a
prefix by default). In addition metrics for CodeGeneration and
HiveExternalCatalog are also reported, these metrics do not make sense in this
context, in addition they have no prefix. Metrics for JVM are reported too
(without application_id prefix), which I am not sure it is wanted either.
```
bin/spark-shell --master yarn \
--conf
"spark.metrics.conf.applicationMaster.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink"
\
--conf "spark.metrics.conf.*.sink.graphite.host"=lc-mytest5 \
--conf "spark.metrics.conf.*.sink.graphite.port"=2003 \
--conf "spark.metrics.conf.*.sink.graphite.period"=10 \
--conf "spark.metrics.conf.*.sink.graphite.unit"=seconds \
--conf "spark.metrics.conf.*.sink.graphite.prefix"="luca" \
--conf
"spark.metrics.conf.*.source.jvm.class"="org.apache.spark.metrics.source.JvmSource"
```
I have used InfluxDB to collect the metrics. This is the output of "show
measurements" in InfluxDB:
```
name: measurements
name
----
CodeGenerator.compilationTime.count
CodeGenerator.compilationTime.max
CodeGenerator.compilationTime.mean
CodeGenerator.compilationTime.min
CodeGenerator.compilationTime.p50
CodeGenerator.compilationTime.p75
CodeGenerator.compilationTime.p95
CodeGenerator.compilationTime.p98
CodeGenerator.compilationTime.p99
CodeGenerator.compilationTime.p999
CodeGenerator.compilationTime.stddev
CodeGenerator.generatedClassSize.count
CodeGenerator.generatedClassSize.max
CodeGenerator.generatedClassSize.mean
CodeGenerator.generatedClassSize.min
CodeGenerator.generatedClassSize.p50
CodeGenerator.generatedClassSize.p75
CodeGenerator.generatedClassSize.p95
CodeGenerator.generatedClassSize.p98
CodeGenerator.generatedClassSize.p99
CodeGenerator.generatedClassSize.p999
CodeGenerator.generatedClassSize.stddev
CodeGenerator.generatedMethodSize.count
CodeGenerator.generatedMethodSize.max
CodeGenerator.generatedMethodSize.mean
CodeGenerator.generatedMethodSize.min
CodeGenerator.generatedMethodSize.p50
CodeGenerator.generatedMethodSize.p75
CodeGenerator.generatedMethodSize.p95
CodeGenerator.generatedMethodSize.p98
CodeGenerator.generatedMethodSize.p99
CodeGenerator.generatedMethodSize.p999
CodeGenerator.generatedMethodSize.stddev
CodeGenerator.sourceCodeSize.count
CodeGenerator.sourceCodeSize.max
CodeGenerator.sourceCodeSize.mean
CodeGenerator.sourceCodeSize.min
CodeGenerator.sourceCodeSize.p50
CodeGenerator.sourceCodeSize.p75
CodeGenerator.sourceCodeSize.p95
CodeGenerator.sourceCodeSize.p98
CodeGenerator.sourceCodeSize.p99
CodeGenerator.sourceCodeSize.p999
CodeGenerator.sourceCodeSize.stddev
HiveExternalCatalog.fileCacheHits.count
HiveExternalCatalog.filesDiscovered.count
HiveExternalCatalog.hiveClientCalls.count
HiveExternalCatalog.parallelListingJobCount.count
HiveExternalCatalog.partitionsFetched.count
application_1516620698330_110908.applicationMaster.numContainersPendingAllocate
application_1516620698330_110908.applicationMaster.numExecutorsFailed
application_1516620698330_110908.applicationMaster.numExecutorsRunning
application_1516620698330_110908.applicationMaster.numLocalityAwareTasks
application_1516620698330_110908.applicationMaster.numReleasedContainers
jvm.PS-MarkSweep.count
jvm.PS-MarkSweep.time
jvm.PS-Scavenge.count
jvm.PS-Scavenge.time
jvm.direct.capacity
jvm.direct.count
jvm.direct.used
jvm.heap.committed
jvm.heap.init
jvm.heap.max
jvm.heap.usage
jvm.heap.used
jvm.mapped.capacity
jvm.mapped.count
jvm.mapped.used
jvm.non-heap.committed
jvm.non-heap.init
jvm.non-heap.max
jvm.non-heap.usage
jvm.non-heap.used
jvm.pools.Code-Cache.committed
jvm.pools.Code-Cache.init
jvm.pools.Code-Cache.max
jvm.pools.Code-Cache.usage
jvm.pools.Code-Cache.used
jvm.pools.Compressed-Class-Space.committed
jvm.pools.Compressed-Class-Space.init
jvm.pools.Compressed-Class-Space.max
jvm.pools.Compressed-Class-Space.usage
jvm.pools.Compressed-Class-Space.used
jvm.pools.Metaspace.committed
jvm.pools.Metaspace.init
jvm.pools.Metaspace.max
jvm.pools.Metaspace.usage
jvm.pools.Metaspace.used
jvm.pools.PS-Eden-Space.committed
jvm.pools.PS-Eden-Space.init
jvm.pools.PS-Eden-Space.max
jvm.pools.PS-Eden-Space.usage
jvm.pools.PS-Eden-Space.used
jvm.pools.PS-Old-Gen.committed
jvm.pools.PS-Old-Gen.init
jvm.pools.PS-Old-Gen.max
jvm.pools.PS-Old-Gen.usage
jvm.pools.PS-Old-Gen.used
jvm.pools.PS-Survivor-Space.committed
jvm.pools.PS-Survivor-Space.init
jvm.pools.PS-Survivor-Space.max
jvm.pools.PS-Survivor-Space.usage
jvm.pools.PS-Survivor-Space.used
jvm.total.committed
jvm.total.init
jvm.total.max
jvm.total.used
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
