[
https://issues.apache.org/jira/browse/BEAM-12032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-12032:
------------------------------------
Description:
Spark runner in some environments (Spark 3 and Databricks) reports a warning on
Unknown metric type. The pipeline works well and the metrics appear correct but
the warning seems to be a problem of the runner.
{quote}
[2021-03-23T15:10:02,222] [INFO ] o.a.s.m.MetricsSystem - Metrics already
registered
java.lang.IllegalArgumentException: Unknown metric type: class
org.apache.beam.runners.spark.metrics.SparkBeamMetric
at
com.codahale.metrics.MetricRegistry.notifyListenerOfAddedMetric(MetricRegistry.java:532)
~[spark-3.0.1-yarn-shuffle.jar:?]
at com.codahale.metrics.MetricRegistry.addListener(MetricRegistry.java:363)
~[spark-3.0.1-yarn-shuffle.jar:?]
at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:93)
~[spark-3.0.1-yarn-shuffle.jar:?]
at
org.apache.spark.metrics.MetricsSystem.registerSource(MetricsSystem.scala:165)
[spark-core_2.12-3.0.1.jar:3.0.1]
at
org.apache.beam.runners.spark.SparkRunner.registerMetricsSource(SparkRunner.java:275)
[beam-runners-spark-2.27.0.jar:?]
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:250)
[beam-runners-spark-2.27.0.jar:?]
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:92)
[beam-runners-spark-2.27.0.jar:?]
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:322)
[beam-sdks-java-harness-2.27.0.jar:?]{quote}
was:
{{Spark runner in some environments (Spark 3 and Databricks) reports a warning
on Unknown metric type. }}{{The pipeline works well and the metrics appear
correct but the warning seems to be a problem of the runner.}}
{quote}
[2021-03-23T15:10:02,222] [INFO ] o.a.s.m.MetricsSystem - Metrics already
registered
java.lang.IllegalArgumentException: Unknown metric type: class
org.apache.beam.runners.spark.metrics.SparkBeamMetric
at
com.codahale.metrics.MetricRegistry.notifyListenerOfAddedMetric(MetricRegistry.java:532)
~[spark-3.0.1-yarn-shuffle.jar:?]
at com.codahale.metrics.MetricRegistry.addListener(MetricRegistry.java:363)
~[spark-3.0.1-yarn-shuffle.jar:?]
at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:93)
~[spark-3.0.1-yarn-shuffle.jar:?]
at
org.apache.spark.metrics.MetricsSystem.registerSource(MetricsSystem.scala:165)
[spark-core_2.12-3.0.1.jar:3.0.1]
at
org.apache.beam.runners.spark.SparkRunner.registerMetricsSource(SparkRunner.java:275)
[beam-runners-spark-2.27.0.jar:?]
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:250)
[beam-runners-spark-2.27.0.jar:?]
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:92)
[beam-runners-spark-2.27.0.jar:?]
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:322)
[beam-sdks-java-harness-2.27.0.jar:?]{quote}
> Spark warns on SparkBeamMetric already registered
> -------------------------------------------------
>
> Key: BEAM-12032
> URL: https://issues.apache.org/jira/browse/BEAM-12032
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: P2
>
> Spark runner in some environments (Spark 3 and Databricks) reports a warning
> on Unknown metric type. The pipeline works well and the metrics appear
> correct but the warning seems to be a problem of the runner.
> {quote}
> [2021-03-23T15:10:02,222] [INFO ] o.a.s.m.MetricsSystem - Metrics already
> registered
> java.lang.IllegalArgumentException: Unknown metric type: class
> org.apache.beam.runners.spark.metrics.SparkBeamMetric
> at
> com.codahale.metrics.MetricRegistry.notifyListenerOfAddedMetric(MetricRegistry.java:532)
> ~[spark-3.0.1-yarn-shuffle.jar:?]
> at com.codahale.metrics.MetricRegistry.addListener(MetricRegistry.java:363)
> ~[spark-3.0.1-yarn-shuffle.jar:?]
> at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:93)
> ~[spark-3.0.1-yarn-shuffle.jar:?]
> at
> org.apache.spark.metrics.MetricsSystem.registerSource(MetricsSystem.scala:165)
> [spark-core_2.12-3.0.1.jar:3.0.1]
> at
> org.apache.beam.runners.spark.SparkRunner.registerMetricsSource(SparkRunner.java:275)
> [beam-runners-spark-2.27.0.jar:?]
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:250)
> [beam-runners-spark-2.27.0.jar:?]
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:92)
> [beam-runners-spark-2.27.0.jar:?]
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:322)
> [beam-sdks-java-harness-2.27.0.jar:?]{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
