Jeremie Piotte created SPARK-33194:
--------------------------------------
Summary: Spark 3.0 streaming metrics are not in Prometheus
Key: SPARK-33194
URL: https://issues.apache.org/jira/browse/SPARK-33194
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 3.0.0
Reporter: Jeremie Piotte
I'm running a Spark 3.0 application (Spark Structured Streaming) on Kubernetes
and I'm trying to use the new native Prometheus metric sink. I'm able to make
it work and get all the metrics described
[here|https://spark.apache.org/docs/latest/monitoring.html#executor-metrics].
However, the metrics I really need are the ones provided upon enabling the
following config: spark.sql.streaming.metricsEnabled, as proposed in [this
Spark Summit
presentation|https://speakerdeck.com/dongjoon/native-support-of-prometheus-monitoring-in-apache-spark-3?slide=32].
Now, even with that config set to "true", I can't see any streaming metrics
under {{/metrics/executors/prometheus}} as advertised. One thing to note is
that I can see them under {{metrics/json}}, therefore, we know that the
configuration was properly applied.
Why aren't streaming metrics sent to the Prometheus sink? Do I need to add some
additional configuration? Is that not supported yet?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
