Jeremie Piotte created SPARK-33194: -------------------------------------- Summary: Spark 3.0 streaming metrics are not in Prometheus Key: SPARK-33194 URL: https://issues.apache.org/jira/browse/SPARK-33194 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 3.0.0 Reporter: Jeremie Piotte
I'm running a Spark 3.0 application (Spark Structured Streaming) on Kubernetes and I'm trying to use the new native Prometheus metric sink. I'm able to make it work and get all the metrics described [here|https://spark.apache.org/docs/latest/monitoring.html#executor-metrics]. However, the metrics I really need are the ones provided upon enabling the following config: spark.sql.streaming.metricsEnabled, as proposed in [this Spark Summit presentation|https://speakerdeck.com/dongjoon/native-support-of-prometheus-monitoring-in-apache-spark-3?slide=32]. Now, even with that config set to "true", I can't see any streaming metrics under {{/metrics/executors/prometheus}} as advertised. One thing to note is that I can see them under {{metrics/json}}, therefore, we know that the configuration was properly applied. Why aren't streaming metrics sent to the Prometheus sink? Do I need to add some additional configuration? Is that not supported yet? -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org