Hi Qian,
Indeed the metrics available with the Prometheus servlet sink (which is marked
still as experimental) are limited, compared to the full instrumentation, and
this is due to the way it is implemented with a servlet and cannot be easily
extended from what I can see.
You can use another supported metrics sink (see
https://spark.apache.org/docs/latest/monitoring.html#metrics ) if you want to
collect all the metrics are exposed by Spark executors.
For example, I use the graphite sink and then collect metrics into an InfluxDB
instance (see https://github.com/cerndb/spark-dashboard )
An additional comment is that there is room for having more sinks available for
Apache Spark metrics, notably for InfluxDB and for Prometheus (gateway), if
someone is interested in working on that.
Best,
Luca
From: Qian Sun <qian.sun2...@gmail.com>
Sent: Friday, February 10, 2023 05:05
To: dev <dev@spark.apache.org>; user.spark <u...@spark.apache.org>
Subject: Executor metrics are missing on prometheus sink
Setting up prometheus sink in this way:
-c spark.ui.prometheus.enabled=true
-c spark.executor.processTreeMetrics.enabled=true
-c spark.metrics.conf=/spark/conf/metric.properties
metric.properties:{}
*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet
*.sink.prometheusServlet.path=/metrics/prometheus
Result:
Both of these endpoints have some metrics
<driver-ip>:4040/metrics/prometheus
<driver-ip>:4040/metrics/executors/prometheus
But the executor one misses metrics under the executor namespace described here:
https://spark.apache.org/docs/latest/monitoring.html#component-instance--executor
How to expose executor metrics on spark exeuctors pod?
Any help will be appreciated.
--
Regards,
Qian Sun
