yalindogusahin commented on code in PR #794: URL: https://github.com/apache/spark-kubernetes-operator/pull/794#discussion_r4015317040
########## docs/operations.md: ########## @@ -174,6 +174,123 @@ Note that this requires a CNI plugin that enforces NetworkPolicy; on clusters wi a plugin the policy is silently ignored. Egress traffic of the operator (Kubernetes API server, DNS) is not restricted by this policy. +## Exposing SparkCluster Worker Metrics + +Every `SparkCluster` gets a generated worker `NetworkPolicy` that only admits ingress from pods +carrying the cluster label or the driver-role label, so a Prometheus scraper is locked out by +default. Opening the worker web UI port (`8081`) is not a safe fix: Spark's built-in +`PrometheusServlet` metrics endpoint is served by the same embedded HTTP server as the web UI, so +admitting that port to any source would expose the whole UI, not just metrics. + +The recommended approach is to attach the community +[Prometheus JMX Exporter](https://github.com/prometheus/jmx_exporter) +(`jmx_prometheus_javaagent`) to the worker JVM as a `-javaagent`. The agent opens its own +dedicated HTTP port that serves Prometheus-format metrics, completely decoupled from the web UI +port. Setting this up takes four steps: + +1. Bake the exporter jar into the Spark image used by the cluster: + + ```dockerfile + FROM apache/spark:4.2.0 + ADD https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar \ + /opt/jmx_exporter/jmx_prometheus_javaagent.jar + ``` + +2. Create a `ConfigMap` holding the exporter's mapping/rules config (which MBeans to expose and + how to name them), and mount it into the worker container via a `volumes` / `volumeMounts` + override under `workerSpec.statefulSetSpec.template.spec`: + + ```yaml + apiVersion: v1 + kind: ConfigMap + metadata: + name: jmx-exporter-config + data: + jmx-exporter-config.yaml: | + lowercaseOutputName: true + rules: + - pattern: ".*" Review Comment: Simplified the docs to show the network policy fragment and link to the full example. Also removed the catch-all exporter rule. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
