GitHub user matyix opened a pull request: https://github.com/apache/spark/pull/19775
Add support for publishing Spark metrics into Prometheus ## What changes were proposed in this pull request? _Originally this PR was submitted to the Spark on K8S fork [here](https://github.com/apache-spark-on-k8s/spark/pull/531) but has been advised to resend it upstream by @erikerlandson and @foxish. K8S specific items were removed from the PR and been reworked for the Apache version._ Publishing Spark metrics into Prometheus - as highlighted in the [JIRA](https://issues.apache.org/jira/browse/SPARK-22343). Implemented a metrics sink that publishes Spark metrics into Prometheus via [Prometheus Pushgateway](https://prometheus.io/docs/instrumenting/pushing/). Metrics data published by Spark is based on [Dropwizard](http://metrics.dropwizard.io/). The format of Spark metrics is not supported natively by Prometheus thus these are converted using [DropwizardExports](https://prometheus.io/client_java/io/prometheus/client/dropwizard/DropwizardExports.html) prior pushing metrics to the pushgateway. Also the default Prometheus pushgateway client API implementation does not support metrics timestamp thus the client API has been ehanced to enrich metrics data with timestamp. ## How was this patch tested? This PR is not affecting the existing code base and not altering the functionality. Nevertheless, I have executed all `unit and integration` tests. Also this setup has been deployed and been monitored via Prometheus (Prometheus 1.7.1 + Pushgateway 0.3.1). `Manual` testing through deploying a Spark cluster, Prometheus server, Pushgateway and ran SparkPi. You can merge this pull request into a Git repository by running: $ git pull https://github.com/banzaicloud/spark apache_master_prometheus_support Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19775.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19775 ---- commit 579cca96af187cf50fbedf5927cdea4e0bbdff26 Author: Janos Matyas <janos.mat...@gmail.com> Date: 2017-10-17T18:51:50Z Add support for prometheus ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org