GitHub user matyix opened a pull request:
https://github.com/apache/spark/pull/19775
Add support for publishing Spark metrics into Prometheus
## What changes were proposed in this pull request?
_Originally this PR was submitted to the Spark on K8S fork
[here](https://github.com/apache-spark-on-k8s/spark/pull/531) but has been
advised to resend it upstream by @erikerlandson and @foxish. K8S specific items
were removed from the PR and been reworked for the Apache version._
Publishing Spark metrics into Prometheus - as highlighted in the
[JIRA](https://issues.apache.org/jira/browse/SPARK-22343). Implemented a
metrics sink that publishes Spark metrics into Prometheus via [Prometheus
Pushgateway](https://prometheus.io/docs/instrumenting/pushing/). Metrics data
published by Spark is based on [Dropwizard](http://metrics.dropwizard.io/). The
format of Spark metrics is not supported natively by Prometheus thus these are
converted using
[DropwizardExports](https://prometheus.io/client_java/io/prometheus/client/dropwizard/DropwizardExports.html)
prior pushing metrics to the pushgateway.
Also the default Prometheus pushgateway client API implementation does not
support metrics timestamp thus the client API has been ehanced to enrich
metrics data with timestamp.
## How was this patch tested?
This PR is not affecting the existing code base and not altering the
functionality. Nevertheless, I have executed all `unit and integration` tests.
Also this setup has been deployed and been monitored via Prometheus (Prometheus
1.7.1 + Pushgateway 0.3.1).
`Manual` testing through deploying a Spark cluster, Prometheus server,
Pushgateway and ran SparkPi.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/banzaicloud/spark
apache_master_prometheus_support
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19775.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19775
----
commit 579cca96af187cf50fbedf5927cdea4e0bbdff26
Author: Janos Matyas <[email protected]>
Date: 2017-10-17T18:51:50Z
Add support for prometheus
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
