This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new d54c88412 [KYUUBI #5944] Introduce Prometheus and Grafana for Kyuubi
Playground
d54c88412 is described below
commit d54c884124d3826a606677d5258eac73a2427ca6
Author: zwangsheng <[email protected]>
AuthorDate: Mon Jan 15 13:49:00 2024 +0800
[KYUUBI #5944] Introduce Prometheus and Grafana for Kyuubi Playground
# :mag: Description
## Issue References ๐
This pull request fixes #5944
## Describe Your Solution ๐ง
Add prometheus and grafana docker image to docker compose yaml, and by
defaults, prometheus will get kyuubi server's metrics and grafana set
prometheus as default datasource.
Help developers quickly build a dashboard development environment.
## Types of changes :bookmark:
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
In local test
- [x] all container start success
- [x] By defaults, prometheus get kyuubi's metrics
- [x] Grafana take the prometheus as default datasource
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklist ๐
- [x] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #5945 from zwangsheng/KYUUBI#5944.
Closes #5944
031deedeb [Binjie Yang] Update docker/playground/README.md
dbf41e37d [zwangsheng] declare metrics related config
e6efe3071 [zwangsheng] make grafana as anonymous
404dd42ff [zwangsheng] fix comments
f7ca4ae1c [zwangsheng] fix prometheus
089f8d08c [zwangsheng] eof
68720788e [zwangsheng] add license header
cfa6edfef [zwangsheng] [KYUUBI #5944][Improvement] Introduce Prometheus and
Grafana for Kyuubi Playground
Lead-authored-by: zwangsheng <[email protected]>
Co-authored-by: Binjie Yang <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
docker/playground/.env | 2 ++
docker/playground/README.md | 2 ++
docker/playground/compose.yml | 25 ++++++++++++++++++++++
docker/playground/conf/kyuubi-defaults.conf | 3 +++
.../{.env => grafana/datasource/prometheus.yaml} | 24 ++++++++++-----------
.../playground/{.env => prometheus/prometheus.yml} | 17 ++++-----------
6 files changed, 47 insertions(+), 26 deletions(-)
diff --git a/docker/playground/.env b/docker/playground/.env
index e8446fd56..c09540fc7 100644
--- a/docker/playground/.env
+++ b/docker/playground/.env
@@ -28,3 +28,5 @@ SPARK_VERSION=3.4.2
SPARK_BINARY_VERSION=3.4
SPARK_HADOOP_VERSION=3.3.4
ZOOKEEPER_VERSION=3.6.3
+PROMETHEUS_VERSION=2.45.2
+GRAFANA_VERSION=10.0.10
diff --git a/docker/playground/README.md b/docker/playground/README.md
index 66dca2af0..04fdebb52 100644
--- a/docker/playground/README.md
+++ b/docker/playground/README.md
@@ -34,6 +34,8 @@ Kyuubi supply some built-in dataset, after Kyuubi started,
you can run the follo
- MinIO: http://localhost:9001
- PostgreSQL localhost:5432 (username: postgres, password: postgres)
- Spark UI: http://localhost:4040 (available after Spark application launching
by Kyuubi, port may be 4041, 4042... if you launch more than one Spark
applications)
+- Prometheus: http://localhost:9090
+- Grafana: http://localhost:3000 (username: admin, password: admin)
### Shutdown
diff --git a/docker/playground/compose.yml b/docker/playground/compose.yml
index 362b3505b..35a794609 100644
--- a/docker/playground/compose.yml
+++ b/docker/playground/compose.yml
@@ -81,3 +81,28 @@ services:
- metastore
- minio
- zookeeper
+
+ prometheus:
+ # leave `v` here for match prometheus docker image tag
+ image: prom/prometheus:v${PROMETHEUS_VERSION}
+ container_name: prometheus
+ hostname: prometheus
+ ports:
+ - 9090:9090
+ - 8123:8123
+ volumes:
+ - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
+
+ grafana:
+ image: grafana/grafana-oss:${GRAFANA_VERSION}
+ container_name: grafana
+ hostname: grafana
+ ports:
+ - 3000:3000
+ environment:
+ - GF_AUTH_ANONYMOUS_ENABLED=true
+ - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
+ volumes:
+ -
./grafana/datasource/prometheus.yaml:/etc/grafana/provisioning/datasources/prometheus.yaml
+ depends_on:
+ - prometheus
diff --git a/docker/playground/conf/kyuubi-defaults.conf
b/docker/playground/conf/kyuubi-defaults.conf
index e4a674634..75c28850f 100644
--- a/docker/playground/conf/kyuubi-defaults.conf
+++ b/docker/playground/conf/kyuubi-defaults.conf
@@ -27,6 +27,9 @@ kyuubi.session.engine.idle.timeout=PT5M
kyuubi.operation.incremental.collect=true
kyuubi.operation.progress.enabled=true
+kyuubi.metrics.reporters=PROMETHEUS
+kyuubi.metrics.prometheus.port=10019
+
kyuubi.engine.session.initialize.sql \
show namespaces in tpcds; \
show namespaces in tpch; \
diff --git a/docker/playground/.env
b/docker/playground/grafana/datasource/prometheus.yaml
similarity index 71%
copy from docker/playground/.env
copy to docker/playground/grafana/datasource/prometheus.yaml
index e8446fd56..c030390b0 100644
--- a/docker/playground/.env
+++ b/docker/playground/grafana/datasource/prometheus.yaml
@@ -15,16 +15,14 @@
# limitations under the License.
#
-AWS_JAVA_SDK_VERSION=1.12.367
-HADOOP_VERSION=3.3.6
-HIVE_VERSION=2.3.9
-ICEBERG_VERSION=1.4.2
-KYUUBI_VERSION=1.8.0
-KYUUBI_HADOOP_VERSION=3.3.6
-POSTGRES_VERSION=12
-POSTGRES_JDBC_VERSION=42.3.4
-SCALA_BINARY_VERSION=2.12
-SPARK_VERSION=3.4.2
-SPARK_BINARY_VERSION=3.4
-SPARK_HADOOP_VERSION=3.3.4
-ZOOKEEPER_VERSION=3.6.3
+apiVersion: 1
+
+datasources:
+- name: prometheus
+ type: prometheus
+ url: http://prometheus:9090
+ orgId: 1
+ isDefault: false
+ access: server
+ editable: true
+ version: 1
diff --git a/docker/playground/.env
b/docker/playground/prometheus/prometheus.yml
similarity index 71%
copy from docker/playground/.env
copy to docker/playground/prometheus/prometheus.yml
index e8446fd56..91218e374 100644
--- a/docker/playground/.env
+++ b/docker/playground/prometheus/prometheus.yml
@@ -15,16 +15,7 @@
# limitations under the License.
#
-AWS_JAVA_SDK_VERSION=1.12.367
-HADOOP_VERSION=3.3.6
-HIVE_VERSION=2.3.9
-ICEBERG_VERSION=1.4.2
-KYUUBI_VERSION=1.8.0
-KYUUBI_HADOOP_VERSION=3.3.6
-POSTGRES_VERSION=12
-POSTGRES_JDBC_VERSION=42.3.4
-SCALA_BINARY_VERSION=2.12
-SPARK_VERSION=3.4.2
-SPARK_BINARY_VERSION=3.4
-SPARK_HADOOP_VERSION=3.3.4
-ZOOKEEPER_VERSION=3.6.3
+scrape_configs:
+ - job_name: kyuubi
+ static_configs:
+ - targets: ['kyuubi:10019']
