This is an automated email from the ASF dual-hosted git repository.
Alanxtl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/dubbo-website.git
The following commit(s) were added to refs/heads/master by this push:
new 343f94b0281 docs: improve metrics monitoring guide (#3219)
343f94b0281 is described below
commit 343f94b02819309762662afdfbc09743595e2c5a
Author: xiaobaicai66695 <[email protected]>
AuthorDate: Sun Aug 9 12:45:10 2026 +0800
docs: improve metrics monitoring guide (#3219)
---
.../tutorial/observability/rpc_metrics.md | 421 +++++++++-----------
.../tutorial/observability/rpc_metrics.md | 427 +++++++++------------
2 files changed, 366 insertions(+), 482 deletions(-)
diff --git
a/content/en/overview/mannual/golang-sdk/tutorial/observability/rpc_metrics.md
b/content/en/overview/mannual/golang-sdk/tutorial/observability/rpc_metrics.md
index b2261b58c1f..1325075c818 100644
---
a/content/en/overview/mannual/golang-sdk/tutorial/observability/rpc_metrics.md
+++
b/content/en/overview/mannual/golang-sdk/tutorial/observability/rpc_metrics.md
@@ -10,346 +10,291 @@ weight: 2
# Dubbo-Go Metrics Monitoring
-Dubbo-Go supports runtime metrics collection and integration with **Prometheus
+ Grafana** to build observability for microservices.
+Dubbo-Go can collect runtime metrics for RPC calls, metadata, registries,
config centers, and other components. This guide uses <a
href="https://github.com/apache/dubbo-go-samples/tree/main/observability/prometheus_grafana";
target="_blank">dubbo-go-samples/observability/prometheus_grafana</a> to show
both Prometheus Pull mode and Pushgateway mode.
-This example covers two monitoring modes:
+> Note: some older descriptions may still mention
`metrics/prometheus_grafana`. The current sample directory is
`observability/prometheus_grafana`.
-* **Pull Mode**: Prometheus scrapes metrics from Dubbo-Go applications.
Recommended for long-running services.
-* **Push Mode**: Dubbo-Go applications push metrics to Pushgateway.
Recommended only for short-lived jobs.
+## 1. Monitoring Modes
-Example source code:
+### 1.1 Pull Mode
->
[https://github.com/apache/dubbo-go-samples/tree/main/metrics](https://github.com/apache/dubbo-go-samples/tree/main/metrics)
-
-## 1. Monitoring Architecture
+```text
+Dubbo-Go application exposes /prometheus or /metrics
+ |
+ v
+Prometheus scrapes the application
+ |
+ v
+Grafana queries Prometheus and renders dashboards
+```
-### 1.1 Pull Mode (Recommended for production)
+Pull mode is the standard Prometheus model and is recommended for long-running
Dubbo-Go providers and consumers. In this sample:
-```
-Dubbo-Go Application ---> Prometheus ---> Grafana
- (exposes /metrics or /prometheus endpoint)
-```
+| Application | Metrics endpoint |
+| --- | --- |
+| go-server | `http://localhost:9099/prometheus` |
+| go-client | `http://localhost:9097/prometheus` |
-Prometheus actively scrapes metrics from Dubbo-Go applications.
+The Dubbo-Go default metrics endpoint is `http://localhost:9090/metrics`. This
sample changes it with `metrics.WithPort(9099)`, `metrics.WithPort(9097)`, and
`metrics.WithPath("/prometheus")`.
-### 1.2 Push Mode (For short-lived jobs)
+### 1.2 Pushgateway Mode
+```text
+Dubbo-Go application pushes metrics
+ |
+ v
+Pushgateway stores the metrics temporarily
+ |
+ v
+Prometheus scrapes Pushgateway
+ |
+ v
+Grafana queries Prometheus and renders dashboards
```
-Dubbo-Go Application ---> Pushgateway ---> Prometheus ---> Grafana
-```
-
-Applications push metrics to Pushgateway. Prometheus scrapes Pushgateway.
-Pushgateway is designed for **short-lived jobs (batch / cron)** and is not
recommended for long-running services.
+Pushgateway is better suited to short-lived jobs such as batch or cron jobs.
Prefer Pull mode for long-running services. If you use Pushgateway, configure
cleanup so stale metrics do not remain after a job exits.
-## 2. Components Overview
+## 2. Prepare the Sample
-| Component | Port | Description |
-| ----------- | ---- | -------------------------------- |
-| Grafana | 3000 | Metrics visualization dashboard |
-| Prometheus | 9090 | Metrics storage and query engine |
-| Pushgateway | 9091 | Receives pushed metrics |
-| go-server metrics endpoint | 9099 in this sample | Provider metrics in Pull
mode |
-| go-client metrics endpoint | 9097 in this sample | Consumer metrics in Pull
mode |
+Clone the samples repository and enter the sample directory:
-If you use Dubbo-Go defaults instead of this sample, the default metrics
endpoint is `http://localhost:9090/metrics`. This sample overrides the metrics
path to `/prometheus`.
+```bash
+git clone --depth 1 https://github.com/apache/dubbo-go-samples.git
+cd dubbo-go-samples/observability/prometheus_grafana
+```
-## 3. Quick Start
+The directory contains:
-### 3.1 Start the Monitoring Stack
+| File or directory | Description |
+| --- | --- |
+| `docker-compose.yml` | Starts ZooKeeper, Prometheus, Pushgateway, and
Grafana |
+| `prometheus_pull.yml` | Prometheus configuration for Pull mode |
+| `prometheus_push.yml` | Prometheus configuration for Pushgateway mode |
+| `go-server/cmd/main.go` | Provider sample, RPC port `20000`, metrics port
`9099` |
+| `go-client/cmd/main.go` | Consumer sample, continuously calls the provider,
metrics port `9097` |
+| `grafana.json` | Dubbo Grafana dashboard JSON |
-Navigate to:
+Start the Docker stack:
```bash
-cd metrics/prometheus_grafana
+docker compose up -d
```
-Start services:
+If your environment still uses Docker Compose v1, run:
```bash
docker-compose up -d
```
-Access:
+After startup, visit:
+
+| Component | URL |
+| --- | --- |
+| Grafana | `http://localhost:3000` |
+| Prometheus | `http://localhost:9090` |
+| Pushgateway | `http://localhost:9091` |
+| ZooKeeper | `127.0.0.1:2181` |
-* Grafana: [http://localhost:3000](http://localhost:3000)
-* Prometheus: [http://localhost:9090](http://localhost:9090)
-* Pushgateway: [http://localhost:9091](http://localhost:9091)
+## 3. Pull Mode Workflow
-### 3.2 Configure Environment Variables
+`docker-compose.yml` mounts `prometheus_pull.yml` by default, so the stack is
ready for Pull mode after startup.
-Both client and server share the same configuration:
+1. Set the registry address:
```bash
export ZK_ADDRESS="127.0.0.1:2181"
-
-# Required for Push mode
-export PUSHGATEWAY_URL="127.0.0.1:9091"
-export JOB_NAME="dubbo-service"
-
-# Optional
-export PUSHGATEWAY_USER="username"
-export PUSHGATEWAY_PASS="1234"
```
-### 3.3 Start Dubbo-Go Server
+For Windows PowerShell:
-```bash
-go run ./go-server/cmd/main.go
+```powershell
+$env:ZK_ADDRESS="127.0.0.1:2181"
```
-### 3.4 Start Dubbo-Go Client
-
-#### Default (Push Mode)
+2. Start the provider:
```bash
-go run ./go-client/cmd/main.go
+go run ./go-server/cmd/main.go --push=false
```
-#### Pull Mode
+3. Open another terminal, enter the same directory, and start the consumer:
```bash
+cd dubbo-go-samples/observability/prometheus_grafana
+export ZK_ADDRESS="127.0.0.1:2181"
go run ./go-client/cmd/main.go --push=false
-go run ./go-server/cmd/main.go --push=false
```
-### 3.5 Verify Metrics
-
-#### Push Mode
+4. Visit the metrics endpoints exposed by the applications:
-Open:
-
-```
-http://localhost:9091/metrics
+```bash
+curl http://localhost:9099/prometheus
+curl http://localhost:9097/prometheus
```
-#### Pull Mode
-
-`<app_port>` means the HTTP metrics port exposed by the Dubbo-Go application
itself, not the Prometheus or Pushgateway port.
-
-In this sample:
-
-* Provider: `http://localhost:9099/prometheus`
-* Consumer: `http://localhost:9097/prometheus`
+The output includes metrics such as `dubbo_provider_`, `dubbo_consumer_`, and
`dubbo_application_`. For example:
-These ports are defined in
[`metrics/prometheus_grafana/prometheus_pull.yml`](https://github.com/apache/dubbo-go-samples/blob/main/metrics/prometheus_grafana/prometheus_pull.yml).
+```text
+dubbo_provider_requests_total
+dubbo_consumer_requests_succeed_total
+```
-If you use your own Dubbo-Go application instead of this sample, replace the
port with your application's metrics port.
+5. Open `http://localhost:9090/targets` and verify that `dubbo-provider` and
`dubbo-consumer` are `UP`.
-## 4. Grafana Configuration
+## 4. Pushgateway Mode Workflow
-### 4.1 Add Prometheus Data Source
+In Pushgateway mode, Prometheus scrapes Pushgateway instead of scraping the
Dubbo-Go applications directly.
-1. Open [http://localhost:3000](http://localhost:3000)
-2. Default credentials: `admin / admin`
-3. Navigate to:
+1. Change the Prometheus volume in `docker-compose.yml` from the Pull
configuration to the Pushgateway configuration:
-```
-Home → Connections → Data sources
+```yaml
+volumes:
+ - ./prometheus_push.yml:/etc/prometheus/prometheus.yml
```
-4. Click **Add new data source**
-5. Select **Prometheus**
-6. Enter:
+2. Restart the monitoring stack:
+```bash
+docker compose down
+docker compose up -d
```
-http://host.docker.internal:9090
-```
-
-> Note: `host.docker.internal` allows Docker containers to access the host
network. Replace with your actual IP if necessary.
-
-7. Click **Save & Test**
-### 4.2 Import Dubbo Dashboard
+3. Set environment variables:
-1. Navigate to:
+```bash
+export ZK_ADDRESS="127.0.0.1:2181"
+export PUSHGATEWAY_URL="127.0.0.1:9091"
+export JOB_NAME="dubbo-service"
+# Set these only when Pushgateway uses Basic Auth
+export PUSHGATEWAY_USER="username"
+export PUSHGATEWAY_PASS="1234"
```
-Home → Dashboards → New → Import
-```
-
-2. Import the dashboard with one of these methods:
-* Upload
[`grafana.json`](https://github.com/apache/dubbo-go-samples/blob/main/metrics/prometheus_grafana/grafana.json)
from the sample directory
-* Enter Grafana dashboard ID `19294` (`Dubbo Observability`) and click **Load**
-* Or download the JSON from [Grafana
Labs](https://grafana.com/grafana/dashboards/19294-dubbo-observability/) and
upload it
+4. Start the provider and consumer:
-3. The sample repository already includes the dashboard file at
`metrics/prometheus_grafana/grafana.json`, so uploading that file is the most
direct option.
-
-4. Select the Prometheus data source
-
-5. Click **Import**
-
-### 4.3 View Dashboard
-
-You will see:
-
-* QPS
-* Success rate
-* Latency (P99)
-* Consumer / Provider request statistics
-* Error rate
-
-Metrics update dynamically as the client continuously calls the server.
-
-## 5. Pushgateway Zombie Metrics Problem
-
-### 5.1 Problem Description
-
-Pushgateway **does not automatically delete old metrics**.
-
-If a job stops:
-
-* Its metrics remain stored
-* This may pollute monitoring data
-
-### 5.2 Solution 1: Application-side Cleanup (Implemented)
-
-Mechanism:
+```bash
+go run ./go-server/cmd/main.go
+```
-* Register `job_pushed_at_seconds`
-* Periodically update timestamp
-* Automatically call DELETE API on graceful shutdown
+In another terminal:
-### 5.3 Solution 2: Production-grade Cleaner (Recommended)
+```bash
+cd dubbo-go-samples/observability/prometheus_grafana
+export ZK_ADDRESS="127.0.0.1:2181"
+export PUSHGATEWAY_URL="127.0.0.1:9091"
+export JOB_NAME="dubbo-service"
+go run ./go-client/cmd/main.go
+```
-Tool repository:
+5. Visit the Pushgateway metrics endpoint:
->
[apache/dubbo-go-samples/tree/main/tools/pgw-cleaner](https://github.com/apache/dubbo-go-samples/tree/main/tools/pgw-cleaner)
+```bash
+curl http://localhost:9091/metrics
+```
-Detailed documentation:
+6. Open `http://localhost:9090/targets` and verify that the `pushgateway`
target is `UP`.
-*
[README.md](https://github.com/apache/dubbo-go-samples/blob/main/tools/pgw-cleaner/README.md)
+The sample registers `job_pushed_at_seconds` and calls the Pushgateway DELETE
API on graceful shutdown. If the process is killed forcefully, cleanup may not
run. For production, use <a
href="https://github.com/apache/dubbo-go-samples/tree/main/tools/pgw-cleaner";
target="_blank">pgw-cleaner</a> or another cleanup process.
-This tool lives in the `apache/dubbo-go-samples` repository, not in the
`apache/dubbo-go` core repository.
+## 5. Prometheus Configuration
-Purpose:
+### 5.1 Pull Configuration
-* Detect expired jobs
-* Automatically clean zombie metrics
+`prometheus_pull.yml` scrapes `/prometheus` from both the provider and
consumer:
-## 6. Troubleshooting
+```yaml
+global:
+ evaluation_interval: 15s
+ scrape_interval: 15s
+scrape_configs:
+ - job_name: dubbo-provider
+ scrape_interval: 15s
+ scrape_timeout: 5s
+ metrics_path: /prometheus
+ static_configs:
+ - targets: ['host.docker.internal:9099']
+ - job_name: dubbo-consumer
+ scrape_interval: 15s
+ scrape_timeout: 5s
+ metrics_path: /prometheus
+ static_configs:
+ - targets: ['host.docker.internal:9097']
+```
-### 6.1 Grafana Shows "No Data"
+`host.docker.internal` lets the Prometheus container reach Dubbo-Go processes
running on the host. If Prometheus and the applications run in the same Docker
network, replace it with the container name. On Linux, if this hostname is
unavailable, use the host IP or add a `host-gateway` mapping in Docker Compose.
-Check:
+### 5.2 Pushgateway Configuration
-* Prometheus data source connection is successful
-* Prometheus → Status → Targets → pushgateway is **UP**
-* Query:
+`prometheus_push.yml` scrapes only Pushgateway:
-```
-dubbo_consumer_requests_succeed_total
+```yaml
+global:
+ evaluation_interval: 15s
+ scrape_interval: 15s
+scrape_configs:
+ - job_name: 'pushgateway'
+ static_configs:
+ - targets: ['host.docker.internal:9091']
+ honor_labels: true
```
-returns results
+`honor_labels: true` keeps labels such as `job` and `instance` that were
pushed by the application, which helps Grafana display data by service
dimension.
-### 6.2 host.docker.internal Not Reachable
+## 6. Import the Grafana Dashboard
-Replace it with your actual host IP:
+1. Open `http://localhost:3000`. The default credentials are `admin` / `admin`.
+2. Go to `Home -> Connections -> Data sources` and click `Add new data source`.
+3. Select `Prometheus` and set the URL to `http://host.docker.internal:9090`.
If Grafana cannot reach this address, use the actual host IP or another address
reachable from the Grafana container.
+4. Click `Save & test`.
+5. Go to `Home -> Dashboards -> New -> Import`.
+6. Upload `grafana.json` from the sample directory, or paste the file content
into the import box. You can also import Grafana dashboard ID `19294` for
`Dubbo Observability`.
+7. Select the Prometheus data source and click `Import`.
-* Update `metrics/prometheus_grafana/prometheus_pull.yml`
-* Update Grafana data source URL
+Once the consumer starts calling the provider, the dashboard panels for QPS,
success rate, P99 latency, consumer/provider request counts, and error rate
will keep updating.
-## 7. Kubernetes Deployment
+## 7. Troubleshooting
-Recommended:
+### 7.1 Port Conflicts
-> kube-prometheus
->
[https://github.com/prometheus-operator/kube-prometheus](https://github.com/prometheus-operator/kube-prometheus)
+If `3000`, `9090`, `9091`, or `2181` is already in use, update the port
mappings in `docker-compose.yml`. If `9099` or `9097` is already in use, update
`metrics.WithPort(...)` in `go-server/cmd/main.go` or `go-client/cmd/main.go`,
and update the targets in `prometheus_pull.yml` accordingly.
-### 7.1 Create PodMonitor
+### 7.2 Prometheus Targets Are DOWN
-Create `dubboPodMonitor.yaml`:
+First verify the endpoints from the host:
-```yaml
-apiVersion: monitoring.coreos.com/v1
-kind: PodMonitor
-metadata:
- name: dubbo-pod-monitor
- namespace: monitoring
-spec:
- namespaceSelector:
- matchNames:
- - dubbo-system
- selector:
- matchLabels:
- app-type: dubbo
- podMetricsEndpoints:
- - port: metrics
- path: /prometheus
+```bash
+curl http://localhost:9099/prometheus
+curl http://localhost:9097/prometheus
+curl http://localhost:9091/metrics
```
-### 7.2 Optional: Add RBAC Permissions When RBAC Is Enabled
-
-If your cluster enforces RBAC, grant Prometheus permission to read Pods in
`dubbo-system`:
+If the host can access them but the Prometheus container cannot, the issue is
usually Docker networking. Replace `host.docker.internal` with the real host
IP, or add this to the Prometheus service on Linux:
```yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
- namespace: dubbo-system
- name: pod-reader
-rules:
- - apiGroups: [""]
- resources: ["pods"]
- verbs: ["get", "list", "watch"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
- name: pod-reader-binding
- namespace: dubbo-system
-roleRef:
- apiGroup: rbac.authorization.k8s.io
- kind: Role
- name: pod-reader
-subjects:
- - kind: ServiceAccount
- name: prometheus-k8s
- namespace: monitoring
+extra_hosts:
+ - "host.docker.internal:host-gateway"
```
-If your Prometheus installation uses a different service account, replace the
`subjects` section accordingly.
+### 7.3 Grafana Shows No Data
-### 7.3 Deploy Application
+Confirm that the Prometheus data source passes `Save & test`; check
`http://localhost:9090/targets` for `UP` targets; then query
`dubbo_consumer_requests_succeed_total` or `dubbo_provider_requests_total` in
Prometheus. If the query is empty, make sure the client is continuously calling
the server and wait for one scrape interval.
-```bash
-kubectl apply -f Deployment.yaml
-```
-
-### 7.4 Verify
+### 7.4 Pushgateway Has Stale Jobs
-Visit:
+Pushgateway does not automatically remove old metrics. Graceful shutdown of
the sample process triggers cleanup; production deployments should use
`pgw-cleaner` or call the Pushgateway DELETE API for expired jobs.
-```
-http://<prometheus-nodeport>/targets
-```
+### 7.5 Should I Visit `/metrics` or `/prometheus`?
-Ensure your pods show:
-
-```
-UP
-```
+Dubbo-Go defaults to port `9090` and path `/metrics`. This sample explicitly
uses `/prometheus`; the server listens on `9099`, and the client listens on
`9097`. Always use the actual values configured by `metrics.WithPort(...)` and
`metrics.WithPath(...)` in your application.
## 8. Production Recommendations
-| Scenario | Recommended Mode |
-| --------------------- | ----------------- |
-| Long-running services | Pull |
-| Short-lived jobs | Push |
-| Kubernetes | Pull + PodMonitor |
-| Pushgateway usage | Use pgw-cleaner |
-
-## 9. Summary
-
-Dubbo-Go provides:
-
-* Pull-based Prometheus integration
-* Push-based Pushgateway integration
-* Docker quick-start stack
-* Kubernetes PodMonitor support
-* Grafana dashboards
-* Zombie metric cleanup support
+| Scenario | Recommended setup |
+| --- | --- |
+| Long-running providers/consumers | Pull mode |
+| Short-lived jobs | Pushgateway mode |
+| Kubernetes | Pull mode + PodMonitor |
+| Pushgateway usage | Cleanup process or DELETE API |
-With this setup, you can build a complete Dubbo-Go observability system.
+In Kubernetes, use Prometheus Operator's PodMonitor to scrape the application
metrics port, and set `path` to the application's actual `/prometheus` or
`/metrics` path.
diff --git
a/content/zh-cn/overview/mannual/golang-sdk/tutorial/observability/rpc_metrics.md
b/content/zh-cn/overview/mannual/golang-sdk/tutorial/observability/rpc_metrics.md
index 61a0da8d7f6..6fba9b03094 100644
---
a/content/zh-cn/overview/mannual/golang-sdk/tutorial/observability/rpc_metrics.md
+++
b/content/zh-cn/overview/mannual/golang-sdk/tutorial/observability/rpc_metrics.md
@@ -8,354 +8,293 @@ type: docs
weight: 2
---
-
# Dubbo-Go 指标监控
-Dubbo-Go 支持采集运行态 Metrics 指标，并接入 **Prometheus + Grafana** 实现微服务可观测性。
-
-当前示例支持两种监控模式：
+Dubbo-Go 可以采集 RPC、元数据、注册中心、配置中心等运行态指标，并通过 Prometheus 与 Grafana 构建可观测性看板。本文以 <a
href="https://github.com/apache/dubbo-go-samples/tree/main/observability/prometheus_grafana";
target="_blank">dubbo-go-samples/observability/prometheus_grafana</a> 为例，说明
Pull 与 Pushgateway 两种运行方式。
-* **Pull 模式**：Prometheus 主动抓取 Dubbo-Go 应用暴露的指标，适合长期运行的服务。
-* **Push 模式**：Dubbo-Go 应用主动将指标推送到 Pushgateway，仅建议用于短生命周期任务。
+> 说明：部分旧文档或任务描述可能仍写作 `metrics/prometheus_grafana`，当前 sample 仓库中的实际目录是
`observability/prometheus_grafana`。
-示例源码：
+## 1. 工作模式
->
[https://github.com/apache/dubbo-go-samples/tree/main/metrics](https://github.com/apache/dubbo-go-samples/tree/main/metrics)
+### 1.1 Pull 模式
-## 1. 监控架构说明
+```text
+Dubbo-Go 应用暴露 /prometheus 或 /metrics
+ |
+ v
+Prometheus 定时拉取
+ |
+ v
+Grafana 查询 Prometheus 并展示大盘
+```
-### 1.1 Pull 模式（推荐生产模式）
+Pull 模式是 Prometheus 的标准模式，适合长期运行的 Dubbo-Go Provider 与 Consumer。当前 sample 中：
-```
-Dubbo-Go 应用 ---> Prometheus ---> Grafana
- (暴露 /metrics 或 /prometheus 接口)
-```
+| 应用 | 指标地址 |
+| --- | --- |
+| go-server | `http://localhost:9099/prometheus` |
+| go-client | `http://localhost:9097/prometheus` |
-Prometheus 主动抓取 Dubbo-Go 应用指标。
+Dubbo-Go 默认指标地址是 `http://localhost:9090/metrics`。sample 通过
`metrics.WithPort(9099)`、`metrics.WithPort(9097)` 和
`metrics.WithPath("/prometheus")` 改成了上面的端口与路径。
-### 1.2 Push 模式（适用于短生命周期任务）
+### 1.2 Pushgateway 模式
-```
-Dubbo-Go 应用 ---> Pushgateway ---> Prometheus ---> Grafana
+```text
+Dubbo-Go 应用推送指标
+ |
+ v
+Pushgateway 暂存指标
+ |
+ v
+Prometheus 拉取 Pushgateway
+ |
+ v
+Grafana 查询 Prometheus 并展示大盘
```
-应用主动推送指标到 Pushgateway，Prometheus 再拉取。
+Pushgateway 更适合 batch、cron job 等短生命周期任务。长期运行的服务优先使用 Pull 模式；如果必须使用
Pushgateway，请配合清理机制，避免停止后的旧指标继续留在 Pushgateway 中。
-Pushgateway 适用于 **短生命周期任务（如 batch / cron job）**，不推荐用于长期运行的服务。
+## 2. 准备示例
-## 2. 示例组件说明
+克隆 samples 仓库并进入示例目录：
-| 组件 | 端口 | 说明 |
-| ----------- | ---- | -------------------- |
-| Grafana | 3000 | 指标可视化 |
-| Prometheus | 9090 | 指标存储与查询 |
-| Pushgateway | 9091 | 接收应用推送指标 |
-| go-server 指标端口 | 本示例中为 9099 | Pull 模式下 Provider 指标暴露端口 |
-| go-client 指标端口 | 本示例中为 9097 | Pull 模式下 Consumer 指标暴露端口 |
-
-如果不使用本示例，而是直接使用 Dubbo-Go 默认配置，则默认指标端点为 `http://localhost:9090/metrics`。当前
sample 将指标路径改成了 `/prometheus`。
+```bash
+git clone --depth 1 https://github.com/apache/dubbo-go-samples.git
+cd dubbo-go-samples/observability/prometheus_grafana
+```
-## 3. 快速开始
+该目录包含：
-### 3.1 启动监控服务栈
+| 文件或目录 | 说明 |
+| --- | --- |
+| `docker-compose.yml` | 启动 ZooKeeper、Prometheus、Pushgateway、Grafana |
+| `prometheus_pull.yml` | Pull 模式的 Prometheus 配置 |
+| `prometheus_push.yml` | Pushgateway 模式的 Prometheus 配置 |
+| `go-server/cmd/main.go` | Provider 示例，RPC 端口 `20000`，指标端口 `9099` |
+| `go-client/cmd/main.go` | Consumer 示例，持续调用 Provider，指标端口 `9097` |
+| `grafana.json` | Dubbo Grafana dashboard JSON |
-进入目录：
+启动 Docker 组件：
```bash
-cd metrics/prometheus_grafana
+docker compose up -d
```
-启动监控组件：
+如果你的环境还在使用 Docker Compose v1，可以执行：
```bash
docker-compose up -d
```
-访问地址：
+启动后可访问：
-* Grafana: [http://localhost:3000](http://localhost:3000)
-* Prometheus: [http://localhost:9090](http://localhost:9090)
-* Pushgateway: [http://localhost:9091](http://localhost:9091)
+| 组件 | 地址 |
+| --- | --- |
+| Grafana | `http://localhost:3000` |
+| Prometheus | `http://localhost:9090` |
+| Pushgateway | `http://localhost:9091` |
+| ZooKeeper | `127.0.0.1:2181` |
-### 3.2 配置环境变量
+## 3. Pull 模式运行流程
-客户端与服务端使用相同环境变量：
+`docker-compose.yml` 默认挂载 `prometheus_pull.yml`，因此直接启动 Docker 组件后即可按 Pull 模式验证。
+
+1. 设置注册中心地址：
```bash
export ZK_ADDRESS="127.0.0.1:2181"
-
-# Push 模式必需
-export PUSHGATEWAY_URL="127.0.0.1:9091"
-export JOB_NAME="dubbo-service"
-
-# 可选
-export PUSHGATEWAY_USER="username"
-export PUSHGATEWAY_PASS="1234"
```
-### 3.3 启动 Dubbo-Go 服务端
+Windows PowerShell 使用：
-```bash
-go run ./go-server/cmd/main.go
+```powershell
+$env:ZK_ADDRESS="127.0.0.1:2181"
```
-### 3.4 启动 Dubbo-Go 客户端
-
-#### 默认 Push 模式
+2. 启动 Provider：
```bash
-go run ./go-client/cmd/main.go
+go run ./go-server/cmd/main.go --push=false
```
-#### 使用 Pull 模式
+3. 新开终端，进入同一目录后启动 Consumer：
```bash
+cd dubbo-go-samples/observability/prometheus_grafana
+export ZK_ADDRESS="127.0.0.1:2181"
go run ./go-client/cmd/main.go --push=false
-go run ./go-server/cmd/main.go --push=false
```
-### 3.5 验证指标
-
-#### Push 模式
+4. 访问应用暴露的指标端点：
-访问：
-
-```
-http://localhost:9091/metrics
+```bash
+curl http://localhost:9099/prometheus
+curl http://localhost:9097/prometheus
```
-#### Pull 模式
-
-`<应用端口>` 指的是 Dubbo-Go 应用自身暴露 Prometheus 指标的 HTTP 端口，不是 Prometheus 或
Pushgateway 的端口。
-
-在当前 sample 中：
-
-* Provider：`http://localhost:9099/prometheus`
-* Consumer：`http://localhost:9097/prometheus`
+输出中可以看到 `dubbo_provider_`、`dubbo_consumer_`、`dubbo_application_` 等指标。例如：
-这两个端口定义在
[`metrics/prometheus_grafana/prometheus_pull.yml`](https://github.com/apache/dubbo-go-samples/blob/main/metrics/prometheus_grafana/prometheus_pull.yml)
中。
+```text
+dubbo_provider_requests_total
+dubbo_consumer_requests_succeed_total
+```
-如果你使用的是自己的 Dubbo-Go 应用，请将端口替换成应用实际配置的指标端口。
+5. 打开 `http://localhost:9090/targets`，确认 `dubbo-provider` 与 `dubbo-consumer`
两个 target 为 `UP`。
-## 4. Grafana 配置
+## 4. Pushgateway 模式运行流程
-### 4.1 添加 Prometheus 数据源
+Pushgateway 模式需要让 Prometheus 拉取 Pushgateway，而不是直接拉取 Dubbo-Go 应用。
-1. 打开 [http://localhost:3000](http://localhost:3000)
-2. 默认账号：admin / admin
-3. 进入：
+1. 将 `docker-compose.yml` 中 Prometheus 的配置文件挂载从 Pull 配置改为 Push 配置：
-```
-Home → Connections → Data sources
+```yaml
+volumes:
+ - ./prometheus_push.yml:/etc/prometheus/prometheus.yml
```
-4. 点击 **Add new data source**
-5. 选择 Prometheus
-6. 填写：
+2. 重新启动监控组件：
-```
-http://host.docker.internal:9090
+```bash
+docker compose down
+docker compose up -d
```
-> 说明：`host.docker.internal` 用于让 Docker 容器访问宿主机网络。如果该地址不可用，请替换为宿主机实际 IP。
+3. 设置环境变量：
-7. 点击 Save & Test
-
-### 4.2 导入 Dubbo 监控大盘
-
-1. 进入：
+```bash
+export ZK_ADDRESS="127.0.0.1:2181"
+export PUSHGATEWAY_URL="127.0.0.1:9091"
+export JOB_NAME="dubbo-service"
+# 如果 Pushgateway 开启了 Basic Auth，再设置以下变量
+export PUSHGATEWAY_USER="username"
+export PUSHGATEWAY_PASS="1234"
```
-Home → Dashboards → New → Import
-```
-
-2. 使用以下任一方式导入：
-
-* 直接上传 sample 目录中的
[`grafana.json`](https://github.com/apache/dubbo-go-samples/blob/main/metrics/prometheus_grafana/grafana.json)
-* 输入 Grafana dashboard ID `19294`（`Dubbo Observability`）后点击 **Load**
-* 或从 [Grafana
Labs](https://grafana.com/grafana/dashboards/19294-dubbo-observability/) 下载
JSON 文件再上传
-3. sample 仓库已经直接提供 `metrics/prometheus_grafana/grafana.json`，因此优先上传这个文件即可。
+4. 启动 Provider 与 Consumer：
-4. 选择 Prometheus 数据源
-5. 点击 Import
-
-### 4.3 查看效果
-
-你将看到：
-
-* QPS
-* 成功率
-* 请求延迟 P99
-* Consumer / Provider 调用统计
-* 错误率
-
-## 5. Pushgateway 僵尸指标问题
-
-### 5.1 问题说明
-
-Pushgateway 默认：
-
-> 不会自动删除旧指标
-
-任务停止后：
-
-* 指标仍然保留
-* 会导致数据污染
-
-### 5.2 方案一：应用侧自动清理（已实现）
-
-机制：
+```bash
+go run ./go-server/cmd/main.go
+```
-* 注册 `job_pushed_at_seconds`
-* 定期更新时间戳
-* 优雅退出时自动 DELETE
+新开终端：
-### 5.3 方案二：运维清理器（推荐生产使用）
+```bash
+cd dubbo-go-samples/observability/prometheus_grafana
+export ZK_ADDRESS="127.0.0.1:2181"
+export PUSHGATEWAY_URL="127.0.0.1:9091"
+export JOB_NAME="dubbo-service"
+go run ./go-client/cmd/main.go
+```
-工具仓库位置：
+5. 访问 Pushgateway 指标：
->
[apache/dubbo-go-samples/tree/main/tools/pgw-cleaner](https://github.com/apache/dubbo-go-samples/tree/main/tools/pgw-cleaner)
+```bash
+curl http://localhost:9091/metrics
+```
-详细说明：
+6. 打开 `http://localhost:9090/targets`，确认 `pushgateway` target 为 `UP`。
-*
[README_CN.md](https://github.com/apache/dubbo-go-samples/blob/main/tools/pgw-cleaner/README_CN.md)
+示例应用注册了 `job_pushed_at_seconds`，并在优雅退出时调用 Pushgateway DELETE API 清理自身
job。直接强制结束进程时可能来不及清理，生产环境建议配合 <a
href="https://github.com/apache/dubbo-go-samples/tree/main/tools/pgw-cleaner";
target="_blank">pgw-cleaner</a> 定期清理过期指标。
-该工具位于 `apache/dubbo-go-samples` 仓库中，不在 `apache/dubbo-go` 主仓库内。
+## 5. Prometheus 配置说明
-用于：
+### 5.1 Pull 配置
-* 自动扫描过期指标
-* 定期清理僵尸 job
+`prometheus_pull.yml` 直接抓取 Provider 与 Consumer 暴露的 `/prometheus`：
-## 6. 常见问题
+```yaml
+global:
+ evaluation_interval: 15s
+ scrape_interval: 15s
+scrape_configs:
+ - job_name: dubbo-provider
+ scrape_interval: 15s
+ scrape_timeout: 5s
+ metrics_path: /prometheus
+ static_configs:
+ - targets: ['host.docker.internal:9099']
+ - job_name: dubbo-consumer
+ scrape_interval: 15s
+ scrape_timeout: 5s
+ metrics_path: /prometheus
+ static_configs:
+ - targets: ['host.docker.internal:9097']
+```
-### 6.1 Grafana 显示 No Data
+`host.docker.internal` 表示 Prometheus 容器访问宿主机上的 Dubbo-Go 进程。如果 Prometheus
与应用都运行在同一个 Docker 网络内，可改成容器名；如果是在 Linux 环境且该域名不可用，请改成宿主机实际 IP，或在 compose 中增加
`host-gateway` 映射。
-请检查：
+### 5.2 Pushgateway 配置
-* Prometheus 数据源是否测试成功
-* Prometheus → Status → Targets 是否为 UP
-* 查询：
+`prometheus_push.yml` 只抓取 Pushgateway：
-```
-dubbo_consumer_requests_succeed_total
+```yaml
+global:
+ evaluation_interval: 15s
+ scrape_interval: 15s
+scrape_configs:
+ - job_name: 'pushgateway'
+ static_configs:
+ - targets: ['host.docker.internal:9091']
+ honor_labels: true
```
-是否有数据
+`honor_labels: true` 会保留应用推送到 Pushgateway 的 `job`、`instance` 等标签，便于 Grafana
dashboard 按服务维度展示。
-### 6.2 host.docker.internal 无法访问
+## 6. Grafana dashboard 导入
-请改为实际 IP 地址：
+1. 打开 `http://localhost:3000`，默认账号密码为 `admin` / `admin`。
+2. 进入 `Home -> Connections -> Data sources`，点击 `Add new data source`。
+3. 选择 `Prometheus`，数据源地址填写 `http://host.docker.internal:9090`。如果 Grafana
无法访问该地址，请改成宿主机实际 IP 或 Prometheus 容器可访问的地址。
+4. 点击 `Save & test`，确认数据源可用。
+5. 进入 `Home -> Dashboards -> New -> Import`。
+6. 上传 sample 目录中的 `grafana.json`，或复制文件内容到导入框。也可以使用 Grafana dashboard ID
`19294` 导入 `Dubbo Observability`。
+7. 选择刚才创建的 Prometheus 数据源，点击 `Import`。
-* 修改 `metrics/prometheus_grafana/prometheus_pull.yml`
-* 修改 Grafana 数据源 URL
+Consumer 启动后会持续调用 Provider，dashboard 中的 QPS、成功率、P99 延迟、Consumer/Provider
请求量和错误率会持续刷新。
-## 7. Kubernetes 部署
+## 7. 常见问题
-推荐使用：
+### 7.1 端口冲突
-> kube-prometheus
->
[https://github.com/prometheus-operator/kube-prometheus](https://github.com/prometheus-operator/kube-prometheus)
+如果 `3000`、`9090`、`9091` 或 `2181` 已被占用，请修改 `docker-compose.yml` 的端口映射；如果 `9099`
或 `9097` 被占用，请修改 `go-server/cmd/main.go` 或 `go-client/cmd/main.go` 中的
`metrics.WithPort(...)`，并同步修改 `prometheus_pull.yml` 的 target。
-### 7.1 添加 PodMonitor
+### 7.2 Prometheus Targets 显示 DOWN
-创建：
+先在宿主机执行：
-```
-dubboPodMonitor.yaml
+```bash
+curl http://localhost:9099/prometheus
+curl http://localhost:9097/prometheus
+curl http://localhost:9091/metrics
```
-内容：
+如果宿主机可以访问而 Prometheus 容器访问失败，通常是 Docker 网络地址问题。把 `host.docker.internal`
改为宿主机实际 IP，或在 Linux Docker Compose 中为 Prometheus 增加：
```yaml
-apiVersion: monitoring.coreos.com/v1
-kind: PodMonitor
-metadata:
- name: dubbo-pod-monitor
- namespace: monitoring
-spec:
- namespaceSelector:
- matchNames:
- - dubbo-system
- selector:
- matchLabels:
- app-type: dubbo
- podMetricsEndpoints:
- - port: metrics
- path: /prometheus
+extra_hosts:
+ - "host.docker.internal:host-gateway"
```
-### 7.2 可选：集群启用 RBAC 时补充权限配置
+### 7.3 Grafana 显示 No Data
-如果集群开启了 RBAC，建议为 Prometheus 增加对 `dubbo-system` 命名空间内 Pod 的读取权限：
+确认 Prometheus 数据源 `Save & test` 成功；在 `http://localhost:9090/targets` 确认 target
为 `UP`；在 Prometheus 查询 `dubbo_consumer_requests_succeed_total` 或
`dubbo_provider_requests_total`。如果查询为空，请确认 client 正在持续调用 server，并等待一个 scrape
interval。
-```yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
- namespace: dubbo-system
- name: pod-reader
-rules:
- - apiGroups: [""]
- resources: ["pods"]
- verbs: ["get", "list", "watch"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
- name: pod-reader-binding
- namespace: dubbo-system
-roleRef:
- apiGroup: rbac.authorization.k8s.io
- kind: Role
- name: pod-reader
-subjects:
- - kind: ServiceAccount
- name: prometheus-k8s
- namespace: monitoring
-```
+### 7.4 Pushgateway 中有旧 job
-如果你的 Prometheus 安装使用的 ServiceAccount 不是 `prometheus-k8s`，请按实际名称修改 `subjects`。
+Pushgateway 默认不会自动删除旧指标。优雅退出 sample 进程可以触发清理逻辑；生产环境建议部署 `pgw-cleaner` 或通过
Pushgateway DELETE API 清理过期 job。
-### 7.3 部署应用
-
-```bash
-kubectl apply -f Deployment.yaml
-```
+### 7.5 `/metrics` 和 `/prometheus` 应该访问哪个
-### 7.4 验证
-
-访问：
-
-```
-http://<prometheus-nodeport>/targets
-```
-
-确认 Pod 状态为：
-
-```
-UP
-```
+Dubbo-Go 默认路径是 `/metrics`，默认端口是 `9090`；当前 sample 显式配置了 `/prometheus`，server 使用
`9099`，client 使用 `9097`。请以你的应用中 `metrics.WithPort(...)` 与
`metrics.WithPath(...)` 的实际配置为准。
## 8. 生产建议
-| 场景 | 推荐模式 |
-| ------- | ----------------- |
-| 长期运行服务 | Pull |
-| 短生命周期任务 | Push |
-| K8s 环境 | Pull + PodMonitor |
-| 需要清理能力 | 配合 pgw-cleaner |
-
-## 9. 总结
-
-Dubbo-Go 当前支持：
-
-* Pull 模式（标准 Prometheus 模式）
-* Push 模式（Pushgateway）
-* Docker 快速部署
-* Kubernetes PodMonitor
-* Grafana 大盘
-* 僵尸指标自动治理
+| 场景 | 推荐方式 |
+| --- | --- |
+| 长期运行的 Provider/Consumer | Pull 模式 |
+| 短生命周期任务 | Pushgateway 模式 |
+| Kubernetes 环境 | Pull 模式 + PodMonitor |
+| 使用 Pushgateway | 配合清理器或 DELETE API |
-通过以上配置，可以实现完整的 Dubbo-Go 可观测性体系。
+如果部署在 Kubernetes 中，可使用 Prometheus Operator 的 PodMonitor 抓取应用暴露的指标端口，并将 `path`
设置为应用实际配置的 `/prometheus` 或 `/metrics`。
