dodjdnh commented on issue #1524:
URL: https://github.com/apache/dubbo-admin/issues/1524#issuecomment-5420184069
Hi @robocanic @chickenlj @nzomkxia，
针对本 Issue 里基于 Grafana 做 metric / trace / log 联动、补齐 log 面板，以及兼容 dubbo-go
metric label 这几项要求，我在 #1537 写过一版较完整的提案。落地时我准备按下面五层来做增量扩展，而不是另起一套可观测平台。
```text
┌─────────────────────────────────────────────────────────────┐
│ UI Layer (Vue3) │
│ GrafanaPage / LogPanel / CorrelationContext / Metric Panel │
└────────────────────────────┬────────────────────────────────┘
│ /api/v1
┌────────────────────────────▼────────────────────────────────┐
│ API Layer (Gin) │
│ /logs/query /logs/health /log-dashboard /metric-profiles │
└────────────────────────────┬────────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────────┐
│ Application Layer │
│ LogService / CorrelationService / MetricCompatibilityService│
└────────────────────────────┬────────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────────┐
│ Adapter Layer │
│ Loki Adapter / ES Adapter / Metric Label Mapper │
└────────────────────────────┬────────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────────┐
│ External Observability Stack │
│ Grafana / Prometheus / Loki / ElasticSearch │
└─────────────────────────────────────────────────────────────┘
```
## 1. UI Layer（Vue3）
- 继续复用 `GrafanaPage.vue` 承载 metric / trace / log 的 Grafana iframe
- 新增 `LogPanel.vue`：过滤条件、时间窗、日志列表、跳转入口
- 引入共享 `CorrelationContext`（可放 composable / provide-inject），在资源页各 tab 间传递：
- `mesh` / `application` / `service` / `instance` / `method`
- `traceId` / `spanId?`
- `start` / `end`
- 应用、实例、服务详情增加 Logging tab：
- `views/resources/applications/tabs/logging.vue`
- `views/resources/instances/tabs/logging.vue`
- `views/resources/services/tabs/logging.vue`
- Logging tab 内两块能力：
- Grafana log dashboard（主可视化）
- Admin 代理查询结果列表（便于按 `traceId` 过滤和一键跳转）
- Metric 兼容不新开独立产品页，优先在现有监控面板上消化 label 差异；必要时加轻量 profile 切换
- 前端 API 封装：`ui-vue3/src/api/service/log.ts`（`queryLogs` / `getLogDashboard`
/ `getMetricProfiles`）
- 前端不直连 Loki / ES / Prometheus，不持有下游凭据
## 2. API Layer（Gin）
在现有 `/api/v1` 上扩展，风格对齐已有 `metric-dashboard` / `trace-dashboard` / `promQL`：
- `GET /api/v1/observability/logs/query`
- 入参对齐
`LogQuery`：provider、mesh、application、service、instance、method、traceId、keywords、start、end、limit、cursor
- 出参走现有 `CommonResp`，data 为统一 `LogResult`
- `GET /api/v1/observability/logs/health`
- 检查当前日志 provider 配置与连通性
- 维度 dashboard 路由（复用 `GetGrafanaDashboard` 的维度分发，扩展 `DashboardType=log`）：
- `GET /api/v1/application/log-dashboard`
- `GET /api/v1/instance/log-dashboard`
- `GET /api/v1/service/log-dashboard`
- `GET /api/v1/observability/metrics/profiles`
- 返回 dubbo-java / dubbo-go 的 label profile 与映射说明，供前端或 dashboard 变量选择
- 错误处理对齐现有 console：超时、下游 4xx/5xx、配置缺失分别转换；token / password / authorization
等字段返回前脱敏
## 3. Application Layer
把业务编排放在 console service，不让 handler 直接感知 LogQL / ES DSL：
- `LogService`
- 读取 observability 配置，选择 Loki / ES / mock provider
- 做查询参数校验、默认时间窗、limit 限流
- 调用 adapter，并把结果归一成 `LogResult`
- `CorrelationService`
- 维护关联上下文到 dashboard 变量、日志过滤条件的映射
- 主路径：`traceId + 时间窗 + 资源标识` 强关联
- 兜底：无 `traceId` 时按 application/service/instance + 时间窗弱关联
- `MetricCompatibilityService`
- 基于现有 instance Prometheus exposition 解析能力做 profile 选择与 label 归一
- 尽量让 Go / Java 走同一套 Grafana metric dashboard，而不是维护两套面板
对应落点：
- `pkg/console/service/log.go`
- `pkg/console/service/correlation.go`（或先内聚在 log/observability service
中，视体积再拆）
- `pkg/console/service/metric_compatibility.go`
- handler 只做绑定参数与 `CommonResp` 包装：`pkg/console/handler/log.go` 等
## 4. Adapter Layer
独立出适配层，便于单测和后续扩展：
```text
pkg/observability/log/
provider.go # LogProvider / LogQuery / LogResult / LogEntry
loki.go # LogQL + Loki query_range
elasticsearch.go # Query DSL + _search
normalize.go # 字段归一化
mock.go # fixture
pkg/observability/metric/
profile.go # dubbojava / dubbogo profile
mapper.go # label mapping
```
- `LokiAdapter`：按 label selector 与时间窗构造 LogQL，解析 `query_range`
- `ElasticsearchAdapter`：按 bool filter/must 构造 Query DSL，解析 `_search`
- `normalize`：把两边差异收敛为统一 `LogEntry`（timestamp / level / message /
application / service / instance / method / traceId / spanId / raw fields）
- `Metric Label
Mapper`：归一常见差异（application/app、interface/service、method、instance/ip/pod 等），再用
fixture 回归，避免改坏现有 Java 面板
配置侧同步扩展：
- `pkg/config/observability`：新增 `logs.loki` /
`logs.elasticsearch`（地址、超时、鉴权、默认索引或 label、启用开关）
- `pkg/config/console`：在 `metricDashboards` / `traceDashboards` 旁增加
`logDashboards`（application/instance/service）
## 5. External Observability Stack
外部系统保持职责不变，Admin 只做接入与关联：
- Grafana：metric / trace / log 面板可视化主入口
- Prometheus：现有 metric 查询与 instance exposition
- Loki / ElasticSearch：日志存储与检索后端
- Admin 不替换、不内嵌这些系统；也不在第一版做独立日志存储、告警或异常检测平台
## 提交计划
由于 OSPP 官方要求，以及这组改动涉及面较大，方向和社区确认前不会提前大规模改主干。若整体方向没有问题，按依赖拆成 4 个 PR：
1. **PR-1** `feat: add log providers and query API`配置扩展 + Adapter 层（Loki /
ES / mock `LogProvider`）+ `/logs/query`、`/logs/health` + 单测
2. **PR-2** `feat: add log dashboards for
app/instance/service``log-dashboard` 路由 + Logging tab + Grafana log 面板接入（应用 /
实例 / 服务）
3. **PR-3** `feat: correlate metric, trace and log
panels``CorrelationContext` 贯通 + Trace → Log → Metric 跳转
4. **PR-4** `feat: support dubbogo metric label profiles`
dubbo-go / dubbo-java metric profile 与 label mapping + fixture + 文档
已有交互原型用于验证跳转和面板形态；如需要可再补链接或录屏。更完整方案见 #1537。
补充两点准备情况：
- 已充分阅读社区开发规范与相关贡献文档，后续会按规范组织提交、测试与文档
- 10 月中旬才开学上课，前期可投入较完整的连续开发时间，保证按上述 PR 节奏推进
## 沟通现状
近期提案审阅较集中，也看到 #1537 里老师回复需要时间，完全理解。此前 19 日和 24
日通过邮件同步过方案和进展，也尝试加入社区讨论群，但目前还没有进一步反馈，所以把实现思路放到本 Issue
公开同步。若整体方向没有大问题，下一步就按上述拆分准备第一个 draft PR。
Related: #1537
感谢审阅。
---
## English version
Hi @robocanic @chickenlj @nzomkxia,
For the requirements in this issue — correlating metrics, traces, and logs
on top of Grafana, adding log panels in Admin, and supporting dubbo-go metric
labels — I already posted a fuller proposal in #1537. For implementation, I
plan to extend the current stack in five layers instead of introducing a
separate observability platform.
```text
┌─────────────────────────────────────────────────────────────┐
│ UI Layer (Vue3) │
│ GrafanaPage / LogPanel / CorrelationContext / Metric Panel │
└────────────────────────────┬────────────────────────────────┘
│ /api/v1
┌────────────────────────────▼────────────────────────────────┐
│ API Layer (Gin) │
│ /logs/query /logs/health /log-dashboard /metric-profiles │
└────────────────────────────┬────────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────────┐
│ Application Layer │
│ LogService / CorrelationService / MetricCompatibilityService│
└────────────────────────────┬────────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────────┐
│ Adapter Layer │
│ Loki Adapter / ES Adapter / Metric Label Mapper │
└────────────────────────────┬────────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────────┐
│ External Observability Stack │
│ Grafana / Prometheus / Loki / ElasticSearch │
└─────────────────────────────────────────────────────────────┘
```
### 1. UI Layer (Vue3)
- Keep using `GrafanaPage.vue` for metric / trace / log Grafana iframes
- Add `LogPanel.vue` for filters, time range, log list, and jump actions
- Introduce a shared `CorrelationContext` (composable / provide-inject)
across resource tabs:
- `mesh` / `application` / `service` / `instance` / `method`
- `traceId` / `spanId?`
- `start` / `end`
- Add Logging tabs on application / instance / service details:
- `views/resources/applications/tabs/logging.vue`
- `views/resources/instances/tabs/logging.vue`
- `views/resources/services/tabs/logging.vue`
- Each Logging tab covers:
- Grafana log dashboard as the main visualization
- Admin-proxied query results for `traceId` filtering and one-click jumps
- Metric compatibility should stay on the existing monitor surface; add a
light profile switch only if needed
- Frontend API wrapper: `ui-vue3/src/api/service/log.ts` (`queryLogs` /
`getLogDashboard` / `getMetricProfiles`)
- The frontend will not talk to Loki / ES / Prometheus directly, and will
not hold downstream credentials
### 2. API Layer (Gin)
Extend the existing `/api/v1` surface in the same style as
`metric-dashboard` / `trace-dashboard` / `promQL`:
- `GET /api/v1/observability/logs/query`
- Input aligned with `LogQuery`: provider, mesh, application, service,
instance, method, traceId, keywords, start, end, limit, cursor
- Output uses the existing `CommonResp`, with `data` as a unified
`LogResult`
- `GET /api/v1/observability/logs/health`
- Check current log provider configuration and connectivity
- Dimension dashboard routes (reuse `GetGrafanaDashboard` dispatch, extend
`DashboardType=log`):
- `GET /api/v1/application/log-dashboard`
- `GET /api/v1/instance/log-dashboard`
- `GET /api/v1/service/log-dashboard`
- `GET /api/v1/observability/metrics/profiles`
- Return dubbo-java / dubbo-go label profiles and mapping notes for
frontend or dashboard variable selection
- Error handling stays consistent with the current console: timeouts,
downstream 4xx/5xx, and missing config are mapped cleanly; token / password /
authorization fields are redacted before response
### 3. Application Layer
Keep orchestration in console services so handlers do not construct LogQL /
ES DSL directly:
- `LogService`
- Read observability config and select Loki / ES / mock provider
- Validate query params, default time window, and limit
- Call adapters and normalize results into `LogResult`
- `CorrelationService`
- Map correlation context to dashboard variables and log filters
- Primary path: strong correlation by `traceId + time window + resource
identity`
- Fallback: weak correlation by application/service/instance + time window
when `traceId` is absent
- `MetricCompatibilityService`
- Build on the existing instance Prometheus exposition parsing for profile
selection and label normalization
- Prefer one shared Grafana metric dashboard for Go / Java instead of
maintaining two panels
Likely code locations:
- `pkg/console/service/log.go`
- `pkg/console/service/correlation.go` (or keep it inside log/observability
service first and split later if needed)
- `pkg/console/service/metric_compatibility.go`
- Handlers only bind params and wrap `CommonResp`, e.g.
`pkg/console/handler/log.go`
### 4. Adapter Layer
Keep adapters isolated for unit testing and future providers:
```text
pkg/observability/log/
provider.go # LogProvider / LogQuery / LogResult / LogEntry
loki.go # LogQL + Loki query_range
elasticsearch.go # Query DSL + _search
normalize.go # field normalization
mock.go # fixtures
pkg/observability/metric/
profile.go # dubbojava / dubbogo profiles
mapper.go # label mapping
```
- `LokiAdapter`: build LogQL from label selectors and time range, parse
`query_range`
- `ElasticsearchAdapter`: build Query DSL with bool filter/must, parse
`_search`
- `normalize`: converge both sides into a unified `LogEntry` (timestamp /
level / message / application / service / instance / method / traceId / spanId
/ raw fields)
- `Metric Label Mapper`: normalize common differences (application/app,
interface/service, method, instance/ip/pod, etc.), then regress with fixtures
so existing Java panels stay intact
Config extensions:
- `pkg/config/observability`: add `logs.loki` / `logs.elasticsearch`
(endpoint, timeout, auth, default index/labels, enable switch)
- `pkg/config/console`: add `logDashboards` next to `metricDashboards` /
`traceDashboards` (application/instance/service)
### 5. External Observability Stack
External systems keep their current roles; Admin only integrates and
correlates:
- Grafana: main visualization entry for metric / trace / log panels
- Prometheus: existing metric queries and instance exposition
- Loki / ElasticSearch: log storage and retrieval backends
- Admin will not replace or embed these systems, and the first version will
not build a separate log storage, alerting, or anomaly-detection platform
### PR plan
Because of OSPP process constraints and the size of this change, I will not
start large trunk commits before the direction is confirmed with the community.
If the overall approach looks fine, I propose splitting the work into 4 PRs:
1. **PR-1** `feat: add log providers and query API`Config extension +
adapter layer (Loki / ES / mock `LogProvider`) + `/logs/query`, `/logs/health`
+ unit tests
2. **PR-2** `feat: add log dashboards for
app/instance/service``log-dashboard` routes + Logging tabs + Grafana log
dashboard integration (application / instance / service)
3. **PR-3** `feat: correlate metric, trace and log panels`Wire
`CorrelationContext` + Trace → Log → Metric navigation
4. **PR-4** `feat: support dubbogo metric label profiles`
dubbo-go / dubbo-java metric profiles and label mapping + fixtures + docs
I already have an interactive prototype for validating the jump flow and
panel shape; I can share a link or recording if useful. The fuller write-up is
in #1537.
Two additional readiness notes:
- I have gone through the community contribution guidelines and related
docs, and will follow them for commits, tests, and documentation
- Classes do not start until mid-October, so I can commit continuous
development time early on and keep the PR cadence above
### Communication note
I understand proposal review is busy right now, and I also saw the note on
#1537 that more time is needed. That is completely fine. I previously followed
up by email on the 19th and 24th, and also tried joining the community
discussion group, but have not received further feedback so far. I am posting
the implementation plan here so we can align in public. If the direction looks
good overall, I will prepare the first draft PR next.
Related: #1537
Thanks for reviewing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
