dodjdnh opened a new issue, #1583:
URL: https://github.com/apache/dubbo-admin/issues/1583
# [Bug] Instance Trace cannot reliably identify the selected Kubernetes Pod
## Motivation / Problem
Dubbo Admin, the Grafana Instance Trace dashboard, and the trace data use
inconsistent meanings for an “instance”.
Dubbo Admin passes `IP:QoSPort`, while the Instance Trace dashboard filters
spans with `net.peer.name=${instance}`. `net.peer.name` describes the remote
peer rather than the Kubernetes Pod that owns the current span. With multiple
replicas of the same application, selecting an instance therefore cannot
reliably isolate that Pod’s traces.
## Current State
| Area | Current behavior |
| --- | --- |
| Instance Metric | Uses `IP:QoSPort` as the instance identity |
| Instance Trace URL | Also passes `IP:QoSPort` |
| Grafana Instance Trace | Filters with `service=${application}` and
`net.peer.name=${instance}` |
| Trace data | Does not provide a reliable own-Pod identity for this query |
## Desired Behavior
Instance Metric should keep its existing identity:
- `var-instance=IP:QoSPort`
Instance Trace should use the selected Kubernetes Pod identity:
- `var-application=<AppName>`
- `var-instance=<bare Pod IP>`
The Instance Trace dashboard should query:
- `service=${application}`
- `k8s.pod.ip=${instance}`
Traces should also carry Pod metadata that can be checked against the
Kubernetes API:
- `k8s.pod.ip`
- `k8s.pod.uid`
- `k8s.pod.name`
- `k8s.namespace.name`
The bare Pod IP is used as the query key because the selected Admin instance
already exposes its Pod IP directly. Pod UID, name, and namespace remain
supporting identity metadata and runtime verification evidence.
## Scope
### Included
- Align the Instance Trace identity used by Dubbo Admin and Grafana.
- Route sample traces through an OpenTelemetry Collector before Jaeger.
- Enrich traces with own-Pod Kubernetes metadata using `k8sattributes`.
- Add the namespace-scoped, read-only Pod access required by the Collector.
- Preserve existing Instance Metric and Service Trace semantics.
### Not Included
- Rebuilding, publishing, or updating the currently referenced `shop-user`
image.
- Log observability or cross-signal navigation such as Trace → Log, Log →
Trace, or Trace → Metric.
- Prometheus behavior or metric pipeline changes.
- Query semantic changes to the other five existing dashboards.
- The disposable Kind test harness or local E2E image artifacts.
## Acceptance Criteria
- Instance Trace receives the selected instance’s `AppName` and bare Pod IP.
- Instance Metric continues to use `IP:QoSPort`.
- The Instance Trace dashboard filters by `service=${application}` and
`k8s.pod.ip=${instance}`.
- Traces contain Pod IP, UID, name, and namespace values matching Kubernetes
truth.
- For two replicas of the same application, querying Pod A by IP returns Pod
A traces and excludes Pod B; the reverse query behaves the same way.
- Existing Service Trace behavior remains unchanged.
- The other five dashboard semantics remain unchanged.
- Relevant Go tests, manifest/dashboard validation, and static regression
checks pass.
## Context / Design Notes
The Admin variable, Grafana filter, and span resource metadata form one
instance-identity contract. Changing only one layer would leave Instance Trace
semantics inconsistent, so they must be verified together.
### Historical introduction
This issue was formed by two historical observability PRs:
- PR #1387 (`2eb4409`) introduced the Grafana Instance Trace dashboard with
`net.peer.name=${instance}` and configured sample traces to go directly to
Jaeger.
- PR #1397 (`d4d7a8f`) set the Admin-side Instance Dashboard variable to
`IP:QoSPort`.
Together, these two layers made the Admin instance value unreliable as a
Kubernetes Pod identity in multi-replica deployments. PR #1302 (`37db89e`) only
migrated the existing code into this repository and is not the main functional
source of this issue. PR #1569 and PR #1573 are prerequisite Service Trace and
observability provisioning changes, not the introducing PRs for this issue.
The prerequisite behavior has since been merged into `upstream/develop`
through PR #1569 and PR #1573.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
