dodjdnh opened a new pull request, #1587: URL: https://github.com/apache/dubbo-admin/pull/1587
## Summary Add Application / Instance log entries to Dubbo Admin, supporting logs stored in Loki or Elasticsearch through Grafana and manual Trace ID searches for related logs within the log page. Provide four log dashboards and Kubernetes automatic loading configuration, plus a field contract, Collector examples, and an ES index template reference for the external collection pipeline. This PR uses a connect-only model: Admin owns the entries and dashboard URLs, Grafana owns log queries, and deployers supply log storage, datasources, and collectors. ## Motivation Upstream already has Metric / Trace entries but lacks corresponding Application / Instance log pages. When troubleshooting application or instance problems, users need to locate the data in an external log system instead of reusing the application and instance identity selected in Admin. This PR adds the first-stage log-viewing capability and supports Loki and Elasticsearch through shared parameters and a field contract, avoiding two separate sets of provider-specific logic in Admin. ## Changes ### 1. Application and Instance Log Entries - The Application log page filters by application name; the Instance log page filters by application name and bare Pod IP, without a QoS port. - The backend returns a Grafana `fullURL`. It does not return log records, connect directly to Loki / Elasticsearch, or perform HTTP redirects. - Both log backends reuse the same Vue pages and backend APIs. Dashboard URLs are selected through `console.logDashboards`, without provider checks. - Preserve existing `var-application` behavior and add `var-applicationRaw`. Existing regex escaping for Application Log remains unchanged, while ES uses the actual raw application name for exact matching. ### 2. Manual Trace ID Search on the Log Page - A shared Vue search component validates complete 16- or 32-character hexadecimal Trace IDs. - The backend queries Jaeger for the earliest valid Span start time in the corresponding trace and converts microseconds to milliseconds. The frontend derives T±5 minutes and then requests a trace-focused log dashboard URL. - Manual search does not retain the current instance filter; it searches logs by Trace ID and time range. It reuses the Application log dashboard instead of adding a third Trace log dashboard. - Distinguish invalid input, a missing trace or invalid start time, missing or unavailable Jaeger configuration, and an unconfigured or unavailable log dashboard. Hide previous results on failure. - Request-version checks prevent late responses from the initial log dashboard request from overwriting manual search results. New protected GET APIs: | API | Behavior | | --- | --- | | `/api/v1/application/log-dashboard` | Return an application log dashboard URL | | `/api/v1/instance/log-dashboard` | Return an instance log dashboard URL | | `/api/v1/trace/start-time` | Return the Trace ID and start time | | `/api/v1/trace/log-dashboard` | Validate the Trace ID and millisecond time range, then return a trace-focused log dashboard URL | ### 3. Loki / ES Dashboards and Deployment Integration - Provide Application and Instance dashboards for each backend, displaying `Time | Level | Pod | Trace ID | Message`. - Loki uses the `app` index label and `pod_ip` structured metadata. Trace IDs may come from structured fields or the log body, with complete 16- or 32-character IDs extracted. - ES uses a raw-mapping contract: `@timestamp`, the original `Body`, and top-level LogRecord attributes. Application, instance, and trace queries use the corresponding `.keyword` fields, without scanning `Body` for Trace IDs or using an ECS adapter. - Table handling covers missing log levels, Pod-name fallback, and null Trace IDs. No Log → Trace Data Link is added. - ConfigMaps and mounts allow the existing Grafana Dashboard provider to load all four dashboards. Default Admin URLs still select Loki; switching to ES requires replacing all three `application`, `instance`, and `trace` URLs. - Deployers still configure external Grafana Loki / ES datasources; this PR does not register them automatically. ### 4. Shared Fixes and Integration Examples - Copy configured URLs when generating dashboard URLs so requests do not mutate shared parameters. - Stabilize container sizing and kiosk embedding for the shared Grafana iframe; these display changes also affect Metric / Trace pages. - Provide Loki / ES reference configurations for OTel Collector Contrib 0.112.0, explaining log-file reading, Pod identity enrichment, field parsing, permissions, and preservation of the original body. - Provide an optional ES composable index template with the required `.keyword` fields. It is a manual-installation example only and does not automatically create indices, alter existing indices, or overwrite production templates. ## Verification Local unit tests, builds, and static checks are complete. Runtime and browser acceptance were manually verified by the author, who confirmed on 2026-10-06 that all checks below passed. These manual acceptance results are not browser automation results. | Area | Check | Result | | --- | --- | --- | | Unit | Relevant Go tests in `service`, `handler`, and `config/console`; `config/observability` has no test files | Complete (PASS) | | Unit | Full Vue suite: 10 test files, 30 tests, using `--hookTimeout 60000` | Complete (PASS) | | Unit | Frontend log feature checkpoint: 18 relevant tests | Complete (PASS) | | Build | `go build ./...`; relevant Go tests and builds at all eight commit checkpoints | Complete (PASS) | | Build | Vite production build, without running the formatter that rewrites source files | Complete (build succeeded with warnings) | | Static | YAML / Dashboard JSON, four dashboards, five-column definitions, ConfigMap mounts, and datasource variables | Complete (PASS) | | Static | Collector raw mapping, ES template JSON, `.keyword` contract, and documentation links | Complete (PASS) | | Static | `git diff --check` | Complete (PASS) | | Regression | Log dashboards match the original log implementation; existing upstream dashboard definitions and instance Metric parameter generation are unchanged | Complete (code / static checks passed) | | Regression | No pending Instance Trace fixes, Trace Collector / Shop deployment changes, Trace→Metric, or Metric compatibility files included | Complete (diff checks passed) | | Runtime | Real Loki / ES log collection, application A+B aggregation, and bidirectional instance A/B isolation | Complete (manual verification passed) | | Runtime | Original Message, actual five-column display, missing-field fallbacks, and complete Trace IDs | Complete (manual verification passed) | | Runtime | Real Jaeger trace → manual search on the log page, T±5-minute range, and error states | Complete (manual verification passed) | | Runtime | ES index template installation, simulation, and actual index field checks | Complete (manual verification passed) | | Browser / Regression | Admin → Grafana log pages and existing Metric / Trace browser behavior | Complete (manual verification passed) | ## Review Focus 1. Whether shared log parameters satisfy both Loki regex semantics and ES exact matching; whether the raw application-name extension remains provider-neutral and instance identity matches the bare Pod IP known to Admin. 2. Whether the application / instance branch and Trace ID `OR` branch in ES queries switch correctly: the default `.*` for an inactive variable is not “match all” in a quoted keyword query and must not be interpreted as a Loki regex. Focus on query scope and branch-switching semantics. 3. Whether five-column transformations, complete Trace IDs, missing-field fallbacks, and the original Message match real Grafana data-frame behavior; whether the Loki structured-metadata and ES raw-mapping contracts are clear. 4. Whether Jaeger start-time lookup, T±5 minutes, backend time-range validation, search failures, and late responses are handled accurately. This is manual search on the log page, not automatic navigation from a Trace page. 5. The impact of shared URL / iframe fixes on existing Metric / Trace behavior, and the boundaries between provisioning and external datasources, Collectors, and ES templates. ## Compatibility / Impact - No Loki / ES-specific Admin branches are added, and existing Metric / Trace APIs or dashboard query definitions are not replaced. - Instance Log uses a bare IP. Existing upstream Instance Metric / Trace parameter generation remains unchanged, and pending Instance Trace fixes are not included. - `var-applicationRaw` is additive; existing log application parameters are unchanged. Manual trace search still uses wildcard application / instance parameters without a raw application parameter. - Ordinary log viewing does not depend on Jaeger; manual Trace ID search requires a reachable Jaeger query endpoint. - Requests no longer mutate shared URLs. Shared iframe display behavior changes, so not all Metric / Trace behavior can be described as completely unaffected. Browser regression in this verification environment was manually verified by the author and passed. - Grafana 12.1.0 is the current dashboard verification version. ES 8.18.0, Loki 3.4.2, and OTel Collector Contrib 0.112.0 are reference integration versions, not a claim that every version combination has been verified. - Admin iframe URLs must be reachable from the user's browser. Default `.svc` example addresses do not guarantee browser access outside the cluster. Affected areas (matching the repository PR template): - [x] Docs - [x] Installation - [x] User Experience - [ ] Dubboctl - [x] Console - [ ] Core Component ## Scope ### Included - Application / Instance log viewing and manual Trace ID search on the log page. - Loki / ES dashboards, automatic dashboard loading, Admin URL configuration, and related shared fixes. - The collection field contract and reference documentation for Collectors and the ES index template. ### Not Included - Service-, namespace-, or cluster-level log views. - Click-through navigation from a Trace page to Log, Log → Trace navigation, or Trace → Metric. - dubbo-go Metric compatibility or Dubbo SDK metric feature changes. - Automatic deployment of log storage, production log collectors, Grafana log datasources, or ES index templates. - ECS adaptation, full-text Body scanning for Trace IDs, or ES provider-specific backend logic. ## Related Issue Fixes #1586 ## Dependency Built on the Service Trace parameter and Kubernetes dashboard provisioning changes already merged into develop (#1569 and #1573). This PR does not depend on the pending Instance Trace bare-IP / Kubernetes Pod-identity PRs: instance lookup and test helpers for the log backend are self-contained. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
