dodjdnh opened a new issue, #3774: URL: https://github.com/apache/dubbo-go/issues/3774
## Motivation / Problem dubbo-go already supports total RPC requests, total failures, and some granular error metrics, but it cannot yet independently observe network errors (Network) and codec errors (Codec / Serialization). When these errors enter the existing RPC metrics collection pipeline, failures whose sources cannot be reliably identified fall into Unknown, making it difficult for operators to distinguish transport failures from serialization failures. This issue aims to complete the pipeline from source identification and classification to metric export for these two error categories, so that Prometheus / Grafana and Dubbo Admin can consume reliable error-classification metrics. The goal is not to copy Java's internal implementation, but to add verifiable capabilities while preserving compatibility with dubbo-go's existing metrics. ## Current State The following findings are based on a review of the `develop` source code in `apache/dubbo-go` on 2026-10-21. | Area | Current behavior | | --- | --- | | Implemented granular error metrics | Timeout, Limit, ServiceUnavailable, BusinessFailed, and Unknown, with corresponding windowed statistics | | Network / Codec classification | `ErrorTypeNetworkFailure` and `ErrorTypeCodec` are reserved, but `classifyError` does not return either category | | Network / Codec metric definitions | Neither the provider nor consumer metric set defines corresponding dedicated cumulative or windowed metrics | | Classification counters | The collector has no update branches for these two categories | | Protocol coverage | Classification currently relies primarily on Triple / gRPC status codes; Dubbo protocol classification remains a TODO | | Error source identification | `CodeInternal` may originate from transport, codec, or other internal errors; some codec errors also use `CodeInvalidArgument`, so Network and Codec cannot be distinguished directly by status code | Main source files: - [Error classifier](https://github.com/apache/dubbo-go/blob/develop/metrics/rpc/error_classifier.go) - [RPC metric definitions](https://github.com/apache/dubbo-go/blob/develop/metrics/rpc/metric_set.go) - [RPC metrics collection](https://github.com/apache/dubbo-go/blob/develop/metrics/rpc/collector.go) ## Desired Behavior 1. When the runtime provides reliable structured error information, network failures and codec failures can be classified as Network and Codec, respectively. 2. Errors whose sources cannot be reliably determined continue to fall into Unknown; do not classify all `CodeInternal` errors as Network or Codec, or infer their sources from error-message strings. 3. Providers and consumers export corresponding cumulative error counts and sliding-window error counts for the failure paths they can each observe, with explicit names, types, units, labels, and initialization behavior. 4. Preserve the original errors and their wrapping relationships; monitoring classification must not change RPC return statuses, retry policies, or business behavior. 5. Preserve the existing accounting semantics for total failures and supported error categories; adding categories must not increment total failures more than once. ## Scope ### Included - Inventory where Network / Codec errors originate in the Triple and Dubbo protocols, how they propagate, and their visibility to metrics collection. - Add reliable error source identification and classification for paths confirmed to be coverable, including wrapped errors; safely fall back to Unknown when identification is not possible. - Complete the corresponding metric definitions, cumulative counter updates, and windowed counter updates for providers and consumers. - Define the boundaries and precedence between Network and existing categories such as Timeout, ServiceUnavailable, and Unknown, avoiding duplicate accounting of the same failure in mutually exclusive categories. - Add classification unit tests, collector / exporter tests, and validation of real RPC failure paths. - Document the metric contract, protocol and caller/provider-side coverage, enablement, query examples, and compatibility impact. ### Not Included - Taking over all Metrics standardization work, renaming existing metrics, or making broad changes to existing labels. - Implementing all Dubbo protocol error classifications; only the Network / Codec paths required by this issue are included, with other categories coordinated separately. - Extending RPC error metrics into network interface, TCP, or operating-system network metrics. - Promising complete coverage of all low-level errors that cannot reach RPC metrics collection points; limitations and follow-up tasks for these paths should be explicitly recorded. - Simultaneously modifying Trace / Log consumers, Dubbo Admin product code, or Grafana Dashboards in this iteration; ## Acceptance Criteria - [ ] Provide an error-path coverage matrix for Triple / Dubbo × provider / consumer, identifying which paths can be classified, which cannot reach collection points, and the reasons for these limitations. - [ ] Establish reproducible error-identification rules for confirmed supported Network / Codec paths; do not substitute status codes or error text for reliable source information. - [ ] Define the contract for new metrics, including names, Counter / Gauge types, count units, labels, window semantics, and zero-value / absent-series behavior; windowed counts must not be used as cumulative Counters. - [ ] Network / Codec failures on paths observable by providers and consumers update the corresponding classified metrics, with consistent event-accounting semantics between cumulative counts and windowed statistics. - [ ] Cover direct errors, wrapped errors, unknown errors, and conflicts with categories such as timeout and service unavailable; retain Unknown when reliable classification is not possible. - [ ] Successful requests do not increment error counters; total failures are not incremented more than once; existing Timeout / Limit / ServiceUnavailable / BusinessFailed behavior passes regression tests. - [ ] New metrics retain bounded RPC identity labels and do not introduce high-cardinality labels such as error messages, full URLs, or Trace IDs. - [ ] Use real RPC fixtures to trigger controlled network and codec failures separately, recording the caller/provider side, protocol, original error, final classification, and exporter / Prometheus metric changes; manually constructed classifier inputs alone must not replace integration validation. - [ ] Record validation failures and uncovered cases separately; do not treat a Dashboard's zero-value fallback as proof that the metric capability has passed validation. - [ ] Relevant unit tests, static checks, and concurrency checks pass, and metric usage and compatibility documentation are provided. These are proposed acceptance criteria; implementation and validation have not yet been completed. ## Context / Design Notes This issue is an independent subtask within the observability plan. It does not replace or close the following overarching issues: - **Related observability plan: [ #3562 — Complete dubbo-go observability implementation and end-to-end integration](https://github.com/apache/dubbo-go/issues/3562)**, covering shared error semantics and Metrics completion. - **Related Metrics task: [ #3337 — Improve Metrics coverage and standardization](https://github.com/apache/dubbo-go/issues/3337)**, which explicitly proposes distinguishing Network / Codec errors and is currently assigned to @Saramanda9988. - **Overlap coordination: [ #3568](https://github.com/apache/dubbo-go/issues/3568)**, which includes Dubbo protocol error classification; duplicate changes to the classifier should be avoided. Existing work should be reused: [#3189](https://github.com/apache/dubbo-go/pull/3189) provides the foundation for the existing granular error metrics; [#3603](https://github.com/apache/dubbo-go/pull/3603) and [#3665](https://github.com/apache/dubbo-go/pull/3665) add classification semantics documentation and tests, but do not complete the dedicated Network / Codec metrics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
