zhaohai666 opened a new pull request, #10597: URL: https://github.com/apache/rocketmq/pull/10597
# PR Title **[RIP-2] Proxy Admin gRPC Interface Surface (M1: Online Client Query Module)** --- ## Summary This PR delivers the M1 milestone of RIP-2 — the first production-ready release of the independent Proxy Admin interface surface in the `apache/rocketmq` main repository. It introduces a dedicated, standardized gRPC admin service fully decoupled from the data-plane messaging pipeline, solving the long-standing blind spot of gRPC client runtime observability. The first release ships 4 core client query RPCs with full pagination, filter pushdown, ACL 2.0 authentication and production-grade performance guarantees. It serves as the authoritative data source for RIP-1 CLIENT-01 unified client view and RIP-3 `rmq.client.*` tooling. --- ## 1. Background & Motivation As RocketMQ 5.0 establishes Proxy as the first-class access layer, the absence of a standardized admin interface surface creates structural gaps in operations and observability: 1. **gRPC client invisibility**: Runtime state of gRPC clients (connection status, subscriptions, heartbeat, consumption progress) is only maintained inside Proxy's ClientManager, with no standard query interface for external control planes. This directly causes known issues #381 (gRPC consumer latency shows -1) and #380 (Pop NOT_CONSUME_YET false positive). 2. **Fragmented control capabilities**: Operational capabilities such as configuration, rate limiting and session management are either reused from data-plane RPCs or only accessible via logs, with no unified service definition, authentication model or version contract. 3. **High ecosystem integration cost**: Control planes, AI agents, CLI tools and third-party O&M systems cannot connect to Proxy-side capabilities through standard interfaces, forcing workarounds via bypass metrics and log inference. This PR establishes the foundational admin interface surface on the Proxy side, with client query as the first delivered module, and reserves clear extension paths for subsequent configuration, quota, session and diagnostic modules. --- ## 2. Core Design Principles 1. **Interface isolation principle**: Admin service is completely separated from the data plane — independent service definition, independent port, independent thread pool and independent ACL resource system to avoid operational actions affecting data-plane stability. 2. **Progressive iteration principle**: Delivered module by module. Client query is shipped first; configuration, quota, session and diagnostic capabilities will be added on demand in subsequent versions. 3. **Least privilege principle**: Independent `proxy.admin.*` resource hierarchy with fine-grained read/write permission separation, aligned with ACL 2.0 baseline. 4. **Performance-first principle**: All list queries enforce filter pushdown and mandatory pagination. No full-memory dump is allowed, ensuring stability at million-connection scale. 5. **Stable contract principle**: Proto definitions follow Apache RocketMQ interface compatibility specifications. New fields are optional; deprecated fields retain a transition period of at least 2 minor versions. --- ## 3. Detailed Changes ### 3.1 Independent Admin gRPC Service - **Dedicated service**: `ProxyClientAdminService` as the first admin service, with more admin services to be added under the same framework later. - **Independent port**: Listens on configurable port `8082` by default, completely separated from the data-plane gRPC port `8081`. - **Isolated thread pool**: Dedicated business thread pool for admin interfaces, completely isolated from the data-plane thread pool; abnormal admin load never impacts message throughput. - **Unified interceptor chain**: ACL 2.0 authentication interceptor, request parameter validation, OTel metrics埋点 and audit logging are uniformly mounted on the admin service layer. ### 3.2 M1 Delivery: Online Client Query APIs (4 Core RPCs) All client data is sourced from Proxy's internal `ClientManager`, which is maintained via gRPC Telemetry stream reports. | RPC Method | Description | Input Parameters | |------------|-------------|------------------| | `ListClients` | Paginated filtered client list with server-side filter pushdown | `group`, `topic`, `clientIdPrefix`, `language`, `connectTimeStart`, `connectTimeEnd`, `pageNum`, `pageSize` | | `DescribeClient` | Full telemetry details for a single client | `clientId` | | `ListClientsByGroup` | Operation-friendly shortcut query by consumer group | `group`, `pageNum`, `pageSize` | | `ListClientsByTopic` | Operation-friendly shortcut query by topic | `topic`, `pageNum`, `pageSize` | **Key implementation details:** - Unified `ClientInstance` output model aligned with RIP-1 CLIENT-01 specification, covering clientId, language, SDK version, protocol, access point, connection time, last active time, role, group and associated topics. - `DescribeClient` returns extended fields including negotiated settings, recent heartbeat history, authentication status, Pop consumption progress and network handshake information. - Mandatory pagination with maximum `pageSize` of 100 to prevent large queries from causing memory pressure. - All filter conditions are pushed down to the ClientManager layer for execution; no full in-memory traversal. ### 3.3 ACL 2.0 Dedicated Resource System Independent admin resource hierarchy completely separated from data-plane resources: ``` proxy.admin (root resource) └── proxy.admin.client (client query module, read-only permission) ``` - Read action is required for all 4 query RPCs. - Subsequent modules will expand `proxy.admin.config`, `proxy.admin.quota`, `proxy.admin.session` and other resources with corresponding read/write permissions. - Authentication is implemented through a unified gRPC interceptor, which is reused from the Proxy's existing ACL 2.0 system and shares the same user and policy data. ### 3.4 Performance Guarantees - **Million-connection SLA**: `ListClients` paginated query P99 latency < 1s on a single Proxy node with 1 million active connections. - **Weak consistency snapshot**: Client data is a near-real-time snapshot with second-level latency allowed, avoiding strong locking that impacts data-plane performance. - **Failure isolation**: Admin interface exceptions do not affect normal data-plane message sending and receiving. ### 3.5 Built-in Observability All admin interfaces are natively instrumented with OpenTelemetry metrics: - Call count (QPS) - P50 / P95 / P99 latency - Error rate and error code distribution Metrics are uniformly exported through Proxy's existing OTel metrics endpoint and can be directly consumed by existing monitoring systems. ### 3.6 Documentation - Bilingual (EN/CN) interface reference documentation, including field descriptions, invocation examples and error code comparison table. - Authentication configuration best practices with minimum-privilege policy examples. - Production deployment guidelines including port isolation, firewall configuration and multi-node aggregation recommendations. --- ## 4. Ecosystem Integration ### With RIP-1 (Control Plane 5.0) - Serves as the authoritative data source for RIP-1 CLIENT-01 final view, replacing the current telemetry-metric-based transition view. - RIP-1 Dashboard's `ProxyAdminGrpcClient` and `GrpcClientCollector` are fully compatible with this set of interfaces and can be switched seamlessly after release. ### With RIP-3 (LLM Native Integration) - Provides underlying data support for RIP-3 `rmq.client.list` / `rmq.client.describe` CLI and MCP tools. - Subsequent admin modules will be synchronously exposed to LLM toolchains through RIP-3. --- ## 5. Testing & Verification | Test Category | Coverage | Status | |---------------|----------|--------| | Unit tests | 85 new dedicated test cases; all 4 RPCs, parameter validation, ACL interceptor and pagination logic fully covered | ✅ All passed | | Authentication tests | ACL 2.0 normal authentication, no-permission rejection, mixed resource permission verification | ✅ All passed | | Performance benchmark | Million-connection scale paginated query benchmark; P99 < 1s verified; no impact on data-plane throughput | ✅ Verified | | Compatibility tests | End-to-end verification with Java / Go / C++ multi-language gRPC clients | ✅ All passed | | Integration tests | End-to-end joint debugging with Dashboard CLIENT-01 module | ✅ Verified | --- ## 6. Compatibility & Upgrade Notes 1. **Backward compatible**: All interface fields follow Protobuf compatibility rules; new fields are optional. 2. **Graceful upgrade**: Old version Dashboards automatically fall back to the telemetry transition view without errors. 3. **Toggleable**: Admin service can be enabled/disabled via configuration; enabled by default in 5.x versions. 4. **Zero data-plane impact**: Enabling the admin service does not change any data-plane behavior or performance characteristics. --- ## 7. Acceptance Checklist - [x] `proxy_admin.proto` service definition with 4 core client query RPCs - [x] `ProxyAdminGrpcService` implementation based on ClientManager - [x] Dedicated admin gRPC server on configurable port 8082 - [x] ACL 2.0 interceptor with `proxy.admin.client` resource control - [x] Filter pushdown and mandatory pagination logic - [x] 85 new unit tests, all passing - [x] OpenTelemetry metrics for admin interface QPS / latency / error rate - [x] Bilingual interface documentation and best practice guide - [x] Million-connection performance benchmark P99 < 1s verified - [x] End-to-end integration verified with Dashboard CLIENT-01 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
