zhaohai666 opened a new issue, #10599:
URL: https://github.com/apache/rocketmq/issues/10599
### Before Creating the Enhancement Request
- [x] I have confirmed that this should be classified as an enhancement
rather than a bug/feature.
### Summary
# Issue Title
**[RIP-2] Proxy Admin gRPC Interface Surface (M1: Online Client Query
Module)**
---
## Issue Description
This tracking issue covers the delivery of RIP-2 (Proxy Admin Interface
Surface) in the `apache/rocketmq` main repository.
The initiative establishes an independent, standardized admin gRPC interface
layer on the Proxy side, fully decoupled from the data-plane messaging
pipeline. The first milestone (M1) delivers online client query capabilities as
the foundational module, resolving the long-standing blind spot of gRPC client
runtime observability. This interface surface will serve as the unified carrier
for all subsequent Proxy-side admin tools, including configuration management,
quota control, session management and diagnostic capabilities, which will be
rolled out progressively based on community demand.
---
## 1. Background & Motivation
As RocketMQ 5.0 elevates Proxy to a first-class access layer with protocol
parsing, authentication, rate limiting, routing and telemetry reporting
capabilities, the lack of a standardized admin interface surface creates
structural operational pain points:
1. **Invisible runtime state**: gRPC client connection status, subscription
configuration, heartbeat liveness and consumption progress are only maintained
internally within Proxy, with no standard query entry for control planes. This
directly causes known issues #381 (gRPC consumer latency displays -1) and #380
(Pop client status false positive).
2. **Fragmented and unstandardized admin capabilities**: Operations such as
configuration query, rate limit adjustment and session management either reuse
data-plane RPC interfaces or rely on log troubleshooting, lacking unified
service definitions, authentication systems and version contracts.
3. **High ecosystem integration cost**: Upper-layer ecosystems including
control panels, AI agents and CLI tools cannot connect to Proxy-side O&M
capabilities through standard interfaces, and can only infer runtime status via
bypass metrics and logs, with insufficient accuracy and timeliness.
---
## 2. Core Goals
1. Establish an independent Proxy Admin gRPC interface surface, fully
decoupled from data-plane interfaces in terms of port, authentication and
service definition.
2. Deliver online client query capabilities in the first release, covering
gRPC client list, details and group/topic dimension filtering, to support the
RIP-1 CLIENT-01 final view.
3. Unify the ACL 2.0 authentication model with the independent
`proxy.admin.*` resource hierarchy for fine-grained permission control.
4. Establish standardized interface contracts and version evolution rules to
ensure backward compatibility, serving as the unified underlying entry for
upper-layer control panels, AI tools and CLIs.
5. Guarantee high performance: paginated query P99 latency < 1s at
million-connection scale, without blocking normal data-plane traffic.
---
## 3. Non-Goals
- Does not modify Broker or NameServer core logic; only targets Proxy-side
admin capability construction.
- The initial release does not enforce cluster-level aggregated views; only
single-Proxy local views are provided, and aggregation can be implemented by
upper-layer control platforms.
- Does not replace existing `mqadmin` O&M commands; serves as a
supplementary gRPC standardized admin channel.
- Planned modules (configuration, quota, session, diagnostics) are not
mandatory for the initial release; they will be iterated version by version
based on community needs.
---
## 4. Core Design Principles
1. **Interface isolation**: Admin interfaces are separated from data-plane
message interfaces in terms of service, port and authentication to prevent
operational actions from affecting data-plane stability.
2. **Progressive iteration**: Delivered module by module. Client query is
delivered first, and capabilities such as configuration, quota and diagnosis
are expanded on demand.
3. **Least privilege**: Independent `proxy.admin.*` resource system with
separated read/write permissions, aligned with the ACL 2.0 authentication
baseline.
4. **Performance first**: Query interfaces support filter pushdown,
pagination and sampling. Full memory dump of data is prohibited to ensure
stability at million-connection scale.
5. **Stable contract**: Proto definitions follow Apache RocketMQ interface
compatibility specifications; new fields are backward compatible, and
deprecated fields reserve a transition period.
6. **Built-in observability**: Admin interfaces natively expose
OpenTelemetry metrics including call count, latency and error rate for
monitoring the health of the admin channel.
---
## 5. Scope of Work
### M1: Initial Delivery — Online Client Query
The first delivered module is the core unit of the Proxy Admin interface
surface, which directly solves the industry pain point of invisible gRPC
clients and supports the final view of RIP-1 CLIENT-01.
#### Core RPC Interfaces (4 in total)
All client data is sourced from the internal `ClientManager` module of
Proxy, maintained based on information reported by the gRPC Telemetry stream.
| RPC Method | Description | Input Parameters |
|------------|-------------|------------------|
| `ListClients` | Paginated filtered client list with server-side filter
pushdown | `group`, `topic`, `clientIdPrefix`, `language`, `connectTimeStart`,
`connectTimeEnd`, `pageNum`, `pageSize` |
| `DescribeClient` | Full telemetry details for a single client instance |
`clientId` |
| `ListClientsByGroup` | O&M-friendly shortcut query by consumer group |
`group`, `pageNum`, `pageSize` |
| `ListClientsByTopic` | O&M-friendly shortcut query by topic | `topic`,
`pageNum`, `pageSize` |
#### ACL 2.0 Resource Model
Independent admin resource hierarchy completely separated from data-plane
resources:
```
proxy.admin (root resource)
└── proxy.admin.client (client query module, read-only permission)
```
#### Performance & Implementation Requirements
- All filter conditions are pushed down to the ClientManager layer; full
in-memory traversal after query is prohibited.
- Mandatory pagination with maximum `pageSize` of 100 to prevent large
queries from causing memory pressure.
- Independent thread pool for admin interfaces, completely isolated from the
data-plane thread pool.
- Weak consistency snapshot: client list is a near-real-time snapshot,
allowing second-level delay to avoid locking that affects data-plane
performance.
- Built-in OpenTelemetry metrics for all admin interfaces: QPS, P95/P99
latency, error rate.
### M2+: Planned Iteration Modules
The following modules are the long-term evolution direction of the Proxy
Admin interface surface. They are not mandatory for the initial release and
will be rolled out version by version based on community demand and priority:
1. **Runtime configuration query & hot update**: `DescribeProxyConfig` /
`UpdateProxyConfig` to query and modify runtime configurations without restart.
2. **Rate limit quota visualization & adjustment**: Query current rate
limiting policies and trigger counts by namespace/topic/group dimensions;
support controlled quota adjustment.
3. **Cross-node session & connection control**: `KickClient` /
`DisconnectChannel` to proactively remove abnormal or lost clients, covering
session synchronization semantics in multi-Proxy deployment.
4. **Advanced semantic O&M diagnosis**: Expose Pop lock holding view,
ACK/NACK count, batch pull window status and NOT_CONSUME_YET trigger reasons
for troubleshooting.
5. **Routing & load balancing observation**: View Proxy → Broker connection
topology, current load distribution, fault removal status and geo-affinity
routing effectiveness.
---
## 6. Acceptance Criteria
### M1 Initial Release Acceptance
- [ ] Proxy Admin overall architecture and proto definitions pass community
review, with clear interface positioning, naming conventions, permission system
and evolution strategies.
- [ ] All 4 core client query RPCs are implemented, with complete functions,
accurate filtering and normal pagination.
- [ ] Paginated query P99 latency < 1s on a single Proxy with million
connections, no memory overflow risk.
- [ ] Fully integrated with ACL 2.0 authentication, with independent
`proxy.admin.client` resource type and minimum permission policy documented.
- [ ] OpenTelemetry metrics for admin interfaces are enabled by default,
meeting observability requirements.
- [ ] End-to-end joint debugging with RIP-1 Dashboard CLIENT-01 passes, and
the gRPC client view is displayed normally.
- [ ] Complete bilingual interface documentation, authentication
configuration examples and best practice manuals are provided.
### Overall Architecture Acceptance
- [ ] Independent admin service framework is established, supporting
subsequent module expansion without structural changes.
- [ ] Admin interfaces are completely isolated from the data plane in terms
of port, thread pool and authentication.
- [ ] Interface contracts follow Apache RocketMQ compatibility
specifications and support smooth version iteration.
### Quality & Documentation
- [ ] Unit test coverage of core modules ≥ 85%.
- [ ] Integration tests cover all interfaces, authentication scenarios and
exception handling.
- [ ] Performance benchmark report for million-connection scale is available.
- [ ] Bilingual (EN/CN) official documents are synchronized with the code
release.
---
## 7. Related RIPs & Dependencies
- **RIP-1 (Control Plane 5.0)**: RIP-1 CLIENT-01 final view depends on the
M1 module of this RIP. Before the M1 module is released, CLIENT-01 uses
telemetry metrics to build a transition view, and switches to the final view
seamlessly after release. The two RIPs can be developed in parallel.
- **RIP-3 (LLM Native Integration)**: RIP-3's `rmq.client.*` series of tools
directly benefit from the M1 module of this RIP. Subsequent planned modules
will also be synchronously exposed to LLM toolchains through RIP-3.
- **Dependency rules**:
- RIP-2 itself does not depend on other RIPs and can be developed and
released independently.
- RIP-1 has a soft dependency on RIP-2 M1; the transition view is
available without RIP-2.
- RIP-3 indirectly depends on RIP-2 through RIP-1's abstraction layer.
---
## 8. Contribution Notes
- All code is delivered in the `proxy` module of the `apache/rocketmq` main
repository.
- All submissions must follow Apache RocketMQ community coding standards,
proto specification and contribution guidelines.
- Sub-tasks can be claimed by commenting on this issue. Please raise design
questions and technical discussions in this thread or the corresponding dev
mailing list.
- Planned modules can be proposed and promoted by interested contributors,
and will be included in the RIP iteration roadmap after community review.
### Motivation
# Issue Title
**[RIP-2] Proxy Admin gRPC Interface Surface (M1: Online Client Query
Module)**
---
## Issue Description
This tracking issue covers the delivery of RIP-2 (Proxy Admin Interface
Surface) in the `apache/rocketmq` main repository.
The initiative establishes an independent, standardized admin gRPC interface
layer on the Proxy side, fully decoupled from the data-plane messaging
pipeline. The first milestone (M1) delivers online client query capabilities as
the foundational module, resolving the long-standing blind spot of gRPC client
runtime observability. This interface surface will serve as the unified carrier
for all subsequent Proxy-side admin tools, including configuration management,
quota control, session management and diagnostic capabilities, which will be
rolled out progressively based on community demand.
---
## 1. Background & Motivation
As RocketMQ 5.0 elevates Proxy to a first-class access layer with protocol
parsing, authentication, rate limiting, routing and telemetry reporting
capabilities, the lack of a standardized admin interface surface creates
structural operational pain points:
1. **Invisible runtime state**: gRPC client connection status, subscription
configuration, heartbeat liveness and consumption progress are only maintained
internally within Proxy, with no standard query entry for control planes. This
directly causes known issues #381 (gRPC consumer latency displays -1) and #380
(Pop client status false positive).
2. **Fragmented and unstandardized admin capabilities**: Operations such as
configuration query, rate limit adjustment and session management either reuse
data-plane RPC interfaces or rely on log troubleshooting, lacking unified
service definitions, authentication systems and version contracts.
3. **High ecosystem integration cost**: Upper-layer ecosystems including
control panels, AI agents and CLI tools cannot connect to Proxy-side O&M
capabilities through standard interfaces, and can only infer runtime status via
bypass metrics and logs, with insufficient accuracy and timeliness.
---
## 2. Core Goals
1. Establish an independent Proxy Admin gRPC interface surface, fully
decoupled from data-plane interfaces in terms of port, authentication and
service definition.
2. Deliver online client query capabilities in the first release, covering
gRPC client list, details and group/topic dimension filtering, to support the
RIP-1 CLIENT-01 final view.
3. Unify the ACL 2.0 authentication model with the independent
`proxy.admin.*` resource hierarchy for fine-grained permission control.
4. Establish standardized interface contracts and version evolution rules to
ensure backward compatibility, serving as the unified underlying entry for
upper-layer control panels, AI tools and CLIs.
5. Guarantee high performance: paginated query P99 latency < 1s at
million-connection scale, without blocking normal data-plane traffic.
---
## 3. Non-Goals
- Does not modify Broker or NameServer core logic; only targets Proxy-side
admin capability construction.
- The initial release does not enforce cluster-level aggregated views; only
single-Proxy local views are provided, and aggregation can be implemented by
upper-layer control platforms.
- Does not replace existing `mqadmin` O&M commands; serves as a
supplementary gRPC standardized admin channel.
- Planned modules (configuration, quota, session, diagnostics) are not
mandatory for the initial release; they will be iterated version by version
based on community needs.
---
## 4. Core Design Principles
1. **Interface isolation**: Admin interfaces are separated from data-plane
message interfaces in terms of service, port and authentication to prevent
operational actions from affecting data-plane stability.
2. **Progressive iteration**: Delivered module by module. Client query is
delivered first, and capabilities such as configuration, quota and diagnosis
are expanded on demand.
3. **Least privilege**: Independent `proxy.admin.*` resource system with
separated read/write permissions, aligned with the ACL 2.0 authentication
baseline.
4. **Performance first**: Query interfaces support filter pushdown,
pagination and sampling. Full memory dump of data is prohibited to ensure
stability at million-connection scale.
5. **Stable contract**: Proto definitions follow Apache RocketMQ interface
compatibility specifications; new fields are backward compatible, and
deprecated fields reserve a transition period.
6. **Built-in observability**: Admin interfaces natively expose
OpenTelemetry metrics including call count, latency and error rate for
monitoring the health of the admin channel.
---
## 5. Scope of Work
### M1: Initial Delivery — Online Client Query
The first delivered module is the core unit of the Proxy Admin interface
surface, which directly solves the industry pain point of invisible gRPC
clients and supports the final view of RIP-1 CLIENT-01.
#### Core RPC Interfaces (4 in total)
All client data is sourced from the internal `ClientManager` module of
Proxy, maintained based on information reported by the gRPC Telemetry stream.
| RPC Method | Description | Input Parameters |
|------------|-------------|------------------|
| `ListClients` | Paginated filtered client list with server-side filter
pushdown | `group`, `topic`, `clientIdPrefix`, `language`, `connectTimeStart`,
`connectTimeEnd`, `pageNum`, `pageSize` |
| `DescribeClient` | Full telemetry details for a single client instance |
`clientId` |
| `ListClientsByGroup` | O&M-friendly shortcut query by consumer group |
`group`, `pageNum`, `pageSize` |
| `ListClientsByTopic` | O&M-friendly shortcut query by topic | `topic`,
`pageNum`, `pageSize` |
#### ACL 2.0 Resource Model
Independent admin resource hierarchy completely separated from data-plane
resources:
```
proxy.admin (root resource)
└── proxy.admin.client (client query module, read-only permission)
```
#### Performance & Implementation Requirements
- All filter conditions are pushed down to the ClientManager layer; full
in-memory traversal after query is prohibited.
- Mandatory pagination with maximum `pageSize` of 100 to prevent large
queries from causing memory pressure.
- Independent thread pool for admin interfaces, completely isolated from the
data-plane thread pool.
- Weak consistency snapshot: client list is a near-real-time snapshot,
allowing second-level delay to avoid locking that affects data-plane
performance.
- Built-in OpenTelemetry metrics for all admin interfaces: QPS, P95/P99
latency, error rate.
### M2+: Planned Iteration Modules
The following modules are the long-term evolution direction of the Proxy
Admin interface surface. They are not mandatory for the initial release and
will be rolled out version by version based on community demand and priority:
1. **Runtime configuration query & hot update**: `DescribeProxyConfig` /
`UpdateProxyConfig` to query and modify runtime configurations without restart.
2. **Rate limit quota visualization & adjustment**: Query current rate
limiting policies and trigger counts by namespace/topic/group dimensions;
support controlled quota adjustment.
3. **Cross-node session & connection control**: `KickClient` /
`DisconnectChannel` to proactively remove abnormal or lost clients, covering
session synchronization semantics in multi-Proxy deployment.
4. **Advanced semantic O&M diagnosis**: Expose Pop lock holding view,
ACK/NACK count, batch pull window status and NOT_CONSUME_YET trigger reasons
for troubleshooting.
5. **Routing & load balancing observation**: View Proxy → Broker connection
topology, current load distribution, fault removal status and geo-affinity
routing effectiveness.
---
## 6. Acceptance Criteria
### M1 Initial Release Acceptance
- [ ] Proxy Admin overall architecture and proto definitions pass community
review, with clear interface positioning, naming conventions, permission system
and evolution strategies.
- [ ] All 4 core client query RPCs are implemented, with complete functions,
accurate filtering and normal pagination.
- [ ] Paginated query P99 latency < 1s on a single Proxy with million
connections, no memory overflow risk.
- [ ] Fully integrated with ACL 2.0 authentication, with independent
`proxy.admin.client` resource type and minimum permission policy documented.
- [ ] OpenTelemetry metrics for admin interfaces are enabled by default,
meeting observability requirements.
- [ ] End-to-end joint debugging with RIP-1 Dashboard CLIENT-01 passes, and
the gRPC client view is displayed normally.
- [ ] Complete bilingual interface documentation, authentication
configuration examples and best practice manuals are provided.
### Overall Architecture Acceptance
- [ ] Independent admin service framework is established, supporting
subsequent module expansion without structural changes.
- [ ] Admin interfaces are completely isolated from the data plane in terms
of port, thread pool and authentication.
- [ ] Interface contracts follow Apache RocketMQ compatibility
specifications and support smooth version iteration.
### Quality & Documentation
- [ ] Unit test coverage of core modules ≥ 85%.
- [ ] Integration tests cover all interfaces, authentication scenarios and
exception handling.
- [ ] Performance benchmark report for million-connection scale is available.
- [ ] Bilingual (EN/CN) official documents are synchronized with the code
release.
---
## 7. Related RIPs & Dependencies
- **RIP-1 (Control Plane 5.0)**: RIP-1 CLIENT-01 final view depends on the
M1 module of this RIP. Before the M1 module is released, CLIENT-01 uses
telemetry metrics to build a transition view, and switches to the final view
seamlessly after release. The two RIPs can be developed in parallel.
- **RIP-3 (LLM Native Integration)**: RIP-3's `rmq.client.*` series of tools
directly benefit from the M1 module of this RIP. Subsequent planned modules
will also be synchronously exposed to LLM toolchains through RIP-3.
- **Dependency rules**:
- RIP-2 itself does not depend on other RIPs and can be developed and
released independently.
- RIP-1 has a soft dependency on RIP-2 M1; the transition view is
available without RIP-2.
- RIP-3 indirectly depends on RIP-2 through RIP-1's abstraction layer.
---
## 8. Contribution Notes
- All code is delivered in the `proxy` module of the `apache/rocketmq` main
repository.
- All submissions must follow Apache RocketMQ community coding standards,
proto specification and contribution guidelines.
- Sub-tasks can be claimed by commenting on this issue. Please raise design
questions and technical discussions in this thread or the corresponding dev
mailing list.
- Planned modules can be proposed and promoted by interested contributors,
and will be included in the RIP iteration roadmap after community review.
### Describe the Solution You'd Like
# Issue Title
**[RIP-2] Proxy Admin gRPC Interface Surface (M1: Online Client Query
Module)**
---
## Issue Description
This tracking issue covers the delivery of RIP-2 (Proxy Admin Interface
Surface) in the `apache/rocketmq` main repository.
The initiative establishes an independent, standardized admin gRPC interface
layer on the Proxy side, fully decoupled from the data-plane messaging
pipeline. The first milestone (M1) delivers online client query capabilities as
the foundational module, resolving the long-standing blind spot of gRPC client
runtime observability. This interface surface will serve as the unified carrier
for all subsequent Proxy-side admin tools, including configuration management,
quota control, session management and diagnostic capabilities, which will be
rolled out progressively based on community demand.
---
## 1. Background & Motivation
As RocketMQ 5.0 elevates Proxy to a first-class access layer with protocol
parsing, authentication, rate limiting, routing and telemetry reporting
capabilities, the lack of a standardized admin interface surface creates
structural operational pain points:
1. **Invisible runtime state**: gRPC client connection status, subscription
configuration, heartbeat liveness and consumption progress are only maintained
internally within Proxy, with no standard query entry for control planes. This
directly causes known issues #381 (gRPC consumer latency displays -1) and #380
(Pop client status false positive).
2. **Fragmented and unstandardized admin capabilities**: Operations such as
configuration query, rate limit adjustment and session management either reuse
data-plane RPC interfaces or rely on log troubleshooting, lacking unified
service definitions, authentication systems and version contracts.
3. **High ecosystem integration cost**: Upper-layer ecosystems including
control panels, AI agents and CLI tools cannot connect to Proxy-side O&M
capabilities through standard interfaces, and can only infer runtime status via
bypass metrics and logs, with insufficient accuracy and timeliness.
---
## 2. Core Goals
1. Establish an independent Proxy Admin gRPC interface surface, fully
decoupled from data-plane interfaces in terms of port, authentication and
service definition.
2. Deliver online client query capabilities in the first release, covering
gRPC client list, details and group/topic dimension filtering, to support the
RIP-1 CLIENT-01 final view.
3. Unify the ACL 2.0 authentication model with the independent
`proxy.admin.*` resource hierarchy for fine-grained permission control.
4. Establish standardized interface contracts and version evolution rules to
ensure backward compatibility, serving as the unified underlying entry for
upper-layer control panels, AI tools and CLIs.
5. Guarantee high performance: paginated query P99 latency < 1s at
million-connection scale, without blocking normal data-plane traffic.
---
## 3. Non-Goals
- Does not modify Broker or NameServer core logic; only targets Proxy-side
admin capability construction.
- The initial release does not enforce cluster-level aggregated views; only
single-Proxy local views are provided, and aggregation can be implemented by
upper-layer control platforms.
- Does not replace existing `mqadmin` O&M commands; serves as a
supplementary gRPC standardized admin channel.
- Planned modules (configuration, quota, session, diagnostics) are not
mandatory for the initial release; they will be iterated version by version
based on community needs.
---
## 4. Core Design Principles
1. **Interface isolation**: Admin interfaces are separated from data-plane
message interfaces in terms of service, port and authentication to prevent
operational actions from affecting data-plane stability.
2. **Progressive iteration**: Delivered module by module. Client query is
delivered first, and capabilities such as configuration, quota and diagnosis
are expanded on demand.
3. **Least privilege**: Independent `proxy.admin.*` resource system with
separated read/write permissions, aligned with the ACL 2.0 authentication
baseline.
4. **Performance first**: Query interfaces support filter pushdown,
pagination and sampling. Full memory dump of data is prohibited to ensure
stability at million-connection scale.
5. **Stable contract**: Proto definitions follow Apache RocketMQ interface
compatibility specifications; new fields are backward compatible, and
deprecated fields reserve a transition period.
6. **Built-in observability**: Admin interfaces natively expose
OpenTelemetry metrics including call count, latency and error rate for
monitoring the health of the admin channel.
---
## 5. Scope of Work
### M1: Initial Delivery — Online Client Query
The first delivered module is the core unit of the Proxy Admin interface
surface, which directly solves the industry pain point of invisible gRPC
clients and supports the final view of RIP-1 CLIENT-01.
#### Core RPC Interfaces (4 in total)
All client data is sourced from the internal `ClientManager` module of
Proxy, maintained based on information reported by the gRPC Telemetry stream.
| RPC Method | Description | Input Parameters |
|------------|-------------|------------------|
| `ListClients` | Paginated filtered client list with server-side filter
pushdown | `group`, `topic`, `clientIdPrefix`, `language`, `connectTimeStart`,
`connectTimeEnd`, `pageNum`, `pageSize` |
| `DescribeClient` | Full telemetry details for a single client instance |
`clientId` |
| `ListClientsByGroup` | O&M-friendly shortcut query by consumer group |
`group`, `pageNum`, `pageSize` |
| `ListClientsByTopic` | O&M-friendly shortcut query by topic | `topic`,
`pageNum`, `pageSize` |
#### ACL 2.0 Resource Model
Independent admin resource hierarchy completely separated from data-plane
resources:
```
proxy.admin (root resource)
└── proxy.admin.client (client query module, read-only permission)
```
#### Performance & Implementation Requirements
- All filter conditions are pushed down to the ClientManager layer; full
in-memory traversal after query is prohibited.
- Mandatory pagination with maximum `pageSize` of 100 to prevent large
queries from causing memory pressure.
- Independent thread pool for admin interfaces, completely isolated from the
data-plane thread pool.
- Weak consistency snapshot: client list is a near-real-time snapshot,
allowing second-level delay to avoid locking that affects data-plane
performance.
- Built-in OpenTelemetry metrics for all admin interfaces: QPS, P95/P99
latency, error rate.
### M2+: Planned Iteration Modules
The following modules are the long-term evolution direction of the Proxy
Admin interface surface. They are not mandatory for the initial release and
will be rolled out version by version based on community demand and priority:
1. **Runtime configuration query & hot update**: `DescribeProxyConfig` /
`UpdateProxyConfig` to query and modify runtime configurations without restart.
2. **Rate limit quota visualization & adjustment**: Query current rate
limiting policies and trigger counts by namespace/topic/group dimensions;
support controlled quota adjustment.
3. **Cross-node session & connection control**: `KickClient` /
`DisconnectChannel` to proactively remove abnormal or lost clients, covering
session synchronization semantics in multi-Proxy deployment.
4. **Advanced semantic O&M diagnosis**: Expose Pop lock holding view,
ACK/NACK count, batch pull window status and NOT_CONSUME_YET trigger reasons
for troubleshooting.
5. **Routing & load balancing observation**: View Proxy → Broker connection
topology, current load distribution, fault removal status and geo-affinity
routing effectiveness.
---
## 6. Acceptance Criteria
### M1 Initial Release Acceptance
- [ ] Proxy Admin overall architecture and proto definitions pass community
review, with clear interface positioning, naming conventions, permission system
and evolution strategies.
- [ ] All 4 core client query RPCs are implemented, with complete functions,
accurate filtering and normal pagination.
- [ ] Paginated query P99 latency < 1s on a single Proxy with million
connections, no memory overflow risk.
- [ ] Fully integrated with ACL 2.0 authentication, with independent
`proxy.admin.client` resource type and minimum permission policy documented.
- [ ] OpenTelemetry metrics for admin interfaces are enabled by default,
meeting observability requirements.
- [ ] End-to-end joint debugging with RIP-1 Dashboard CLIENT-01 passes, and
the gRPC client view is displayed normally.
- [ ] Complete bilingual interface documentation, authentication
configuration examples and best practice manuals are provided.
### Overall Architecture Acceptance
- [ ] Independent admin service framework is established, supporting
subsequent module expansion without structural changes.
- [ ] Admin interfaces are completely isolated from the data plane in terms
of port, thread pool and authentication.
- [ ] Interface contracts follow Apache RocketMQ compatibility
specifications and support smooth version iteration.
### Quality & Documentation
- [ ] Unit test coverage of core modules ≥ 85%.
- [ ] Integration tests cover all interfaces, authentication scenarios and
exception handling.
- [ ] Performance benchmark report for million-connection scale is available.
- [ ] Bilingual (EN/CN) official documents are synchronized with the code
release.
---
## 7. Related RIPs & Dependencies
- **RIP-1 (Control Plane 5.0)**: RIP-1 CLIENT-01 final view depends on the
M1 module of this RIP. Before the M1 module is released, CLIENT-01 uses
telemetry metrics to build a transition view, and switches to the final view
seamlessly after release. The two RIPs can be developed in parallel.
- **RIP-3 (LLM Native Integration)**: RIP-3's `rmq.client.*` series of tools
directly benefit from the M1 module of this RIP. Subsequent planned modules
will also be synchronously exposed to LLM toolchains through RIP-3.
- **Dependency rules**:
- RIP-2 itself does not depend on other RIPs and can be developed and
released independently.
- RIP-1 has a soft dependency on RIP-2 M1; the transition view is
available without RIP-2.
- RIP-3 indirectly depends on RIP-2 through RIP-1's abstraction layer.
---
## 8. Contribution Notes
- All code is delivered in the `proxy` module of the `apache/rocketmq` main
repository.
- All submissions must follow Apache RocketMQ community coding standards,
proto specification and contribution guidelines.
- Sub-tasks can be claimed by commenting on this issue. Please raise design
questions and technical discussions in this thread or the corresponding dev
mailing list.
- Planned modules can be proposed and promoted by interested contributors,
and will be included in the RIP iteration roadmap after community review.
### Describe Alternatives You've Considered
# Issue Title
**[RIP-2] Proxy Admin gRPC Interface Surface (M1: Online Client Query
Module)**
---
## Issue Description
This tracking issue covers the delivery of RIP-2 (Proxy Admin Interface
Surface) in the `apache/rocketmq` main repository.
The initiative establishes an independent, standardized admin gRPC interface
layer on the Proxy side, fully decoupled from the data-plane messaging
pipeline. The first milestone (M1) delivers online client query capabilities as
the foundational module, resolving the long-standing blind spot of gRPC client
runtime observability. This interface surface will serve as the unified carrier
for all subsequent Proxy-side admin tools, including configuration management,
quota control, session management and diagnostic capabilities, which will be
rolled out progressively based on community demand.
---
## 1. Background & Motivation
As RocketMQ 5.0 elevates Proxy to a first-class access layer with protocol
parsing, authentication, rate limiting, routing and telemetry reporting
capabilities, the lack of a standardized admin interface surface creates
structural operational pain points:
1. **Invisible runtime state**: gRPC client connection status, subscription
configuration, heartbeat liveness and consumption progress are only maintained
internally within Proxy, with no standard query entry for control planes. This
directly causes known issues #381 (gRPC consumer latency displays -1) and #380
(Pop client status false positive).
2. **Fragmented and unstandardized admin capabilities**: Operations such as
configuration query, rate limit adjustment and session management either reuse
data-plane RPC interfaces or rely on log troubleshooting, lacking unified
service definitions, authentication systems and version contracts.
3. **High ecosystem integration cost**: Upper-layer ecosystems including
control panels, AI agents and CLI tools cannot connect to Proxy-side O&M
capabilities through standard interfaces, and can only infer runtime status via
bypass metrics and logs, with insufficient accuracy and timeliness.
---
## 2. Core Goals
1. Establish an independent Proxy Admin gRPC interface surface, fully
decoupled from data-plane interfaces in terms of port, authentication and
service definition.
2. Deliver online client query capabilities in the first release, covering
gRPC client list, details and group/topic dimension filtering, to support the
RIP-1 CLIENT-01 final view.
3. Unify the ACL 2.0 authentication model with the independent
`proxy.admin.*` resource hierarchy for fine-grained permission control.
4. Establish standardized interface contracts and version evolution rules to
ensure backward compatibility, serving as the unified underlying entry for
upper-layer control panels, AI tools and CLIs.
5. Guarantee high performance: paginated query P99 latency < 1s at
million-connection scale, without blocking normal data-plane traffic.
---
## 3. Non-Goals
- Does not modify Broker or NameServer core logic; only targets Proxy-side
admin capability construction.
- The initial release does not enforce cluster-level aggregated views; only
single-Proxy local views are provided, and aggregation can be implemented by
upper-layer control platforms.
- Does not replace existing `mqadmin` O&M commands; serves as a
supplementary gRPC standardized admin channel.
- Planned modules (configuration, quota, session, diagnostics) are not
mandatory for the initial release; they will be iterated version by version
based on community needs.
---
## 4. Core Design Principles
1. **Interface isolation**: Admin interfaces are separated from data-plane
message interfaces in terms of service, port and authentication to prevent
operational actions from affecting data-plane stability.
2. **Progressive iteration**: Delivered module by module. Client query is
delivered first, and capabilities such as configuration, quota and diagnosis
are expanded on demand.
3. **Least privilege**: Independent `proxy.admin.*` resource system with
separated read/write permissions, aligned with the ACL 2.0 authentication
baseline.
4. **Performance first**: Query interfaces support filter pushdown,
pagination and sampling. Full memory dump of data is prohibited to ensure
stability at million-connection scale.
5. **Stable contract**: Proto definitions follow Apache RocketMQ interface
compatibility specifications; new fields are backward compatible, and
deprecated fields reserve a transition period.
6. **Built-in observability**: Admin interfaces natively expose
OpenTelemetry metrics including call count, latency and error rate for
monitoring the health of the admin channel.
---
## 5. Scope of Work
### M1: Initial Delivery — Online Client Query
The first delivered module is the core unit of the Proxy Admin interface
surface, which directly solves the industry pain point of invisible gRPC
clients and supports the final view of RIP-1 CLIENT-01.
#### Core RPC Interfaces (4 in total)
All client data is sourced from the internal `ClientManager` module of
Proxy, maintained based on information reported by the gRPC Telemetry stream.
| RPC Method | Description | Input Parameters |
|------------|-------------|------------------|
| `ListClients` | Paginated filtered client list with server-side filter
pushdown | `group`, `topic`, `clientIdPrefix`, `language`, `connectTimeStart`,
`connectTimeEnd`, `pageNum`, `pageSize` |
| `DescribeClient` | Full telemetry details for a single client instance |
`clientId` |
| `ListClientsByGroup` | O&M-friendly shortcut query by consumer group |
`group`, `pageNum`, `pageSize` |
| `ListClientsByTopic` | O&M-friendly shortcut query by topic | `topic`,
`pageNum`, `pageSize` |
#### ACL 2.0 Resource Model
Independent admin resource hierarchy completely separated from data-plane
resources:
```
proxy.admin (root resource)
└── proxy.admin.client (client query module, read-only permission)
```
#### Performance & Implementation Requirements
- All filter conditions are pushed down to the ClientManager layer; full
in-memory traversal after query is prohibited.
- Mandatory pagination with maximum `pageSize` of 100 to prevent large
queries from causing memory pressure.
- Independent thread pool for admin interfaces, completely isolated from the
data-plane thread pool.
- Weak consistency snapshot: client list is a near-real-time snapshot,
allowing second-level delay to avoid locking that affects data-plane
performance.
- Built-in OpenTelemetry metrics for all admin interfaces: QPS, P95/P99
latency, error rate.
### M2+: Planned Iteration Modules
The following modules are the long-term evolution direction of the Proxy
Admin interface surface. They are not mandatory for the initial release and
will be rolled out version by version based on community demand and priority:
1. **Runtime configuration query & hot update**: `DescribeProxyConfig` /
`UpdateProxyConfig` to query and modify runtime configurations without restart.
2. **Rate limit quota visualization & adjustment**: Query current rate
limiting policies and trigger counts by namespace/topic/group dimensions;
support controlled quota adjustment.
3. **Cross-node session & connection control**: `KickClient` /
`DisconnectChannel` to proactively remove abnormal or lost clients, covering
session synchronization semantics in multi-Proxy deployment.
4. **Advanced semantic O&M diagnosis**: Expose Pop lock holding view,
ACK/NACK count, batch pull window status and NOT_CONSUME_YET trigger reasons
for troubleshooting.
5. **Routing & load balancing observation**: View Proxy → Broker connection
topology, current load distribution, fault removal status and geo-affinity
routing effectiveness.
---
## 6. Acceptance Criteria
### M1 Initial Release Acceptance
- [ ] Proxy Admin overall architecture and proto definitions pass community
review, with clear interface positioning, naming conventions, permission system
and evolution strategies.
- [ ] All 4 core client query RPCs are implemented, with complete functions,
accurate filtering and normal pagination.
- [ ] Paginated query P99 latency < 1s on a single Proxy with million
connections, no memory overflow risk.
- [ ] Fully integrated with ACL 2.0 authentication, with independent
`proxy.admin.client` resource type and minimum permission policy documented.
- [ ] OpenTelemetry metrics for admin interfaces are enabled by default,
meeting observability requirements.
- [ ] End-to-end joint debugging with RIP-1 Dashboard CLIENT-01 passes, and
the gRPC client view is displayed normally.
- [ ] Complete bilingual interface documentation, authentication
configuration examples and best practice manuals are provided.
### Overall Architecture Acceptance
- [ ] Independent admin service framework is established, supporting
subsequent module expansion without structural changes.
- [ ] Admin interfaces are completely isolated from the data plane in terms
of port, thread pool and authentication.
- [ ] Interface contracts follow Apache RocketMQ compatibility
specifications and support smooth version iteration.
### Quality & Documentation
- [ ] Unit test coverage of core modules ≥ 85%.
- [ ] Integration tests cover all interfaces, authentication scenarios and
exception handling.
- [ ] Performance benchmark report for million-connection scale is available.
- [ ] Bilingual (EN/CN) official documents are synchronized with the code
release.
---
## 7. Related RIPs & Dependencies
- **RIP-1 (Control Plane 5.0)**: RIP-1 CLIENT-01 final view depends on the
M1 module of this RIP. Before the M1 module is released, CLIENT-01 uses
telemetry metrics to build a transition view, and switches to the final view
seamlessly after release. The two RIPs can be developed in parallel.
- **RIP-3 (LLM Native Integration)**: RIP-3's `rmq.client.*` series of tools
directly benefit from the M1 module of this RIP. Subsequent planned modules
will also be synchronously exposed to LLM toolchains through RIP-3.
- **Dependency rules**:
- RIP-2 itself does not depend on other RIPs and can be developed and
released independently.
- RIP-1 has a soft dependency on RIP-2 M1; the transition view is
available without RIP-2.
- RIP-3 indirectly depends on RIP-2 through RIP-1's abstraction layer.
---
## 8. Contribution Notes
- All code is delivered in the `proxy` module of the `apache/rocketmq` main
repository.
- All submissions must follow Apache RocketMQ community coding standards,
proto specification and contribution guidelines.
- Sub-tasks can be claimed by commenting on this issue. Please raise design
questions and technical discussions in this thread or the corresponding dev
mailing list.
- Planned modules can be proposed and promoted by interested contributors,
and will be included in the RIP iteration roadmap after community review.
### Additional Context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
