chenBright opened a new pull request, #3720:
URL: https://github.com/apache/brpc/pull/3720
### What problem does this PR solve?
Issue Number: resolve
Problem Summary:
The existing RDMA transport allocates receive queues, buffers and completion
queues per connection. These
resources remain reserved even when connections are idle, making this model
expensive for workloads with
many connections and sparse RPC traffic.
### What is changed and the side effects?
Changed:
- Add opt-in, event-driven SRQ support with one shared receive queue,
receive pool and TX/RX completion
queues per worker.
- Separate private-RQ and SRQ implementations behind the `RdmaConnection`
interface while keeping handshake
handling in `RdmaEndpoint`.
- Negotiate SRQ through handshake v3. Use initial credits and soft/hard
credit requests with grants, allowing the
first request to be sent without an additional credit-request round trip.
- Preserve IOBuf zero-copy ownership across sends, receives and asynchronous
teardown. Reuse receive buffers
directly for small-message copies and control messages.
- Grow registered receive memory asynchronously within a configurable limit
and wake workers when blocks are returned.
- Batch shared TX CQ reservations, reserve capacity for control traffic, and
handle partial submissions and resource retirement.
Side effects:
Performance effects: Recorded comparison with 10,000 established connections
and 64-byte echo attachments:
- Client and server used separate physical RDMA NICs on the same host.
- Four senders rotated through the connections, sleeping 1 ms after each RPC.
- Both modes used event-driven progress.
- Each configuration ran twice, with 5 seconds of warmup and 30 seconds of
measurement per run.
- Connection establishment was excluded from RPC latency.
| Metric | Private RQ | SRQ |
|---|---:|---:|
| Client RSS after connection establishment | ~12,252 MiB | ~1,095 MiB |
| Average RPC latency | 62.1–63.4 µs | 62.7–64.5 µs |
| P50 RPC latency | 58–60 µs | 60 µs |
| P99 RPC latency | 105–118 µs | 89–94 µs |
Client RSS decreased by approximately 91%, while average and median latency
remained comparable and P99 was lower
in this sparse workload. Per-connection QP and SQ resources are still
required.
Ranges represent individual results from two runs. These measurements come
from the November 6 benchmark snapshot,
before subsequent refactors. They do not establish throughput limits or
cross-host performance.
Breaking backward compatibility:
SRQ is disabled by default. Existing private-RQ behavior and v2 handshakes
remain supported. SRQ requires
v3-capable peers with compatible SRQ settings. Failed SRQ negotiation
between v3 peers falls back to TCP.
SRQ cannot be combined with polling mode.
---
### Check List:
- Please make sure your changes are compilable.
- When providing us with a new feature, it is best to add related tests.
- Please follow [Contributor Covenant Code of
Conduct](https://github.com/apache/brpc/blob/master/CODE_OF_CONDUCT.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
