lizining1231 opened a new pull request, #3706:
URL: https://github.com/apache/dubbo-go/pull/3706
### Description
Fixes #3694
Triple unary RPCs go through `duplexHTTPCall`, which allocates an `io.Pipe`,
a per-request `makeRequest` goroutine and a `responseReady` signal per call.
For small payloads (CPU-dominated hot path) this pipe hand-off machinery is
pure overhead. On top of that, the fast path never actually ran end-to-end
because of a protocol-selection defect:
1. **Default protocol defect**: `newClientManager` only passed
`tri.WithTriple()` on the HTTP/1.1 branch; HTTP/2 (the actual default
transport) silently fell back to the gRPC protocol , the
`triple_protocol.Client` default inherited from connect-go. Every end-to-end
benchmark therefore measured the gRPC compatibility path, and a fast path
living on `tripleClient.NewConn` was unreachable.
2. **Per-request pipe overhead**: each unary call creates an `io.Pipe`
writer task, spawns a `makeRequest` goroutine, and blocks on a `responseReady`
close-signal hand-off, two execution contexts where one synchronous caller
context suffices.
### Changes
**1. Config switch: add an opt-in `UnaryFastPath` toggle, off by default**
`global/triple_config.go`:
- Add `UnaryFastPath bool` to `TripleConfig` (yaml/json/property mapped) and
copy it in `Clone`.
`protocol/triple/triple_protocol/option.go` / `client.go` / `protocol.go`:
- Add `WithUnaryFastPath()` client option and propagate it through
`clientConfig` / `protocolClientParams` into `NewConn`.
**2. Fast path implementation: replace the duplex pipe machinery with a
synchronous call**
`protocol/triple/triple_protocol/unary_fastpath.go` (new):
- `unaryFastPathCall` performs the unary round trip synchronously in the
caller goroutine: no `io.Pipe`, no `makeRequest` goroutine; the `responseReady`
signal is resolved synchronously instead of from a separate goroutine;
- Uses pooled buffers, declares `Content-Length`, reads the response
directly;
- The write side (`Send` / `CloseRequest`) is mutex-guarded; `RequestHeader`
is safe to call concurrently because headers are never mutated after
construction, so the `StreamingClientConn` concurrency-safe is preserved.
`protocol/triple/triple_protocol/protocol_triple.go`:
- `tripleClient.NewConn` selects `newUnaryFastPathCall` for unary stream
types when the switch is on; streaming keeps the duplex path.
`protocol/triple/client.go`:
- Hoist `tri.WithTriple()` so HTTP/1.1, HTTP/2 and HTTP/3 all explicitly
select the Triple protocol (HTTP/2 previously fell back to gRPC, making the
fast path unreachable), and gate `tri.WithUnaryFastPath()` on the config switch.
**3. Tests: conn routing, production bench baseline, body round-trip and
race coverage**
`protocol/triple/triple_protocol/unary_fastpath_conn_type_test.go`:
- Asserts `NewConn` routes unary to the fast path and streaming to duplex.
`protocol/triple/triple_protocol/unary_fastpath_bench_test.go`:
- `BenchmarkUnaryFastPathProduction` drives the production fast path with
`WithTriple + WithUnaryFastPath` for an A/B baseline against duplex.
`protocol/triple/triple_protocol/unary_fastpath_body_test.go`:
- Request body read/write round-trips, `Content-Length` declaration and
pooled buffer reuse.
`protocol/triple/triple_protocol/unary_fastpath_concurrency_test.go`:
- Concurrent `Send` / `RequestHeader` / `CloseRequest` stay race-free under
`-race`.
### Test
`unary_fastpath_conn_type_test.go`
| Test | Description |
|---|---|
| `TestUnaryFastPathNewConnType` | Switch on routes unary calls to the fast
path, off falls back to duplex, streaming always uses duplex |
`unary_fastpath_behavior_test.go`
| Test | Description |
|---|---|
| `TestUnaryFastPathWriteAccumulates` | Write appends into the pooled buffer
without touching the network; CloseWrite hands the whole payload to the
transport exactly once with an exact Content-Length |
| `TestUnaryFastPathEmptyBodyUsesNoBody` | An empty request body uses
`http.NoBody` with Content-Length=0, avoiding chunked encoding and background
writes |
| `TestUnaryFastPathConcurrentReadClose` | 8 readers racing 8 closes,
verified race-free under `-race` |
| `TestUnaryFastPathRequestHeaderConcurrent` | RequestHeader races with
Write and CloseWrite safely, per the streaming client conn contract |
| `TestUnaryFastPathWriteAfterTransportError` | After a transport failure
writes are rejected with io.EOF and Read surfaces CodeUnavailable |
| `TestUnaryFastPathSetErrorRejectsWrite` | A stored error rejects writes
before the body is sent, surfacing the concrete error |
| `TestUnaryFastPathWireConsistent` | The fast path wire bytes are identical
to duplex, so the protocol does not drift |
| `TestUnaryFastPathContextCancel` | Context cancellation aborts the
in-flight request and Read surfaces CodeCanceled |
| `TestUnaryFastPathEndToEnd` | End-to-end against a real HTTP/2 server
through the real NewConn entry, covering header/trailer interop and response
reading |
| `TestUnaryFastPathProtoJSONCodec` | The JSON codec works end to end,
matching duplex behavior to keep codec generality |
| `TestUnaryFastPathGRPCKeepsDuplex` | gRPC protocol calls are unaffected by
the switch and keep using duplex even when enabled, preserving gRPC behavior |
`unary_fastpath_body_test.go`
| Test | Description |
|---|---|
| `TestUnaryFastPathBodyReturnOnAsyncClose` | Close returns the buffer to
the pool exactly once, Read returns EOF afterwards, and pooled reuse stays
clean |
| `TestUnaryFastPathBodyAbortServer` | A non-2xx early response still
returns the buffer to the pool, surfaces CodePermissionDenied, and propagates
the error metadata |
`unary_fastpath_concurrency_test.go`
| Test | Description |
|---|---|
| `TestUnaryFastPathWriteCloseConcurrent` | Concurrent Write/CloseWrite over
30 rounds with no torn body observed by the server, pinning the write-side lock
boundary |
| `TestUnaryFastPathWriteAfterClose` | Write after bodySent returns io.EOF,
mirroring duplex close semantics |
`unary_fastpath_bench_test.go`
| Benchmark | Description |
|---|---|
| `BenchmarkUnaryDuplex` | Control group: the generated client with the fast
path disabled (duplex) |
| `BenchmarkUnaryFastPathProduction` | Treatment group: differs from the
control only by the WithUnaryFastPath option |
### Validation
# Dubbo-Go Triple Protocol Performance Benchmark Summary Report
### 128 bytes payload
| Metric | Concurrency | duplex | fastpath | Improvement |
| --- | --- | --- | --- | --- |
| QPS | 50 | 5,329.3 | 6,997.9 | +31.3% ↑ |
| QPS | 100 | 5,409.2 | 6,024.1 | +11.4% ↑ |
| P99 latency (ms) | 50 | 15.99 | 12.67 | -20.8% ↑ |
| P99 latency (ms) | 100 | 30.58 | 29.74 | -2.7% ↑ |
### 1024 bytes payload
| Metric | Concurrency | duplex | fastpath | Improvement |
| --- | --- | --- | --- | --- |
| QPS | 50 | 4,300.3 | 5,196.4 | +20.8% ↑ |
| QPS | 100 | 4,368.5 | 5,605.8 | +28.3% ↑ |
| P99 latency (ms) | 50 | 20.40 | 18.22 | -10.7% ↑ |
| P99 latency (ms) | 100 | 37.48 | 26.95 | -28.1% ↑ |
### 16384 bytes payload
| Metric | Concurrency | duplex | fastpath | Improvement |
| --- | --- | --- | --- | --- |
| QPS | 50 | 2,584.2 | 2,871.6 | +11.1% ↑ |
| QPS | 100 | 2,428.2 | 2,815.7 | +16.0% ↑ |
| P99 latency (ms) | 50 | 37.28 | 30.90 | -17.1% ↑ |
| P99 latency (ms) | 100 | 81.24 | 69.07 | -15.0% ↑ |
### 1048576 bytes (1MiB) payload
| Metric | Concurrency | duplex | fastpath | Improvement |
| --- | --- | --- | --- | --- |
| QPS | 50 | 101.3 | 101.7 | +0.4% ↑ |
| QPS | 100 | 104.3 | 114.4 | +9.8% ↑ |
| P99 latency (ms) | 50 | 1,430.17 | 1,479.12 | +3.4% ↓ |
| P99 latency (ms) | 100 | 3,179.88 | 2,874.32 | -9.6% ↑ |
## benchstat significance test (count=10)
```bash
goos: linux
goarch: amd64
pkg: dubbo.apache.org/dubbo-go/v3/protocol/triple/triple_protocol
cpu: Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz
│ duplex10_1mib.txt │ fastpath10_1mib.txt │
│ sec/op │ sec/op vs base │
Unary/128B-12 147.1µ ± 8% 117.6µ ± 11% -20.08% (p=0.000 n=10)
Unary/1024B-12 169.9µ ± 18% 106.0µ ± 30% -37.59% (p=0.000 n=10)
Unary/16384B-12 226.1µ ± 26% 173.1µ ± 12% -23.43% (p=0.000 n=10)
Unary/1MiB-12 3.662m ± 19% 3.450m ± 10% ~ (p=0.912 n=10)
geomean 379.3µ 293.8µ -22.55%
```
*Note: the names above come from a unified `BenchmarkUnary` run; in the
current
codebase the control is `BenchmarkUnaryDuplex` and the treatment is
`BenchmarkUnaryFastPathProduction`.*
| Payload | duplex (sec/op) | fastpath (sec/op) | Change | p-value |
Significance |
| --- | --- | --- | --- | --- | --- |
| 128B | 147.1µs | 117.6µs | -20.08% | 0.000 | Significant |
| 1KiB | 169.9µs | 106.0µs | -37.59% | 0.000 | Significant |
| 16KiB | 226.1µs | 173.1µs | -23.43% | 0.000 | Significant |
| 1MiB | 3.662ms | 3.450ms | -5.8% | 0.912 | Not significant |
| geomean | 379.3µs | 293.8µs | -22.55% | - | Overall positive |
# -race test
```bash
lizining@Y:~/projects/dubbo-go$ go test -race ./protocol/triple/... 2>&1 |
grep -Ev "no test files|\(cached\)"
go test -race -count=1 ./protocol/triple/triple_protocol/
# 3 rounds of writer-side concurrency testing
go test -race -count=3 -run
'TestUnaryFastPathWriteCloseConcurrent|TestUnaryFastPathSetErrorRejectsWrite|TestUnaryFastPathWriteAfterTransportError|TestUnaryFastPathRequestHeaderConcurrent|TestUnaryFastPathConcurrentReadClose|TestUnaryFastPathWriteAfterClose|TestUnaryFastPathGRPCKeepsDuplex|TestUnaryFastPathEndToEnd'
./protocol/triple/triple_protocol/
go test -race ./protocol/triple/... 2>&1 | grep FAIL || echo "无失败"
ok dubbo.apache.org/dubbo-go/v3/protocol/triple/triple_protocol
31.804s
ok dubbo.apache.org/dubbo-go/v3/protocol/triple/triple_protocol
1.457s
无失败
lizining@Y:~/projects/dubbo-go$
```
# pprof Benchmark
**Scenario:** Triple unary · 128B · 100 concurrent requests
The fast path removes the per-request `io.Pipe` pipe, the standalone
`makeRequest` goroutine, and the `responseReady` channel handoff. Total
goroutines decrease by 31%, and `syscall`/`futex` CPU usage drops.
### Goroutine Profile (310 → 213, -31%)
duplex
<img width="1600" height="900" alt="image"
src="https://github.com/user-attachments/assets/23d1ff6e-1b44-4bc6-b47f-0e1a019d375a";
/>
fastpath
<img width="1600" height="1040" alt="image"
src="https://github.com/user-attachments/assets/a305c789-981c-4f7f-8e47-4eb5920318a3";
/>
| | duplex | fastpath |
| :--- | :---: | :---: |
| Total goroutines | 310 | 213 |
| http2 connection-related frames | 26 | 16 |
| pipe-related frames | 3 | 1 |
### CPU Profile
duplex
<img width="1600" height="820" alt="image"
src="https://github.com/user-attachments/assets/2ff6cc33-02c3-41a8-a768-490f01f8166a";
/>
fastpath
<img width="1600" height="1060" alt="image"
src="https://github.com/user-attachments/assets/f32230c3-d174-4722-b423-59016ab9112c";
/>
| Hotspot | duplex | fastpath | Change |
| :--- | :---: | :---: | :---: |
| `syscall.Syscall6` | 28.82% | 24.64% | **-4.2%** |
| `runtime.futex` | 12.41% | 10.25% | **-2.2%** |
### Checklist
- [x] I confirm the target branch is `develop`
- [x] I have run `make fmt` to format my code
- [x] I have run `make test` to run local tests
- [x] I have added tests that prove my fix is effective or that my feature
works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
