AlinsRan opened a new pull request, #13912:
URL: https://github.com/apache/apisix/pull/13912
### Description
Adds TLS passthrough to the stream proxy: a `stream_proxy.tcp` listen can
now forward the encrypted stream to the upstream untouched while still picking
that upstream from the SNI.
Until now "route by SNI" and "don't decrypt" were mutually exclusive in the
stream subsystem — the only SNI source was a handshake the worker performed
itself, so any SNI-matched stream route implied terminating TLS at the gateway.
A backend that owns its own certificate, or an operator who does not want
private keys on the gateway, had no way to get both.
#### Three modes per listen
```yaml
apisix:
proxy_mode: "http&stream"
stream_proxy:
tcp:
- addr: 9100
tls: true # terminate (unchanged)
- addr: 9101
tls_passthrough: true # pass everything through
- addr: 9102
tls: true # mixed: the matched stream_route decides
tls_passthrough: true
```
A passthrough listen is rendered without `ssl`, with `ssl_preread on`, and
`apisix/ssl.lua` reads the SNI captured by `ngx_stream_ssl_preread_module`
instead of `ngx.ssl.server_name()`.
On a **mixed** port each connection is terminated or passed through
according to `tls_passthrough` on the `stream_route` it matches (new field,
default `false`). That comes from etcd, so moving a service between the two
modes needs no gateway config change or restart.
#### Why the port mode itself is static
`ssl_preread on` and `listen ... ssl` are config-time directives with no
runtime switch, and on one server they are incompatible — measured: the
handshake completes, the preread phase then never sees a ClientHello and the
connection stalls. So a port is structurally either terminate or preread.
What *can* be dynamic is the choice per connection. The two behaviours move
into internal servers reachable over unix sockets; the preread phase of the
public listen matches the route and sets the `proxy_pass` target:
```nginx
server {
listen 9102; # no ssl
ssl_preread on;
proxy_protocol on; # carries the client across the
internal hop
access_log off;
set $stream_tls_target "";
preread_by_lua_block { apisix.stream_tls_route_phase("…_terminate_2",
"…_passthrough_2") }
proxy_pass $stream_tls_target;
}
server { # internal: terminates
listen unix:…/stls-t2.sock ssl proxy_protocol;
set_real_ip_from unix:;
ssl_certificate_by_lua_block { apisix.ssl_phase() }
preread_by_lua_block { apisix.stream_preread_phase(nil, true) }
proxy_pass apisix_backend;
}
server { # internal: passes through
listen unix:…/stls-p2.sock proxy_protocol;
set_real_ip_from unix:;
ssl_preread on; # the ClientHello was inspected, never
consumed
preread_by_lua_block { apisix.stream_preread_phase(true, true) }
proxy_pass apisix_backend;
}
```
Plugins and the log phase run **once**, on whichever internal server owns
the connection; the outer block matches the route and does nothing else. The
client address crosses the hop in a PROXY protocol header and is restored with
`set_real_ip_from unix:`; `$server_addr`/`$server_port` are restored from
`$proxy_protocol_server_addr`/`_port`, so a `stream_route` carrying
`server_addr`/`server_port` matches the same on both sides of the hop.
Only mixed ports pay that hop. `tls: true` and `tls_passthrough: true` ports
keep their direct path.
#### Consequences of not terminating
These are inherent to passthrough, not implementation limits:
- payload-inspecting stream plugins (`mqtt-proxy`, `xrpc`, `redis`) and
gateway mTLS do not apply; mTLS moves to the backend;
- an upstream with `"scheme": "tls"` is refused with a `503` and a
diagnostic rather than silently double-wrapping — a second handshake would send
the client's ClientHello to the upstream as payload;
- `apisix_stream_metrics_zone` counts nginx sessions and has no per-server
switch, so a mixed port counts each client connection twice.
It is an explicit opt-in per listen port, never a default.
### Changes
| File | Change |
|---|---|
| `apisix/cli/schema.lua` | per-listen `tls_passthrough` |
| `apisix/cli/ops.lua` | groups stream listens by TLS mode (plain /
passthrough / mixed) × `proxy_protocol_to_upstream`; rejects two listens on one
address that would need different server blocks; names and length-checks the
internal sockets |
| `apisix/cli/ngx_tpl.lua` | renders `ssl_preread on`, and for mixed listens
the outer preread block plus the two internal servers and their upstreams |
| `apisix/ssl.lua` | `server_name(clienthello, preread)` — reads
`$ssl_preread_server_name` when nothing was terminated locally, normalising
`""` to `nil` so `fallback_sni` still applies to a no-SNI ClientHello |
| `apisix/init.lua` | `stream_preread_phase(tls_passthrough,
behind_mixed_hop)` skips `verify_tls_client` and restores the listen address;
new `stream_tls_route_phase()` for the outer block |
| `apisix/schema_def.lua` | `stream_route.tls_passthrough` |
| `apisix/stream/router/ip_port.lua` | passes the mode through; no routing
logic change |
| `apisix/upstream.lua` | `scheme: tls` under passthrough returns 503, ahead
of the traffic-split short circuit so an inline upstream from that plugin is
covered too |
| `t/APISIX.pm` | two test-framework fixes needed by the new `.t`, see below
|
Grouping listens by their TLS mode extends the existing
`proxy_protocol_to_upstream` split rather than replacing it;
`t/cli/test_stream_proxy_protocol.sh` (11 assertions on that grouping) stays
green.
#### Test framework
Both fixes are required by the new `.t` and both were latent bugs:
- a bare `--- stream_tls_verify` section carries an empty value, which Perl
treats as false, so the springboard called `sslhandshake` with verification
disabled. With `defined`, a stream TLS test can actually verify the served
chain. Checked both directions: pointing `custom_trusted_cert` at a certificate
that does not sign the backend's now fails the case, pointing it at the
backend's CA passes.
- a block that sets `stream_server_config` alongside `stream_tls_request`
rendered a second, conflicting `stream {}` block, because on that path the
stream block lives in the main config.
### Tests
`t/stream-node/tls-passthrough.t` — 9 cases, 27 assertions:
1. the prereaded SNI selects an upstream, which terminates the handshake
itself. The passthrough listen holds no certificate at all, the backend
presents one the gateway has no copy of, and the client verifies that chain
against its CA — point the trust store at the gateway's own certificate instead
and the case fails, so a terminating gateway could not satisfy it;
2. a second SNI on the same listen routes independently;
3. a ClientHello with no SNI falls back to `ssl.fallback_sni` instead of
matching on `""`;
4. the mixed-mode phase picks its target from
`stream_route.tls_passthrough`. The two internal upstreams are cross-mapped
against the routes' own upstreams, so the backend that answers can only be
explained by the target the phase picked;
5. an upstream with `scheme: tls` is refused, with the diagnostic.
**13 of the 27 assertions fail on the parent commit** (with only the
`t/APISIX.pm` framework fixes applied, so the failures are attributable to the
feature and not to the harness), and all 27 pass with the change.
`t/cli/test_stream_tls_passthrough.sh` — 8 assertions, only what a `.t`
structurally cannot reach. Test::Nginx builds its own `nginx.conf` and never
runs `apisix/cli/ops.lua` or `ngx_tpl.lua`, so the half of this feature that
*is* config generation has no other home:
1. `tls` + `tls_passthrough` renders a mixed port — plain listen,
`ssl_preread on`, `proxy_pass $stream_tls_target`, PROXY protocol out,
`access_log off`; the internal terminating server has the certificate phases
and `set_real_ip_from`, the internal passthrough server prereads and terminates
nothing;
2. two listens on one address needing different server blocks are rejected,
including a port range overlapping a single port;
3. a pure passthrough listen renders without `ssl`, with `ssl_preread on`,
no termination or re-encryption;
4. terminating and plain listens keep their own block, and `ssl_preread`
does not leak into them; a pure passthrough listen pays no mixed-mode hop and
creates no internal sockets;
5. passthrough combines with `proxy_protocol_to_upstream`;
6. mixed mode end to end on a real generated config: one port,
`admin.apisix.dev` reaches a TLS backend untouched (backend's own certificate),
`test.com` is terminated by the gateway and proxied plaintext to a plain
backend. Both routes are pinned to `server_port: 9100`, which only matches if
the router reads the address the PROXY header carried rather than the unix
socket's own;
7. an `ip-restriction` whitelist on the terminating route passes and a
blacklist denies — the internal hop reports the real client address, not
`unix:`.
`t/stream-node/` and `t/stream-plugin/` were run file by file against the
merge base with a clean etcd between runs: identical results on both sides,
plus the new file passing.
#### Which issue(s) this PR fixes:
Ported from api7/api7-ee-3-gateway#2173, where it has already been reviewed.
### Checklist
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change
- [x] I have verified that this change is backward compatible (If not,
please discuss on the [APISIX mailing
list](https://github.com/apache/apisix/tree/master#community) first)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
