This is an automated email from the ASF dual-hosted git repository.
Alanxtl pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/dubbo-go.git
The following commit(s) were added to refs/heads/develop by this push:
new 630e2839f fix(triple): stop HTTP/2/HTTP/3 startup from hanging Run
(#3645)
630e2839f is described below
commit 630e2839fe8cf043ab8640a1d652e427c824997f
Author: Li Zining <[email protected]>
AuthorDate: Sat Aug 22 21:58:49 2026 +0800
fix(triple): stop HTTP/2/HTTP/3 startup from hanging Run (#3645)
* fix(triple): pre-bind TCP and UDP sockets and guard the dual-protocol
startup path
startHttp2AndHttp3 launched HTTP/2 and HTTP/3 inside a plain
errgroup.Group{} whose Wait() only returns after ALL goroutines finish.
When either the TCP or the UDP port was occupied, the failing side
returned its bind error while the successful side kept listening, so
Wait() never returned and Run hung forever, leaving the successful
listener unclosed.
Bind both sockets up front — net.Listen for TCP (HTTP/2) and
net.ListenPacket for UDP (HTTP/3) — and return the bind error immediately
when either fails. Validate TLS readiness (Certificates / GetCertificate /
GetConfigForClient) before binding so ServeTLS cannot fail inside a
goroutine, and when either Serve exits abnormally, close the other side
explicitly so eg.Wait does not block on the still-listening socket. The
servers are then started concurrently on the pre-bound sockets via
ServeTLS and http3 Serve.
Fixes: #3640
Signed-off-by: lizining <[email protected]>
* test(triple): add regression tests for dual-protocol startup and shutdown
Cover the startup and shutdown paths of the dual-protocol server, one
test per scenario:
- Run fails fast with a bind error when the TCP or the UDP port is occupied
- Run returns an error when TLS has no usable certificate
- Run returns the Serve error and releases the other side's port when
either Serve exits abnormally after a successful bind
- the TCP and UDP ports can be rebound after Stop and GracefulStop
- concurrent start-and-stop cycles pass under the race detector, guarding
the uatomic.Pointer server fields against regression
Signed-off-by: lizining <[email protected]>
* fix(triple): abort dual-protocol startup when Stop lands during bind
Stop and GracefulStop only close the servers already stored in the
uatomic.Pointer fields. During the pre-bind and store steps of
startHttp2AndHttp3 both fields are still nil, so a concurrent Stop
found nothing to close, returned success, and startup went on to serve —
the caller believed the server was stopped while it kept listening.
Every Stop and GracefulStop now increments a stop count that startup
snapshots before binding; the startup aborts before serving when the
count changed. The deferred closes then release the pre-bound sockets.
The count (instead of a boolean flag) keeps repeated start/stop cycles
working, and atomic loads make the handoff race-free.
Fixes: #3640
Signed-off-by: lizining <[email protected]>
* test(triple): cover Stop landing during the dual-protocol startup window
Inject Stop from the netListen hook, right after the TCP socket is
pre-bound while both server fields are still nil. Run must then abort
instead of serving: the tests assert that Run returns without serving
(no connection can be established), the TCP and UDP ports can be
rebound, and no listener leaks. They fail before the fix (Run hangs)
and pass after it.
Signed-off-by: lizining <[email protected]>
* test(triple): cover Stop landing before Run executes
Register the epoch synchronously, Stop, then release Run: the entry
checkpoint must abort without binding any socket. Cover both the
dual-protocol and the single-protocol path, and assert the TCP port can
still be rebound so no listener leaks. These tests hang before the fix
and pass after it.
Signed-off-by: lizining <[email protected]>
* fix(triple): abort server startup when Stop lands before Run
The startup epoch was snapshotted inside the transport goroutine, so a
Stop that completed before the goroutine read the stop count was absorbed:
Run loaded the already-incremented value as its new baseline and kept
listening, so the first Stop returned while the server still served.
Register the epoch synchronously in startTransport before the goroutine
launches, pass it into Run, and check it at Run's entry before any socket
is bound or served. The entry checkpoint guards every protocol path, so a
Stop landing after registration but before Run aborts the startup.
Fixes: #3640
Signed-off-by: lizining <[email protected]>
* test(triple): build the test TLS config before the transport goroutine
TestServer_HTTP2AndHTTP3_StopBeforeRunAbortsStartup called
newTestTLSConfig(t) inside the transport goroutine, but the helper
asserts via t, which violates testifylint's go-require rule and made
make lint report one issue. Build the config in the test goroutine and
pass the value into the transport goroutine; behavior is unchanged.
Signed-off-by: lizining <[email protected]>
* fix(triple): re-check the epoch after publishing the server
The single-protocol paths had no second epoch check between publishing
the server and listening, so a Stop landing there returned success while
the startup kept listening. Re-check the epoch after the store in
startHttp2 and startHttp3, mirroring the dual-protocol path.
Fixes: #3640
Signed-off-by: lizining <[email protected]>
* test(triple): deterministically cover the single-protocol checkpoints
Add StopBetweenCheckpointAndServe tests that Stop first, then call
startHttp2 and startHttp3 directly with a stale epoch so the post-store
checkpoint aborts deterministically, and assert the port stays free.
Signed-off-by: lizining <[email protected]>
* fix(triple): restore two-argument Run and hide the startup epoch
The exported Run(callProtocol, tlsConf) was widened to three arguments
to carry the startup epoch, breaking source compatibility for pre-3640
consumers.
Keep Run two-argument: it snapshots the epoch itself and delegates to a
private run(callProtocol, tlsConf, epoch). A new exported Start snapshots
the epoch synchronously before the transport goroutine runs, which the
adaptation layer now calls instead of snapshotting by hand.
Fixes: #3640
* test(triple): guard two-argument Run signature
Add an external-package test that calls Run with the pre-3640
two-argument shape and serves a real HTTP/2 start/stop cycle. The
external package sees only the exported API, so a future signature
widening fails compilation here.
---------
Signed-off-by: lizining <[email protected]>
---
protocol/triple/active_notify_test.go | 4 +-
protocol/triple/server.go | 10 +-
protocol/triple/triple_protocol/server.go | 115 ++++-
protocol/triple/triple_protocol/server_ext_test.go | 86 ++++
.../triple_protocol/server_lifecycle_test.go | 513 ++++++++++++++++++++-
protocol/triple/triple_protocol/server_test.go | 4 +-
6 files changed, 709 insertions(+), 23 deletions(-)
diff --git a/protocol/triple/active_notify_test.go
b/protocol/triple/active_notify_test.go
index 6ffcf7c03..2bcbeabeb 100644
--- a/protocol/triple/active_notify_test.go
+++ b/protocol/triple/active_notify_test.go
@@ -121,9 +121,7 @@ func TestTripleHealthWatchEmitsClosingEvent(t *testing.T) {
)
require.NoError(t, err)
- go func() {
- _ = server.Run(constant.CallHTTP2, nil)
- }()
+ server.Start(constant.CallHTTP2, nil)
defer func() {
_ = server.Stop()
}()
diff --git a/protocol/triple/server.go b/protocol/triple/server.go
index 9200263a3..cb894b114 100644
--- a/protocol/triple/server.go
+++ b/protocol/triple/server.go
@@ -232,12 +232,10 @@ func (s *Server) startTransport(callProtocol string,
tlsConf *tls.Config) {
return
}
s.transportStarted = true
-
- go func() {
- if runErr := s.triServer.Run(callProtocol, tlsConf); runErr !=
nil {
- logger.Errorf("[Triple][Server] server serve failed,
err=%v", runErr)
- }
- }()
+ // Start snapshots the startup epoch synchronously before the transport
+ // goroutine runs, so run's checkpoint detects a Stop that completes
+ // before the goroutine executes.
+ s.triServer.Start(callProtocol, tlsConf)
}
func (s *Server) registerServiceHandlers(invoker base.Invoker, info
*common.ServiceInfo, handlerOpts []tri.HandlerOption) {
diff --git a/protocol/triple/triple_protocol/server.go
b/protocol/triple/triple_protocol/server.go
index 41b12edb8..531069f50 100644
--- a/protocol/triple/triple_protocol/server.go
+++ b/protocol/triple/triple_protocol/server.go
@@ -21,6 +21,7 @@ import (
"context"
"crypto/tls"
"fmt"
+ "net"
"net/http"
)
@@ -47,12 +48,20 @@ import (
"dubbo.apache.org/dubbo-go/v3/protocol/triple/openapi"
)
+// netListen and netListenPacket create the pre-bound sockets in
+// startHttp2AndHttp3. Tests override them to simulate Serve failures.
+var (
+ netListen = net.Listen
+ netListenPacket = net.ListenPacket
+)
+
type Server struct {
addr string
mux *methodRouteMux
handlers map[string]*Handler
httpSrv uatomic.Pointer[http.Server]
http3Srv uatomic.Pointer[http3.Server]
+ stopCount uatomic.Uint32
tripleConfig *global.TripleConfig // Configuration for the triple
protocol
openapiIntegration *openapi.OpenAPIIntegration
}
@@ -184,27 +193,69 @@ func (s *Server) SetFallbackHTTPHandler(h http.Handler) {
s.mux.SetFallbackHandler(h)
}
+// Start starts the server for the given protocol without blocking. It
+// snapshots the startup epoch synchronously before the transport goroutine
+// runs, so run's checkpoint detects a Stop that completes before the
+// goroutine executes. Serve errors are logged; use Run when the error must
+// be returned synchronously.
+func (s *Server) Start(callProtocol string, tlsConf *tls.Config) {
+ epoch := s.beginStart()
+ go func() {
+ if runErr := s.run(callProtocol, tlsConf, epoch); runErr != nil
{
+ logger.Errorf("[Triple][Server] server serve failed,
err=%v", runErr)
+ }
+ }()
+}
+
+// beginStart snapshots the startup epoch before the transport goroutine
+// runs. It must be called synchronously on the start path so run's
+// checkpoint can detect a Stop that completes before run reads the counter.
+func (s *Server) beginStart() uint32 {
+ return s.stopCount.Load()
+}
+
+// Run starts the server for the given protocol and blocks until the server
+// is closed. It keeps the pre-3640 two-argument signature: the startup epoch
+// is snapshotted here, so a Stop that completes before this call executes
+// cannot be detected. Callers that need that guarantee use Start.
func (s *Server) Run(callProtocol string, tlsConf *tls.Config) error {
+ return s.run(callProtocol, tlsConf, s.stopCount.Load())
+}
+
+func (s *Server) run(callProtocol string, tlsConf *tls.Config, epoch uint32)
error {
+ // A Stop that completed after the synchronous epoch snapshot but before
+ // this checkpoint aborts the startup here, before any socket is bound
or
+ // served.
+ if s.stopCount.Load() != epoch {
+ return nil
+ }
+
// Support for starting HTTP/2 and HTTP/3 servers simultaneously.
switch callProtocol {
case constant.CallHTTP2:
- return s.startHttp2(tlsConf)
+ return s.startHttp2(tlsConf, epoch)
case constant.CallHTTP3:
- return s.startHttp3(tlsConf)
+ return s.startHttp3(tlsConf, epoch)
case constant.CallHTTP2AndHTTP3:
- return s.startHttp2AndHttp3(tlsConf)
+ return s.startHttp2AndHttp3(tlsConf, epoch)
default:
return fmt.Errorf("unsupported protocol: %s, only http2, http3,
or http2-and-http3 are supported", callProtocol)
}
}
-func (s *Server) startHttp2(tlsConf *tls.Config) error {
+func (s *Server) startHttp2(tlsConf *tls.Config, epoch uint32) error {
s.httpSrv.Store(&http.Server{
Addr: s.addr,
Handler: h2c.NewHandler(s.mux, &http2.Server{}),
TLSConfig: tlsConf,
})
+ // A Stop that landed after the entry checkpoint but before this server
+ // was published closed nothing; abort so no listener is served.
+ if s.stopCount.Load() != epoch {
+ return nil
+ }
+
logger.Debugf("[Triple][Server] triple HTTP/2 Server starting on %v",
s.addr)
srv := s.httpSrv.Load()
@@ -221,7 +272,7 @@ func (s *Server) startHttp2(tlsConf *tls.Config) error {
return nil
}
-func (s *Server) startHttp3(tlsConf *tls.Config) error {
+func (s *Server) startHttp3(tlsConf *tls.Config, epoch uint32) error {
if tlsConf == nil {
return fmt.Errorf("TRIPLE HTTP/3 Server must have TLS config,
but TLS config is nil")
}
@@ -246,6 +297,12 @@ func (s *Server) startHttp3(tlsConf *tls.Config) error {
QUICConfig: quicConfig,
})
+ // A Stop that landed after the entry checkpoint but before this server
+ // was published closed nothing; abort so no listener is served.
+ if s.stopCount.Load() != epoch {
+ return nil
+ }
+
logger.Debugf("[Triple][Server] triple HTTP/3 Server starting on %v",
s.addr)
err = s.http3Srv.Load().ListenAndServe()
@@ -255,7 +312,7 @@ func (s *Server) startHttp3(tlsConf *tls.Config) error {
return nil
}
-func (s *Server) startHttp2AndHttp3(tlsConf *tls.Config) error {
+func (s *Server) startHttp2AndHttp3(tlsConf *tls.Config, epoch uint32) error {
// Check if TLS config is provided for HTTP/3
if tlsConf == nil {
return fmt.Errorf("TRIPLE HTTP/2 and HTTP/3 Server must have
TLS config, but TLS config is nil")
@@ -271,6 +328,28 @@ func (s *Server) startHttp2AndHttp3(tlsConf *tls.Config)
error {
return err
}
+ if len(tlsConf.Certificates) == 0 &&
+ tlsConf.GetCertificate == nil &&
+ tlsConf.GetConfigForClient == nil {
+ return fmt.Errorf("TRIPLE HTTP/2 and HTTP/3 Server must have a
TLS certificate configured, but none of
Certificates/GetCertificate/GetConfigForClient is set")
+ }
+
+ // Pre-bind the TCP (HTTP/2) listener before serving any request:
+ // fail fast with the bind error when the port is occupied.
+ tcpLn, err := netListen("tcp", s.addr)
+ if err != nil {
+ return fmt.Errorf("HTTP/2 server bind error: %w", err)
+ }
+ defer tcpLn.Close()
+
+ // Pre-bind the UDP (HTTP/3) socket as well; on failure close the
+ // already-bound TCP listener and return, no request has been served
yet.
+ udpConn, err := netListenPacket("udp", s.addr)
+ if err != nil {
+ return fmt.Errorf("HTTP/3 server bind error: %w", err)
+ }
+ defer udpConn.Close()
+
// Start HTTP/3 server first to get its configuration
s.http3Srv.Store(&http3.Server{
Addr: s.addr,
@@ -293,14 +372,23 @@ func (s *Server) startHttp2AndHttp3(tlsConf *tls.Config)
error {
TLSConfig: tlsConf,
})
+ // A Stop during the bind or store steps closed nothing; abort so the
+ // deferred closes release the sockets.
+ if s.stopCount.Load() != epoch {
+ return nil
+ }
+
logger.Debugf("[Triple][Server] triple HTTP/2 and HTTP/3 Server
starting on %v", s.addr)
// Use errgroup to manage concurrent server startup
- eg := &errgroup.Group{}
+ eg, _ := errgroup.WithContext(context.Background())
// Start HTTP/2 server in a goroutine
eg.Go(func() error {
- if err := s.httpSrv.Load().ListenAndServeTLS("", ""); err !=
nil && err != http.ErrServerClosed {
+ if err := s.httpSrv.Load().ServeTLS(tcpLn, "", ""); err != nil
&& err != http.ErrServerClosed {
+ // Close the HTTP/3 server so its Serve call returns and
+ // eg.Wait does not block on the still-listening UDP
socket.
+ _ = s.http3Srv.Load().Close()
return fmt.Errorf("HTTP/2 server error: %w", err)
}
return nil
@@ -308,7 +396,10 @@ func (s *Server) startHttp2AndHttp3(tlsConf *tls.Config)
error {
// Start HTTP/3 server in a goroutine
eg.Go(func() error {
- if err := s.http3Srv.Load().ListenAndServe(); err != nil && err
!= http.ErrServerClosed {
+ if err := s.http3Srv.Load().Serve(udpConn); err != nil && err
!= http.ErrServerClosed {
+ // Close the HTTP/2 server so its Serve call returns and
+ // eg.Wait does not block on the still-listening TCP
listener.
+ _ = s.httpSrv.Load().Close()
return fmt.Errorf("HTTP/3 server error: %w", err)
}
return nil
@@ -320,6 +411,9 @@ func (s *Server) startHttp2AndHttp3(tlsConf *tls.Config)
error {
// Stop the Triple server for both HTTP/2 and HTTP/3.
func (s *Server) Stop() error {
+ // Record the stop first so an in-flight startup aborts at its
checkpoint.
+ s.stopCount.Add(1)
+
eg, _ := errgroup.WithContext(context.Background())
// stop HTTP server
@@ -348,6 +442,9 @@ func (s *Server) Stop() error {
// Gracefulstop shutdown the Triple server for both HTTP/2 and HTTP/3
gracefully.
func (s *Server) GracefulStop(ctx context.Context) error {
+ // Record the stop first so an in-flight startup aborts at its
checkpoint.
+ s.stopCount.Add(1)
+
eg, ctx := errgroup.WithContext(ctx)
// shutdown HTTP server
diff --git a/protocol/triple/triple_protocol/server_ext_test.go
b/protocol/triple/triple_protocol/server_ext_test.go
new file mode 100644
index 000000000..a4d94a3d7
--- /dev/null
+++ b/protocol/triple/triple_protocol/server_ext_test.go
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package triple_protocol_test
+
+import (
+ "net"
+ "testing"
+ "time"
+)
+
+import (
+ "dubbo.apache.org/dubbo-go/v3/common/constant"
+ triple "dubbo.apache.org/dubbo-go/v3/protocol/triple/triple_protocol"
+
"dubbo.apache.org/dubbo-go/v3/protocol/triple/triple_protocol/internal/assert"
+)
+
+// TestServer_Run_KeepsTwoArgumentSignature verifies that the exported Run
+// keeps its pre-3640 two-argument signature: the exact call shape a
+// pre-3640 consumer writes still compiles and serves, then stops cleanly.
+func TestServer_Run_KeepsTwoArgumentSignature(t *testing.T) {
+ addr := freeAddr(t)
+ srv := triple.NewServer(addr, nil)
+ errCh := make(chan error, 1)
+ go func() {
+ errCh <- srv.Run(constant.CallHTTP2, nil)
+ }()
+
+ // Wait until the listener is up so the stop below closes a serving
+ // server instead of racing the startup.
+ waitForListener(t, addr)
+
+ assert.Nil(t, srv.Stop())
+ select {
+ case err := <-errCh:
+ // A clean Stop is the normal end of the single-protocol path:
the
+ // shutdown filter suppresses http.ErrServerClosed, so Run
returns nil.
+ assert.Nil(t, err)
+ case <-time.After(5 * time.Second):
+ t.Fatalf("server did not exit within 5s")
+ }
+}
+
+// freeAddr returns a free TCP address on loopback for the test server.
+func freeAddr(t *testing.T) string {
+ t.Helper()
+
+ l, err := net.Listen("tcp", "127.0.0.1:0")
+ assert.Nil(t, err)
+ addr := l.Addr().String()
+ assert.Nil(t, l.Close())
+ return addr
+}
+
+// waitForListener polls until the address accepts TCP connections or the
+// deadline expires.
+func waitForListener(t *testing.T, addr string) {
+ t.Helper()
+
+ deadline := time.Now().Add(3 * time.Second)
+ for {
+ conn, err := net.DialTimeout("tcp", addr, 100*time.Millisecond)
+ if err == nil {
+ _ = conn.Close()
+ return
+ }
+ if time.Now().After(deadline) {
+ t.Fatalf("server did not accept connections within 3s:
%v", err)
+ }
+ time.Sleep(20 * time.Millisecond)
+ }
+}
diff --git a/protocol/triple/triple_protocol/server_lifecycle_test.go
b/protocol/triple/triple_protocol/server_lifecycle_test.go
index 25b7a4442..2043c46e6 100644
--- a/protocol/triple/triple_protocol/server_lifecycle_test.go
+++ b/protocol/triple/triple_protocol/server_lifecycle_test.go
@@ -26,8 +26,10 @@ import (
"crypto/x509"
"crypto/x509/pkix"
"errors"
+ "fmt"
"math/big"
"net"
+ "sync"
"syscall"
"testing"
"time"
@@ -89,12 +91,13 @@ func getFreeAddr(t *testing.T) string {
}
// runServer starts the server in a goroutine and returns the channel that
-// receives the error returned by Run. ListenAndServe is blocking, so the
-// server must always be started this way in tests.
+// receives the error returned by the startup. ListenAndServe is blocking, so
+// the server must always be started this way in tests.
func runServer(srv *Server, protocol string, tlsConf *tls.Config) chan error {
errCh := make(chan error, 1)
+ epoch := srv.beginStart()
go func() {
- errCh <- srv.Run(protocol, tlsConf)
+ errCh <- srv.run(protocol, tlsConf, epoch)
}()
return errCh
}
@@ -263,6 +266,88 @@ func TestServer_HTTP2AndHTTP3_StartAndGracefulStop(t
*testing.T) {
require.NoError(t, waitForServerExit(t, errCh, 5*time.Second))
}
+// TestServer_HTTP2AndHTTP3_StopReleasesTCPPort verifies that Stop releases
+// the TCP socket, so the same address can be bound again afterwards.
+func TestServer_HTTP2AndHTTP3_StopReleasesTCPPort(t *testing.T) {
+ cfg := &global.TripleConfig{
+ Http3: &global.Http3Config{Enable: true},
+ }
+ srv, errCh := startTestServer(t, constant.CallHTTP2AndHTTP3, cfg,
newTestTLSConfig(t))
+ waitForTCPReady(t, srv.addr, 3*time.Second)
+ waitForHTTP3Stored(t, srv)
+
+ require.NoError(t, srv.Stop())
+ require.NoError(t, waitForServerExit(t, errCh, 5*time.Second))
+
+ // The TCP socket must be released once Run has returned.
+ tcpLn, err := net.Listen("tcp", srv.addr)
+ require.NoError(t, err)
+ defer tcpLn.Close()
+}
+
+// TestServer_HTTP2AndHTTP3_StopReleasesUDPPort verifies that Stop releases
+// the UDP socket, so the same address can be bound again afterwards.
+func TestServer_HTTP2AndHTTP3_StopReleasesUDPPort(t *testing.T) {
+ cfg := &global.TripleConfig{
+ Http3: &global.Http3Config{Enable: true},
+ }
+ srv, errCh := startTestServer(t, constant.CallHTTP2AndHTTP3, cfg,
newTestTLSConfig(t))
+ waitForTCPReady(t, srv.addr, 3*time.Second)
+ waitForHTTP3Stored(t, srv)
+
+ require.NoError(t, srv.Stop())
+ require.NoError(t, waitForServerExit(t, errCh, 5*time.Second))
+
+ // The UDP socket must be released once Run has returned.
+ udpConn, err := net.ListenPacket("udp", srv.addr)
+ require.NoError(t, err)
+ defer udpConn.Close()
+}
+
+// TestServer_HTTP2AndHTTP3_GracefulStopReleasesTCPPort verifies that
+// GracefulStop releases the TCP socket, so the same address can be bound again
+// afterwards.
+func TestServer_HTTP2AndHTTP3_GracefulStopReleasesTCPPort(t *testing.T) {
+ cfg := &global.TripleConfig{
+ Http3: &global.Http3Config{Enable: true},
+ }
+ srv, errCh := startTestServer(t, constant.CallHTTP2AndHTTP3, cfg,
newTestTLSConfig(t))
+ waitForTCPReady(t, srv.addr, 3*time.Second)
+ waitForHTTP3Stored(t, srv)
+
+ graceCtx, cancel := context.WithTimeout(context.Background(),
constant.DefaultGracefulShutdownTimeout)
+ defer cancel()
+ require.NoError(t, srv.GracefulStop(graceCtx))
+ require.NoError(t, waitForServerExit(t, errCh, 5*time.Second))
+
+ // The TCP socket must be released once Run has returned.
+ tcpLn, err := net.Listen("tcp", srv.addr)
+ require.NoError(t, err)
+ defer tcpLn.Close()
+}
+
+// TestServer_HTTP2AndHTTP3_GracefulStopReleasesUDPPort verifies that
+// GracefulStop releases the UDP socket, so the same address can be bound again
+// afterwards.
+func TestServer_HTTP2AndHTTP3_GracefulStopReleasesUDPPort(t *testing.T) {
+ cfg := &global.TripleConfig{
+ Http3: &global.Http3Config{Enable: true},
+ }
+ srv, errCh := startTestServer(t, constant.CallHTTP2AndHTTP3, cfg,
newTestTLSConfig(t))
+ waitForTCPReady(t, srv.addr, 3*time.Second)
+ waitForHTTP3Stored(t, srv)
+
+ graceCtx, cancel := context.WithTimeout(context.Background(),
constant.DefaultGracefulShutdownTimeout)
+ defer cancel()
+ require.NoError(t, srv.GracefulStop(graceCtx))
+ require.NoError(t, waitForServerExit(t, errCh, 5*time.Second))
+
+ // The UDP socket must be released once Run has returned.
+ udpConn, err := net.ListenPacket("udp", srv.addr)
+ require.NoError(t, err)
+ defer udpConn.Close()
+}
+
func TestServer_StopBeforeStart(t *testing.T) {
srv := NewServer(getFreeAddr(t), nil)
require.NoError(t, srv.Stop())
@@ -275,6 +360,117 @@ func TestServer_GracefulStopBeforeStart(t *testing.T) {
require.NoError(t, srv.GracefulStop(graceCtx))
}
+// TestServer_HTTP2AndHTTP3_StopBeforeRunAbortsStartup verifies that a Stop
+// completed before run executes is still detected: beginStart registers the
+// epoch synchronously, Stop increments the counter, and run's checkpoint
+// aborts the startup without binding any socket.
+func TestServer_HTTP2AndHTTP3_StopBeforeRunAbortsStartup(t *testing.T) {
+ addr := getFreeAddr(t)
+ srv := NewServer(addr, nil)
+
+ // Register the startup epoch, then Stop before run gets to execute: the
+ // checkpoint must abort instead of serving on the pre-bound sockets.
+ epoch := srv.beginStart()
+ require.NoError(t, srv.Stop())
+
+ // newTestTLSConfig uses t for assertions, so build it in the test
+ // goroutine and only pass the value into the transport goroutine.
+ tlsConf := newTestTLSConfig(t)
+ errCh := make(chan error, 1)
+ go func() {
+ errCh <- srv.run(constant.CallHTTP2AndHTTP3, tlsConf, epoch)
+ }()
+
+ select {
+ case err := <-errCh:
+ require.NoError(t, err)
+ case <-time.After(3 * time.Second):
+ require.FailNow(t, "Run did not abort within 3s: a Stop
completed before Run must still be detected")
+ }
+
+ // The abort must not leave any listener behind: the TCP port is free.
+ tcpLn, err := net.Listen("tcp", addr)
+ require.NoError(t, err)
+ require.NoError(t, tcpLn.Close())
+}
+
+// TestServer_HTTP2_StopBeforeRunAbortsStartup verifies that run's checkpoint
+// guards the single-protocol path too: a Stop completed after beginStart but
+// before run executes aborts the HTTP/2 startup without binding any socket.
+func TestServer_HTTP2_StopBeforeRunAbortsStartup(t *testing.T) {
+ addr := getFreeAddr(t)
+ srv := NewServer(addr, nil)
+
+ epoch := srv.beginStart()
+ require.NoError(t, srv.Stop())
+
+ errCh := make(chan error, 1)
+ go func() {
+ errCh <- srv.run(constant.CallHTTP2, nil, epoch)
+ }()
+
+ select {
+ case err := <-errCh:
+ require.NoError(t, err)
+ case <-time.After(3 * time.Second):
+ require.FailNow(t, "Run did not abort within 3s: a Stop
completed before Run must still be detected")
+ }
+
+ tcpLn, err := net.Listen("tcp", addr)
+ require.NoError(t, err)
+ require.NoError(t, tcpLn.Close())
+}
+
+// TestServer_HTTP2_StopBetweenCheckpointAndServeAbortsStartup verifies that
+// startHttp2 re-checks the epoch after publishing the server: a Stop that
+// landed after the entry checkpoint aborts the HTTP/2 startup before the
+// listener serves, leaving the TCP port free.
+func TestServer_HTTP2_StopBetweenCheckpointAndServeAbortsStartup(t *testing.T)
{
+ addr := getFreeAddr(t)
+ srv := NewServer(addr, nil)
+
+ // Stop after the epoch snapshot, then start HTTP/2 with the stale
epoch:
+ // the checkpoint after the store must abort deterministically.
+ epoch := srv.beginStart()
+ require.NoError(t, srv.Stop())
+
+ err := srv.startHttp2(nil, epoch)
+ require.NoError(t, err)
+ // The server was published before the abort, so the per-protocol
+ // checkpoint, not the entry check, aborted the startup.
+ require.NotNil(t, srv.httpSrv.Load())
+
+ // The abort must not leave any listener behind: the TCP port is free.
+ tcpLn, err := net.Listen("tcp", addr)
+ require.NoError(t, err)
+ require.NoError(t, tcpLn.Close())
+}
+
+// TestServer_HTTP3_StopBetweenCheckpointAndServeAbortsStartup verifies that
+// startHttp3 re-checks the epoch after publishing the server: a Stop that
+// landed after the entry checkpoint aborts the HTTP/3 startup before the
+// listener serves, leaving the UDP port free.
+func TestServer_HTTP3_StopBetweenCheckpointAndServeAbortsStartup(t *testing.T)
{
+ addr := getFreeAddr(t)
+ srv := NewServer(addr, nil)
+
+ // Stop after the epoch snapshot, then start HTTP/3 with the stale
epoch:
+ // the checkpoint after the store must abort deterministically.
+ epoch := srv.beginStart()
+ require.NoError(t, srv.Stop())
+
+ err := srv.startHttp3(newTestTLSConfig(t), epoch)
+ require.NoError(t, err)
+ // The server was published before the abort, so the per-protocol
+ // checkpoint, not the entry check, aborted the startup.
+ require.NotNil(t, srv.http3Srv.Load())
+
+ // The abort must not leave any listener behind: the UDP port is free.
+ udpConn, err := net.ListenPacket("udp", addr)
+ require.NoError(t, err)
+ require.NoError(t, udpConn.Close())
+}
+
func TestServer_Run_HTTP3WithoutTLS(t *testing.T) {
srv := NewServer(getFreeAddr(t), nil)
err := srv.Run(constant.CallHTTP3, nil)
@@ -282,6 +478,52 @@ func TestServer_Run_HTTP3WithoutTLS(t *testing.T) {
assert.Contains(t, err.Error(), "must have TLS config")
}
+// TestServer_HTTP2AndHTTP3_StartFailsOnOccupiedTCP verifies that Run fails
+// fast with a bind error when the TCP port is occupied, instead of hanging on
+// errgroup.Wait while the UDP side keeps listening.
+func TestServer_HTTP2AndHTTP3_StartFailsOnOccupiedTCP(t *testing.T) {
+ addr := getFreeAddr(t)
+ tcpLn, err := net.Listen("tcp", addr)
+ require.NoError(t, err)
+ defer tcpLn.Close()
+
+ srv := NewServer(addr, nil)
+ errCh := runServer(srv, constant.CallHTTP2AndHTTP3, newTestTLSConfig(t))
+
+ select {
+ case err := <-errCh:
+ require.Error(t, err)
+ require.True(t, isAddrInUse(err), "expected address in use
error, got: %v", err)
+ case <-time.After(3 * time.Second):
+ require.FailNow(t, "Run did not return within 3s: it must fail
fast when the TCP port is occupied")
+ }
+}
+
+// TestServer_HTTP2AndHTTP3_StartFailsOnOccupiedUDP is the symmetric case:
+// Run must fail fast with a bind error when the UDP port is occupied, instead
+// of hanging while the HTTP/2 side keeps listening.
+func TestServer_HTTP2AndHTTP3_StartFailsOnOccupiedUDP(t *testing.T) {
+ udpConn, err := net.ListenPacket("udp", "127.0.0.1:0")
+ require.NoError(t, err)
+ defer udpConn.Close()
+ addr := udpConn.LocalAddr().String()
+
+ srv := NewServer(addr, nil)
+ errCh := runServer(srv, constant.CallHTTP2AndHTTP3, newTestTLSConfig(t))
+
+ select {
+ case err := <-errCh:
+ require.Error(t, err)
+ require.True(t, isAddrInUse(err), "expected address in use
error, got: %v", err)
+ // The bind error must come from the UDP pre-bind step; if it
comes
+ // from the TCP step instead, this test is not exercising the
case it
+ // claims to cover.
+ assert.Contains(t, err.Error(), "HTTP/3 server bind error")
+ case <-time.After(3 * time.Second):
+ require.FailNow(t, "Run did not return within 3s: it must fail
fast when the UDP port is occupied")
+ }
+}
+
func TestServer_Run_HTTP2AndHTTP3WithoutTLS(t *testing.T) {
srv := NewServer(getFreeAddr(t), nil)
err := srv.Run(constant.CallHTTP2AndHTTP3, nil)
@@ -289,6 +531,15 @@ func TestServer_Run_HTTP2AndHTTP3WithoutTLS(t *testing.T) {
assert.Contains(t, err.Error(), "must have TLS config")
}
+// TestServer_Run_HTTP2AndHTTP3_TLSWithoutCert verifies that Run fails when
+// the TLS config carries no certificate source.
+func TestServer_Run_HTTP2AndHTTP3_TLSWithoutCert(t *testing.T) {
+ srv := NewServer(getFreeAddr(t), nil)
+ err := srv.Run(constant.CallHTTP2AndHTTP3, &tls.Config{})
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "must have a TLS certificate
configured")
+}
+
func TestServer_RunUnsupportedProtocol(t *testing.T) {
srv := NewServer(getFreeAddr(t), nil)
err := srv.Run("tcp", nil)
@@ -334,6 +585,262 @@ func TestServer_RepeatedStartStop(t *testing.T) {
}
}
+// TestServer_HTTP2AndHTTP3_ServeFailsOnHTTP2Side verifies that when the
+// HTTP/2 Serve fails after a successful bind, the HTTP/3 side is closed and
+// Run returns the error instead of hanging on errgroup.Wait.
+func TestServer_HTTP2AndHTTP3_ServeFailsOnHTTP2Side(t *testing.T) {
+ oldListen := netListen
+ netListen = func(network, addr string) (net.Listener, error) {
+ ln, err := net.Listen(network, addr)
+ if err != nil {
+ return nil, err
+ }
+ defer ln.Close() // the returned listener stays closed so Serve
fails
+ return ln, nil
+ }
+ t.Cleanup(func() { netListen = oldListen })
+
+ srv := NewServer(getFreeAddr(t), nil)
+ errCh := runServer(srv, constant.CallHTTP2AndHTTP3, newTestTLSConfig(t))
+
+ select {
+ case err := <-errCh:
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "HTTP/2 server error")
+ case <-time.After(3 * time.Second):
+ require.FailNow(t, "Run did not return within 3s: it must fail
when the HTTP/2 Serve exits with an error")
+ }
+
+ // The HTTP/3 side must have been closed, so the UDP port can be
rebound.
+ udpConn, err := net.ListenPacket("udp", srv.addr)
+ require.NoError(t, err)
+ defer udpConn.Close()
+}
+
+// TestServer_HTTP2AndHTTP3_ServeFailsOnHTTP3Side verifies that when the
+// HTTP/3 Serve fails after a successful bind, the HTTP/2 side is closed and
+// Run returns the error instead of hanging on errgroup.Wait.
+func TestServer_HTTP2AndHTTP3_ServeFailsOnHTTP3Side(t *testing.T) {
+ oldListenPacket := netListenPacket
+ netListenPacket = func(network, addr string) (net.PacketConn, error) {
+ conn, err := net.ListenPacket(network, addr)
+ if err != nil {
+ return nil, err
+ }
+ defer conn.Close() // the returned conn stays closed so Serve
fails
+ return conn, nil
+ }
+ t.Cleanup(func() { netListenPacket = oldListenPacket })
+
+ srv := NewServer(getFreeAddr(t), nil)
+ errCh := runServer(srv, constant.CallHTTP2AndHTTP3, newTestTLSConfig(t))
+
+ select {
+ case err := <-errCh:
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "HTTP/3 server error")
+ case <-time.After(3 * time.Second):
+ require.FailNow(t, "Run did not return within 3s: it must fail
when the HTTP/3 Serve exits with an error")
+ }
+
+ // The HTTP/2 side must have been closed, so the TCP port can be
rebound.
+ tcpLn, err := net.Listen("tcp", srv.addr)
+ require.NoError(t, err)
+ defer tcpLn.Close()
+}
+
+// TestServer_HTTP2AndHTTP3_ConcurrentStartAndStop runs concurrent
+// start-and-stop cycles so the Store calls in startHttp2AndHttp3 overlap
+// with the Load calls in Stop, which the race detector reports if the
+// server fields are accessed without synchronization.
+func TestServer_HTTP2AndHTTP3_ConcurrentStartAndStop(t *testing.T) {
+ cfg := &global.TripleConfig{
+ Http3: &global.Http3Config{Enable: true},
+ }
+
+ const iterations = 10
+ errCh := make(chan error, 2*iterations)
+ var wg sync.WaitGroup
+ for range iterations {
+ wg.Go(func() {
+
+ srv := NewServer(getFreeAddr(t), cfg)
+ runCh := runServer(srv, constant.CallHTTP2AndHTTP3,
newTestTLSConfig(t))
+
+ // The first Stop may run while Run is still storing the
+ // servers, so it can close nothing. The second Stop
after the
+ // stores are visible always closes them.
+ _ = srv.Stop()
+ for i := 0; srv.http3Srv.Load() == nil && i < 100; i++ {
+ time.Sleep(time.Millisecond)
+ }
+ errCh <- srv.Stop()
+
+ select {
+ case err := <-runCh:
+ errCh <- err
+ case <-time.After(5 * time.Second):
+ errCh <- fmt.Errorf("Run did not return within
5s")
+ }
+ })
+ }
+
+ done := make(chan struct{})
+ go func() {
+ wg.Wait()
+ close(done)
+ }()
+ select {
+ case <-done:
+ case <-time.After(30 * time.Second):
+ require.FailNow(t, "concurrent start/stop did not finish within
30s")
+ }
+ close(errCh)
+ for err := range errCh {
+ require.NoError(t, err)
+ }
+}
+
+// TestServer_HTTP2AndHTTP3_ConcurrentStartAndGracefulStop runs concurrent
+// start-and-graceful-stop cycles so the Store calls in startHttp2AndHttp3
+// overlap with the Load calls in GracefulStop, which the race detector
+// reports if the server fields are accessed without synchronization.
+func TestServer_HTTP2AndHTTP3_ConcurrentStartAndGracefulStop(t *testing.T) {
+ cfg := &global.TripleConfig{
+ Http3: &global.Http3Config{Enable: true},
+ }
+
+ const iterations = 10
+ errCh := make(chan error, 2*iterations)
+ var wg sync.WaitGroup
+ for range iterations {
+ wg.Go(func() {
+
+ srv := NewServer(getFreeAddr(t), cfg)
+ runCh := runServer(srv, constant.CallHTTP2AndHTTP3,
newTestTLSConfig(t))
+
+ graceCtx, cancel :=
context.WithTimeout(context.Background(),
constant.DefaultGracefulShutdownTimeout)
+ defer cancel()
+
+ // The first GracefulStop may run while Run is still
storing
+ // the servers, so it can close nothing. The second
+ // GracefulStop after the stores are visible always
closes them.
+ _ = srv.GracefulStop(graceCtx)
+ for i := 0; srv.http3Srv.Load() == nil && i < 100; i++ {
+ time.Sleep(time.Millisecond)
+ }
+ errCh <- srv.GracefulStop(graceCtx)
+
+ select {
+ case err := <-runCh:
+ errCh <- err
+ case <-time.After(5 * time.Second):
+ errCh <- fmt.Errorf("Run did not return within
5s")
+ }
+ })
+ }
+
+ done := make(chan struct{})
+ go func() {
+ wg.Wait()
+ close(done)
+ }()
+ select {
+ case <-done:
+ case <-time.After(30 * time.Second):
+ require.FailNow(t, "concurrent start/graceful-stop did not
finish within 30s")
+ }
+ close(errCh)
+ for err := range errCh {
+ require.NoError(t, err)
+ }
+}
+
+// TestServer_HTTP2AndHTTP3_StopDuringStartup verifies that a Stop arriving
+// while the startup is still in progress cancels the startup: Run returns
+// without serving and the pre-bound sockets are released. The Stop is
+// injected from the pre-bind step, when both httpSrv and http3Srv are still
+// nil, so a Stop that only closes the stored servers would miss them.
+func TestServer_HTTP2AndHTTP3_StopDuringStartup(t *testing.T) {
+ cfg := &global.TripleConfig{
+ Http3: &global.Http3Config{Enable: true},
+ }
+ addr := getFreeAddr(t)
+ srv := NewServer(addr, cfg)
+
+ oldListen := netListen
+ netListen = func(network, addr string) (net.Listener, error) {
+ ln, err := net.Listen(network, addr)
+ if err != nil {
+ return nil, err
+ }
+ if err := srv.Stop(); err != nil {
+ _ = ln.Close() // release the bound socket before
returning the error
+ return nil, err
+ }
+ return ln, nil
+ }
+ t.Cleanup(func() { netListen = oldListen })
+
+ errCh := runServer(srv, constant.CallHTTP2AndHTTP3, newTestTLSConfig(t))
+
+ // Run must return: the startup was canceled instead of serving.
+ require.NoError(t, waitForServerExit(t, errCh, 3*time.Second))
+
+ // The pre-bound sockets must be released, otherwise the ports leak.
+ tcpLn, err := net.Listen("tcp", srv.addr)
+ require.NoError(t, err)
+ defer tcpLn.Close()
+
+ udpConn, err := net.ListenPacket("udp", srv.addr)
+ require.NoError(t, err)
+ defer udpConn.Close()
+}
+
+// TestServer_HTTP2AndHTTP3_StopDuringStartupDoesNotServe verifies that an
+// aborted startup leaves nothing listening: after Run returns, connecting
+// to the address fails and both sockets can be rebound.
+func TestServer_HTTP2AndHTTP3_StopDuringStartupDoesNotServe(t *testing.T) {
+ cfg := &global.TripleConfig{
+ Http3: &global.Http3Config{Enable: true},
+ }
+ addr := getFreeAddr(t)
+ srv := NewServer(addr, cfg)
+
+ oldListen := netListen
+ netListen = func(network, addr string) (net.Listener, error) {
+ ln, err := net.Listen(network, addr)
+ if err != nil {
+ return nil, err
+ }
+ if err := srv.Stop(); err != nil {
+ _ = ln.Close() // release the bound socket before
returning the error
+ return nil, err
+ }
+ return ln, nil
+ }
+ t.Cleanup(func() { netListen = oldListen })
+
+ errCh := runServer(srv, constant.CallHTTP2AndHTTP3, newTestTLSConfig(t))
+ require.NoError(t, waitForServerExit(t, errCh, 3*time.Second))
+
+ // The aborted startup must not serve: a connection attempt fails.
+ conn, err := net.DialTimeout("tcp", addr, 200*time.Millisecond)
+ require.Error(t, err)
+ if conn != nil {
+ _ = conn.Close()
+ }
+
+ // Both sockets must be free to rebind.
+ tcpLn, err := net.Listen("tcp", addr)
+ require.NoError(t, err)
+ defer tcpLn.Close()
+
+ udpConn, err := net.ListenPacket("udp", addr)
+ require.NoError(t, err)
+ defer udpConn.Close()
+}
+
// TestServerRunReturnsBindErrorWhenPortInUse verifies that Run propagates a
// genuine serve error (here a TCP port conflict) instead of swallowing it
// together with http.ErrServerClosed. It guards the boundary of the shutdown
diff --git a/protocol/triple/triple_protocol/server_test.go
b/protocol/triple/triple_protocol/server_test.go
index c9ef72dd5..e43f14758 100644
--- a/protocol/triple/triple_protocol/server_test.go
+++ b/protocol/triple/triple_protocol/server_test.go
@@ -110,7 +110,7 @@ func TestServer_HTTP3PathsUseQUICConfigHelper(t *testing.T)
{
},
})
- err := srv.startHttp3(&tls.Config{})
+ err := srv.startHttp3(&tls.Config{}, srv.beginStart())
require.Error(t, err)
require.ErrorContains(t, err, "keep-alive-period")
assert.Nil(t, srv.http3Srv.Load())
@@ -123,7 +123,7 @@ func TestServer_HTTP3PathsUseQUICConfigHelper(t *testing.T)
{
},
})
- err := srv.startHttp2AndHttp3(&tls.Config{})
+ err := srv.startHttp2AndHttp3(&tls.Config{}, srv.beginStart())
require.Error(t, err)
require.ErrorContains(t, err, "max-idle-timeout")
assert.Nil(t, srv.http3Srv.Load())
