This is an automated email from the ASF dual-hosted git repository.
Alanxtl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/dubbo-website.git
The following commit(s) were added to refs/heads/master by this push:
new 93fc40783ae docs(golang-sdk): improve graceful shutdown flow and
verification docs (#3224)
93fc40783ae is described below
commit 93fc40783ae610afe2e9bd1de7ae634e7f64e4a1
Author: XiaoFei <[email protected]>
AuthorDate: Thu Aug 13 09:13:24 2026 +0800
docs(golang-sdk): improve graceful shutdown flow and verification docs
(#3224)
* docs(golang-sdk): improve graceful shutdown flow and verification docs
* docs(golang-sdk): fix graceful shutdown flow and timeout example
* docs: add graceful shutdown new API examples
---
.../tutorial/deployment/graceful_shutdown.md | 319 ++++++++++++++++++--
.../tutorial/deployment/graceful_shutdown.md | 327 +++++++++++++++++++--
2 files changed, 587 insertions(+), 59 deletions(-)
diff --git
a/content/en/overview/mannual/golang-sdk/tutorial/deployment/graceful_shutdown.md
b/content/en/overview/mannual/golang-sdk/tutorial/deployment/graceful_shutdown.md
index 10969f744ec..60411ba8b29 100644
---
a/content/en/overview/mannual/golang-sdk/tutorial/deployment/graceful_shutdown.md
+++
b/content/en/overview/mannual/golang-sdk/tutorial/deployment/graceful_shutdown.md
@@ -13,49 +13,304 @@ type: docs
Sample source: <a
href="https://github.com/apache/dubbo-go-samples/tree/main/graceful_shutdown"
target="_blank">dubbo-go-samples/graceful_shutdown</a>.
-## Background
+## 1. Background
-In a stable production environment, container scheduling is fully managed by
k8s, and microservice governance is maintained and managed by the service
framework or operations personnel. In scenarios like releasing a new version or
scaling up/down, old container instances will be terminated and replaced with
new ones. If this replacement process is not handled properly in high-traffic
online production environments, it can lead to a large number of erroneous
requests in a short time, trig [...]
+During version releases, scaling, or node migration, Kubernetes, systemd, or
an operations platform stops old instances and starts new ones. If an old
instance exits immediately, the registry, client-side invoker cache, and
upstream traffic may be briefly inconsistent: the registry may not have
propagated the delete event, and upstream requests may still be running. For
high-traffic services, this can cause connection resets, request failures,
timeout retries, and alert storms.
-## Feature Description
+Dubbo-go graceful shutdown drains traffic before the process exits. It is
designed to ensure that:
-In a complete RPC call process, intermediate services often act as both
service providers and consumers. After receiving a request from the upstream
service, the intermediate service processes the request and returns the result
to the upstream service, and then calls the downstream service's interface as
needed. Therefore, the graceful shutdown function needs to ensure stability on
both the service provider and consumer sides, which can be broken down into the
following steps:
+- A provider notifies the registry and long-connection consumers before it
stops accepting new business requests.
+- Requests that have already reached the provider can finish within a bounded
timeout.
+- A consumer can wait for its own outbound requests before releasing
downstream references.
+- Resource destruction, port closing, and custom shutdown callbacks run in
order.
-1. Unregister from the registry, destroying the service information registered
in the registry.
-2. As a service provider, wait for a period to ensure that the client
successfully updates service information and completes upstream task
processing, then refuse to accept new requests.
-3. As a service consumer, wait for a period to ensure that requests to the
downstream service receive a response, then cancel the subscription to the
registry.
-4. Destroy references to downstream tasks and ports exposed for service
provision.
-5. Execute the user's custom callback operations.
+## 2. How It Works
-By following these steps, it ensures that the dubbo-go service instance stops
safely and smoothly, without impacting ongoing business.
+In a normal Dubbo-go call chain, a service instance often acts as both a
provider and a consumer. Graceful shutdown therefore handles both sides.
-> Note: Canceling the subscription to the registry cannot be performed in step
1, as changes to downstream service information may occur when the intermediate
service sends requests to the downstream service.
+### 2.1 Shutdown Trigger
-## Usage Scenarios
+When the application receives an exit signal such as `SIGTERM` or `SIGINT`,
Dubbo-go enters the built-in graceful shutdown flow if
`dubbo.shutdown.internal-signal` is `true`. Typical triggers include Kubernetes
pod deletion, rolling upgrades, `kubectl delete pod`, `kill -TERM <pid>`, and
pressing `Ctrl+C` in a terminal.
-Stop the dubbo-go instance using the ` kill pid ` command.
+> Do not use `kill -9` for graceful shutdown. `SIGKILL` cannot be caught by
the process, so Dubbo-go cannot run the shutdown flow.
-## Usage
+### 2.2 Provider Flow
-The following are configurable settings that users can customize in the yaml
configuration file:
+The provider-side goal is to remove the instance from traffic first, then wait
for in-flight requests.
-- As a service provider, the dubbo-go instance needs to wait for the client to
successfully update service information during the offline period. This time
corresponds to the `consumer-update-wait-time` field in the configuration,
defaulting to 3s.
-- As a service provider, if requests from upstream tasks are not yet
processed, it needs to wait for them to complete. As a service consumer, it
must wait for responses to downstream requests. This time corresponds to the
`step-timeout` field, defaulting to 3s.
-- As a service provider, if requests from upstream tasks are already
processed, it needs to wait for a window time. If no new requests are received
during this time, proceed with subsequent steps. This corresponds to the
`offline-request-window-timeout` field, defaulting to 0s.
-- Users can customize whether to enable graceful shutdown functionality; this
corresponds to the `internal-signal` field in the configuration, which is
enabled by default.
-- The dubbo-go instance may freeze during graceful shutdown due to exceptions.
A timeout can be configured in the settings to forcibly close the instance
after this time. This corresponds to the `timeout` field, defaulting to 60s.
+1. Unregister the current instance from the registry so consumers can refresh
their provider list.
+2. Send closing notices to Triple long-connection consumers so they can avoid
the closing invoker earlier.
+3. Continue accepting requests during `consumer-update-wait-time` to allow
registry events and client caches to converge.
+4. Before turning on request rejection, wait for provider active requests to
reach zero within `step-timeout` and `offline-request-window-timeout`.
+5. Enter the reject-new-request phase. New requests are rejected or handled by
the custom `reject-handler`.
+6. Destroy protocols, close listening ports, and release resources.
+
+`offline-request-window-timeout` is useful for high-traffic services. After
active requests temporarily reach zero, Dubbo-go observes a short window. If no
new request arrives during that window, it is safer to continue destroying
resources.
+
+### 2.3 Consumer Flow
+
+The consumer-side goal is to avoid providers that are closing and to finish
its own outbound requests before exit.
+
+When the local instance exits as a consumer:
+
+1. Wait for outbound downstream requests to finish. The wait budget is
controlled by `step-timeout`.
+2. Unsubscribe from the registry.
+3. Destroy consumer invokers, connections, and references.
+
+When a remote provider is shutting down, the consumer can mark the
corresponding invoker as closing based on active closing notices, response
attachments, or connection-closing errors. During
`closing-invoker-expire-time`, the consumer avoids routing to that invoker. The
invoker is removed when the registry delete event arrives; if the delete event
does not arrive in time, the expiration prevents the invoker from staying
unavailable forever.
+
+> A consumer should not unsubscribe from the registry at the beginning of
shutdown, because the instance may still need service discovery data to
complete outbound calls that are already in progress.
+
+## 3. Configuration
+
+Configure graceful shutdown in `dubbogo.yaml`:
```yaml
dubbo:
shutdown:
- timeout:60
- step-timeout:3
- consumer-update-wait-time:3
- internal-signal:true
- offline-request-window-timeout:0
+ timeout: 60s
+ step-timeout: 3s
+ notify-timeout: 5s
+ consumer-update-wait-time: 3s
+ offline-request-window-timeout: 3s
+ closing-invoker-expire-time: 30s
+ internal-signal: true
+```
+
+If you initialize a Dubbo instance with the new API, you can also configure
common graceful shutdown parameters with `dubbo.WithShutdown(...)` and
`graceful_shutdown.WithXXX(...)` options:
+
+```go
+package main
+
+import (
+ "time"
+
+ dubbo "dubbo.apache.org/dubbo-go/v3"
+ "dubbo.apache.org/dubbo-go/v3/graceful_shutdown"
+)
+
+func main() {
+ ins, err := dubbo.NewInstance(
+ dubbo.WithShutdown(
+ graceful_shutdown.WithTimeout(60*time.Second),
+ graceful_shutdown.WithStepTimeout(3*time.Second),
+ graceful_shutdown.WithNotifyTimeout(5*time.Second),
+
graceful_shutdown.WithConsumerUpdateWaitTime(3*time.Second),
+
graceful_shutdown.WithOfflineRequestWindowTimeout(3*time.Second),
+ // Dubbo-go listens for shutdown signals internally by
default.
+ // Uncomment the next line if the application handles
signals itself.
+ // graceful_shutdown.WithoutInternalSignal(),
+ ),
+ )
+ if err != nil {
+ panic(err)
+ }
+ _ = ins
+}
+```
+
+`internal-signal` is enabled by default. Use
`graceful_shutdown.WithoutInternalSignal()` in the new API when you need to
disable it. Fields without dedicated `WithXXX` options can still be configured
in `dubbogo.yaml`.
+
+| Field | Default | Description |
+| --- | --- | --- |
+| `timeout` | `60s` | Maximum duration of the whole graceful shutdown flow.
The application continues exiting after this budget to avoid hanging forever. |
+| `step-timeout` | `3s` | Timeout for a single wait phase, mainly request
draining. It should be greater than the service P99/P999 latency. |
+| `notify-timeout` | `5s` | Timeout budget for actively notifying
long-connection consumers. It controls only the notify step and does not
replace request draining. |
+| `consumer-update-wait-time` | `3s` | Time a provider waits after
unregistering so consumers can refresh their provider list. |
+| `offline-request-window-timeout` | `3s` | Observation window after provider
active requests reach zero, used to confirm no new request is still arriving. |
+| `closing-invoker-expire-time` | `30s` | Expiration time for a consumer-side
invoker that has been marked as closing. |
+| `internal-signal` | `true` | Whether Dubbo-go listens for process shutdown
signals internally. If disabled, the application must trigger shutdown itself. |
+| `reject-handler` | default handler | Rejection handler used after the
provider starts rejecting new requests. It usually does not need customization.
|
+
+Recommended tuning rules:
+
+- `timeout` should cover registry propagation, long-connection notification,
request draining, and shutdown callbacks. It should be lower than Kubernetes
`terminationGracePeriodSeconds`.
+- `step-timeout` should cover the normal latency of most requests. Increase it
for slow methods.
+- `consumer-update-wait-time` depends on registry push latency and the time
consumers need to refresh provider lists.
+- `closing-invoker-expire-time` should not be too short, otherwise a consumer
may route to a provider that is still closing before registry convergence
finishes.
+
+## 4. Local Verification
+
+`dubbo-go-samples/graceful_shutdown` provides a Triple local sample for
verifying long-connection notices, request draining, and timing knobs. The
sample does not include a registry, so it can verify protocol-level behavior
but cannot directly show registry unregister propagation.
+
+### 4.1 Start The Sample
+
+Clone the samples repository and run commands from the repository root:
+
+```bash
+git clone --depth 1 https://github.com/apache/dubbo-go-samples.git
+cd dubbo-go-samples
+```
+
+Start the server in the first terminal:
+
+```bash
+go run ./graceful_shutdown/go-server/cmd -timeout=60s -step-timeout=5s
-delay=2s
+```
+
+Start the client in a second terminal:
+
+```bash
+go run ./graceful_shutdown/go-client/cmd -addr=tri://127.0.0.1:20000
-concurrency=3 -interval=300ms -request-timeout=6s
+```
+
+Then press `Ctrl+C` in the server terminal and watch the logs.
+
+Expected behavior:
+
+- Server logs print graceful shutdown phases in order.
+- Requests that already reached the server still have a chance to complete.
+- After shutdown begins, new requests gradually fail or are avoided by the
consumer.
+
+### 4.2 Important Address Format And Flags
+
+For direct client calls, `-addr` must include the protocol prefix, such as
`tri://127.0.0.1:20000`. If you only pass `127.0.0.1:20000`, the direct
reference may be parsed incorrectly in some scenarios.
+
+The sample server CLI flags are slightly shorter than the YAML configuration
keys. The main flags are:
+
+- `-port=20000`
+- `-timeout=60s`
+- `-step-timeout=3s`
+- `-consumer-update-wait=3s`
+- `-offline-window=3s`
+- `-delay=0s`
+
+`-delay` adds fixed processing latency to every request so you can verify
in-flight request draining during shutdown.
+
+The sample client supports these main flags:
+
+- `-addr=tri://127.0.0.1:20000`
+- `-interval=200ms`
+- `-concurrency=1`
+- `-request-timeout=5s`
+- `-short=true|false`
+- `-name-prefix=hello`
+- `-max-requests=0`
+- `-min-successes=0`
+- `-min-failures=0`
+
+For long-connection testing, keep `-short=false`. `-max-requests`,
`-min-successes`, and `-min-failures` are mainly for automated verification;
the client panics if the configured minimum counts are not reached before exit.
+
+### 4.3 Verify Request Draining
+
+Use `-delay` to simulate slow requests and set `step-timeout` greater than the
request delay:
+
+```bash
+go run ./graceful_shutdown/go-server/cmd -delay=2s -step-timeout=5s
+```
+
+Keep the client sending concurrent requests:
+
+```bash
+go run ./graceful_shutdown/go-client/cmd -addr=tri://127.0.0.1:20000
-concurrency=3 -interval=300ms -request-timeout=6s
+```
+
+Stop the server while requests are running. Some in-flight requests should
complete during shutdown; after the wait budget is consumed, the server
continues exiting.
+
+### 4.4 Verify Long-Connection Notice
+
+The default client uses long connections, which is the preferred path for
verifying active closing notices:
+
+```bash
+go run ./graceful_shutdown/go-client/cmd -addr=tri://127.0.0.1:20000
+```
+
+To compare short-connection behavior, add `-short=true`:
+
+```bash
+go run ./graceful_shutdown/go-client/cmd -addr=tri://127.0.0.1:20000
-short=true
+```
+
+### 4.5 Observe Active Notice And Request Draining Together
+
+Use a short consumer update wait so the server starts rejecting new work
earlier while existing requests still have a draining window:
+
+```bash
+go run ./graceful_shutdown/go-server/cmd -delay=2s -timeout=15s
-step-timeout=2s -consumer-update-wait=0s
+```
+
+Keep the client sending concurrent requests:
+
+```bash
+go run ./graceful_shutdown/go-client/cmd -addr=tri://127.0.0.1:20000
-concurrency=2 -interval=200ms -request-timeout=4s
+```
+
+After pressing `Ctrl+C` in the server terminal, watch for:
+
+- Server logs print the full graceful shutdown sequence.
+- Some in-flight requests still complete after shutdown starts.
+- Newer requests fail earlier than with the default configuration.
+- Client logs show the Triple long-connection active notice path.
+
+### 4.6 Short Request Draining Window
+
+Use a draining budget shorter than the request delay to compare server log
behavior:
+
+```bash
+go run ./graceful_shutdown/go-server/cmd -delay=2s -step-timeout=1s
-consumer-update-wait=0s
+```
+
+Keep the client sending concurrent requests:
+
+```bash
+go run ./graceful_shutdown/go-client/cmd -addr=tri://127.0.0.1:20000
-concurrency=2 -interval=200ms -request-timeout=4s
+```
+
+This scenario is mainly for observing how the graceful shutdown flow continues
when `step-timeout` is shorter than the in-flight request processing time. The
overall `timeout` has a lower bound in Dubbo-go, so use `step-timeout` for
local short-budget draining verification.
+
+### 4.7 Integration Test
+
+The sample is also included in the samples repository integration test script:
+
+```bash
+./integrate_test.sh graceful_shutdown
+```
+
+The script starts the Triple server, runs the client in the background, waits
until at least one request succeeds, sends an interrupt signal to trigger
graceful shutdown, and verifies that the client observes both successful
requests and shutdown-time failures.
+
+## 5. Kubernetes Deployment Tips
+
+In Kubernetes, pod deletion or rolling upgrade runs `preStop`, sends `SIGTERM`
to the container main process, and finally kills the container when
`terminationGracePeriodSeconds` expires. A Dubbo-go application must finish
graceful shutdown within this window.
+
+Recommended configuration:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: dubbo-go-provider
+spec:
+ template:
+ spec:
+ terminationGracePeriodSeconds: 90
+ containers:
+ - name: provider
+ image: your-registry/dubbo-go-provider:latest
+ lifecycle:
+ preStop:
+ exec:
+ command: ["/bin/sh", "-c", "sleep 5"]
+```
+
+Configuration recommendations:
+
+- `terminationGracePeriodSeconds` should be greater than
`dubbo.shutdown.timeout`, with extra time reserved for log flushing, metrics
reporting, and container exit.
+- A short `sleep` in `preStop` can give Kubernetes Endpoints, Service load
balancing, and external gateways time to remove the pod from traffic, but it
should not replace Dubbo-go graceful shutdown.
+- The container entrypoint should use `exec ./your-app` or a signal-forwarding
init process such as `tini` so `SIGTERM` reaches the Go process.
+- If the application exposes health checks, make readiness fail quickly after
shutdown starts to reduce the window in which Kubernetes Service still forwards
new traffic.
+
+Verification commands:
+
+```bash
+kubectl delete pod <pod-name>
+kubectl logs -f <pod-name>
```
-Additionally, if users wish to execute custom callback operations after the
offline logic is completely finished, they can use the following code:
+Keep a client sending requests at the same time, then check server logs,
client success/failure counts, registry instance lists, and pod exit duration.
+
+## 6. Custom Callback
+
+If you need cleanup logic after the shutdown flow, such as flushing logs,
reporting metrics, or closing business resources, register a custom callback:
```go
extension.AddCustomShutdownCallback(func() {
@@ -63,6 +318,16 @@ extension.AddCustomShutdownCallback(func() {
})
```
-## References
+Callbacks should be short and bounded. Long-running callbacks are still
constrained by the total `timeout` budget.
+
+## 7. Troubleshooting Checklist
+
+- If graceful shutdown does not start, make sure the process was not killed by
`kill -9` and `internal-signal` was not disabled.
+- If rolling upgrades still produce many failures, check whether
`consumer-update-wait-time` is shorter than registry propagation and consumer
refresh latency.
+- If slow requests are interrupted, check whether `step-timeout` and
Kubernetes `terminationGracePeriodSeconds` are large enough.
+- If consumers still call a provider that is closing, check whether Triple
long connections are used, registry events are normal, and
`closing-invoker-expire-time` is not too short.
+- If Kubernetes force kills the pod, increase `terminationGracePeriodSeconds`
or shorten application shutdown callbacks.
+
+## 8. References
[【Dubbo-go Elegant Up and Down Design and
Practice】](https://developer.aliyun.com/article/860775)
diff --git
a/content/zh-cn/overview/mannual/golang-sdk/tutorial/deployment/graceful_shutdown.md
b/content/zh-cn/overview/mannual/golang-sdk/tutorial/deployment/graceful_shutdown.md
index aa603e4f385..bef93afc41e 100644
---
a/content/zh-cn/overview/mannual/golang-sdk/tutorial/deployment/graceful_shutdown.md
+++
b/content/zh-cn/overview/mannual/golang-sdk/tutorial/deployment/graceful_shutdown.md
@@ -3,60 +3,313 @@ aliases:
- /zh/docs3-v2/golang-sdk/tutorial/governance/traffic/graceful_shutdown/
- /zh-cn/docs3-v2/golang-sdk/tutorial/governance/traffic/graceful_shutdown/
- /zh-cn/overview/mannual/golang-sdk/tutorial/deploy2/graceful_shutdown/
-description: 本文主要介绍优雅下线的基本步骤和使用说明
-keywords: 优雅下线
-title: 优雅下线
+description: 本文主要介绍 Dubbo-go 优雅停机的整体流程、配置、验证方式和 Kubernetes 部署建议
+keywords: 优雅停机
+title: 优雅停机
type: docs
---
-# 优雅下线
+# 优雅停机
示例源码:<a
href="https://github.com/apache/dubbo-go-samples/tree/main/graceful_shutdown"
target="_blank">dubbo-go-samples/graceful_shutdown</a>。
-## 背景
+## 1. 背景
-在稳定生产的过程中,容器调度完全由 k8s
管控,微服务治理完全由服务框架或者运维人员进行维护和管理。而在发布新版本,或者扩缩容的场景下,会终止旧的容器实例,并使用新的容器实例进行替换,对于承载高流量的线上生产环境,这个替换过程的衔接如果不合理,将在短时间内造成大量的错误请求,触发报警甚至影响正常业务。对于体量较大的厂家,发布过程出现问题所造成的损失会是巨大的。
-因此,优雅下线的诉求被提出。这要求服务框架在拥有稳定服务调用能力、传统服务治理能力的基础之上,应当提供服务下线过程中稳定的保障,从而减少运维成本,提高应用稳定性。
+在发布新版本、扩缩容或迁移节点时,Kubernetes、systemd
或运维平台会终止旧实例并启动新实例。如果旧实例直接退出,注册中心、客户端连接池和上游流量之间会出现短暂的不一致:注册中心可能还没有推送摘除事件,上游请求也可能仍在处理中。对于高流量服务,这会带来连接断开、请求失败、超时重试和告警放大。
-## 特性说明
+Dubbo-go 的优雅停机用于在进程退出前完成摘流和请求排空,尽量保证:
-在一次完整的RPC调用过程中,中间服务往往充当服务提供者和服务消费者两个角色。中间服务在接收到来自上游服务的请求之后,处理请求得到结果返回给上游服务,然后根据需要调用下游服务提供的接口使用下游服务。因此优雅下线功能需要兼顾服务作为服务提供者和服务消费者两侧的稳定性,具体可以分为以下几步:
+- Provider 不再接收新的业务请求之前,先从注册中心反注册并通知长连接 Consumer。
+- 已经进入 Provider 的请求可以在超时时间内完成。
+- Consumer 本身正在发出的下游调用可以在退出前拿到响应。
+- 资源销毁、端口释放和用户自定义回调按顺序执行。
-1. 向注册中心进行反注册,销毁在注册中心注册的服务信息
-2. 作为服务提供者,要等待一段时间,保证客户端成功更新服务信息以及上游任务请求处理完毕,然后拒绝接收新的请求
-3. 作为服务消费者,要等待一段时间,保证使用下游服务的请求得到响应,然后取消对注册中心的订阅
-4. 销毁对下游任务的引用,销毁对外提供服务暴露的端口
-5. 执行用户的自定义回调操作
+## 2. 工作机制
-通过以上步骤,可以保证dubbo-go服务实例安全平稳停止,不对进行中的业务产生影响。
+一次 Dubbo-go 调用链路中,某个服务实例通常既是 Provider,也是 Consumer。因此优雅停机会同时处理服务提供端和服务消费端。
-> 注意:取消对注册中心的订阅不能在步骤1中执行,这是因为中间服务对下游服务发送请求的时候可能存在下游服务信息的变动
+### 2.1 触发停机
-## 使用场景
+当应用收到 `SIGTERM`、`SIGINT` 等退出信号时,如果 `dubbo.shutdown.internal-signal` 为
`true`,Dubbo-go 会进入内置优雅停机流程。Kubernetes 删除 Pod、滚动发布、`kubectl delete pod`、`kill
-TERM <pid>` 和终端 `Ctrl+C` 都属于典型触发方式。
-对dubbo-go实例使用` kill pid `命令停止实例
+> 不要使用 `kill -9` 触发停机。`SIGKILL` 无法被进程捕获,Dubbo-go 没有机会执行优雅停机流程。
-## 使用方式
+### 2.2 Provider 侧流程
-以下是在yaml配置文件中,用户可以自定义的配置
+Provider 侧目标是先摘流,再等待正在执行的请求完成。
--
作为服务提供者,dubbo-go实例下线时需要等待客户端成功更新服务信息,这段时间在配置中对应的字段为consumer-update-wait-time,默认3s
--
作为服务提供者,dubbo-go实例下线时如果来自上游任务的请求暂未处理完毕,需要等待上游任务请求处理完毕。作为服务消费者,dubbo-go实例需要等待对下游的请求收到回复。这段时间在配置中对应的字段为step-timeout,默认3s
--
作为服务提供者,dubbo-go实例下线时如果来自上游任务的请求已经处理完毕,需要等待一段窗口时间,如果在窗口时间内没有接收到新的请求,再执行后续步骤。这段时间在配置中对应的字段为offline-request-window-timeout,默认0s
-- 用户可以自定义是否开启优雅下线功能,在配置中对应的字段为internal-signal,默认开启。
--
dubbo-go实例在优雅下线过程中可能因为异常导致卡死,在配置中可以配置超时时间,实例在超时之后强制关闭。这在配置中对应的字段为timeout,默认60s
+1. 从注册中心反注册当前实例,触发 Consumer 端服务列表更新。
+2. 对 Triple 长连接 Consumer 发送关闭通知,使 Consumer 可以尽快避开正在关闭的 Invoker。
+3. 在 `consumer-update-wait-time` 时间内继续接收请求,给客户端和注册中心事件传播留出窗口。
+4. 在开启拒绝新请求前,在 `step-timeout` 和 `offline-request-window-timeout` 约束下等待
Provider 侧活跃请求数归零。
+5. 进入拒绝新请求阶段,后续新请求会被拒绝或由自定义 `reject-handler` 处理。
+6. 销毁协议、关闭监听端口并释放资源。
+
+`offline-request-window-timeout`
主要用于高流量场景:当活跃请求暂时归零后,再观察一个短窗口。如果窗口内没有新的请求进入,就可以更安全地继续销毁资源。
+
+### 2.3 Consumer 侧流程
+
+Consumer 侧目标是避免继续调用正在关闭的 Provider,并保证自身退出时下游请求尽量完成。
+
+当本实例作为 Consumer 退出时:
+
+1. 等待正在发出的下游请求完成,等待预算由 `step-timeout` 控制。
+2. 取消对注册中心的订阅。
+3. 销毁 Consumer 端 Invoker、连接和引用。
+
+当远端 Provider 正在关闭时,Consumer 会根据主动关闭通知、响应中的关闭标记或连接关闭错误,将对应 Invoker 临时标记为
closing,并在 `closing-invoker-expire-time` 期间避免继续路由到该 Invoker。注册中心删除事件到达后,Invoker
会被正式移除;如果删除事件没有及时到达,过期时间可以避免 Invoker 永久不可用。
+
+> Consumer 取消注册中心订阅不能放在最开始执行,因为服务在退出期间仍可能需要完成已经发出的下游调用。
+
+## 3. 配置说明
+
+可以在 `dubbogo.yaml` 中配置优雅停机参数:
```yaml
dubbo:
shutdown:
- timeout:60
- step-timeout:3
- consumer-update-wait-time:3
- internal-signal:true
- offline-request-window-timeout:0
+ timeout: 60s
+ step-timeout: 3s
+ notify-timeout: 5s
+ consumer-update-wait-time: 3s
+ offline-request-window-timeout: 3s
+ closing-invoker-expire-time: 30s
+ internal-signal: true
+```
+
+如果使用 new API 初始化 Dubbo 实例,也可以通过 `dubbo.WithShutdown(...)` 搭配
`graceful_shutdown.WithXXX(...)` 配置常用停机参数:
+
+```go
+package main
+
+import (
+ "time"
+
+ dubbo "dubbo.apache.org/dubbo-go/v3"
+ "dubbo.apache.org/dubbo-go/v3/graceful_shutdown"
+)
+
+func main() {
+ ins, err := dubbo.NewInstance(
+ dubbo.WithShutdown(
+ graceful_shutdown.WithTimeout(60*time.Second),
+ graceful_shutdown.WithStepTimeout(3*time.Second),
+ graceful_shutdown.WithNotifyTimeout(5*time.Second),
+
graceful_shutdown.WithConsumerUpdateWaitTime(3*time.Second),
+
graceful_shutdown.WithOfflineRequestWindowTimeout(3*time.Second),
+ // 默认启用 Dubbo-go 内置信号监听;如需应用自行处理信号,可取消下一行注释。
+ // graceful_shutdown.WithoutInternalSignal(),
+ ),
+ )
+ if err != nil {
+ panic(err)
+ }
+ _ = ins
+}
+```
+
+`internal-signal` 默认启用,new API 中需要关闭时使用
`graceful_shutdown.WithoutInternalSignal()`;没有专用 `WithXXX` 的字段可以继续保留在
`dubbogo.yaml` 中配置。
+
+| 字段 | 默认值 | 说明 |
+| --- | --- | --- |
+| `timeout` | `60s` | 整个优雅停机流程的最大耗时。超过后应用会继续退出,避免进程长期卡住。 |
+| `step-timeout` | `3s` | 单个等待阶段的超时时间,主要用于请求排空。建议大于服务 P99/P999 响应时间。 |
+| `notify-timeout` | `5s` | 主动通知长连接 Consumer 的超时预算,只控制通知阶段,不替代请求排空时间。 |
+| `consumer-update-wait-time` | `3s` | Provider 反注册后继续等待 Consumer 更新地址列表的时间。 |
+| `offline-request-window-timeout` | `3s` | Provider
活跃请求归零后的观察窗口,用于确认没有新的请求继续进入。 |
+| `closing-invoker-expire-time` | `30s` | Consumer 将远端 Invoker 标记为 closing
后的过期时间。 |
+| `internal-signal` | `true` | 是否启用 Dubbo-go 内置信号监听。关闭后需要应用自行调用停机逻辑。 |
+| `reject-handler` | 默认处理器 | Provider 开始拒绝新请求时使用的拒绝处理器,通常无需配置。 |
+
+建议根据业务实际耗时调整这些参数:
+
+- `timeout` 应覆盖注册中心传播、长连接通知、请求排空和回调执行,并且小于 Kubernetes
`terminationGracePeriodSeconds`。
+- `step-timeout` 应至少覆盖绝大多数请求的正常响应时间,慢接口可适当调大。
+- `consumer-update-wait-time` 取决于注册中心推送延迟和 Consumer 更新地址列表的耗时。
+- `closing-invoker-expire-time` 不宜过短,否则 Consumer 可能在注册中心事件未收敛前重新选择正在关闭的
Provider。
+
+## 4. 本地验证
+
+`dubbo-go-samples/graceful_shutdown` 提供了 Triple
协议的本地验证示例,用于观察长连接通知、请求排空和停机时间参数的效果。该示例不包含注册中心,因此可以验证协议层行为,但不能直接观察注册中心反注册传播。
+
+### 4.1 启动示例
+
+在本地克隆示例仓库,并从仓库根目录执行命令:
+
+```bash
+git clone --depth 1 https://github.com/apache/dubbo-go-samples.git
+cd dubbo-go-samples
+```
+
+在第一个终端启动服务端:
+
+```bash
+go run ./graceful_shutdown/go-server/cmd -timeout=60s -step-timeout=5s
-delay=2s
+```
+
+在第二个终端启动客户端:
+
+```bash
+go run ./graceful_shutdown/go-client/cmd -addr=tri://127.0.0.1:20000
-concurrency=3 -interval=300ms -request-timeout=6s
+```
+
+然后在服务端终端按 `Ctrl+C`,观察日志。
+
+预期现象:
+
+- 服务端按顺序输出优雅停机阶段日志。
+- 已经进入服务端的请求仍有机会完成。
+- 停机开始后,新请求会逐渐失败或被 Consumer 避开。
+
+### 4.2 重要地址格式和参数
+
+进行直连调用时,`-addr` 必须带协议前缀,例如 `tri://127.0.0.1:20000`。如果只传
`127.0.0.1:20000`,在某些场景下可能会被错误解析。
+
+示例服务端命令行参数与 YAML 配置名不完全相同,主要支持:
+
+- `-port=20000`
+- `-timeout=60s`
+- `-step-timeout=3s`
+- `-consumer-update-wait=3s`
+- `-offline-window=3s`
+- `-delay=0s`
+
+其中 `-delay` 会给每次请求增加固定处理延迟,用于观察停机时的在途请求排空效果。
+
+示例客户端主要支持:
+
+- `-addr=tri://127.0.0.1:20000`
+- `-interval=200ms`
+- `-concurrency=1`
+- `-request-timeout=5s`
+- `-short=true|false`
+- `-name-prefix=hello`
+- `-max-requests=0`
+- `-min-successes=0`
+- `-min-failures=0`
+
+长连接验证时建议保持 `-short=false`。`-max-requests`、`-min-successes` 和 `-min-failures`
主要用于自动化验证;如果客户端退出前没有达到最小阈值,会直接 `panic` 让集成测试失败。
+
+### 4.3 验证请求排空
+
+使用 `-delay` 模拟慢请求,并让 `step-timeout` 大于请求耗时:
+
+```bash
+go run ./graceful_shutdown/go-server/cmd -delay=2s -step-timeout=5s
+```
+
+客户端保持并发调用:
+
+```bash
+go run ./graceful_shutdown/go-client/cmd -addr=tri://127.0.0.1:20000
-concurrency=3 -interval=300ms -request-timeout=6s
+```
+
+当请求正在执行时停止服务端。正常情况下,部分已进入服务端的请求会在停机流程中完成;超过等待预算后,服务端继续退出。
+
+### 4.4 验证长连接通知
+
+默认客户端使用长连接,更适合验证主动关闭通知:
+
+```bash
+go run ./graceful_shutdown/go-client/cmd -addr=tri://127.0.0.1:20000
+```
+
+如需对比短连接行为,可以增加 `-short=true`:
+
+```bash
+go run ./graceful_shutdown/go-client/cmd -addr=tri://127.0.0.1:20000
-short=true
+```
+
+### 4.5 同时观察主动通知与请求排空
+
+缩短 Consumer 更新等待时间,可以让服务端更早拒绝新请求,同时保留已有请求的排空窗口:
+
+```bash
+go run ./graceful_shutdown/go-server/cmd -delay=2s -timeout=15s
-step-timeout=2s -consumer-update-wait=0s
+```
+
+客户端持续并发调用:
+
+```bash
+go run ./graceful_shutdown/go-client/cmd -addr=tri://127.0.0.1:20000
-concurrency=2 -interval=200ms -request-timeout=4s
+```
+
+在服务端终端按 `Ctrl+C` 后,重点观察:
+
+- 服务端日志打印完整优雅停机序列。
+- 部分在途请求会在停机开始后继续完成。
+- 新进入的请求会比默认配置更早失败。
+- 客户端日志会体现 Triple 长连接主动通知路径。
+
+### 4.6 缩短请求排空窗口
+
+可以让请求排空预算短于请求处理耗时,用于对比服务端日志行为:
+
+```bash
+go run ./graceful_shutdown/go-server/cmd -delay=2s -step-timeout=1s
-consumer-update-wait=0s
+```
+
+客户端保持并发调用:
+
+```bash
+go run ./graceful_shutdown/go-client/cmd -addr=tri://127.0.0.1:20000
-concurrency=2 -interval=200ms -request-timeout=4s
+```
+
+该场景主要用于观察 `step-timeout` 短于在途请求处理耗时时,优雅停机流程如何继续推进。Dubbo-go 的整体 `timeout`
存在下限,本地验证短等待预算时建议调整 `step-timeout`。
+
+### 4.7 集成测试
+
+示例也接入了 samples 仓库的集成测试脚本:
+
+```bash
+./integrate_test.sh graceful_shutdown
+```
+
+脚本会启动 Triple 服务端,后台运行客户端,等待至少一次请求成功后发送中断信号触发优雅停机,并校验客户端在停机期间能观察到成功请求和失败请求。
+
+## 5. Kubernetes 部署建议
+
+在 Kubernetes 中,Pod 删除或滚动升级时会先执行 `preStop`,然后向容器主进程发送 `SIGTERM`,最后在
`terminationGracePeriodSeconds` 到期后强制结束容器。Dubbo-go 应用需要在这个时间窗口内完成优雅停机。
+
+推荐配置:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: dubbo-go-provider
+spec:
+ template:
+ spec:
+ terminationGracePeriodSeconds: 90
+ containers:
+ - name: provider
+ image: your-registry/dubbo-go-provider:latest
+ lifecycle:
+ preStop:
+ exec:
+ command: ["/bin/sh", "-c", "sleep 5"]
+```
+
+配置建议:
+
+- `terminationGracePeriodSeconds` 应大于
`dubbo.shutdown.timeout`,并预留日志刷新、指标上报和容器退出时间。
+- `preStop` 中的短暂 `sleep` 可为 Kubernetes Endpoint、Service 负载均衡和外部网关摘流留出时间,但不要用过长
`sleep` 替代 Dubbo-go 自身的优雅停机。
+- 容器入口脚本应使用 `exec ./your-app` 或 `tini` 等方式确保 `SIGTERM` 能传递给 Go 进程。
+- 如果应用提供健康检查,建议在进入停机后尽快让 readiness 失败,减少 Kubernetes Service 继续转发新流量的窗口。
+
+验证方式:
+
+```bash
+kubectl delete pod <pod-name>
+kubectl logs -f <pod-name>
```
-此外,如果用户希望在下线逻辑彻底结束后,执行一些自定义的回调操作,可以使用如下代码
+同时保持客户端持续调用,观察服务端日志、客户端成功/失败请求数量、注册中心实例列表以及 Pod 退出耗时。
+
+## 6. 自定义回调
+
+如果需要在停机流程结束后执行清理动作,例如刷新日志、上报指标或关闭业务资源,可以注册自定义回调:
```go
extension.AddCustomShutdownCallback(func() {
@@ -64,6 +317,16 @@ extension.AddCustomShutdownCallback(func() {
})
```
-## 参考资料
+回调应尽量短小可控。如果回调耗时过长,仍会受到 `timeout` 总预算限制。
+
+## 7. 排查清单
+
+- 如果没有进入优雅停机流程,先确认不是 `kill -9`,并确认 `internal-signal` 没有被关闭。
+- 如果滚动发布仍出现大量失败,检查 `consumer-update-wait-time` 是否小于注册中心推送和 Consumer 地址刷新耗时。
+- 如果慢请求被中断,检查 `step-timeout` 和 Kubernetes `terminationGracePeriodSeconds` 是否足够。
+- 如果 Consumer 仍打到正在关闭的 Provider,检查是否使用 Triple
长连接、注册中心事件是否正常、`closing-invoker-expire-time` 是否过短。
+- 如果 Pod 被 Kubernetes 强制杀死,调大 `terminationGracePeriodSeconds` 或缩短应用侧停机回调耗时。
+
+## 8. 参考资料
[【Dubbo-go 优雅上下线的设计与实践】](https://developer.aliyun.com/article/860775)