hanahmily opened a new pull request, #1163:
URL: https://github.com/apache/skywalking-banyandb/pull/1163
### Problem
When the FODC proxy pod is replaced (rollout, restart, or rescheduling),
every `fodc-agent` drops to `agents_online: 0` and **never re-registers** —
recovery requires a manual `kubectl rollout restart` of the agent workloads.
### Root cause
On a broken proxy connection the agent runs `reconnect()`, which first tears
down the heartbeat ticker and all stream receive-loops, then re-establishes
them. If any stream (re)start fails transiently — e.g. the **new** proxy pod is
not yet accepting RPCs (`connection refused` to the service VIP during the swap
window) — `reconnect()` simply `return`ed:
- no retry was scheduled,
- the heartbeat ticker and stream receive-loops (the only things that
re-trigger a reconnect) had already been torn down.
So a single failed re-establishment left the agent permanently disconnected.
The transient connect-error branch also called `Disconnect()`, which latches
`disconnected=true` and would suppress any future reconnect.
### Fix
`reconnect()` now calls a new `scheduleReconnect()` on **any** connect or
stream-establishment failure, re-arming another attempt after
`--reconnect-interval` until the proxy is reachable again.
`scheduleReconnect()` is single-flight-safe and re-checks `ctx`/`disconnected`,
so a timer firing after intentional shutdown is a no-op. The transient
connect-error path no longer calls `Disconnect()`.
### Testing
- New unit tests
`TestScheduleReconnect_{ArmsRetryWhenHealthy,SkipsWhenDisconnected,SkipsWhenContextDone}`
(via a small test-only `reconnectFn` seam).
- Local CI green on the `fodc/agent` module: `golangci-lint` (31 linters) +
`revive` → **0 issues**; `go test ./fodc/agent/...` → **11/11 packages pass**.
### Live verification (GKE showcase)
Rolled the fix onto all 7 agents, then scaled the proxy `Deployment` to **0
for 75s** (forcing `connection refused` on every reconnect attempt) and back to
1:
```
23:38:39 connection refused -> "Failed to restart registration stream; will
retry"
23:38:39 "Scheduling proxy reconnect retry" (retry_in=10s)
... repeats every 10s through the 75s outage (6 attempts per agent) ...
23:39:39 "Agent registered with Proxy" -> "Successfully reconnected to
Proxy"
```
All 7 agents auto-recovered to `agents_online: 7` ~16s after the proxy
returned, **with no agent restart**. Before the fix, the agents stalled at `0`
after the first failed attempt and stayed there until a manual restart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
