Narayanaraj commented on issue #10915:
URL: https://github.com/apache/cloudstack/issues/10915#issuecomment-6582551813
We hit this on **4.23.0.0** with **one** management server and KVM agents
(Ubuntu 24.04, OpenJDK 17), and **not** through maintenance mode, which I think
answers the question above about whether it can trigger with a single
management IP:
- A plain `systemctl restart cloudstack-management` (in our case triggered
by an OS package update whose post-install hook restarted the service, and once
by a reboot) leaves every agent that was connected `Disconnected` for days. The
agent logs `Lost connection to host: <ms-ip>. Attempting reconnection while we
still have 0 commands in progress.` and then nothing at all. Restarting
`cloudstack-agent` fixes it. A host whose agent happened to restart later
reconnected normally.
- No connection attempt to port 8250 leaves the agent (checked with `ss` for
40 s), although the port is reachable.
- Thread dump of an agent about three days after the disconnect: the
reconnect thread is alive and looping, not blocked:
```
"Agent-Handler-4" #52 prio=5 os_prio=0 ... waiting on condition
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep([email protected]/Native Method)
at
com.cloud.utils.backoff.impl.ConstantTimeBackoff.waitBeforeRetry(ConstantTimeBackoff.java:49)
at com.cloud.agent.Agent.reconnect(Agent.java:659)
at com.cloud.agent.Agent.reconnect(Agent.java:627)
at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1342)
```
Reading 4.23.0.0, the chain matches the description in this issue, and it
does not need maintenance mode:
1. `AgentManagerImpl` (around line 1719) answers pings with `avoidMsList =
_mshostDao.listNonUpStateMsIPs()`, every management server not in state `Up`.
2. During a **graceful** shutdown the management server is no longer `Up`
while it still answers pings, so its last ping answers list **itself**.
3. The agent replaces its avoid list on every ping answer
(`processPingAnswer` → `shell.setAvoidHosts(...)`).
4. `Agent.reconnect()` only builds a `NioClient` (and logs `Reconnecting to
host`) for a host not in the avoid list. With one configured host in the list,
it sleeps and picks the same host again, forever, without logging.
A hard crash would not set the list, which matches what we saw: it only
happens on clean stops.
Possible fixes, any of:
- in `reconnect()`, ignore (or clear) the avoid list when every configured
host is in it;
- don't put the answering management server's own IP into its ping answers,
or clear the avoid list when the connection drops;
- at least log at INFO when a host is skipped because it is in the avoid
list. The silence is what made this hard to find.
Workaround we run now: restart the agents after any planned management
restart, plus a watchdog on each KVM host that restarts an agent with no
ESTABLISHED connection to 8250 for 10 minutes while the port answers.
Happy to test a patch on 4.23.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
