scornet256 opened a new issue, #14390:
URL: https://github.com/apache/cloudstack/issues/14390
### problem
We run 5 management servers. after a restart of one of them, that server
keeps stopping the console proxy vm about every 30 seconds with:
```
INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[...]) Stop console
proxy Console VM instance {"id":35,...}
```
Because we are currently in `ResetSuspending` management mode the other
management servers start it again, so the console proxy keeps flapping between
Up and Disconnected. we counted 200+ stops in about 1hours time. The agent
connection itself is fine, `StartupProxyCommand` and `ReadyCommand` get
answered, then it gets stopped again.
The db is fine while this happens:
- `consoleproxy.management.state = Auto`
- `consoleproxy.management.state.last = Auto`
- both global only, no zone level values, and `isdynamic: false` in
listConfigurations
The other 4 management servers dont log any `ResetSuspending`. restarting
the affected management server once more fixes it, 0 stops after that.
We had this twice now, both times on the same management server right after
it restarted. the first time was shortly after we uploaded console proxy
certificates with `uploadCustomCertificate`. The second time we can't link it
to an upload, so we dont know what had `ResetSuspending` in the db when that
server read it.
Looks like the regression from #11415: these settings are read through
ConfigKey now and are not dynamic, so every management server caches the value
once and never reads it again. In a cluster one server can stay stuck in
`ResetSuspending` while all the others are on `Auto`, and they keep fighting
over the console proxy.
### versions
- ACS 4.22.1.1, 5 management servers
- KVM on AlmaLinux 9.8, agent 4.22.1.0
- MariaDB Galera cluster
### The steps to reproduce the bug
Not 100% reproducible for us, what we think happens:
1. Run more than one management server
2. Upload a console proxy certificate, this sets
`consoleproxy.management.state` to `ResetSuspending` in the database
3. Restart one management server before the state is back on `Auto`, so it
reads `ResetSuspending` on startup
4. The state goes back to `Auto` in the database, but the restarted server
keeps `ResetSuspending` in memory and stops the console proxy every scan
### What to do about it?
#13202 makes these two settings dynamic, which should fix this since the
stuck server would read Auto again. It has been approved but has merge
conflicts since september, would be good to get it into 4.22.2.
For now the workaround is: after restarting management servers, grep the
logs for "currently in ResetSuspending" and restart the server that shows it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
