Hi Peter, This seems to be caused by database connection problems. Try to restart cloudstack management service, and also mariadb if it still doesn't work.
-Wei On Tue, 30 Nov 2021 at 18:37, Peter Stine <peter.st...@granddial.com.invalid> wrote: > Hello everyone, > > I was getting a warning for high CPU usage on my management servers and > found that the spikes were in line with the cloudstack logs showing a > message that the Zone is not prepared to launch the consoleproxy and > secondary storage VMs. > > 2021-11-30 09:40:58,957 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-ff6437c2) (logid:b30a1545) Zone 2 has no host available > which is enabled and in Up state > 2021-11-30 09:40:58,958 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-ff6437c2) (logid:b30a1545) Zone 2 is not ready to launch > secondary storage VM yet > 2021-11-30 09:40:59,751 DEBUG [c.c.c.ConsoleProxyManagerImpl] > (consoleproxy-1:ctx-95609472) (logid:6b4db700) Zone 2 has no host available > which is enabled and in Up state > 2021-11-30 09:40:59,751 DEBUG [c.c.c.ConsoleProxyManagerImpl] > (consoleproxy-1:ctx-95609472) (logid:6b4db700) Zone 2 is not ready to > launch console proxy yet > > The odd thing is that both of those VMs are currently running, accessible, > and can be modified from the webUI. The zone also has 2 hosts, both of > which seem to be operating just fine. > However, there seems to be a Lockmaster1 java exception on the management > servers for the db connection. I think this might be part of the issue, but > I cannot find anything about it. > > This post seems relevant ( > https://blog.widodh.nl/2013/04/cloudstack-zone-x-is-is-not-ready-to-launch-console-proxy-yet/), > but seems only to refer to an issue when the VMs won't launch. > > Has anyone run across this before and found a solution? > > Cloudstack version: 4.15.2 > My systems are running Ubuntu 20.04 > Hypervisor is KVM. > My database (MariaDB) is running in a galera cluster (I have confirmed > that I can log into the database from each server) > My 3 management servers are in a corosync with a HA IP (That IP is where > the database connections are pointed) > > Management logs: > Management 2 (main at this point): > https://gist.github.com/PeterS-gd/10aff04f4204ebebf9066537dea06bac > Management 1 (in cluster): > https://gist.github.com/PeterS-gd/e9e29e030def574313a9168b581d71ed > Management 3 (in cluster): > https://gist.github.com/PeterS-gd/4effc4ddfe7d4a360cc518c121407ee3 > > On the server that has the HA IP, the sql error log shows this: > 2021-11-30 14:31:14 489 [Warning] Aborted connection 489 to db: 'cloud' > user: 'cloud' host: 'gd-cs-control-server' (Got an error reading > communication packets) > 2021-11-30 14:31:14 461 [Warning] Aborted connection 461 to db: 'cloud' > user: 'cloud' host: 'gd-cs-control-server' (Got an error reading > communication packets) > 2021-11-30 14:31:14 521 [Warning] Aborted connection 521 to db: 'cloud' > user: 'cloud' host: 'gd-cs-control-server' (Got an error reading > communication packets) > 2021-11-30 14:31:14 515 [Warning] Aborted connection 515 to db: 'cloud' > user: 'cloud' host: 'gd-cs-control-server' (Got an error reading > communication packets) > 2021-11-30 14:31:14 251 [Warning] Aborted connection 251 to db: > 'cloud_usage' user: 'cloud' host: 'gd-cs-control-server' (Got an error > reading communication packets) > 2021-11-30 14:31:14 490 [Warning] Aborted connection 490 to db: 'cloud' > user: 'cloud' host: 'gd-cs-control-server' (Got an error reading > communication packets) > 2021-11-30 14:31:14 452 [Warning] Aborted connection 452 to db: 'cloud' > user: 'cloud' host: 'gd-cs-control-server' (Got an error reading > communication packets) > 2021-11-30 14:31:14 445 [Warning] Aborted connection 445 to db: 'cloud' > user: 'cloud' host: 'gd-cs-control-server' (Got an error reading > communication packets) > 2021-11-30 14:31:14 514 [Warning] Aborted connection 514 to db: 'cloud' > user: 'cloud' host: 'gd-cs-control-server' (Got an error reading > communication packets) > 2021-11-30 14:31:14 502 [Warning] Aborted connection 502 to db: 'cloud' > user: 'cloud' host: 'gd-cs-control-server' (Got an error reading > communication packets) > > Thanks! > Peter > > >
