Hi Wei, Apparently, my symlinks got deleted somehow. Recreating them fixed the issue. The CPUs are no longer spiking as high as before. Thanks for your help!
Peter ---------------------------------------- From: "Wei ZHOU" <ustcweiz...@gmail.com> Sent: 12/1/21 12:00 PM To: peter.st...@granddial.com Subject: Re: Zone not ready to launch system VMs though they are already running... Hi Peter, Could you please check /etc/cloudstack/usage/db.properties, it should be a symlink to /etc/cloudstack/management/db.properties -Wei On Wed, 1 Dec 2021 at 16:25, Peter Stine <peter.st...@granddial.com> wrote: Hi Slavka, I checked the db.properties on all the management servers and all of those parameters were filled. db.cloud.maxActive=750 db.cloud.maxIdle=90 db.cloud.maxWait=10000 db.cloud.validationQuery=SELECT 1 db.cloud.testOnBorrow=true db.cloud.testWhileIdle=true db.cloud.timeBetweenEvictionRunsMillis=40000 db.cloud.minEvictableIdleTimeMillis=240000 db.cloud.poolPreparedStatements=false db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC I did try raising the numbers for max active and max idle, but after restarting the service, it didn't seem to make a difference as the warning still persists. Peter ---------------------------------------- From: "Slavka Peleva" <slav...@storpool.com> Sent: 12/1/21 3:59 AM To: users@cloudstack.apache.org, peter.st...@granddial.com Cc: Wei ZHOU <ustcweiz...@gmail.com> Subject: Re: Zone not ready to launch system VMs though they are already running... Hi Peter, The error from your usage.log says that the `db.cloud.maxActive` from your `db.properties` file is empty, and you have to provide a value for it. Also, if those properties are empty too - `db.cloud.maxIdle` and `db.cloud.maxWait` - the init of DB will fail again. Best regards, Slavka On Wed, Dec 1, 2021 at 9:55 AM Peter Stine <peter.st...@granddial.com.invalid> wrote: Wei, That got the Zone to recognize as up and ready. To the database point, in the usage log, I now see that is is failing to load the db configuration. As I am not using default logins, this means it won't correctly connect to the database. I am not sure what is stopping cloudstack from loading the database configuration. Here is the warning: 2021-11-30 18:09:34,747 WARN [db.Transaction.Transaction] (main:null) (logid:) Unable to load db configuration, using defaults with 5 connections. Falling back on assumed datasource on localhost:3306 using username:password=cloud:cloud. Please check your configuration java.lang.NumberFormatException: null at java.base/java.lang.Integer.parseInt(Integer.java:614) at java.base/java.lang.Integer.parseInt(Integer.java:770) at com.cloud.utils.db.TransactionLegacy.initDataSource(TransactionLegacy.java:1030) at com.cloud.utils.db.TransactionLegacy.<clinit>(TransactionLegacy.java:1008) at com.cloud.utils.db.TransactionContextBuilder.interceptStart(TransactionContextBuilder.java:49) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:116) at com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_56282efc.createPartialSelectSql(<generated>) at com.cloud.utils.db.GenericDaoBase.<init>(GenericDaoBase.java:238) at com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl.<init>(UsagePortForwardingRuleDaoImpl.java:50) at com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_56282efc.<init>(<generated>) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:291) at net.sf.cglib.proxy.Enhancer$EnhancerFactoryData.newInstance(Enhancer.java:448) at net.sf.cglib.proxy.Enhancer.nextInstance(Enhancer.java:700) at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:298) at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:480) at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:305) at com.cloud.utils.component.ComponentInstantiationPostProcessor.postProcessBeforeInstantiation(ComponentInstantiationPostProcessor.java:92) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1144) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1119) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:923) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:588) at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:144) at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:85) at com.cloud.usage.UsageServer.start(UsageServer.java:52) at com.cloud.usage.UsageServer.main(UsageServer.java:41) ---------------------------------------- From: "Wei ZHOU" <ustcweiz...@gmail.com> Sent: 11/30/21 1:00 PM To: users <users@cloudstack.apache.org>, peter.st...@granddial.com Subject: Re: Zone not ready to launch system VMs though they are already running... Hi Peter, This seems to be caused by database connection problems. Try to restart cloudstack management service, and also mariadb if it still doesn't work. -Wei On Tue, 30 Nov 2021 at 18:37, Peter Stine <peter.st...@granddial.com.invalid> wrote: Hello everyone, I was getting a warning for high CPU usage on my management servers and found that the spikes were in line with the cloudstack logs showing a message that the Zone is not prepared to launch the consoleproxy and secondary storage VMs. 2021-11-30 09:40:58,957 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-ff6437c2) (logid:b30a1545) Zone 2 has no host available which is enabled and in Up state 2021-11-30 09:40:58,958 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-ff6437c2) (logid:b30a1545) Zone 2 is not ready to launch secondary storage VM yet 2021-11-30 09:40:59,751 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-95609472) (logid:6b4db700) Zone 2 has no host available which is enabled and in Up state 2021-11-30 09:40:59,751 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-95609472) (logid:6b4db700) Zone 2 is not ready to launch console proxy yet The odd thing is that both of those VMs are currently running, accessible, and can be modified from the webUI. The zone also has 2 hosts, both of which seem to be operating just fine. However, there seems to be a Lockmaster1 java exception on the management servers for the db connection. I think this might be part of the issue, but I cannot find anything about it. This post seems relevant (https://blog.widodh.nl/2013/04/cloudstack-zone-x-is-is-not-ready-to-launch-console-proxy-yet/), but seems only to refer to an issue when the VMs won't launch. Has anyone run across this before and found a solution? Cloudstack version: 4.15.2 My systems are running Ubuntu 20.04 Hypervisor is KVM. My database (MariaDB) is running in a galera cluster (I have confirmed that I can log into the database from each server) My 3 management servers are in a corosync with a HA IP (That IP is where the database connections are pointed) Management logs: Management 2 (main at this point): https://gist.github.com/PeterS-gd/10aff04f4204ebebf9066537dea06bac Management 1 (in cluster): https://gist.github.com/PeterS-gd/e9e29e030def574313a9168b581d71ed Management 3 (in cluster): https://gist.github.com/PeterS-gd/4effc4ddfe7d4a360cc518c121407ee3 On the server that has the HA IP, the sql error log shows this: 2021-11-30 14:31:14 489 [Warning] Aborted connection 489 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets) 2021-11-30 14:31:14 461 [Warning] Aborted connection 461 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets) 2021-11-30 14:31:14 521 [Warning] Aborted connection 521 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets) 2021-11-30 14:31:14 515 [Warning] Aborted connection 515 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets) 2021-11-30 14:31:14 251 [Warning] Aborted connection 251 to db: 'cloud_usage' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets) 2021-11-30 14:31:14 490 [Warning] Aborted connection 490 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets) 2021-11-30 14:31:14 452 [Warning] Aborted connection 452 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets) 2021-11-30 14:31:14 445 [Warning] Aborted connection 445 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets) 2021-11-30 14:31:14 514 [Warning] Aborted connection 514 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets) 2021-11-30 14:31:14 502 [Warning] Aborted connection 502 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets) Thanks! Peter
