GitHub user TadiosAbebe closed a discussion: Usage server not working
Hello everyone, I can't seem to figure out the following issue i have with the usage server. I remember it was working before but now the usage.err and usage.out logs are throwing `Unable to find on DB, due to: (conn=122596) Table 'cloud.usage_job' doesn't exist`. I did not know what i mess up. usage.err ``` SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Exception in thread "main" java.lang.RuntimeException: Unable to start com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_480aaa40 at com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(ComponentContext.java:155) at com.cloud.usage.UsageServer.start(UsageServer.java:55) at com.cloud.usage.UsageServer.main(UsageServer.java:42) Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to find on DB, due to: (conn=122596) Table 'cloud.usage_job' doesn't exist at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:441) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$searchIncludingRemoved$51(<generated>) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.searchIncludingRemoved(<generated>) at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:368) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$searchIncludingRemoved$50(<generated>) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.searchIncludingRemoved(<generated>) at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:357) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$search$18(<generated>) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.search(<generated>) at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:1463) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$search$16(<generated>) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.search(<generated>) at com.cloud.usage.dao.UsageJobDaoImpl.getLastJob(UsageJobDaoImpl.java:150) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$getLastJob$5(<generated>) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.getLastJob(<generated>) at com.cloud.usage.UsageManagerImpl.start(UsageManagerImpl.java:344) at com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(ComponentContext.java:149) ... 2 more ``` usage.out ``` 08:43:49.671 [Usage-HB-1] ERROR com.cloud.usage.UsageManagerImpl.Heartbeat - error in heartbeat com.cloud.utils.exception.CloudRuntimeException: Unable to find on DB, due to: (conn=122596) Table 'cloud.usage_job' doesn't exist at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:441) ~[cloud-framework-db-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$searchIncludingRemoved$51(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[cglib-nodep-3.3.0.jar:?] at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) ~[cloud-utils-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.searchIncludingRemoved(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:368) ~[cloud-framework-db-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$searchIncludingRemoved$50(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[cglib-nodep-3.3.0.jar:?] at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) ~[cloud-utils-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.searchIncludingRemoved(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:357) ~[cloud-framework-db-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$search$18(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[cglib-nodep-3.3.0.jar:?] at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) ~[cloud-utils-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.search(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:1463) ~[cloud-framework-db-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$search$16(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[cglib-nodep-3.3.0.jar:?] at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) ~[cloud-utils-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.search(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl.getNextImmediateJob(UsageJobDaoImpl.java:178) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$getNextImmediateJob$6(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[cglib-nodep-3.3.0.jar:?] at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) ~[cloud-utils-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.getNextImmediateJob(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0] at com.cloud.usage.UsageManagerImpl$Heartbeat.runInContext(UsageManagerImpl.java:2192) ~[cloudstack-usage.jar:4.20.0.0] at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) ~[cloud-framework-managed-context-4.20.0.0.jar:4.20.0.0] at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) ~[cloud-framework-managed-context-4.20.0.0.jar:4.20.0.0] at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) ~[cloud-framework-managed-context-4.20.0.0.jar:4.20.0.0] at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) ~[cloud-framework-managed-context-4.20.0.0.jar:4.20.0.0] at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) ~[cloud-framework-managed-context-4.20.0.0.jar:4.20.0.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] 08:44:49.659 [Usage-HB-1] ERROR com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628 - DB Exception on: HikariProxyPreparedStatement@438883728 wrapping ServerPreparedStatement{sql:'SELECT usage_job.id, usage_job.host, usage> java.sql.SQLSyntaxErrorException: (conn=122596) Table 'cloud.usage_job' doesn't exist at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:282) ~[mariadb-java-client-3.1.4.jar:?] at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:370) ~[mariadb-java-client-3.1.4.jar:?] at org.mariadb.jdbc.message.client.PrepareExecutePacket.readPacket(PrepareExecutePacket.java:153) ~[mariadb-java-client-3.1.4.jar:?] ``` I have tried to remove and re-install the cloudstack-usage service but the issue is still the same The management server log also have similar log every few minutes management-server.log ``` 2025-05-24 05:22:12,322 ERROR [c.c.u.d.UsageJobDaoImpl] (HA-3:[ctx-7cfd0587]) (logid:585b600e) DB Exception on: HikariProxyPreparedStatement@1302901535 wrapping ServerPreparedStatement{sql:'SELECT usage_job.id, usage_job.host, usage_job.pid, usage_job.job_type, usage_job.scheduled, usage_job.start_millis, usage_job.end_millis, usage_job.exec_time, usage_job.start_date, usage_job.end_date, usage_job.success, usage_job.heartbeat FROM usage_job ORDER BY usage_job.heartbeat DESC LIMIT 0, 1', parameters:[]} java.sql.SQLSyntaxErrorException: (conn=121197) Table 'cloud.usage_job' doesn't exist at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:282) at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:370) at org.mariadb.jdbc.message.client.PrepareExecutePacket.readPacket(PrepareExecutePacket.java:153) at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:872) at org.mariadb.jdbc.client.impl.StandardClient.readResults(StandardClient.java:811) at org.mariadb.jdbc.client.impl.StandardClient.execute(StandardClient.java:670) at org.mariadb.jdbc.client.impl.MultiPrimaryClient.execute(MultiPrimaryClient.java:347) at org.mariadb.jdbc.ServerPreparedStatement.executePipeline(ServerPreparedStatement.java:117) at org.mariadb.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:95) at org.mariadb.jdbc.ServerPreparedStatement.executeQuery(ServerPreparedStatement.java:431) at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java) at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:434) at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:368) at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:357) at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:1463) at com.cloud.usage.dao.UsageJobDaoImpl.getLastHeartbeat(UsageJobDaoImpl.java:190) at jdk.internal.reflect.GeneratedMethodAccessor382.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:569) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) at jdk.proxy3/jdk.proxy3.$Proxy302.getLastHeartbeat(Unknown Source) at com.cloud.ha.HighAvailabilityManagerExtImpl$UsageServerMonitorTask.runInContext(HighAvailabilityManagerExtImpl.java:73) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecuto ``` Anyone has idea on how i can resolve the above issue or why it happened ? I am using cloudstack 4.20. GitHub link: https://github.com/apache/cloudstack/discussions/10914 ---- This is an automatically sent email for users@cloudstack.apache.org. To unsubscribe, please send an email to: users-unsubscr...@cloudstack.apache.org
