Hi,
a week ago we discovered an issue with the usage server version (4.9).
After around two month from upgrading to 4.9.0 usage server stopped working and
produced following error messages in /var/log/cloudstack/usage/usage.log:
ERROR [usage.dao.UsageDaoImpl] (Usage-Job-1:null) (logid:) error saving account
to cloud_usage db
java.lang.NullPointerException
at com.cloud.usage.dao.UsageDaoImpl.saveAccounts(UsageDaoImpl.java:116)
at
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7.CGLIB$saveAccounts$15(<generated>)
at
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7_FastClassByCloudStack_80eceb8.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:122)
at
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7.saveAccounts(<generated>)
at com.cloud.usage.UsageManagerImpl.parse(UsageManagerImpl.java:516)
at
com.cloud.usage.UsageManagerImpl.runInContextInternal(UsageManagerImpl.java:384)
at
com.cloud.usage.UsageManagerImpl$1.runInContext(UsageManagerImpl.java:326)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at com.cloud.usage.UsageManagerImpl.run(UsageManagerImpl.java:323)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
ERROR [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) Usage Manager
error
com.cloud.utils.exception.CloudRuntimeException
at com.cloud.usage.dao.UsageDaoImpl.saveAccounts(UsageDaoImpl.java:135)
at
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7.CGLIB$saveAccounts$15(<generated>)
at
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7_FastClassByCloudStack_80eceb8.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:122)
at
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7.saveAccounts(<generated>)
at com.cloud.usage.UsageManagerImpl.parse(UsageManagerImpl.java:516)
at
com.cloud.usage.UsageManagerImpl.runInContextInternal(UsageManagerImpl.java:384)
at
com.cloud.usage.UsageManagerImpl$1.runInContext(UsageManagerImpl.java:326)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at com.cloud.usage.UsageManagerImpl.run(UsageManagerImpl.java:323)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
This happened on our testing and prod environment on Ubuntu.
We also tried to fix it with the migrate-dynamicroles.py script, but that
didn't fixed it.
The problem was gone after we did an Upgrade to 4.9.2.
So if someone has this problem too then upgrade to 4.9.2.
BR
Alexander
