GitHub user muthukrishnang1100 added a comment to the discussion: OOBM Issue
during KVM HA
when i enable OOBM on first host it will affect the third host error like this
2026-01-03 08:26:15,452 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Vm-Operations-Cleanup-1:ctx-541bbd5f) (logid:45ace2f5) VM Operation Thread
Running
2026-01-03 08:26:15,481 DEBUG [o.a.c.v.s.VMSchedulerImpl]
(VMSchedulerPollTask:ctx-ecbf1dfd) (logid:c5c0bab1) VM scheduler.poll is being
called at 2026-01-03 08:26:00 GMT
2026-01-03 08:26:15,492 DEBUG [o.a.c.v.s.VMSchedulerImpl]
(VMSchedulerPollTask:ctx-ecbf1dfd) (logid:c5c0bab1) Got 0 scheduled jobs to be
executed at 2026-01-03 08:26:00 GMT
2026-01-03 08:26:15,497 INFO [o.a.c.v.s.VMSchedulerImpl]
(VMSchedulerPollTask:ctx-ecbf1dfd) (logid:c5c0bab1) Cleaned up 0 VM scheduled
job entries
2026-01-03 08:26:15,738 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-1:null)
(logid:f750332a) Preparing command [/usr/bin/ipmitool -I lanplus -R 1 -v -H
192.168.10.117 -p 623 -U ADMIN -P ***** chassis power status] to execute.
2026-01-03 08:26:15,739 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-1:null)
(logid:f750332a) Submitting command [/usr/bin/ipmitool -I lanplus -R 1 -v -H
192.168.10.117 -p 623 -U ADMIN -P ***** chassis power status].
2026-01-03 08:26:15,739 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-1:null)
(logid:f750332a) Waiting for a response from command [/usr/bin/ipmitool -I
lanplus -R 1 -v -H 192.168.10.117 -p 623 -U ADMIN -P ***** chassis power
status]. Defined timeout: [60].
2026-01-03 08:26:15,802 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-1:null)
(logid:f750332a) Process standard output for command [/usr/bin/ipmitool -I
lanplus -R 1 -v -H 192.168.10.117 -p 623 -U ADMIN -P ***** chassis power
status]: [Chassis Power is on
].
2026-01-03 08:26:15,802 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-1:null)
(logid:f750332a) Process standard error output command [/usr/bin/ipmitool -I
lanplus -R 1 -v -H 192.168.10.117 -p 623 -U ADMIN -P ***** chassis power
status]: [Loading IANA PEN Registry...
Using best available cipher suite 3
Running Get VSO Capabilities my_addr 0x20, transit 0, target 0x20
Invalid completion code received: Invalid command
Discovered IPMB address 0x0
].
2026-01-03 08:28:15,834 DEBUG [o.a.c.o.d.i.IpmitoolOutOfBandManagementDriver]
(pool-1-thread-2:null) (logid:05dbe44f) The command [/usr/bin/ipmitool -I
lanplus -R 1 -v -H 192.168.10.117 -p **** -U ADMIN -P **** chassis power
status] was successful and got the result [Chassis Power is on].
2026-01-03 08:28:16,790 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-67852447) (logid:e29c6d96) HA health check
task is running...
2026-01-03 08:28:18,528 DEBUG [c.c
------------------------------------------------------------------------------------------------------------------------------------------------------
After wait some minutes the DB error comes on the same third host
2026-01-03 08:29:51,056 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
(qtp341748265-313:ctx-9e12f4f5 ctx-fc18c527) (logid:1e5886ee) API rate limiting
is disabled. We will not use ApiRateLimitService.
2026-01-03 08:29:51,057 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
(qtp341748265-22:ctx-db55deee ctx-762f2a41) (logid:93061ece) API rate limiting
is disabled. We will not use ApiRateLimitService.
2026-01-03 08:29:51,066 DEBUG [c.c.a.ApiServlet] (qtp341748265-313:ctx-9e12f4f5
ctx-fc18c527) (logid:1e5886ee) ===END=== 10.1.12.54 -- GET
page=1&pagesize=100&details=min&listall=true&showicon=true&command=listProjects&response=json&sessionkey=Id__9JN7wYWDV2te4E-9s5S5HXo
2026-01-03 08:29:51,084 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1058)
at com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:1035)
at com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:981)
at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy97.findById(Unknown Source)
at
org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.commandCanBypassHostMaintenance(CloudStackPrimaryDataStoreDriverImpl.java:227)
at
org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.deleteAsync(CloudStackPrimaryDataStoreDriverImpl.java:238)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.expungeVolumeAsync(VolumeServiceImpl.java:440)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.cleanupVolumes(VolumeOrchestrator.java:1286)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy241.cleanupVolumes(Unknown Source)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:667)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:600)
at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:2462)
at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:3357)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy186.destroyVm(Unknown Source)
at
org.apache.cloudstack.api.command.user.vm.DestroyVMCmd.execute(DestroyVMCmd.java:137)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications
link failure
The last packet successfully received from the server was 1 milliseconds ago.
The last packet sent successfully to the server was 1 milliseconds ago.
at
com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:175)
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at
com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2022)
... 69 more
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications
link failure
The last packet successfully received from the server was 1 milliseconds ago.
The last packet sent successfully to the server was 1 milliseconds ago.
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
... 81 more
Caused by: java.net.SocketException: Connection reset
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at
com.mysql.cj.protocol.ReadAheadInputStream.fill(ReadAheadInputStream.java:107)
... 87 more
2026-01-03 08:29:51,091 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Complete async
job-1578, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"DB
Exception on: null"}
2026-01-03 08:29:51,092 DEBUG [c.c.a.ApiServlet] (qtp341748265-328:ctx-386dbf87
ctx-14af538e) (logid:0cd05586) ===END=== 10.1.12.54 -- GET
listall=true&details=group,nics,secgrp,tmpl,servoff,diskoff,iso,volume,affgrp,backoff&isvnf=false&page=1&pagesize=20&showIcon=true&command=listVirtualMachines&response=json&sessionkey=Id__9JN7wYWDV2te4E-9s5S5HXo
2026-01-03 08:29:51,094 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Publish async
job-1578 complete on message bus
2026-01-03 08:29:51,094 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Wake up jobs
related to job-1578
2026-01-03 08:29:51,094 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Update db status
for job-1578
2026-01-03 08:29:51,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Wake up jobs joined
with job-1578 and disjoin all subjobs created from job- 1578
2026-01-03 08:29:51,102 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Done executing
org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin for job-1578
--------------------------------------------------------------------------------------------------------------------------------------------------------
after first host error like this
2026-01-03 08:41:02,803 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-10:null) (logid:) Seq 3-6899233154154889263: MgmtId
200525778825546: Resp: Routing to peer
2026-01-03 08:41:02,803 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-10:null) (logid:) Seq 3-6899233154154889263: No more
commands found
2026-01-03 08:41:02,907 WARN [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-7:null) (logid:) Caught:
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:864)
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:819)
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:1490)
at
com.cloud.vm.dao.ConsoleProxyDaoImpl.update(ConsoleProxyDaoImpl.java:138)
at jdk.internal.reflect.GeneratedMethodAccessor282.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy191.update(Unknown Source)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.updateConsoleProxyStatus(ConsoleProxyManagerImpl.java:1655)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl$VmBasedAgentHook.onLoadReport(ConsoleProxyManagerImpl.java:282)
at
com.cloud.consoleproxy.ConsoleProxyListener.processControlCommand(ConsoleProxyListener.java:56)
at
com.cloud.agent.manager.AgentManagerImpl.handleControlCommand(AgentManagerImpl.java:307)
at
com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1408)
at
com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1499)
at
com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:709)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout
waiting for idle object
at
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:147)
at
com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at
com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
... 30 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:356)
at
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:141)
... 32 more
2026-01-03 08:41:02,908 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-7:null) (logid:) SeqA 10-10207: Sending Seq 10-10207: {
Ans: , MgmtId: 270790104347868, via: 10, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"DB Exception on:
null","wait":"0","bypassHostMaintenance":"false"}}] }
2026-01-03 08:41:02,910 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-14:null) (logid:) SeqA 10-10214: Processing Seq 10-10214:
{ Cmd , MgmtId: -1, via: 10, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"2","_loadInfo":"{
"connections": [],
"removedSessions": []
}","wait":"0","bypassHostMaintenance":"false"}}] }
2026-01-03 08:41:04,810 ERROR [c.c.s.StatsCollector]
(StatsCollector-3:ctx-d4726101) (logid:85fb8547) Error trying to retrieve VM
stats
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at
com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:433)
at
com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:362)
at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:351)
at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:1411)
at jdk.internal.reflect.GeneratedMethodAccessor101.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy96.search(Unknown Source)
at
com.cloud.server.StatsCollector$VmStatsCollector.runInContext(StatsCollector.java:1193)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout
waiting for idle object
at
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:147)
at
com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at
com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
... 29 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:356)
at
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:141)
-------------------------------------------------------------------------------------------------------------------------------------------------------
After this i cant able to access the browser dasboard for host 1
<img width="1887" height="992" alt="Screenshot 2026-01-03 141214"
src="https://github.com/user-attachments/assets/7dc555e4-8159-433a-8399-3c5a098928d5";
/>
and also this error shows for 2nd and 3rd host
2026-01-03 08:42:47,318 DEBUG [c.c.s.StatsCollector]
(StatsCollector-2:ctx-59cb42e3) (logid:1448cbc8) AutoScaling Monitor is
running...
2026-01-03 08:42:47,322 DEBUG [c.c.s.StatsCollector]
(StatsCollector-2:ctx-59cb42e3) (logid:1448cbc8) Skipping AutoScaling Monitor
2026-01-03 08:42:47,445 DEBUG [c.c.s.d.VolumeStatsDaoImpl]
(StatsCollector-6:ctx-0dc9cd57) (logid:7ed4936e) Starting to remove all
volume_stats rows older than [Fri Jan 02 20:42:47 UTC 2026].
2026-01-03 08:42:47,449 INFO [c.c.s.d.VolumeStatsDaoImpl]
(StatsCollector-6:ctx-0dc9cd57) (logid:7ed4936e) Removed a total of [0]
volume_stats rows older than [Fri Jan 02 20:42:47 UTC 2026].
2026-01-03 08:42:47,516 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-c6510625) (logid:9d8ba108) Detected management node
left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:47,516 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-c6510625) (logid:9d8ba108) Trying to connect to
192.168.170.25
2026-01-03 08:42:47,517 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-c6510625) (logid:9d8ba108) Management node 1 is
detected inactive by timestamp but is pingable
2026-01-03 08:42:47,934 DEBUG [c.c.v.d.VmStatsDaoImpl]
(StatsCollector-5:ctx-878cdf24) (logid:9ab5fc97) Starting to remove all
vm_stats rows older than [Fri Jan 02 20:42:47 UTC 2026].
2026-01-03 08:42:47,957 INFO [c.c.v.d.VmStatsDaoImpl]
(StatsCollector-5:ctx-878cdf24) (logid:9ab5fc97) Removed a total of [4]
vm_stats rows older than [Fri Jan 02 20:42:47 UTC 2026].
2026-01-03 08:42:48,265 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-747221a1) (logid:6609a8eb) HA health check
task is running...
2026-01-03 08:42:49,016 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-1d71274d) (logid:ea11cb26) Detected management node
left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:49,016 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-1d71274d) (logid:ea11cb26) Trying to connect to
192.168.170.25
2026-01-03 08:42:49,016 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-1d71274d) (logid:ea11cb26) Management node 1 is
detected inactive by timestamp but is pingable
2026-01-03 08:42:50,377 DEBUG [c.c.a.ApiServlet]
(qtp341748265-330:ctx-ff157751) (logid:bb22092b) ===START=== 10.1.12.54 -- GET
command=readyForShutdown&response=json&sessionkey=HUEhk8e7DRkUnItZrBXJj8wcyPY
2026-01-03 08:42:50,377 DEBUG [c.c.a.ApiServlet]
(qtp341748265-330:ctx-ff157751) (logid:bb22092b) Two factor authentication is
already verified for the user 2, so skipping
2026-01-03 08:42:50,442 DEBUG [c.c.a.ApiServer] (qtp341748265-330:ctx-ff157751
ctx-107a9660) (logid:bb22092b) CIDRs from which account 'Account
[{"accountName":"admin","id":2,"uuid":"373ce4b1-e2fc-11f0-99f1-f6483bfa74dc"}]'
is allowed to perform API calls: 0.0.0.0/0,::/0
2026-01-03 08:42:50,459 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker]
(qtp341748265-330:ctx-ff157751 ctx-107a9660) (logid:bb22092b) RoleService is
enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-01-03 08:42:50,461 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
(qtp341748265-330:ctx-ff157751 ctx-107a9660) (logid:bb22092b) API rate limiting
is disabled. We will not use ApiRateLimitService.
2026-01-03 08:42:50,468 DEBUG [c.c.a.ApiServlet] (qtp341748265-330:ctx-ff157751
ctx-107a9660) (logid:bb22092b) ===END=== 10.1.12.54 -- GET
command=readyForShutdown&response=json&sessionkey=HUEhk8e7DRkUnItZrBXJj8wcyPY
2026-01-03 08:42:50,517 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-c00bc642) (logid:691dfd91) Detected management node
left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:50,517 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-c00bc642) (logid:691dfd91) Trying to connect to
192.168.170.25
2026-01-03 08:42:50,518 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-c00bc642) (logid:691dfd91) Management node 1 is
detected inactive by timestamp but is pingable
2026-01-03 08:42:51,028 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-fc33bca6) (logid:6a1d468e) Enabled non-edge zones available
for scan: 3
2026-01-03 08:42:51,031 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-fc33bca6) (logid:6a1d468e) Zone [3] is not ready to launch
secondary storage VM.
2026-01-03 08:42:51,238 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-6f970a3d) (logid:f1022f8c) Enabled non-edge zones available
for scan: 3
2026-01-03 08:42:51,241 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-6f970a3d) (logid:f1022f8c) Zone 3 is not ready to launch
console proxy yet
2026-01-03 08:42:52,017 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-a950b9e6) (logid:1307e641) Detected management node
left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:52,017 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-a950b9e6) (logid:1307e641) Trying to connect to
192.168.170.25
2026-01-03 08:42:52,018 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-a950b9e6) (logid:1307e641) Management node 1 is
detected inactive by timestamp but is pingable
2026-01-03 08:42:52,315 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-9faba4d1) (logid:b77f852d) HA health check
task is running...
2026-01-03 08:42:53,517 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-d870df33) (logid:78abf199) Detected management node
left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:53,517 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-d870df33) (logid:78abf199) Trying to connect to
192.168.170.25
2026-01-03 08:42:53,517 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-d870df33) (logid:78abf199) Management node 1 is
detected inactive by timestamp but is pingable
2026-01-03 08:42:55,016 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-61681d00) (logid:9cab442a) Detected management node
left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:55,016 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-61681d00) (logid:9cab442a) Trying to connect to
192.168.170.25
2026-01-03 08:42:55,016 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-61681d00) (logid:9cab442a) Management node 1 is
detected inactive by timestamp but is pingable
2026-01-03 08:42:55,379 DEBUG [c.c.a.ApiServlet] (qtp341748265-23:ctx-71bde22b)
(logid:736b2752) ===START=== 10.1.12.54 -- GET
command=readyForShutdown&response=json&sessionkey=HUEhk8e7DRkUnItZrBXJj8wcyPY
2026-01-03 08:42:55,379 DEBUG [c.c.a.ApiServlet] (qtp341748265-23:ctx-71bde22b)
(logid:736b2752) Two factor authentication is already verified for the user 2,
so skipping
2026-01-03 08:42:55,412 DEBUG [c.c.a.ApiServer] (qtp341748265-23:ctx-71bde22b
ctx-6d4332bd) (logid:736b2752) CIDRs from which account 'Account
[{"accountName":"admin","id":2,"uuid":"373ce4b1-e2fc-11f0-99f1-f6483bfa74dc"}]'
is allowed to perform API calls: 0.0.0.0/0,::/0
2026-01-03 08:42:55,429 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker]
(qtp341748265-23:ctx-71bde22b ctx-6d4332bd) (logid:736b2752) RoleService is
enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-01-03 08:42:55,431 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
(qtp341748265-23:ctx-71bde22b ctx-6d4332bd) (logid:736b2752) API rate limiting
is disabled. We will not use ApiRateLimitService.
2026-01-03 08:42:55,438 DEBUG [c.c.a.ApiServlet] (qtp341748265-23:ctx-71bde22b
ctx-6d4332bd) (logid:736b2752) ===END=== 10.1.12.54 -- GET
command=readyForShutdown&response=json&sessionkey=HUEhk8e7DRkUnItZrBXJj8wcyPY
2026-01-03 08:42:56,371 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-f757e80f) (logid:50d360f6) HA health check
task is running...
2026-01-03 08:42:56,517 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-53e4ff9e) (logid:cb381e1c) Detected management node
left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:56,517 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-53e4ff9e) (logid:cb381e1c) Trying to connect to
192.168.170.25
2026-01-03 08:42:56,517 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-53e4ff9e) (logid:cb381e1c) Management node 1 is
detected inactive by timestamp but is pingable
2026-01-03 08:42:58,017 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-077362ee) (logid:75659ed9) Detected management node
left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:58,017 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-077362ee) (logid:75659ed9) Trying to connect to
192.168.170.25
2026-01-03 08:42:58,018 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-077362ee) (logid:75659ed9) Management node 1 is
detected inactive by timestamp but is pingable
2026-01-03 08:42:59,518 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-985e3a16) (logid:5bea8e90) Detected management node
left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:59,518 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-985e3a16) (logid:5bea8e90) Trying to connect to
192.168.170.25
2026-01-03 08:42:59,518 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-985e3a16) (logid:5bea8e90) Management node 1 is
detected inactive by timestamp but is pingable
2026-01-03 08:43:00,428 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-9b2ad256) (logid:d5b4b84f) HA health check
task is running...
2026-01-03 08:43:01,016 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-9290a1cc) (logid:e7200c92) Detected management node
left, id:1, nodeIP:192.168.170.25
2026-01-03 08:43:01,016 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-9290a1cc) (logid:e7200c92) Trying to connect to
192.168.170.25
2026-01-03 08:43:01,017 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-9290a1cc) (logid:e7200c92) Management node 1 is
detected inactive by timestamp but is pingable
2026-01-03 08:43:02,367 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-f7f04c55) (logid:b1bb1740) Found 0 routers to update
status.
2026-01-03 08:43:02,376 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-f7f04c55) (logid:b1bb1740) Found 1 VPC's to update
Redundant State.
2026-01-03 08:43:02,380 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-f7f04c55) (logid:b1bb1740) Found 0 networks to
update RvR status.
2026-01-03 08:43:02,386 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RedundantRouterStatusMonitor-5:ctx-a441b866) (logid:3b5c05ec) Skip router pair
(r-11-VM,r-12-VM) due to not belong to this mgmt server
2026-01-03 08:43:02,512 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-72950dce) (logid:a71762c1) Detected management node
left, id:1, nodeIP:192.168.170.25
2026-01-03 08:43:02,512 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-72950dce) (logid:a71762c1) Trying to connect to
192.168.170.25
2026-01-03 08:43:02,513 INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-72950dce) (logid:a71762c1) Manag
GitHub link:
https://github.com/apache/cloudstack/discussions/12353#discussioncomment-15396143
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
