kiranchavala opened a new issue, #12256: URL: https://github.com/apache/cloudstack/issues/12256
### problem Unable to destroy/expunge systemvm in a multihypervisor cluster/environment ### versions ACS 4.22 ### The steps to reproduce the bug 1. Create a Cloudstack environment with 2 clusters ( vmware , KVM) 2. Launch systemvm's on the vmware cluster, make sure they are in running state 3. Shutdown the vmware vcenter cluster , the system vm's will be running state , but the agent will be in a disconnected state 4. Mark the zone is disabled and try to stop or destroy the systemvm's 5. Exception observed <img width="1632" height="734" alt="Image" src="https://github.com/user-attachments/assets/e205eeec-c2e6-4157-8c3c-6c007918195f"; /> Logs ``` 2025-12-15 05:08:35,162 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Wait time setting on org.apache.cloudstack.storage.command.DeleteCommand is 1800 seconds 2025-12-15 05:08:35,162 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected 2025-12-15 05:08:35,162 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-14:[ctx-63c6fb05, job-152, ctx-8ae693e9]) (logid:cb1a3aae) Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected 2025-12-15 05:08:35,162 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Unable to destroy volume [id: 9, uuid: 5113804b-2cd3-42ce-b76f-bdfa97885503] com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:141) at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.deleteAsync(CloudStackPrimaryDataStoreDriverImpl.java:259) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.expungeVolumeAsync(VolumeServiceImpl.java:444) at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.cleanupVolumes(VolumeOrchestrator.java:1355) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:569) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ... Tried reducing the value of the expunge.delay and expunge.interval Workaround Manually update the database and set the removed date mysql>update vm_instance set removed=now() where id=<>; ### What to do about it? In order to deploy the systemvm's on the kvm cluster end user has to update the database Provide an option to force remove the systemvm which are struck or introduce a zone level setting which can specify the cluster for the systemvm deployment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
