kiranchavala opened a new issue, #13174: URL: https://github.com/apache/cloudstack/issues/13174
### problem Unable to remove vcenter/esxi host from cloudstack with force/cleanly if the vcenter is deleted externally ### versions ACS 4.22 ### The steps to reproduce the bug 1. Set up Cloudstack env with a vcenter 2. Vcenter which is got deleted externally 3. Admin user goes to the Ui > Zones > Remove vmware details from zone <img width="1623" height="159" alt="Image" src="https://github.com/user-attachments/assets/a854a0cd-c0d3-4c47-8a71-e29d45c7ee30"; /> Exception : <img width="1400" height="609" alt="Image" src="https://github.com/user-attachments/assets/ce0b4ebc-b37c-48c6-9dbb-e88ee3726c56"; /> Logs ``` 2026-05-18 06:08:03,827 DEBUG [c.c.a.ApiServlet] (qtp2038105753-25:[ctx-04dda2ab]) (logid:f6e52355) ===START=== 10.0.3.251 -- POST command=removeVmwareDc response=json zoneid=7696337c-f532-41a9-a37a-a974b6fff240 sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8 2026-05-18 06:08:03,827 DEBUG [c.c.a.ApiServlet] (qtp2038105753-25:[ctx-04dda2ab]) (logid:f6e52355) Two factor authentication is already verified for the user 2, so skipping 2026-05-18 06:08:03,832 DEBUG [c.c.a.ApiServer] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"970585ac-4f5e-11f1-9840-1e00e50001f3"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2026-05-18 06:08:03,834 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) Account for user id 97067b1a-4f5e-11f1-9840-1e00e50001f3 is Root Admin or Domain Admin, all APIs are allowed. 2026-05-18 06:08:03,834 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2026-05-18 06:08:03,834 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) API rate limiting is disabled. We will not use ApiRateLimitService. 2026-05-18 06:08:03,840 WARN [o.a.c.a.c.a.z.RemoveVmwareDcCmd] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) The zone has one or more resources (like cluster), hence not able to remove VMware datacenter from zone. Please remove all resource from zone, and retry. Exception: com.cloud.exception.ResourceInUseException: Zone has one or more clusters. Can't remove VMware datacenter to zone which already has clusters. at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.validateZoneWithResources(VmwareManagerImpl.java:1433) at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.removeVmwareDatacenter(VmwareManagerImpl.java:1350) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ``` 4. Next admin tries to delete the host in the vmware cluster from cloudstack by keeping the host in maintenance mode. There is no option to force delete the host <img width="1252" height="637" alt="Image" src="https://github.com/user-attachments/assets/36d3d336-1a0a-4834-8ba8-83e912ed01b0"; /> <img width="468" height="258" alt="Image" src="https://github.com/user-attachments/assets/be24e90e-e0ad-495d-a3e0-0d2d5abded69"; /> logs ``` 2026-05-18 06:14:03,175 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-c7171a5b]) (logid:3f7a543d) ===START=== 10.0.3.251 -- POST command=prepareHostForMaintenance response=json id=4a0c6597-86e8-4409-adfe-63970f27ca09 sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8 2026-05-18 06:14:03,175 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-c7171a5b]) (logid:3f7a543d) Two factor authentication is already verified for the user 2, so skipping 2026-05-18 06:14:03,179 DEBUG [c.c.a.ApiServer] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"970585ac-4f5e-11f1-9840-1e00e50001f3"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2026-05-18 06:14:03,180 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) Account for user id 97067b1a-4f5e-11f1-9840-1e00e50001f3 is Root Admin or Domain Admin, all APIs are allowed. 2026-05-18 06:14:03,180 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2026-05-18 06:14:03,180 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) API rate limiting is disabled. We will not use ApiRateLimitService. 2026-05-18 06:14:03,193 DEBUG [c.c.a.ApiServer] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) Retrieved cmdEventType from job info: MAINT.PREPARE 2026-05-18 06:14:03,195 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-33:[ctx-1f2955e7, job-41]) (logid:1e20138f) Add job-41 into job monitoring 2026-05-18 06:14:03,197 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) submit async job-41, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.host.PrepareForHostMaintenanceCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"QAZorHbuhisqM_pl6NxnR1wC3b8\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"154\",\"id\":\"4a0c6597-86e8-4409-adfe-63970f27ca09\",\"ctxDetails\":\"{\\\"interface com.cloud.host.Host\\\":\\\"4a0c6597-86e8-4409-adfe-63970f27ca09\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"4a0c6597-86e8-4409-adfe-63970f27ca09\",\"cmdEventType\":\"MAINT.PREPARE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":41,"initMsid":32989190816243,"instanceId":1,"instanceType":"Host","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"8aa5c307-73fc-4cad-bd3b-5caaeafb483a"} 2026-05-18 06:14:03,198 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) ===END=== 10.0.3.251 -- POST command=prepareHostForMaintenance response=json id=4a0c6597-86e8-4409-adfe-63970f27ca09 sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8 2026-05-18 06:14:03,198 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-33:[ctx-1f2955e7, job-41]) (logid:8aa5c307) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.host.PrepareForHostMaintenanceCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"QAZorHbuhisqM_pl6NxnR1wC3b8\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"154\",\"id\":\"4a0c6597-86e8-4409-adfe-63970f27ca09\",\"ctxDetails\":\"{\\\"interface com.cloud.host.Host\\\":\\\"4a0c6597-86e8-4409-adfe-63970f27ca09\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"4a0c6597-86e8-4409-adfe-63970f27ca09\",\"cmdEventType\":\"MAINT.PREPARE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":41,"initMsid":32989190816243,"instanceId":1,"instanceType":"Host","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"8aa5c307-73fc-4cad-bd3b-5caaeafb483a"} 2026-05-18 06:14:03,213 INFO [c.c.r.ResourceManagerImpl] (API-Job-Executor-33:[ctx-1f2955e7, job-41, ctx-c137d8d5]) (logid:8aa5c307) Maintenance: attempting maintenance of host Host {"id":1,"name":"10.0.32.191","type":"Routing","uuid":"4a0c6597-86e8-4409-adfe-63970f27ca09"} 2026-05-18 06:14:03,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-33:[ctx-1f2955e7, job-41, ctx-c137d8d5]) (logid:8aa5c307) Can not send command com.cloud.agent.api.MaintainCommand due to Host Host {"id":1,"name":"10.0.32.191","type":"Routing","uuid":"4a0c6597-86e8-4409-adfe-63970f27ca09"} not being up 2026-05-18 06:14:03,216 WARN [c.c.r.ResourceManagerImpl] (API-Job-Executor-33:[ctx-1f2955e7, job-41, ctx-c137d8d5]) (logid:8aa5c307) Unable to send MaintainCommand to host: Host {"id":1,"name":"10.0.32.191","type":"Routing","uuid":"4a0c6597-86e8-4409-adfe-63970f27ca09"} ``` 5. Next admin tries to delete the primary storage associated with the VMware cluster <img width="1618" height="551" alt="Image" src="https://github.com/user-attachments/assets/8cf8d2c8-217e-464e-804d-084f7a7d2129"; /> logs ``` 2026-05-18 06:18:34,608 DEBUG [c.c.s.StorageManagerImpl] (qtp2038105753-23:[ctx-eef67450, ctx-7f6db4e9]) (logid:2da339a0) Cannot delete storage pool StoragePool {"id":1,"name":"ref-trl-11757-v-Mol9-harikrishna-patnala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"4ac4c575-ace8-391c-892a-b207bc1248c6"} as the following non-destroyed volumes are on it: [Volume [36d6c20e-33f6-4403-a4fb-64efa7e2cdde] (attached to VM [4195861e-b559-40fc-8829-61217d75d9da])]. 2026-05-18 06:18:34,608 ERROR [c.c.a.ApiServer] (qtp2038105753-23:[ctx-eef67450, ctx-7f6db4e9]) (logid:2da339a0) unhandled exception executing api command: [Ljava.lang.String;@7c5be73a com.cloud.utils.exception.CloudRuntimeException: Cannot delete pool StoragePool {"id":1,"name":"ref-trl-11757-v-Mol9-harikrishna-patnala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"4ac4c575-ace8-391c-892a-b207bc1248c6"} as there are non-destroyed volumes associated to this pool. at com.cloud.storage.StorageManagerImpl.deleteDataStoreInternal(StorageManagerImpl.java:1765) ``` 6. Admin user tries to expunge the vm which is having volumes in the primary storage <img width="1614" height="400" alt="Image" src="https://github.com/user-attachments/assets/9e18df4c-1239-41d3-952d-00e0d624d790"; /> logs ``` 2026-05-18 06:20:16,910 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Cleaning storage for VM [VM instance {"id":3,"instanceName":"i-2-3-VM","state":"Expunging","type":"User","uuid":"4195861e-b559-40fc-8829-61217d75d9da"}]. 2026-05-18 06:20:16,911 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Skipping destroy for the volume [{"name":"ROOT-3","uuid":"36d6c20e-33f6-4403-a4fb-64efa7e2cdde"}] as it is in [Destroy] state. 2026-05-18 06:20:16,931 DEBUG [c.c.h.XenServerGuru] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) We are returning the default host to execute commands because the command is not of Copy type. 2026-05-18 06:20:16,931 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Wait time setting on org.apache.cloudstack.storage.command.DeleteCommand is 1800 seconds 2026-05-18 06:20:16,932 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Failed to send command, due to Agent:2, com.cloud.exception.AgentUnavailableException: Resource [Host:2] is unreachable: Host 2: Host with specified id is not in the right state: Alert 2026-05-18 06:20:16,932 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Unable to destroy volume [id: 3, uuid: 36d6c20e-33f6-4403-a4fb-64efa7e2cdde] com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:2, com.cloud.exception.AgentUnavailableException: Resource [Host:2] is unreachable: Host 2: Host with specified id is not in the right state: Alert at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:141) at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.deleteAsync(CloudStackPrimaryDataStoreDriverImpl.java:260) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.expungeVolumeAsync(VolumeServiceImpl.java:459) at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.cleanupVolumes(VolumeOrchestrator.java:1357) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ``` 7. User Tries to destroy the volume <img width="1624" height="585" alt="Image" src="https://github.com/user-attachments/assets/f6fab222-f147-4958-bea4-6918eb22c05a"; /> Logs ``` 2026-05-18 06:24:41,199 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-7f941028]) (logid:ffa4161f) ===START=== 10.0.3.251 -- POST command=deleteVolume response=json id=36d6c20e-33f6-4403-a4fb-64efa7e2cdde sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8 2026-05-18 06:24:41,199 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-7f941028]) (logid:ffa4161f) Two factor authentication is already verified for the user 2, so skipping 2026-05-18 06:24:41,204 DEBUG [c.c.a.ApiServer] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"970585ac-4f5e-11f1-9840-1e00e50001f3"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2026-05-18 06:24:41,206 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) Account for user id 97067b1a-4f5e-11f1-9840-1e00e50001f3 is Root Admin or Domain Admin, all APIs are allowed. 2026-05-18 06:24:41,206 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2026-05-18 06:24:41,206 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) API rate limiting is disabled. We will not use ApiRateLimitService. 2026-05-18 06:24:41,213 DEBUG [c.c.u.AccountManagerImpl] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) Account [Account [{"accountName":"admin","id":2,"uuid":"970585ac-4f5e-11f1-9840-1e00e50001f3"}]] has access to resource. 2026-05-18 06:24:41,229 INFO [c.c.a.ApiServer] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) Please specify a volume that is not attached to any VM. 2026-05-18 06:24:41,230 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) ===END=== 10.0.3.251 -- POST command=deleteVolume response=json id=36d6c20e-33f6-4403-a4fb-64efa7e2cdde sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8 ``` Admin user has to manually execute the DB queries to remove the vcenter details from the cloudstack zone ### What to do about it? Expected behaviour Provide an option of force removing the vcenter details from cloudstack if the vcenter is destroyed externally and is unrecoverable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
