Hi David, Can you check in the logs if the volume allocation worked fine for SSVM? Is there a record present in the volumes table?
-Jithin From: David Larsen <[email protected]> Date: Thursday, 30 March 2023 at 1:31 AM To: [email protected] <[email protected]> Subject: SSVM - Getting "Unable to create deployment, no usable volumes found for the VM" after Cloudstack/Mysql/Ubuntu upgrade. Hi Today i upgraded our Cloudstack (4.17.2 --> 4.18.0) , mysql (5.7--> 8) and ubuntu (18.04-->22.04) in production. I thought I had tested it well in a lab... and of course the only important thing I couldn't test went wrong!! Secondary Storage VM... At first it looked fine. I successfully upgraded console proxy system vm to 4.18.0. But when I tried to do the same with secondarystoragevm, stop it and wait for it to upgrade, Cloudstack deleted the secondary systemvm from vmware and when it started the copy job it complained about a locked file on the vmware volume....(?) I didn't find any locked files in the s-1177-VM vm directory on the vmware volume. >From the error log bellow: "Unable to create deployment, no usable volumes >found for the VM: 1177" Every five vmware volumes have about 50% free space, so enough space is not an issue... In the case of locked file on vmware I have restarted vcenter and management agents on every host, so every locked files should be released. No change. Then I reverted to cloudstack 4.17.2 on ubuntu 22.04 on both management and mysql server hoping cloudstack would create the SSVM from scratch... it did not. Same error in the log and no S-xxxx-VM created... In Cloudstack dashboard it show S-1177-VM as stopped, but in vmware, it's gone... Everything else I have checked is running fine. Start/stop vm ++ Can it be any ubuntu 22.04 upgrade issues here? Until I upgraded today, everything was running fine on ubuntu 18.04, Cloudstack 4.17.2 and Mysql 5.7 Any new settings in db.properties regarding new mysql or ubuntu? 2023-03-29 20:58:18,866 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-0dd13ec7) (logid:c5bd94b0) Found a stopped secondary storage VM instance {id: "1177", name: "s-1177-VM", uuid: "602280ac-a809-4ac9-b229-ea43c0ac0145", type="SecondaryStorageVm"}, starting it. 2023-03-29 20:58:19,743 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-93:ctx-c7b556c4 job-12132/job-12228 ctx-8a5aa0c5) (logid:4c5d9052) orchestrating VM start for 's-1177-VM' com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d<mailto:com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d> set to null 2023-03-29 20:58:19,753 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-93:ctx-c7b556c4 job-12132/job-12228 ctx-8a5aa0c5) (logid:4c5d9052) VM instance {id: "1177", name: "s-1177-VM", uuid: "602280ac-a809-4ac9-b229-ea43c0ac0145", type="SecondaryStorageVm"} state transited from [Stopped] to [Starting] with event [StartRequested]. VM's original host: null, new host: null, host before state transition: null 2023-03-29 20:58:19,753 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-93:ctx-c7b556c4 job-12132/job-12228 ctx-8a5aa0c5) (logid:4c5d9052) Successfully transitioned to start state for VM instance {id: "1177", name: "s-1177-VM", uuid: "602280ac-a809-4ac9-b229-ea43c0ac0145", type="SecondaryStorageVm"} reservation id = 30f7264c-1598-45aa-a71e-b2b0f88a4519 2023-03-29 20:58:19,886 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-93:ctx-c7b556c4 job-12132/job-12228 ctx-8a5aa0c5) (logid:4c5d9052) VM instance {id: "1177", name: "s-1177-VM", uuid: "602280ac-a809-4ac9-b229-ea43c0ac0145", type="SecondaryStorageVm"} state transited from [Starting] to [Stopped] with event [OperationFailed]. VM's original host: null, new host: null, host before state transition: null 2023-03-29 20:58:19,887 WARN [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-93:ctx-c7b556c4 job-12132/job-12228 ctx-8a5aa0c5) (logid:4c5d9052) Unable to orchestrate start VM instance {id: "1177", name: "s-1177-VM", uuid: "602280ac-a809-4ac9-b229-ea43c0ac0145", type="SecondaryStorageVm"} due to [Unable to create deployment, no usable volumes found for the VM: 1177]. 2023-03-29 20:58:19,943 WARN [c.c.v.SystemVmLoadScanner] (secstorage-1:ctx-0dd13ec7) (logid:c5bd94b0) Unexpected exception Unable to orchestrate start VM instance {id: "1177", name: "s-1177-VM", uuid: "602280ac-a809-4ac9-b229-ea43c0ac0145", type="SecondaryStorageVm"} due to [Unable to create deployment, no usable volumes found for the VM: 1177]. com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate start VM instance {id: "1177", name: "s-1177-VM", uuid: "602280ac-a809-4ac9-b229-ea43c0ac0145", type="SecondaryStorageVm"} due to [Unable to create deployment, no usable volumes found for the VM: 1177]. Thanks -David
