Here are the logs from daemon.log of the failing SSVM Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed to locate executable /usr/local/cloud/systemvm/_run.sh: No such file or directory Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed at step EXEC spawning /usr/local/cloud/systemvm/_run.sh: No such file or directory Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Main process exited, code=exited, status=203/EXEC Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Failed with result 'exit-code'. Feb 21 02:36:16 systemvm systemd[1]: systemd-update-utmp-runlevel.service: Succeeded. Feb 21 02:36:16 systemvm systemd[1]: Finished Update UTMP about System Runlevel Changes. Feb 21 02:36:16 systemvm systemd[1]: Startup finished in 2.285s (kernel) + 2min 16.468s (userspace) = 2min 18.754s. Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Scheduled restart job, restart counter is at 1. Feb 21 02:36:16 systemvm systemd[1]: Stopped CloudStack Agent service.
This repeats 5 times and fails The /usr/local/cloud/systemvm/ directory does not exist on the SSVM. Antoine Boucher [email protected] [o] +1-226-505-9734 www.haltondc.com “Data security made simple”  Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system. > On Feb 21, 2023, at 7:53 AM, Antoine Boucher <[email protected]> wrote: > > They are not old they are the new ones. I have deleted them a number of > times after the upgrade but the issue prevails. > > Antoine Boucher > [email protected] > [o] +1-226-505-9734 > www.haltondc.com > > “Data security made simple and affordable” > > > On Feb 21, 2023, at 00:24, Rohit Yadav <[email protected]> wrote: > > > You can destroy these old systemvms or do a stop and start on them. > > Regards. > > > > > From: Antoine Boucher <[email protected]> > Sent: Tuesday, February 21, 2023 10:40:46 AM > To: users <[email protected]> > Subject: Failed to setup certificates for system vm error on 2nd Zone after > upgrading to 4.17.2 > > After upgrading my two zone ACS from 4.16.2 to 4.17.2, the system VM (and > VRs) of the first zone upgraded without issues but the system VMs (s-318-VM > and v-317-VM) of the second zone (Kitchener1) are no longer able to establish > connection with the management server. I have rebooted the host they are on, > deleted the system vm to be recreated, I also logged into the two system VM > and found nothing so far except fo the following management server logs: > > 2023-02-20 21:34:10,063 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) > (logid:5d1ede3b) Failed to setup certificate for system vm: s-318-VM > 2023-02-20 21:34:10,747 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) Failed to setup certificate for system vm: v-317-VM > > The host of these 2 system VM was missed in the original update, such that is > was updated after the Management Server was running with 4.17.2 instead of > before. > > Would anyone have suggestion to resolve the issue? > > > > > > 2023-02-20 21:34:09,599 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) Seq 44-1267200345151373322: Received: { Ans: , MgmtId: > 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { StartAnswer, > CheckSshAnswer } } > 2023-02-20 21:34:09,623 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) VM instance {id: "317", name: "v-317-VM", uuid: > "0fbbd751-3bbe-46f6-bec0-e2887516fa6a", type="ConsoleProxy"} state transited > from [Starting] to [Running] with event [OperationSucceeded]. VM's original > host: Host {"id": "44", "name": "kit1-kvm03", "uuid": > "3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}, new host: Host > {"id": "44", "name": "kit1-kvm03", "uuid": > "3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}, host before state > transition: Host {"id": "44", "name": "kit1-kvm03", "uuid": > "3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"} > 2023-02-20 21:34:09,625 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) > (logid:5d1ede3b) Seq 44-1267200345151373326: Sending { Cmd , MgmtId: > 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 100011, > [{"org.apache.cloudstack.ca.SetupCertificateCommand":{"handleByAgent":"false","accessDetails":{"router.name":"s-318-VM","Control":"169.254.254.63","Storage":"10.91.6.249","Public":"64.32.40.68","Management":"10.91.0.244","router.ip":"169.254.254.63"},"wait":"60","bypassHostMaintenance":"false"}}] > } > 2023-02-20 21:34:09,626 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) Start completed for VM VM instance {id: "317", name: > "v-317-VM", uuid: "0fbbd751-3bbe-46f6-bec0-e2887516fa6a", type="ConsoleProxy"} > 2023-02-20 21:34:10,063 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:null) > (logid:) Seq 44-1267200345151373326: Processing: { Ans: , MgmtId: > 130593671224, via: 44, Ver: v1, Flags: 10, > [{"org.apache.cloudstack.ca.SetupCertificateAnswer":{"result":"false","wait":"0","bypassHostMaintenance":"false"}}] > } > 2023-02-20 21:34:10,063 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) > (logid:5d1ede3b) Seq 44-1267200345151373326: Received: { Ans: , MgmtId: > 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { > SetupCertificateAnswer } } > 2023-02-20 21:34:10,063 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) > (logid:5d1ede3b) Failed to setup certificate for system vm: s-318-VM > 2023-02-20 21:34:10,064 DEBUG [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) > (logid:5d1ede3b) Done executing VM work job: > com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":318,"handlerName":"VirtualMachineManagerImpl"} > 2023-02-20 21:34:10,071 DEBUG [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401) (logid:5d1ede3b) Done > with run of VM work job: com.cloud.vm.VmWorkStart for VM 318, job origin: > 20367 > 2023-02-20 21:34:10,072 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401) (logid:5d1ede3b) > Remove job-20401 from job monitoring > 2023-02-20 21:34:10,081 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] > (secstorage-1:ctx-bedcece7) (logid:304260be) received secondary storage vm > alert > 2023-02-20 21:34:10,082 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] > (secstorage-1:ctx-bedcece7) (logid:304260be) Secondary Storage Vm is up, > zone: Kitchener1, secStorageVm: s-318-VM, public > 2023-02-20 21:34:10,083 WARN [c.c.a.AlertManagerImpl] > (secstorage-1:ctx-bedcece7) (logid:304260be) alertType=[19] dataCenterId=[4] > podId=[4] clusterId=[null] message=[Secondary Storage Vm up in zone: > Kitchener1, secStorageVm: s-318-VM, public IP: 64.32.40.68, private IP: > 10.91.0.244]. > 2023-02-20 21:34:10,085 INFO [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-bedcece7) (logid:304260be) Secondary storage VM instance > {id: "318", name: "s-318-VM", uuid: "be765f00-9b3f-47c8-b056-6a41e7befb87", > type="SecondaryStorageVm"} was started. > 2023-02-20 21:34:10,085 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] > (secstorage-1:ctx-bedcece7) (logid:304260be) Primary secondary storage is not > even started, wait until next turn > 2023-02-20 21:34:10,433 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) Seq 44-1267200345151373327: Sending { Cmd , MgmtId: > 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 100011, > [{"org.apache.cloudstack.ca.SetupCertificateCommand":{"handleByAgent":"false","accessDetails":{"router.name":"v-317-VM","Control":"169.254.21.136","Public":"64.32.40.66","Management":"10.91.0.237","router.ip":"169.254.21.136"},"wait":"60","bypassHostMaintenance":"false"}}] > } > 2023-02-20 21:34:10,622 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-342:ctx-4fd75e0f) (logid:63ad79d6) Seq 1-4502473727463655978: > Executing request > 2023-02-20 21:34:10,639 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-342:ctx-4fd75e0f) (logid:c3a060c9) Seq 1-4502473727463655978: > Response Received: > 2023-02-20 21:34:10,639 DEBUG [c.c.a.t.Request] > (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Seq 1-4502473727463655978: > Received: { Ans: , MgmtId: 130593671224, via: 1(cs-xcp01), Ver: v1, Flags: > 10, { GetVolumeStatsAnswer } } > 2023-02-20 21:34:10,639 DEBUG [c.c.a.m.AgentManagerImpl] > (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing class > com.cloud.agent.api.GetVolumeStatsCommand: > 2023-02-20 21:34:10,682 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-86:ctx-520aafa5) (logid:65ac2df2) Seq 2-4712172585113094682: > Executing request > 2023-02-20 21:34:10,724 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-86:ctx-520aafa5) (logid:c3a060c9) Seq 2-4712172585113094682: > Response Received: > 2023-02-20 21:34:10,724 DEBUG [c.c.a.t.Request] > (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Seq 2-4712172585113094682: > Received: { Ans: , MgmtId: 130593671224, via: 2(cs-xcp02), Ver: v1, Flags: > 10, { GetVolumeStatsAnswer } } > 2023-02-20 21:34:10,724 DEBUG [c.c.a.m.AgentManagerImpl] > (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing class > com.cloud.agent.api.GetVolumeStatsCommand: > 2023-02-20 21:34:10,747 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null) > (logid:) Seq 44-1267200345151373327: Processing: { Ans: , MgmtId: > 130593671224, via: 44, Ver: v1, Flags: 10, > [{"org.apache.cloudstack.ca.SetupCertificateAnswer":{"result":"false","wait":"0","bypassHostMaintenance":"false"}}] > } > 2023-02-20 21:34:10,747 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) Seq 44-1267200345151373327: Received: { Ans: , MgmtId: > 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { > SetupCertificateAnswer } } > 2023-02-20 21:34:10,747 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) Failed to setup certificate for system vm: v-317-VM > 2023-02-20 21:34:10,748 DEBUG [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) Done executing VM work job: > com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":317,"handlerName":"VirtualMachineManagerImpl"} > 2023-02-20 21:34:10,756 DEBUG [c.c.a.t.Request] > (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Seq 9-6833649484581308478: > Received: { Ans: , MgmtId: 130593671224, via: 9(cs-kvm01), Ver: v1, Flags: > 10, { GetVolumeStatsAnswer } } > 2023-02-20 21:34:10,756 DEBUG [c.c.a.m.AgentManagerImpl] > (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing class > com.cloud.agent.api.GetVolumeStatsCommand: > 2023-02-20 21:34:10,762 DEBUG [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400) (logid:352477d9) Done > with run of VM work job: com.cloud.vm.VmWorkStart for VM 317, job origin: > 20366 > 2023-02-20 21:34:10,764 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400) (logid:352477d9) > Remove job-20400 from job monitoring > 2023-02-20 21:34:10,773 INFO [c.c.c.ConsoleProxyManagerImpl] > (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) Console proxy v-317-VM is > started > 2023-02-20 21:34:10,773 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-2:null) (logid:) SeqA 45-791665: Processing Seq > 45-791665: { Cmd , MgmtId: -1, via: 45, Ver: v1, Flags: 11, > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"290","_loadInfo":"{ > "connections": [] > }","wait":"0","bypassHostMaintenance":"false"}}] } > 2023-02-20 21:34:10,773 DEBUG [c.c.a.ConsoleProxyAlertAdapter] > (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) received console proxy alert > 2023-02-20 21:34:10,775 DEBUG [c.c.a.ConsoleProxyAlertAdapter] > (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) Console proxy is up, zone: > Kitchener1, proxy: v-317-VM > 2023-02-20 21:34:10,776 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-2:null) (logid:) SeqA 45-791665: Sending Seq 45-791665: > { Ans: , MgmtId: 130593671224, via: 45, Ver: v1, Flags: 100010, > [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] > } > 2023-02-20 21:34:10,776 WARN [c.c.a.AlertManagerImpl] > (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) alertType=[10] > dataCenterId=[4] podId=[4] clusterId=[null] message=[Console proxy up in > zone: Kitchener1, proxy: v-317-VM, public IP: 64.32.40.66, private IP: > 10.91.0.237]. > > > > Regards, > Antoine Boucher >
