can you check the /etc/cloudstack-release in ssvm ? -Wei
On Tuesday, 21 February 2023, Antoine Boucher <[email protected]> wrote: > Here are the logs from daemon.log of the failing SSVM > > Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed to locate > executable /usr/local/cloud/systemvm/_run.sh: No such file or directory > > Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed at step EXEC > spawning /usr/local/cloud/systemvm/_run.sh: No such file or directory > > Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Main process exited, > code=exited, status=203/EXEC > > Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Failed with result > 'exit-code'. > > Feb 21 02:36:16 systemvm systemd[1]: systemd-update-utmp-runlevel.service: > Succeeded. > > Feb 21 02:36:16 systemvm systemd[1]: Finished Update UTMP about System > Runlevel Changes. > > Feb 21 02:36:16 systemvm systemd[1]: Startup finished in 2.285s (kernel) + > 2min 16.468s (userspace) = 2min 18.754s. > > Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Scheduled restart job, > restart counter is at 1. > > Feb 21 02:36:16 systemvm systemd[1]: Stopped CloudStack Agent service. > > This repeats 5 times and fails > > The /usr/local/cloud/systemvm/ directory does not exist on the SSVM. > > > > > *Antoine Boucher* > [email protected] > [o] +1-226-505-9734 > www.haltondc.com > > “Data security made simple” > > > [image: HDClogo7-small.png] > > > Confidentiality Warning: This message and any attachments are intended > only for the use of the intended recipient(s), are confidential, and may be > privileged. If you are not the intended recipient, you are hereby notified > that any review, retransmission, conversion to hard copy, > copying, circulation or other use of this message and any attachments is > strictly prohibited. If you are not the intended recipient, please notify > the sender immediately by return e-mail, and delete this message and any > attachments from your system. > > > On Feb 21, 2023, at 7:53 AM, Antoine Boucher <[email protected]> > wrote: > > They are not old they are the new ones. I have deleted them a number of > times after the upgrade but the issue prevails. > > Antoine Boucher > [email protected] > [o] +1-226-505-9734 > www.haltondc.com > > “Data security made simple and affordable” > > > On Feb 21, 2023, at 00:24, Rohit Yadav <[email protected]> wrote: > > > You can destroy these old systemvms or do a stop and start on them. > > Regards. > > > > > ------------------------------ > *From:* Antoine Boucher <[email protected]> > *Sent:* Tuesday, February 21, 2023 10:40:46 AM > *To:* users <[email protected]> > *Subject:* Failed to setup certificates for system vm error on 2nd Zone > after upgrading to 4.17.2 > > After upgrading my two zone ACS from 4.16.2 to 4.17.2, the system VM (and > VRs) of the first zone upgraded without issues but the system VMs (s-318-VM > and v-317-VM) of the second zone (Kitchener1) are no longer able to > establish connection with the management server. I have rebooted the host > they are on, deleted the system vm to be recreated, I also logged into the > two system VM and found nothing so far except fo the following management > server logs: > > *2023-02-20 21:34:10,063 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) > (logid:5d1ede3b) Failed to setup certificate for system vm: s-318-VM* > *2023-02-20 21:34:10,747 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) Failed to setup certificate for system vm: v-317-VM* > > The host of these 2 system VM was missed in the original update, such that > is was updated after the Management Server was running with 4.17.2 instead > of before. > > Would anyone have suggestion to resolve the issue? > > > [image: Screenshot 2023-02-20 at 11.32.41 PM.png] > > > 2023-02-20 21:34:09,599 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-14:ctx-502066d6 > job-20366/job-20400 ctx-d7f724f4) (logid:352477d9) Seq > 44-1267200345151373322: Received: { Ans: , MgmtId: 130593671224, via: > 44(kit1-kvm03), Ver: v1, Flags: 10, { StartAnswer, CheckSshAnswer } } > 2023-02-20 21:34:09,623 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) VM instance {id: "317", name: "v-317-VM", uuid: > "0fbbd751-3bbe-46f6-bec0-e2887516fa6a", type="ConsoleProxy"} state > transited from [Starting] to [Running] with event [OperationSucceeded]. > VM's original host: Host {"id": "44", "name": "kit1-kvm03", "uuid": > "3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}, new host: Host > {"id": "44", "name": "kit1-kvm03", "uuid": > "3bde1185-8db6-414a-9c59-f788a2e0e9cf", > "type"="Routing"}, host before state transition: Host {"id": "44", "name": > "kit1-kvm03", "uuid": "3bde1185-8db6-414a-9c59-f788a2e0e9cf", > "type"="Routing"} > 2023-02-20 21:34:09,625 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-15:ctx-34675425 > job-20367/job-20401 ctx-d67c7cd1) (logid:5d1ede3b) Seq > 44-1267200345151373326: Sending { Cmd , MgmtId: 130593671224, via: > 44(kit1-kvm03), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.ca. > SetupCertificateCommand":{"handleByAgent":"false","accessDetails":{" > router.name":"s-318-VM","Control":"169.254.254.63","Storage":"10.91.6. > 249","Public":"64.32.40.68","Management":"10.91.0.244"," > router.ip":"169.254.254.63"},"wait":"60","bypassHostMaintenance":"false"}}] > } > 2023-02-20 21:34:09,626 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) Start completed for VM VM instance {id: "317", name: > "v-317-VM", uuid: "0fbbd751-3bbe-46f6-bec0-e2887516fa6a", > type="ConsoleProxy"} > 2023-02-20 21:34:10,063 DEBUG [c.c.a.t.Request] > (AgentManager-Handler-8:null) (logid:) Seq 44-1267200345151373326: > Processing: { Ans: , MgmtId: 130593671224, via: 44, Ver: v1, Flags: 10, [{" > org.apache.cloudstack.ca.SetupCertificateAnswer":{" > result":"false","wait":"0","bypassHostMaintenance":"false"}}] } > 2023-02-20 21:34:10,063 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-15:ctx-34675425 > job-20367/job-20401 ctx-d67c7cd1) (logid:5d1ede3b) Seq > 44-1267200345151373326: Received: { Ans: , MgmtId: 130593671224, via: > 44(kit1-kvm03), Ver: v1, Flags: 10, { SetupCertificateAnswer } } > *2023-02-20 21:34:10,063 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) > (logid:5d1ede3b) Failed to setup certificate for system vm: s-318-VM* > 2023-02-20 21:34:10,064 DEBUG [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) > (logid:5d1ede3b) Done executing VM work job: com.cloud.vm.VmWorkStart{" > dcId":0,"userId":1,"accountId":1,"vmId":318,"handlerName":" > VirtualMachineManagerImpl"} > 2023-02-20 21:34:10,071 DEBUG [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401) (logid:5d1ede3b) > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 318, job > origin: 20367 > 2023-02-20 21:34:10,072 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401) (logid:5d1ede3b) > Remove job-20401 from job monitoring > 2023-02-20 21:34:10,081 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] > (secstorage-1:ctx-bedcece7) (logid:304260be) received secondary storage vm > alert > 2023-02-20 21:34:10,082 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] > (secstorage-1:ctx-bedcece7) (logid:304260be) Secondary Storage Vm is up, > zone: Kitchener1, secStorageVm: s-318-VM, public > 2023-02-20 21:34:10,083 WARN [c.c.a.AlertManagerImpl] > (secstorage-1:ctx-bedcece7) (logid:304260be) alertType=[19] > dataCenterId=[4] podId=[4] clusterId=[null] message=[Secondary Storage Vm > up in zone: Kitchener1, secStorageVm: s-318-VM, public IP: 64.32.40.68, > private IP: 10.91.0.244]. > 2023-02-20 21:34:10,085 INFO [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-bedcece7) (logid:304260be) Secondary storage VM instance > {id: "318", name: "s-318-VM", uuid: "be765f00-9b3f-47c8-b056-6a41e7befb87", > type="SecondaryStorageVm"} was started. > 2023-02-20 21:34:10,085 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] > (secstorage-1:ctx-bedcece7) (logid:304260be) Primary secondary storage is > not even started, wait until next turn > 2023-02-20 21:34:10,433 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-14:ctx-502066d6 > job-20366/job-20400 ctx-d7f724f4) (logid:352477d9) Seq > 44-1267200345151373327: Sending { Cmd , MgmtId: 130593671224, via: > 44(kit1-kvm03), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.ca. > SetupCertificateCommand":{"handleByAgent":"false","accessDetails":{" > router.name":"v-317-VM","Control":"169.254.21.136","Public":"64.32.40.66" > ,"Management":"10.91.0.237","router.ip":"169.254.21.136"},"wait":"60"," > bypassHostMaintenance":"false"}}] } > 2023-02-20 21:34:10,622 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-342:ctx-4fd75e0f) (logid:63ad79d6) Seq 1-4502473727463655978: > Executing request > 2023-02-20 21:34:10,639 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-342:ctx-4fd75e0f) (logid:c3a060c9) Seq 1-4502473727463655978: > Response Received: > 2023-02-20 21:34:10,639 DEBUG [c.c.a.t.Request] > (StatsCollector-3:ctx-597a0e4b) > (logid:c3a060c9) Seq 1-4502473727463655978: Received: { Ans: , MgmtId: > 130593671224, via: 1(cs-xcp01), Ver: v1, Flags: 10, { GetVolumeStatsAnswer > } } > 2023-02-20 21:34:10,639 DEBUG [c.c.a.m.AgentManagerImpl] > (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing > class com.cloud.agent.api.GetVolumeStatsCommand: > 2023-02-20 21:34:10,682 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-86:ctx-520aafa5) (logid:65ac2df2) Seq 2-4712172585113094682: > Executing request > 2023-02-20 21:34:10,724 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-86:ctx-520aafa5) (logid:c3a060c9) Seq 2-4712172585113094682: > Response Received: > 2023-02-20 21:34:10,724 DEBUG [c.c.a.t.Request] > (StatsCollector-3:ctx-597a0e4b) > (logid:c3a060c9) Seq 2-4712172585113094682: Received: { Ans: , MgmtId: > 130593671224, via: 2(cs-xcp02), Ver: v1, Flags: 10, { GetVolumeStatsAnswer > } } > 2023-02-20 21:34:10,724 DEBUG [c.c.a.m.AgentManagerImpl] > (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing > class com.cloud.agent.api.GetVolumeStatsCommand: > 2023-02-20 21:34:10,747 DEBUG [c.c.a.t.Request] > (AgentManager-Handler-3:null) (logid:) Seq 44-1267200345151373327: > Processing: { Ans: , MgmtId: 130593671224, via: 44, Ver: v1, Flags: 10, [{" > org.apache.cloudstack.ca.SetupCertificateAnswer":{" > result":"false","wait":"0","bypassHostMaintenance":"false"}}] } > 2023-02-20 21:34:10,747 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-14:ctx-502066d6 > job-20366/job-20400 ctx-d7f724f4) (logid:352477d9) Seq > 44-1267200345151373327: Received: { Ans: , MgmtId: 130593671224, via: > 44(kit1-kvm03), Ver: v1, Flags: 10, { SetupCertificateAnswer } } > *2023-02-20 21:34:10,747 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) Failed to setup certificate for system vm: v-317-VM* > 2023-02-20 21:34:10,748 DEBUG [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) > (logid:352477d9) Done executing VM work job: com.cloud.vm.VmWorkStart{" > dcId":0,"userId":1,"accountId":1,"vmId":317,"handlerName":" > VirtualMachineManagerImpl"} > 2023-02-20 21:34:10,756 DEBUG [c.c.a.t.Request] > (StatsCollector-3:ctx-597a0e4b) > (logid:c3a060c9) Seq 9-6833649484581308478: Received: { Ans: , MgmtId: > 130593671224, via: 9(cs-kvm01), Ver: v1, Flags: 10, { GetVolumeStatsAnswer > } } > 2023-02-20 21:34:10,756 DEBUG [c.c.a.m.AgentManagerImpl] > (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing > class com.cloud.agent.api.GetVolumeStatsCommand: > 2023-02-20 21:34:10,762 DEBUG [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400) (logid:352477d9) > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 317, job > origin: 20366 > 2023-02-20 21:34:10,764 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400) (logid:352477d9) > Remove job-20400 from job monitoring > 2023-02-20 21:34:10,773 INFO [c.c.c.ConsoleProxyManagerImpl] > (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) Console proxy v-317-VM is > started > 2023-02-20 21:34:10,773 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-2:null) (logid:) SeqA 45-791665: Processing Seq > 45-791665: { Cmd , MgmtId: -1, via: 45, Ver: v1, Flags: 11, > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand" > :{"_proxyVmId":"290","_loadInfo":"{ > "connections": [] > }","wait":"0","bypassHostMaintenance":"false"}}] } > 2023-02-20 21:34:10,773 DEBUG [c.c.a.ConsoleProxyAlertAdapter] > (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) received console proxy alert > 2023-02-20 21:34:10,775 DEBUG [c.c.a.ConsoleProxyAlertAdapter] > (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) Console proxy is up, zone: > Kitchener1, proxy: v-317-VM > 2023-02-20 21:34:10,776 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-2:null) (logid:) SeqA 45-791665: Sending Seq > 45-791665: { Ans: , MgmtId: 130593671224, via: 45, Ver: v1, Flags: 100010, > [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0"," > bypassHostMaintenance":"false"}}] } > 2023-02-20 21:34:10,776 WARN [c.c.a.AlertManagerImpl] > (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) alertType=[10] > dataCenterId=[4] podId=[4] clusterId=[null] message=[Console proxy up in > zone: Kitchener1, proxy: v-317-VM, public IP: 64.32.40.66, private IP: > 10.91.0.237]. > > > > Regards, > *Antoine Boucher* > > >
