I compared the system template fro KVM of both zone and they seem identical.
The /usr/local/cloud/systemvm/ directory exist on SSVM on zone 1 exist but the /usr/local/cloud/ directory of the zone 2 SSVM does not exist? Antoine Boucher [email protected] [o] +1-226-505-9734 www.haltondc.com “Data security made simple”  Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system. > On Feb 21, 2023, at 1:31 PM, Antoine Boucher <[email protected]> wrote: > > 4.17.2 > > Antoine Boucher > [email protected] > [o] +1-226-505-9734 > www.haltondc.com > > “Data security made simple and affordable” > > > On Feb 21, 2023, at 11:51, Wei ZHOU <[email protected]> wrote: > > can you check the /etc/cloudstack-release in ssvm ? > > -Wei > > On Tuesday, 21 February 2023, Antoine Boucher <[email protected] > <mailto:[email protected]>> wrote: >> Here are the logs from daemon.log of the failing SSVM >> >> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed to locate >> executable /usr/local/cloud/systemvm/_run.sh: No such file or directory >> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed at step EXEC >> spawning /usr/local/cloud/systemvm/_run.sh: No such file or directory >> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Main process exited, >> code=exited, status=203/EXEC >> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Failed with result >> 'exit-code'. >> Feb 21 02:36:16 systemvm systemd[1]: systemd-update-utmp-runlevel.service: >> Succeeded. >> Feb 21 02:36:16 systemvm systemd[1]: Finished Update UTMP about System >> Runlevel Changes. >> Feb 21 02:36:16 systemvm systemd[1]: Startup finished in 2.285s (kernel) + >> 2min 16.468s (userspace) = 2min 18.754s. >> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Scheduled restart job, >> restart counter is at 1. >> Feb 21 02:36:16 systemvm systemd[1]: Stopped CloudStack Agent service. >> >> This repeats 5 times and fails >> >> The /usr/local/cloud/systemvm/ directory does not exist on the SSVM. >> >> >> >> >> Antoine Boucher >> [email protected] <mailto:[email protected]> >> [o] +1-226-505-9734 >> www.haltondc.com <http://www.haltondc.com/> >> >> “Data security made simple” >> >> >> >> >> >> Confidentiality Warning: This message and any attachments are intended only >> for the use of the intended recipient(s), are confidential, and may be >> privileged. If you are not the intended recipient, you are hereby notified >> that any review, retransmission, conversion to hard copy, copying, >> circulation or other use of this message and any attachments is strictly >> prohibited. If you are not the intended recipient, please notify the sender >> immediately by return e-mail, and delete this message and any attachments >> from your system. >> >> >>> On Feb 21, 2023, at 7:53 AM, Antoine Boucher <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> They are not old they are the new ones. I have deleted them a number of >>> times after the upgrade but the issue prevails. >>> >>> Antoine Boucher >>> [email protected] <mailto:[email protected]> >>> [o] +1-226-505-9734 >>> www.haltondc.com <http://www.haltondc.com/> >>> >>> “Data security made simple and affordable” >>> >>> >>> On Feb 21, 2023, at 00:24, Rohit Yadav <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> >>> You can destroy these old systemvms or do a stop and start on them. >>> >>> Regards. >>> >>> >>> >>> >>> From: Antoine Boucher <[email protected] <mailto:[email protected]>> >>> Sent: Tuesday, February 21, 2023 10:40:46 AM >>> To: users <[email protected] <mailto:[email protected]>> >>> Subject: Failed to setup certificates for system vm error on 2nd Zone after >>> upgrading to 4.17.2 >>> >>> After upgrading my two zone ACS from 4.16.2 to 4.17.2, the system VM (and >>> VRs) of the first zone upgraded without issues but the system VMs (s-318-VM >>> and v-317-VM) of the second zone (Kitchener1) are no longer able to >>> establish connection with the management server. I have rebooted the host >>> they are on, deleted the system vm to be recreated, I also logged into the >>> two system VM and found nothing so far except fo the following management >>> server logs: >>> >>> 2023-02-20 21:34:10,063 ERROR [c.c.v.VirtualMachineManagerImpl] >>> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) >>> (logid:5d1ede3b) Failed to setup certificate for system vm: s-318-VM >>> 2023-02-20 21:34:10,747 ERROR [c.c.v.VirtualMachineManagerImpl] >>> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >>> (logid:352477d9) Failed to setup certificate for system vm: v-317-VM >>> >>> The host of these 2 system VM was missed in the original update, such that >>> is was updated after the Management Server was running with 4.17.2 instead >>> of before. >>> >>> Would anyone have suggestion to resolve the issue? >>> >>> >>> >>> >>> >>> 2023-02-20 21:34:09,599 DEBUG [c.c.a.t.Request] >>> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >>> (logid:352477d9) Seq 44-1267200345151373322: Received: { Ans: , MgmtId: >>> 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { StartAnswer, >>> CheckSshAnswer } } >>> 2023-02-20 21:34:09,623 DEBUG [c.c.c.CapacityManagerImpl] >>> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >>> (logid:352477d9) VM instance {id: "317", name: "v-317-VM", uuid: >>> "0fbbd751-3bbe-46f6-bec0-e2887516fa6a", type="ConsoleProxy"} state >>> transited from [Starting] to [Running] with event [OperationSucceeded]. >>> VM's original host: Host {"id": "44", "name": "kit1-kvm03", "uuid": >>> "3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}, new host: Host >>> {"id": "44", "name": "kit1-kvm03", "uuid": >>> "3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}, host before >>> state transition: Host {"id": "44", "name": "kit1-kvm03", "uuid": >>> "3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"} >>> 2023-02-20 21:34:09,625 DEBUG [c.c.a.t.Request] >>> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) >>> (logid:5d1ede3b) Seq 44-1267200345151373326: Sending { Cmd , MgmtId: >>> 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 100011, >>> [{"org.apache.cloudstack.ca >>> <http://org.apache.cloudstack.ca/>.SetupCertificateCommand":{"handleByAgent":"false","accessDetails":{"router.name >>> >>> <http://router.name/>":"s-318-VM","Control":"169.254.254.63","Storage":"10.91.6.249","Public":"64.32.40.68","Management":"10.91.0.244","router.ip":"169.254.254.63"},"wait":"60","bypassHostMaintenance":"false"}}] >>> } >>> 2023-02-20 21:34:09,626 DEBUG [c.c.v.VirtualMachineManagerImpl] >>> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >>> (logid:352477d9) Start completed for VM VM instance {id: "317", name: >>> "v-317-VM", uuid: "0fbbd751-3bbe-46f6-bec0-e2887516fa6a", >>> type="ConsoleProxy"} >>> 2023-02-20 21:34:10,063 DEBUG [c.c.a.t.Request] >>> (AgentManager-Handler-8:null) (logid:) Seq 44-1267200345151373326: >>> Processing: { Ans: , MgmtId: 130593671224, via: 44, Ver: v1, Flags: 10, >>> [{"org.apache.cloudstack.ca >>> <http://org.apache.cloudstack.ca/>.SetupCertificateAnswer":{"result":"false","wait":"0","bypassHostMaintenance":"false"}}] >>> } >>> 2023-02-20 21:34:10,063 DEBUG [c.c.a.t.Request] >>> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) >>> (logid:5d1ede3b) Seq 44-1267200345151373326: Received: { Ans: , MgmtId: >>> 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { >>> SetupCertificateAnswer } } >>> 2023-02-20 21:34:10,063 ERROR [c.c.v.VirtualMachineManagerImpl] >>> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) >>> (logid:5d1ede3b) Failed to setup certificate for system vm: s-318-VM >>> 2023-02-20 21:34:10,064 DEBUG [c.c.v.VmWorkJobHandlerProxy] >>> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) >>> (logid:5d1ede3b) Done executing VM work job: >>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":318,"handlerName":"VirtualMachineManagerImpl"} >>> 2023-02-20 21:34:10,071 DEBUG [c.c.v.VmWorkJobDispatcher] >>> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401) (logid:5d1ede3b) >>> Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 318, job >>> origin: 20367 >>> 2023-02-20 21:34:10,072 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401) (logid:5d1ede3b) >>> Remove job-20401 from job monitoring >>> 2023-02-20 21:34:10,081 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] >>> (secstorage-1:ctx-bedcece7) (logid:304260be) received secondary storage vm >>> alert >>> 2023-02-20 21:34:10,082 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] >>> (secstorage-1:ctx-bedcece7) (logid:304260be) Secondary Storage Vm is up, >>> zone: Kitchener1, secStorageVm: s-318-VM, public >>> 2023-02-20 21:34:10,083 WARN [c.c.a.AlertManagerImpl] >>> (secstorage-1:ctx-bedcece7) (logid:304260be) alertType=[19] >>> dataCenterId=[4] podId=[4] clusterId=[null] message=[Secondary Storage Vm >>> up in zone: Kitchener1, secStorageVm: s-318-VM, public IP: 64.32.40.68, >>> private IP: 10.91.0.244]. >>> 2023-02-20 21:34:10,085 INFO [o.a.c.s.SecondaryStorageManagerImpl] >>> (secstorage-1:ctx-bedcece7) (logid:304260be) Secondary storage VM instance >>> {id: "318", name: "s-318-VM", uuid: "be765f00-9b3f-47c8-b056-6a41e7befb87", >>> type="SecondaryStorageVm"} was started. >>> 2023-02-20 21:34:10,085 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] >>> (secstorage-1:ctx-bedcece7) (logid:304260be) Primary secondary storage is >>> not even started, wait until next turn >>> 2023-02-20 21:34:10,433 DEBUG [c.c.a.t.Request] >>> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >>> (logid:352477d9) Seq 44-1267200345151373327: Sending { Cmd , MgmtId: >>> 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 100011, >>> [{"org.apache.cloudstack.ca >>> <http://org.apache.cloudstack.ca/>.SetupCertificateCommand":{"handleByAgent":"false","accessDetails":{"router.name >>> >>> <http://router.name/>":"v-317-VM","Control":"169.254.21.136","Public":"64.32.40.66","Management":"10.91.0.237","router.ip":"169.254.21.136"},"wait":"60","bypassHostMaintenance":"false"}}] >>> } >>> 2023-02-20 21:34:10,622 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-342:ctx-4fd75e0f) (logid:63ad79d6) Seq 1-4502473727463655978: >>> Executing request >>> 2023-02-20 21:34:10,639 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-342:ctx-4fd75e0f) (logid:c3a060c9) Seq 1-4502473727463655978: >>> Response Received: >>> 2023-02-20 21:34:10,639 DEBUG [c.c.a.t.Request] >>> (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Seq 1-4502473727463655978: >>> Received: { Ans: , MgmtId: 130593671224, via: 1(cs-xcp01), Ver: v1, Flags: >>> 10, { GetVolumeStatsAnswer } } >>> 2023-02-20 21:34:10,639 DEBUG [c.c.a.m.AgentManagerImpl] >>> (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing >>> class com.cloud.agent.api.GetVolumeStatsCommand: >>> 2023-02-20 21:34:10,682 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-86:ctx-520aafa5) (logid:65ac2df2) Seq 2-4712172585113094682: >>> Executing request >>> 2023-02-20 21:34:10,724 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-86:ctx-520aafa5) (logid:c3a060c9) Seq 2-4712172585113094682: >>> Response Received: >>> 2023-02-20 21:34:10,724 DEBUG [c.c.a.t.Request] >>> (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Seq 2-4712172585113094682: >>> Received: { Ans: , MgmtId: 130593671224, via: 2(cs-xcp02), Ver: v1, Flags: >>> 10, { GetVolumeStatsAnswer } } >>> 2023-02-20 21:34:10,724 DEBUG [c.c.a.m.AgentManagerImpl] >>> (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing >>> class com.cloud.agent.api.GetVolumeStatsCommand: >>> 2023-02-20 21:34:10,747 DEBUG [c.c.a.t.Request] >>> (AgentManager-Handler-3:null) (logid:) Seq 44-1267200345151373327: >>> Processing: { Ans: , MgmtId: 130593671224, via: 44, Ver: v1, Flags: 10, >>> [{"org.apache.cloudstack.ca >>> <http://org.apache.cloudstack.ca/>.SetupCertificateAnswer":{"result":"false","wait":"0","bypassHostMaintenance":"false"}}] >>> } >>> 2023-02-20 21:34:10,747 DEBUG [c.c.a.t.Request] >>> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >>> (logid:352477d9) Seq 44-1267200345151373327: Received: { Ans: , MgmtId: >>> 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { >>> SetupCertificateAnswer } } >>> 2023-02-20 21:34:10,747 ERROR [c.c.v.VirtualMachineManagerImpl] >>> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >>> (logid:352477d9) Failed to setup certificate for system vm: v-317-VM >>> 2023-02-20 21:34:10,748 DEBUG [c.c.v.VmWorkJobHandlerProxy] >>> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >>> (logid:352477d9) Done executing VM work job: >>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":317,"handlerName":"VirtualMachineManagerImpl"} >>> 2023-02-20 21:34:10,756 DEBUG [c.c.a.t.Request] >>> (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Seq 9-6833649484581308478: >>> Received: { Ans: , MgmtId: 130593671224, via: 9(cs-kvm01), Ver: v1, Flags: >>> 10, { GetVolumeStatsAnswer } } >>> 2023-02-20 21:34:10,756 DEBUG [c.c.a.m.AgentManagerImpl] >>> (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing >>> class com.cloud.agent.api.GetVolumeStatsCommand: >>> 2023-02-20 21:34:10,762 DEBUG [c.c.v.VmWorkJobDispatcher] >>> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400) (logid:352477d9) >>> Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 317, job >>> origin: 20366 >>> 2023-02-20 21:34:10,764 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400) (logid:352477d9) >>> Remove job-20400 from job monitoring >>> 2023-02-20 21:34:10,773 INFO [c.c.c.ConsoleProxyManagerImpl] >>> (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) Console proxy v-317-VM is >>> started >>> 2023-02-20 21:34:10,773 DEBUG [c.c.a.m.AgentManagerImpl] >>> (AgentManager-Handler-2:null) (logid:) SeqA 45-791665: Processing Seq >>> 45-791665: { Cmd , MgmtId: -1, via: 45, Ver: v1, Flags: 11, >>> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"290","_loadInfo":"{ >>> "connections": [] >>> }","wait":"0","bypassHostMaintenance":"false"}}] } >>> 2023-02-20 21:34:10,773 DEBUG [c.c.a.ConsoleProxyAlertAdapter] >>> (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) received console proxy alert >>> 2023-02-20 21:34:10,775 DEBUG [c.c.a.ConsoleProxyAlertAdapter] >>> (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) Console proxy is up, zone: >>> Kitchener1, proxy: v-317-VM >>> 2023-02-20 21:34:10,776 DEBUG [c.c.a.m.AgentManagerImpl] >>> (AgentManager-Handler-2:null) (logid:) SeqA 45-791665: Sending Seq >>> 45-791665: { Ans: , MgmtId: 130593671224, via: 45, Ver: v1, Flags: 100010, >>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] >>> } >>> 2023-02-20 21:34:10,776 WARN [c.c.a.AlertManagerImpl] >>> (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) alertType=[10] >>> dataCenterId=[4] podId=[4] clusterId=[null] message=[Console proxy up in >>> zone: Kitchener1, proxy: v-317-VM, public IP: 64.32.40.66, private IP: >>> 10.91.0.237]. >>> >>> >>> >>> Regards, >>> Antoine Boucher >>> >>
