Hi Wei, Thank you for your information. It turns out that the host was not properly updated to 4.17.2
All work now. Thanks again! Regards, Antoine > On Feb 22, 2023, at 7:51 AM, Wei ZHOU <ustcweiz...@gmail.com> wrote: > > The scripts are transferred from kvm host via scp command. It would be good > to check > - kvm host has cloudstack-agent installed and upgraded > - scp to ssvm/router vms via cloud0 is allowed on kvm host > > -Wei > > > > On Wednesday, 22 February 2023, Antoine Boucher <antoi...@haltondc.com > <mailto:antoi...@haltondc.com>> > wrote: > >> Is there any reason why the /usr/local/cloud/systemvm directory would be >> missing from the SSVM one one zone but not on an other? >> >> Is the directory added during the systemvm creation or is it part of the >> base system VM template? >> >> Both /etc/cloudstack-release show: Cloudstack Release 4.17.2 Fri 09 Dec >> 2022 12:51:18 PM UTC >> >> Regards, >> Antoine >> >> >> >>> On Feb 21, 2023, at 3:31 PM, Antoine Boucher <antoi...@haltondc.com> >> wrote: >>> >>> I compared the system template fro KVM of both zone and they seem >> identical. >>> >>> >>> The /usr/local/cloud/systemvm/ directory exist on SSVM on zone 1 exist >> but the /usr/local/cloud/ directory of the zone 2 SSVM does not exist? >>> >>> >>> >>> Antoine Boucher >>> antoi...@haltondc.com >>> [o] +1-226-505-9734 >>> www.haltondc.com >>> >>> “Data security made simple” >>> >>> >>> <HDClogo7-small.png> >>> >>> >>> Confidentiality Warning: This message and any attachments are intended >> only for the use of the intended recipient(s), are confidential, and may be >> privileged. If you are not the intended recipient, you are hereby notified >> that any review, retransmission, conversion to hard copy, copying, >> circulation or other use of this message and any attachments is strictly >> prohibited. If you are not the intended recipient, please notify the sender >> immediately by return e-mail, and delete this message and any attachments >> from your system. >>> >>> >>>> On Feb 21, 2023, at 1:31 PM, Antoine Boucher <antoi...@haltondc.com> >> wrote: >>>> >>>> 4.17.2 >>>> >>>> Antoine Boucher >>>> antoi...@haltondc.com >>>> [o] +1-226-505-9734 >>>> www.haltondc.com >>>> >>>> “Data security made simple and affordable” >>>> >>>> >>>> On Feb 21, 2023, at 11:51, Wei ZHOU <ustcweiz...@gmail.com> wrote: >>>> >>>> can you check the /etc/cloudstack-release in ssvm ? >>>> >>>> -Wei >>>> >>>> On Tuesday, 21 February 2023, Antoine Boucher <antoi...@haltondc.com >> <mailto:antoi...@haltondc.com>> wrote: >>>>> Here are the logs from daemon.log of the failing SSVM >>>>> >>>>> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed to >> locate executable /usr/local/cloud/systemvm/_run.sh: No such file or >> directory >>>>> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed at step >> EXEC spawning /usr/local/cloud/systemvm/_run.sh: No such file or directory >>>>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Main process >> exited, code=exited, status=203/EXEC >>>>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Failed with result >> 'exit-code'. >>>>> Feb 21 02:36:16 systemvm systemd[1]: systemd-update-utmp-runlevel.service: >> Succeeded. >>>>> Feb 21 02:36:16 systemvm systemd[1]: Finished Update UTMP about System >> Runlevel Changes. >>>>> Feb 21 02:36:16 systemvm systemd[1]: Startup finished in 2.285s >> (kernel) + 2min 16.468s (userspace) = 2min 18.754s. >>>>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Scheduled restart >> job, restart counter is at 1. >>>>> Feb 21 02:36:16 systemvm systemd[1]: Stopped CloudStack Agent service. >>>>> >>>>> This repeats 5 times and fails >>>>> >>>>> The /usr/local/cloud/systemvm/ directory does not exist on the SSVM. >>>>> >>>>> >>>>> >>>>> >>>>> Antoine Boucher >>>>> antoi...@haltondc.com <mailto:antoi...@haltondc.com> >>>>> <mailto:antoi...@haltondc.com> >>>>> [o] +1-226-505-9734 >>>>> www.haltondc.com <http://www.haltondc.com/> <http://www.haltondc.com/> >>>>> >>>>> “Data security made simple” >>>>> >>>>> >>>>> <HDClogo7-small.png> >>>>> >>>>> >>>>> Confidentiality Warning: This message and any attachments are intended >> only for the use of the intended recipient(s), are confidential, and may be >> privileged. If you are not the intended recipient, you are hereby notified >> that any review, retransmission, conversion to hard copy, copying, >> circulation or other use of this message and any attachments is strictly >> prohibited. If you are not the intended recipient, please notify the sender >> immediately by return e-mail, and delete this message and any attachments >> from your system. >>>>> >>>>> >>>>>> On Feb 21, 2023, at 7:53 AM, Antoine Boucher <antoi...@haltondc.com >>>>>> <mailto:antoi...@haltondc.com> >> <mailto:antoi...@haltondc.com>> wrote: >>>>>> >>>>>> They are not old they are the new ones. I have deleted them a number >> of times after the upgrade but the issue prevails. >>>>>> >>>>>> Antoine Boucher >>>>>> antoi...@haltondc.com <mailto:antoi...@haltondc.com> >>>>>> <mailto:antoi...@haltondc.com> >>>>>> [o] +1-226-505-9734 >>>>>> www.haltondc.com <http://www.haltondc.com/> <http://www.haltondc.com/> >>>>>> >>>>>> “Data security made simple and affordable” >>>>>> >>>>>> >>>>>> On Feb 21, 2023, at 00:24, Rohit Yadav <rohit.ya...@shapeblue.com >>>>>> <mailto:rohit.ya...@shapeblue.com> >> <mailto:rohit.ya...@shapeblue.com>> wrote: >>>>>> >>>>>> >>>>>> You can destroy these old systemvms or do a stop and start on them. >>>>>> >>>>>> Regards. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> From: Antoine Boucher <antoi...@haltondc.com >>>>>> <mailto:antoi...@haltondc.com> <mailto: >> antoi...@haltondc.com <mailto:antoi...@haltondc.com>>> >>>>>> Sent: Tuesday, February 21, 2023 10:40:46 AM >>>>>> To: users <users@cloudstack.apache.org >>>>>> <mailto:users@cloudstack.apache.org> <mailto:users@cloudstack. >> apache.org <http://apache.org/>>> >>>>>> Subject: Failed to setup certificates for system vm error on 2nd Zone >> after upgrading to 4.17.2 >>>>>> >>>>>> After upgrading my two zone ACS from 4.16.2 to 4.17.2, the system VM >> (and VRs) of the first zone upgraded without issues but the system VMs >> (s-318-VM and v-317-VM) of the second zone (Kitchener1) are no longer able >> to establish connection with the management server. I have rebooted the >> host they are on, deleted the system vm to be recreated, I also logged into >> the two system VM and found nothing so far except fo the following >> management server logs: >>>>>> >>>>>> 2023-02-20 21:34:10,063 ERROR [c.c.v.VirtualMachineManagerImpl] >> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) >> (logid:5d1ede3b) Failed to setup certificate for system vm: s-318-VM >>>>>> 2023-02-20 21:34:10,747 ERROR [c.c.v.VirtualMachineManagerImpl] >> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >> (logid:352477d9) Failed to setup certificate for system vm: v-317-VM >>>>>> >>>>>> The host of these 2 system VM was missed in the original update, such >> that is was updated after the Management Server was running with 4.17.2 >> instead of before. >>>>>> >>>>>> Would anyone have suggestion to resolve the issue? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> 2023-02-20 21:34:09,599 DEBUG [c.c.a.t.Request] >> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >> (logid:352477d9) Seq 44-1267200345151373322: Received: { Ans: , MgmtId: >> 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { StartAnswer, >> CheckSshAnswer } } >>>>>> 2023-02-20 21:34:09,623 DEBUG [c.c.c.CapacityManagerImpl] >> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >> (logid:352477d9) VM instance {id: "317", name: "v-317-VM", uuid: >> "0fbbd751-3bbe-46f6-bec0-e2887516fa6a", type="ConsoleProxy"} state >> transited from [Starting] to [Running] with event [OperationSucceeded]. >> VM's original host: Host {"id": "44", "name": "kit1-kvm03", "uuid": >> "3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}, new host: Host >> {"id": "44", "name": "kit1-kvm03", "uuid": >> "3bde1185-8db6-414a-9c59-f788a2e0e9cf", >> "type"="Routing"}, host before state transition: Host {"id": "44", "name": >> "kit1-kvm03", "uuid": "3bde1185-8db6-414a-9c59-f788a2e0e9cf", >> "type"="Routing"} >>>>>> 2023-02-20 21:34:09,625 DEBUG [c.c.a.t.Request] >> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) >> (logid:5d1ede3b) Seq 44-1267200345151373326: Sending { Cmd , MgmtId: >> 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 100011, [{" >> org.apache.cloudstack.ca <http://org.apache.cloudstack.ca/> >> <http://org.apache.cloudstack.ca/ >>> .SetupCertificateCommand":{"handleByAgent":"false","accessDetails":{" >> router.name <http://router.name/>":"s-318-VM","Control":"169.254.254.63" >> ,"Storage":"10.91.6.249","Public":"64.32.40.68"," >> Management":"10.91.0.244","router.ip":"169.254.254.63"},"wait":"60"," >> bypassHostMaintenance":"false"}}] } >>>>>> 2023-02-20 21:34:09,626 DEBUG [c.c.v.VirtualMachineManagerImpl] >> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >> (logid:352477d9) Start completed for VM VM instance {id: "317", name: >> "v-317-VM", uuid: "0fbbd751-3bbe-46f6-bec0-e2887516fa6a", >> type="ConsoleProxy"} >>>>>> 2023-02-20 21:34:10,063 DEBUG [c.c.a.t.Request] >> (AgentManager-Handler-8:null) (logid:) Seq 44-1267200345151373326: >> Processing: { Ans: , MgmtId: 130593671224, via: 44, Ver: v1, Flags: 10, [{" >> org.apache.cloudstack.ca <http://org.apache.cloudstack.ca/> >> <http://org.apache.cloudstack.ca/ >>> .SetupCertificateAnswer":{"result":"false","wait":"0"," >> bypassHostMaintenance":"false"}}] } >>>>>> 2023-02-20 21:34:10,063 DEBUG [c.c.a.t.Request] >> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) >> (logid:5d1ede3b) Seq 44-1267200345151373326: Received: { Ans: , MgmtId: >> 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { >> SetupCertificateAnswer } } >>>>>> 2023-02-20 21:34:10,063 ERROR [c.c.v.VirtualMachineManagerImpl] >> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) >> (logid:5d1ede3b) Failed to setup certificate for system vm: s-318-VM >>>>>> 2023-02-20 21:34:10,064 DEBUG [c.c.v.VmWorkJobHandlerProxy] >> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) >> (logid:5d1ede3b) Done executing VM work job: com.cloud.vm.VmWorkStart{" >> dcId":0,"userId":1,"accountId":1,"vmId":318,"handlerName":" >> VirtualMachineManagerImpl"} >>>>>> 2023-02-20 21:34:10,071 DEBUG [c.c.v.VmWorkJobDispatcher] >> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401) (logid:5d1ede3b) >> Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 318, job >> origin: 20367 >>>>>> 2023-02-20 21:34:10,072 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401) (logid:5d1ede3b) >> Remove job-20401 from job monitoring >>>>>> 2023-02-20 21:34:10,081 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] >> (secstorage-1:ctx-bedcece7) (logid:304260be) received secondary storage vm >> alert >>>>>> 2023-02-20 21:34:10,082 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] >> (secstorage-1:ctx-bedcece7) (logid:304260be) Secondary Storage Vm is up, >> zone: Kitchener1, secStorageVm: s-318-VM, public >>>>>> 2023-02-20 21:34:10,083 WARN [c.c.a.AlertManagerImpl] >> (secstorage-1:ctx-bedcece7) (logid:304260be) alertType=[19] >> dataCenterId=[4] podId=[4] clusterId=[null] message=[Secondary Storage Vm >> up in zone: Kitchener1, secStorageVm: s-318-VM, public IP: 64.32.40.68, >> private IP: 10.91.0.244]. >>>>>> 2023-02-20 21:34:10,085 INFO [o.a.c.s.SecondaryStorageManagerImpl] >> (secstorage-1:ctx-bedcece7) (logid:304260be) Secondary storage VM instance >> {id: "318", name: "s-318-VM", uuid: "be765f00-9b3f-47c8-b056-6a41e7befb87", >> type="SecondaryStorageVm"} was started. >>>>>> 2023-02-20 21:34:10,085 INFO >>>>>> [o.a.c.s.PremiumSecondaryStorageManagerImpl] >> (secstorage-1:ctx-bedcece7) (logid:304260be) Primary secondary storage is >> not even started, wait until next turn >>>>>> 2023-02-20 21:34:10,433 DEBUG [c.c.a.t.Request] >> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >> (logid:352477d9) Seq 44-1267200345151373327: Sending { Cmd , MgmtId: >> 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 100011, [{" >> org.apache.cloudstack.ca <http://org.apache.cloudstack.ca/> >> <http://org.apache.cloudstack.ca/ >>> .SetupCertificateCommand":{"handleByAgent":"false","accessDetails":{" >> router.name <http://router.name/>":"v-317-VM","Control":"169.254.21.136" >> ,"Public":"64.32.40.66","Management":"10.91.0.237"," >> router.ip":"169.254.21.136"},"wait":"60","bypassHostMaintenance":"false"}}] >> } >>>>>> 2023-02-20 21:34:10,622 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-342:ctx-4fd75e0f) (logid:63ad79d6) Seq 1-4502473727463655978: >> Executing request >>>>>> 2023-02-20 21:34:10,639 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-342:ctx-4fd75e0f) (logid:c3a060c9) Seq 1-4502473727463655978: >> Response Received: >>>>>> 2023-02-20 21:34:10,639 DEBUG [c.c.a.t.Request] >>>>>> (StatsCollector-3:ctx-597a0e4b) >> (logid:c3a060c9) Seq 1-4502473727463655978: Received: { Ans: , MgmtId: >> 130593671224, via: 1(cs-xcp01), Ver: v1, Flags: 10, { GetVolumeStatsAnswer >> } } >>>>>> 2023-02-20 21:34:10,639 DEBUG [c.c.a.m.AgentManagerImpl] >> (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing >> class com.cloud.agent.api.GetVolumeStatsCommand: >>>>>> 2023-02-20 21:34:10,682 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-86:ctx-520aafa5) (logid:65ac2df2) Seq 2-4712172585113094682: >> Executing request >>>>>> 2023-02-20 21:34:10,724 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-86:ctx-520aafa5) (logid:c3a060c9) Seq 2-4712172585113094682: >> Response Received: >>>>>> 2023-02-20 21:34:10,724 DEBUG [c.c.a.t.Request] >>>>>> (StatsCollector-3:ctx-597a0e4b) >> (logid:c3a060c9) Seq 2-4712172585113094682: Received: { Ans: , MgmtId: >> 130593671224, via: 2(cs-xcp02), Ver: v1, Flags: 10, { GetVolumeStatsAnswer >> } } >>>>>> 2023-02-20 21:34:10,724 DEBUG [c.c.a.m.AgentManagerImpl] >> (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing >> class com.cloud.agent.api.GetVolumeStatsCommand: >>>>>> 2023-02-20 21:34:10,747 DEBUG [c.c.a.t.Request] >> (AgentManager-Handler-3:null) (logid:) Seq 44-1267200345151373327: >> Processing: { Ans: , MgmtId: 130593671224, via: 44, Ver: v1, Flags: 10, [{" >> org.apache.cloudstack.ca <http://org.apache.cloudstack.ca/> >> <http://org.apache.cloudstack.ca/ >>> .SetupCertificateAnswer":{"result":"false","wait":"0"," >> bypassHostMaintenance":"false"}}] } >>>>>> 2023-02-20 21:34:10,747 DEBUG [c.c.a.t.Request] >> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >> (logid:352477d9) Seq 44-1267200345151373327: Received: { Ans: , MgmtId: >> 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { >> SetupCertificateAnswer } } >>>>>> 2023-02-20 21:34:10,747 ERROR [c.c.v.VirtualMachineManagerImpl] >> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >> (logid:352477d9) Failed to setup certificate for system vm: v-317-VM >>>>>> 2023-02-20 21:34:10,748 DEBUG [c.c.v.VmWorkJobHandlerProxy] >> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) >> (logid:352477d9) Done executing VM work job: com.cloud.vm.VmWorkStart{" >> dcId":0,"userId":1,"accountId":1,"vmId":317,"handlerName":" >> VirtualMachineManagerImpl"} >>>>>> 2023-02-20 21:34:10,756 DEBUG [c.c.a.t.Request] >>>>>> (StatsCollector-3:ctx-597a0e4b) >> (logid:c3a060c9) Seq 9-6833649484581308478: Received: { Ans: , MgmtId: >> 130593671224, via: 9(cs-kvm01), Ver: v1, Flags: 10, { GetVolumeStatsAnswer >> } } >>>>>> 2023-02-20 21:34:10,756 DEBUG [c.c.a.m.AgentManagerImpl] >> (StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing >> class com.cloud.agent.api.GetVolumeStatsCommand: >>>>>> 2023-02-20 21:34:10,762 DEBUG [c.c.v.VmWorkJobDispatcher] >> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400) (logid:352477d9) >> Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 317, job >> origin: 20366 >>>>>> 2023-02-20 21:34:10,764 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400) (logid:352477d9) >> Remove job-20400 from job monitoring >>>>>> 2023-02-20 21:34:10,773 INFO [c.c.c.ConsoleProxyManagerImpl] >> (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) Console proxy v-317-VM is >> started >>>>>> 2023-02-20 21:34:10,773 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentManager-Handler-2:null) (logid:) SeqA 45-791665: Processing Seq >> 45-791665: { Cmd , MgmtId: -1, via: 45, Ver: v1, Flags: 11, >> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand" >> :{"_proxyVmId":"290","_loadInfo":"{ >>>>>> "connections": [] >>>>>> }","wait":"0","bypassHostMaintenance":"false"}}] } >>>>>> 2023-02-20 21:34:10,773 DEBUG [c.c.a.ConsoleProxyAlertAdapter] >> (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) received console proxy alert >>>>>> 2023-02-20 21:34:10,775 DEBUG [c.c.a.ConsoleProxyAlertAdapter] >> (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) Console proxy is up, zone: >> Kitchener1, proxy: v-317-VM >>>>>> 2023-02-20 21:34:10,776 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentManager-Handler-2:null) (logid:) SeqA 45-791665: Sending Seq >> 45-791665: { Ans: , MgmtId: 130593671224, via: 45, Ver: v1, Flags: 100010, >> [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0"," >> bypassHostMaintenance":"false"}}] } >>>>>> 2023-02-20 21:34:10,776 WARN [c.c.a.AlertManagerImpl] >> (consoleproxy-1:ctx-2caa6d85) (logid:5554e671) alertType=[10] >> dataCenterId=[4] podId=[4] clusterId=[null] message=[Console proxy up in >> zone: Kitchener1, proxy: v-317-VM, public IP: 64.32.40.66, private IP: >> 10.91.0.237]. >>>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> Antoine Boucher
