They are not old they are the new ones. I have deleted them a number of times
after the upgrade but the issue prevails.
Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com
“Data security made simple and affordable”
Confidentiality Warning: This message and any attachments are intended only for
the use of the intended recipient(s), are confidential, and may be privileged.
If you are not the intended recipient, you are hereby notified that any review,
retransmission, conversion to hard copy, copying, circulation or other use of
this message and any attachments is strictly prohibited. If you are not the
intended recipient, please notify the sender immediately by return e-mail, and
delete this message and any attachments from your system.
On Feb 21, 2023, at 00:24, Rohit Yadav <rohit.ya...@shapeblue.com> wrote:
You can destroy these old systemvms or do a stop and start on them.
Regards.
From: Antoine Boucher <antoi...@haltondc.com>
Sent: Tuesday, February 21, 2023 10:40:46 AM
To: users <users@cloudstack.apache.org>
Subject: Failed to setup certificates for system vm error on 2nd Zone after
upgrading to 4.17.2
After upgrading my two zone ACS from 4.16.2 to 4.17.2, the system VM (and VRs)
of the first zone upgraded without issues but the system VMs (s-318-VM and
v-317-VM) of the second zone (Kitchener1) are no longer able to establish
connection with the management server. I have rebooted the host they are on,
deleted the system vm to be recreated, I also logged into the two system VM and
found nothing so far except fo the following management server logs:
2023-02-20 21:34:10,063 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1)
(logid:5d1ede3b) Failed to setup certificate for system vm: s-318-VM
2023-02-20 21:34:10,747 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4)
(logid:352477d9) Failed to setup certificate for system vm: v-317-VM
The host of these 2 system VM was missed in the original update, such that is
was updated after the Management Server was running with 4.17.2 instead of
before.
Would anyone have suggestion to resolve the issue?
2023-02-20 21:34:09,599 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4)
(logid:352477d9) Seq 44-1267200345151373322: Received: { Ans: , MgmtId:
130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { StartAnswer,
CheckSshAnswer } }
2023-02-20 21:34:09,623 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4)
(logid:352477d9) VM instance {id: "317", name: "v-317-VM", uuid:
"0fbbd751-3bbe-46f6-bec0-e2887516fa6a", type="ConsoleProxy"} state transited
from [Starting] to [Running] with event [OperationSucceeded]. VM's original
host: Host {"id": "44", "name": "kit1-kvm03", "uuid":
"3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}, new host: Host
{"id": "44", "name": "kit1-kvm03", "uuid":
"3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}, host before state
transition: Host {"id": "44", "name": "kit1-kvm03", "uuid":
"3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}
2023-02-20 21:34:09,625 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1)
(logid:5d1ede3b) Seq 44-1267200345151373326: Sending { Cmd , MgmtId:
130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.ca.SetupCertificateCommand":{"handleByAgent":"false","accessDetails":{"router.name":"s-318-VM","Control":"169.254.254.63","Storage":"10.91.6.249","Public":"64.32.40.68","Management":"10.91.0.244","router.ip":"169.254.254.63"},"wait":"60","bypassHostMaintenance":"false"}}]
}
2023-02-20 21:34:09,626 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4)
(logid:352477d9) Start completed for VM VM instance {id: "317", name:
"v-317-VM", uuid: "0fbbd751-3bbe-46f6-bec0-e2887516fa6a", type="ConsoleProxy"}
2023-02-20 21:34:10,063 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:null)
(logid:) Seq 44-1267200345151373326: Processing: { Ans: , MgmtId:
130593671224, via: 44, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.ca.SetupCertificateAnswer":{"result":"false","wait":"0","bypassHostMaintenance":"false"}}]
}
2023-02-20 21:34:10,063 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1)
(logid:5d1ede3b) Seq 44-1267200345151373326: Received: { Ans: , MgmtId:
130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { SetupCertificateAnswer
} }
2023-02-20 21:34:10,063 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1)
(logid:5d1ede3b) Failed to setup certificate for system vm: s-318-VM
2023-02-20 21:34:10,064 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1)
(logid:5d1ede3b) Done executing VM work job:
com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":318,"handlerName":"VirtualMachineManagerImpl"}
2023-02-20 21:34:10,071 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-15:ctx-34675425 job-20367/job-20401) (logid:5d1ede3b) Done
with run of VM work job: com.cloud.vm.VmWorkStart for VM 318, job origin: 20367
2023-02-20 21:34:10,072 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-15:ctx-34675425 job-20367/job-20401) (logid:5d1ede3b) Remove
job-20401 from job monitoring
2023-02-20 21:34:10,081 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter]
(secstorage-1:ctx-bedcece7) (logid:304260be) received secondary storage vm alert
2023-02-20 21:34:10,082 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter]
(secstorage-1:ctx-bedcece7) (logid:304260be) Secondary Storage Vm is up, zone:
Kitchener1, secStorageVm: s-318-VM, public
2023-02-20 21:34:10,083 WARN [c.c.a.AlertManagerImpl]
(secstorage-1:ctx-bedcece7) (logid:304260be) alertType=[19] dataCenterId=[4]
podId=[4] clusterId=[null] message=[Secondary Storage Vm up in zone:
Kitchener1, secStorageVm: s-318-VM, public IP: 64.32.40.68, private IP:
10.91.0.244].
2023-02-20 21:34:10,085 INFO [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-bedcece7) (logid:304260be) Secondary storage VM instance {id:
"318", name: "s-318-VM", uuid: "be765f00-9b3f-47c8-b056-6a41e7befb87",
type="SecondaryStorageVm"} was started.
2023-02-20 21:34:10,085 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl]
(secstorage-1:ctx-bedcece7) (logid:304260be) Primary secondary storage is not
even started, wait until next turn
2023-02-20 21:34:10,433 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4)
(logid:352477d9) Seq 44-1267200345151373327: Sending { Cmd , MgmtId:
130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.ca.SetupCertificateCommand":{"handleByAgent":"false","accessDetails":{"router.name":"v-317-VM","Control":"169.254.21.136","Public":"64.32.40.66","Management":"10.91.0.237","router.ip":"169.254.21.136"},"wait":"60","bypassHostMaintenance":"false"}}]
}
2023-02-20 21:34:10,622 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-342:ctx-4fd75e0f) (logid:63ad79d6) Seq 1-4502473727463655978:
Executing request
2023-02-20 21:34:10,639 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-342:ctx-4fd75e0f) (logid:c3a060c9) Seq 1-4502473727463655978:
Response Received:
2023-02-20 21:34:10,639 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-597a0e4b)
(logid:c3a060c9) Seq 1-4502473727463655978: Received: { Ans: , MgmtId:
130593671224, via: 1(cs-xcp01), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } }
2023-02-20 21:34:10,639 DEBUG [c.c.a.m.AgentManagerImpl]
(StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing class
com.cloud.agent.api.GetVolumeStatsCommand:
2023-02-20 21:34:10,682 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-86:ctx-520aafa5) (logid:65ac2df2) Seq 2-4712172585113094682:
Executing request
2023-02-20 21:34:10,724 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-86:ctx-520aafa5) (logid:c3a060c9) Seq 2-4712172585113094682:
Response Received:
2023-02-20 21:34:10,724 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-597a0e4b)
(logid:c3a060c9) Seq 2-4712172585113094682: Received: { Ans: , MgmtId:
130593671224, via: 2(cs-xcp02), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } }
2023-02-20 21:34:10,724 DEBUG [c.c.a.m.AgentManagerImpl]
(StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing class
com.cloud.agent.api.GetVolumeStatsCommand:
2023-02-20 21:34:10,747 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null)
(logid:) Seq 44-1267200345151373327: Processing: { Ans: , MgmtId:
130593671224, via: 44, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.ca.SetupCertificateAnswer":{"result":"false","wait":"0","bypassHostMaintenance":"false"}}]
}
2023-02-20 21:34:10,747 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4)
(logid:352477d9) Seq 44-1267200345151373327: Received: { Ans: , MgmtId:
130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { SetupCertificateAnswer
} }
2023-02-20 21:34:10,747 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4)
(logid:352477d9) Failed to setup certificate for system vm: v-317-VM
2023-02-20 21:34:10,748 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4)
(logid:352477d9) Done executing VM work job:
com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":317,"handlerName":"VirtualMachineManagerImpl"}
2023-02-20 21:34:10,756 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-597a0e4b)
(logid:c3a060c9) Seq 9-6833649484581308478: Received: { Ans: , MgmtId:
130593671224, via: 9(cs-kvm01), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } }
2023-02-20 21:34:10,756 DEBUG [c.c.a.m.AgentManagerImpl]
(StatsCollector-3:ctx-597a0e4b) (logid:c3a060c9) Details from executing class
com.cloud.agent.api.GetVolumeStatsCommand:
2023-02-20 21:34:10,762 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400) (logid:352477d9) Done
with run of VM work job: com.cloud.vm.VmWorkStart for VM 317, job origin: 20366
2023-02-20 21:34:10,764 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400) (logid:352477d9) Remove
job-20400 from job monitoring
2023-02-20 21:34:10,773 INFO [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-2caa6d85) (logid:5554e671) Console proxy v-317-VM is started
2023-02-20 21:34:10,773 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-2:null) (logid:) SeqA 45-791665: Processing Seq
45-791665: { Cmd , MgmtId: -1, via: 45, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"290","_loadInfo":"{
"connections": []
}","wait":"0","bypassHostMaintenance":"false"}}] }
2023-02-20 21:34:10,773 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
(consoleproxy-1:ctx-2caa6d85) (logid:5554e671) received console proxy alert
2023-02-20 21:34:10,775 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
(consoleproxy-1:ctx-2caa6d85) (logid:5554e671) Console proxy is up, zone:
Kitchener1, proxy: v-317-VM
2023-02-20 21:34:10,776 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-2:null) (logid:) SeqA 45-791665: Sending Seq 45-791665:
{ Ans: , MgmtId: 130593671224, via: 45, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
}
2023-02-20 21:34:10,776 WARN [c.c.a.AlertManagerImpl]
(consoleproxy-1:ctx-2caa6d85) (logid:5554e671) alertType=[10] dataCenterId=[4]
podId=[4] clusterId=[null] message=[Console proxy up in zone: Kitchener1,
proxy: v-317-VM, public IP: 64.32.40.66, private IP: 10.91.0.237].
Regards,
Antoine Boucher
