Wei, Yes, both hosts have /etc/libvirt/hooks/qemu and python3 installed. In this case, they both have the same names for their interfaces (enp3s0f0 and enp3s0f1), but host 1 is using enp3s0f0 and host 2 is using enp3s0f1.
~Peter ---------------------------------------- From: "Wei ZHOU" <ustcweiz...@gmail.com> Sent: 11/24/21 3:14 PM To: peter.st...@granddial.com Subject: Re: Libvirt exception on live migrate Hi Peter, I checked /etc/libvirt/hooks/qemu on hosts once again. I think it is able to handle migrations between hosts with different device names. My questions are (1) Do the hosts have /etc/libvirt/hooks/qemu ? Is python3 installed ? (2) Does the destination host have the devices enp3s0f0 and enp3s0f1 ? -Wei On Wed, 24 Nov 2021 at 21:07, Peter Stine <peter.st...@granddial.com> wrote: Hi Wei, Hmm... that's a little frustrating, since not all of the hardware I'm working with is all the same. Sure. Here's the github link: https://github.com/apache/cloudstack/issues/5717 ~Peter ---------------------------------------- From: "Wei ZHOU" <ustcweiz...@gmail.com> Sent: 11/24/21 2:23 PM To: peter.st...@granddial.com Subject: Re: Libvirt exception on live migrate Hi Peter, Virtual routers have nics on Public and Guest network(both on cloudbr0), but not on Management network, therefore it is expected that VR does not connect cloudbr1 (management). Currently cloudstack does not support vm migration between hosts with different device names. VM migration can succeed with some changes in /etc/libvirt/hooks/qemu (for example, phyDev = enp3s0f1). However it is just a workaround, your hosts should have the network device name (use bonding or rename device name) Could you please create an issue on github ?. -Wei On Wed, 24 Nov 2021 at 15:49, Peter Stine <peter.st...@granddial.com> wrote: Here are the agents' properties: https://gist.github.com/PeterS-gd/230c6e2b843901b71feb54c39270a4a8 Cloudstack can migrate a VM after I shutdown the VM, so it can recognize it in some way at least, but it cannot live migrate. Traffic labels: Management: cloudbr1 Guest: cloudbr0 Public: cloudbr0 ---------------------------------------- From: "Wei ZHOU" <ustcweiz...@gmail.com> Sent: 11/24/21 9:26 AM To: peter.st...@granddial.com Subject: Re: Libvirt exception on live migrate Hi Peter, You need to set the correct network label for traffic types (Public/Guest) in zone-> physical networks. Please check the values. cloudstack should be able to handle the migration between hosts with different network device names. Could you please share the /etc/cloudstack/agent/agent.properties on hosts ? -Wei On Wed, 24 Nov 2021 at 14:50, Peter Stine <peter.st...@granddial.com> wrote: Sure. Cloudstack version is 4.15.2 All of my systems are running Ubuntu server 20.04 (My hypervisor is KVM) Yes, the network device names are different on the hosts (see below). this is part of the issue, the virtual router does not seem to connect to the cloudbr1 or cloudbr0. Host 1 interface that is created by virtual router: 50: enp3s0f0.200@enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brenp3s0f0-200 state UP group default qlen 1000 link/ether d4:85:64:2f:37:b0 brd ff:ff:ff:ff:ff:ff inet6 fe80::d685:64ff:fe2f:37b0/64 scope link valid_lft forever preferred_lft forever 51: brenp3s0f0-200: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether d4:85:64:2f:37:b0 brd ff:ff:ff:ff:ff:ff inet6 fe80::6433:6dff:fe95:1055/64 scope link valid_lft forever preferred_lft forever Interfaces on Host 2: 39: enp3s0f1.200@enp3s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brenp3s0f1-200 state UP group default qlen 1000 link/ether 00:10:18:ab:b0:6e brd ff:ff:ff:ff:ff:ff inet6 fe80::210:18ff:feab:b06e/64 scope link valid_lft forever preferred_lft forever 40: brenp3s0f1-200: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:10:18:ab:b0:6e brd ff:ff:ff:ff:ff:ff inet6 fe80::6894:9ff:fef4:dfc4/64 scope link valid_lft forever preferred_lft forever Brctl show for host 1: bridge name bridge id STP enabled interfaces brenp3s0f0-200 8000.d485642f37b0 no enp3s0f0.200 vnet13 vnet2 vnet5 vnet7 brenp3s0f0-3128 8000.d485642f37b0 no enp3s0f0.3128 vnet11 cloud0 8000.fe00a9fe0dfd no vnet0 vnet12 vnet3 cloudbr0 8000.d485642f37b0 no enp3s0f0 vcable0 vnet8 cloudbr1 8000.7a90d2bda3a3 no ethvcable1.204 vnet1 vnet4 cloudbr2 8000.7a90d2bda3a3 no vcable1.206 vnet6 brctl show on host 2: bridge name bridge id STP enabled interfaces brenp3s0f1-200 8000.001018abb06e no enp3s0f1.200 vnet0 vnet2 vnet3 brenp3s0f1-3128 8000.001018abb06e no enp3s0f1.3128 cloud0 8000.fe00a9fe0eda no vnet1 cloudbr0 8000.001018abb06e no enp3s0f1 vcable0 cloudbr1 8000.6629bdef525a no ethvcable1.204 cloudbr2 8000.6629bdef525a no vcable1.206 Host1 agent log (source of migration): https://gist.github.com/PeterS-gd/221ad3affeb77da0828323ca2c2ae119 Host2 agent log (destination of migration): https://gist.github.com/PeterS-gd/d250c240f63538a6bd541b7870dffe0d ---------------------------------------- From: "Wei ZHOU" <ustcweiz...@gmail.com> Sent: 11/24/21 4:56 AM To: users <users@cloudstack.apache.org>, peter.st...@granddial.com Subject: Re: Libvirt exception on live migrate Hi Peter, The information you provided is not enough to find out the root cause. Could you share more information ? cloudstack version os distribution and version network device name on source host and destination host output of `brctl show` or `bridge link` on source host and destination host logs in /var/log/cloudstack/agent/agent.log on source host and destination host. I guess the network device name on hosts are different. -Wei On Wed, 24 Nov 2021 at 09:31, Peter Stine <peter.st...@granddial.com.invalid> wrote: Good afternoon everyone, I have gotten my cloudstack running, but I am unable to live migrate any of my VMs. I get an error like this: "Exception during migrate: org.libvirt.LibvirtException: Cannot get interface MTU on 'brenp3s0f0-3128': No such device" I have tried using an L2, isolated, and shared network. I get the same result (though the exact interface name changes) When I set up these networks, it seems to directly tap the baremetal interface instead of the cloudbr1. Because of this, the name does not remain consistent for migration. Is there a reason / workaround for this? I was under the impression, that everything was supposed to use the cloud bridges to permit migration for High Availability. This issue seems relevant (https://issues.apache.org/jira/browse/CLOUDSTACK-7743), but there is no solution listed. Thanks! Peter
