Re: [Openstack] [Quantum/OVS] Br-int not responsive after network node reboot
Hi,
I progressed in investigating the bug. I forgot to mention I was
following Provided Router/single tenancy setup.
So, at reboot, my tap/qg/qr network interfaces were down :
7: qg-c39e5df4-7f: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN
link/ether fe:10:8c:d8:d8:ca brd ff:ff:ff:ff:ff:ff
inet 192.168.10.1/16 brd 192.168.255.255 scope global qg-c39e5df4-7f
inet 192.168.10.2/32 brd 192.168.10.2 scope global qg-c39e5df4-7f
8: qr-f76e4668-fa: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN
link/ether 52:93:35:8b:11:ff brd ff:ff:ff:ff:ff:ff
inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-f76e4668-fa
9: tap2ed3cd8a-03: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN
link/ether 0e:3e:e0:61:5e:b0 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.2/24 brd 10.0.0.255 scope global tap2ed3cd8a-03
So, a short 'ip link set dev up' for the three above fixed my issue.
Other terms, GRE tunneling is fine at reboot, only interface status is
wrong.
Do you have any idea how to fix it ? I can't issue a 'post-up' statement
in /etc/network/interfaces and /etc/rc.local is quite a ugly patch I think.
Thanks,
-Sylvain
Le 18/02/2013 18:06, Sylvain Bauza a écrit :
Hi,
I'll try to be clear. I do follow a classic setup for Quantum with 2
NICs and a GRE tunnel in between nodes with br-int/br-tun.
Everything is fine at first install, but when rebooting the network
node (including quantum-ovs-plugin-agent, quantum-l3-agent and
quantum-dhcp-agent), I notice that DHCP assignation is failing for my
VMs.
A tcpdump shows at physical level that GRE packets are arriving on
eth0 (mgmt NIC) on network node (for DHCP request) but no reply is
done by the network node.
The workaround I found is to delete br-int and br-tun on network node,
create only br-int and restart both services
(quantum-{ovs-plugin,l3,dhcp} ) to get things done.
This is quite brutal. Do you know if it's a known bug, or something
bad with my setup ?
Thanks,
-Sylvain
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Quantum/OVS] Br-int not responsive after network node reboot
On 02/19/2013 01:57 PM, Sylvain Bauza wrote:
Hi,
I progressed in investigating the bug. I forgot to mention I was
following Provided Router/single tenancy setup.
So, at reboot, my tap/qg/qr network interfaces were down :
7: qg-c39e5df4-7f: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN
link/ether fe:10:8c:d8:d8:ca brd ff:ff:ff:ff:ff:ff
inet 192.168.10.1/16 brd 192.168.255.255 scope global qg-c39e5df4-7f
inet 192.168.10.2/32 brd 192.168.10.2 scope global qg-c39e5df4-7f
8: qr-f76e4668-fa: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN
link/ether 52:93:35:8b:11:ff brd ff:ff:ff:ff:ff:ff
inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-f76e4668-fa
9: tap2ed3cd8a-03: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN
link/ether 0e:3e:e0:61:5e:b0 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.2/24 brd 10.0.0.255 scope global tap2ed3cd8a-03
So, a short 'ip link set dev up' for the three above fixed my issue.
Other terms, GRE tunneling is fine at reboot, only interface status is
wrong.
Do you have any idea how to fix it ? I can't issue a 'post-up'
statement in /etc/network/interfaces and /etc/rc.local is quite a ugly
patch I think.
The problem is as follows:
When you reboot the host the openvswitch will create the interfaces on
restart. this causes problems with the dhcp and the l3 agents.
the solution to this is to run the quantum-ovs-cleanup utility on reboot
prior to the dhcp and the l3 agents.
thanks
Gary
Thanks,
-Sylvain
Le 18/02/2013 18:06, Sylvain Bauza a écrit :
Hi,
I'll try to be clear. I do follow a classic setup for Quantum with 2
NICs and a GRE tunnel in between nodes with br-int/br-tun.
Everything is fine at first install, but when rebooting the network
node (including quantum-ovs-plugin-agent, quantum-l3-agent and
quantum-dhcp-agent), I notice that DHCP assignation is failing for my
VMs.
A tcpdump shows at physical level that GRE packets are arriving on
eth0 (mgmt NIC) on network node (for DHCP request) but no reply is
done by the network node.
The workaround I found is to delete br-int and br-tun on network
node, create only br-int and restart both services
(quantum-{ovs-plugin,l3,dhcp} ) to get things done.
This is quite brutal. Do you know if it's a known bug, or something
bad with my setup ?
Thanks,
-Sylvain
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Quantum/OVS] Br-int not responsive after network node reboot
Le 19/02/2013 13:31, Gary Kotton a écrit : On 02/19/2013 01:57 PM, Sylvain Bauza wrote: Hi, I progressed in investigating the bug. I forgot to mention I was following Provided Router/single tenancy setup. So, at reboot, my tap/qg/qr network interfaces were down : 7: qg-c39e5df4-7f: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN link/ether fe:10:8c:d8:d8:ca brd ff:ff:ff:ff:ff:ff inet 192.168.10.1/16 brd 192.168.255.255 scope global qg-c39e5df4-7f inet 192.168.10.2/32 brd 192.168.10.2 scope global qg-c39e5df4-7f 8: qr-f76e4668-fa: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN link/ether 52:93:35:8b:11:ff brd ff:ff:ff:ff:ff:ff inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-f76e4668-fa 9: tap2ed3cd8a-03: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN link/ether 0e:3e:e0:61:5e:b0 brd ff:ff:ff:ff:ff:ff inet 10.0.0.2/24 brd 10.0.0.255 scope global tap2ed3cd8a-03 So, a short 'ip link set dev up' for the three above fixed my issue. Other terms, GRE tunneling is fine at reboot, only interface status is wrong. Do you have any idea how to fix it ? I can't issue a 'post-up' statement in /etc/network/interfaces and /etc/rc.local is quite a ugly patch I think. The problem is as follows: When you reboot the host the openvswitch will create the interfaces on restart. this causes problems with the dhcp and the l3 agents. the solution to this is to run the quantum-ovs-cleanup utility on reboot prior to the dhcp and the l3 agents. thanks Gary Thanks, got the bug# : https://bugs.launchpad.net/quantum/+bug/1091605 As the fix is released in 2012.2.3, I have to patch manually or keep my method (pref.) of deleting/creating bridges (as only Quantum is using these bridges). Thanks for the quick answer. -Sylvain ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Quantum/OVS] Br-int not responsive after network node reboot
On 02/19/2013 03:47 PM, Sylvain Bauza wrote: Le 19/02/2013 13:31, Gary Kotton a écrit : On 02/19/2013 01:57 PM, Sylvain Bauza wrote: Hi, I progressed in investigating the bug. I forgot to mention I was following Provided Router/single tenancy setup. So, at reboot, my tap/qg/qr network interfaces were down : 7: qg-c39e5df4-7f: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN link/ether fe:10:8c:d8:d8:ca brd ff:ff:ff:ff:ff:ff inet 192.168.10.1/16 brd 192.168.255.255 scope global qg-c39e5df4-7f inet 192.168.10.2/32 brd 192.168.10.2 scope global qg-c39e5df4-7f 8: qr-f76e4668-fa: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN link/ether 52:93:35:8b:11:ff brd ff:ff:ff:ff:ff:ff inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-f76e4668-fa 9: tap2ed3cd8a-03: BROADCAST,MULTICAST mtu 1500 qdisc noop state DOWN link/ether 0e:3e:e0:61:5e:b0 brd ff:ff:ff:ff:ff:ff inet 10.0.0.2/24 brd 10.0.0.255 scope global tap2ed3cd8a-03 So, a short 'ip link set dev up' for the three above fixed my issue. Other terms, GRE tunneling is fine at reboot, only interface status is wrong. Do you have any idea how to fix it ? I can't issue a 'post-up' statement in /etc/network/interfaces and /etc/rc.local is quite a ugly patch I think. The problem is as follows: When you reboot the host the openvswitch will create the interfaces on restart. this causes problems with the dhcp and the l3 agents. the solution to this is to run the quantum-ovs-cleanup utility on reboot prior to the dhcp and the l3 agents. thanks Gary Thanks, got the bug# : https://bugs.launchpad.net/quantum/+bug/1091605 As the fix is released in 2012.2.3, I have to patch manually or keep my method (pref.) of deleting/creating bridges (as only Quantum is using these bridges). I am not sure. I know that in Fedora and in RHEL we ensure that the utility is run on boot (if the plugin os openvswitch). Not sure if the Ubuntu/Debian packagers have done the same. It is worthwhile notifying them about this. I think that Dan may know the relevant people (I do not know their names off hand) Thanks Gary Thanks for the quick answer. -Sylvain ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Quantum/OVS] Br-int not responsive after network node reboot
Le 19/02/2013 13:31, Gary Kotton a écrit :
The problem is as follows:
When you reboot the host the openvswitch will create the interfaces on
restart. this causes problems with the dhcp and the l3 agents.
the solution to this is to run the quantum-ovs-cleanup utility on
reboot prior to the dhcp and the l3 agents.
thanks
Gary
Hi Gary,
As I was stresstesting Quantum, I did noticed that the issue is similar
on my compute node : instances are unable to get DHCP address.
A quick look at network level shows that there is no GRE encapsulation
over DHCP packets.
I tried removing/adding br-int/br-tun, but the virtual network
interfaces (qvo) don't bridge to br-int if the VM was previously launched.
Spawning a new VM is successful, but rebooting (either by virsh
destroy/start, or thru Horizon) doesn't allow the qvo interface to be
attached again. A restart of nova-compute doesn't do the trick either.
Any idea ?
Thanks,
-Sylvain
Thanks,
-Sylvain
Le 18/02/2013 18:06, Sylvain Bauza a écrit :
Hi,
I'll try to be clear. I do follow a classic setup for Quantum with 2
NICs and a GRE tunnel in between nodes with br-int/br-tun.
Everything is fine at first install, but when rebooting the network
node (including quantum-ovs-plugin-agent, quantum-l3-agent and
quantum-dhcp-agent), I notice that DHCP assignation is failing for
my VMs.
A tcpdump shows at physical level that GRE packets are arriving on
eth0 (mgmt NIC) on network node (for DHCP request) but no reply is
done by the network node.
The workaround I found is to delete br-int and br-tun on network
node, create only br-int and restart both services
(quantum-{ovs-plugin,l3,dhcp} ) to get things done.
This is quite brutal. Do you know if it's a known bug, or something
bad with my setup ?
Thanks,
-Sylvain
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
[Openstack] [Quantum/OVS] Br-int not responsive after network node reboot
Hi,
I'll try to be clear. I do follow a classic setup for Quantum with 2
NICs and a GRE tunnel in between nodes with br-int/br-tun.
Everything is fine at first install, but when rebooting the network node
(including quantum-ovs-plugin-agent, quantum-l3-agent and
quantum-dhcp-agent), I notice that DHCP assignation is failing for my VMs.
A tcpdump shows at physical level that GRE packets are arriving on eth0
(mgmt NIC) on network node (for DHCP request) but no reply is done by
the network node.
The workaround I found is to delete br-int and br-tun on network node,
create only br-int and restart both services
(quantum-{ovs-plugin,l3,dhcp} ) to get things done.
This is quite brutal. Do you know if it's a known bug, or something bad
with my setup ?
Thanks,
-Sylvain
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
