In short term, we use veth pairs with namespace to fix the issue if performance 
is not impacted (Hopefully:)

If performance downgrade too much, we may consider the following:

1) DHCP agent: use veth pairs with namespace since it is not critical path.

2) L3 agent: don't create port in OSV.  Connect L3 agent without open switch

VM <---> Linux Bridge <---> open switch <--> (int-br-eth, phy-br-eth) <---> 
physical switch <----> vlan interface with namespace <---> L3agent

In long term, we may implement namespace and remove linux bridge.

-----Original Message-----
From: Ihar Hrachyshka [] 
Sent: Friday, February 13, 2015 8:15 PM
To: openstack-dev
Cc: Jiri Benc;
Subject: [openstack-dev] [neutron] moving openvswitch ports between namespaces 
considered harmful

Hash: SHA1

Hi neutroners,

we** had several conversations recently with our Red Hat fellows who work on 
openvswitch (Jiri Benc and Jiri Pirko) regarding the way neutron utilizes their 
software. Those were beneficial to both sides to understand what we do right 
and wrong. I was asked to share some of the points from those discussions with 
broader community.


One of the issues that came up during discussions is the way neutron connects 
ovs ports to namespaces. The short story is that openvswitch is not designed 
with namespaces in mind, and the fact that moving its ports into a different 
namespace works for neutron is mere coincidence, and is actually considered as 
a bug by openvswitch guys.

It's not just broken in theory from software design standpoint, but also in 
practice. Specifically,

1. ovsdb dump does not work properly for namespaces:
- -

This results in wrong statistics and other data collected for these ports;

2. We suspect that the following kernel crash is triggered because of our usage 
of the feature that is actually a bug:
- -

Quoting Jiri Benc,

"The problem is openvswitch does not support its ports to be moved to a 
different name space. The fact that it's possible to do that is a bug - such 
operation should have been denied. Unfortunately, due to a missing check, it's 
not been denied. Such setup does not work reliably, though, and leads to 
various issues from incorrect resource accounting to kernel crashes.

We're aware of the bug but we don't have any easy way to fix it. The most 
obvious option, disabling moving of ovs ports to different name spaces, would 
be easy to do but it would break Neutron. The other option, making ovs to work 
across net name spaces, is hard and will require addition of different kernel 
APIs and large changes in ovs user space daemons. This constitutes tremendous 
amount of work."

The tracker bug on openvswitch side is:
- -

So in the best case, we may expect openvswitch to properly support the feature 
in long term, but short term it won't work, especially while neutron expects 
other features implemented in openvswitch for it (like NAT, or connection 
tracking, or ipv6 tunnel endpoints, to name a few).

We could try to handle the issue neutron side. We can fix it by utilizing veth 
pairs to get into namespaces, but it may mean worse performance, and will 
definitely require proper benchmarking to see whether we can live with 
performance drop.


There were other suggestions on how we can enhance our way of usage of 
openvswitch. Among those, getting rid of linux bridge used for security groups, 
with special attention on getting rid of ebtables
(sic!) for they are a lot slower than iptables; getting rid of veth pair for 
instance ports.


I really encourage everyone to check the following video from 2015 
on all that and more at:

- -

Among other things, you will find presentation of plotnetcfg tool to create 
nice graphs of openstack state.

If you're lazy enough and want to switch directly to the analysis of neutron 
problems, skip to ~18:00.

I also encourage to check our the video around 30:00 on the way out of 
openvswitch for neutron (tc/eBPF). As crazy as encouraging.


While at it, I encourage everyone interested in controlling switches the open 
source way to check out presentation of the new kernel subsystem for that (I 
guess vaporware atm):

- -


So, that's what I have for you now. I really want to hear from everyone what is 
our plan to solve the problem neutron side.



**Jakub Libosvar and me
Version: GnuPG v1


OpenStack Development Mailing List (not for usage questions)

OpenStack Development Mailing List (not for usage questions)

Reply via email to