Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful
I agree with Kevin that we should adopt veth pairs for fixing the issue in the short term, at least until CT gets merged and distributed in OVS. At that point the transition to a OVS based solution will make a lot of sense, given that the numbers show that it's worth of course ;) On Sun Feb 15 2015 at 7:17:39 AM Thomas Graf tg...@noironetworks.com wrote: FYI: Ivar (CCed) is also working on collecting numbers to compare both architectures to kick of a discussion at the next summit. Ivar, can you link to the talk proposal? Thanks for bringing this up Thomas! Here is the link to the talk proposal [0]. Any with suggestion, idea or random comment is very welcome! Ivar. [0] https://www.openstack.org/vote-paris/presentation/taking-security-groups-to-ludicrous-speed-with-open-vswitch __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful
[Sorry for the resend, I had to subscribe to openstack-dev first, maybe worth removing the subscribe requirement for outsiders] [Copying ovs-dev] On 02/13/15 at 01:47pm, Miguel Ángel Ajo wrote: Sorry, I forgot about 5) If we put all our OVS/OF bridge logic in just one bridge (instead of N: br-tun, br-int, br-ex, br-xxx), the performance should be yet higher, since, as far as I understood, flow rule lookup could be more optimized into the kernel megaflows without forwarding and re-starting evaluation due to patch ports. (Please correct me here where I’m wrong, I just have very high level view of this). Some preliminary numbers were presented at the OVS Fall Conference 2014 which indicate that a pure OVS ACL solution scales better as the number of rules changes. You can find the number on slide 9 here: http://www.openvswitch.org/support/ovscon2014/17/1030-conntrack_nat.pdf Another obvious advantage is that since we have to go through the OVS flow table anyway, traversing any additional (linear) ruleset is likely to have more overhead. FYI: Ivar (CCed) is also working on collecting numbers to compare both architectures to kick of a discussion at the next summit. Ivar, can you link to the talk proposal? On Friday, 13 de February de 2015 at 13:42, Miguel Ángel Ajo wrote: I’m working on the following items: 1) Doing Openflow traffic filtering (stateful firewall) based on OVS+CT[1] patch, which may eventually merge. Here I want to build a good amount of benchmarks to be able to compare the current network iptables+LB solution to just openflow. Openflow filtering should be fast, as it’s quite smart at using hashes to match OF rules in the kernel megaflows (thanks Jiri T.Graf for explaining me this) The only bad part is that we would have to dynamically change more rules based on security group changes (now we do that via ip sets without reloading all the rules). Worth pointing out that it is entirely possible to integrate ipset with OVS in the datapath in case representing ipsets with individual wildcard flows is not sufficient. I guess we'll know when we have more numbers. To do this properly, we may want to make the OVS plugin a real OF controller to be able to push OF rules to the bridge without the need of calling ovs-ofctl on the command line all the time. We should synchronize this effort with the OVN effort. There is a lot of overlap. 2) Using OVS+OF to do QoS other interesting stuff to look at: 3) Doing routing in OF too, thanks to the NAT capabilities of having OVS+CT Just want to point out that this is still WIP with several issues outstanding. I think everybody agrees that it's tremendously useful to have, we need to be able to implement it properly. I'll let you and anybody else interested know as soon as it's ready for testing. 4) The namespace problem, what kinds of statistics get broken by moving ports into namespaces now?, the short-term fix could be using vets, but “namespaceable” OVS ports would be perfect, yet I understand the change is a big feature. If we had 1 3, may be 4 wouldn’t be a problem anymore. Implementing VRF in OVS will hide (4) for OpenStack but we should still fix it in OVS as Ben suggested in the Bugzilla. It looks feasible to support netns properly in OVS. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful
What is the status of the conntrack integration with respect to availability in distributions? The lack of state tracking has blocked the ability for us to get rid of namespaces for the L3 agent (because of SNAT) and the filtering bridge between the VM and OVS (stateful firewall for security groups). It has been known for a long time that these are suboptimal, but our hands are sort of tied because we don't want to require kernel code changes to use Neutron. Are Ubuntu 1404 or CentOS 7 shipping openvswitch kernel modules with conntrack integration? If not, I don't see a feasible way of eliminating any of these problems with a pure OVS solution. (faking a stateful firewall with flag matching doesn't count) In the short term, we should probably switch back to veth pairs to handle the namespace issue for the dhcp agent and the L3 agent. [Sorry for the resend, I had to subscribe to openstack-dev first, maybe worth removing the subscribe requirement for outsiders] [Copying ovs-dev] On 02/13/15 at 01:47pm, Miguel Ángel Ajo wrote: Sorry, I forgot about 5) If we put all our OVS/OF bridge logic in just one bridge (instead of N: br-tun, br-int, br-ex, br-xxx), the performance should be yet higher, since, as far as I understood, flow rule lookup could be more optimized into the kernel megaflows without forwarding and re-starting evaluation due to patch ports. (Please correct me here where I’m wrong, I just have very high level view of this). Some preliminary numbers were presented at the OVS Fall Conference 2014 which indicate that a pure OVS ACL solution scales better as the number of rules changes. You can find the number on slide 9 here: http://www.openvswitch.org/support/ovscon2014/17/1030-conntrack_nat.pdf Another obvious advantage is that since we have to go through the OVS flow table anyway, traversing any additional (linear) ruleset is likely to have more overhead. FYI: Ivar (CCed) is also working on collecting numbers to compare both architectures to kick of a discussion at the next summit. Ivar, can you link to the talk proposal? On Friday, 13 de February de 2015 at 13:42, Miguel Ángel Ajo wrote: I’m working on the following items: 1) Doing Openflow traffic filtering (stateful firewall) based on OVS+CT[1] patch, which may eventually merge. Here I want to build a good amount of benchmarks to be able to compare the current network iptables+LB solution to just openflow. Openflow filtering should be fast, as it’s quite smart at using hashes to match OF rules in the kernel megaflows (thanks Jiri T.Graf for explaining me this) The only bad part is that we would have to dynamically change more rules based on security group changes (now we do that via ip sets without reloading all the rules). Worth pointing out that it is entirely possible to integrate ipset with OVS in the datapath in case representing ipsets with individual wildcard flows is not sufficient. I guess we'll know when we have more numbers. To do this properly, we may want to make the OVS plugin a real OF controller to be able to push OF rules to the bridge without the need of calling ovs-ofctl on the command line all the time. We should synchronize this effort with the OVN effort. There is a lot of overlap. 2) Using OVS+OF to do QoS other interesting stuff to look at: 3) Doing routing in OF too, thanks to the NAT capabilities of having OVS+CT Just want to point out that this is still WIP with several issues outstanding. I think everybody agrees that it's tremendously useful to have, we need to be able to implement it properly. I'll let you and anybody else interested know as soon as it's ready for testing. 4) The namespace problem, what kinds of statistics get broken by moving ports into namespaces now?, the short-term fix could be using vets, but “namespaceable” OVS ports would be perfect, yet I understand the change is a big feature. If we had 1 3, may be 4 wouldn’t be a problem anymore. Implementing VRF in OVS will hide (4) for OpenStack but we should still fix it in OVS as Ben suggested in the Bugzilla. It looks feasible to support netns properly in OVS. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/13/2015 01:42 PM, Miguel Ángel Ajo wrote: Hi, Ihar Jiri, thank you for pointing this out. I’m working on the following items: 1) Doing Openflow traffic filtering (stateful firewall) based on OVS+CT[1] patch, which may eventually merge. Here I want to build a good amount of benchmarks to be able to compare the current network iptables+LB solution to just openflow. Openflow filtering should be fast, as it’s quite smart at using hashes to match OF rules in the kernel megaflows (thanks Jiri T.Graf for explaining me this) The only bad part is that we would have to dynamically change more rules based on security group changes (now we do that via ip sets without reloading all the rules). To do this properly, we may want to make the OVS plugin a real OF controller to be able to push OF rules to the bridge without the need of calling ovs-ofctl on the command line all the time. I am actually a bit worried about this project being bound so tightly to openvswitch. There are common linux alternatives in the wild that do not require it at all (tc, eBPF). Of course, openvswitch could be utilized in some exotic cases, like controlling HyperV instances in Windows environment, but for usual Linux platform, utilizing something like openvswitch seems to be an overkill, while we don't use any of controller separation features and do propagate updates via AMQP to each controlled node. 2) Using OVS+OF to do QoS other interesting stuff to look at: 3) Doing routing in OF too, thanks to the NAT capabilities of having OVS+CT 4) The namespace problem, what kinds of statistics get broken by moving ports into namespaces now?, I think some of those are already mentioned in the ovs tracker bug: - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 Specifically, mac_in_use that is used to track mac addresses attached to ports; or ifindex that is used for SNMP. the short-term fix could be using vets, but “namespaceable” OVS ports would be perfect, yet I understand the change is a big feature. If we had 1 3, may be 4 wouldn’t be a problem anymore. I don't believe so. We still need namespaces for e.g. DHCP namespaces, or in any other cases where we need to run a service that is not capable of binding to a specific interface. [1] https://github.com/justinpettit/ovs/tree/conntrack Miguel Ángel Ajo On Friday, 13 de February de 2015 at 13:14, Ihar Hrachyshka wrote: Hi neutroners, we** had several conversations recently with our Red Hat fellows who work on openvswitch (Jiri Benc and Jiri Pirko) regarding the way neutron utilizes their software. Those were beneficial to both sides to understand what we do right and wrong. I was asked to share some of the points from those discussions with broader community. === One of the issues that came up during discussions is the way neutron connects ovs ports to namespaces. The short story is that openvswitch is not designed with namespaces in mind, and the fact that moving its ports into a different namespace works for neutron is mere coincidence, and is actually considered as a bug by openvswitch guys. It's not just broken in theory from software design standpoint, but also in practice. Specifically, 1. ovsdb dump does not work properly for namespaces: - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 This results in wrong statistics and other data collected for these ports; 2. We suspect that the following kernel crash is triggered because of our usage of the feature that is actually a bug: - https://bugs.launchpad.net/neutron/+bug/1418097 Quoting Jiri Benc, The problem is openvswitch does not support its ports to be moved to a different name space. The fact that it's possible to do that is a bug - such operation should have been denied. Unfortunately, due to a missing check, it's not been denied. Such setup does not work reliably, though, and leads to various issues from incorrect resource accounting to kernel crashes. We're aware of the bug but we don't have any easy way to fix it. The most obvious option, disabling moving of ovs ports to different name spaces, would be easy to do but it would break Neutron. The other option, making ovs to work across net name spaces, is hard and will require addition of different kernel APIs and large changes in ovs user space daemons. This constitutes tremendous amount of work. The tracker bug on openvswitch side is: - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 So in the best case, we may expect openvswitch to properly support the feature in long term, but short term it won't work, especially while neutron expects other features implemented in openvswitch for it (like NAT, or connection tracking, or ipv6 tunnel endpoints, to name a few). We could try to handle the issue neutron side. We can fix it by utilizing veth pairs to
Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful
In short term, we use veth pairs with namespace to fix the issue if performance is not impacted (Hopefully:) If performance downgrade too much, we may consider the following: 1) DHCP agent: use veth pairs with namespace since it is not critical path. 2) L3 agent: don't create port in OSV. Connect L3 agent without open switch VM --- Linux Bridge --- open switch -- (int-br-eth, phy-br-eth) --- physical switch vlan interface with namespace --- L3agent In long term, we may implement namespace and remove linux bridge. -Original Message- From: Ihar Hrachyshka [mailto:ihrac...@redhat.com] Sent: Friday, February 13, 2015 8:15 PM To: openstack-dev Cc: Jiri Benc; jpi...@redhat.com Subject: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi neutroners, we** had several conversations recently with our Red Hat fellows who work on openvswitch (Jiri Benc and Jiri Pirko) regarding the way neutron utilizes their software. Those were beneficial to both sides to understand what we do right and wrong. I was asked to share some of the points from those discussions with broader community. === One of the issues that came up during discussions is the way neutron connects ovs ports to namespaces. The short story is that openvswitch is not designed with namespaces in mind, and the fact that moving its ports into a different namespace works for neutron is mere coincidence, and is actually considered as a bug by openvswitch guys. It's not just broken in theory from software design standpoint, but also in practice. Specifically, 1. ovsdb dump does not work properly for namespaces: - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 This results in wrong statistics and other data collected for these ports; 2. We suspect that the following kernel crash is triggered because of our usage of the feature that is actually a bug: - - https://bugs.launchpad.net/neutron/+bug/1418097 Quoting Jiri Benc, The problem is openvswitch does not support its ports to be moved to a different name space. The fact that it's possible to do that is a bug - such operation should have been denied. Unfortunately, due to a missing check, it's not been denied. Such setup does not work reliably, though, and leads to various issues from incorrect resource accounting to kernel crashes. We're aware of the bug but we don't have any easy way to fix it. The most obvious option, disabling moving of ovs ports to different name spaces, would be easy to do but it would break Neutron. The other option, making ovs to work across net name spaces, is hard and will require addition of different kernel APIs and large changes in ovs user space daemons. This constitutes tremendous amount of work. The tracker bug on openvswitch side is: - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 So in the best case, we may expect openvswitch to properly support the feature in long term, but short term it won't work, especially while neutron expects other features implemented in openvswitch for it (like NAT, or connection tracking, or ipv6 tunnel endpoints, to name a few). We could try to handle the issue neutron side. We can fix it by utilizing veth pairs to get into namespaces, but it may mean worse performance, and will definitely require proper benchmarking to see whether we can live with performance drop. === There were other suggestions on how we can enhance our way of usage of openvswitch. Among those, getting rid of linux bridge used for security groups, with special attention on getting rid of ebtables (sic!) for they are a lot slower than iptables; getting rid of veth pair for instance ports. === I really encourage everyone to check the following video from devconf.cz 2015 on all that and more at: - - https://www.youtube.com/watch?v=BlLD-qh9EBQ Among other things, you will find presentation of plotnetcfg tool to create nice graphs of openstack state. If you're lazy enough and want to switch directly to the analysis of neutron problems, skip to ~18:00. I also encourage to check our the video around 30:00 on the way out of openvswitch for neutron (tc/eBPF). As crazy as encouraging. === While at it, I encourage everyone interested in controlling switches the open source way to check out presentation of the new kernel subsystem for that (I guess vaporware atm): - - https://www.youtube.com/watch?v=awekaJ7xWuw === So, that's what I have for you now. I really want to hear from everyone what is our plan to solve the problem neutron side. Comments? /Ihar **Jakub Libosvar and me -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEcBAEBAgAGBQJU3eq+AAoJEC5aWaUY1u57uyIH/2MRnU7Xr2ivfzDsqg1T1djN WgE6j87hVyIUnw/p/+vD1eDpOURPmZUcE/S7B6SCVv5KNB+j0pr22os5JM0cjCox zt63xz4GR/LibiJhyPnWtmSOqYdGFeTIdOj2TvovvOqtmI4MRmHoZy4fwShq0jXd RX00Z/o2DCxB+0KfJYQiWbFgXO43/zrdNGe9ME3XWI5TvVXQx49DMwv5K1jYN45Q oHsyqIGovi1bpYDFCaqe
Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/13/2015 01:47 PM, Miguel Ángel Ajo wrote: Sorry, I forgot about 5) If we put all our OVS/OF bridge logic in just one bridge (instead of N: br-tun, br-int, br-ex, br-xxx), the performance should be yet higher, since, as far as I understood, flow rule lookup could be more optimized into the kernel megaflows without forwarding and re-starting evaluation due to patch ports. (Please correct me here where I’m wrong, I just have very high level view of this). Indeed, that was also mentioned by Jiri in our private talks. That said, I'm as unaware of details here as you probably are (or more). Best, Miguel Ángel Ajo On Friday, 13 de February de 2015 at 13:42, Miguel Ángel Ajo wrote: Hi, Ihar Jiri, thank you for pointing this out. I’m working on the following items: 1) Doing Openflow traffic filtering (stateful firewall) based on OVS+CT[1] patch, which may eventually merge. Here I want to build a good amount of benchmarks to be able to compare the current network iptables+LB solution to just openflow. Openflow filtering should be fast, as it’s quite smart at using hashes to match OF rules in the kernel megaflows (thanks Jiri T.Graf for explaining me this) The only bad part is that we would have to dynamically change more rules based on security group changes (now we do that via ip sets without reloading all the rules). To do this properly, we may want to make the OVS plugin a real OF controller to be able to push OF rules to the bridge without the need of calling ovs-ofctl on the command line all the time. 2) Using OVS+OF to do QoS other interesting stuff to look at: 3) Doing routing in OF too, thanks to the NAT capabilities of having OVS+CT 4) The namespace problem, what kinds of statistics get broken by moving ports into namespaces now?, the short-term fix could be using vets, but “namespaceable” OVS ports would be perfect, yet I understand the change is a big feature. If we had 1 3, may be 4 wouldn’t be a problem anymore. [1] https://github.com/justinpettit/ovs/tree/conntrack Miguel Ángel Ajo On Friday, 13 de February de 2015 at 13:14, Ihar Hrachyshka wrote: Hi neutroners, we** had several conversations recently with our Red Hat fellows who work on openvswitch (Jiri Benc and Jiri Pirko) regarding the way neutron utilizes their software. Those were beneficial to both sides to understand what we do right and wrong. I was asked to share some of the points from those discussions with broader community. === One of the issues that came up during discussions is the way neutron connects ovs ports to namespaces. The short story is that openvswitch is not designed with namespaces in mind, and the fact that moving its ports into a different namespace works for neutron is mere coincidence, and is actually considered as a bug by openvswitch guys. It's not just broken in theory from software design standpoint, but also in practice. Specifically, 1. ovsdb dump does not work properly for namespaces: - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 This results in wrong statistics and other data collected for these ports; 2. We suspect that the following kernel crash is triggered because of our usage of the feature that is actually a bug: - https://bugs.launchpad.net/neutron/+bug/1418097 Quoting Jiri Benc, The problem is openvswitch does not support its ports to be moved to a different name space. The fact that it's possible to do that is a bug - such operation should have been denied. Unfortunately, due to a missing check, it's not been denied. Such setup does not work reliably, though, and leads to various issues from incorrect resource accounting to kernel crashes. We're aware of the bug but we don't have any easy way to fix it. The most obvious option, disabling moving of ovs ports to different name spaces, would be easy to do but it would break Neutron. The other option, making ovs to work across net name spaces, is hard and will require addition of different kernel APIs and large changes in ovs user space daemons. This constitutes tremendous amount of work. The tracker bug on openvswitch side is: - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 So in the best case, we may expect openvswitch to properly support the feature in long term, but short term it won't work, especially while neutron expects other features implemented in openvswitch for it (like NAT, or connection tracking, or ipv6 tunnel endpoints, to name a few). We could try to handle the issue neutron side. We can fix it by utilizing veth pairs to get into namespaces, but it may mean worse performance, and will definitely require proper benchmarking to see whether we can live with performance drop. === There were other suggestions on how we can enhance our way of usage of openvswitch. Among those, getting rid of linux bridge used for
Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful
Sorry, I forgot about 5) If we put all our OVS/OF bridge logic in just one bridge (instead of N: br-tun, br-int, br-ex, br-xxx), the performance should be yet higher, since, as far as I understood, flow rule lookup could be more optimized into the kernel megaflows without forwarding and re-starting evaluation due to patch ports. (Please correct me here where I’m wrong, I just have very high level view of this). Best, Miguel Ángel Ajo On Friday, 13 de February de 2015 at 13:42, Miguel Ángel Ajo wrote: Hi, Ihar Jiri, thank you for pointing this out. I’m working on the following items: 1) Doing Openflow traffic filtering (stateful firewall) based on OVS+CT[1] patch, which may eventually merge. Here I want to build a good amount of benchmarks to be able to compare the current network iptables+LB solution to just openflow. Openflow filtering should be fast, as it’s quite smart at using hashes to match OF rules in the kernel megaflows (thanks Jiri T.Graf for explaining me this) The only bad part is that we would have to dynamically change more rules based on security group changes (now we do that via ip sets without reloading all the rules). To do this properly, we may want to make the OVS plugin a real OF controller to be able to push OF rules to the bridge without the need of calling ovs-ofctl on the command line all the time. 2) Using OVS+OF to do QoS other interesting stuff to look at: 3) Doing routing in OF too, thanks to the NAT capabilities of having OVS+CT 4) The namespace problem, what kinds of statistics get broken by moving ports into namespaces now?, the short-term fix could be using vets, but “namespaceable” OVS ports would be perfect, yet I understand the change is a big feature. If we had 1 3, may be 4 wouldn’t be a problem anymore. [1] https://github.com/justinpettit/ovs/tree/conntrack Miguel Ángel Ajo On Friday, 13 de February de 2015 at 13:14, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi neutroners, we** had several conversations recently with our Red Hat fellows who work on openvswitch (Jiri Benc and Jiri Pirko) regarding the way neutron utilizes their software. Those were beneficial to both sides to understand what we do right and wrong. I was asked to share some of the points from those discussions with broader community. === One of the issues that came up during discussions is the way neutron connects ovs ports to namespaces. The short story is that openvswitch is not designed with namespaces in mind, and the fact that moving its ports into a different namespace works for neutron is mere coincidence, and is actually considered as a bug by openvswitch guys. It's not just broken in theory from software design standpoint, but also in practice. Specifically, 1. ovsdb dump does not work properly for namespaces: - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 This results in wrong statistics and other data collected for these ports; 2. We suspect that the following kernel crash is triggered because of our usage of the feature that is actually a bug: - - https://bugs.launchpad.net/neutron/+bug/1418097 Quoting Jiri Benc, The problem is openvswitch does not support its ports to be moved to a different name space. The fact that it's possible to do that is a bug - such operation should have been denied. Unfortunately, due to a missing check, it's not been denied. Such setup does not work reliably, though, and leads to various issues from incorrect resource accounting to kernel crashes. We're aware of the bug but we don't have any easy way to fix it. The most obvious option, disabling moving of ovs ports to different name spaces, would be easy to do but it would break Neutron. The other option, making ovs to work across net name spaces, is hard and will require addition of different kernel APIs and large changes in ovs user space daemons. This constitutes tremendous amount of work. The tracker bug on openvswitch side is: - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 So in the best case, we may expect openvswitch to properly support the feature in long term, but short term it won't work, especially while neutron expects other features implemented in openvswitch for it (like NAT, or connection tracking, or ipv6 tunnel endpoints, to name a few). We could try to handle the issue neutron side. We can fix it by utilizing veth pairs to get into namespaces, but it may mean worse performance, and will definitely require proper benchmarking to see whether we can live with performance drop. === There were other suggestions on how we can enhance our way of usage of openvswitch. Among those, getting rid of linux bridge used for security groups, with
[openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi neutroners, we** had several conversations recently with our Red Hat fellows who work on openvswitch (Jiri Benc and Jiri Pirko) regarding the way neutron utilizes their software. Those were beneficial to both sides to understand what we do right and wrong. I was asked to share some of the points from those discussions with broader community. === One of the issues that came up during discussions is the way neutron connects ovs ports to namespaces. The short story is that openvswitch is not designed with namespaces in mind, and the fact that moving its ports into a different namespace works for neutron is mere coincidence, and is actually considered as a bug by openvswitch guys. It's not just broken in theory from software design standpoint, but also in practice. Specifically, 1. ovsdb dump does not work properly for namespaces: - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 This results in wrong statistics and other data collected for these ports; 2. We suspect that the following kernel crash is triggered because of our usage of the feature that is actually a bug: - - https://bugs.launchpad.net/neutron/+bug/1418097 Quoting Jiri Benc, The problem is openvswitch does not support its ports to be moved to a different name space. The fact that it's possible to do that is a bug - such operation should have been denied. Unfortunately, due to a missing check, it's not been denied. Such setup does not work reliably, though, and leads to various issues from incorrect resource accounting to kernel crashes. We're aware of the bug but we don't have any easy way to fix it. The most obvious option, disabling moving of ovs ports to different name spaces, would be easy to do but it would break Neutron. The other option, making ovs to work across net name spaces, is hard and will require addition of different kernel APIs and large changes in ovs user space daemons. This constitutes tremendous amount of work. The tracker bug on openvswitch side is: - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 So in the best case, we may expect openvswitch to properly support the feature in long term, but short term it won't work, especially while neutron expects other features implemented in openvswitch for it (like NAT, or connection tracking, or ipv6 tunnel endpoints, to name a few). We could try to handle the issue neutron side. We can fix it by utilizing veth pairs to get into namespaces, but it may mean worse performance, and will definitely require proper benchmarking to see whether we can live with performance drop. === There were other suggestions on how we can enhance our way of usage of openvswitch. Among those, getting rid of linux bridge used for security groups, with special attention on getting rid of ebtables (sic!) for they are a lot slower than iptables; getting rid of veth pair for instance ports. === I really encourage everyone to check the following video from devconf.cz 2015 on all that and more at: - - https://www.youtube.com/watch?v=BlLD-qh9EBQ Among other things, you will find presentation of plotnetcfg tool to create nice graphs of openstack state. If you're lazy enough and want to switch directly to the analysis of neutron problems, skip to ~18:00. I also encourage to check our the video around 30:00 on the way out of openvswitch for neutron (tc/eBPF). As crazy as encouraging. === While at it, I encourage everyone interested in controlling switches the open source way to check out presentation of the new kernel subsystem for that (I guess vaporware atm): - - https://www.youtube.com/watch?v=awekaJ7xWuw === So, that's what I have for you now. I really want to hear from everyone what is our plan to solve the problem neutron side. Comments? /Ihar **Jakub Libosvar and me -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEcBAEBAgAGBQJU3eq+AAoJEC5aWaUY1u57uyIH/2MRnU7Xr2ivfzDsqg1T1djN WgE6j87hVyIUnw/p/+vD1eDpOURPmZUcE/S7B6SCVv5KNB+j0pr22os5JM0cjCox zt63xz4GR/LibiJhyPnWtmSOqYdGFeTIdOj2TvovvOqtmI4MRmHoZy4fwShq0jXd RX00Z/o2DCxB+0KfJYQiWbFgXO43/zrdNGe9ME3XWI5TvVXQx49DMwv5K1jYN45Q oHsyqIGovi1bpYDFCaqe+CPuRh5xO8SVrOHa+JHURgW8N0JHzDSN31zLT45roMp4 AqmBkZgjAG6WvsJWwYcQdoBEUdy5l0ml/86qysC5yFVdntHe2pfzMkpeZyLFNho= =4OWY -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful
Hi, Ihar Jiri, thank you for pointing this out. I’m working on the following items: 1) Doing Openflow traffic filtering (stateful firewall) based on OVS+CT[1] patch, which may eventually merge. Here I want to build a good amount of benchmarks to be able to compare the current network iptables+LB solution to just openflow. Openflow filtering should be fast, as it’s quite smart at using hashes to match OF rules in the kernel megaflows (thanks Jiri T.Graf for explaining me this) The only bad part is that we would have to dynamically change more rules based on security group changes (now we do that via ip sets without reloading all the rules). To do this properly, we may want to make the OVS plugin a real OF controller to be able to push OF rules to the bridge without the need of calling ovs-ofctl on the command line all the time. 2) Using OVS+OF to do QoS other interesting stuff to look at: 3) Doing routing in OF too, thanks to the NAT capabilities of having OVS+CT 4) The namespace problem, what kinds of statistics get broken by moving ports into namespaces now?, the short-term fix could be using vets, but “namespaceable” OVS ports would be perfect, yet I understand the change is a big feature. If we had 1 3, may be 4 wouldn’t be a problem anymore. [1] https://github.com/justinpettit/ovs/tree/conntrack Miguel Ángel Ajo On Friday, 13 de February de 2015 at 13:14, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi neutroners, we** had several conversations recently with our Red Hat fellows who work on openvswitch (Jiri Benc and Jiri Pirko) regarding the way neutron utilizes their software. Those were beneficial to both sides to understand what we do right and wrong. I was asked to share some of the points from those discussions with broader community. === One of the issues that came up during discussions is the way neutron connects ovs ports to namespaces. The short story is that openvswitch is not designed with namespaces in mind, and the fact that moving its ports into a different namespace works for neutron is mere coincidence, and is actually considered as a bug by openvswitch guys. It's not just broken in theory from software design standpoint, but also in practice. Specifically, 1. ovsdb dump does not work properly for namespaces: - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 This results in wrong statistics and other data collected for these ports; 2. We suspect that the following kernel crash is triggered because of our usage of the feature that is actually a bug: - - https://bugs.launchpad.net/neutron/+bug/1418097 Quoting Jiri Benc, The problem is openvswitch does not support its ports to be moved to a different name space. The fact that it's possible to do that is a bug - such operation should have been denied. Unfortunately, due to a missing check, it's not been denied. Such setup does not work reliably, though, and leads to various issues from incorrect resource accounting to kernel crashes. We're aware of the bug but we don't have any easy way to fix it. The most obvious option, disabling moving of ovs ports to different name spaces, would be easy to do but it would break Neutron. The other option, making ovs to work across net name spaces, is hard and will require addition of different kernel APIs and large changes in ovs user space daemons. This constitutes tremendous amount of work. The tracker bug on openvswitch side is: - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340 So in the best case, we may expect openvswitch to properly support the feature in long term, but short term it won't work, especially while neutron expects other features implemented in openvswitch for it (like NAT, or connection tracking, or ipv6 tunnel endpoints, to name a few). We could try to handle the issue neutron side. We can fix it by utilizing veth pairs to get into namespaces, but it may mean worse performance, and will definitely require proper benchmarking to see whether we can live with performance drop. === There were other suggestions on how we can enhance our way of usage of openvswitch. Among those, getting rid of linux bridge used for security groups, with special attention on getting rid of ebtables (sic!) for they are a lot slower than iptables; getting rid of veth pair for instance ports. === I really encourage everyone to check the following video from devconf.cz (http://devconf.cz) 2015 on all that and more at: - - https://www.youtube.com/watch?v=BlLD-qh9EBQ Among other things, you will find presentation of plotnetcfg tool to create nice graphs of openstack state. If you're lazy enough and want to switch directly to the analysis of neutron problems, skip to ~18:00. I also encourage to check our the video around 30:00 on the way out of openvswitch for neutron (tc/eBPF).
Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful
Surely eliminating linux bridge for iptables by ovs+tc is quite important for performance. On Fri, Feb 13, 2015 at 01:57:46PM +0100, Ihar Hrachyshka ihrac...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/13/2015 01:47 PM, Miguel Ángel Ajo wrote: Sorry, I forgot about 5) If we put all our OVS/OF bridge logic in just one bridge (instead of N: br-tun, br-int, br-ex, br-xxx), the performance should be yet higher, since, as far as I understood, flow rule lookup could be more optimized into the kernel megaflows without forwarding and re-starting evaluation due to patch ports. (Please correct me here where I’m wrong, I just have very high level view of this). Indeed, that was also mentioned by Jiri in our private talks. That said, I'm as unaware of details here as you probably are (or more). The current ovs instantiates only kernel datapath and optimizes out patch port in kernel. the patch port logic is handled by ovs-vswitchd side when building flow rules for kernel datapath. I don't know which version, so you may be using old versions... Or are you referring to recirculation? 2) Using OVS+OF to do QoS other interesting stuff to look at: What exactly do you mean? marking packet or tc queuing? thanks, -- Isaku Yamahata isaku.yamah...@gmail.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
