Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful

2015-02-16 Thread Ivar Lazzaro
I agree with Kevin that we should adopt veth pairs for fixing the issue in
the short term, at least until CT gets merged and distributed in OVS. At
that point the transition to a OVS based solution will make a lot of sense,
given that the numbers show that it's worth of course ;)

On Sun Feb 15 2015 at 7:17:39 AM Thomas Graf tg...@noironetworks.com
wrote:

 FYI: Ivar (CCed) is also working on collecting numbers to compare both
 architectures to kick of a discussion at the next summit. Ivar, can
 you link to the talk proposal?


Thanks for bringing this up Thomas! Here is the link to the talk proposal
[0].
Any with suggestion, idea or random comment is very welcome!

Ivar.

[0]
https://www.openstack.org/vote-paris/presentation/taking-security-groups-to-ludicrous-speed-with-open-vswitch
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful

2015-02-15 Thread Thomas Graf
[Sorry for the resend, I had to subscribe to openstack-dev first,
 maybe worth removing the subscribe requirement for outsiders]

[Copying ovs-dev]

On 02/13/15 at 01:47pm, Miguel Ángel Ajo wrote:
 Sorry, I forgot about   
 
 5)  If we put all our OVS/OF bridge logic in just one bridge (instead of N: 
 br-tun, br-int, br-ex, br-xxx),
  the performance should be yet higher, since, as far as I understood, 
 flow rule lookup could be more
  optimized into the kernel megaflows without forwarding and re-starting 
 evaluation due to patch ports.
  (Please correct me here where I’m wrong, I just have very high level 
 view of this).

Some preliminary numbers were presented at the OVS Fall Conference 2014
which indicate that a pure OVS ACL solution scales better as the
number of rules changes. You can find the number on slide 9 here:

http://www.openvswitch.org/support/ovscon2014/17/1030-conntrack_nat.pdf

Another obvious advantage is that since we have to go through the OVS
flow table anyway, traversing any additional (linear) ruleset is
likely to have more overhead.

FYI: Ivar (CCed) is also working on collecting numbers to compare both
architectures to kick of a discussion at the next summit. Ivar, can
you link to the talk proposal?

 On Friday, 13 de February de 2015 at 13:42, Miguel Ángel Ajo wrote:
 
  I’m working on the following items:
   
  1) Doing Openflow traffic filtering (stateful firewall) based on OVS+CT[1] 
  patch, which may
  eventually merge. Here I want to build a good amount of benchmarks to 
  be able to compare
  the current network iptables+LB solution to just openflow.
   
   Openflow filtering should be fast, as it’s quite smart at using hashes 
  to match OF rules
   in the kernel megaflows (thanks Jiri  T.Graf for explaining me this)
  
   The only bad part is that we would have to dynamically change more 
  rules based on security
  group changes (now we do that via ip sets without reloading all the rules).

Worth pointing out that it is entirely possible to integrate ipset
with OVS in the datapath in case representing ipsets with individual
wildcard flows is not sufficient. I guess we'll know when we have more
numbers.

To do this properly, we may want to make the OVS plugin a real OF 
  controller to be able to
  push OF rules to the bridge without the need of calling ovs-ofctl on the 
  command line all the time.

We should synchronize this effort with the OVN effort. There is a lot
of overlap.

  2) Using OVS+OF to do QoS
   
  other interesting stuff to look at:
   
  3) Doing routing in OF too, thanks to the NAT capabilities of having OVS+CT 
   

Just want to point out that this is still WIP with several issues
outstanding. I think everybody agrees that it's tremendously useful
to have, we need to be able to implement it properly. I'll let you
and anybody else interested know as soon as it's ready for testing.

  4) The namespace problem, what kinds of statistics get broken by moving 
  ports into namespaces now?,
  the short-term fix could be using vets, but “namespaceable” OVS ports 
  would be perfect, yet I understand
  the change is a big feature.
   
  If we had 1  3, may be 4 wouldn’t be a problem anymore.

Implementing VRF in OVS will hide (4) for OpenStack but we should
still fix it in OVS as Ben suggested in the Bugzilla. It looks
feasible to support netns properly in OVS.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful

2015-02-15 Thread Kevin Benton
What is the status of the conntrack integration with respect to
availability in distributions? The lack of state tracking has blocked the
ability for us to get rid of namespaces for the L3 agent (because of SNAT)
and the filtering bridge between the VM and OVS (stateful firewall for
security groups).

It has been known for a long time that these are suboptimal, but our hands
are sort of tied because we don't want to require kernel code changes to
use Neutron.

Are Ubuntu 1404 or CentOS 7 shipping openvswitch kernel modules with
conntrack integration? If not, I don't see a feasible way of eliminating
any of these problems with a pure OVS solution. (faking a stateful firewall
with flag matching doesn't count)

In the short term, we should probably switch back to veth pairs to handle
the namespace issue for the dhcp agent and the L3 agent.
[Sorry for the resend, I had to subscribe to openstack-dev first,
 maybe worth removing the subscribe requirement for outsiders]

[Copying ovs-dev]

On 02/13/15 at 01:47pm, Miguel Ángel Ajo wrote:
 Sorry, I forgot about

 5)  If we put all our OVS/OF bridge logic in just one bridge (instead of
N: br-tun, br-int, br-ex, br-xxx),
  the performance should be yet higher, since, as far as I understood,
flow rule lookup could be more
  optimized into the kernel megaflows without forwarding and
re-starting evaluation due to patch ports.
  (Please correct me here where I’m wrong, I just have very high level
view of this).

Some preliminary numbers were presented at the OVS Fall Conference 2014
which indicate that a pure OVS ACL solution scales better as the
number of rules changes. You can find the number on slide 9 here:

http://www.openvswitch.org/support/ovscon2014/17/1030-conntrack_nat.pdf

Another obvious advantage is that since we have to go through the OVS
flow table anyway, traversing any additional (linear) ruleset is
likely to have more overhead.

FYI: Ivar (CCed) is also working on collecting numbers to compare both
architectures to kick of a discussion at the next summit. Ivar, can
you link to the talk proposal?

 On Friday, 13 de February de 2015 at 13:42, Miguel Ángel Ajo wrote:

  I’m working on the following items:
 
  1) Doing Openflow traffic filtering (stateful firewall) based on
OVS+CT[1] patch, which may
  eventually merge. Here I want to build a good amount of benchmarks
to be able to compare
  the current network iptables+LB solution to just openflow.
 
   Openflow filtering should be fast, as it’s quite smart at using
hashes to match OF rules
   in the kernel megaflows (thanks Jiri  T.Graf for explaining me
this)
 
   The only bad part is that we would have to dynamically change more
rules based on security
  group changes (now we do that via ip sets without reloading all the
rules).

Worth pointing out that it is entirely possible to integrate ipset
with OVS in the datapath in case representing ipsets with individual
wildcard flows is not sufficient. I guess we'll know when we have more
numbers.

To do this properly, we may want to make the OVS plugin a real OF
controller to be able to
  push OF rules to the bridge without the need of calling ovs-ofctl on
the command line all the time.

We should synchronize this effort with the OVN effort. There is a lot
of overlap.

  2) Using OVS+OF to do QoS
 
  other interesting stuff to look at:
 
  3) Doing routing in OF too, thanks to the NAT capabilities of having
OVS+CT

Just want to point out that this is still WIP with several issues
outstanding. I think everybody agrees that it's tremendously useful
to have, we need to be able to implement it properly. I'll let you
and anybody else interested know as soon as it's ready for testing.

  4) The namespace problem, what kinds of statistics get broken by moving
ports into namespaces now?,
  the short-term fix could be using vets, but “namespaceable” OVS
ports would be perfect, yet I understand
  the change is a big feature.
 
  If we had 1  3, may be 4 wouldn’t be a problem anymore.

Implementing VRF in OVS will hide (4) for OpenStack but we should
still fix it in OVS as Ben suggested in the Bugzilla. It looks
feasible to support netns properly in OVS.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful

2015-02-13 Thread Ihar Hrachyshka
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/13/2015 01:42 PM, Miguel Ángel Ajo wrote:
 Hi, Ihar  Jiri, thank you for pointing this out.
 
 I’m working on the following items:
 
 1) Doing Openflow traffic filtering (stateful firewall) based on
 OVS+CT[1] patch, which may
 eventually merge. Here I want to build a good amount of benchmarks
 to be able to compare
 the current network iptables+LB solution to just openflow.
 
  Openflow filtering should be fast, as it’s quite smart at using
 hashes to match OF rules
  in the kernel megaflows (thanks Jiri  T.Graf for explaining me this)

  The only bad part is that we would have to dynamically change more
 rules based on security
 group changes (now we do that via ip sets without reloading all the rules).
 
   To do this properly, we may want to make the OVS plugin a real OF
 controller to be able to
 push OF rules to the bridge without the need of calling ovs-ofctl on the
 command line all the time.

I am actually a bit worried about this project being bound so tightly to
openvswitch. There are common linux alternatives in the wild that do not
require it at all (tc, eBPF). Of course, openvswitch could be utilized
in some exotic cases, like controlling HyperV instances in Windows
environment, but for usual Linux platform, utilizing something like
openvswitch seems to be an overkill, while we don't use any of
controller separation features and do propagate updates via AMQP to each
controlled node.

 
 2) Using OVS+OF to do QoS
 
 other interesting stuff to look at:
 
 3) Doing routing in OF too, thanks to the NAT capabilities of having OVS+CT 
 
 4) The namespace problem, what kinds of statistics get broken by moving
 ports into namespaces now?,

I think some of those are already mentioned in the ovs tracker bug:
- - https://bugzilla.redhat.com/show_bug.cgi?id=1160340

Specifically, mac_in_use that is used to track mac addresses attached to
ports; or ifindex that is used for SNMP.

 the short-term fix could be using vets, but “namespaceable” OVS
 ports would be perfect, yet I understand
 the change is a big feature.
 
 If we had 1  3, may be 4 wouldn’t be a problem anymore.

I don't believe so. We still need namespaces for e.g. DHCP namespaces,
or in any other cases where we need to run a service that is not capable
of binding to a specific interface.

 
 [1] https://github.com/justinpettit/ovs/tree/conntrack
 
 Miguel Ángel Ajo
 
 On Friday, 13 de February de 2015 at 13:14, Ihar Hrachyshka wrote:
 
 Hi neutroners,
 
 we** had several conversations recently with our Red Hat fellows who
 work on openvswitch (Jiri Benc and Jiri Pirko) regarding the way
 neutron utilizes their software. Those were beneficial to both sides
 to understand what we do right and wrong. I was asked to share some of
 the points from those discussions with broader community.
 
 ===
 
 One of the issues that came up during discussions is the way neutron
 connects ovs ports to namespaces. The short story is that openvswitch
 is not designed with namespaces in mind, and the fact that moving its
 ports into a different namespace works for neutron is mere
 coincidence, and is actually considered as a bug by openvswitch guys.
 
 It's not just broken in theory from software design standpoint, but
 also in practice. Specifically,
 
 1. ovsdb dump does not work properly for namespaces:
 - https://bugzilla.redhat.com/show_bug.cgi?id=1160340
 
 This results in wrong statistics and other data collected for these ports;
 
 2. We suspect that the following kernel crash is triggered because of
 our usage of the feature that is actually a bug:
 - https://bugs.launchpad.net/neutron/+bug/1418097
 
 Quoting Jiri Benc,
 
 The problem is openvswitch does not support its ports to be moved to
 a different name space. The fact that it's possible to do that is a
 bug - such operation should have been denied. Unfortunately, due to a
 missing check, it's not been denied. Such setup does not work
 reliably, though, and leads to various issues from incorrect resource
 accounting to kernel crashes.
 
 We're aware of the bug but we don't have any easy way to fix it. The
 most obvious option, disabling moving of ovs ports to different name
 spaces, would be easy to do but it would break Neutron. The other
 option, making ovs to work across net name spaces, is hard and will
 require addition of different kernel APIs and large changes in ovs
 user space daemons. This constitutes tremendous amount of work.
 
 The tracker bug on openvswitch side is:
 - https://bugzilla.redhat.com/show_bug.cgi?id=1160340
 
 So in the best case, we may expect openvswitch to properly support the
 feature in long term, but short term it won't work, especially while
 neutron expects other features implemented in openvswitch for it (like
 NAT, or connection tracking, or ipv6 tunnel endpoints, to name a few).
 
 We could try to handle the issue neutron side. We can fix it by
 utilizing veth pairs to 

Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful

2015-02-13 Thread Guo, Ruijing
In short term, we use veth pairs with namespace to fix the issue if performance 
is not impacted (Hopefully:)

If performance downgrade too much, we may consider the following:

1) DHCP agent: use veth pairs with namespace since it is not critical path.

2) L3 agent: don't create port in OSV.  Connect L3 agent without open switch

VM --- Linux Bridge --- open switch -- (int-br-eth, phy-br-eth) --- 
physical switch  vlan interface with namespace --- L3agent

In long term, we may implement namespace and remove linux bridge.

-Original Message-
From: Ihar Hrachyshka [mailto:ihrac...@redhat.com] 
Sent: Friday, February 13, 2015 8:15 PM
To: openstack-dev
Cc: Jiri Benc; jpi...@redhat.com
Subject: [openstack-dev] [neutron] moving openvswitch ports between namespaces 
considered harmful

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi neutroners,

we** had several conversations recently with our Red Hat fellows who work on 
openvswitch (Jiri Benc and Jiri Pirko) regarding the way neutron utilizes their 
software. Those were beneficial to both sides to understand what we do right 
and wrong. I was asked to share some of the points from those discussions with 
broader community.

===

One of the issues that came up during discussions is the way neutron connects 
ovs ports to namespaces. The short story is that openvswitch is not designed 
with namespaces in mind, and the fact that moving its ports into a different 
namespace works for neutron is mere coincidence, and is actually considered as 
a bug by openvswitch guys.

It's not just broken in theory from software design standpoint, but also in 
practice. Specifically,

1. ovsdb dump does not work properly for namespaces:
- - https://bugzilla.redhat.com/show_bug.cgi?id=1160340

This results in wrong statistics and other data collected for these ports;

2. We suspect that the following kernel crash is triggered because of our usage 
of the feature that is actually a bug:
- - https://bugs.launchpad.net/neutron/+bug/1418097

Quoting Jiri Benc,

The problem is openvswitch does not support its ports to be moved to a 
different name space. The fact that it's possible to do that is a bug - such 
operation should have been denied. Unfortunately, due to a missing check, it's 
not been denied. Such setup does not work reliably, though, and leads to 
various issues from incorrect resource accounting to kernel crashes.

We're aware of the bug but we don't have any easy way to fix it. The most 
obvious option, disabling moving of ovs ports to different name spaces, would 
be easy to do but it would break Neutron. The other option, making ovs to work 
across net name spaces, is hard and will require addition of different kernel 
APIs and large changes in ovs user space daemons. This constitutes tremendous 
amount of work.

The tracker bug on openvswitch side is:
- - https://bugzilla.redhat.com/show_bug.cgi?id=1160340

So in the best case, we may expect openvswitch to properly support the feature 
in long term, but short term it won't work, especially while neutron expects 
other features implemented in openvswitch for it (like NAT, or connection 
tracking, or ipv6 tunnel endpoints, to name a few).

We could try to handle the issue neutron side. We can fix it by utilizing veth 
pairs to get into namespaces, but it may mean worse performance, and will 
definitely require proper benchmarking to see whether we can live with 
performance drop.

===

There were other suggestions on how we can enhance our way of usage of 
openvswitch. Among those, getting rid of linux bridge used for security groups, 
with special attention on getting rid of ebtables
(sic!) for they are a lot slower than iptables; getting rid of veth pair for 
instance ports.

===

I really encourage everyone to check the following video from devconf.cz 2015 
on all that and more at:

- - https://www.youtube.com/watch?v=BlLD-qh9EBQ

Among other things, you will find presentation of plotnetcfg tool to create 
nice graphs of openstack state.

If you're lazy enough and want to switch directly to the analysis of neutron 
problems, skip to ~18:00.

I also encourage to check our the video around 30:00 on the way out of 
openvswitch for neutron (tc/eBPF). As crazy as encouraging.

===

While at it, I encourage everyone interested in controlling switches the open 
source way to check out presentation of the new kernel subsystem for that (I 
guess vaporware atm):

- - https://www.youtube.com/watch?v=awekaJ7xWuw

===

So, that's what I have for you now. I really want to hear from everyone what is 
our plan to solve the problem neutron side.

Comments?

/Ihar

**Jakub Libosvar and me
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEcBAEBAgAGBQJU3eq+AAoJEC5aWaUY1u57uyIH/2MRnU7Xr2ivfzDsqg1T1djN
WgE6j87hVyIUnw/p/+vD1eDpOURPmZUcE/S7B6SCVv5KNB+j0pr22os5JM0cjCox
zt63xz4GR/LibiJhyPnWtmSOqYdGFeTIdOj2TvovvOqtmI4MRmHoZy4fwShq0jXd
RX00Z/o2DCxB+0KfJYQiWbFgXO43/zrdNGe9ME3XWI5TvVXQx49DMwv5K1jYN45Q

Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful

2015-02-13 Thread Ihar Hrachyshka
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/13/2015 01:47 PM, Miguel Ángel Ajo wrote:
 Sorry, I forgot about
 
 5)  If we put all our OVS/OF bridge logic in just one bridge
 (instead of N: br-tun, br-int, br-ex, br-xxx), the performance
 should be yet higher, since, as far as I understood, flow rule
 lookup could be more optimized into the kernel megaflows without
 forwarding and re-starting evaluation due to patch ports. (Please
 correct me here where I’m wrong, I just have very high level view
 of this).

Indeed, that was also mentioned by Jiri in our private talks. That
said, I'm as unaware of details here as you probably are (or more).

 
 Best, Miguel Ángel Ajo
 
 On Friday, 13 de February de 2015 at 13:42, Miguel Ángel Ajo
 wrote:
 
 Hi, Ihar  Jiri, thank you for pointing this out.
 
 I’m working on the following items:
 
 1) Doing Openflow traffic filtering (stateful firewall) based on 
 OVS+CT[1] patch, which may eventually merge. Here I want to build
 a good amount of benchmarks to be able to compare the current
 network iptables+LB solution to just openflow.
 
 Openflow filtering should be fast, as it’s quite smart at using 
 hashes to match OF rules in the kernel megaflows (thanks Jiri 
 T.Graf for explaining me this)
 
 The only bad part is that we would have to dynamically change 
 more rules based on security group changes (now we do that via ip
 sets without reloading all the rules).
 
 To do this properly, we may want to make the OVS plugin a real OF
 controller to be able to push OF rules to the bridge without the
 need of calling ovs-ofctl on the command line all the time.
 
 2) Using OVS+OF to do QoS
 
 other interesting stuff to look at:
 
 3) Doing routing in OF too, thanks to the NAT capabilities of
 having OVS+CT
 
 4) The namespace problem, what kinds of statistics get broken by 
 moving ports into namespaces now?, the short-term fix could be
 using vets, but “namespaceable” OVS ports would be perfect, yet I
 understand the change is a big feature.
 
 If we had 1  3, may be 4 wouldn’t be a problem anymore.
 
 [1] https://github.com/justinpettit/ovs/tree/conntrack
 
 Miguel Ángel Ajo
 
 On Friday, 13 de February de 2015 at 13:14, Ihar Hrachyshka
 wrote:
 
 Hi neutroners,
 
 we** had several conversations recently with our Red Hat fellows
 who work on openvswitch (Jiri Benc and Jiri Pirko) regarding the
 way neutron utilizes their software. Those were beneficial to both
 sides to understand what we do right and wrong. I was asked to
 share some of the points from those discussions with broader
 community.
 
 ===
 
 One of the issues that came up during discussions is the way
 neutron connects ovs ports to namespaces. The short story is that
 openvswitch is not designed with namespaces in mind, and the fact
 that moving its ports into a different namespace works for neutron
 is mere coincidence, and is actually considered as a bug by
 openvswitch guys.
 
 It's not just broken in theory from software design standpoint,
 but also in practice. Specifically,
 
 1. ovsdb dump does not work properly for namespaces: -
 https://bugzilla.redhat.com/show_bug.cgi?id=1160340
 
 This results in wrong statistics and other data collected for
 these ports;
 
 2. We suspect that the following kernel crash is triggered because
 of our usage of the feature that is actually a bug: -
 https://bugs.launchpad.net/neutron/+bug/1418097
 
 Quoting Jiri Benc,
 
 The problem is openvswitch does not support its ports to be moved
 to a different name space. The fact that it's possible to do that
 is a bug - such operation should have been denied. Unfortunately,
 due to a missing check, it's not been denied. Such setup does not
 work reliably, though, and leads to various issues from incorrect
 resource accounting to kernel crashes.
 
 We're aware of the bug but we don't have any easy way to fix it.
 The most obvious option, disabling moving of ovs ports to different
 name spaces, would be easy to do but it would break Neutron. The
 other option, making ovs to work across net name spaces, is hard
 and will require addition of different kernel APIs and large
 changes in ovs user space daemons. This constitutes tremendous
 amount of work.
 
 The tracker bug on openvswitch side is: -
 https://bugzilla.redhat.com/show_bug.cgi?id=1160340
 
 So in the best case, we may expect openvswitch to properly support
 the feature in long term, but short term it won't work, especially
 while neutron expects other features implemented in openvswitch for
 it (like NAT, or connection tracking, or ipv6 tunnel endpoints, to
 name a few).
 
 We could try to handle the issue neutron side. We can fix it by 
 utilizing veth pairs to get into namespaces, but it may mean worse 
 performance, and will definitely require proper benchmarking to
 see whether we can live with performance drop.
 
 ===
 
 There were other suggestions on how we can enhance our way of usage
 of openvswitch. Among those, getting rid of linux bridge used for 
 

Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful

2015-02-13 Thread Miguel Ángel Ajo
Sorry, I forgot about   

5)  If we put all our OVS/OF bridge logic in just one bridge (instead of N: 
br-tun, br-int, br-ex, br-xxx),
 the performance should be yet higher, since, as far as I understood, flow 
rule lookup could be more
 optimized into the kernel megaflows without forwarding and re-starting 
evaluation due to patch ports.
 (Please correct me here where I’m wrong, I just have very high level view 
of this).

Best,
Miguel Ángel Ajo


On Friday, 13 de February de 2015 at 13:42, Miguel Ángel Ajo wrote:

 Hi, Ihar  Jiri, thank you for pointing this out.
  
 I’m working on the following items:
  
 1) Doing Openflow traffic filtering (stateful firewall) based on OVS+CT[1] 
 patch, which may
 eventually merge. Here I want to build a good amount of benchmarks to be 
 able to compare
 the current network iptables+LB solution to just openflow.
  
  Openflow filtering should be fast, as it’s quite smart at using hashes 
 to match OF rules
  in the kernel megaflows (thanks Jiri  T.Graf for explaining me this)
 
  The only bad part is that we would have to dynamically change more rules 
 based on security
 group changes (now we do that via ip sets without reloading all the rules).
  
   To do this properly, we may want to make the OVS plugin a real OF 
 controller to be able to
 push OF rules to the bridge without the need of calling ovs-ofctl on the 
 command line all the time.
  
 2) Using OVS+OF to do QoS
  
 other interesting stuff to look at:
  
 3) Doing routing in OF too, thanks to the NAT capabilities of having OVS+CT  
  
 4) The namespace problem, what kinds of statistics get broken by moving ports 
 into namespaces now?,
 the short-term fix could be using vets, but “namespaceable” OVS ports 
 would be perfect, yet I understand
 the change is a big feature.
  
 If we had 1  3, may be 4 wouldn’t be a problem anymore.
  
 [1] https://github.com/justinpettit/ovs/tree/conntrack  
  
 Miguel Ángel Ajo
  
  
 On Friday, 13 de February de 2015 at 13:14, Ihar Hrachyshka wrote:
  
  -BEGIN PGP SIGNED MESSAGE-
  Hash: SHA1
   
  Hi neutroners,
   
  we** had several conversations recently with our Red Hat fellows who
  work on openvswitch (Jiri Benc and Jiri Pirko) regarding the way
  neutron utilizes their software. Those were beneficial to both sides
  to understand what we do right and wrong. I was asked to share some of
  the points from those discussions with broader community.
   
  ===
   
  One of the issues that came up during discussions is the way neutron
  connects ovs ports to namespaces. The short story is that openvswitch
  is not designed with namespaces in mind, and the fact that moving its
  ports into a different namespace works for neutron is mere
  coincidence, and is actually considered as a bug by openvswitch guys.
   
  It's not just broken in theory from software design standpoint, but
  also in practice. Specifically,
   
  1. ovsdb dump does not work properly for namespaces:
  - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340
   
  This results in wrong statistics and other data collected for these ports;
   
  2. We suspect that the following kernel crash is triggered because of
  our usage of the feature that is actually a bug:
  - - https://bugs.launchpad.net/neutron/+bug/1418097
   
  Quoting Jiri Benc,
   
  The problem is openvswitch does not support its ports to be moved to
  a different name space. The fact that it's possible to do that is a
  bug - such operation should have been denied. Unfortunately, due to a
  missing check, it's not been denied. Such setup does not work
  reliably, though, and leads to various issues from incorrect resource
  accounting to kernel crashes.
   
  We're aware of the bug but we don't have any easy way to fix it. The
  most obvious option, disabling moving of ovs ports to different name
  spaces, would be easy to do but it would break Neutron. The other
  option, making ovs to work across net name spaces, is hard and will
  require addition of different kernel APIs and large changes in ovs
  user space daemons. This constitutes tremendous amount of work.
   
  The tracker bug on openvswitch side is:
  - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340
   
  So in the best case, we may expect openvswitch to properly support the
  feature in long term, but short term it won't work, especially while
  neutron expects other features implemented in openvswitch for it (like
  NAT, or connection tracking, or ipv6 tunnel endpoints, to name a few).
   
  We could try to handle the issue neutron side. We can fix it by
  utilizing veth pairs to get into namespaces, but it may mean worse
  performance, and will definitely require proper benchmarking to see
  whether we can live with performance drop.
   
  ===
   
  There were other suggestions on how we can enhance our way of usage of
  openvswitch. Among those, getting rid of linux bridge used for
  security groups, with 

Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful

2015-02-13 Thread Miguel Ángel Ajo
Hi, Ihar  Jiri, thank you for pointing this out.

I’m working on the following items:

1) Doing Openflow traffic filtering (stateful firewall) based on OVS+CT[1] 
patch, which may
eventually merge. Here I want to build a good amount of benchmarks to be 
able to compare
the current network iptables+LB solution to just openflow.

 Openflow filtering should be fast, as it’s quite smart at using hashes to 
match OF rules
 in the kernel megaflows (thanks Jiri  T.Graf for explaining me this)

 The only bad part is that we would have to dynamically change more rules 
based on security
group changes (now we do that via ip sets without reloading all the rules).

  To do this properly, we may want to make the OVS plugin a real OF 
controller to be able to
push OF rules to the bridge without the need of calling ovs-ofctl on the 
command line all the time.

2) Using OVS+OF to do QoS

other interesting stuff to look at:

3) Doing routing in OF too, thanks to the NAT capabilities of having OVS+CT  

4) The namespace problem, what kinds of statistics get broken by moving ports 
into namespaces now?,
the short-term fix could be using vets, but “namespaceable” OVS ports would 
be perfect, yet I understand
the change is a big feature.

If we had 1  3, may be 4 wouldn’t be a problem anymore.

[1] https://github.com/justinpettit/ovs/tree/conntrack  

Miguel Ángel Ajo


On Friday, 13 de February de 2015 at 13:14, Ihar Hrachyshka wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
  
 Hi neutroners,
  
 we** had several conversations recently with our Red Hat fellows who
 work on openvswitch (Jiri Benc and Jiri Pirko) regarding the way
 neutron utilizes their software. Those were beneficial to both sides
 to understand what we do right and wrong. I was asked to share some of
 the points from those discussions with broader community.
  
 ===
  
 One of the issues that came up during discussions is the way neutron
 connects ovs ports to namespaces. The short story is that openvswitch
 is not designed with namespaces in mind, and the fact that moving its
 ports into a different namespace works for neutron is mere
 coincidence, and is actually considered as a bug by openvswitch guys.
  
 It's not just broken in theory from software design standpoint, but
 also in practice. Specifically,
  
 1. ovsdb dump does not work properly for namespaces:
 - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340
  
 This results in wrong statistics and other data collected for these ports;
  
 2. We suspect that the following kernel crash is triggered because of
 our usage of the feature that is actually a bug:
 - - https://bugs.launchpad.net/neutron/+bug/1418097
  
 Quoting Jiri Benc,
  
 The problem is openvswitch does not support its ports to be moved to
 a different name space. The fact that it's possible to do that is a
 bug - such operation should have been denied. Unfortunately, due to a
 missing check, it's not been denied. Such setup does not work
 reliably, though, and leads to various issues from incorrect resource
 accounting to kernel crashes.
  
 We're aware of the bug but we don't have any easy way to fix it. The
 most obvious option, disabling moving of ovs ports to different name
 spaces, would be easy to do but it would break Neutron. The other
 option, making ovs to work across net name spaces, is hard and will
 require addition of different kernel APIs and large changes in ovs
 user space daemons. This constitutes tremendous amount of work.
  
 The tracker bug on openvswitch side is:
 - - https://bugzilla.redhat.com/show_bug.cgi?id=1160340
  
 So in the best case, we may expect openvswitch to properly support the
 feature in long term, but short term it won't work, especially while
 neutron expects other features implemented in openvswitch for it (like
 NAT, or connection tracking, or ipv6 tunnel endpoints, to name a few).
  
 We could try to handle the issue neutron side. We can fix it by
 utilizing veth pairs to get into namespaces, but it may mean worse
 performance, and will definitely require proper benchmarking to see
 whether we can live with performance drop.
  
 ===
  
 There were other suggestions on how we can enhance our way of usage of
 openvswitch. Among those, getting rid of linux bridge used for
 security groups, with special attention on getting rid of ebtables
 (sic!) for they are a lot slower than iptables; getting rid of veth
 pair for instance ports.
  
 ===
  
 I really encourage everyone to check the following video from
 devconf.cz (http://devconf.cz) 2015 on all that and more at:
  
 - - https://www.youtube.com/watch?v=BlLD-qh9EBQ
  
 Among other things, you will find presentation of plotnetcfg tool to
 create nice graphs of openstack state.
  
 If you're lazy enough and want to switch directly to the analysis of
 neutron problems, skip to ~18:00.
  
 I also encourage to check our the video around 30:00 on the way out of
 openvswitch for neutron (tc/eBPF). 

Re: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful

2015-02-13 Thread Isaku Yamahata
Surely eliminating linux bridge for iptables by ovs+tc is quite important
for performance.


On Fri, Feb 13, 2015 at 01:57:46PM +0100,
Ihar Hrachyshka ihrac...@redhat.com wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 On 02/13/2015 01:47 PM, Miguel Ángel Ajo wrote:
  Sorry, I forgot about
  
  5)  If we put all our OVS/OF bridge logic in just one bridge
  (instead of N: br-tun, br-int, br-ex, br-xxx), the performance
  should be yet higher, since, as far as I understood, flow rule
  lookup could be more optimized into the kernel megaflows without
  forwarding and re-starting evaluation due to patch ports. (Please
  correct me here where I’m wrong, I just have very high level view
  of this).
 
 Indeed, that was also mentioned by Jiri in our private talks. That
 said, I'm as unaware of details here as you probably are (or more).

The current ovs instantiates only kernel datapath and optimizes out
patch port in kernel. the patch port logic is handled by ovs-vswitchd
side when building flow rules for kernel datapath.
I don't know which version, so you may be using old versions...

Or are you referring to recirculation?



  2) Using OVS+OF to do QoS
  
  other interesting stuff to look at:

What exactly do you mean? marking packet or tc queuing?


thanks,
-- 
Isaku Yamahata isaku.yamah...@gmail.com

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev