Re: [Openstack-operators] Raising the degree of the scandal
George, are you willing to commit development resources to help address issues in Neutron, and improve the support for Linux Bridge, since All neutron/ovs stuff is horrible in term of performance ? Linux Bridge parity has been identified as a step towards a single network stack for OpenStack. We are in the process of enabling Linux Bridge testing at the gate to prevent bit-rot. We will need resources from the community to ensure that Linux Bridge is maintained, since Neutron's mission is to support many different networking solutions. I will also note that arp spoofing is the 3rd item on our priority list. https://etherpad.openstack.org/p/YVR-nova-network If anyone else wishes to volunteer, now is the time. Thanks -- Sean M. Collins ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Raising the degree of the scandal
On 05/17/2015 04:33 PM, Miguel Ángel Ajo wrote: Probably the solution is not selected to be backported because: * It’s an intrusive change * Introduces new dependencies * Probably it’s going to introduce a performance penalty because eatables is slow. I’m asking in reviews for this feature to be enabled/disabled via a flag. I'm understand that. All neutron/ovs stuff is horrible in term of performance. But it makes compromise between security fix and 'concerns'. I do not worry about problem, I worry about trading security for something else. If this is not fixed, why bother to talk about 'supported/unsupported' versions of openstack? Btw: If it is good enough for Libre, why it is not good enough for Kilo? ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Raising the degree of the scandal
This whole discussion is basically pointless because the ebtables work isn't even done. There is no merged ebtables fix to back-port. Additionally, some of us don’t want to run OVS Well OVS is the reference right now. If you choose to use something else, there are going to be feature gaps like this. Did you consider this before trying to remove OVS? so an OVS only solution is effectively imho - crap You mean for people that choose not to use the gate-tested reference driver? Anyone can just as easily argue that an ebtables solution is crap because it assumes a filtering bridge so it doesn't work with any direct plugging setups. However, we try to discourage attacking contributions that didn't happen to fit a narrow use-case. It's unproductive and poisonous behavior that discourages people from doing anything. you need to run OVS - seems like a stretch if you aren't using GRE/vxlan tunneling for all of your guests. Tunneling has little to do with OVS. Linux bridge also supports vxlan and GRE. You can just say why you don't like OVS (e.g. hard to debug, tooling is different, admins don't know how to use it, etc). This can help motivate the movement to restart development on the linux bridge driver. Since, flat networks are a 100% supported Who is supporting Linux bridge used as a backend for shared networks and is claiming they are secure? Whoever is doing that should be on the receiving end of your complaints. Please be clear about what you really want here. It sounds like you want ARP filtering support in the Linux bridge driver. Is that correct? On Mon, May 18, 2015 at 12:22 AM, Kris G. Lindgren klindg...@godaddy.com wrote: I always thought that ebtables was below the stack in the iptables schema - but still part of netfilter - thus should be reasonably fast (I would argue faster than a user space lookup to openvswitchd). Considering the rules being added are small in number and trivial (on this port allow src/dst mac of blah). Do you have any performance data showing that its slow? A quick google search shows nothing relevant :-). Additionally, libvirt anti-spoofing rules configured via nova using nwfilter uses ebtables to do the anti spoofing rules. No one seems to complain about the performance of that... Additionally, some of us don’t want to run OVS, so an OVS only solution is effectively imho - crap. We are actively looking at removing OVS from our environment due to a number of reasons. So saying if you run neutron and care to have *REAL* network protection - you need to run OVS - seems like a stretch if you aren't using GRE/vxlan tunneling for all of your guests. I personally agree with George, this stance of We never said that neutron would provide anti-spoofing on flat networks thus we wont backport this, because it now brings into scope ebtables, seems to be a pretty huge gap in what neutron says it does and the protection it actually provides. We supply security group rules, but stealing someone else's IP or the gateway that doesn't belong to you - yep that totally cool with us. It strikes me that neutrons goal to deliver networking is basically two fold. 1.) Provide multi-tenant networking 2.) Make sure tenants cant stomp on each other. Without number 2 (anti-spoofing) you kinda fail to provide number 1. Since, flat networks are a 100% supported and viable way of doing tenant networking I would say that this is a bug and should be backported to kilo/juno. I don’t personly buy the new dependency reason, new to neutron - maybe - but not new to people running nova/libvirt. Ebtables is used by libvirt for nwfilter, assuming an pretty standard libvirt install ebtables should be installed by default. Additionally, this would have already been a dependency in nova-compute due to anti-spoofing support there. Kris Lindgren Senior Linux Systems Engineer GoDaddy, LLC. From: Miguel Ángel Ajo majop...@redhat.com Date: Sunday, May 17, 2015 at 6:33 AM To: George Shuklin george.shuk...@gmail.com Cc: openstack-operators@lists.openstack.org openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] Raising the degree of the scandal Probably the solution is not selected to be backported because: * It’s an intrusive change * Introduces new dependencies * Probably it’s going to introduce a performance penalty because eatables is slow. I’m asking in reviews for this feature to be enabled/disabled via a flag. In the future I hope OVS with connection tracking to be merged, so then we can finally have a proper openvswitch_firewall_driver supporting stateful firewalling without reflective rules or flag matching (one is slow, the other is insecure…) Miguel Ángel Ajo On Saturday, 16 de May de 2015 at 23:28, George Shuklin wrote: On 05/15/2015 07:48 PM, Jay Pipes wrote: On 05/15/2015 12:38 PM, George Shuklin wrote: Just to let everyone know: broken
Re: [Openstack-operators] Raising the degree of the scandal
I always thought that ebtables was below the stack in the iptables schema - but still part of netfilter - thus should be reasonably fast (I would argue faster than a user space lookup to openvswitchd). Considering the rules being added are small in number and trivial (on this port allow src/dst mac of blah). Do you have any performance data showing that its slow? A quick google search shows nothing relevant :-). Additionally, libvirt anti-spoofing rules configured via nova using nwfilter uses ebtables to do the anti spoofing rules. No one seems to complain about the performance of that... Additionally, some of us don't want to run OVS, so an OVS only solution is effectively imho - crap. We are actively looking at removing OVS from our environment due to a number of reasons. So saying if you run neutron and care to have *REAL* network protection - you need to run OVS - seems like a stretch if you aren't using GRE/vxlan tunneling for all of your guests. I personally agree with George, this stance of We never said that neutron would provide anti-spoofing on flat networks thus we wont backport this, because it now brings into scope ebtables, seems to be a pretty huge gap in what neutron says it does and the protection it actually provides. We supply security group rules, but stealing someone else's IP or the gateway that doesn't belong to you - yep that totally cool with us. It strikes me that neutrons goal to deliver networking is basically two fold. 1.) Provide multi-tenant networking 2.) Make sure tenants cant stomp on each other. Without number 2 (anti-spoofing) you kinda fail to provide number 1. Since, flat networks are a 100% supported and viable way of doing tenant networking I would say that this is a bug and should be backported to kilo/juno. I don't personly buy the new dependency reason, new to neutron - maybe - but not new to people running nova/libvirt. Ebtables is used by libvirt for nwfilter, assuming an pretty standard libvirt install ebtables should be installed by default. Additionally, this would have already been a dependency in nova-compute due to anti-spoofing support there. Kris Lindgren Senior Linux Systems Engineer GoDaddy, LLC. From: Miguel Ángel Ajo majop...@redhat.commailto:majop...@redhat.com Date: Sunday, May 17, 2015 at 6:33 AM To: George Shuklin george.shuk...@gmail.commailto:george.shuk...@gmail.com Cc: openstack-operators@lists.openstack.orgmailto:openstack-operators@lists.openstack.org openstack-operators@lists.openstack.orgmailto:openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] Raising the degree of the scandal Probably the solution is not selected to be backported because: * It's an intrusive change * Introduces new dependencies * Probably it's going to introduce a performance penalty because eatables is slow. I'm asking in reviews for this feature to be enabled/disabled via a flag. In the future I hope OVS with connection tracking to be merged, so then we can finally have a proper openvswitch_firewall_driver supporting stateful firewalling without reflective rules or flag matching (one is slow, the other is insecure...) Miguel Ángel Ajo On Saturday, 16 de May de 2015 at 23:28, George Shuklin wrote: On 05/15/2015 07:48 PM, Jay Pipes wrote: On 05/15/2015 12:38 PM, George Shuklin wrote: Just to let everyone know: broken antispoofing is not an 'security issue' and the fix is not planned to be backported to Juno/kilo. https://bugs.launchpad.net/bugs/1274034 What can I say? All hail devstack! Who care about production? George, I can understand you are frustrated with this issue and feel strongly about it. However, I don't think notes like this are all that productive. Would a more productive action be to tell the operator community a bit about the vulnerability and suggest appropriate remedies to take? Ok, sorry. Short issue: If few tenants use same network (shared network) one tenant may disrupt network activities of other tenant by sending a specially crafted ARP packets on behave of the victim. Normally, Openstack prohibit usage of unauthorized addresses (this feature is called 'antispoofing' and it is essential for multi-tenant clouds). This feature were subtly broken (malicious tenant may not use other addresses but still may disrupt activities of other tenants). Finally, that bug has been fixed. But now they says 'oh, it is not that important, we will not backport it to current releases, only to Libery' because of new etables dependency. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.orgmailto:OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin
Re: [Openstack-operators] Raising the degree of the scandal
neutron plugins. (Ala linux bridge+iptables+ now ebtables). Please be clear about what you really want here. It sounds like you want ARP filtering support in the Linux bridge driver. Is that correct? On Mon, May 18, 2015 at 12:22 AM, Kris G. Lindgren klindg...@godaddy.com wrote: I always thought that ebtables was below the stack in the iptables schema - but still part of netfilter - thus should be reasonably fast (I would argue faster than a user space lookup to openvswitchd). Considering the rules being added are small in number and trivial (on this port allow src/dst mac of blah). Do you have any performance data showing that its slow? A quick google search shows nothing relevant :-). Additionally, libvirt anti-spoofing rules configured via nova using nwfilter uses ebtables to do the anti spoofing rules. No one seems to complain about the performance of that... Additionally, some of us don¹t want to run OVS, so an OVS only solution is effectively imho - crap. We are actively looking at removing OVS from our environment due to a number of reasons. So saying if you run neutron and care to have *REAL* network protection - you need to run OVS - seems like a stretch if you aren't using GRE/vxlan tunneling for all of your guests. I personally agree with George, this stance of We never said that neutron would provide anti-spoofing on flat networks thus we wont backport this, because it now brings into scope ebtables, seems to be a pretty huge gap in what neutron says it does and the protection it actually provides. We supply security group rules, but stealing someone else's IP or the gateway that doesn't belong to you - yep that totally cool with us. It strikes me that neutrons goal to deliver networking is basically two fold. 1.) Provide multi-tenant networking 2.) Make sure tenants cant stomp on each other. Without number 2 (anti-spoofing) you kinda fail to provide number 1. Since, flat networks are a 100% supported and viable way of doing tenant networking I would say that this is a bug and should be backported to kilo/juno. I don¹t personly buy the new dependency reason, new to neutron - maybe - but not new to people running nova/libvirt. Ebtables is used by libvirt for nwfilter, assuming an pretty standard libvirt install ebtables should be installed by default. Additionally, this would have already been a dependency in nova-compute due to anti-spoofing support there. Kris Lindgren Senior Linux Systems Engineer GoDaddy, LLC. From: Miguel Ángel Ajo majop...@redhat.com Date: Sunday, May 17, 2015 at 6:33 AM To: George Shuklin george.shuk...@gmail.com Cc: openstack-operators@lists.openstack.org openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] Raising the degree of the scandal Probably the solution is not selected to be backported because: * It¹s an intrusive change * Introduces new dependencies * Probably it¹s going to introduce a performance penalty because eatables is slow. I¹m asking in reviews for this feature to be enabled/disabled via a flag. In the future I hope OVS with connection tracking to be merged, so then we can finally have a proper openvswitch_firewall_driver supporting stateful firewalling without reflective rules or flag matching (one is slow, the other is insecureŠ) Miguel Ángel Ajo On Saturday, 16 de May de 2015 at 23:28, George Shuklin wrote: On 05/15/2015 07:48 PM, Jay Pipes wrote: On 05/15/2015 12:38 PM, George Shuklin wrote: Just to let everyone know: broken antispoofing is not an 'security issue' and the fix is not planned to be backported to Juno/kilo. https://bugs.launchpad.net/bugs/1274034 What can I say? All hail devstack! Who care about production? George, I can understand you are frustrated with this issue and feel strongly about it. However, I don't think notes like this are all that productive. Would a more productive action be to tell the operator community a bit about the vulnerability and suggest appropriate remedies to take? Ok, sorry. Short issue: If few tenants use same network (shared network) one tenant may disrupt network activities of other tenant by sending a specially crafted ARP packets on behave of the victim. Normally, Openstack prohibit usage of unauthorized addresses (this feature is called 'antispoofing' and it is essential for multi-tenant clouds). This feature were subtly broken (malicious tenant may not use other addresses but still may disrupt activities of other tenants). Finally, that bug has been fixed. But now they says 'oh, it is not that important, we will not backport it to current releases, only to Libery' because of new etables dependency. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Raising the degree of the scandal
Probably the solution is not selected to be backported because: * It’s an intrusive change * Introduces new dependencies * Probably it’s going to introduce a performance penalty because eatables is slow. I’m asking in reviews for this feature to be enabled/disabled via a flag. In the future I hope OVS with connection tracking to be merged, so then we can finally have a proper openvswitch_firewall_driver supporting stateful firewalling without reflective rules or flag matching (one is slow, the other is insecure…) Miguel Ángel Ajo On Saturday, 16 de May de 2015 at 23:28, George Shuklin wrote: On 05/15/2015 07:48 PM, Jay Pipes wrote: On 05/15/2015 12:38 PM, George Shuklin wrote: Just to let everyone know: broken antispoofing is not an 'security issue' and the fix is not planned to be backported to Juno/kilo. https://bugs.launchpad.net/bugs/1274034 What can I say? All hail devstack! Who care about production? George, I can understand you are frustrated with this issue and feel strongly about it. However, I don't think notes like this are all that productive. Would a more productive action be to tell the operator community a bit about the vulnerability and suggest appropriate remedies to take? Ok, sorry. Short issue: If few tenants use same network (shared network) one tenant may disrupt network activities of other tenant by sending a specially crafted ARP packets on behave of the victim. Normally, Openstack prohibit usage of unauthorized addresses (this feature is called 'antispoofing' and it is essential for multi-tenant clouds). This feature were subtly broken (malicious tenant may not use other addresses but still may disrupt activities of other tenants). Finally, that bug has been fixed. But now they says 'oh, it is not that important, we will not backport it to current releases, only to Libery' because of new etables dependency. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org (mailto:OpenStack-operators@lists.openstack.org) http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Raising the degree of the scandal
On 05/15/2015 07:48 PM, Jay Pipes wrote: On 05/15/2015 12:38 PM, George Shuklin wrote: Just to let everyone know: broken antispoofing is not an 'security issue' and the fix is not planned to be backported to Juno/kilo. https://bugs.launchpad.net/bugs/1274034 What can I say? All hail devstack! Who care about production? George, I can understand you are frustrated with this issue and feel strongly about it. However, I don't think notes like this are all that productive. Would a more productive action be to tell the operator community a bit about the vulnerability and suggest appropriate remedies to take? Ok, sorry. Short issue: If few tenants use same network (shared network) one tenant may disrupt network activities of other tenant by sending a specially crafted ARP packets on behave of the victim. Normally, Openstack prohibit usage of unauthorized addresses (this feature is called 'antispoofing' and it is essential for multi-tenant clouds). This feature were subtly broken (malicious tenant may not use other addresses but still may disrupt activities of other tenants). Finally, that bug has been fixed. But now they says 'oh, it is not that important, we will not backport it to current releases, only to Libery' because of new etables dependency. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Raising the degree of the scandal
On 2015-05-15 12:48:36 -0400 (-0400), Jay Pipes wrote: [...] Would a more productive action be to tell the operator community a bit about the vulnerability and suggest appropriate remedies to take? Perhaps by pointing them to the security note published about this last September... https://wiki.openstack.org/wiki/OSSN/OSSN-0027 -- Jeremy Stanley signature.asc Description: Digital signature ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
