See inline. ____________________________________________ Kris Lindgren Senior Linux Systems Engineer GoDaddy, LLC.
On 5/18/15, 1:40 AM, "Kevin Benton" <[email protected]> wrote: >This whole discussion is basically pointless because the ebtables work >isn't even done. There is no merged ebtables fix to back-port. Fair. > >>Additionally, some of us don¹t want to run OVS > >Well OVS is the reference right now. If you choose to use something >else, there are going to be feature gaps like this. Did you consider >this before trying to remove OVS? > >>so an OVS only solution is effectively imho - crap > >You mean for people that choose not to use the gate-tested reference >driver? > >Anyone can just as easily argue that an ebtables solution is crap >because it assumes a filtering bridge so it doesn't work with any >direct plugging setups. However, we try to discourage attacking >contributions that didn't happen to fit a narrow use-case. It's >unproductive and poisonous behavior that discourages people from doing >anything. You mean how security groups was/is currently and has always been implemented in neutron + ovs. Creating linux bridge per vm so you can do filtering via iptables over the bridge? To be fair, the OVS solution does solve the problem for some people - so you are right I shouldn't have called it crap. I guess it's when that solution is called out as an acceptable work around that the more complete/robust solution wont be back ported because it add's new dependancies, I guess that to me is what is crap. BTW, to be clear that since we package our own packages - we will pull this in and being running it as soon as it lands. But that doesn't do anyone else any good. > >>you need to run OVS - seems like a stretch if you aren't using GRE/vxlan >>tunneling for all of your guests. > >Tunneling has little to do with OVS. Linux bridge also supports vxlan >and GRE. You can just say why you don't like OVS (e.g. hard to debug, >tooling is different, admins don't know how to use it, etc). This can >help motivate the movement to restart development on the linux bridge >driver. The references configuration all say or at minimum imply that if you are doing gre/overlay networks you need to be using OVS. I don¹t really want to get into an ovs bashing/discussion here, but in general since neutron creates a linux bridge to apply security group rules to for vms. We effectively make an architecture that uses both OVS and linux bridge for every vm. If I also have an architecture that happens to use vlans and nothing else - what does ovs gain me? Complexity - yep, additonal overhead, yep. Since a packet gets ran through kernel space for ovs + possibly a trip to user space opensvswitchd + linux bridging code + bridge mac learning code + iptables + tap code. Now tell me in that chain whats the one thing that doesn¹t need to be there? I hope we both said Openvswitch. Last I heard the megaflows that allowed for the huge speed ups no longer work when you start doing specific L4 matches (IE ip/port/porto filtering). > >>Since, flat networks are a 100% supported > >Who is supporting Linux bridge used as a backend for shared networks >and is claiming they are secure? Whoever is doing that should be on >the receiving end of your complaints. We are using neutron + ovs + shared networks (on real vlans). But neutron + ml2 + linux bridge mechanism driver is totally a supported thing ;-). We also aren't using any of the router stuff in neutron either. Because neutron is suppose to apply "anti-spoofing rules" to prevent vm's from doing bad things to other vm's. So I think the complaints are appropriately justified, since the original commit that added security groups to quantum had a method named: "_arp_spoofing_rule" [1] that the original intent was to have neutron preventing arp spoofing. It just never worked for the 3+ years, because as it turns out you can't filter arp via iptables - since iptables is layer3-7 and arp is layer2. It was also further hidden during the addition of allowed address-pairs [2], which effectively reverse implemented the previous logic [3] [1] - https://github.com/openstack/neutron/commit/f14af5dc755706c7297a96fa504acdf e15ac1957#diff-65b266f9e013df37c4934f0b1007897cR168 [2] - https://github.com/openstack/neutron/commit/b67b20832a5bfccd1bbf8d1e63ebcd7 061856881 [3] - https://github.com/openstack/neutron/commit/0efce6195fa7be80e110bd841dc9b35 37a94c376#diff-abf220de4c2165d9e5bfd6dde12b3f4fR205 So, to be clear. I want the bug fix - dare I say security/vunerability fix - when it lands to be back-ported to the current supported stable release. This is to close a 3+ year old security issue in that neutron tries to provide anti-spoofing rules - but fails to implement them in a way that actual work. Yet at a minimum alludes to the fact that it does stuff like nova to prevent vm's from doing arp spoofing/poisoning. Additionally, I want done in a way that is as generic as possible, so it works with the most amount of supported neutron plugins. (Ala linux bridge+iptables+ now ebtables). > >Please be clear about what you really want here. It sounds like you >want ARP filtering support in the Linux bridge driver. Is that >correct? > >On Mon, May 18, 2015 at 12:22 AM, Kris G. Lindgren ><[email protected]> wrote: >> I always thought that ebtables was below the stack in the iptables >>schema - >> but still part of netfilter - thus should be reasonably fast (I would >>argue >> faster than a user space lookup to openvswitchd). Considering the rules >> being added are small in number and trivial (on this port allow src/dst >>mac >> of blah). Do you have any performance data showing that its slow? A >>quick >> google search shows nothing relevant :-). Additionally, libvirt >> anti-spoofing rules configured via nova using nwfilter uses ebtables to >>do >> the anti spoofing rules. No one seems to complain about the >>performance of >> that... >> >> Additionally, some of us don¹t want to run OVS, so an OVS only solution >>is >> effectively imho - crap. We are actively looking at removing OVS from >>our >> environment due to a number of reasons. So saying if you run neutron >>and >> care to have *REAL* network protection - you need to run OVS - seems >>like a >> stretch if you aren't using GRE/vxlan tunneling for all of your guests. >> >> I personally agree with George, this stance of "We never said that >>neutron >> would provide anti-spoofing on flat networks thus we wont backport this, >> because it now brings into scope ebtables", seems to be a pretty huge >>gap in >> what neutron says it does and the protection it actually provides. We >> supply security group rules, but stealing someone else's IP or the >>gateway >> that doesn't belong to you - yep that totally cool with us. It strikes >>me >> that neutrons goal to deliver networking is basically two fold. 1.) >>Provide >> multi-tenant networking 2.) Make sure tenants cant stomp on each other. >> Without number 2 (anti-spoofing) you kinda fail to provide number 1. >>Since, >> flat networks are a 100% supported and viable way of doing tenant >>networking >> I would say that this is a bug and should be backported to kilo/juno. >> >> I don¹t personly buy the new dependency reason, new to neutron - maybe >>- but >> not new to people running nova/libvirt. Ebtables is used by libvirt for >> nwfilter, assuming an pretty standard libvirt install ebtables should be >> installed by default. Additionally, this would have already been a >> dependency in nova-compute due to anti-spoofing support there. >> ____________________________________________ >> >> Kris Lindgren >> Senior Linux Systems Engineer >> GoDaddy, LLC. >> >> >> From: Miguel Ángel Ajo <[email protected]> >> Date: Sunday, May 17, 2015 at 6:33 AM >> To: George Shuklin <[email protected]> >> Cc: "[email protected]" >> <[email protected]> >> Subject: Re: [Openstack-operators] Raising the degree of the scandal >> >> Probably the solution is not selected to be backported because: >> >> * It¹s an intrusive change >> * Introduces new dependencies >> * Probably it¹s going to introduce a performance penalty because >>eatables >> is slow. >> >> I¹m asking in reviews for this feature to be enabled/disabled via a >>flag. >> >> In the future I hope OVS with connection tracking to be merged, so then >>we >> can >> finally have a proper openvswitch_firewall_driver supporting stateful >> firewalling >> without reflective rules or flag matching (one is slow, the other is >> insecureŠ) >> >> Miguel Ángel Ajo >> >> On Saturday, 16 de May de 2015 at 23:28, George Shuklin wrote: >> >> On 05/15/2015 07:48 PM, Jay Pipes wrote: >> >> On 05/15/2015 12:38 PM, George Shuklin wrote: >> >> Just to let everyone know: broken antispoofing is not an 'security >> issue' and the fix is not planned to be backported to Juno/kilo. >> >> https://bugs.launchpad.net/bugs/1274034 >> >> What can I say? All hail devstack! Who care about production? >> >> >> George, I can understand you are frustrated with this issue and feel >> strongly about it. However, I don't think notes like this are all that >> productive. >> >> Would a more productive action be to tell the operator community a bit >> about the vulnerability and suggest appropriate remedies to take? >> >> Ok, sorry. >> >> Short issue: If few tenants use same network (shared network) one tenant >> may disrupt network activities of other tenant by sending a specially >> crafted ARP packets on behave of the victim. Normally, Openstack >> prohibit usage of unauthorized addresses (this feature is called >> 'antispoofing' and it is essential for multi-tenant clouds). This >> feature were subtly broken (malicious tenant may not use other addresses >> but still may disrupt activities of other tenants). >> >> Finally, that bug has been fixed. But now they says 'oh, it is not that >> important, we will not backport it to current releases, only to >> "Libery"' because of new etables dependency. >> >> >> _______________________________________________ >> OpenStack-operators mailing list >> [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> >> >> >> _______________________________________________ >> OpenStack-operators mailing list >> [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> > > > >-- >Kevin Benton _______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
