Re: [openstack-dev] [Nova][Neutron] Neutron + Nova + OVS security group fix
Hi Nachi and the teams, (2014/03/26 9:57), Salvatore Orlando wrote: I hope we can sort this out on the mailing list IRC, without having to schedule emergency meetings. Salvatore On 25 March 2014 22:58, Nachi Ueno na...@ntti3.com mailto:na...@ntti3.com wrote: Hi Nova, Neturon Team I would like to discuss issue of Neutron + Nova + OVS security group fix. We have a discussion in IRC today, but the issue is complicated so we will have a conf call tomorrow 17:00 UST (10AM PDT). #openstack-neutron (I'll put conf call information in IRC) thanks, but I'd prefer you discuss the matter on IRC. I won't be available at that time and having IRC logs on eavesdrop will allow me to catch up without having to ask people or waiting for minutes on the mailing list. I can't join the meeting too. It is midnight. -- Please let me know if this time won't work with you. Bug Report https://bugs.launchpad.net/neutron/+bug/1297469 Background of this issue: ML2 + OVSDriver + IptablesBasedFirewall combination is a default plugin setting in the Neutron. In this case, we need a special handing in VIF. Because OpenVSwitch don't support iptables, we are using linuxbride + openvswitch bridge. We are calling this as hybrid driver. The hybrid solution in Neutron has been around for such a long time that I would hardly call it a special handling. To summarize, the VIF is plugged into a linux bridge, which has another leg plugged in the OVS integration bridge. On the other discussion, we generalized the Nova side VIF plugging to the Libvirt GenericVIFDriver. The idea is let neturon tell the VIF plugging configration details to the GenericDriver, and GerericDriver takes care of it. The downside of the generic driver is that so far it's assuming local configuration values are sufficient to correctly determine VIF plugging. The generic VIF driver will use the hybrid driver if get_firewall_required is true. And this will happen if the firewall driver is anything different from the NoOp driver. This was uncovered by a devstack commit (1143f7e). When I previously discussed with the people involved this issue, I was under the impression that the devstack patch introduced the problem. Apparently the Generic VIF driver is not taking at the moments hints from neutron regarding the driver to use, and therefore, from what I gather, makes a decision based on nova conf flags only. So a quick fix would be to tell the Generic VIF driver to always use hybrid plugging when neutron is enabled (which can be gathered by nova conf flags). This will fix the issue for ML2, but will either break or insert an unnecessary extra hop for other plugins. When the generic VIF driver is introduced, OVS VIF driver and the hybrid VIF driver are considered same as e as both are pluggged into OVS and the hybrid driver is implemeted as a variation of OVS driver, but the thing is not so simple than the first thought. The hybrid driver solution lives such a long time and IMO the hybrid VIF driver should be considered as a different one from OVS VIF driver. I start to think VIF_TYPE_OVS_HYBRID is a good way as Savaltore mentioned below. Another point to be discussed is whether passing vif secuirty attributes work from now on. Even when neutron security group is enabled, do we need to do some port security mechanism (anti-spoofing, ) on nova-compute side (such as libvirt nwfilter) or not? Unfortunatly, HybridDriver is removed before GenericDriver is ready for security group. The drivers were marked for deprecation in Havana, and if we thought the GenericDriver was not good for neutron security groups we had enough time to scream. This makes ML2 + OVSDriver + IptablesBasedFirewall combination unfunctional. We were working on realfix, but we can't make it until Icehouse release due to design discussions [1]. # Even if neturon side patch isn't merged yet. So we are proposing a workaround fix to Nova side. In this fix, we are adding special version of the GenericVIFDriver which can work with the combination. There is two point on this new Driver. (1) It prevent set conf.filtername. Because we should use NoopFirewallDriver, we need conf.filtername should be None when we use it. (2) use plug_ovs_hybrid and unplug_ovs_hybrid by enforcing get_require_firewall as True. IIUC, the original intention of get_firewall_required() is to control whether nwfilter is enabled or not, not to control hybird plugging. As a plan, get_firewall_required() is changed to look binding:attribute (binding:capablity:port_filter or binding:vif_security:iptable_required if I use the concept discussed so far). What we need is a way to determine hybrid plugging is required or not. Changing the meaning of get_firewall_required is not a good idea to me.
Re: [openstack-dev] [Nova][Neutron] Neutron + Nova + OVS security group fix
The thread branched, and it's getting long. I'm trying to summarize the discussion for other people to quickly catch up. - The bug being targeted is https://bugs.launchpad.net/neutron/+bug/1297469 It has also been reported as https://bugs.launchpad.net/neutron/+bug/1252620 and as https://bugs.launchpad.net/nova/+bug/1248859 The fix for bug 1112912 had a fix for it. - The problem is the generic VIF driver does not perform hybrid plugging which is required by Neutron when running with ML2 plugin and OVS mech driver - The proposed patches (#21946 and #44596) are however very unlikely to merge in icehouse - An alternative approach has been proposed ( https://review.openstack.org/#/c/82904/); this will 'specialize' the GenericVIF driver for use with neutron. It is meant to be a temporary workaround pending permanent solution. It's not adding conf variables, but has probably docImpact. If that works for nova core, that works for me as well - An idea regarding leveraging VIF_TYPE and fixing the issue has been launched. This will constitute a fix which might be improved in the future, and is still small and targeted. However we still need to look at the issue Nachi's pointing out regarding the fact that a libvirt network filter name should not be added in guest config. Salvatore On 26 March 2014 05:57, Akihiro Motoki mot...@da.jp.nec.com wrote: Hi Nachi and the teams, (2014/03/26 9:57), Salvatore Orlando wrote: I hope we can sort this out on the mailing list IRC, without having to schedule emergency meetings. Salvatore On 25 March 2014 22:58, Nachi Ueno na...@ntti3.com mailto: na...@ntti3.com wrote: Hi Nova, Neturon Team I would like to discuss issue of Neutron + Nova + OVS security group fix. We have a discussion in IRC today, but the issue is complicated so we will have a conf call tomorrow 17:00 UST (10AM PDT). #openstack-neutron (I'll put conf call information in IRC) thanks, but I'd prefer you discuss the matter on IRC. I won't be available at that time and having IRC logs on eavesdrop will allow me to catch up without having to ask people or waiting for minutes on the mailing list. I can't join the meeting too. It is midnight. -- Please let me know if this time won't work with you. Bug Report https://bugs.launchpad.net/neutron/+bug/1297469 Background of this issue: ML2 + OVSDriver + IptablesBasedFirewall combination is a default plugin setting in the Neutron. In this case, we need a special handing in VIF. Because OpenVSwitch don't support iptables, we are using linuxbride + openvswitch bridge. We are calling this as hybrid driver. The hybrid solution in Neutron has been around for such a long time that I would hardly call it a special handling. To summarize, the VIF is plugged into a linux bridge, which has another leg plugged in the OVS integration bridge. On the other discussion, we generalized the Nova side VIF plugging to the Libvirt GenericVIFDriver. The idea is let neturon tell the VIF plugging configration details to the GenericDriver, and GerericDriver takes care of it. The downside of the generic driver is that so far it's assuming local configuration values are sufficient to correctly determine VIF plugging. The generic VIF driver will use the hybrid driver if get_firewall_required is true. And this will happen if the firewall driver is anything different from the NoOp driver. This was uncovered by a devstack commit (1143f7e). When I previously discussed with the people involved this issue, I was under the impression that the devstack patch introduced the problem. Apparently the Generic VIF driver is not taking at the moments hints from neutron regarding the driver to use, and therefore, from what I gather, makes a decision based on nova conf flags only. So a quick fix would be to tell the Generic VIF driver to always use hybrid plugging when neutron is enabled (which can be gathered by nova conf flags). This will fix the issue for ML2, but will either break or insert an unnecessary extra hop for other plugins. When the generic VIF driver is introduced, OVS VIF driver and the hybrid VIF driver are considered same as e as both are pluggged into OVS and the hybrid driver is implemeted as a variation of OVS driver, but the thing is not so simple than the first thought. The hybrid driver solution lives such a long time and IMO the hybrid VIF driver should be considered as a different one from OVS VIF driver. I start to think VIF_TYPE_OVS_HYBRID is a good way as Savaltore mentioned below. Another point to be discussed is whether passing vif secuirty attributes work from now on. Even when neutron security group is enabled, do we need to do some port security mechanism (anti-spoofing, ) on nova-compute side (such as libvirt nwfilter) or not? Unfortunatly, HybridDriver is
Re: [openstack-dev] [Nova][Neutron] Neutron + Nova + OVS security group fix
Aplogies for double-posting. I've commented again regarding the alternatives to the solution currently on review. Please find more info on the bug report [1] I've attached patches for a hackish but (imo) cleaner solution, and patches for a compact fix along the lines of what I and Akihiro were saying. Salvatore [1] https://bugs.launchpad.net/neutron/+bug/1297469 On 26 March 2014 09:02, Salvatore Orlando sorla...@nicira.com wrote: The thread branched, and it's getting long. I'm trying to summarize the discussion for other people to quickly catch up. - The bug being targeted is https://bugs.launchpad.net/neutron/+bug/1297469 It has also been reported as https://bugs.launchpad.net/neutron/+bug/1252620 and as https://bugs.launchpad.net/nova/+bug/1248859 The fix for bug 1112912 had a fix for it. - The problem is the generic VIF driver does not perform hybrid plugging which is required by Neutron when running with ML2 plugin and OVS mech driver - The proposed patches (#21946 and #44596) are however very unlikely to merge in icehouse - An alternative approach has been proposed ( https://review.openstack.org/#/c/82904/); this will 'specialize' the GenericVIF driver for use with neutron. It is meant to be a temporary workaround pending permanent solution. It's not adding conf variables, but has probably docImpact. If that works for nova core, that works for me as well - An idea regarding leveraging VIF_TYPE and fixing the issue has been launched. This will constitute a fix which might be improved in the future, and is still small and targeted. However we still need to look at the issue Nachi's pointing out regarding the fact that a libvirt network filter name should not be added in guest config. Salvatore On 26 March 2014 05:57, Akihiro Motoki mot...@da.jp.nec.com wrote: Hi Nachi and the teams, (2014/03/26 9:57), Salvatore Orlando wrote: I hope we can sort this out on the mailing list IRC, without having to schedule emergency meetings. Salvatore On 25 March 2014 22:58, Nachi Ueno na...@ntti3.com mailto: na...@ntti3.com wrote: Hi Nova, Neturon Team I would like to discuss issue of Neutron + Nova + OVS security group fix. We have a discussion in IRC today, but the issue is complicated so we will have a conf call tomorrow 17:00 UST (10AM PDT). #openstack-neutron (I'll put conf call information in IRC) thanks, but I'd prefer you discuss the matter on IRC. I won't be available at that time and having IRC logs on eavesdrop will allow me to catch up without having to ask people or waiting for minutes on the mailing list. I can't join the meeting too. It is midnight. -- Please let me know if this time won't work with you. Bug Report https://bugs.launchpad.net/neutron/+bug/1297469 Background of this issue: ML2 + OVSDriver + IptablesBasedFirewall combination is a default plugin setting in the Neutron. In this case, we need a special handing in VIF. Because OpenVSwitch don't support iptables, we are using linuxbride + openvswitch bridge. We are calling this as hybrid driver. The hybrid solution in Neutron has been around for such a long time that I would hardly call it a special handling. To summarize, the VIF is plugged into a linux bridge, which has another leg plugged in the OVS integration bridge. On the other discussion, we generalized the Nova side VIF plugging to the Libvirt GenericVIFDriver. The idea is let neturon tell the VIF plugging configration details to the GenericDriver, and GerericDriver takes care of it. The downside of the generic driver is that so far it's assuming local configuration values are sufficient to correctly determine VIF plugging. The generic VIF driver will use the hybrid driver if get_firewall_required is true. And this will happen if the firewall driver is anything different from the NoOp driver. This was uncovered by a devstack commit (1143f7e). When I previously discussed with the people involved this issue, I was under the impression that the devstack patch introduced the problem. Apparently the Generic VIF driver is not taking at the moments hints from neutron regarding the driver to use, and therefore, from what I gather, makes a decision based on nova conf flags only. So a quick fix would be to tell the Generic VIF driver to always use hybrid plugging when neutron is enabled (which can be gathered by nova conf flags). This will fix the issue for ML2, but will either break or insert an unnecessary extra hop for other plugins. When the generic VIF driver is introduced, OVS VIF driver and the hybrid VIF driver are considered same as e as both are pluggged into OVS and the hybrid driver is implemeted as a variation of OVS driver, but the thing is not so simple than the first thought. The hybrid driver solution lives such a long time and IMO the
Re: [openstack-dev] [Nova][Neutron] Neutron + Nova + OVS security group fix
I hope we can sort this out on the mailing list IRC, without having to schedule emergency meetings. Salvatore On 25 March 2014 22:58, Nachi Ueno na...@ntti3.com wrote: Hi Nova, Neturon Team I would like to discuss issue of Neutron + Nova + OVS security group fix. We have a discussion in IRC today, but the issue is complicated so we will have a conf call tomorrow 17:00 UST (10AM PDT). #openstack-neutron (I'll put conf call information in IRC) thanks, but I'd prefer you discuss the matter on IRC. I won't be available at that time and having IRC logs on eavesdrop will allow me to catch up without having to ask people or waiting for minutes on the mailing list. -- Please let me know if this time won't work with you. Bug Report https://bugs.launchpad.net/neutron/+bug/1297469 Background of this issue: ML2 + OVSDriver + IptablesBasedFirewall combination is a default plugin setting in the Neutron. In this case, we need a special handing in VIF. Because OpenVSwitch don't support iptables, we are using linuxbride + openvswitch bridge. We are calling this as hybrid driver. The hybrid solution in Neutron has been around for such a long time that I would hardly call it a special handling. To summarize, the VIF is plugged into a linux bridge, which has another leg plugged in the OVS integration bridge. On the other discussion, we generalized the Nova side VIF plugging to the Libvirt GenericVIFDriver. The idea is let neturon tell the VIF plugging configration details to the GenericDriver, and GerericDriver takes care of it. The downside of the generic driver is that so far it's assuming local configuration values are sufficient to correctly determine VIF plugging. The generic VIF driver will use the hybrid driver if get_firewall_required is true. And this will happen if the firewall driver is anything different from the NoOp driver. This was uncovered by a devstack commit (1143f7e). When I previously discussed with the people involved this issue, I was under the impression that the devstack patch introduced the problem. Apparently the Generic VIF driver is not taking at the moments hints from neutron regarding the driver to use, and therefore, from what I gather, makes a decision based on nova conf flags only. So a quick fix would be to tell the Generic VIF driver to always use hybrid plugging when neutron is enabled (which can be gathered by nova conf flags). This will fix the issue for ML2, but will either break or insert an unnecessary extra hop for other plugins. Unfortunatly, HybridDriver is removed before GenericDriver is ready for security group. The drivers were marked for deprecation in Havana, and if we thought the GenericDriver was not good for neutron security groups we had enough time to scream. This makes ML2 + OVSDriver + IptablesBasedFirewall combination unfunctional. We were working on realfix, but we can't make it until Icehouse release due to design discussions [1]. # Even if neturon side patch isn't merged yet. So we are proposing a workaround fix to Nova side. In this fix, we are adding special version of the GenericVIFDriver which can work with the combination. There is two point on this new Driver. (1) It prevent set conf.filtername. Because we should use NoopFirewallDriver, we need conf.filtername should be None when we use it. (2) use plug_ovs_hybrid and unplug_ovs_hybrid by enforcing get_require_firewall as True. Here is patchs with UT. Workaournd fix: Nova https://review.openstack.org/#/c/82904/ Devstack patch for ML2 (Tested with 82904) https://review.openstack.org/#/c/82937/ Are there other plugins which need the hybrid driver for sec groups to work? I think so. And also - the patch does not seem to work according to Jenkins. The failures look genuine to me. We have tested the patch 82904 with following test, and this works. - Launch VM - Assign floating ip - make sure ping to the floating ip is failing from GW - modify security group rule to allow ping from anywhere - make sure ping is working You can actually run your devstack patch with your patch under review in the check queue. Check what Aaron did here: https://review.openstack.org/#/c/78694/11 I wonder if instead of putting this bit of tape, we could just leverage the VIF_TYPE attribute of the port binding extension. Most plugins use VIF_TYPE_OVS already. It's a pity the ml2 plugin with the OVS mech driver did not specify VIF_TYPE_OVS_HYBRID. But, in my opinion if we fix the relevant constants in the plugins which require hybrid plugging, and we slightly change the generic VIF driver logic to make a decision according to the VIF_TYPE binding attribute we should fine, as we'll end up with two small, contained patches, which, IMHO, are not even much ugly. But again, I'm far from being a subject matter expert when it comes to nova/neutron integration and the ML2 plugin. [1] Real fix: (defered to Juno) Improve vif attributes related with firewalling
Re: [openstack-dev] [Nova][Neutron] Neutron + Nova + OVS security group fix
Hi Salvatore 2014-03-25 17:57 GMT-07:00 Salvatore Orlando sorla...@nicira.com: I hope we can sort this out on the mailing list IRC, without having to schedule emergency meetings. Russel requested to have a conf call on this, so let him decide it. Salvatore On 25 March 2014 22:58, Nachi Ueno na...@ntti3.com wrote: Hi Nova, Neturon Team I would like to discuss issue of Neutron + Nova + OVS security group fix. We have a discussion in IRC today, but the issue is complicated so we will have a conf call tomorrow 17:00 UST (10AM PDT). #openstack-neutron (I'll put conf call information in IRC) thanks, but I'd prefer you discuss the matter on IRC. I won't be available at that time and having IRC logs on eavesdrop will allow me to catch up without having to ask people or waiting for minutes on the mailing list. -- Please let me know if this time won't work with you. Bug Report https://bugs.launchpad.net/neutron/+bug/1297469 Background of this issue: ML2 + OVSDriver + IptablesBasedFirewall combination is a default plugin setting in the Neutron. In this case, we need a special handing in VIF. Because OpenVSwitch don't support iptables, we are using linuxbride + openvswitch bridge. We are calling this as hybrid driver. The hybrid solution in Neutron has been around for such a long time that I would hardly call it a special handling. To summarize, the VIF is plugged into a linux bridge, which has another leg plugged in the OVS integration bridge. On the other discussion, we generalized the Nova side VIF plugging to the Libvirt GenericVIFDriver. The idea is let neturon tell the VIF plugging configration details to the GenericDriver, and GerericDriver takes care of it. The downside of the generic driver is that so far it's assuming local configuration values are sufficient to correctly determine VIF plugging. The generic VIF driver will use the hybrid driver if get_firewall_required is true. And this will happen if the firewall driver is anything different from the NoOp driver. This was uncovered by a devstack commit (1143f7e). When I previously discussed with the people involved this issue, I was under the impression that the devstack patch introduced the problem. Apparently the Generic VIF driver is not taking at the moments hints from neutron regarding the driver to use, and therefore, from what I gather, makes a decision based on nova conf flags only. So a quick fix would be to tell the Generic VIF driver to always use hybrid plugging when neutron is enabled (which can be gathered by nova conf flags). This will fix the issue for ML2, but will either break or insert an unnecessary extra hop for other plugins. get_firewall_required = True won't fix this issue. We need to make sure we won't set config.filtername in this case. Unfortunatly, HybridDriver is removed before GenericDriver is ready for security group. The drivers were marked for deprecation in Havana, and if we thought the GenericDriver was not good for neutron security groups we had enough time to scream. The reason we missed this issue is we lack the negative test for security groups.. so we couldn't realize this. This makes ML2 + OVSDriver + IptablesBasedFirewall combination unfunctional. We were working on realfix, but we can't make it until Icehouse release due to design discussions [1]. # Even if neturon side patch isn't merged yet. So we are proposing a workaround fix to Nova side. In this fix, we are adding special version of the GenericVIFDriver which can work with the combination. There is two point on this new Driver. (1) It prevent set conf.filtername. Because we should use NoopFirewallDriver, we need conf.filtername should be None when we use it. (2) use plug_ovs_hybrid and unplug_ovs_hybrid by enforcing get_require_firewall as True. Here is patchs with UT. Workaournd fix: Nova https://review.openstack.org/#/c/82904/ Devstack patch for ML2 (Tested with 82904) https://review.openstack.org/#/c/82937/ Are there other plugins which need the hybrid driver for sec groups to work? I think so. And also - the patch does not seem to work according to Jenkins. The failures look genuine to me. I agere with you, however we should start with minimal fix for this. We can work for another plugin in another patch. This patch will fail in Jenkins because it needs 82904. We have tested the patch 82904 with following test, and this works. - Launch VM - Assign floating ip - make sure ping to the floating ip is failing from GW - modify security group rule to allow ping from anywhere - make sure ping is working You can actually run your devstack patch with your patch under review in the check queue. Check what Aaron did here: https://review.openstack.org/#/c/78694/11 Nice hack. let me try it I wonder if instead of putting this bit of tape, we could just leverage the VIF_TYPE attribute of the port binding extension. Most plugins use
