Re: [openstack-dev] [neutron] blueprint ovs-firewall-driver: OVS implementation of security groups
Salvatore, The 80% distinction came from a discussion I had at summit, representing that the majority of features described by the current security groups could be implemented today with OVS without connection tracking. It’s not based on any mathematical calculation… more of a pseudo-application of Pareto’s principle. :) Correct, the OVS tcp_flags feature will be used to implement an emulated statefulness for TCP flows whereas non-TCP flows would use the source-port-range-min, source-port-range-max extended API to implement stateless flows. Performance measurements would have to come after implementations are made for the proposed blueprint. Although, benchmarks of the two existing FirewallDriver implementations can be done today. We can measure number of concurrent connections until failure, overall bandwidth as percentage of line rate, etc. Are there any other specific metrics you would like to see in the benchmark? Amir On Jun 3, 2014, at 2:51 AM, Salvatore Orlando sorla...@nicira.commailto:sorla...@nicira.com wrote: I would like to understand how did we get to this 80%/20% distinction. In other terms, it seems conntrack's RELATED features won't be supported for non-tcp traffic. What about the ESTABLISHED feature? The blueprint specs refers to tcp_flags=ack. Or will that be supported through the source port matching extension which is being promoted? More comments inline. On 3 June 2014 01:22, Amir Sadoughi amir.sadou...@rackspace.commailto:amir.sadou...@rackspace.com wrote: Hi all, In the Neutron weekly meeting today[0], we discussed the ovs-firewall-driver blueprint[1]. Moving forward, OVS features today will give us 80% of the iptables security groups behavior. Specifically, OVS lacks connection tracking so it won’t have a RELATED feature or stateful rules for non-TCP flows. (OVS connection tracking is currently under development, to be released by 2015[2]). To make the “20% difference more explicit to the operator and end user, we have proposed feature configuration to provide security group rules API validation that would validate based on connection tracking ability, for example. I am stilly generally skeptic of API changes which surface backend details on user-facing APIs. I understand why you are proposing this however, and I think it would be good to get first an assessment of the benefits brought by such a change before making a call on changing API behaviour to reflect security group implementation on the backend. Several ideas floated up during the chat today, I wanted to expand the discussion to the mailing list for further debate. Some ideas include: - marking ovs-firewall-driver as experimental in Juno - What does it mean to be marked as “experimental”? In this case experimental would be a way to say not 100% functional. You would not expect a public service provider exposing neutron APIs backed by this driver, but maybe in some private deployments where the missing features are not a concern it could be used. - performance improvements under a new OVS firewall driver untested so far (vthapar is working on this) From the last comment in your post it seems you already have proof of the performance improvement, perhaps you can add those to the Performance Impact section on the spec. - incomplete implementation will cause confusion, educational burden It's more about technical debt in my opinion, but this is not necessarily the case. - debugging OVS is new to users compared to debugging old iptables This won't be a concern as long as we have good documentation to back the implementation. As Neutron is usually sloppy with documentation - then it's a concern. - waiting for upstream OVS to implement (OpenStack K- or even L- cycle) In my humble opinion, merging the blueprint for Juno will provide us a viable, more performant security groups implementation than what we have available today. Amir [0] http://eavesdrop.openstack.org/meetings/networking/2014/networking.2014-06-02-21.01.log.html [1] https://review.openstack.org/#/c/89712/ [2] http://openvswitch.org/pipermail/dev/2014-May/040567.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] blueprint ovs-firewall-driver: OVS implementation of security groups
Paul, Beyond explicit configuration for the cloud operator, documentation and API validation for the end user, is there anything specific you would like to see as a “warning label”? Does iptables do TCP sequence number validation? Where we can, we should strive to match iptables behavior. Regarding OVS flows and security groups, we can provide a tool to explain how security group rules are mapped to the integration bridge. In the proposed solution contained in the blueprint, security group rule flows would be distinguished from other agent’s flows via cookie. Regarding packet logging, I don’t know if OVS is capable of it. If iptables in Neutron does not currently support that feature, I don’t think Neutron should explicitly support out-of-tree features. Amir On Jun 3, 2014, at 6:59 AM, CARVER, PAUL pc2...@att.commailto:pc2...@att.com wrote: Amir Sadoughi wrote: Specifically, OVS lacks connection tracking so it won’t have a RELATED feature or stateful rules for non-TCP flows. (OVS connection tracking is currently under development, to be released by 2015 It definitely needs a big obvious warning label on this. A stateless firewall hasn’t been acceptable in serious security environments for at least a decade. “Real” firewalls do things like TCP sequence number validation to ensure that someone isn’t hi-jacking an existing connection and TCP flag validation to make sure that someone isn’t “fuzzing” by sending invalid combinations of flags in order to uncover bugs in servers behind the firewall. - debugging OVS is new to users compared to debugging old iptables This one is very important in my opinion. There absolutely needs to be a section in the documentation on displaying and interpreting the rules generated by Neutron. I’m pretty sure that if you tell anyone with Linux admin experience that Neutron security groups are iptables based, they should be able to figure their way around iptables –L or iptables –S without much help. If they haven’t touched iptables in a while, five minutes reading “man iptables” should be enough for them to figure out the important options and they can readily see the relationship between what they put in a security group and what shows up in the iptables chain. I don’t think there’s anywhere near that ease of use on how to list the OvS ruleset for a VM and see how it corresponds to the Neutron security group. Finally, logging of packets (including both dropped and permitted connections) is mandatory in many environments. Does OvS have the ability to do the necessary logging? Although Neutron security groups don’t currently enable logging, the capabilities are present in the underlying iptables and can be enabled with some work. If OvS doesn’t support logging of connections then this feature definitely needs to be clearly marked as “not a firewall substitute” so that admins are clearly informed that they still need a “real” firewall for audit compliance and may only consider OvS based Neutron security groups as an additional layer of protection behind the “real” firewall. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] blueprint ovs-firewall-driver: OVS implementation of security groups
Carl, You are correct in both distinctions. Like I mentioned to Paul, beyond explicit configuration for the cloud operator, documentation and API validation for the end user, is there anything specific you would like to see as a “warning label”? Amir On Jun 3, 2014, at 9:01 AM, Carl Baldwin c...@ecbaldwin.netmailto:c...@ecbaldwin.net wrote: How does ovs handle tcp flows? Does it include stateful tracking of tcp -- as your wording below implies -- or does it do stateless inspection of returning tcp packets? It appears it is the latter. This isn't the same as providing a stateful ESTABLISHED feature. Many users may not fully understand the differences. One of the most basic use cases, which is to ping an outside Ip address from inside a nova instance would not work without connection tracking with the default security groups which don't allow ingress except related and established. This may surprise many. Carl Hi all, In the Neutron weekly meeting today[0], we discussed the ovs-firewall-driver blueprint[1]. Moving forward, OVS features today will give us 80% of the iptables security groups behavior. Specifically, OVS lacks connection tracking so it won’t have a RELATED feature or stateful rules for non-TCP flows. (OVS connection tracking is currently under development, to be released by 2015[2]). To make the “20% difference more explicit to the operator and end user, we have proposed feature configuration to provide security group rules API validation that would validate based on connection tracking ability, for example. Several ideas floated up during the chat today, I wanted to expand the discussion to the mailing list for further debate. Some ideas include: - marking ovs-firewall-driver as experimental in Juno - What does it mean to be marked as “experimental”? - performance improvements under a new OVS firewall driver untested so far (vthapar is working on this) - incomplete implementation will cause confusion, educational burden - debugging OVS is new to users compared to debugging old iptables - waiting for upstream OVS to implement (OpenStack K- or even L- cycle) In my humble opinion, merging the blueprint for Juno will provide us a viable, more performant security groups implementation than what we have available today. Amir [0] http://eavesdrop.openstack.org/meetings/networking/2014/networking.2014-06-02-21.01.log.html [1] https://review.openstack.org/#/c/89712/ [2] http://openvswitch.org/pipermail/dev/2014-May/040567.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] blueprint ovs-firewall-driver: OVS implementation of security groups
I would like to understand how did we get to this 80%/20% distinction. In other terms, it seems conntrack's RELATED features won't be supported for non-tcp traffic. What about the ESTABLISHED feature? The blueprint specs refers to tcp_flags=ack. Or will that be supported through the source port matching extension which is being promoted? More comments inline. On 3 June 2014 01:22, Amir Sadoughi amir.sadou...@rackspace.com wrote: Hi all, In the Neutron weekly meeting today[0], we discussed the ovs-firewall-driver blueprint[1]. Moving forward, OVS features today will give us 80% of the iptables security groups behavior. Specifically, OVS lacks connection tracking so it won’t have a RELATED feature or stateful rules for non-TCP flows. (OVS connection tracking is currently under development, to be released by 2015[2]). To make the “20% difference more explicit to the operator and end user, we have proposed feature configuration to provide security group rules API validation that would validate based on connection tracking ability, for example. I am stilly generally skeptic of API changes which surface backend details on user-facing APIs. I understand why you are proposing this however, and I think it would be good to get first an assessment of the benefits brought by such a change before making a call on changing API behaviour to reflect security group implementation on the backend. Several ideas floated up during the chat today, I wanted to expand the discussion to the mailing list for further debate. Some ideas include: - marking ovs-firewall-driver as experimental in Juno - What does it mean to be marked as “experimental”? In this case experimental would be a way to say not 100% functional. You would not expect a public service provider exposing neutron APIs backed by this driver, but maybe in some private deployments where the missing features are not a concern it could be used. - performance improvements under a new OVS firewall driver untested so far (vthapar is working on this) From the last comment in your post it seems you already have proof of the performance improvement, perhaps you can add those to the Performance Impact section on the spec. - incomplete implementation will cause confusion, educational burden It's more about technical debt in my opinion, but this is not necessarily the case. - debugging OVS is new to users compared to debugging old iptables This won't be a concern as long as we have good documentation to back the implementation. As Neutron is usually sloppy with documentation - then it's a concern. - waiting for upstream OVS to implement (OpenStack K- or even L- cycle) In my humble opinion, merging the blueprint for Juno will provide us a viable, more performant security groups implementation than what we have available today. Amir [0] http://eavesdrop.openstack.org/meetings/networking/2014/networking.2014-06-02-21.01.log.html [1] https://review.openstack.org/#/c/89712/ [2] http://openvswitch.org/pipermail/dev/2014-May/040567.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] blueprint ovs-firewall-driver: OVS implementation of security groups
Amir Sadoughi wrote: Specifically, OVS lacks connection tracking so it won't have a RELATED feature or stateful rules for non-TCP flows. (OVS connection tracking is currently under development, to be released by 2015 It definitely needs a big obvious warning label on this. A stateless firewall hasn't been acceptable in serious security environments for at least a decade. Real firewalls do things like TCP sequence number validation to ensure that someone isn't hi-jacking an existing connection and TCP flag validation to make sure that someone isn't fuzzing by sending invalid combinations of flags in order to uncover bugs in servers behind the firewall. - debugging OVS is new to users compared to debugging old iptables This one is very important in my opinion. There absolutely needs to be a section in the documentation on displaying and interpreting the rules generated by Neutron. I'm pretty sure that if you tell anyone with Linux admin experience that Neutron security groups are iptables based, they should be able to figure their way around iptables -L or iptables -S without much help. If they haven't touched iptables in a while, five minutes reading man iptables should be enough for them to figure out the important options and they can readily see the relationship between what they put in a security group and what shows up in the iptables chain. I don't think there's anywhere near that ease of use on how to list the OvS ruleset for a VM and see how it corresponds to the Neutron security group. Finally, logging of packets (including both dropped and permitted connections) is mandatory in many environments. Does OvS have the ability to do the necessary logging? Although Neutron security groups don't currently enable logging, the capabilities are present in the underlying iptables and can be enabled with some work. If OvS doesn't support logging of connections then this feature definitely needs to be clearly marked as not a firewall substitute so that admins are clearly informed that they still need a real firewall for audit compliance and may only consider OvS based Neutron security groups as an additional layer of protection behind the real firewall. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] blueprint ovs-firewall-driver: OVS implementation of security groups
How does ovs handle tcp flows? Does it include stateful tracking of tcp -- as your wording below implies -- or does it do stateless inspection of returning tcp packets? It appears it is the latter. This isn't the same as providing a stateful ESTABLISHED feature. Many users may not fully understand the differences. One of the most basic use cases, which is to ping an outside Ip address from inside a nova instance would not work without connection tracking with the default security groups which don't allow ingress except related and established. This may surprise many. Carl Hi all, In the Neutron weekly meeting today[0], we discussed the ovs-firewall-driver blueprint[1]. Moving forward, OVS features today will give us 80% of the iptables security groups behavior. Specifically, OVS lacks connection tracking so it won’t have a RELATED feature or stateful rules for non-TCP flows. (OVS connection tracking is currently under development, to be released by 2015[2]). To make the “20% difference more explicit to the operator and end user, we have proposed feature configuration to provide security group rules API validation that would validate based on connection tracking ability, for example. Several ideas floated up during the chat today, I wanted to expand the discussion to the mailing list for further debate. Some ideas include: - marking ovs-firewall-driver as experimental in Juno - What does it mean to be marked as “experimental”? - performance improvements under a new OVS firewall driver untested so far (vthapar is working on this) - incomplete implementation will cause confusion, educational burden - debugging OVS is new to users compared to debugging old iptables - waiting for upstream OVS to implement (OpenStack K- or even L- cycle) In my humble opinion, merging the blueprint for Juno will provide us a viable, more performant security groups implementation than what we have available today. Amir [0] http://eavesdrop.openstack.org/meetings/networking/2014/networking.2014-06-02-21.01.log.html [1] https://review.openstack.org/#/c/89712/ [2] http://openvswitch.org/pipermail/dev/2014-May/040567.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
