Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
As I found out there already is a change made by Xurong Yang that assigns conntrack zones to ports https://review.openstack.org/#/c/118274/6 If merged, it will make connection tracking easier and will allow to add functionality for closing connections after updating or deleting security group rules. I will try to add this functionality basing on the existing patch and see if it works. I believe that the change requires attention and discussion as there are some concerns regarding it. Perhaps somebody could take a look it? On Fri, Oct 24, 2014 at 9:28 PM, Carl Baldwin c...@ecbaldwin.net wrote: On Fri, Oct 24, 2014 at 6:17 AM, Salvatore Orlando sorla...@nicira.com wrote: Assigning a distinct ct zone to each port sounds more scalable. This should keep the number of zones per host Agree that zones could be a good solution to this problem. +1 to zone / port for scalability. Though it will take a bit more code and complexity to kill the right connections than it would with zone / rule. Once we identify the ports, and therefore the ct zones, then we'd still need to find the connections matching the rules which were removed. This does not sound like being too difficult, but it can result in searches over long lists - think about an instance hosting a DB or web server. Are you thinking of listing all connections and then iterating over the list with some code to match it to a security group rule being removed/updated? Or, map the security group rule to conntrack filter arguments to send to a call to conntrack -D? This could be problematic. What if security group rules were redundant and an update or removal of a rule should not really affect any existing connections? If all connections were compared instead against the set of *remaining* security group rules then this wouldn't be a problem. This sounds non-trivial to me. I'm probably thinking too hard about this. ;) The above two considerations made me suggest the idea of associating ct zones with rules, but it is probably true that this can cause us to go beyond the 2^16 limit. I agree we'd hit this limit. Carl ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
Hi, Elena Ezhova, thanks for your work to this problem! I agree with your analysis, this why I commit this bug but don't submit patch for it. I have want to use conntrack to solve this bug, but I also thought the problem you have said: The problem here is that it is sometimes impossible to tell which connection should be killed. For example there may be two instances running in different namespaces that have the same ip addresses. As a compute doesn't know anything about namespaces, it cannot distinguish between the two seemingly identical connections: $ sudo conntrack -L | grep 10.0.0.5 tcp 6 431954 ESTABLISHED src=10.0.0.3 dst=10.0.0.5 sport=60723 dport=22 src=10.0.0.5 dst=10.0.0.3 sport=22 dport=60723 [ASSURED] mark=0 use=1 tcp 6 431976 ESTABLISHED src=10.0.0.3 dst=10.0.0.5 sport=60729 dport=22 src=10.0.0.5 dst=10.0.0.3 sport=22 dport=60729 [ASSURED] mark=0 use=1 1. I think this problem is due to that in compute node, all tenants instances (if use ovs agent, it use vlan to isolate different tenant instance) are in same namespace, so it can't distinguish the connection as above use case. 2. the ip_conntrack works above L3, so it can't search for a connection by destination MAC I am not clear as ajo said: I'm not sure if removing all the conntrack rules that match the certain filter would be OK enough, as it may only lead to full reevaluation of rules for the next packet of the cleared connections (may be I'm missing some corner detail, which could be). 在 2014-10-23 18:22:46，Elena Ezhova eezh...@mirantis.com 写道： Hi! I am working on a bug ping still working once connected even after related security group rule is deleted (https://bugs.launchpad.net/neutron/+bug/1335375). The gist of the problem is the following: when we delete a security group rule the corresponding rule in iptables is also deleted, but the connection, that was allowed by that rule, is not being destroyed. The reason for such behavior is that in iptables we have the following structure of a chain that filters input packets for an interface of an istance: Chain neutron-openvswi-i830fa99f-3 (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/00.0.0.0/0 state INVALID /* Drop packets that are not associated with a state. */ 0 0 RETURN all -- * * 0.0.0.0/00.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ 0 0 RETURN udp -- * * 10.0.0.3 0.0.0.0/0 udp spt:67 dpt:68 0 0 RETURN all -- * * 0.0.0.0/00.0.0.0/0 match-set IPv43a0d3610-8b38-43f2-8 src 0 0 RETURN tcp -- * * 0.0.0.0/00.0.0.0/0 tcp dpt:22 rule that allows ssh on port 22 184 RETURN icmp -- * * 0.0.0.0/00.0.0.0/0 0 0 neutron-openvswi-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0/* Send unmatched traffic to the fallback chain. */ So, if we delete rule that allows tcp on port 22, then all connections that are already established won't be closed, because all packets would satisfy the rule: 0 0 RETURN all -- * * 0.0.0.0/00.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ I seek advice on the way how to deal with the problem. There are a couple of ideas how to do it (more or less realistic): Kill the connection using conntrack The problem here is that it is sometimes impossible to tell which connection should be killed. For example there may be two instances running in different namespaces that have the same ip addresses. As a compute doesn't know anything about namespaces, it cannot distinguish between the two seemingly identical connections: $ sudo conntrack -L | grep 10.0.0.5 tcp 6 431954 ESTABLISHED src=10.0.0.3 dst=10.0.0.5 sport=60723 dport=22 src=10.0.0.5 dst=10.0.0.3 sport=22 dport=60723 [ASSURED] mark=0 use=1 tcp 6 431976 ESTABLISHED src=10.0.0.3 dst=10.0.0.5 sport=60729 dport=22 src=10.0.0.5 dst=10.0.0.3 sport=22 dport=60729 [ASSURED] mark=0 use=1 I wonder whether there is any way to search for a connection by destination MAC? Delete iptables rule that directs packets associated with a known session to the RETURN chain It will force all packets to go through the full chain each time and this will definitely make the connection close. But this will strongly affect the performance. Probably there may be created a timeout after which this rule will be restored, but it is uncertain
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
I think the root cause of the problem here is that we are losing segregation between tenants at the conntrack level. The compute side plugs everything into the same namespace and we have no guarantees about uniqueness of any other fields kept by conntrack. Because of this loss of uniqueness, I think there may be another lurking bug here as well. One tenant establishing connections between IPs that overlap with another tenant will create the possibility that a connection the other tenant attempts will match the conntrack entry from the original connection. Then whichever closes the connection first will result in the conntrack entry being removed and the return traffic from the remaining connection being dropped. I think the correct way forward here is to isolate each tenant (or even compute interface) into its own conntrack zone.[1] This will provide isolation against that imaginary unlikely scenario I just presented. :-) More importantly, it will allow us to clear connections for a specific tenant (or compute interface) without interfering with others because conntrack can delete by zone.[2] 1. https://github.com/torvalds/linux/commit/5d0aa2ccd4699a01cfdf14886191c249d7b45a01 2. see the -w option. http://manpages.ubuntu.com/manpages/raring/man8/conntrack.8.html On Thu, Oct 23, 2014 at 3:22 AM, Elena Ezhova eezh...@mirantis.com wrote: Hi! I am working on a bug ping still working once connected even after related security group rule is deleted ( https://bugs.launchpad.net/neutron/+bug/1335375). The gist of the problem is the following: when we delete a security group rule the corresponding rule in iptables is also deleted, but the connection, that was allowed by that rule, is not being destroyed. The reason for such behavior is that in iptables we have the following structure of a chain that filters input packets for an interface of an istance: Chain neutron-openvswi-i830fa99f-3 (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0state INVALID /* Drop packets that are not associated with a state. */ 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ 0 0 RETURN udp -- * * 10.0.0.3 0.0.0.0/0udp spt:67 dpt:68 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0match-set IPv43a0d3610-8b38-43f2-8 src 0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0tcp dpt:22 rule that allows ssh on port 22 184 RETURN icmp -- * * 0.0.0.0/0 0.0.0.0/0 0 0 neutron-openvswi-sg-fallback all -- * * 0.0.0.0/00.0.0.0/0/* Send unmatched traffic to the fallback chain. */ So, if we delete rule that allows tcp on port 22, then all connections that are already established won't be closed, because all packets would satisfy the rule: 0 0 RETURN all -- * * 0.0.0.0/00.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ I seek advice on the way how to deal with the problem. There are a couple of ideas how to do it (more or less realistic): - Kill the connection using conntrack The problem here is that it is sometimes impossible to tell which connection should be killed. For example there may be two instances running in different namespaces that have the same ip addresses. As a compute doesn't know anything about namespaces, it cannot distinguish between the two seemingly identical connections: $ sudo conntrack -L | grep 10.0.0.5 tcp 6 431954 ESTABLISHED src=10.0.0.3 dst=10.0.0.5 sport=60723 dport=22 src=10.0.0.5 dst=10.0.0.3 sport=22 dport=60723 [ASSURED] mark=0 use=1 tcp 6 431976 ESTABLISHED src=10.0.0.3 dst=10.0.0.5 sport=60729 dport=22 src=10.0.0.5 dst=10.0.0.3 sport=22 dport=60729 [ASSURED] mark=0 use=1 I wonder whether there is any way to search for a connection by destination MAC? - Delete iptables rule that directs packets associated with a known session to the RETURN chain It will force all packets to go through the full chain each time and this will definitely make the connection close. But this will strongly affect the performance. Probably there may be created a timeout after which this rule will be restored, but it is uncertain how long should it be. Please share your thoughts on how it would be better to handle it. Thanks in advance, Elena ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Kevin Benton ___
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
Just like Kevin I was considering using conntrack zones to segregate connections. However, I don't know whether this would be feasible as I've never used iptables CT target in real applications. Segregation should probably happen at the security group level - or even at the rule level - rather than the tenant level. Indeed the same situation could occur even with two security groups belonging to the same tenant. Probably each rule can be associated with a different conntrack zone. So when it's matched, the corresponding conntrack entries will be added to the appropriate zone. And therefore when the rules are removed the corresponding connections to kill can be filtered by zone as explained by Kevin. This approach will add a good number of rules to the RAW table however, so its impact on control/data plane scalability should be assessed, as it might turn as bad as the solution where connections where explicitly dropped with an ad-hoc iptables rule. Salvatore On 24 October 2014 09:32, Kevin Benton blak...@gmail.com wrote: I think the root cause of the problem here is that we are losing segregation between tenants at the conntrack level. The compute side plugs everything into the same namespace and we have no guarantees about uniqueness of any other fields kept by conntrack. Because of this loss of uniqueness, I think there may be another lurking bug here as well. One tenant establishing connections between IPs that overlap with another tenant will create the possibility that a connection the other tenant attempts will match the conntrack entry from the original connection. Then whichever closes the connection first will result in the conntrack entry being removed and the return traffic from the remaining connection being dropped. I think the correct way forward here is to isolate each tenant (or even compute interface) into its own conntrack zone.[1] This will provide isolation against that imaginary unlikely scenario I just presented. :-) More importantly, it will allow us to clear connections for a specific tenant (or compute interface) without interfering with others because conntrack can delete by zone.[2] 1. https://github.com/torvalds/linux/commit/5d0aa2ccd4699a01cfdf14886191c249d7b45a01 2. see the -w option. http://manpages.ubuntu.com/manpages/raring/man8/conntrack.8.html On Thu, Oct 23, 2014 at 3:22 AM, Elena Ezhova eezh...@mirantis.com wrote: Hi! I am working on a bug ping still working once connected even after related security group rule is deleted ( https://bugs.launchpad.net/neutron/+bug/1335375). The gist of the problem is the following: when we delete a security group rule the corresponding rule in iptables is also deleted, but the connection, that was allowed by that rule, is not being destroyed. The reason for such behavior is that in iptables we have the following structure of a chain that filters input packets for an interface of an istance: Chain neutron-openvswi-i830fa99f-3 (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0state INVALID /* Drop packets that are not associated with a state. */ 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ 0 0 RETURN udp -- * * 10.0.0.3 0.0.0.0/0udp spt:67 dpt:68 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0match-set IPv43a0d3610-8b38-43f2-8 src 0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0tcp dpt:22 rule that allows ssh on port 22 184 RETURN icmp -- * * 0.0.0.0/0 0.0.0.0/0 0 0 neutron-openvswi-sg-fallback all -- * * 0.0.0.0/00.0.0.0/0/* Send unmatched traffic to the fallback chain. */ So, if we delete rule that allows tcp on port 22, then all connections that are already established won't be closed, because all packets would satisfy the rule: 0 0 RETURN all -- * * 0.0.0.0/00.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ I seek advice on the way how to deal with the problem. There are a couple of ideas how to do it (more or less realistic): - Kill the connection using conntrack The problem here is that it is sometimes impossible to tell which connection should be killed. For example there may be two instances running in different namespaces that have the same ip addresses. As a compute doesn't know anything about namespaces, it cannot distinguish between the two seemingly identical connections: $ sudo conntrack -L | grep 10.0.0.5 tcp 6 431954 ESTABLISHED src=10.0.0.3 dst=10.0.0.5 sport=60723 dport=22 src=10.0.0.5
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
Nice!, It sounds like a good mechanism to handle this. Defining a good mechanism here is crucial, we must be aware of the 2^16 zones limit [1], and that ipset rules will coalesce connections to lots of different IPs over the same rule. May be a good option is to tag connections per rule (we limit ourself to 2^16 rules) AND per ip address / port / protocol, etc. (having an average of 5 in / 5 out rules per port, that's a 6553 ports limit per machine). Or, if we need this to scale to more ports, tag connections per port, and target them by rule AND ip address / port / protocol. [1] https://github.com/torvalds/linux/commit/5d0aa2ccd4699a01cfdf14886191c249d7b45a01#diff-4d53dd1f3ad5275bc2e79f2c12af6e68R8 Best, Miguel Ángel. - Original Message - Just like Kevin I was considering using conntrack zones to segregate connections. However, I don't know whether this would be feasible as I've never used iptables CT target in real applications. Segregation should probably happen at the security group level - or even at the rule level - rather than the tenant level. Indeed the same situation could occur even with two security groups belonging to the same tenant. Probably each rule can be associated with a different conntrack zone. So when it's matched, the corresponding conntrack entries will be added to the appropriate zone. And therefore when the rules are removed the corresponding connections to kill can be filtered by zone as explained by Kevin. This approach will add a good number of rules to the RAW table however, so its impact on control/data plane scalability should be assessed, as it might turn as bad as the solution where connections where explicitly dropped with an ad-hoc iptables rule. Salvatore On 24 October 2014 09:32, Kevin Benton blak...@gmail.com wrote: I think the root cause of the problem here is that we are losing segregation between tenants at the conntrack level. The compute side plugs everything into the same namespace and we have no guarantees about uniqueness of any other fields kept by conntrack. Because of this loss of uniqueness, I think there may be another lurking bug here as well. One tenant establishing connections between IPs that overlap with another tenant will create the possibility that a connection the other tenant attempts will match the conntrack entry from the original connection. Then whichever closes the connection first will result in the conntrack entry being removed and the return traffic from the remaining connection being dropped. I think the correct way forward here is to isolate each tenant (or even compute interface) into its own conntrack zone.[1] This will provide isolation against that imaginary unlikely scenario I just presented. :-) More importantly, it will allow us to clear connections for a specific tenant (or compute interface) without interfering with others because conntrack can delete by zone.[2] 1. https://github.com/torvalds/linux/commit/5d0aa2ccd4699a01cfdf14886191c249d7b45a01 2. see the -w option. http://manpages.ubuntu.com/manpages/raring/man8/conntrack.8.html On Thu, Oct 23, 2014 at 3:22 AM, Elena Ezhova eezh...@mirantis.com wrote: Hi! I am working on a bug ping still working once connected even after related security group rule is deleted ( https://bugs.launchpad.net/neutron/+bug/1335375 ). The gist of the problem is the following: when we delete a security group rule the corresponding rule in iptables is also deleted, but the connection, that was allowed by that rule, is not being destroyed. The reason for such behavior is that in iptables we have the following structure of a chain that filters input packets for an interface of an istance: Chain neutron-openvswi-i830fa99f-3 (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that are not associated with a state. */ 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ 0 0 RETURN udp -- * * 10.0.0.3 0.0.0.0/0 udp spt:67 dpt:68 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set IPv43a0d3610-8b38-43f2-8 src 0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 rule that allows ssh on port 22 1 84 RETURN icmp -- * * 0.0.0.0/0 0.0.0.0/0 0 0 neutron-openvswi-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ So, if we delete rule that allows tcp on port 22, then all connections that are already established won't be closed, because all packets would satisfy the rule: 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
While a zone per rule would be nice because we can easily delete connection state by only referencing a zone, that's probably overkill. We only need enough to disambiguate between overlapping IPs so we can then delete connection state by matching standard L3/4 headers again, right? I think a conntrack zone per port would be the easiest from an accounting perspective. We already setup an iptables chain per port so the grouping is already there (/me sweeps the complexity of choosing zone numbers under the rug). On Fri, Oct 24, 2014 at 2:25 AM, Salvatore Orlando sorla...@nicira.com wrote: Just like Kevin I was considering using conntrack zones to segregate connections. However, I don't know whether this would be feasible as I've never used iptables CT target in real applications. Segregation should probably happen at the security group level - or even at the rule level - rather than the tenant level. Indeed the same situation could occur even with two security groups belonging to the same tenant. Probably each rule can be associated with a different conntrack zone. So when it's matched, the corresponding conntrack entries will be added to the appropriate zone. And therefore when the rules are removed the corresponding connections to kill can be filtered by zone as explained by Kevin. This approach will add a good number of rules to the RAW table however, so its impact on control/data plane scalability should be assessed, as it might turn as bad as the solution where connections where explicitly dropped with an ad-hoc iptables rule. Salvatore On 24 October 2014 09:32, Kevin Benton blak...@gmail.com wrote: I think the root cause of the problem here is that we are losing segregation between tenants at the conntrack level. The compute side plugs everything into the same namespace and we have no guarantees about uniqueness of any other fields kept by conntrack. Because of this loss of uniqueness, I think there may be another lurking bug here as well. One tenant establishing connections between IPs that overlap with another tenant will create the possibility that a connection the other tenant attempts will match the conntrack entry from the original connection. Then whichever closes the connection first will result in the conntrack entry being removed and the return traffic from the remaining connection being dropped. I think the correct way forward here is to isolate each tenant (or even compute interface) into its own conntrack zone.[1] This will provide isolation against that imaginary unlikely scenario I just presented. :-) More importantly, it will allow us to clear connections for a specific tenant (or compute interface) without interfering with others because conntrack can delete by zone.[2] 1. https://github.com/torvalds/linux/commit/5d0aa2ccd4699a01cfdf14886191c249d7b45a01 2. see the -w option. http://manpages.ubuntu.com/manpages/raring/man8/conntrack.8.html On Thu, Oct 23, 2014 at 3:22 AM, Elena Ezhova eezh...@mirantis.com wrote: Hi! I am working on a bug ping still working once connected even after related security group rule is deleted ( https://bugs.launchpad.net/neutron/+bug/1335375). The gist of the problem is the following: when we delete a security group rule the corresponding rule in iptables is also deleted, but the connection, that was allowed by that rule, is not being destroyed. The reason for such behavior is that in iptables we have the following structure of a chain that filters input packets for an interface of an istance: Chain neutron-openvswi-i830fa99f-3 (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0state INVALID /* Drop packets that are not associated with a state. */ 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ 0 0 RETURN udp -- * * 10.0.0.3 0.0.0.0/0udp spt:67 dpt:68 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0match-set IPv43a0d3610-8b38-43f2-8 src 0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0tcp dpt:22 rule that allows ssh on port 22 184 RETURN icmp -- * * 0.0.0.0/0 0.0.0.0/0 0 0 neutron-openvswi-sg-fallback all -- * * 0.0.0.0/00.0.0.0/0/* Send unmatched traffic to the fallback chain. */ So, if we delete rule that allows tcp on port 22, then all connections that are already established won't be closed, because all packets would satisfy the rule: 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ I seek advice on the way how to deal with the
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
Kevin, I agree, with you, 1 zone per port should be reasonable. The 2^16 rule limit will force us into keeping state (to tie ports to zones across reboots), may be this state can be just recovered by reading the iptables rules at boot, and reconstructing the current openvswitch-agent local port/zone association. Best, Miguel Ángel. - Original Message - While a zone per rule would be nice because we can easily delete connection state by only referencing a zone, that's probably overkill. We only need enough to disambiguate between overlapping IPs so we can then delete connection state by matching standard L3/4 headers again, right? I think a conntrack zone per port would be the easiest from an accounting perspective. We already setup an iptables chain per port so the grouping is already there (/me sweeps the complexity of choosing zone numbers under the rug). On Fri, Oct 24, 2014 at 2:25 AM, Salvatore Orlando sorla...@nicira.com wrote: Just like Kevin I was considering using conntrack zones to segregate connections. However, I don't know whether this would be feasible as I've never used iptables CT target in real applications. Segregation should probably happen at the security group level - or even at the rule level - rather than the tenant level. Indeed the same situation could occur even with two security groups belonging to the same tenant. Probably each rule can be associated with a different conntrack zone. So when it's matched, the corresponding conntrack entries will be added to the appropriate zone. And therefore when the rules are removed the corresponding connections to kill can be filtered by zone as explained by Kevin. This approach will add a good number of rules to the RAW table however, so its impact on control/data plane scalability should be assessed, as it might turn as bad as the solution where connections where explicitly dropped with an ad-hoc iptables rule. Salvatore On 24 October 2014 09:32, Kevin Benton blak...@gmail.com wrote: I think the root cause of the problem here is that we are losing segregation between tenants at the conntrack level. The compute side plugs everything into the same namespace and we have no guarantees about uniqueness of any other fields kept by conntrack. Because of this loss of uniqueness, I think there may be another lurking bug here as well. One tenant establishing connections between IPs that overlap with another tenant will create the possibility that a connection the other tenant attempts will match the conntrack entry from the original connection. Then whichever closes the connection first will result in the conntrack entry being removed and the return traffic from the remaining connection being dropped. I think the correct way forward here is to isolate each tenant (or even compute interface) into its own conntrack zone.[1] This will provide isolation against that imaginary unlikely scenario I just presented. :-) More importantly, it will allow us to clear connections for a specific tenant (or compute interface) without interfering with others because conntrack can delete by zone.[2] 1. https://github.com/torvalds/linux/commit/5d0aa2ccd4699a01cfdf14886191c249d7b45a01 2. see the -w option. http://manpages.ubuntu.com/manpages/raring/man8/conntrack.8.html On Thu, Oct 23, 2014 at 3:22 AM, Elena Ezhova eezh...@mirantis.com wrote: Hi! I am working on a bug ping still working once connected even after related security group rule is deleted ( https://bugs.launchpad.net/neutron/+bug/1335375 ). The gist of the problem is the following: when we delete a security group rule the corresponding rule in iptables is also deleted, but the connection, that was allowed by that rule, is not being destroyed. The reason for such behavior is that in iptables we have the following structure of a chain that filters input packets for an interface of an istance: Chain neutron-openvswi-i830fa99f-3 (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that are not associated with a state. */ 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ 0 0 RETURN udp -- * * 10.0.0.3 0.0.0.0/0 udp spt:67 dpt:68 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set IPv43a0d3610-8b38-43f2-8 src 0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 rule that allows ssh on port 22 1 84 RETURN icmp -- * * 0.0.0.0/0 0.0.0.0/0 0 0 neutron-openvswi-sg-fallback all -- * * 0.0.0.0/0
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
sorry: when I said boot, I mean openvswitch agent restart. - Original Message - Kevin, I agree, with you, 1 zone per port should be reasonable. The 2^16 rule limit will force us into keeping state (to tie ports to zones across reboots), may be this state can be just recovered by reading the iptables rules at boot, and reconstructing the current openvswitch-agent local port/zone association. Best, Miguel Ángel. - Original Message - While a zone per rule would be nice because we can easily delete connection state by only referencing a zone, that's probably overkill. We only need enough to disambiguate between overlapping IPs so we can then delete connection state by matching standard L3/4 headers again, right? I think a conntrack zone per port would be the easiest from an accounting perspective. We already setup an iptables chain per port so the grouping is already there (/me sweeps the complexity of choosing zone numbers under the rug). On Fri, Oct 24, 2014 at 2:25 AM, Salvatore Orlando sorla...@nicira.com wrote: Just like Kevin I was considering using conntrack zones to segregate connections. However, I don't know whether this would be feasible as I've never used iptables CT target in real applications. Segregation should probably happen at the security group level - or even at the rule level - rather than the tenant level. Indeed the same situation could occur even with two security groups belonging to the same tenant. Probably each rule can be associated with a different conntrack zone. So when it's matched, the corresponding conntrack entries will be added to the appropriate zone. And therefore when the rules are removed the corresponding connections to kill can be filtered by zone as explained by Kevin. This approach will add a good number of rules to the RAW table however, so its impact on control/data plane scalability should be assessed, as it might turn as bad as the solution where connections where explicitly dropped with an ad-hoc iptables rule. Salvatore On 24 October 2014 09:32, Kevin Benton blak...@gmail.com wrote: I think the root cause of the problem here is that we are losing segregation between tenants at the conntrack level. The compute side plugs everything into the same namespace and we have no guarantees about uniqueness of any other fields kept by conntrack. Because of this loss of uniqueness, I think there may be another lurking bug here as well. One tenant establishing connections between IPs that overlap with another tenant will create the possibility that a connection the other tenant attempts will match the conntrack entry from the original connection. Then whichever closes the connection first will result in the conntrack entry being removed and the return traffic from the remaining connection being dropped. I think the correct way forward here is to isolate each tenant (or even compute interface) into its own conntrack zone.[1] This will provide isolation against that imaginary unlikely scenario I just presented. :-) More importantly, it will allow us to clear connections for a specific tenant (or compute interface) without interfering with others because conntrack can delete by zone.[2] 1. https://github.com/torvalds/linux/commit/5d0aa2ccd4699a01cfdf14886191c249d7b45a01 2. see the -w option. http://manpages.ubuntu.com/manpages/raring/man8/conntrack.8.html On Thu, Oct 23, 2014 at 3:22 AM, Elena Ezhova eezh...@mirantis.com wrote: Hi! I am working on a bug ping still working once connected even after related security group rule is deleted ( https://bugs.launchpad.net/neutron/+bug/1335375 ). The gist of the problem is the following: when we delete a security group rule the corresponding rule in iptables is also deleted, but the connection, that was allowed by that rule, is not being destroyed. The reason for such behavior is that in iptables we have the following structure of a chain that filters input packets for an interface of an istance: Chain neutron-openvswi-i830fa99f-3 (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that are not associated with a state. */ 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ 0 0 RETURN udp -- * * 10.0.0.3 0.0.0.0/0 udp spt:67 dpt:68 0 0 RETURN all -- * *
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
Assigning a distinct ct zone to each port sounds more scalable. This should keep the number of zones per host contained. What should the workflow when rules are updated or deleted be? 1) From the rule security group find ports on host where it's applied 2) kill all matching connections for those ports I'm just thinking aloud here, but can #1 be achieved without doing a call from the agent to the server? Otherwise one could pack the set of affect ports in messages for security group updates. Once we identify the ports, and therefore the ct zones, then we'd still need to find the connections matching the rules which were removed. This does not sound like being too difficult, but it can result in searches over long lists - think about an instance hosting a DB or web server. The above two considerations made me suggest the idea of associating ct zones with rules, but it is probably true that this can cause us to go beyond the 2^16 limit. Salvatore On 24 October 2014 11:16, Miguel Angel Ajo Pelayo mangel...@redhat.com wrote: sorry: when I said boot, I mean openvswitch agent restart. - Original Message - Kevin, I agree, with you, 1 zone per port should be reasonable. The 2^16 rule limit will force us into keeping state (to tie ports to zones across reboots), may be this state can be just recovered by reading the iptables rules at boot, and reconstructing the current openvswitch-agent local port/zone association. Best, Miguel Ángel. - Original Message - While a zone per rule would be nice because we can easily delete connection state by only referencing a zone, that's probably overkill. We only need enough to disambiguate between overlapping IPs so we can then delete connection state by matching standard L3/4 headers again, right? I think a conntrack zone per port would be the easiest from an accounting perspective. We already setup an iptables chain per port so the grouping is already there (/me sweeps the complexity of choosing zone numbers under the rug). On Fri, Oct 24, 2014 at 2:25 AM, Salvatore Orlando sorla...@nicira.com wrote: Just like Kevin I was considering using conntrack zones to segregate connections. However, I don't know whether this would be feasible as I've never used iptables CT target in real applications. Segregation should probably happen at the security group level - or even at the rule level - rather than the tenant level. Indeed the same situation could occur even with two security groups belonging to the same tenant. Probably each rule can be associated with a different conntrack zone. So when it's matched, the corresponding conntrack entries will be added to the appropriate zone. And therefore when the rules are removed the corresponding connections to kill can be filtered by zone as explained by Kevin. This approach will add a good number of rules to the RAW table however, so its impact on control/data plane scalability should be assessed, as it might turn as bad as the solution where connections where explicitly dropped with an ad-hoc iptables rule. Salvatore On 24 October 2014 09:32, Kevin Benton blak...@gmail.com wrote: I think the root cause of the problem here is that we are losing segregation between tenants at the conntrack level. The compute side plugs everything into the same namespace and we have no guarantees about uniqueness of any other fields kept by conntrack. Because of this loss of uniqueness, I think there may be another lurking bug here as well. One tenant establishing connections between IPs that overlap with another tenant will create the possibility that a connection the other tenant attempts will match the conntrack entry from the original connection. Then whichever closes the connection first will result in the conntrack entry being removed and the return traffic from the remaining connection being dropped. I think the correct way forward here is to isolate each tenant (or even compute interface) into its own conntrack zone.[1] This will provide isolation against that imaginary unlikely scenario I just presented. :-) More importantly, it will allow us to clear connections for a specific tenant (or compute interface) without interfering with others because conntrack can delete by zone.[2] 1. https://github.com/torvalds/linux/commit/5d0aa2ccd4699a01cfdf14886191c249d7b45a01 2. see the -w option. http://manpages.ubuntu.com/manpages/raring/man8/conntrack.8.html On Thu, Oct 23, 2014 at 3:22 AM, Elena Ezhova eezh...@mirantis.com wrote: Hi! I am working on a
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
Hi Elena, On Thu, Oct 23, 2014 at 4:22 AM, Elena Ezhova eezh...@mirantis.com wrote: Kill the connection using conntrack The problem here is that it is sometimes impossible to tell which connection should be killed. For example there may be two instances running in different namespaces that have the same ip addresses. As a compute doesn't know anything about namespaces, it cannot distinguish between the two seemingly identical connections: If it really were different namespaces -- Linux network namespaces -- then it wouldn't matter. The conntrack context is isolated between namespaces. A similar fix for floating IPs takes advantage of this fact [1]. However, I don't think you meant Linux network namespaces when you said different namespaces above. I think you probably just meant to use different namespaces as a more generic term for different network traffic domains that are isolated from each other. So, moving on... In the case of security groups on compute nodes isolation is accomplished using VLANs, linux bridges, etc. So, yes, the conntrack context is shared. I can't think of a way to use conntrack to solve this problem. I'm not advocating the following approach at the moment, just brainstorming here... I wonder if we could just ignore the potential IP address overlap and kill the connections anyway. What would happen? Some connections that were supposed to survive will actually recover if nf_conntrack_tcp_loose is enabled. But, some may not recover. So, maybe this isn't a brilliant idea. I wonder whether there is any way to search for a connection by destination MAC? Well, for one thing, the mac is not guaranteed unique either. For another, I don't even know if conntrack is aware of or cares about any L2 details about the connection since it is kind of an L3/L4 sort of thing. Delete iptables rule that directs packets associated with a known session to the RETURN chain It will force all packets to go through the full chain each time and this will definitely make the connection close. But this will strongly affect the performance. Probably there may be created a timeout after which this rule will be restored, but it is uncertain how long should it be. I don't think this is just about performance, this will cause other connections to stop working. If I'm not mistaken, removing these rules makes it behave more like a stateless firewall. In stateless firewalls, rules must be spelled out independently for ingress and egress packets whereas in a stateful firewall, one typically thinks of the whole connection as ingress or egress (after adding those RELATED/ESTABLISHED rules to allow the other direction through based on conntrack state). Here's another brainstorming idea... I wonder if something could be done with the NOTRACK target in the raw table. Could such a rule be added to squash active connections by hiding them from conntrack? Rules could be added with a timeout so that they don't stay in the table forever. The rule would have to be written to match security group rules that were just deleted. This could be tricky. But, once added, they could be removed on the next iptables update after a given period of time or on agent restart. Carl [1] https://review.openstack.org/#/c/103475/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
Miguel Ángel, On Thu, Oct 23, 2014 at 5:56 AM, Miguel Angel Ajo Pelayo mangel...@redhat.com wrote: Temporarily removing this entry doesn't seem like a good solution to me as we can't really know how long do we need to remove this rule to induce the connection to close at both ends (it will only close if any new activity happens, and timeout is exhausted afterwards). I think you're right here. I think any activity will keep the connection alive in conntrack. So, we are at the mercy of the timeouts at both ends. Assuming an attacker has control over at least the external endpoint, it could be kept open indefinitely generating activity. Carl ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
On 10/23/2014 08:57 PM, Brian Haley wrote: On 10/23/14 6:22 AM, Elena Ezhova wrote: Hi! I am working on a bug ping still working once connected even after related security group rule is deleted (https://bugs.launchpad.net/neutron/+bug/1335375). The gist of the problem is the following: when we delete a security group rule the corresponding rule in iptables is also deleted, but the connection, that was allowed by that rule, is not being destroyed. The reason for such behavior is that in iptables we have the following structure of a chain that filters input packets for an interface of an istance: snip Like Miguel said, there's no easy way to identify this on the compute node since neither the MAC nor the interface are going to be in the conntrack command output. And you don't want to drop the wrong tenant's connections. Just wondering, if you remove the conntrack entries using the IP/port from the router namespace does it drop the connection? Or will it just start working again on the next packet? Doesn't work for VM to VM packets, but those packets are probably less interesting. It's just my first guess. Presumably this issue affects other conntrack users, no? What does upstream conntrack have to say about the matter? I tend to avoid such things where I can, but what do real firewalls do with such matters? If one removes a rule which allowed a given connection through, do they actually go ahead and nuke existing connections? rick jones ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
On Fri, Oct 24, 2014 at 6:17 AM, Salvatore Orlando sorla...@nicira.com wrote: Assigning a distinct ct zone to each port sounds more scalable. This should keep the number of zones per host Agree that zones could be a good solution to this problem. +1 to zone / port for scalability. Though it will take a bit more code and complexity to kill the right connections than it would with zone / rule. Once we identify the ports, and therefore the ct zones, then we'd still need to find the connections matching the rules which were removed. This does not sound like being too difficult, but it can result in searches over long lists - think about an instance hosting a DB or web server. Are you thinking of listing all connections and then iterating over the list with some code to match it to a security group rule being removed/updated? Or, map the security group rule to conntrack filter arguments to send to a call to conntrack -D? This could be problematic. What if security group rules were redundant and an update or removal of a rule should not really affect any existing connections? If all connections were compared instead against the set of *remaining* security group rules then this wouldn't be a problem. This sounds non-trivial to me. I'm probably thinking too hard about this. ;) The above two considerations made me suggest the idea of associating ct zones with rules, but it is probably true that this can cause us to go beyond the 2^16 limit. I agree we'd hit this limit. Carl ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
Hi! This is an interesting topic, I don't know if there's any way to target connection tracker rules by MAC, but that'd be the ideal solution. I also understand the RETURN for RELATED,ESTABLISHED is there for performance reasons, and removing it would lead to longer table evaluation, and degraded packet throughput. Temporarily removing this entry doesn't seem like a good solution to me as we can't really know how long do we need to remove this rule to induce the connection to close at both ends (it will only close if any new activity happens, and timeout is exhausted afterwards). Also, I'm not sure if removing all the conntrack rules that match the certain filter would be OK enough, as it may only lead to full reevaluation of rules for the next packet of the cleared connections (may be I'm missing some corner detail, which could be). Best regards, Miguel Ángel. - Original Message - Hi! I am working on a bug ping still working once connected even after related security group rule is deleted ( https://bugs.launchpad.net/neutron/+bug/1335375 ). The gist of the problem is the following: when we delete a security group rule the corresponding rule in iptables is also deleted, but the connection, that was allowed by that rule, is not being destroyed. The reason for such behavior is that in iptables we have the following structure of a chain that filters input packets for an interface of an istance: Chain neutron-openvswi-i830fa99f-3 (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that are not associated with a state. */ 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ 0 0 RETURN udp -- * * 10.0.0.3 0.0.0.0/0 udp spt:67 dpt:68 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set IPv43a0d3610-8b38-43f2-8 src 0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 rule that allows ssh on port 22 1 84 RETURN icmp -- * * 0.0.0.0/0 0.0.0.0/0 0 0 neutron-openvswi-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ So, if we delete rule that allows tcp on port 22, then all connections that are already established won't be closed, because all packets would satisfy the rule: 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ I seek advice on the way how to deal with the problem. There are a couple of ideas how to do it (more or less realistic): * Kill the connection using conntrack The problem here is that it is sometimes impossible to tell which connection should be killed. For example there may be two instances running in different namespaces that have the same ip addresses. As a compute doesn't know anything about namespaces, it cannot distinguish between the two seemingly identical connections: $ sudo conntrack -L | grep 10.0.0.5 tcp 6 431954 ESTABLISHED src=10.0.0.3 dst=10.0.0.5 sport=60723 dport=22 src=10.0.0.5 dst=10.0.0.3 sport=22 dport=60723 [ASSURED] mark=0 use=1 tcp 6 431976 ESTABLISHED src=10.0.0.3 dst=10.0.0.5 sport=60729 dport=22 src=10.0.0.5 dst=10.0.0.3 sport=22 dport=60729 [ASSURED] mark=0 use=1 I wonder whether there is any way to search for a connection by destination MAC? * Delete iptables rule that directs packets associated with a known session to the RETURN chain It will force all packets to go through the full chain each time and this will definitely make the connection close. But this will strongly affect the performance. Probably there may be created a timeout after which this rule will be restored, but it is uncertain how long should it be. Please share your thoughts on how it would be better to handle it. Thanks in advance, Elena ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
If you exec conntrack inside the namespace with ip netns exec does it still show both connections? Vish On Oct 23, 2014, at 3:22 AM, Elena Ezhova eezh...@mirantis.com wrote: Hi! I am working on a bug ping still working once connected even after related security group rule is deleted (https://bugs.launchpad.net/neutron/+bug/1335375). The gist of the problem is the following: when we delete a security group rule the corresponding rule in iptables is also deleted, but the connection, that was allowed by that rule, is not being destroyed. The reason for such behavior is that in iptables we have the following structure of a chain that filters input packets for an interface of an istance: Chain neutron-openvswi-i830fa99f-3 (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/00.0.0.0/0 state INVALID /* Drop packets that are not associated with a state. */ 0 0 RETURN all -- * * 0.0.0.0/00.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ 0 0 RETURN udp -- * * 10.0.0.3 0.0.0.0/0 udp spt:67 dpt:68 0 0 RETURN all -- * * 0.0.0.0/00.0.0.0/0 match-set IPv43a0d3610-8b38-43f2-8 src 0 0 RETURN tcp -- * * 0.0.0.0/00.0.0.0/0 tcp dpt:22 rule that allows ssh on port 22 184 RETURN icmp -- * * 0.0.0.0/00.0.0.0/0 0 0 neutron-openvswi-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0/* Send unmatched traffic to the fallback chain. */ So, if we delete rule that allows tcp on port 22, then all connections that are already established won't be closed, because all packets would satisfy the rule: 0 0 RETURN all -- * * 0.0.0.0/00.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ I seek advice on the way how to deal with the problem. There are a couple of ideas how to do it (more or less realistic): Kill the connection using conntrack The problem here is that it is sometimes impossible to tell which connection should be killed. For example there may be two instances running in different namespaces that have the same ip addresses. As a compute doesn't know anything about namespaces, it cannot distinguish between the two seemingly identical connections: $ sudo conntrack -L | grep 10.0.0.5 tcp 6 431954 ESTABLISHED src=10.0.0.3 dst=10.0.0.5 sport=60723 dport=22 src=10.0.0.5 dst=10.0.0.3 sport=22 dport=60723 [ASSURED] mark=0 use=1 tcp 6 431976 ESTABLISHED src=10.0.0.3 dst=10.0.0.5 sport=60729 dport=22 src=10.0.0.5 dst=10.0.0.3 sport=22 dport=60729 [ASSURED] mark=0 use=1 I wonder whether there is any way to search for a connection by destination MAC? Delete iptables rule that directs packets associated with a known session to the RETURN chain It will force all packets to go through the full chain each time and this will definitely make the connection close. But this will strongly affect the performance. Probably there may be created a timeout after which this rule will be restored, but it is uncertain how long should it be. Please share your thoughts on how it would be better to handle it. Thanks in advance, Elena ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Killing connection after security group rule deletion
On 10/23/14 6:22 AM, Elena Ezhova wrote: Hi! I am working on a bug ping still working once connected even after related security group rule is deleted (https://bugs.launchpad.net/neutron/+bug/1335375). The gist of the problem is the following: when we delete a security group rule the corresponding rule in iptables is also deleted, but the connection, that was allowed by that rule, is not being destroyed. The reason for such behavior is that in iptables we have the following structure of a chain that filters input packets for an interface of an istance: snip Like Miguel said, there's no easy way to identify this on the compute node since neither the MAC nor the interface are going to be in the conntrack command output. And you don't want to drop the wrong tenant's connections. Just wondering, if you remove the conntrack entries using the IP/port from the router namespace does it drop the connection? Or will it just start working again on the next packet? Doesn't work for VM to VM packets, but those packets are probably less interesting. It's just my first guess. -Brian ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
