Re: [openstack-dev] [OSSN 0020] Disassociating floating IPs does not terminate NAT connections with Neutron L3 agent
Awesome! Hope it reaches Juno! :-) This is important... Best, Thiago On 16 September 2014 13:17, Carl Baldwin wrote: > Hi, > > There is current work in review to use conntrack to terminate these > connections [1][2] much like you suggested. I hope to get this in to > RC1 but it needs another iteration. > > For Kilo, I'd like to explore stateless forwarding for floating ips. > Since conntrack is the root of the security issue in the first place, > the idea here is to eliminate it from the floating ip data path > altogether [3]. The patch I have up is really just a placeholder with > some notes on how it might be accomplished. My hope is that this > stateless NAT for floating ips could ease some of the pressure that > I've observed on conntrack and increase performance. It needs some > more investigation. > > Carl > > [1] https://bugs.launchpad.net/neutron/+bug/1334926 > [2] https://review.openstack.org/#/c/103475 > [3] https://review.openstack.org/#/c/121689/ > > On Mon, Sep 15, 2014 at 11:46 PM, Martinx - ジェームズ > wrote: > > Hey stackers, > > > > Let me ask something about this... Why not use Linux Conntrack Table at > each > > Tenant Namespace (L3 Router) to detect which connections were > > made/established over a Floating IP ? > > > > Like this, on the Neutron L3 Router: > > > > -- > > apt-get install conntrack > > > > ip netns exec qrouter-09b72faa-a5ef-4a52-80b5-1dcbea23b1b6 conntrack -L | > > grep ESTABLISHED > > > > tcp 6 431998 ESTABLISHED src=192.168.3.5 dst=193.16.15.250 > sport=36476 > > dport=8333 src=193.16.15.250 dst=187.1.93.67 sport=8333 dport=36476 > > [ASSURED] mark=0 use=1 > > -- > > > > Floating IP: 187.1.93.67 > > Instance IP: 192.168.3.5 > > > > http://conntrack-tools.netfilter.org/manual.html#conntrack > > > > > > > > Or, as a workaround, right after removing the Floating IP, Neutron might > > insert a temporary firewall rule (for about 5~10 minutes?), to drop the > > connections of that previous "Floating IP + Instance IP couple"... It > looks > > really ugly but, at least, it will make sure that nothing will pass right > > after removing a Floating IP... Effectively terminating (dropping) the > NAT > > connections after disassociating a Floating IP... ;-) > > > > > > > > Also, I think that NFTables can bring some light here... I truly believe > > that if OpenStack moves to a "NFTables_Driver", it will be much easier > to: > > manage firewall rules, logging, counters, IDS/IPS, atomic replacements of > > rules, even NAT66... All under a single implementation... Maybe with some > > kind of "real-time connection monitoring"... I mean, with NFTables, it > > becomes easier to implement a firewall ruleset with a Intrusion > Prevention > > System (IPS), take a look: > > > > https://home.regit.org/2014/02/suricata-and-nftables/ > > > > So, if NFTables can make Suricata's life easier, why not give Suricata's > > power to Netutron L3 Router? Starting with a new NFTables_Driver... > =) > > > > I'm not an expert on NFTables but, from what I'm seeing, it perfectly > fits > > in OpenStack, in fact, NFTables will make OpenStack better. > > > > https://home.regit.org/2014/01/why-you-will-love-nftables/ > > > > Best! > > Thiago > > > > On 15 September 2014 20:49, Nathan Kinder wrote: > >> > >> -BEGIN PGP SIGNED MESSAGE- > >> Hash: SHA1 > >> > >> Disassociating floating IPs does not terminate NAT connections with > >> Neutron L3 agent > >> - --- > >> > >> ### Summary ### > >> Every virtual instance is automatically assigned a private IP address. > >> You may optionally assign public IP addresses to instances. OpenStack > >> uses the term "floating IP" to refer to an IP address (typically > >> public) that can be dynamically added to a running virtual instance. > >> The Neutron L3 agent uses Network Address Translation (NAT) to assign > >> floating IPs to virtual instances. Floating IPs can be dynamically > >> released from a running virtual instance but any active connections are > >> not terminated with this release as expected when using the Neutron L3 > >> agent. > >> > >> ### Affected Services / Software ### > >> Neutron, Icehouse, Havana, Grizzly, Folsom > >> > >> ### Discussion ### > >> When creating a virtual instance, a floating IP address is not > >> allocated by default. After a virtual instance is created, a user can > >> explicitly associate a floating IP address to that instance. Users can > >> create connections to the virtual instance using this floating IP > >> address. Also, this floating IP address can be disassociated from any > >> running instance without shutting that instance down. > >> > >> If a user initiates a connection using the floating IP address, this > >> connection remains alive and accessible even after the floating IP > >> address is released from that instance. This potentially violates > >> restrictive policies which are only being applied to new connections. > >> These policies are ignored for pre-existing connections and the
Re: [openstack-dev] [OSSN 0020] Disassociating floating IPs does not terminate NAT connections with Neutron L3 agent
Now there is already a bug:https://bugs.launchpad.net/neutron/+bug/1334926 for this problem, meanwhile the security group also has same problem, I have report a bug: https://bugs.launchpad.net/neutron/+bug/1335375 在 2014-09-16 01:46:11，"Martinx - ジェームズ" 写道： Hey stackers, Let me ask something about this... Why not use Linux Conntrack Table at each Tenant Namespace (L3 Router) to detect which connections were made/established over a Floating IP ? Like this, on the Neutron L3 Router: -- apt-get install conntrack ip netns exec qrouter-09b72faa-a5ef-4a52-80b5-1dcbea23b1b6 conntrack -L | grep ESTABLISHED tcp 6 431998 ESTABLISHED src=192.168.3.5 dst=193.16.15.250 sport=36476 dport=8333 src=193.16.15.250 dst=187.1.93.67 sport=8333 dport=36476 [ASSURED] mark=0 use=1 -- Floating IP: 187.1.93.67 Instance IP: 192.168.3.5 http://conntrack-tools.netfilter.org/manual.html#conntrack Or, as a workaround, right after removing the Floating IP, Neutron might insert a temporary firewall rule (for about 5~10 minutes?), to drop the connections of that previous "Floating IP + Instance IP couple"... It looks really ugly but, at least, it will make sure that nothing will pass right after removing a Floating IP... Effectively terminating (dropping) the NAT connections after disassociating a Floating IP... ;-) Also, I think that NFTables can bring some light here... I truly believe that if OpenStack moves to a "NFTables_Driver", it will be much easier to: manage firewall rules, logging, counters, IDS/IPS, atomic replacements of rules, even NAT66... All under a single implementation... Maybe with some kind of "real-time connection monitoring"... I mean, with NFTables, it becomes easier to implement a firewall ruleset with a Intrusion Prevention System (IPS), take a look: https://home.regit.org/2014/02/suricata-and-nftables/ So, if NFTables can make Suricata's life easier, why not give Suricata's power to Netutron L3 Router? Starting with a new NFTables_Driver... =) I'm not an expert on NFTables but, from what I'm seeing, it perfectly fits in OpenStack, in fact, NFTables will make OpenStack better. https://home.regit.org/2014/01/why-you-will-love-nftables/ Best! Thiago On 15 September 2014 20:49, Nathan Kinder wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Disassociating floating IPs does not terminate NAT connections with Neutron L3 agent - --- ### Summary ### Every virtual instance is automatically assigned a private IP address. You may optionally assign public IP addresses to instances. OpenStack uses the term "floating IP" to refer to an IP address (typically public) that can be dynamically added to a running virtual instance. The Neutron L3 agent uses Network Address Translation (NAT) to assign floating IPs to virtual instances. Floating IPs can be dynamically released from a running virtual instance but any active connections are not terminated with this release as expected when using the Neutron L3 agent. ### Affected Services / Software ### Neutron, Icehouse, Havana, Grizzly, Folsom ### Discussion ### When creating a virtual instance, a floating IP address is not allocated by default. After a virtual instance is created, a user can explicitly associate a floating IP address to that instance. Users can create connections to the virtual instance using this floating IP address. Also, this floating IP address can be disassociated from any running instance without shutting that instance down. If a user initiates a connection using the floating IP address, this connection remains alive and accessible even after the floating IP address is released from that instance. This potentially violates restrictive policies which are only being applied to new connections. These policies are ignored for pre-existing connections and the virtual instance remains accessible from the public network. This issue is only known to affect Neutron when using the L3 agent. Nova networking is not affected. ### Recommended Actions ### There is unfortunately no easy way to detect which connections were made over a floating IP address from a virtual instance, as the NAT is performed at the Neutron router. The only safe way of terminating all connections made over a floating IP address is to terminate the virtual instance itself. The following recommendations should be followed when using the Neutron L3 agent: - - Only attach a floating IP address to a virtual instance when that instance should be accessible from networks outside the cloud. - - Terminate or stop the instance along with disassociating the floating IP address to ensure that all connections are closed. The Neutron development team plans to address this issue in a future version of Neutron. ### Contacts / References ### This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0020 Original LaunchPad Bug : https://bugs.launchpad.net/neutron/+bug/1334926 OpenStack Security ML : openstack-secur...@l
Re: [openstack-dev] [OSSN 0020] Disassociating floating IPs does not terminate NAT connections with Neutron L3 agent
Hi, There is current work in review to use conntrack to terminate these connections [1][2] much like you suggested. I hope to get this in to RC1 but it needs another iteration. For Kilo, I'd like to explore stateless forwarding for floating ips. Since conntrack is the root of the security issue in the first place, the idea here is to eliminate it from the floating ip data path altogether [3]. The patch I have up is really just a placeholder with some notes on how it might be accomplished. My hope is that this stateless NAT for floating ips could ease some of the pressure that I've observed on conntrack and increase performance. It needs some more investigation. Carl [1] https://bugs.launchpad.net/neutron/+bug/1334926 [2] https://review.openstack.org/#/c/103475 [3] https://review.openstack.org/#/c/121689/ On Mon, Sep 15, 2014 at 11:46 PM, Martinx - ジェームズ wrote: > Hey stackers, > > Let me ask something about this... Why not use Linux Conntrack Table at each > Tenant Namespace (L3 Router) to detect which connections were > made/established over a Floating IP ? > > Like this, on the Neutron L3 Router: > > -- > apt-get install conntrack > > ip netns exec qrouter-09b72faa-a5ef-4a52-80b5-1dcbea23b1b6 conntrack -L | > grep ESTABLISHED > > tcp 6 431998 ESTABLISHED src=192.168.3.5 dst=193.16.15.250 sport=36476 > dport=8333 src=193.16.15.250 dst=187.1.93.67 sport=8333 dport=36476 > [ASSURED] mark=0 use=1 > -- > > Floating IP: 187.1.93.67 > Instance IP: 192.168.3.5 > > http://conntrack-tools.netfilter.org/manual.html#conntrack > > > > Or, as a workaround, right after removing the Floating IP, Neutron might > insert a temporary firewall rule (for about 5~10 minutes?), to drop the > connections of that previous "Floating IP + Instance IP couple"... It looks > really ugly but, at least, it will make sure that nothing will pass right > after removing a Floating IP... Effectively terminating (dropping) the NAT > connections after disassociating a Floating IP... ;-) > > > > Also, I think that NFTables can bring some light here... I truly believe > that if OpenStack moves to a "NFTables_Driver", it will be much easier to: > manage firewall rules, logging, counters, IDS/IPS, atomic replacements of > rules, even NAT66... All under a single implementation... Maybe with some > kind of "real-time connection monitoring"... I mean, with NFTables, it > becomes easier to implement a firewall ruleset with a Intrusion Prevention > System (IPS), take a look: > > https://home.regit.org/2014/02/suricata-and-nftables/ > > So, if NFTables can make Suricata's life easier, why not give Suricata's > power to Netutron L3 Router? Starting with a new NFTables_Driver... =) > > I'm not an expert on NFTables but, from what I'm seeing, it perfectly fits > in OpenStack, in fact, NFTables will make OpenStack better. > > https://home.regit.org/2014/01/why-you-will-love-nftables/ > > Best! > Thiago > > On 15 September 2014 20:49, Nathan Kinder wrote: >> >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> Disassociating floating IPs does not terminate NAT connections with >> Neutron L3 agent >> - --- >> >> ### Summary ### >> Every virtual instance is automatically assigned a private IP address. >> You may optionally assign public IP addresses to instances. OpenStack >> uses the term "floating IP" to refer to an IP address (typically >> public) that can be dynamically added to a running virtual instance. >> The Neutron L3 agent uses Network Address Translation (NAT) to assign >> floating IPs to virtual instances. Floating IPs can be dynamically >> released from a running virtual instance but any active connections are >> not terminated with this release as expected when using the Neutron L3 >> agent. >> >> ### Affected Services / Software ### >> Neutron, Icehouse, Havana, Grizzly, Folsom >> >> ### Discussion ### >> When creating a virtual instance, a floating IP address is not >> allocated by default. After a virtual instance is created, a user can >> explicitly associate a floating IP address to that instance. Users can >> create connections to the virtual instance using this floating IP >> address. Also, this floating IP address can be disassociated from any >> running instance without shutting that instance down. >> >> If a user initiates a connection using the floating IP address, this >> connection remains alive and accessible even after the floating IP >> address is released from that instance. This potentially violates >> restrictive policies which are only being applied to new connections. >> These policies are ignored for pre-existing connections and the virtual >> instance remains accessible from the public network. >> >> This issue is only known to affect Neutron when using the L3 agent. >> Nova networking is not affected. >> >> ### Recommended Actions ### >> There is unfortunately no easy way to detect which connections were >> made over a floating IP address from a virtual instance, as the NAT is >> perf
Re: [openstack-dev] [OSSN 0020] Disassociating floating IPs does not terminate NAT connections with Neutron L3 agent
Hey stackers, Let me ask something about this... Why not use Linux Conntrack Table at each Tenant Namespace (L3 Router) to detect which connections were made/established over a Floating IP ? Like this, on the Neutron L3 Router: -- apt-get install conntrack ip netns exec qrouter-09b72faa-a5ef-4a52-80b5-1dcbea23b1b6 conntrack -L | grep ESTABLISHED tcp 6 431998 ESTABLISHED src=192.168.3.5 dst=193.16.15.250 sport=36476 dport=8333 src=193.16.15.250 dst=187.1.93.67 sport=8333 dport=36476 [ASSURED] mark=0 use=1 -- Floating IP: 187.1.93.67 Instance IP: 192.168.3.5 http://conntrack-tools.netfilter.org/manual.html#conntrack Or, *as a workaround*, right after removing the Floating IP, Neutron might insert a temporary firewall rule (for about 5~10 minutes?), to drop the connections of that previous "Floating IP + Instance IP couple"... It looks really ugly but, at least, it will make sure that nothing will pass right after removing a Floating IP... Effectively terminating (dropping) the NAT connections after disassociating a Floating IP... ;-) Also, I think that NFTables can bring some light here... I truly believe that if OpenStack moves to a "NFTables_Driver", it will be much easier to: manage firewall rules, logging, counters, IDS/IPS, atomic replacements of rules, even NAT66... All under a single implementation... Maybe with some kind of "real-time connection monitoring"... I mean, with NFTables, it becomes easier to implement a firewall ruleset with a Intrusion Prevention System (IPS), take a look: https://home.regit.org/2014/02/suricata-and-nftables/ So, if NFTables can make Suricata's life easier, why not give Suricata's power to Netutron L3 Router? Starting with a new NFTables_Driver... =) I'm not an expert on NFTables but, from what I'm seeing, it perfectly fits in OpenStack, in fact, NFTables will make OpenStack better. https://home.regit.org/2014/01/why-you-will-love-nftables/ Best! Thiago On 15 September 2014 20:49, Nathan Kinder wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Disassociating floating IPs does not terminate NAT connections with > Neutron L3 agent > - --- > > ### Summary ### > Every virtual instance is automatically assigned a private IP address. > You may optionally assign public IP addresses to instances. OpenStack > uses the term "floating IP" to refer to an IP address (typically > public) that can be dynamically added to a running virtual instance. > The Neutron L3 agent uses Network Address Translation (NAT) to assign > floating IPs to virtual instances. Floating IPs can be dynamically > released from a running virtual instance but any active connections are > not terminated with this release as expected when using the Neutron L3 > agent. > > ### Affected Services / Software ### > Neutron, Icehouse, Havana, Grizzly, Folsom > > ### Discussion ### > When creating a virtual instance, a floating IP address is not > allocated by default. After a virtual instance is created, a user can > explicitly associate a floating IP address to that instance. Users can > create connections to the virtual instance using this floating IP > address. Also, this floating IP address can be disassociated from any > running instance without shutting that instance down. > > If a user initiates a connection using the floating IP address, this > connection remains alive and accessible even after the floating IP > address is released from that instance. This potentially violates > restrictive policies which are only being applied to new connections. > These policies are ignored for pre-existing connections and the virtual > instance remains accessible from the public network. > > This issue is only known to affect Neutron when using the L3 agent. > Nova networking is not affected. > > ### Recommended Actions ### > There is unfortunately no easy way to detect which connections were > made over a floating IP address from a virtual instance, as the NAT is > performed at the Neutron router. The only safe way of terminating all > connections made over a floating IP address is to terminate the virtual > instance itself. > > The following recommendations should be followed when using the Neutron > L3 agent: > > - - Only attach a floating IP address to a virtual instance when that > instance should be accessible from networks outside the cloud. > - - Terminate or stop the instance along with disassociating the floating > IP address to ensure that all connections are closed. > > The Neutron development team plans to address this issue in a future > version of Neutron. > > ### Contacts / References ### > This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0020 > Original LaunchPad Bug : https://bugs.launchpad.net/neutron/+bug/1334926 > OpenStack Security ML : openstack-secur...@lists.openstack.org > OpenStack Security Group : https://launchpad.net/~openstack-ossg > -BEGIN PGP SIGNATURE- > Version: GnuPG v1 > > iQEcBAEBAgAGBQJUF3r6AAoJEJa+6E7Ri+EVo+AH/i4GhZsFD3OJ
[openstack-dev] [OSSN 0020] Disassociating floating IPs does not terminate NAT connections with Neutron L3 agent
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Disassociating floating IPs does not terminate NAT connections with Neutron L3 agent - --- ### Summary ### Every virtual instance is automatically assigned a private IP address. You may optionally assign public IP addresses to instances. OpenStack uses the term "floating IP" to refer to an IP address (typically public) that can be dynamically added to a running virtual instance. The Neutron L3 agent uses Network Address Translation (NAT) to assign floating IPs to virtual instances. Floating IPs can be dynamically released from a running virtual instance but any active connections are not terminated with this release as expected when using the Neutron L3 agent. ### Affected Services / Software ### Neutron, Icehouse, Havana, Grizzly, Folsom ### Discussion ### When creating a virtual instance, a floating IP address is not allocated by default. After a virtual instance is created, a user can explicitly associate a floating IP address to that instance. Users can create connections to the virtual instance using this floating IP address. Also, this floating IP address can be disassociated from any running instance without shutting that instance down. If a user initiates a connection using the floating IP address, this connection remains alive and accessible even after the floating IP address is released from that instance. This potentially violates restrictive policies which are only being applied to new connections. These policies are ignored for pre-existing connections and the virtual instance remains accessible from the public network. This issue is only known to affect Neutron when using the L3 agent. Nova networking is not affected. ### Recommended Actions ### There is unfortunately no easy way to detect which connections were made over a floating IP address from a virtual instance, as the NAT is performed at the Neutron router. The only safe way of terminating all connections made over a floating IP address is to terminate the virtual instance itself. The following recommendations should be followed when using the Neutron L3 agent: - - Only attach a floating IP address to a virtual instance when that instance should be accessible from networks outside the cloud. - - Terminate or stop the instance along with disassociating the floating IP address to ensure that all connections are closed. The Neutron development team plans to address this issue in a future version of Neutron. ### Contacts / References ### This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0020 Original LaunchPad Bug : https://bugs.launchpad.net/neutron/+bug/1334926 OpenStack Security ML : openstack-secur...@lists.openstack.org OpenStack Security Group : https://launchpad.net/~openstack-ossg -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEcBAEBAgAGBQJUF3r6AAoJEJa+6E7Ri+EVo+AH/i4GhZsFD3OJWlasq+XxkqqO W7g/6YQuKgRndl63UjnWAfpvJCA8Bl1msryb2K0tTZpDByVpgupPAf6+/NMZXvCT 37YF236Ig/a/iLNjAdHRNHzq8Bhxe7tIikm1ICUH+Hyhob7soBlAC52lEJz9cFwb Hazo2K0jjt4TEyxAae06KsIuOV/n+tO7ginYxxv2g8DkhKik5PMi4x8j//DYFz92 +SwPvUKeWiZ3JmD1M84Yj4VgPxah6fKDtCYKdTdcv7pYJGlcac8DTXbJkoFVd6H/ v+XbBGWjg7+M7WlZJmDlC2XfWLVKBsREs3BAN/hagE6aKAyImT/gfyT0WxLpVIU= =Gk3u -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
