Re: [openstack-dev] Fwd: [Neutron][DVR]Neutron distributed SNAT

2015-02-18 Thread Kevin Benton
If I understand correctly, for southbound traffic there would be
hair-pinning via the L3 agent that the upstream router happened to pick out
of the ECMP group since it doesn't know where the hypervisors are. On the
other hand northbound traffic could egress directly (assuming an l3 agent
is running on each compute node in the DVR fashion).

If we went down this route, we would require a dynamic routing protocol to
run between the agents and the upstream router. Additionally, we would have
to tweak our addressing scheme a bit so the l3 agents could have separate
addresses to use for their BGP session (or whatever routing protocol we
choose) since the gateway address would be shared amongst them.

Did I get what you were proposing correctly?

On Wed, Feb 18, 2015 at 5:28 PM, Angus Lees g...@inodes.org wrote:

 On Mon Feb 16 2015 at 9:37:22 PM Kevin Benton blak...@gmail.com wrote:

 It's basically very much like floating IPs, only you're handing out a
 sub-slice of a floating-IP to each machine - if you like.

 This requires participation of the upstream router (L4 policy routing
 pointing to next hops that distinguish each L3 agent) or intervention on
 the switches between the router an L3 agents (a few OpenFlow rules would
 make this simple). Both approaches need to adapt to L3 agent changes so
 static configuration is not adequate. Unfortunately, both of these are
 outside of the control of Neutron so I don't see an easy way to push this
 state in a generic fashion.


 (Just to continue this thought experiment)

 The L3 agents that would need to forward ingress traffic to the right
 hypervisor only need to know which [IP+port range] has been assigned to
 which hypervisor.  This information is fairly static, so these forwarders
 are effectively stateless and can be trivially replicated to deal with the
 desired ingress volume and reliability.

 When I've built similar systems in the past, the easy way to interface
 with the rest of the provider network was to use whatever dynamic routing
 protocol was already in use, and just advertise multiple ECMP routes for
 the SNAT source IPs from the forwarders (ideally advertising from the
 forwarders themselves, so they stop if there's a connectivity issue).  All
 the cleverness then happens on the forwarding hosts (we could call them
 L3 agents).  It's simple and works well, but I agree we have no precedent
 in neutron at present.

 On Mon, Feb 16, 2015 at 12:33 AM, Robert Collins 
 robe...@robertcollins.net wrote:

 Or a pool of SNAT addresses ~= to the size of the hypervisor count.


 Oh yeah. If we can afford to assign a unique SNAT address per hypervisor
 then we're done - at that point it really is just like floating-ips.

  - Gus

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Kevin Benton
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Fwd: [Neutron][DVR]Neutron distributed SNAT

2015-02-18 Thread Angus Lees
On Mon Feb 16 2015 at 9:37:22 PM Kevin Benton blak...@gmail.com wrote:

 It's basically very much like floating IPs, only you're handing out a
 sub-slice of a floating-IP to each machine - if you like.

 This requires participation of the upstream router (L4 policy routing
 pointing to next hops that distinguish each L3 agent) or intervention on
 the switches between the router an L3 agents (a few OpenFlow rules would
 make this simple). Both approaches need to adapt to L3 agent changes so
 static configuration is not adequate. Unfortunately, both of these are
 outside of the control of Neutron so I don't see an easy way to push this
 state in a generic fashion.


(Just to continue this thought experiment)

The L3 agents that would need to forward ingress traffic to the right
hypervisor only need to know which [IP+port range] has been assigned to
which hypervisor.  This information is fairly static, so these forwarders
are effectively stateless and can be trivially replicated to deal with the
desired ingress volume and reliability.

When I've built similar systems in the past, the easy way to interface with
the rest of the provider network was to use whatever dynamic routing
protocol was already in use, and just advertise multiple ECMP routes for
the SNAT source IPs from the forwarders (ideally advertising from the
forwarders themselves, so they stop if there's a connectivity issue).  All
the cleverness then happens on the forwarding hosts (we could call them
L3 agents).  It's simple and works well, but I agree we have no precedent
in neutron at present.

On Mon, Feb 16, 2015 at 12:33 AM, Robert Collins robe...@robertcollins.net
 wrote:

 Or a pool of SNAT addresses ~= to the size of the hypervisor count.


Oh yeah. If we can afford to assign a unique SNAT address per hypervisor
then we're done - at that point it really is just like floating-ips.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Fwd: [Neutron][DVR]Neutron distributed SNAT

2015-02-16 Thread Assaf Muller


- Original Message -
 Has there been any work to use conntrack synchronization similar to L3 HA in
 DVR so failover is fast on the SNAT node?
 

https://review.openstack.org/#/c/139686/
https://review.openstack.org/#/c/143169/

These changes have taken a back seat to improving the DVR job reliability
and L3 agent refactoring. Michael and Rajeev could expand.

 On Sat, Feb 14, 2015 at 1:31 PM, Carl Baldwin  c...@ecbaldwin.net  wrote:
 
 
 
 
 
 On Feb 10, 2015 2:36 AM, Wilence Yao  wilence@gmail.com  wrote:
  
  
  Hi all,
  After OpenStack Juno, floating ip is handled by dvr, but SNAT is still
  handled by l3agent on network node. The distributed SNAT is in future
  plans for DVR. In my opinion, SNAT can move to DVR as well as floating ip.
  I have searched in blueprint, there is little about distributed SNAT. Is
  there any different between distributed floating ip and distributed SNAT?
 
 The difference is that a shared snat address is shared among instances on
 multiple compute nodes. A floating ip is exclusive to a single instance on
 one compute node. I'm interested to hear your ideas for distributing it.
 
 Carl
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 --
 Kevin Benton
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Fwd: [Neutron][DVR]Neutron distributed SNAT

2015-02-16 Thread Kevin Benton
Or a pool of SNAT addresses ~= to the size of the hypervisor count.

This had originally come up as an option in the early DVR discussions. IIRC
it was going to be a tunable parameter since it results in a tradeoff
between spent public addresses and distributed-ness. However, due to time
constraints and complexity, the burning of extra IPs to distribute SNAT
wasn't implemented because it required changes to the data model (multiple
IPs per router gateway interface) and changes to when IPs were assigned
(dynamically adding more IPs to the gateway interface as tenant ports were
instantiated on new nodes). Someone from the DVR team can correct me if I'm
missing the reasons behind some of these decisions.


Conntrack synchronisation gets us HA on the SNAT node, but that's a long
way from distributed SNAT.

Definitely, I was not paying close attention and thought this thread was
just about the HA of the SNAT node.

It's basically very much like floating IPs, only you're handing out a
sub-slice of a floating-IP to each machine - if you like.

This requires participation of the upstream router (L4 policy routing
pointing to next hops that distinguish each L3 agent) or intervention on
the switches between the router an L3 agents (a few OpenFlow rules would
make this simple). Both approaches need to adapt to L3 agent changes so
static configuration is not adequate. Unfortunately, both of these are
outside of the control of Neutron so I don't see an easy way to push this
state in a generic fashion.



On Mon, Feb 16, 2015 at 12:33 AM, Robert Collins robe...@robertcollins.net
wrote:

 On 16 February 2015 at 21:29, Angus Lees g...@inodes.org wrote:
  Conntrack synchronisation gets us HA on the SNAT node, but that's a long
 way
  from distributed SNAT.
 
  Distributed SNAT (in at least one implementation) needs a way to allocate
  unique [IP + ephemeral port ranges] to hypervisors, and then some sort of
  layer4 loadbalancer capable of forwarding the ingress traffic to that IP
  back to the right hypervisor/guest based on the ephemeral port range.
 It's
  basically very much like floating IPs, only you're handing out a
 sub-slice
  of a floating-IP to each machine - if you like.

 Or a pool of SNAT addresses ~= to the size of the hypervisor count.

 -Rob


 --
 Robert Collins rbtcoll...@hp.com
 Distinguished Technologist
 HP Converged Cloud

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Kevin Benton
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Fwd: [Neutron][DVR]Neutron distributed SNAT

2015-02-16 Thread Robert Collins
On 16 February 2015 at 21:29, Angus Lees g...@inodes.org wrote:
 Conntrack synchronisation gets us HA on the SNAT node, but that's a long way
 from distributed SNAT.

 Distributed SNAT (in at least one implementation) needs a way to allocate
 unique [IP + ephemeral port ranges] to hypervisors, and then some sort of
 layer4 loadbalancer capable of forwarding the ingress traffic to that IP
 back to the right hypervisor/guest based on the ephemeral port range.  It's
 basically very much like floating IPs, only you're handing out a sub-slice
 of a floating-IP to each machine - if you like.

Or a pool of SNAT addresses ~= to the size of the hypervisor count.

-Rob


-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Fwd: [Neutron][DVR]Neutron distributed SNAT

2015-02-16 Thread Angus Lees
Conntrack synchronisation gets us HA on the SNAT node, but that's a long
way from distributed SNAT.

Distributed SNAT (in at least one implementation) needs a way to allocate
unique [IP + ephemeral port ranges] to hypervisors, and then some sort of
layer4 loadbalancer capable of forwarding the ingress traffic to that IP
back to the right hypervisor/guest based on the ephemeral port range.  It's
basically very much like floating IPs, only you're handing out a sub-slice
of a floating-IP to each machine - if you like.

On Mon Feb 16 2015 at 6:12:33 PM Kevin Benton blak...@gmail.com wrote:

 Has there been any work to use conntrack synchronization similar to L3 HA
 in DVR so failover is fast on the SNAT node?

 On Sat, Feb 14, 2015 at 1:31 PM, Carl Baldwin c...@ecbaldwin.net wrote:


 On Feb 10, 2015 2:36 AM, Wilence Yao wilence@gmail.com wrote:
 
 
  Hi all,
After OpenStack Juno, floating ip is handled by dvr, but SNAT is
 still handled by l3agent on network node. The distributed SNAT is in future
 plans for DVR. In my opinion, SNAT can move to DVR as well as floating ip.
 I have searched in blueprint, there is little  about distributed SNAT. Is
 there any different between distributed floating ip and distributed SNAT?

 The difference is that a shared snat address is shared among instances on
 multiple compute nodes.  A floating ip is exclusive to a single instance on
 one compute node.  I'm interested to hear your ideas for distributing it.

 Carl

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Kevin Benton
  
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Fwd: [Neutron][DVR]Neutron distributed SNAT

2015-02-15 Thread Kevin Benton
Has there been any work to use conntrack synchronization similar to L3 HA
in DVR so failover is fast on the SNAT node?

On Sat, Feb 14, 2015 at 1:31 PM, Carl Baldwin c...@ecbaldwin.net wrote:


 On Feb 10, 2015 2:36 AM, Wilence Yao wilence@gmail.com wrote:
 
 
  Hi all,
After OpenStack Juno, floating ip is handled by dvr, but SNAT is still
 handled by l3agent on network node. The distributed SNAT is in future plans
 for DVR. In my opinion, SNAT can move to DVR as well as floating ip. I have
 searched in blueprint, there is little  about distributed SNAT. Is there
 any different between distributed floating ip and distributed SNAT?

 The difference is that a shared snat address is shared among instances on
 multiple compute nodes.  A floating ip is exclusive to a single instance on
 one compute node.  I'm interested to hear your ideas for distributing it.

 Carl

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Kevin Benton
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev