Ivan, I am a bit confused with the configuration you pictured below. NVE is on server, the server has more than one nic so each has IP interface address. But there is no routing protocol running on server and use of MPLS/VPN.
To use MPLS/VPN, you have a IGP network as underlay network first. If NVE acts as PE. The server need to participate IGP first, the loopback address of server is advertised in IGP. It does not matter how many nics exist, the ecmp make it work. If the server does not run IGP protocol and act as a IP host in underlay network. This seems not aligning with the nvo3 framework and MPLS/VPN architecture. Regards, Lucy -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Ivan Pepelnjak Sent: Friday, September 07, 2012 9:54 AM To: 'Yakov Rekhter' Cc: 'Balus, Florin Stelian (Florin)'; 'Somesh Gupta'; [email protected] Subject: Re: [nvo3] Support for multi-homed NVEs Yakov, The scenario that worries me is a hypervisor-based NVE with more than one transport (underlay) IP address, with each IP address tied to a NIC (or multiple bonded NICs), and potentially being in a different IP subnet (because it's connected to a different ToR switch). I'm further assuming that the hypervisor is not running a routing protocol and thus behaves like an IP host locally multihomed to multiple logical networks. Assuming we use something like MPLS/VPN for NVO3, the BGP sessions from NVE to its BGP neighbors (example: a set of route reflectors) would run from one of the underlay IP addresses. If another underlay IP address becomes unreachable, we cannot use BGP session drop to detect that. Furthermore, we won't have /32 routes to BGP next hops any more (that would not scale to the several million endpoints from the NVO3 charter). The first L3 switch to which the server is connected (a ToR switch or a spine switch) will originate an IP prefix ==> we cannot use NH tracking. Finally, the server might not experience a NIC link loss if it loses connection to first-hop IP gateway. Example: blade enclosure switch might be a L2-only switch ==> we cannot expect the hypervisor-based NVE to revoke prefixes or change BGP next hop if its transport address becomes unreachable. Using BFD from the NVE to detect first-hop gateway reachability and using that information to set BGP next hops in BGP prefix origination process (as proposed by Diego Garcia Del Rio) sounds like the simplest option. Does this make sense? Ivan > -----Original Message----- > From: Yakov Rekhter [mailto:[email protected]] > Sent: Friday, September 07, 2012 12:50 AM > To: Ivan Pepelnjak > Cc: 'Balus, Florin Stelian (Florin)'; [email protected]; 'Somesh Gupta'; > [email protected] > Subject: Re: [nvo3] Support for multi-homed NVEs > > Ivan, > > > Florin, > > > > All the VPNs you've mentioned share the data-plane principles with > > LISP (that's why I mentioned them in the same sentence): IP packet is > > encapsulated into an envelope with a known destination endpoint > > (PE-router, ETR), and sent on its way. The big question is "is the > > destination endpoint reachable?" > > > > In most MPLS-based implementations we assume the endpoint is reachable > > if we're getting VPN routing/signalling updates from it and if it's > > reachable through the IP routing table. Some implementations might be > > more cautious and use GRE encapsulation if there's no LSP to the > > endpoint, others might use MPLS OAM. Usually these techniques are > > glossed over and left as implementation details. > > In 2547 VPNs (or E-VPNs) it is the PE that originates routes to the VPN > sites connected to that PE. So, if the PE goes down, the routes get > withdrawn. This is "vanilla" BGP, and as such has be supported by any non- > broken BGP implementation. In addition, there are several enhancements to > speed up connectivity restoration after egress PE failure, such as Next- > Hop tracking, egress PE protection, etc... > that have been implemented by vendors. > > > On the other hand, LISP defines several mechanisms that can be used to > > check the ETR liveliness. > > Yes, except that all these LISP mechanisms are broken in one way or > another. > > > Coming back to nvo3, we'll have serious problems with NVE in a > > hypervisor using more than one underlay IP address, more so if its > > control-plane session uses only one of them. We'll never know whether > > the other IP addresses are reachable (the problem becomes worse if you > > have a DC transport infrastructure that 's a mixture of L2 and L3). > > > In situation where NVE has more than one IP address used by nvo3, we > > need (in my opinion) something that checks the liveliness of the > > remote NVE IP address ... and it's not an implementation detail, it's > > a mandatory requirement. > > "check the liveness" is a bit underspecified, as it does not say what > should be the upper bound on the detection time. > While discussing the upper bound, we should probably keep in mind that the > goal here is to minimize connectivity disruption after a failure. Handling > connectivity diruption could be done via either global or local repair > techniques (both of these have been employed in 2547 VPNs to deal with PEs > failures). > > Yakov. > > > > Hope this makes more sense > > Ivan > > > > > -----Original Message----- > > > From: Balus, Florin Stelian (Florin) [mailto:florin.balus@alcatel- > > > lucent.com] > > > Sent: Tuesday, September 04, 2012 11:46 PM > > > To: Ivan Pepelnjak > > > Cc: Somesh Gupta; [email protected] > > > Subject: RE: [nvo3] Support for multi-homed NVEs > > > > > > Ivan, > > > See in-line... > > > > > > > -----Original Message----- > > > > From: Ivan Pepelnjak [mailto:[email protected]] > > > > Sent: Tuesday, September 04, 2012 12:03 PM > > > > To: Balus, Florin Stelian (Florin) > > > > Cc: Somesh Gupta; [email protected] > > > > Subject: Re: [nvo3] Support for multi-homed NVEs > > > > > > > > A) Not per prefix (at least not without serious amount of > > > > end-to-end BGP multipathing+BGP AddPath or multiple RDs) > > > > > > > > B) There's no liveliness check in *VPN (apart from LISP) > > > [FB>] How are the VPN specifications (IP VPN, VPLS, VPWS or incoming > > > EVPN) connected to LISP? Also can you be more specific on the > > > absence of liveliness check in *VPN. Are you talking about a certain > > > deployment/implementation environment? > > > Thanks, > > > Florin > > > > > > > > > > > In MPLS/VPN we rely on the host route to PE loopback and/or LSP to > > > > the > > > > /32 prefix to indicate next hop validity. That won't work for > > > > hypervisor-based NVEs > > > > > > > > On 9/4/12 8:00 PM, Balus, Florin Stelian (Florin) wrote: > > > > > AFAIK the text in NVO3 framework and requirements drafts allows > > > > multiple IPs per NVE. We have multiple IPs per PE in current VPN > > > > implementations. This is an implementation matter though in my > opinion. > > > > > > > > > >> -----Original Message----- > > > > >> From: Somesh Gupta [mailto:[email protected]] > > > > >> Sent: Tuesday, September 04, 2012 10:51 AM > > > > >> To: Ivan Pepelnjak; Balus, Florin Stelian (Florin) > > > > >> Cc: [email protected] > > > > >> Subject: RE: [nvo3] Support for multi-homed NVEs > > > > >> > > > > >> Florin, > > > > >> > > > > >> Regarding the multi-homing, my assumption is that the NVE in > > > > >> the hypervisor would not (want to) run a routing protocol. So > > > > >> as Ivan points out, the standard would need to accommodate > > > > >> multiple IP addresses per NVE. > > > > >> > > > > >> Somesh > > > > >> > > > > >>> -----Original Message----- > > > > >>> From: Ivan Pepelnjak [mailto:[email protected]] > > > > >>> Sent: Tuesday, September 04, 2012 9:58 AM > > > > >>> To: Balus, Florin Stelian (Florin) > > > > >>> Cc: Somesh Gupta; [email protected] > > > > >>> Subject: Re: [nvo3] Support for multi-homed NVEs > > > > >>> > > > > >>> Ah, that other can of worms ;) Mine was simpler. > > > > >>> > > > > >>> On the underlay side, we might decide that NVEs have a single > > > > >>> IP address or multiple IP addresses (like some NVGRE load > > > > >>> balancing proposals). If we decide NVEs have a single IP > > > > >>> address (potential per virtual network segment), then the rest > > > > >>> is implementation details > > > > >> (and > > > > >>> we're back to MLAG/SMLT land for true redundancy). > > > > >>> Alternatively we might implement the option of having multiple > > > > >>> IP addresses per NVE, and the NVEs might use the > > > > >>> IP-address-per-link option (thus no need for L2 or MLAG at all). > > > > >>> > > > > >>> On the overlay side, the real problem (as you stated) is the > > > > >>> multi-homing of NVO3-to-legacy gateways. I don't see any other > > > > >>> need for overlay NVE multihoming. > > > > >>> > > > > >>> BTW, Nicira has nicely solved the NVO3 gateway multihoming - > > > > >>> the > > > > >> whole > > > > >>> NVO3 network works exactly like VMware's vSwitch: split > > > > >>> horizon bridging (thus no forwarding loops through NVO3), with > > > > >>> every VM MAC address being dynamically assigned to one of the > > > > >>> gateways, which also solves the return path issues (dynamic > > > > >>> MAC learning in legacy network takes care of that). Maybe we > > > > >>> should just use the wheel > > > > that > > > > >>> has already been invented? > > > > >>> > > > > >>> Kind regards, > > > > >>> Ivan > > > > >>> > > > > >>> On 9/4/12 6:45 PM, Balus, Florin Stelian (Florin) wrote: > > > > >>>> I understand the discussion below is about the NVE > > > > >>>> multi-homing > > > > >>> towards the IP core, on the tunnel side. > > > > >>>> We did not focus in the framework draft on the core > > > > >>>> redundancy as > > > > >> in > > > > >>> our opinion there was no need to standardize anything here. > > > > >>> There are no differences from what is available today in > > > > >>> regular IP > > > > networks: > > > > >> if > > > > >>> NVEs are multi-homed directly to the next IP router, regular > > > > routing > > > > >>> will take care of it. If there is Ethernet switching in > > > > >>> between NVE and the next IP hop, L2 resiliency mechanisms need > > > > >>> to be > > > employed. > > > > >>> From what I read below it looks more of an implementation > > > > >>> discussion than a standardization requirement. Am I right? > > > > >>>> By Multi-homed NVEs one can also understand a set of NVEs > > > > >>>> multi-homed > > > > >>> on the access side to other devices. That is a discussion we > > > > >>> need > > > > to > > > > >>> have in my opinion. An use case example: NVO3 network - NVE > > > > >>> GWs > > > > >> multi- > > > > >>> homed to external non-NVO3 networks. Handoff can be VLANs, > > > > >>> VPLS > > > > PWs, > > > > >>> or BGP EVPN labels... > > > > >>>> I think the latter is worth discussing although there are > > > > >>>> some > > > > >>> mechanisms and some standardization initiatives in place > already. > > > > >>>> > > > > >>>>> -----Original Message----- > > > > >>>>> From: [email protected] [mailto:[email protected]] > > > > >>>>> On Behalf > > > > >>> Of > > > > >>>>> Ivan Pepelnjak > > > > >>>>> Sent: Friday, August 31, 2012 12:23 AM > > > > >>>>> To: 'Somesh Gupta'; [email protected] > > > > >>>>> Subject: Re: [nvo3] Support for multi-homed NVEs > > > > >>>>> > > > > >>>>> This is definitely an interesting can of worms ;) > > > > >>>>> > > > > >>>>> While I don't think we should go down the path of IP-A/IP-B > > > > >>>>> networks similar to some other DC technology, we will face > > > > >>>>> the reality of > > > > >>> some > > > > >>>>> NVE elements (hypervisor soft switches) not being underlay > > > > >>>>> IP > > > > >>> routers. > > > > >>>>> We could either: > > > > >>>>> > > > > >>>>> (A) ignore the issue and expect the network designer to > > > > >>>>> solve it > > > > >>> using > > > > >>>>> any one of the existing NIC teaming/MLAG kludges while > > > > >>>>> retaining > > > > a > > > > >>>>> single encapsulation IP address per NVE; > > > > >>>>> > > > > >>>>> (B) provide support for multiple encapsulation addresses per > > > > >>>>> NVE > > > > >> so > > > > >>> a > > > > >>>>> multi-homed NVE could have one IP address per physical > > > > >>>>> interface and send and receive nvo3-encapsulated frames > > > > >>>>> using more than one > > > > >>> address. > > > > >>>>> Option (A) is the easy way out similar to existing MPLS/VPN > > > > >>>>> behavior and would fit well with existing DC deployments. It > > > > would > > > > >>>>> also > > > > >>> retain > > > > >>>>> all the server-to-ToR multihoming complexity. > > > > >>>>> > > > > >>>>> Option (B) would reduce the complexity of the underlay DC > > > > >>>>> network (which would become a simple L3 network with > > > > >>>>> single-homed IP addresses), but we'd have to deal with a > > > > >>>>> bunch of additional > > > > >>> problems > > > > >>>>> (peer IP address liveliness check). > > > > >>>>> > > > > >>>>> Just speculating ... > > > > >>>>> Ivan > > > > >>>>> > > > > >>>>>> -----Original Message----- > > > > >>>>>> From: [email protected] [mailto:[email protected]] > > > > >>>>>> On > > > > >>> Behalf > > > > >>>>>> Of Somesh Gupta > > > > >>>>>> Sent: Friday, August 31, 2012 6:58 AM > > > > >>>>>> To: [email protected] > > > > >>>>>> Subject: [nvo3] Support for multi-homed NVEs > > > > >>>>>> > > > > >>>>>> I did not see any mention of multi-homed NVEs in > > > > >>>>>> draft-lasserre- > > > > >>> nvo3- > > > > >>>>>> framework-03.txt. NVEs are connected together by an L3 > > > > >>>>>> network > > > > >>>>>> - > > > > >>> does > > > > >>>>>> that mean only one? > > > > >>>>>> Can it be multi-homes to two L3 networks? > > > > >>>>>> > > > > >>>>>> Somesh > > > > >>>>>> _______________________________________________ > > > > >>>>>> nvo3 mailing list > > > > >>>>>> [email protected] > > > > >>>>>> https://www.ietf.org/mailman/listinfo/nvo3 > > > > >>>>> _______________________________________________ > > > > >>>>> nvo3 mailing list > > > > >>>>> [email protected] > > > > >>>>> https://www.ietf.org/mailman/listinfo/nvo3 > > > > _______________________________________________ > > nvo3 mailing list > > [email protected] > > https://www.ietf.org/mailman/listinfo/nvo3 _______________________________________________ nvo3 mailing list [email protected] https://www.ietf.org/mailman/listinfo/nvo3 _______________________________________________ nvo3 mailing list [email protected] https://www.ietf.org/mailman/listinfo/nvo3
