Weiguo, Comment inline.
Yours Irrespectively, John > -----Original Message----- > From: BESS [mailto:[email protected]] On Behalf Of Haoweiguo > Sent: Saturday, November 21, 2015 2:18 AM > To: UTTARO, JAMES; John E Drake; EXT - [email protected]; Lucy > yong; Henderickx, Wim (Wim); [email protected] > Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc > > Hi Jim & John, > Firstly i think option-C interconnection between cloud data center and MPLS > VPN network exists. Secondly, i think for different encapsulations, different > interconnecting methods are needed. For VXLAN/NVGRE, VTEP IP/UDP Port > allocation solutions are needed. [JD] I don't think this is either the best or the only solution and I don't think we should be in any rush to adopt it. > Only for MPLSoGRE/oUDP, both standard > MPLS+MPLS+GRE/UDP and VTEP IP/UDP Port allocation solution can work, it > shoud be measured which one is more suitable or both solutions are > proposed to the industry for reference. > Thanks, > weiguo > > ________________________________________ > From: BESS [[email protected]] on behalf of UTTARO, JAMES > [[email protected]] > Sent: Saturday, November 21, 2015 1:51 > To: John E Drake; EXT - [email protected]; Lucy yong; Henderickx, > Wim (Wim); [email protected] > Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc > > +1 > > -----Original Message----- > From: BESS [mailto:[email protected]] On Behalf Of John E Drake > Sent: Friday, November 20, 2015 12:19 PM > To: EXT - [email protected]; Lucy yong; Henderickx, Wim (Wim); > [email protected] > Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc > > Lucy, > > My apologies, I misunderstood. > > I think we have to accept the fact that we will have to deal with a > multiplicity > of different encapsulations in the data plane along a packet's e2e path and > that we should take a more measured approach to deciding how to deal with > this in a general and extensible way before accepting any solutions. > > Yours Irrespectively, > > John > > > -----Original Message----- > > From: [email protected] [mailto:[email protected]] > > Sent: Friday, November 20, 2015 12:04 PM > > To: John E Drake; Lucy yong; Henderickx, Wim (Wim); [email protected] > > Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc > > > > 2015-11-20, John E Drake: > > > That presupposes that the group likes either of the two proposed > > > solutions > > in your draft. > > > > John, I think Lucy's "two solutions" was referring to > > draft-hao-bess-inter- nvo3-vpn-optionc solution and the 3-label > > Optionc MPLS/MPLS/UDP solution described by Wim. > > > > -Thomas > > > > > > > > > > > >> -----Original Message----- > > >> From: BESS [mailto:[email protected]] On Behalf Of Lucy yong > > >> Sent: Friday, November 20, 2015 11:49 AM > > >> To: EXT - [email protected]; Henderickx, Wim (Wim); > > >> [email protected] > > >> Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc > > >> > > >> Share my 2 cent. > > >> > > >> Cloud providers want to tunnel its customer traffic through DC (AS)BR. > > >> Option C is a way to realize it. Both solutions summarized by > > >> Thomas have no change on WAN VPN side and seamlessly work with > WAN > > >> VPN > > option C. > > >> However, to support either solution, DC has to do some enhancement > > >> on DC BR or ToR switch, etc, which dictates to different > > >> implementations within a DC. Option C pro and con remains > > >> regardless what implementation is used in a DC. > > >> > > >> Two solutions should not coexist in one DC (does not make a sense), > > >> but it does not matter if one DC operator prefers to use one > > >> solution and another DC prefers to use another solution. Since > > >> there are many cloud providers today, it is worth for the WG to > > >> document both solutions and point out the implementation > > >> requirements on impacted components. Then, up to vendors and > > >> operators to select a solution for > > their DC. > > >> > > >> It does not make a sense for us to vote which solution is better > > >> here because a better solution for a DC depends on many factors. > > >> > > >> > > >> Lucy > > >> > > >> > > >> > > >> > > >> > > >> -----Original Message----- > > >> From: BESS [mailto:[email protected]] On Behalf Of > > >> [email protected] > > >> Sent: Friday, November 20, 2015 3:02 AM > > >> To: Henderickx, Wim (Wim); [email protected] > > >> Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc > > >> > > >> 2015-11-19, Henderickx, Wim (Wim): > > >>> WH> I vote for a an evolution of switches/TORs that have proper > > >>> support for this. I hope some HW vendors of TOR chips shime in, > > >>> but I am told the MPLS solution is possible in the next generation > > >>> chips they are working on. > > >> > > >> Well, it looks like the key questions are: > > >> - when would ToR chips support MPLS/MPLS/UDP ? (the generation > > >> that has been released recently but not present in most shipping > > >> ToRs yet, the next one ?) > > >> - do we want an interim VXLAN-based solution ? (that will involve > > >> at best a performance penalty with existing chips, and at worse > > >> impossible to implement -- we haven't seen clear information in > > >> this > > >> thread) > > >> > > >> -Thomas > > >> > > >> > > >>>> Zhuangshunwan : > > >>>>> > > >>>>> Hi Diego, > > >>>>> > > >>>>> Thanks for your comments. Pls see inline with [Vincent]. > > >>>>> > > >>>>> Vincent > > >>>>> > > >>>>> *发件人:*BESS [mailto:[email protected]] *代表 *Diego > Garcia > > >> del Rio > > >>>>> *发送时间:*2015年11月18日14:25 *收件人:*[email protected] *主 > 题 > > >> :*Re: [bess] > > >>>>> draft-hao-bess-inter-nvo3-vpn-optionc > > >>>>> > > >>>>> Some comments from my side, I think the draft makes quite a few > > >>>>> assumptions on specific implementation details that are way too > > >>>>> general to be considered widely available. Most of the TOR > > >>>>> devices today already pay a double-pass penalty when doing > > >>>>> routing of traffic in/out of vxlan-type tunnels. Only the newest > > >>>>> generation can route into tunnels without additional passes. And > > >>>>> are definitively limited in being able to arbitrary select UDP > > >>>>> ports on a per tunnel basis. In fact, most are even limited at > > >>>>> using more than one VNID per "service" of sorts. [Vincent]: Yes, > > >>>>> the new generation BCM chipset has already supported VXLAN > > >>>>> routing > > without > > >>>>> additional passes. For OVS/TOR, VXLAN encapsulation is more > > >>>>> popular than MPLSoGRE/UDP, and has better performance. The > > >>>>> IP-addressed based implementation which would, I assume, imply > > >>>>> assigning one or more CIDRs to a loopback interface on the > > >>>>> ASBR-d is also quite arbitrary and does not seem like a > > >>>>> technically "clean" solution. (besides burning tons of IPs). As > > >>>>> a side-note, most PE-grade routers i've worked with were quite > > >>>>> limited in terms of IP addresses used for tunnel termination and > > >>>>> it wasn't that just a simple pool can be used. [Vincent]: I > > >>>>> think the larger VTEP IP address range on ASBR-d has no limitations. > > >>>>> For the hardware on ASBR-d, it has capability to terminate > > >>>>> multiple VXLAN tunnels with arbitrary destination VTEP IP > > >>>>> addresses. Wim's mentions on cases where the Application itself, > > >>>>> hosted in a datacenter, would be part of the option-C > > >>>>> interconnect, was dismissed without much discussion so far, > > >>>>> while, if we look in detail at the type of users which will even > > >>>>> consider a complex topology like model-C its most likely users > > >>>>> and operators very familiar with MPLS VPNs in the WAN. Those > > >>>>> type of operators will most likely be very interested in > > >>>>> deploying MPLS or WAN-grade applications (i.e., virtual-routers > > >>>>> or other > > >>>>> VNFs) in the DC and thus its highly likely that the interconnect > > >>>>> would not terminate at the NVE itself but rather the TS (the > > >>>>> virtual machine). Also, the use of UDP ports at random would > > >>>>> imply quite complex logic on the ASBR-d IMHO. Im not saying its > > >>>>> impossible, but since the received packet now not only has to be > > >>>>> received on a random (though locally chosen) UDP port and this > > >>>>> information preserved in the pipeline to be able to do the > > >>>>> double-tunnel-stitching, it sounds quite complex. I am sure > > >>>>> someone in the list will mention this has already been > > >>>>> implemented somewhere, and I won't argue with that. And let's > > >>>>> not even bring into account what this would do to any DC > > >>>>> middlebox that now has to look at vxlan over almost any random > > >>>>> port. We have to go back to the "is it a 4 or is it a 6 in byte > > >>>>> x" heuristics to try to guess whether the packet is vxlan or > > >>>>> just something entirely different running over IP. [Vincent]: > > >>>>> Using NP or multi-core CPU hardware technology, it can be > > >>>>> implemented although deeper packet inspection is needed to > > >>>>> perform UDP port and MPLS stitching. In general I feel the > > >>>>> proposed solution seems to be fitting of a specific use-case which is > not really detailed > > >>>>> in the draft and does not describe a solution that would > > >>>>> "elegantly" solve the issues at hand. It just feels like we're > > >>>>> using any available bit-space to stuff data into places that do > > >>>>> not necesarily belong. Yes, MPLS encapsulations on virtual > > >>>>> switches are not yet fully available, and there can be some > > >>>>> performance penalty on the TORs, but the solutions are much > > >>>>> cleaner from a control and data plane point of view. Maybe I'm > > >>>>> too utopic. [Vincent]: I think pure VXLAN solution has its > > >>>>> scenario, it's general rather than specific. We can't require > > >>>>> all OVS/NVEs support VXLAN + MPLSoGRE at the same time. Best > > >>>>> regards, Diego > > >>>>> ---------------------------------------------------------------- > > >>>>> -- > > >>>>> -- > > >>>>> ------------- > > >>>>> > > >>>>> > > >> Hi, > > >>>>> The problem we are trying to solve is to reduce data center > > >>>>> GW/ASBR-d's forwarding table size, the motivation is same as > > >>>>> traditional MPLS VPN option-C. Currently, the most common > > >>>>> practise on ASBR-d is to terminate VXLAN encapsulation, look up > > >>>>> local routing table, and then perform MPLS encapsulation to the > > >>>>> WAN > > network. > > >>>>> ASBR-d needs to maintain all VM's MAC/IP. In Option-C method, > > >>>>> only transport layer information needed to be maintained at > > >>>>> GW/ASBR-d, the scalability will be greatly enhanced. Traditonal > > >>>>> Option-C is only for MPLS VPN network interworking, because > > >>>>> VXLAN is > > becoming > > >>>>> pervasive in data center, the solution in this draft was > > >>>>> proposed for the heterogeneous network interworking. The > > >>>>> advantage of this solution is that only VXLAN encapsulation is > required for OVS/TOR. > > >>>>> Unlike Wim's solution, east-west bound traffic uses VXLAN encap, > > >>>>> while north-south bound traffic uses MPLSoGRE/UDP encap. There > > are > > >>>>> two solutions in this draft: 1. Using VXLAN tunnel destination > > >>>>> IP for stitching at ASBR-d. No data plane modification > > >>>>> requirements on OVS or TOR switches, only hardware changes on > > >>>>> ASBR-d. ASBR-d normally is router, it has capability to realize > > >>>>> the hardware changes. It will consume many IP addresses and the > > >>>>> IP pool for allocation needs to be configured on ASBR-d > > >>>>> beforehand. 2. Using VXLAN destination UDP port for stitching at > > >>>>> ASBR-d. Compared with solution 1, less IP address will be > > >>>>> consumed for allocation. If UDP port range is too large, we can > combine with solution 1 and 2. > > >>>>> In this solution, both data plane modification changes are > > >>>>> needed at OVS/TOR and ASBR-d. ASBR-d also has capability to > > >>>>> realize the hardware changes. For OVS, it also can realize the > > >>>>> data plane changes. For TOR switch, it normally can't realize this > function. > > >>>>> This solution mainly focuses on pure software based overlay > > >>>>> network, it has more scalability. In public cloud data center, > > >>>>> software based overlay network is the majority case. Whether > > >>>>> using solution 1 or 2 depends on the operators real envionment. > > >>>>> So I think our solution has no flaws, it works fine. > > >>>>> Thanks, weiguo ________________________________ From: > BESS > > >>>>> [[email protected] <mailto:[email protected]>] on > behalf > > >>>>> of John E Drake [[email protected] <mailto:[email protected]>] > > >>>>> Sent: Wednesday, November 18, 2015 2:49 To: Henderickx, Wim > > (Wim); > > >>>>> EXT - [email protected] > > <mailto:[email protected]>; > > >> BESS > > >>>>> Subject: Re: [bess] draft-hao-bess-inter-nvo3-vpn-optionc Hi, I > > >>>>> think Wim has conclusively demonstrated that this draft has > > >>>>> fatal flaws and I don’t support it. I also agree with his > > >>>>> suggestion that we first figure out what problem we are trying > > >>>>> to solve before solving it. Yours Irrespectively, John From: > > >>>>> BESS [mailto:[email protected] > > >>>>> <mailto:[email protected]>] On Behalf Of Henderickx, Wim > > >>>>> (Wim) Sent: Tuesday, November 17, 2015 > > >>>>> 12:49 PM To: EXT - [email protected] > > >>>>> <mailto:[email protected]>; BESS Subject: Re: [bess] > > >>>>> draft-hao-bess-inter-nvo3-vpn-optionc ― Snip ― No, the spec as > > >>>>> it is can be implemented in its VXLAN variant with existing > > >>>>> vswitches (e.g. OVS allows to choose the VXLAN destination port, > > >>>>> ditto for the linux kernel stack). (ToR is certainly another > > >>>>> story, most of them not having a flexible enough VXLAN dataplane > > >>>>> nor support for any > > >>>>> MPLS-over-IP.) WH> and how many ports simultaneously would > they > > >>>>> support? For this to work every tenant needs a different VXLAN > > >>>>> UDP destination port/receive port. There might be SW elements > > >>>>> that could do some of this, but IETF defines solutions which > > >>>>> should be implemented across the board HW/SW/etc. > > >>>>> Even if some SW switches can do this, the proposal will impose > > >>>>> so many issues in HW/data-plane engines that I cannot be behind > > >>>>> this solution. To make this work generically we will have to > > >>>>> make changes anyhow. Given this, we better do it in the right > > >>>>> way and guide the industry to a solution which does not imply > > >>>>> those > > complexities. > > >>>>> Otherwise we will stick with these specials forever with all > > >>>>> consequences (bugs, etc). - snip - From: > > >>>>> "[email protected] > > >>>>> > > >> > > <mailto:[email protected]><mailto:[email protected] > > >>>>> <mailto:[email protected]>>" > <[email protected] > > >>>>> > > >> > > <mailto:[email protected]><mailto:[email protected] > > >>>>> <mailto:[email protected]>>> Organization: Orange Date: > > >>>>> Tuesday 17 November 2015 at 01:37 To: Wim Henderickx > > >>>>> <[email protected] > > >>>>> <mailto:wim.henderickx@alcatel- > > >> lucent.com><mailto:wim.henderickx@alc > > >>>>> atel-lucent.com > > >>>>> > > >>>>> > > >> <mailto:[email protected]>>>, BESS <[email protected] > > >>>>> <mailto:[email protected]><mailto:[email protected] > > >>>>> <mailto:[email protected]>>> Subject: Re: [bess] > > >>>>> draft-hao-bess-inter-nvo3-vpn-optionc Hi Wim, WG, 2015-11-16, > > >>>>> Henderickx, Wim (Wim): 2015-11-13, Henderickx, Wim (Wim): > > Thomas, > > >> we > > >>>>> can discuss forever and someone need to describe requirements, > > >>>>> but the current proposal I cannot agree to for the reasons > explained. > > >>>>> TM> Well, although discussing forever is certainly not the goal, > > >>>>> TM> the > > >>>>> reasons for rejecting a proposal need to be thoroughly understood. > > >>>>> WH> my point is what is the real driver for supporting a plain > > >>>>> WH> VXLAN > > >>>>> data-plane here, the use cases I have seen in this txt is always > > >>>>> where an application behind a NVE/TOR is demanding option c, but > > >>>>> none of the NVE/TOR elements. > > >>>>> My understanding is that the applications are contexts where > > >>>>> overlays are present is when workloads (VMs or baremetal) need > > >>>>> to be interconnected with VPNs. In these contexts, there can be > > >>>>> reasons to want Option C to reduce the state on ASBRs. In these > > >>>>> context, its not the workload (VM or baremetal) that would > > >>>>> typically handle VRFs, but really the vswitch or ToR. WH2> can > > >>>>> it not > > be all cases: > > >>>>> TOR/vswitch/Application. I would make the solution flexible to > > >>>>> support all of these not? 2015-11-13, Henderickx, Wim (Wim): TM> > > >>>>> The right trade-off to make may in fact depend on whether you > > prefer: > > >>>>> (a) a new dataplane stitching behavior on DC ASBRs (the behavior > > >>>>> specified in this draft) or > > >>>>> (b) an evolution of the encaps on the vswitches and ToRs to > > >>>>> support MPLS/MPLS/(UDP or GRE) WH> b depends on the use case > I > > >>>>> don't get what you mean by "b depends on the use case". WH> see > > my > > >>>>> above comment. If the real use case is an application behind > > >>>>> NVE/TOR requiring model C, than all the discussion on impact on > > >>>>> NVE/TOR is void. As such I want to have a discussion on the real > > >>>>> driver/requirement for option c interworking with an IP based > > >>>>> Fabric. Although I can agree than detailing requirements can > > >>>>> always help, I don't think one can assume a certain application > > >>>>> to dismiss the proposal. WH> for me the proposal is not > > >>>>> acceptable for the reasons explained: too much impact on the > > >>>>> data-planes I wrote the above based on the idea that the encap > > >>>>> used in MPLS/MPLS/(UDP or GRE), which hence has to be > supported > > >>>>> on the > > ToRs and vswitches. > > >>>>> Another possibility would be service-label/middle-label/Ethernet > > >>>>> assuming an L2 adjacency between vswitches/ToRs and ASBRs, but > > >>>>> this certainly does not match your typical DC architecture. Or > > >>>>> perhaps had you something else in mind ? WH> see above. The > > >>>>> draft right now also requires changes in existing TOR/NVE so for > > >>>>> me all this discussion/debate is void. No, the spec as it is can > > >>>>> be implemented in its VXLAN variant with existing vswitches > > >>>>> (e.g. OVS allows to choose the VXLAN destination port, ditto for > > >>>>> the linux kernel stack). (ToR is certainly another story, most > > >>>>> of them not having a flexible enough VXLAN dataplane nor support > > >>>>> for any > > >>>>> MPLS-over-IP.) > > >>>>> WH> and how many ports simultaneously would they support? WH> > > and > > >>>>> depending on implementation you don’t need to change any of the > > >>>>> TOR/vswitches. Does this mean that for some implementations you > > >>>>> may not need to change any of the TOR/vswitches, but that for > > >>>>> some others you may ? WH> any proposal on the table requires > > >>>>> changes, so for me this is not a valid discussion See above, the > > >>>>> proposal in the draft does not necessarily need changes in > > >>>>> vswitches. Let me take a practical example : while I can quite > > >>>>> easily see how to implement the procedures in > > >>>>> draft-hao-bess-inter-nvo3-vpn-optionc > > >>>>> based on current vswitch implementations of VXLAN, the lack of > > >>>>> MPLS/MPLS/(UDP, GRE) support in commonplace vswitches seems > to > > >> me as > > >>>>> making that alternate solution you suggest harder to implement. > > >>>>> WH> I would disagree to this. Tell me which switch/TOR handles > > >>>>> multiple UDP ports for VXLAN ? I mentioned _v_switches, and many > > >>>>> do support a variable destination port for VXLAN, which is > > >>>>> sufficient to implement what the draft proposes. -Thomas From: > > >>>>> Thomas Morin <[email protected] > > >>>>> > > >> > > <mailto:[email protected]><mailto:[email protected] > > >>>>> <mailto:[email protected]>>> Organization: Orange Date: > > >>>>> Friday 13 November 2015 at 09:57 To: Wim Henderickx > > >>>>> <[email protected] > > >>>>> <mailto:wim.henderickx@alcatel- > > >> lucent.com><mailto:wim.henderickx@alc > > >>>>> atel-lucent.com > > >>>>> > > >>>>> > > >> <mailto:[email protected]>>> > > >>>>> Cc: "[email protected] <mailto:[email protected]><mailto:[email protected] > > >>>>> <mailto:[email protected]>>" <[email protected] > > >>>>> <mailto:[email protected]><mailto:[email protected] > > >>>>> <mailto:[email protected]>>> Subject: Re: [bess] > > >>>>> draft-hao-bess-inter-nvo3-vpn-optionc Hi Wim, I agree on the > > >>>>> analysis that this proposal is restricted to implementations > > >>>>> that supports the chosen encap with non-IANA ports (which may be > > >>>>> hard to achieve for instance on hardware implementations, as you > > >>>>> suggest), or to context where managing multiple IPs would be > > >>>>> operationally viable. However, it does not seem obvious to me > > >>>>> how the alternative you propose [relying on 3-label option C > > >>>>> with an > > >>>>> MPLS/MPLS/(UDP|GRE) encap] addresses the issue of whether the > > >> encap > > >>>>> behavior is supported or not (e.g. your typical ToR chipset > > >>>>> possibly may not support this kind of encap, and even > > >>>>> software-based switches may not be ready to support that today). > > >>>>> My take is that having different options to adapt to various > > >>>>> implementations constraints we may have would have value. (+ one > > >>>>> question below on VXLAN...) -Thomas 2015-11-12, Henderickx, Wim > > >>>>> (Wim): On VXLAN/NVGRE, do you challenge the fact that they > would > > >>>>> be used with a dummy MAC address that would be replaced by the > > >>>>> right MAC by a sender based on an ARP request when needed ? Is > > >>>>> the above the issue you had in mind about VXLAN and NVGRE ? > WH> > > >>>>> yes I you don't mind me asking : why do you challenge that ? > > >>>>> > > >> > > >> > > > __________________________________________________________ > > >> > > > __________________________________________________________ > > >> _____ > > >> > > >> Ce message et ses pieces jointes peuvent contenir des informations > > >> confidentielles ou privilegiees et ne doivent donc pas etre > > >> diffuses, exploites ou copies sans autorisation. Si vous avez recu > > >> ce message par erreur, veuillez le signaler a l'expediteur et le > > >> detruire ainsi que les pieces jointes. Les messages electroniques > > >> etant susceptibles d'alteration, France Telecom - Orange decline > > >> toute responsabilite si ce message a ete altere, deforme ou > > >> falsifie. Merci > > >> > > >> This message and its attachments may contain confidential or > > >> privileged information that may be protected by law; they should > > >> not be distributed, used or copied without authorization. > > >> If you have received this email in error, please notify the sender > > >> and delete this message and its attachments. > > >> As emails may be altered, France Telecom - Orange shall not be > > >> liable if this message was modified, changed or falsified. > > >> Thank you. > > >> > > >> _______________________________________________ > > >> BESS mailing list > > >> [email protected] > > >> https://www.ietf.org/mailman/listinfo/bess > > >> _______________________________________________ > > >> BESS mailing list > > >> [email protected] > > >> https://www.ietf.org/mailman/listinfo/bess > > > > > > > __________________________________________________________ > > > __________________________________________________________ > > _____ > > > > Ce message et ses pieces jointes peuvent contenir des informations > > confidentielles ou privilegiees et ne doivent donc pas etre diffuses, > > exploites ou copies sans autorisation. Si vous avez recu ce message > > par erreur, veuillez le signaler a l'expediteur et le detruire ainsi > > que les pieces jointes. Les messages electroniques etant susceptibles > > d'alteration, France Telecom - Orange decline toute responsabilite si > > ce message a ete altere, deforme ou falsifie. Merci > > > > This message and its attachments may contain confidential or > > privileged information that may be protected by law; they should not > > be distributed, used or copied without authorization. > > If you have received this email in error, please notify the sender and > > delete this message and its attachments. > > As emails may be altered, France Telecom - Orange shall not be liable > > if this message was modified, changed or falsified. > > Thank you. > > _______________________________________________ > BESS mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/bess > _______________________________________________ > BESS mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/bess > _______________________________________________ > BESS mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/bess _______________________________________________ BESS mailing list [email protected] https://www.ietf.org/mailman/listinfo/bess
