IMHO, very larger DC doesn't mean the subnets there must be very large in the number of hosts within each subnet. Take the Iaas-based public cloud data center as an example, most tenants are SMEs which usually don't require too many VMs and those VMs of each tenant could be located within a single subnet for simplicity. Hence most subnets there would not be very large.
Best regards, Xiaohu > -----邮件原件----- > 发件人: [email protected] [mailto:[email protected]] 代表 Aldrin > Isaac > 发送时间: 2012年7月11日 11:43 > 收件人: Stiliadis, Dimitrios (Dimitri) > 抄送: [email protected]; [email protected]; [email protected]; Linda > Dunbar; Pedro Roque Marques; Joel M. Halpern > 主题: Re: [nvo3] inter-CUG traffic [was Re: call for adoption: > draft-narten-nvo3-overlay-problem-statement-02] > > How are these very large DCs with one large flat L2 subnet dealing with the > issue of broadcast/arp today? > > > On Jul 10, 2012, at 2:39 PM, Stiliadis, Dimitrios (Dimitri) wrote: > > > \> > >> I thought that we are discussing if it is necessary for NVo3 to consider > >> cross > >> subnet communications. For the example you give where all applications in > the > >> DC are all in one subnet and those applications don't communicate with > peers > >> in other subnets, I would think they are pretty small DC, which is not > >> really > >> what NVo3 is targeted for, is it? > >> > > > > On the contrary. Lots of very large DCs are there for such applications. > > > > (most Big Data related data centers are like that). > > > > Dimitri > > > > > >> Linda > >> > >>> -----Original Message----- > >>> From: Stiliadis, Dimitrios (Dimitri) [mailto:dimitri.stiliadis@alcatel- > >>> lucent.com] > >>> Sent: Monday, July 09, 2012 5:41 PM > >>> To: Linda Dunbar; Joel M. Halpern > >>> Cc: Pedro Roque Marques; [email protected]; [email protected]; > >>> [email protected] > >>> Subject: RE: [nvo3] inter-CUG traffic [was Re: call for adoption: > >>> draft-narten-nvo3-overlay-problem-statement-02] > >>> > >>> There is no DMZ in such applications. They don't reside on the DMZ. > >>> > >>> Applications maintain Tbytes of data in a hadoop cluster and running > >>> mapreduce > >>> on them. There is a single application and single tenant on this whole > >>> cluster. > >>> mapreduce nodes will send lots of data between the map and reduce > >>> phase > >>> and there is no need for any L3 separation in these cases. > >>> > >>> This is just one example. We can't possibly make the assumption that > >>> any > >>> app in the DC is related to a web-service on the DMZ. > >>> > >>> > >>> Dimitri > >>> > >>> > >>> > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: Linda Dunbar [mailto:[email protected]] > >>>> Sent: Monday, July 09, 2012 2:59 PM > >>>> To: Stiliadis, Dimitrios (Dimitri); Joel M. Halpern > >>>> Cc: Pedro Roque Marques; [email protected]; [email protected]; > >>> [email protected] > >>>> Subject: RE: [nvo3] inter-CUG traffic [was Re: call for adoption: > >>> draft- > >>>> narten-nvo3-overlay-problem-statement-02] > >>>> > >>>> Dimitrios, > >>>> > >>>> Do you mean that all applications in "a large hddop/mapredure > >>> cluster" are in > >>>> one subnet? > >>>> And the applications in "one cluster" never communicate with another > >>>> "cluster" in the same data center? Really? > >>>> I would think applications in DMZ would be at least put in a > >>> different subnet. > >>>> > >>>> Linda > >>>> > >>>>> -----Original Message----- > >>>>> From: Stiliadis, Dimitrios (Dimitri) > >>> [mailto:dimitri.stiliadis@alcatel- > >>>>> lucent.com] > >>>>>> What kind of DCs which only have intra subnet communication? > >>>>> > >>>>> A large hadoop/mapreduce cluster will not bother about different > >>>>> subnets, > >>>>> and the traffic requirements can be huge. Thousands of enterprise > >>>>> applications > >>>>> will also do the same, since they assume they are in a protected > >>>>> environment. > >>>>> > >>>>> I agree with Joel's earlier statement : > >>>>> > >>>>>> I have seen descriptions of data centers for which it is almost > >>>>> "all", > >>>>>> and other data centers for which it is negligible. > >>>>> > >>>>> Dimitri > >>>>> > >>>>> > >>>>>> > >>>>>> Linda > >>>>>>> -----Original Message----- > >>>>>>> From: Joel M. Halpern [mailto:[email protected]] > >>>>>>> Sent: Monday, July 09, 2012 3:00 PM > >>>>>>> To: Linda Dunbar > >>>>>>> Cc: Pedro Roque Marques; [email protected]; > [email protected]; > >>>>>>> [email protected] > >>>>>>> Subject: Re: [nvo3] inter-CUG traffic [was Re: call for > >>> adoption: > >>>>>>> draft-narten-nvo3-overlay-problem-statement-02] > >>>>>>> > >>>>>>> I have not seen data to suggest "most". Do you have a source > >>> for > >>>>> that? > >>>>>>> I have seen descriptions of data centers for which it is > >>> almost > >>>>> "all", > >>>>>>> and other data centers for which it is negligible. > >>>>>>> Yours, > >>>>>>> Joel > >>>>>>> > >>>>>>> On 7/9/2012 3:51 PM, Linda Dunbar wrote: > >>>>>>>> Joel, > >>>>>>>> > >>>>>>>> I agree with you that VPN related WG (e.g. L2VPN, L3VPN) may > >>> not > >>>>> need > >>>>>>> to address too much on across VN communications. > >>>>>>>> > >>>>>>>> But most traffic in Data Center are across subnets. Given > >>> NVo3 is > >>>>> for > >>>>>>> identifying issues associated with data centers, I think that > >>> cross > >>>>>>> Subnet traffic should not be ignored. > >>>>>>>> > >>>>>>>> Linda > >>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: [email protected] [mailto:[email protected]] > >>> On > >>>>> Behalf > >>>>>>> Of > >>>>>>>>> Joel M. Halpern > >>>>>>>>> Sent: Friday, June 29, 2012 9:32 PM > >>>>>>>>> To: Pedro Roque Marques > >>>>>>>>> Cc: [email protected]; [email protected]; [email protected] > >>>>>>>>> Subject: Re: [nvo3] inter-CUG traffic [was Re: call for > >>> adoption: > >>>>>>>>> draft-narten-nvo3-overlay-problem-statement-02] > >>>>>>>>> > >>>>>>>>> I would not be so bold as to insist that all deployments can > >>>>> safely > >>>>>>>>> ignore inter-VPN intra-data-center traffic. But there are > >>> MANY > >>>>>>> cases > >>>>>>>>> where that is not an important part of the traffic mix. > >>>>>>>>> So I was urging that we not mandate optimal inter-subnet > >>> routing > >>>>> as > >>>>>>>>> part > >>>>>>>>> of the NVO3 requirements. > >>>>>>>>> I would not want to prohibit it either, as there are > >>> definitely > >>>>>>> cases > >>>>>>>>> where it matters, some along the lines you alude to. > >>>>>>>>> > >>>>>>>>> Yours, > >>>>>>>>> Joel > >>>>>>>>> > >>>>>>>>> On 6/29/2012 9:40 PM, Pedro Roque Marques wrote: > >>>>>>>>>> Joel, > >>>>>>>>>> A very common model currently is to have a 3 tier app where > >>>>> each > >>>>>>> tier > >>>>>>>>> is in its VLAN. You will find that web-servers for instance > >>>>> don't > >>>>>>>>> actually talk much to each other... although they are on the > >>>>> same > >>>>>>> VLAN > >>>>>>>>> 100% of their traffic goes outside VLAN. Very similar story > >>>>> applies > >>>>>>> to > >>>>>>>>> app logic tier. The database tier may have some replication > >>>>> traffic > >>>>>>>>> within its VLAN but hopefully than is less than the requests > >>>>> that it > >>>>>>>>> serves. > >>>>>>>>>> > >>>>>>>>>> There isn't a whole lot of intra-CUG/subnet traffic under > >>> that > >>>>>>>>> deployment model. A problem statement that assumes > >>> (implicitly) > >>>>> that > >>>>>>>>> most or a significant part of the traffic stays local to a > >>>>>>>>> VLAN/subnet/CUG is not a good match for the common 3-tier > >>>>>>> application > >>>>>>>>> model. Even if you assume that web and app tiers use a > >>>>>>> VLAN/subnet/CUG > >>>>>>>>> per tenant (which really is an application in enterprise) > >>> the > >>>>>>> database > >>>>>>>>> is typically common for a large number of apps/tenants. > >>>>>>>>>> > >>>>>>>>>> Pedro. > >>>>>>>>>> > >>>>>>>>>> On Jun 29, 2012, at 5:26 PM, Joel M. Halpern wrote: > >>>>>>>>>> > >>>>>>>>>>> Depending upon what portion of the traffic needs inter- > >>> region > >>>>>>>>> handling (inter vpn, inter-vlan, ...) it is not obvious that > >>>>>>> "optimal" > >>>>>>>>> is an important goal. As a general rule, perfect is the > >>> enemy > >>>>> of > >>>>>>> good. > >>>>>>>>>>> > >>>>>>>>>>> Yours, > >>>>>>>>>>> Joel > >>>>>>>>>>> > >>>>>>>>>>> On 6/29/2012 7:54 PM, [email protected] wrote: > >>>>>>>>>>>> Pedro, > >>>>>>>>>>>> > >>>>>>>>>>>>> Can you please describe an example of how you could set > >>> up > >>>>> such > >>>>>>>>>>>>> straightforward routing, assuming two Hosts belong to > >>>>> different > >>>>>>>>> "CUGs" such > >>>>>>>>>>>>> that these can be randomly spread across the DC ? My > >>>>> question > >>>>>>> is > >>>>>>>>> where is the > >>>>>>>>>>>>> "gateway", how is it provisioned and how can traffic > >>> paths > >>>>> be > >>>>>>>>> guaranteed to > >>>>>>>>>>>>> be optimal. > >>>>>>>>>>>> > >>>>>>>>>>>> Ok, I see your point - the routing functionality is > >>>>>>> straightforward > >>>>>>>>> to move over, > >>>>>>>>>>>> but ensuring optimal pathing is significantly more work, > >>> as > >>>>> noted > >>>>>>>>> in another one > >>>>>>>>>>>> of your messages: > >>>>>>>>>>>> > >>>>>>>>>>>>> Conceptually, that means that the functionality of the > >>>>> "gateway" > >>>>>>>>> should be > >>>>>>>>>>>>> implemented at the overlay ingress and egress points, > >>> rather > >>>>>>> than > >>>>>>>>> requiring > >>>>>>>>>>>>> a mid-box. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> --David > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>>> From: Pedro Roque Marques > >>> [mailto:[email protected]] > >>>>>>>>>>>>> Sent: Friday, June 29, 2012 7:38 PM > >>>>>>>>>>>>> To: Black, David > >>>>>>>>>>>>> Cc: [email protected]; [email protected] > >>>>>>>>>>>>> Subject: Re: [nvo3] inter-CUG traffic [was Re: call for > >>>>> adoption: > >>>>>>>>> draft- > >>>>>>>>>>>>> narten-nvo3-overlay-problem-statement-02] > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Jun 29, 2012, at 4:02 PM, <[email protected]> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>>> There is an underlying assumption in NVO3 that > >>> isolating > >>>>>>> tenants > >>>>>>>>> from > >>>>>>>>>>>>>>> each other is a key reason to use overlays. If 90% of > >>> the > >>>>>>>>> traffic is > >>>>>>>>>>>>>>> actually between different tenants, it is not > >>> immediately > >>>>>>> clear > >>>>>>>>> to me > >>>>>>>>>>>>>>> why one has set up a system with a lot of "inter > >>> tenant" > >>>>>>> traffic. > >>>>>>>>> Is > >>>>>>>>>>>>>>> this is a case we need to focus on optimizing? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> A single tenant may have multiple virtual networks with > >>>>> routing > >>>>>>>>> used to > >>>>>>>>>>>>>> provide/control access among them. The crucial thing > >>> is to > >>>>>>> avoid > >>>>>>>>> assuming > >>>>>>>>>>>>>> that a tenant or other administrative entity has a > >>> single > >>>>>>> virtual > >>>>>>>>> network > >>>>>>>>>>>>>> (or CUG in Pedro's email). For example, consider > >>> moving a > >>>>>>>>> portion of > >>>>>>>>>>>>>> a single data center that uses multiple VLANs and > >>> routers > >>>>> to > >>>>>>>>> selectively > >>>>>>>>>>>>>> connect them into an nvo3 environment - each VLAN gets > >>>>> turned > >>>>>>>>> into a virtual > >>>>>>>>>>>>>> network, and the routers now route among virtual > >>> networks > >>>>>>> instead > >>>>>>>>> of VLANs. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> One of the things that's been pointed out to me in > >>> private > >>>>> is > >>>>>>>>> that the level > >>>>>>>>>>>>>> of importance that one places on routing across virtual > >>>>>>> networks > >>>>>>>>> may depend > >>>>>>>>>>>>>> on one's background. If one is familiar with VLANs and > >>>>> views > >>>>>>>>> nvo3 overlays > >>>>>>>>>>>>>> providing VLAN-like functionality, IP routing among > >>> virtual > >>>>>>>>> networks is a > >>>>>>>>>>>>>> straightforward application of IP routing among VLANs > >>> (e.g., > >>>>>>> the > >>>>>>>>> previous > >>>>>>>>>>>>>> mention of L2/L3 IRB functionality that is common in > >>> data > >>>>>>> center > >>>>>>>>> network > >>>>>>>>>>>>>> switches). > >>>>>>>>>>>>> > >>>>>>>>>>>>> Can you please describe an example of how you could set > >>> up > >>>>> such > >>>>>>>>>>>>> straightforward routing, assuming two Hosts belong to > >>>>> different > >>>>>>>>> "CUGs" such > >>>>>>>>>>>>> that these can be randomly spread across the DC ? My > >>>>> question > >>>>>>> is > >>>>>>>>> where is the > >>>>>>>>>>>>> "gateway", how is it provisioned and how can traffic > >>> paths > >>>>> be > >>>>>>>>> guaranteed to > >>>>>>>>>>>>> be optimal. > >>>>>>>>>>>>> > >>>>>>>>>>>>>> OTOH, if one is familiar with VPNs where access > >>> among > >>>>>>>>>>>>>> otherwise-closed groups has to be explicitly configured, > >>>>>>>>> particularly > >>>>>>>>>>>>>> L3 VPNs where one cannot look to L2 to help with > >>> grouping > >>>>> the > >>>>>>> end > >>>>>>>>> systems, > >>>>>>>>>>>>>> this sort of cross-group access can be a significant > >>> area > >>>>> of > >>>>>>>>> functionality. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Considering that in a VPN one can achieve inter-CUG > >>> traffic > >>>>>>>>> exchange without > >>>>>>>>>>>>> an gateway in the middle via policy, it is unclear why > >>> you > >>>>>>> suggest > >>>>>>>>> that "look > >>>>>>>>>>>>> to L2" would help. > >>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>> --David > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>>>>> From: [email protected] [mailto:nvo3- > >>> [email protected]] > >>>>> On > >>>>>>>>> Behalf Of > >>>>>>>>>>>>> Thomas > >>>>>>>>>>>>>>> Narten > >>>>>>>>>>>>>>> Sent: Friday, June 29, 2012 5:56 PM > >>>>>>>>>>>>>>> To: Pedro Roque Marques > >>>>>>>>>>>>>>> Cc: [email protected] > >>>>>>>>>>>>>>> Subject: [nvo3] inter-CUG traffic [was Re: call for > >>>>> adoption: > >>>>>>>>> draft-narten- > >>>>>>>>>>>>>>> nvo3-overlay-problem-statement-02] > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Pedro Roque Marques <[email protected]> > writes: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I object to the document on the following points: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 3) Does not discuss the requirements for inter-CUG > >>>>> traffic. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Given that the problem statement is not supposed to be > >>> the > >>>>>>>>>>>>>>> requirements document,, what exactly should the > >>> problem > >>>>>>>>> statement say > >>>>>>>>>>>>>>> about this topic? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> <[email protected]> writes: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Inter-VN traffic (what you refer to as inter-CUG > >>> traffic) > >>>>> is > >>>>>>>>> handled > >>>>>>>>>>>>>>>> by a straightforward application of IP routing to the > >>>>> inner > >>>>>>> IP > >>>>>>>>>>>>>>>> headers; this is similar to the well-understood > >>>>> application > >>>>>>> of > >>>>>>>>> IP > >>>>>>>>>>>>>>>> routing to forward traffic across VLANs. We should > >>> talk > >>>>>>> about > >>>>>>>>> VRFs > >>>>>>>>>>>>>>>> as something other than a limitation of current > >>>>> approaches - > >>>>>>>>> for > >>>>>>>>>>>>>>>> VLANs, VRFs (separate instances of routing) are > >>>>> definitely a > >>>>>>>>>>>>>>>> feature, and I expect this to carry forward to nvo3 > >>> VNs. > >>>>> In > >>>>>>>>>>>>>>>> addition, we need to make changes to address > >>> Dimitri's > >>>>>>> comments > >>>>>>>>>>>>>>>> about problems with the current VRF text. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Pedro Roque Marques <[email protected]> > writes: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> That is where again the differences between different > >>>>> types > >>>>>>> of > >>>>>>>>>>>>>>>> data-centers do play in. If for instance 90% of a VMs > >>>>> traffic > >>>>>>>>>>>>>>>> happens to be between the Host OS and a network > >>> attached > >>>>>>>>> storage > >>>>>>>>>>>>>>>> file system run as-a-Service (with the appropriate > >>> multi- > >>>>>>> tenent > >>>>>>>>>>>>>>>> support) then the question of where are the routers > >>>>> becomes a > >>>>>>>>> very > >>>>>>>>>>>>>>>> important issue. In a large scale data-center where > >>> the > >>>>> Host > >>>>>>> VM > >>>>>>>>> and > >>>>>>>>>>>>>>>> the CPU that hosts the filesystem block can be > >>> randomly > >>>>>>> spread > >>>>>>>>>>>>>>>> where is the router ? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Where is what router? Are you assuming the Host OS and > >>> NAS > >>>>> are > >>>>>>>>> in the > >>>>>>>>>>>>>>> different VNs? And hence, traffic has to (at least > >>>>>>> conceptually) > >>>>>>>>> exit > >>>>>>>>>>>>>>> one VN and reenter another whenever there is HostOS - > >>> NAS > >>>>>>>>> traffic? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Is every switch a router ? Does it have all the CUGs > >>>>> present ? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> The underlay can be a mixture of switches and > >>> routers... > >>>>> that > >>>>>>> is > >>>>>>>>> not > >>>>>>>>>>>>>>> our concern. So long as the underlay delivers traffic > >>>>> sourced > >>>>>>> by > >>>>>>>>> an > >>>>>>>>>>>>>>> ingress NVE to the appropriate egress NVE, we are good. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> If there are issues with the actual path taken being > >>>>>>> suboptimal > >>>>>>>>> in > >>>>>>>>>>>>>>> some sense, that is an underlay problem to solve, not > >>> for > >>>>> the > >>>>>>>>> overlay. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> In some DC designs the problem to solve is the inter- > >>> CUG > >>>>>>>>>>>>>>>> traffic. With L2 headers being totally irrelevant. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> There is an underlying assumption in NVO3 that > >>> isolating > >>>>>>> tenants > >>>>>>>>> from > >>>>>>>>>>>>>>> each other is a key reason to use overlays. If 90% of > >>> the > >>>>>>>>> traffic is > >>>>>>>>>>>>>>> actually between different tenants, it is not > >>> immediately > >>>>>>> clear > >>>>>>>>> to me > >>>>>>>>>>>>>>> why one has set up a system with a lot of "inter > >>> tenant" > >>>>>>> traffic. > >>>>>>>>> Is > >>>>>>>>>>>>>>> this is a case we need to focus on optimizing? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> But in any case, if one does have inter-VN traffic, > >>> that > >>>>> will > >>>>>>>>> have to > >>>>>>>>>>>>>>> get funneled through a "gateway" between VNs, at least > >>>>>>>>> conceptually, I > >>>>>>>>>>>>>>> would assume that an implementation of overlays would > >>>>> provide > >>>>>>> at > >>>>>>>>> least > >>>>>>>>>>>>>>> one, and likely more such gateways on each VN. How > >>> many > >>>>> and > >>>>>>>>> where to > >>>>>>>>>>>>>>> place them will presumably depend on many factors but > >>>>> would be > >>>>>>>>> done > >>>>>>>>>>>>>>> based on traffic patterns and network layout. I would > >>> not > >>>>>>> think > >>>>>>>>> every > >>>>>>>>>>>>>>> NVE has to provide such functionality. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> What do you propose needs saying in the problem > >>> statement > >>>>>>> about > >>>>>>>>> that? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thomas > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>>>>> nvo3 mailing list > >>>>>>>>>>>>>>> [email protected] > >>>>>>>>>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3 > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>> nvo3 mailing list > >>>>>>>>>>>> [email protected] > >>>>>>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3 > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> nvo3 mailing list > >>>>>>>>> [email protected] > >>>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3 > >>>>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> nvo3 mailing list > >>>>>> [email protected] > >>>>>> https://www.ietf.org/mailman/listinfo/nvo3 > > _______________________________________________ > > nvo3 mailing list > > [email protected] > > https://www.ietf.org/mailman/listinfo/nvo3 > > _______________________________________________ > nvo3 mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/nvo3 _______________________________________________ nvo3 mailing list [email protected] https://www.ietf.org/mailman/listinfo/nvo3
