Sorry for late reply due to travel and holiday. Please see my reply inline w/ [Lucy1]
From: Pankaj Garg [mailto:[email protected]] Sent: Monday, August 26, 2013 1:50 PM To: Lucy yong; [email protected] Cc: [email protected]; [email protected] Subject: RE: comments on draft-sridharan-virtualization-nvgre-03 My comments are inline. From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Lucy yong Sent: Thursday, August 22, 2013 4:46 PM To: Pankaj Garg; [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: [nvo3] comments on draft-sridharan-virtualization-nvgre-03 Hi Pankaj, Please see my reply in-line below. Regards, Lucy From: Pankaj Garg [mailto:[email protected]] Sent: Thursday, August 22, 2013 4:18 PM To: Lucy yong; [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: RE: comments on draft-sridharan-virtualization-nvgre-03 Hi Lucy, My comments are inline. Thanks Pankaj From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Lucy yong Sent: Tuesday, August 20, 2013 2:49 PM To: [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: [nvo3] comments on draft-sridharan-virtualization-nvgre-03 Hi Authors, I read this new version and have a major concern on the ECMP support. * The new version adds ECMP support section (4.1) and require switches and routers SHOULD provide ECMP on the NVGRE packet using the outer frame fields and entire Key field (32-bit). This means that the solution not only requires the change on the endpoint devices but also the change on the intermediate switches and routers, which put the high bar for the network. Rare switches and routers today support GRE key based load balance. [PG] - This requirement has always been there as without using the key field, there is not enough entropy for ECMP in the network. There are two options for GRE based network virtualization, either use the outer frame fields and the key field or calculate entropy based on the inner packet fields. Many of the newer current generation switches and router silicon supports ECMP hashing for NVGRE packets using outer frame fields and key field. However, the older devices would not be able to do that. [Lucy] There is another option beside these two. That is described in the draft mentioned below, which has no change request on intermediate routers/switches. Many older devices are there now. I don't see the reason to replace them for supporting this feature. In addition, both described options create complex logic to existing LB process. Today switches/routers widely support 5 tuple based LB, which we should leverage instead of adding unnecessary complex to it. [Lucy1] We should promote this approach as part of NVGRE solution? * To support GRE Key based load balancing, the hardware has to check the GRE Key presence and the protocol type to avoid a mis-operation on other GRE usages, which is bad. * Using 8 bits in the Key field for flow entropy is not sufficient for all the applications. [PG] - The entropy is defined by 4 fields in NVGRE, outer source IP, outer destination IP, VSID and FlowId. The VSID is the only thing that is static for one virtual network. All other fields can change and that can provide additional entropy needed in the large networks. [Lucy] I mean that one virtual network may contain many and many flows, mapping all the flows into 8 bits flow entropy does not give a good granularity, which may potentially cause uneven load balancing issue. Hashing based load balancing is based on statistical. Without a large amount of flow entropy, it will not work well. [PG] This is incorrect. The entropy is defined by outer source IP, destination IP, VSID and FlowId. In a specific virtual network, only VSID is constant, hence the flow would be mapped based on source IP, destination IP and FlowId. Did you mean to say that between two NVE sending traffic on same VSID, all flows would map based on 8-bit entropy? If yes, then I agree, however different source IP can be used to add additional entropy in such cases if necessary. [Lucy1] Yes, I mean to say that two NVE sending traffic on same VSID, all flows would map based on 8-bit entropy. That may not provide good flow granularity for the load balance in core. You can use difference source IP addresses, but it adds the complex in many aspects and not scale. Source IP address is used as the tunnel EP identification and has some special properties. * Directly disclosing VSID to underlying network can be a security concern in some cases. [PG] - In theory, it is possible to use IPsec as the transport to protect the GRE header, however then you lose ECMP etc. I am also not sure why this is a concern since the virtual network ID is disclosed in the network in all network virtualization tunneling proposals out there, correct? [Lucy] My point is that the fields that are processed by underlying network (intermediate nodes) should not be the fields belonging to the virtual network. If we want the overlay virtual network is decoupled from the physical infrastructure, we should do that consistently. To let overlay and underlying operate on the same field on packets is bed idea, IMHO. It creates some dependency and restriction for both networks. For example, what if virtual networks span across several WAN networks, and some WAN operator want to create its own flow entropy by rehashing some fields on the packets? Your two options do not give that flexibility. IMHO, it is important to have flow entropy field on packet header for the underlying network and allow the end point to fill the flow entropy there. [PG] Yes, this is something not allowed in the spec i.e. modification of GRE key field by transit devices. [Lucy1] if we separate GRE key field and flow entropy field. Underlying operator can manipulate the flow entropy field if they want to. * The draft further suggests the interim solution for ECMP support (without any hardware upgrade). It is to assign multiple physical addresses (PA, outer address) to NVGRE end point and use policy controlling the choice of PA to be used. This makes operation complex. [PG] - There is some operational overhead, but I believe it is minimum as one can assign multiple source IP to each NVE to gain additional entropy for ECMP. The tunneling operation can select the source IP based on the hash of inner packet headers. [Lucy] I don't believe that is simple to manage. [PG] Actually it is not hard, because all you need to do is assign more than one IP to the NVE. However, it depends upon the operation model. In Hyper-V Network Virtualization, the network virtualization policy is handled by a central controller and hence it is easy to configure the NVE to provide more hashing if needed. [Lucy1] As I mentioned about, outer IP address is used as Tunnel EP identification. It is routed in underling network. Having many source IP or destination IP address for flow entropy purpose, it may add unnecessary routing burden for core networks. It may also adds complex to other functions such as OAM. We can have a better solution to support ECMP. Most switches and routers today support 5 tuple based load balance. Five tuple are IP src/dst addr, tcp|udp src/dst ports, and IP protocol type. draft-yong-tsvwg-gre-in-udp-encap-01 proposes the gre-in-udp encapsulation for GRE encapsulated protocols to be tunneled over IP networks where ECMP exists. This solution supports 16 bits flow entropy, does not require any change on intermediate switches and routers, and applies to any GRE encapsulated protocol. It also gives the ingress end point flexibility to generate the flow entropy without explicitly exposing VSID. I highly recommend NVGRE proposal to adopt this method for ECMP support. [PG] - Thanks, I would take a look at the proposal, however I treat this as another tunneling protocol in addition to three existing ones i.e. NVGRE, VxLAN and STT. What are the specific advantages of this over other protocols? Also many of current generation NIC hardware support offloads for the existing three protocols. Wouldn't using GRE-IN-UDP break those offloads? [Lucy] If NIC hardware support VxLAN which use UDP port for the flow entropy, it should be easy to support gre-in-udp encaps as well. Will be great to see chip vendor comment on this. Please take a look at the gre-in-udp encapsulation proposal (below), welcome comment on it. [PG] Sure, I would take a look at it. [Lucy1] Have you look at it? It is a simple draft. Lucy Regards, Lucy Here is the draft. The TSVWG will adopt it as WG draft soon. http://datatracker.ietf.org/doc/draft-yong-tsvwg-gre-in-udp-encap/ Regards, Lucy
_______________________________________________ nvo3 mailing list [email protected] https://www.ietf.org/mailman/listinfo/nvo3
