Jeff, I agree that supporting LAG/ECMP at the NIC->host interface is the issue for both VXLAN/NVGRE. However if NVGRE uses gre-in-udp, it will make future NIC enhancement for this capability much simpler since both encapsulations use udp src port for the flow entropy that is used by LB. Why do you think that inner payload inspection is necessary for ECMP LB?
Thank, Lucy -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Jeff Wheeler Sent: Sunday, September 22, 2013 1:13 PM Cc: [email protected] Subject: [nvo3] LAG/ECMP load-balancing problems facing overlay networks On Sat, Sep 21, 2013 at 11:57 AM, Lucy yong <[email protected]> wrote: > * Requirement: For performance reasons, multipath over LAG and ECMP > paths SHOULD be supported. > > VXLAN supports common five tuple based LB. NVGRE requests LB to use > GRE header, which is not commonly supported by underlying IP network. I'd like to note that it is becoming increasingly common for the underlying L3 or L2 network to support load-balancing based on GRE inner-header fields. In fact, this can be largely relied on for high-bandwidth GRE tunnels over the public Internet (!) today, and within many datacenter networks. Where this is not so reliable is in the NIC->Host interface. This is an issue facing NVGRE, and in some circumstance, VXLAN also. For example, the most popular 10GE server NIC chipsets will deliver all NVGRE packets to a host DMA ring buffer based on the GRE outer-header only. One example is the Intel 82599, which everyone must be familiar with to have an informed opinion on vRouter-related topics. Even if the NIC is configured with several DMA rings, and the vRouter is able to use different CPU cores to service those rings for distribution of the work, all NVGRE traffic (having same outer-header) will arrive at only one DMA ring, and other CPU cores may not be utilized. If vRouters are used to support network-heavy applications, NIC vendors must be encouraged to support additional header inspection which may then be used for load-balancing across host DMA rings. One would hope same vendors will eventually implement hardware offload of VXLAN/NVGRE entirely, and these two requirements share a common underlying need for the NIC to understand the VXLAN/NVGRE encapsulation. In other words, if a NIC vendor plans to support deeper header inspection for load-balancing, they have already done some of the work needed to support hardware offload. This list is very focused on procedural items and support in the networks; but nearly zero attention is paid to NIC->Host interface issues. For overlay technologies to deliver acceptable performance for network-heavy workloads, NIC->Host interface must be more intelligent than it is today. Next-generation NICs must implement new capabilities. It is therefore useful to consider NICs in any discussion of load-balancing across the datacenter network, if for no other reason than to foster greater understanding of this challenge. -- Jeff S Wheeler <[email protected]> Sr Network Operator / Innovative Network Concepts _______________________________________________ nvo3 mailing list [email protected] https://www.ietf.org/mailman/listinfo/nvo3 _______________________________________________ nvo3 mailing list [email protected] https://www.ietf.org/mailman/listinfo/nvo3
