Hi Linda, Thank you for adding this section to the draft. Some comments in line...
> > L3 Address Migration > > When the attachment to NVE is L3 based, TS migration can cause one > subnetwork to be scatted among many NVEs, or fragmented addresses. > > The outbound traffic of fragmented L3 addresses doesn’t have the same issue > as L2 address migration, but the inbound traffic has the same issues as L2 > address migration (Section 6). In theory, host hosting at DCBR can achieve > the optimal path forwarding in very fragmented network. But host routing can > be challenging in a very large and highly virtualized data center, there > could be hundreds of thousands of hosts/VMs, sometimes in millions, due to > business demand and highly advanced server virtualization technologies. > > Optimal routing of TS's inbound traffic. This means that as a given TS moves > from one server to another, the (inbound) traffic originated outside of the > TS's directly attached NVE, and destined to that TS be routed optimally to > the NVE to which the server presently hosting that TS, without first > traversing some other NVEs. This is also known as avoiding "triangular > routing". > Security should be considered here also. Security is strongest when applied end-to-end. For example, if we are encrypting at the ingress NVE, decryption should happen at the final NVE-- if the NVEs are on host then this implies packets are encrypted during all of transit. Also, when encrypting this may be over the full inner packet so that virtual addresses are not visible to intermediate devices. With ESP/GUE the header stack might be IP/UDP/GUE/ESP/IP; vnid is visible, but virtual addresses aren't. Even if we are just using a security cookie to validate a VNID, the outer IP addresses can be considered for anti-spoofing. Triangular routing convolutes both of these cases I believe. > In theory, host hosting by every NVE (including the DCBR) can achieve the > optimal path forwarding in very fragmented network. But host routing can be > challenging in a very large and highly virtualized data center, there could > be hundreds of thousands of hosts/VMs, sometimes in millions, due to > business demand and highly advanced server virtualization technologies. > This is true, but there are some mitigating factors favoring host routes. 1) a particular NVE should only be communicating with a subset of virtual addresses at a given time. 2) Migrations are fairly rare events and latency to adapt is already assumed to be in 10's of msecs. An extra round trip or so to relearn a host route after migration is not unreasonable. So the host routes could be in a cache containing the working set where there is a mechanism to resolve them. > ECMP can be used by the DCBR or any NVEs that don’t support host routing or > can’t access NVA to distribute traffic equally to any of the NVEs that > support the subnet (VN). If an NVE doesn’t have the destination of a data > packet directly attached, it can query NVA for the target NVE to which the > destination is attached, and encapsulate the packet with the target NVE as > outer destination before sending it out. > > Another approach is to designate one or two NVEs as designated forwarder for > a specific subnet when the subnet is spread across many NVEs. For example, > if high percentage of TSs of one subnet is attached to NVE “X”, the > remaining small percentage of the subnet is spread around many NVEs. > Designating NVE “X” as the designated forwarder for the subnet can greatly > reduce the “triangular routing” for the traffic destined to TSs in this > subnet. > I'm not sure this is practical. While there are reasons to schedule VMs with physical locality in the DC, there are also reasons we wouldn't do this (like we don't want a single device failure to be able to take out all VM's of a customer). Also, I don't think we want to constrain the job scheduler any more than it already is-- it's likely over enough time that entropy will prevail so that VMs for particular subnets are randomly distributed across the DC. If we designate NVE "X" as a forwarder like you suggest, then when it gets a packet for which there is a better route it could send the equivalent of an ICMP redirect to back to the originator to eliminate the triangular routing. Furthermore, if instead of acting as a forwarder, NVE "X" is a resolver then a resolution protocol could be implemented (ARP model) so that packets might never forwarded using triangular routing which could address the end to end security issue I posed above. Thanks, Tom > > > Linda Dunbar > > _______________________________________________ nvo3 mailing list [email protected] https://www.ietf.org/mailman/listinfo/nvo3
