Thomas has made some very valid points, and I would like to add the following observations: 1) 16,000: that's the number of remote endpoint entries (across all tenants) supported in today's top-of-the-line ASICs which can do VXLAN/NVGRE.You don't have room in there to store all the endpoints from all tenants. The asic vendors could make room, but that'd be a more expensive product.
2) The provider doesn't make any money from the networking software running on the server. To the extent that the networking software consumes resources, you have that much fewer resources available for the VMs (which is where the provider makes money). Further, unlike the VMs which can be moved around when they consume too many resources, the switch software is a permanent resident. In addition, switches implemented in the hypervisor kernel have to contend with a limited stack size (typically 4 or 8kb), and virtual memory constraints. You don't want the standard to force switch implementations to be in a VM, away from the direct data path in the kernel. Again, I'm not advocating doing away with the push model. I'm just saying that you need to build in support for pull. -- Sunny From: [email protected] To: Kireeti Kompella <[email protected]>, Cc: "[email protected]" <[email protected]> Date: 09/26/2012 11:05 AM Subject: Re: [nvo3] Push or pull? Sent by: [email protected] Hi Kireeti. Kireeti Kompella <[email protected]> writes: > I'm glad you brought this up. Actually, this conversation has > happened several times, to my knowledge, without a firm > conclusion. I doubt we can close it, but at least, let's air it. > Push: send route updates to everyone (first see Aldrin's comment about > RT Constraint) as soon as you (the AUTHORITY/ORACLE) get them. > Pull: sit on updates you get until someone asks for them. > I could try to convince you what a terrible idea Pull is. I could > refer to the Internet, which is all Push, and scales reasonably > well. You mean like DNS or ARP? I do not think we should say "push is good, pull is bad". That is just too categorical a statement. > I could ask you what happens to packets while the Pull is being > responded to, or a bunch of related questions. I won't. They get queued. Or dropped. Or possibly something else. Yes, there are implications to that. But not necessarily a show stopper either. > > In my view, this puts an unnecessary load on NVEs. > Let's talk instead about the "unnecessary load". Can someone quantify > this? Is it CPU? memory? messaging? What's the bottleneck or pain > point? Some or all of the above. If typical VNs are smallish, I agree that an NVE can preload full tables with no problem. But what about for very large VNs? Should the architecture *force* such preloading of full tables, even if the working set of routes is actually very small? And what about for very large VNs where there is a lot of VM mobility? Should all NVEs be required to get update info even for destinations they don't care about? > Here's my back-of-the-envelop calculation for memory, normalized to a > VM. Let's say a VM has 10,000 friends in the DC that it might possibly > want to talk to, but only one that it really wants to talk to. Let's > say that a FIB route entry takes 100 bytes. That adds up to a possible > total of 1MB vs. an actual of 100 bytes. Is 1MB really something one > should optimize, especially considering that the VM has probably been > allocated 4GB? Are you really arguing that the difference between 1MB and 100 bytes is just noise? And who says this is in conventional memory on a host? I could see this being done in the ASIC... > Maybe there is a dimension to this that really is an issue. I would > love to know, especially with numbers backing it up. But let's first > convince ourselves that this is a problem worth solving before > spending cycles solving it. I do not think we should today require that the NVO3 architecture (in a MUST sense) support only push. I think we should allow for either push or pull, or some combination. I can see benefits with both approaches. Note also that we may be looking at the problem from different perspectives. For example, in a single data center, I can imagine a centralized directory service holding the complete address mapping information for all the VNs in the DC. An NVE in such cases can query such a mapping system with very very low latency. Thomas _______________________________________________ nvo3 mailing list [email protected] https://www.ietf.org/mailman/listinfo/nvo3
_______________________________________________ nvo3 mailing list [email protected] https://www.ietf.org/mailman/listinfo/nvo3
