Garrett D'Amore writes: > Andrew Gallatin wrote: > > Garrett D'Amore writes: > > > Andrew Gallatin wrote: > > > > Garrett D'Amore writes: > > > > > The problem here is that the only reason to lower the MTU is to > > deal > > > > > with cases where Path MTU discovery fails. For example, lowering > > the > > > > > MTU because your upstream provider doesn't properly deal with > > frames > > > > > larger than a PPP size or somesuch. > > > > > > > > > > Its frustrating that these cases still exist, but they do. In > > general, > > > > > I agree, that lowering the MTU should not be necessary. And > > indeed, > > > > > frankly nobody should need to touch the values provided by the > > media > > > > > drivers when everything works properly. > > > > > > > > You may want to touch the values in order to reduce memory useage if > > > > you know you cannot use jubmo frames. Since most drivers manage their > > > > own receive buffers, this can add up. For example, my 10GbE driver, > > > > depending on load, may allocate up to a (tunable) maximum of 4096 > > > > receive buffers. The difference between 4096 1500b and 9000b frames > > > > is nearly 30MB. > > > > > > > > It would be nice if the driver could be notified that the MTU is > > > > changing so that it can re-allocate appropriately sized receive > > > > buffers. Every other *nix that I've worked with does this. > > > > > > > > > > Okay, fair enough. :-) > > > > > > Btw, I am *hopeful* that one day in the future Nemo will provide buffer > > > management on behalf of drivers. This will address some of the > > > long-standing races with "loan-up", and free drivers from making poor > > > decisions as to when to bcopy or use loan up. (Or maybe just allocate > > a > > > new DMA or DVMA buffer....) > > > > Or maybe just fix the IOMMU problem.. > > > > The main reason drivers have to do any of this loaning or bcopying > > nonsense is because translating a kernel virtual to a DMA address on > > IOMMU infected systems is so horribly expensive. The one (only?) > > thing MacOSX got right in its network buffer management is that it > > pre-enters all network buffers into the IOMMU(s), so that obtaining a > > DMA address is a just a simple table lookup, without any hardware > > interaction. > > > > But some Sun drivers do this as well... hence dvma_reserve(). > > The problem, as I understand it, is that even this requires buffers to > be reused. For packets that are loaned up in the stack, there is no > guarantee that they will be returned in a timely fashion to the driver. > So we still wind up seeing the cost of bcopy come up from time to time.
What I'm proposing, and I may be all wet, is making allocb() do the equivalent of dvma_reserve for all the memory it manages. This would have the advantage of avoiding IOMMU overheads on the transmit side as well. > Of course, in general, the stack does return large buffers back to > userland ... it is most likely to "hang on" to smaller packets, which > may be better served by a bcopy anyway. In general, but you can always contrive a special case where you've got a ton of non-consuming sockets with large socket buffer sizes.. Drew _______________________________________________ networking-discuss mailing list [email protected]
