Andrew Gallatin wrote:
Garrett D'Amore writes:
> Andrew Gallatin wrote:
> > Garrett D'Amore writes:
> > > Andrew Gallatin wrote:
> > > > Garrett D'Amore writes:
> > > > > The problem here is that the only reason to lower the MTU is to deal
> > > > > with cases where Path MTU discovery fails. For example, lowering the
> > > > > MTU because your upstream provider doesn't properly deal with frames
> > > > > larger than a PPP size or somesuch.
> > > > >
> > > > > Its frustrating that these cases still exist, but they do. In general,
> > > > > I agree, that lowering the MTU should not be necessary. And indeed,
> > > > > frankly nobody should need to touch the values provided by the media
> > > > > drivers when everything works properly.
> > > >
> > > > You may want to touch the values in order to reduce memory useage if
> > > > you know you cannot use jubmo frames. Since most drivers manage their
> > > > own receive buffers, this can add up. For example, my 10GbE driver,
> > > > depending on load, may allocate up to a (tunable) maximum of 4096
> > > > receive buffers. The difference between 4096 1500b and 9000b frames
> > > > is nearly 30MB.
> > > >
> > > > It would be nice if the driver could be notified that the MTU is
> > > > changing so that it can re-allocate appropriately sized receive
> > > > buffers. Every other *nix that I've worked with does this.
> > > >
> > >
> > > Okay, fair enough. :-)
> > >
> > > Btw, I am *hopeful* that one day in the future Nemo will provide buffer
> > > management on behalf of drivers. This will address some of the
> > > long-standing races with "loan-up", and free drivers from making poor
> > > decisions as to when to bcopy or use loan up. (Or maybe just allocate a
> > > new DMA or DVMA buffer....)
> >
> > Or maybe just fix the IOMMU problem..
> >
> > The main reason drivers have to do any of this loaning or bcopying
> > nonsense is because translating a kernel virtual to a DMA address on
> > IOMMU infected systems is so horribly expensive. The one (only?)
> > thing MacOSX got right in its network buffer management is that it
> > pre-enters all network buffers into the IOMMU(s), so that obtaining a
> > DMA address is a just a simple table lookup, without any hardware
> > interaction.
> >
>
> But some Sun drivers do this as well... hence dvma_reserve().
>
> The problem, as I understand it, is that even this requires buffers to
> be reused. For packets that are loaned up in the stack, there is no
> guarantee that they will be returned in a timely fashion to the driver.
> So we still wind up seeing the cost of bcopy come up from time to time.
What I'm proposing, and I may be all wet, is making allocb() do the
equivalent of dvma_reserve for all the memory it manages. This would
have the advantage of avoiding IOMMU overheads on the transmit side as
well.
Hmm... maybe a special version of allocb()? (Using it for all
allocb()'s would be a terrible idea, IMO. Because mblks are used all
over the system, and a lot of them don't ever touch hardware. E.g. for
DLPI control messages.)
It certainly bears some more consideration.
> Of course, in general, the stack does return large buffers back to
> userland ... it is most likely to "hang on" to smaller packets, which
> may be better served by a bcopy anyway.
In general, but you can always contrive a special case where
you've got a ton of non-consuming sockets with large socket
buffer sizes..
True.
-- Garrett
Drew
_______________________________________________
networking-discuss mailing list
[email protected]
