Hi Brian,
The overhead of it is not only dma binding, but also unbinding.
And some other shortages are:
1) we have to hold the MBLKs until packet transmition complete. With
bcopy we are able to free them immediately. So when the system are near
to running out of MBLKs, bcopy works better.
2) In some driver like bge, it only has a small number of TX buffer
descriptor. With bcopy, it ensures one BD per transmit packet, while it
may require more than one with dma_bind. so using dma bind, it will run
out of Tx BD more quicker during heavy traffic.
That's part of the reasons why we use both bcopy and dma_bind in our NIC
driver. I agree we need a more faster dma binding and unbinding solution.
Regards,
Zeeshanul Huq
Garrett D'Amore wrote:
Brian Xu - Sun Microsystems - Beijing China wrote:
Hi there,
I have a question here:
Why all of the NIC drivers have to bcopy the MBLKs for transmit?
(some of them bcopy always, and some others bcopy under a threshold
of the packet length).
I think one of the reason is the overhead of the setup of dma on the
fly is greater than the overhead of bcopy for short packets. I want
to know if this is the case and if there are any other reasons.
Yes. For any packet reasonably sized bcopy (ETHERMTU or smaller) is
faster on *all* recent hardware. (This is confirmed on even an older
300MHz Via C3.) (Hmm... I've heard that for some Niagra systems this
might not be true, however. But I've not tested it myself.)
I think the situation is different with jumbo frames, though.
If what I guess is the major cause, I have a proposal and I want to
hear your advice whether it makes sense.
The most time-consuming action for the dma setup is the dma bind,
more specific, calling into the VM layer to get the PFN for the
vaddr(hat_getpfnum()), since it need to search the huge page table.
While for the MBLKs, essentially which are slab objects, the PFN has
already been determined in the slab layer, and for most of their
usage, we only touch the magazine layer, where the PFN is a pre
determined one. That is, the PFN should be considered as a
constructed state, but we don't leverage it for dma bind.
In storage, we have a field 'b_shadow' in buf(9S) to store the pages
which are recently used, through which the PFNs can be easily got. so in
the case that b_shadow works, ddi_dma_buf_bind_handle() is much
faster than the ddi_dma_mem_bind_handle().
Another example, moving the dma bind of the HBA driver(mpt) from Tx
path to the kmem cache constrcutor, mpt driver got 26% throughput
increment. See CR6707308.
If the mblk could store the PFN info and we had a
ddi_dma_mblk_bind_handle() like interface, then I think it will
benefit the performance of the NIC drivers. I consulted the PAE, and
got a answer that the bcopy is typically about 10-15% of a NIC TX
workload.
There are things that can do to make DMA faster, better, and simpler.
In an ideal world, the GLDv3 could do most of this work, and the mblk
could just carry the ddi_dma_cookie with it.
-- Garrett
Thanks,
Brian
_______________________________________________
driver-discuss mailing list
driver-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/driver-discuss
_______________________________________________
networking-discuss mailing list
networking-disc...@opensolaris.org
_______________________________________________
driver-discuss mailing list
driver-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/driver-discuss
