Brian Xu - Sun Microsystems - Beijing China wrote:
Garrett D'Amore wrote:
Brian Xu - Sun Microsystems - Beijing China wrote:
zeeshanul huq - Sun Microsystems - Beijing China wrote:
Hi Brian,
The overhead of it is not only dma binding, but also unbinding.
If no copybuf is used, the overhead of the unbinding is quite quite
small comparing to the binding.
For small packets, even the unbinding can start to be expensive.
Lock contention becomes a concern.
And some other shortages are:
1) we have to hold the MBLKs until packet transmition complete.
With bcopy we are able to free them immediately. So when the system
are near to running out of MBLKs, bcopy works better.
I don't know when running out of MBLKs occurs. When the system is
short of kernel memory? If it is the case, then the extra bcopy also
consumes kernel memory.
Actually, IOMMU resources are a bigger issue.
With bcopy, the pre-allocated dma also occupies IOMMU entry. without
bcopy, more IOMMU entries are needed and are allocated on the fly. So
do you mean there may be not enough IOMMU entries? Please clarify.
Without bcopy, you might have to allocate more IOMMU entries. Its a
bigger problem on the rx path when you do loanup and buffer recycling
(using esballoc), but even on the tx side, if you have a packet that is
spread across multiple pages (or chained mbufs even!), then you might
need more IOMMU entries. And, usually you still have the IOMMU entries
for bcopy because you *really* want to bcopy for small packets unless
you want to have terrible small packet performance.
-- Garrett
Thanks,
Brian
2) In some driver like bge, it only has a small number of TX buffer
descriptor. With bcopy, it ensures one BD per transmit packet,
while it may require more than one with dma_bind. so using dma
bind, it will run out of Tx BD more quicker during heavy traffic.
Yes. This is reasonable.
That's part of the reasons why we use both bcopy and dma_bind in
our NIC driver. I agree we need a more faster dma binding and
unbinding solution.
What I suggested is one way to get much faster dma binding.
Of course, the original binding is also kept to meet the bcopy
requirement.
Sure. I think a lot more analysis is required here before we do any
significant changes, though.
-- Garrett
Thanks,
Brian
Regards,
Zeeshanul Huq
Garrett D'Amore wrote:
Brian Xu - Sun Microsystems - Beijing China wrote:
Hi there,
I have a question here:
Why all of the NIC drivers have to bcopy the MBLKs for transmit?
(some of them bcopy always, and some others bcopy under a
threshold of the packet length).
I think one of the reason is the overhead of the setup of dma on
the fly is greater than the overhead of bcopy for short packets.
I want to know if this is the case and if there are any other
reasons.
Yes. For any packet reasonably sized bcopy (ETHERMTU or smaller)
is faster on *all* recent hardware. (This is confirmed on even an
older 300MHz Via C3.) (Hmm... I've heard that for some Niagra
systems this might not be true, however. But I've not tested it
myself.)
I think the situation is different with jumbo frames, though.
If what I guess is the major cause, I have a proposal and I want
to hear your advice whether it makes sense.
The most time-consuming action for the dma setup is the dma bind,
more specific, calling into the VM layer to get the PFN for the
vaddr(hat_getpfnum()), since it need to search the huge page
table. While for the MBLKs, essentially which are slab objects,
the PFN has already been determined in the slab layer, and for
most of their usage, we only touch the magazine layer, where the
PFN is a pre determined one. That is, the PFN should be
considered as a constructed state, but we don't leverage it for
dma bind.
In storage, we have a field 'b_shadow' in buf(9S) to store the
pages which are recently used, through which the PFNs can be
easily got. so in
the case that b_shadow works, ddi_dma_buf_bind_handle() is much
faster than the ddi_dma_mem_bind_handle().
Another example, moving the dma bind of the HBA driver(mpt) from
Tx path to the kmem cache constrcutor, mpt driver got 26%
throughput increment. See CR6707308.
If the mblk could store the PFN info and we had a
ddi_dma_mblk_bind_handle() like interface, then I think it will
benefit the performance of the NIC drivers. I consulted the PAE,
and got a answer that the bcopy is typically about 10-15% of a
NIC TX workload.
There are things that can do to make DMA faster, better, and
simpler. In an ideal world, the GLDv3 could do most of this work,
and the mblk could just carry the ddi_dma_cookie with it.
-- Garrett
Thanks,
Brian
_______________________________________________
driver-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/driver-discuss
_______________________________________________
networking-discuss mailing list
[email protected]
_______________________________________________
networking-discuss mailing list
[email protected]
