Garrett D'Amore wrote:
Andrew Gallatin wrote:
Garrett D'Amore wrote:
Andrew Gallatin wrote:
Garrett D'Amore wrote:
*) For tx, just keep the addresses in low space for now. There
should be enough space to find a few hundred MB of VA space for
packet buffers under the 4GB limit. I don't think we need to
support gigabytes of these after all. When the pool is exhausted,
the system could internally resort to bcopy.
So we cripple performance on good NICs to help crippled NICs?
That's just so backwards. How about we only use the pool and
resort to bcopy if the NIC cannot do 64-bit DMA?
I'd be OK with that... if there was a nice way to represent it.
(Perhaps a "mblk_get_restricted_paddr()" or some such that does the
copy if the mblk needs it.)
I was thinking more along the lines of some kind of setup function
which took as an argument a ddi_dma_attr_t, so that it could know
what the driver could handle. If the driver couldn't handle the
resulting address, it would copy for the driver into something it
could handle, and maybe increment a counter so the driver author
could know what was happening.
The problem with that, is that you don't necessarily no apriori which
NIC a packet will be used for TX.
So you need to have some kind of default global pool, and then a
fallback function to remap packets that need it. (And it might not be
just due to address bits, but also alignment or segmentation concerns,
that force a driver to have to remap.)
I think we're in basic agreement on that. You have a default
global pool, which *tries* to use memory below 4GB.
If it cannot, it is no big deal, and it uses memory from
anywhere. When the driver tries to do dma, it triggers
a copy to a fallback buffer that was pre-allocated at
driver attach time.
If we use the 32-bit pool, then in the common case all NICs will get
good performance. Exhaustion of the pool should ideally be a rare
enough event that it can be slowpathed in either case, without a
significant impact on performance.
I'm scared of that kind of "nobody could ever need more than 640K"
thinking. That's what gets us stuck with weird performance landmines
years down the road. Heck, my NIC alone supports 8 TX queues per
port, with 4K elements each. Assuming everything is mapped for DMA,
that's
about 256MB right there for a dual port NIC. I can only imagine
that queues will get bigger, and then there are some NIC drivers
which don't immediately free buffers after transmit, etc.
256 MB! Ouch. Why so many elements? Do you really need this many to
get decent performance?
We increased it especially for Solaris. I can forward you the
@sun.com address of some people who asked for it, if you like. :)
I suppose 32-bit limitations could at some point become a limiting
factor, and then the pool can be converted. But that can be done on a
system wide default. Forcing the pool to require DAC right now on all
systems is not, IMO, a good idea.
I'm not suggesting that. I'm just saying it should prefer non-DAC,
but not force *everybody* to the slow path if memory below the
4GB boundary is exhausted.
Most desktop/laptop systems still don't have more than 4GB of *real*
memory.
Well, Vista is helping with that :) I just built my wife
a new FreeeBSD desktop with 6GB of ram. The memory cost me around $100.
Most people don't run Solaris with 10GbE NICs
on a desktop/laptop, and most server machines I've seen recently
have a minimum of 8GB. 64GB is not uncommon.
Drew
_______________________________________________
networking-discuss mailing list
[email protected]
