Andrew Gallatin wrote: > Nitin Hande wrote: >> Andrew Gallatin wrote: >>> Hi, >>> >>> Can somebody shed some light on how crossbow hashes outgoing >>> packets to different transmit rings (not ring groups)? >>> >>> My 10GbE driver has multiple rings (and a single group). Each >>> transmit ring shares an interrupt with a corresponding receive >>> ring. We call a set of 1 TX ring, 1 RX ring, and interrupt handler >>> state a "slice". Transmit completions are handled from the interrupt >>> handler. On OSes which support multiple transmit routes, >>> we've found that ensuring that a particular connection is always >>> hashed to the same slice by the host and the NIC helps quite a bit >>> with performance (improves CPU locality, reduces cache misses, decreases >>> power consumption). >>> >>> Some OSes (like FreeBSD) allow a driver to assist in tagging a >>> connection so as to ensure that it is easy to hash >>> traffic for the same connection into the same slice in the host >>> and the NIC. Others (Linux, S10) allow the driver to hash the >>> outgoing packets to provide this locality. >>> >>> So.. Where is the transmit hashing done in crossbow? Is it tunable? >>> Is there a hook where I can do provide a hash routine (like Linux)? >>> Can I tag packets (like FreeBSD)? Is it at least something standard >>> like Toeplitz? >> >> If your driver has advertised multiple tx rings, then look for >> mac_tx_fanout_mode() which in turn computes the hash on fanout hint >> passed from ip. Providing hooks for additional hash routines has been >> suggested. > > I guess my best bet might be to lie, and say I have only one TX > ring, then fanout things myself, like I used to before Crossbow. > Is there any non-obvious disadvantage to that?
I don't know if this is non-obvious but exposing Tx rings to mac layer would help in: 1) virtualization like VNICs getting their own Tx rings. 2) Flow control when multiple Tx rings is present (presently done for UDP). If a Tx ring is blocked (out of desc), then only conn_t's sending on that Tx rings gets blocked. We want all the NICs to expose their Tx rings to the MAC layer. > When looking at this, I noticed mac_tx_serializer_mode(). Am I reading > this right, in that is serializes a single queue? That seems lacking, > compared to the nxge_serialize stuff it replaces. It is a generic solution for NICs that do not have good locking on the Tx side. mac_tx_serializer_mode() is used when you have a single Tx ring. nxge would not use that mode. It exposes multiple Tx rings. When multiple Tx rings are present, mac_tx_fanout_mode() is used. mac_tx_fanout_mode() can operate in serialized mode also in which case there would be a serializer (soft ring) for each Tx ring. Nxge uses that mode. -krgopi > > Drew > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss --
