On Mon, Jan 19, 2009 at 9:07 AM, Andrew Gallatin <[email protected]> wrote: > Min Miles Xu wrote: >> Hi Darren, >> I'm from Solaris NIC team. Some questions you raise is also what I >> concerned before. >> >> Darren Reed wrote: >>> Over time there has been an increase in the number >>> of features that NICs provide, to the point where >>> we are at now with some providing full TOEs. So far >>> there has been little in the way of knobs to turn >>> things on or off with, aside from do_hwcksum. >>> >> Currently TOEs such as HW checksum are toggled in the .conf file of > > Are we talking about stateful TOEs ("TCP Offload Engines"), which > have their own TCP/IP stack, and bypass the Solaris TCP/IP stack > entirely? Or are we talking about stateless offloads, like > hardware checksum offload, TCP Segmentation Offload (LSO), > and TCP Receive Offload (LRO)? When I think "TOE" I think "stateful" > >> drivers or /etc/system. /etc/system seems to be a more consistent >> interface since the name of TOEs in different drivers varies a lot. It's >> natural to think about a program to provide a unique interface of >> configurations such as dladm. And we already have Project Brussels. The >> actual barrier I notice is that GLD interface _m_getcapab() is not >> flexible. The driver can't update its TOEs capability interactively >> through _m_getcapab() at runtime (For e1000g LSO, it has practical >> usage). Only thing you can do after changing the configurations is to >> reboot the machine. So if we can have some changes in the stack, >> consolidating the configurations to _m_getprop(), it's better. > > Well, you can edit the driver's .conf file and rmdrv/add_drv to > re-load the driver, assuming you have a connection into the > machine independent of the NIC in question. > > But in general, I agree there should be some way to change what > stateless offloads are in use at runtime. Solaris is way behind > here. For example, the BSDs do it via ifconfig (to disable > TSO on "mxge0": ifconfig mxge0 -tso), Linux does it via the > horribly cryptic ethtool (to disable TSO on eth2: > ethtool -K eth2 tso off).
What your describing could be done very easily by brussels today I think. dladm set-linkprop -p lso=yes mylink0 Whether this is an exposed property (or kept as a hidden one i.e. 'lso' vs. '_lso') is probably work some discussion. What can't be done is the driver itself changing the state of this on its own (and still have it managed by _m_getprop()/_m_setprop()). I'm not sure this is a huge deal here though. If a particular revision of a card must keep LSO disabled for some reason, the driver can always reject requests to enable it via set-linkprop. I believe all persistent settings (done via set-linkprop) are merely applied via _m_setprop() during the boot process. > > >>> Something that might provide food for thought about >>> what sort of knobs we do need is how well those TOEs >>> play with other network components. For example, if >>> there is an interoperability problem with the TOE on >>> a NIC and some HBA, how do we manage that? How do we >>> disable use of the TOE on a card until a firmware >>> upgrade is available for it? >>> >>> And what sort of extra observability does that NIC >>> need in order to allow diagnosing problems with traffic >>> flow between it and the HBA? >>> >> If the issue can be isolated to the NIC only, we usually dump the NIC >> descriptors and the buffers with mdb to watch what's going on. But it >> requires knowledge about the specific NIC. Not sure what's the problem >> you meet. >>> If snoop and /dev/net are providing us with a view of >>> traffic that is between IP and the driver, how do we >>> get a look at what the NIC is really sending? >>> >> Well, without the help of special devices to observe the internal states >> of the NIC chip, you can still observe what the NIC is actually sending >> by monitoring the other end of the network. Sure, you should ensure that >> NIC on the other end is good in advance. > > And make sure it is not using some offloads of its own (like LRO) which > can influence what the traffic coming through snoop looks like. > > Drew > _______________________________________________ > networking-discuss mailing list > [email protected] > _______________________________________________ networking-discuss mailing list [email protected]
