On May 6, 2009, at 5:49 AM, Andrew Gallatin wrote: > Nicolas Droux wrote: >> [Bcc'ed driver-discuss at opensolaris.org and networking-discuss at >> opensolaris.org >> ] >> I am pleased to announce the availability of the first revision of >> the "Crossbow APIs for Device Drivers" document, available at the >> following location: > > I recently ported a 10GbE driver to Crossbow. My driver currently > has a single ring-group, and a configurable number of rings. The > NIC hashes received traffic to the rings in hardware. > > > I'm having a strange issue which I do not see in the non-crossbow > version of the driver. When I run TCP benchmarks, I'm seeing > what seems like packet loss. Specifically, netstat shows > tcpInUnorderBytes and tcpInDupBytes increasing at a rapid rate, > and bandwidth is terrible (~1Gb/s for crossbow, 7Gb/s non-crossbow > on the same box with the same OS revision). > > The first thing I suspected was that packets were getting dropped > due to my having the wrong generation number, but a dtrace probe > doesn't show any drops there. > > Now I'm wondering if perhaps the interupt handler is in > the middle of a call to mac_rx_ring() when interrupts > are disabled. Am I supposed to ensure that my interrupt handler is not > calling mac_rx_ring() before my rx_ring_intr_disable() > routine returns? Or does the mac layer serialize this?
Can you reproduce the problem with only one RX ring enabled? If so, something to try would be to bind the poll thread to the same CPU as the MSI for that single RX ring. To find the CPU the MSI is bound to, run ::interrupts from mdb, then assign the CPU to use for the poll thread by doing a "dladm setlinkprop -p cpus=<cpuid> <link>". There might be a race between the poll thread and the thread trying to deliver the chain through mac_rx_ring() from interrupt context, since we currently don't rebind the MSIs to the same CPUs as their corresponding poll threads. We are planning to do the rebinding of MSIs, but we are depending on interrupt rebinding APIs which are still being worked on. The experiment above would allow us to confirm whether it the issue seen here or if need to look somewhere else. BTW, which ONNV build are you currently using? Nicolas. > > > Thanks, > > Drew -- Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc. nicolas.droux at sun.com - http://blogs.sun.com/droux
