On May 11, 2009, at 1:57 PM, Andrew Gallatin wrote:
> Nicolas Droux wrote:
>> Andrew,
>> Thanks for the additional info. We'd like to verify that interrupts
>> are getting disabled from interrupt context itself. If you don't
>> mind, could you gather an aggregation of the callers of
>> mac_hwring_disable_intr() during one of your runs? You should be
>> able to do this with "dtrace -n fbt::
>> mac_hwring_disable_intr:entry'{...@[stack()] = count()}'"
>
> Yes, they're all coming from the interrupt context, via my rx
> interrupt handler:
>
> dtrace: description 'fbt::mac_hwring_disable_intr:entry' matched 1
> probe
> ^C
>
>
> mac`mac_rx_srs_drain+0x359
> mac`mac_rx_srs_process+0x1db
> mac`mac_rx+0x94
> mac`mac_rx_ring+0x4c
> myri10ge`myri10ge_intr_rx+0x70
> myri10ge`myri10ge_intr+0xa2
> unix`av_dispatch_autovect+0x7c
> unix`dispatch_hardint+0x33
> unix`switch_sp_and_call+0x13
> 66481
OK that's good to know.
>
>
> FWIW, I was able to reproduce the problem on another machine
> running build 111. I BFU'ed to 112, 113, and 114, and I see
> the same thing in each build.
>
> I think that I must be doing something wrong in my driver.
I wouldn't rule this out from the result of these experiments.
One key requirement is that you need to ensure that the driver will
not pass new packets through mac_rx_ring() for a ring once its
interrupt disable entry point for that ring returns. Otherwise the
mac_rx_ring() thread can race against the poll thread which can cause
packet reordering. This could be the issue here. The driver is
required to implement the locking needed for this. If the driver needs
to change state from the interrupt disable entry point and query that
state from the interrupt handler before invoking mac_rx_ring(), both
pieces of code need to protect their access to that state via a common
lock.
Nicolas.
>
> Is there some sort of counter or dtrace script that one
> use to track down out-of-order packets?
>
> I'm not sure if this is helpful, but the 2 stacks I see for
> the increment of mib:::tcpInDataUnorderSegs are:
>
> # dtrace -n mib:::tcpInDataUnorderSegs'{...@[stack()] = count()}'
> dtrace: description 'mib:::tcpInDataUnorderSegs' matched 1 probe
> ^C
>
>
> ip`tcp_rput_data+0xdd0
> ip`squeue_enter+0x330
> ip`ip_input+0xc17
> mac`mac_rx_soft_ring_drain+0xdf
> mac`mac_soft_ring_worker+0x111
> unix`thread_start+0x8
> 1710
>
> ip`tcp_rput_data+0xdd0
> ip`squeue_drain+0x179
> ip`squeue_enter+0x3f4
> ip`ip_input+0xc17
> mac`mac_rx_soft_ring_drain+0xdf
> mac`mac_soft_ring_worker+0x111
> unix`thread_start+0x8
> 195335
>
>
> Thanks,
> Drew
--
Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc.
nicolas.droux at sun.com - http://blogs.sun.com/droux
