On May 11, 2009, at 1:57 PM, Andrew Gallatin wrote:
Nicolas Droux wrote:
Andrew,
Thanks for the additional info. We'd like to verify that interrupts
are getting disabled from interrupt context itself. If you don't
mind, could you gather an aggregation of the callers of
mac_hwring_disable_intr() during one of your runs? You should be
able to do this with "dtrace -n fbt::
mac_hwring_disable_intr:entry'{...@[stack()] = count()}'"
Yes, they're all coming from the interrupt context, via my rx
interrupt handler:
dtrace: description 'fbt::mac_hwring_disable_intr:entry' matched 1
probe
^C
mac`mac_rx_srs_drain+0x359
mac`mac_rx_srs_process+0x1db
mac`mac_rx+0x94
mac`mac_rx_ring+0x4c
myri10ge`myri10ge_intr_rx+0x70
myri10ge`myri10ge_intr+0xa2
unix`av_dispatch_autovect+0x7c
unix`dispatch_hardint+0x33
unix`switch_sp_and_call+0x13
66481
OK that's good to know.
FWIW, I was able to reproduce the problem on another machine
running build 111. I BFU'ed to 112, 113, and 114, and I see
the same thing in each build.
I think that I must be doing something wrong in my driver.
I wouldn't rule this out from the result of these experiments.
One key requirement is that you need to ensure that the driver will
not pass new packets through mac_rx_ring() for a ring once its
interrupt disable entry point for that ring returns. Otherwise the
mac_rx_ring() thread can race against the poll thread which can cause
packet reordering. This could be the issue here. The driver is
required to implement the locking needed for this. If the driver needs
to change state from the interrupt disable entry point and query that
state from the interrupt handler before invoking mac_rx_ring(), both
pieces of code need to protect their access to that state via a common
lock.
Nicolas.
Is there some sort of counter or dtrace script that one
use to track down out-of-order packets?
I'm not sure if this is helpful, but the 2 stacks I see for
the increment of mib:::tcpInDataUnorderSegs are:
# dtrace -n mib:::tcpInDataUnorderSegs'{...@[stack()] = count()}'
dtrace: description 'mib:::tcpInDataUnorderSegs' matched 1 probe
^C
ip`tcp_rput_data+0xdd0
ip`squeue_enter+0x330
ip`ip_input+0xc17
mac`mac_rx_soft_ring_drain+0xdf
mac`mac_soft_ring_worker+0x111
unix`thread_start+0x8
1710
ip`tcp_rput_data+0xdd0
ip`squeue_drain+0x179
ip`squeue_enter+0x3f4
ip`ip_input+0xc17
mac`mac_rx_soft_ring_drain+0xdf
mac`mac_soft_ring_worker+0x111
unix`thread_start+0x8
195335
Thanks,
Drew
--
Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc.
[email protected] - http://blogs.sun.com/droux
_______________________________________________
networking-discuss mailing list
[email protected]
