On Mon, 5 Jan 2009, Terry Kennedy wrote:
I may have missed this earlier in the thread, but I don't see a kernel
stack trace of the stuck thread/process. Could you grab one using procstat
-k, DDB, or KGDB? I'd like to confirm that the 'sbwait' really reflects
waiting to send, rather than waiting to receive, which (for better or
worse) uses the same wmesg. procstat -k may be the simplest of the above
to do if your system is reasonable recent.
I didn't post that earlier as no-one had asked for it 8-)
Indeed :-).
The system is current as of December 29th. Here's the relevant info:
Could I ask you to also send me procstat -f output?
More below the quote.
(0:10) test4:/sysprog/terry# uname -a
FreeBSD test4.tmk.com 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #0: Mon Dec 29
11:48:04 EST 2008 [email protected]:/usr/obj/usr/src/sys/PE1550 i386
(0:11) test4:/sysprog/terry# ps -axwww | grep dump
UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
0 4436 4411 0 8 0 35896 34552 wait I+ p1 0:00.70
/sbin/rdump 0uLa -b 64 -C 32 -f server /usr (rdump)
0 4439 4436 0 4 0 35896 34784 sbwait I+ p1 0:03.05 rdump:
/dev/amrd0s1f: pass 4: 18.48% done, finished in 0:17 at Sat Jan 3 21:02:05
2009 (rdump)
0 4440 4439 0 20 0 35896 34624 pause I+ p1 0:05.26
/sbin/rdump 0uLa -b 64 -C 32 -f server /usr (rdump)
0 4441 4439 0 20 0 35896 34624 pause I+ p1 0:05.26
/sbin/rdump 0uLa -b 64 -C 32 -f server /usr (rdump)
0 4442 4439 0 4 0 35896 34624 sbwait I+ p1 0:05.26
/sbin/rdump 0uLa -b 64 -C 32 -f server /usr (rdump)
(0:12) test4:/sysprog/terry# procstat -k 4436
PID TID COMM TDNAME KSTACK
4436 100115 rdump - mi_switch sleepq_switch
sleepq_catch_signals sleepq_wait_sig _sleep kern_wait wait4 syscall
Xint0x80_syscall (0:13) test4:/sysprog/terry# procstat -k 4439
PID TID COMM TDNAME KSTACK
4439 100127 rdump - mi_switch sleepq_switch
sleepq_catch_signals sleepq_wait_sig _sleep sbwait soreceive_generic
soreceive soo_read dofileread kern_readv read syscall Xint0x80_syscall (0:14)
test4:/sysprog/terry# procstat -k 4440
PID TID COMM TDNAME KSTACK
4440 100131 rdump - mi_switch sleepq_switch
sleepq_catch_signals sleepq_wait_sig _sleep kern_sigsuspend sigsuspend
syscall Xint0x80_syscall (0:15) test4:/sysprog/terry# procstat -k 4441
PID TID COMM TDNAME KSTACK
4441 100105 rdump - mi_switch sleepq_switch
sleepq_catch_signals sleepq_wait_sig _sleep kern_sigsuspend sigsuspend
syscall Xint0x80_syscall (0:16) test4:/sysprog/terry# procstat -k 4442
PID TID COMM TDNAME KSTACK
4442 100135 rdump - mi_switch sleepq_switch
sleepq_catch_signals sleepq_wait_sig _sleep sbwait soreceive_generic
soreceive soo_read dofileread kern_readv read syscall Xint0x80_syscall
As I understand it, the processes in sbwait state are waiting to receive.
That would seem to indicate that they don't see the ACKs from the other end,
despite the tcpdump showing that they were received.
In general, being blocked in soreceive() means that the application at the
other end hasn't sent data, or the other end hasn't received or correctly
processed ACKs from the local end, so isn't sending more data that it has
queued up. The condition you describe sounds more like what would happen in a
sender: that it has data to send, but the remote side hasn't ACK'd
sufficiently to send it all. If you have kgdb handy, it would be useful to
look at *so and *so->so_domain in the soreceive_generic frame of proc 4439.
If it's an inet socket, we'd like to see *(struct inpcb *)so->so_pcb, and if
it's a TCP socket, *(struct tcpcb *)((struct inpcb *)so->so_pcb)->inp_ppcb.
Robert N M Watson
Computer Laboratory
University of Cambridge
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[email protected]"
