On 25.11.2008 14:54, Anton Yuzhaninov wrote:
On 25.11.2008 02:38, Anton Yuzhaninov wrote:
On 25.11.2008 01:48, Anton Yuzhaninov wrote:
Box with fresh RELENG_7 panic under heavy network load (more than 50k
connections).
This panics seems to be senfile(2) related, because when sendfile
disabled in nginx, I can't reproduce the problem.
Backtrace in all cases like this:
# kgdb kernel /spool/crash/vmcore.1
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
panic: vm_page_unwire: invalid wire count: 0
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x182
vm_page_unwire() at vm_page_unwire+0x84
sf_buf_mext() at sf_buf_mext+0x3c
mb_free_ext() at mb_free_ext+0x99
sbdrop_internal() at sbdrop_internal+0x1e8
tcp_do_segment() at tcp_do_segment+0x1512
tcp_input() at tcp_input+0x7f7
ip_input() at ip_input+0xa8
ether_demux() at ether_demux+0x1b4
ether_input() at ether_input+0x1bb
bge_intr() at bge_intr+0x3ca
ithread_loop() at ithread_loop+0x180
fork_exit() at fork_exit+0x11f
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffea28fd30, rbp = 0 ---
Uptime: 36m47s
Physical memory: 4087 MB
Dumping 708 MB: 693 677 661 645 629 613 597 581 565 549 533 517 501
485 469 453 437 421 405 389 373 357 341 325 309 293 277 261 245 229
213 197 181 165 149 133 117 101 85 69 53 37 21 5
#0 doadump () at pcpu.h:195
195 __asm __volatile("movq %%gs:0,%0" : "=r" (td));
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xffffffff8031adf8 in boot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:418
#2 0xffffffff8031b25c in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3 0xffffffff8044a084 in vm_page_unwire (m=Variable "m" is not
available.
) at /usr/src/sys/vm/vm_page.c:1410
#4 0xffffffff80379a4c in sf_buf_mext (addr=Variable "addr" is not
available.
) at /usr/src/sys/kern/uipc_syscalls.c:1720
#5 0xffffffff8036e9c9 in mb_free_ext (m=0xffffff0081f93d00) at
/usr/src/sys/kern/uipc_mbuf.c:257
May be it is wire_count integer overflow?
wire_count type is u_short...
Yes, It clearly wire_count integer overflow
On kernel with INVARIANTS panic string is:
vm_page_wire: wire_count overflow m=0xffffff00d7d270c8
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xffffffff8030b806 in boot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:418
#2 0xffffffff8030bc6c in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3 0xffffffff8042e289 in vm_page_wire (m=Variable "m" is not available.
) at /usr/src/sys/vm/vm_page.c:1358
#4 0xffffffff8042f62e in vm_page_grab (object=0xffffff0003cfa340,
pindex=7, allocflags=672)
at /usr/src/sys/vm/vm_page.c:1695
#5 0xffffffff8036a9ca in kern_sendfile (td=0xffffff0003870a50,
uap=0xffffffffec6e1bf0, hdr_uio=0x1000,
trl_uio=0xffffff001ea7e6c0, compat=Variable "compat" is not available.
) at /usr/src/sys/kern/uipc_syscalls.c:2050
#6 0xffffffff8036b0f6 in sendfile (td=0xffffff0003870a50,
uap=0xffffffffec6e1bf0) at /usr/src/sys/kern/uipc_syscalls.c:1775
#7 0xffffffff804593ec in syscall (frame=0xffffffffec6e1c80) at
/usr/src/sys/amd64/amd64/trap.c:907
I see 2 ways to fix this bug:
1. Change wire_count type to u_int - it bad, because vm_page will eats
more memory.
I have tested with u_int wire_count - panic don't repeated.
It seems to be good solution.
due to alignment vm_page has same size
On unpatched kernel (u_short wire_count):
(kgdb) print sizeof(struct vm_page)
$1 = 120
On patched kernel (u_int wire_count):
(kgdb) print sizeof(struct vm_page)
$1 = 120
arch is amd64 on both boxes.
--
Anton Yuzhaninov
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
