RE: Kernel Panic on 9.0 and 9.1 with carp on BCE network interface
Hi, Is there anything else I can provide you? the core and kernel.debug?
The panic is hapening once or twice a day on this server.
Thank you again for your help
> Date: Fri, 7 Sep 2012 17:14:41 +0400
> From: gleb...@freebsd.org
> To: jl.dup...@outlook.com
> CC: freebsd-stable@FreeBSD.org
> Subject: Re: Kernel Panic on 9.0 and 9.1 with carp on BCE network interface
>
> On Thu, Aug 30, 2012 at 02:39:10PM +, Jean-Luc Dupont wrote:
> J> Sorry, it seems that I didn't put the right backtrace :
> J>
> J> #0 doadump (textdump=Variable "textdump" is not available.
> J> ) at /usr/src/sys/kern/kern_shutdown.c:271
> J> 271 dumpsys(&dumper);
> J> (kgdb) #0 doadump (textdump=Variable "textdump" is not available.
> J> ) at /usr/src/sys/kern/kern_shutdown.c:271
> J> #1 0x807fdf02 in kern_reboot (howto=260)
> J> at /usr/src/sys/kern/kern_shutdown.c:448
> J> #2 0x807fe3e3 in panic (fmt=0x104 )
> J> at /usr/src/sys/kern/kern_shutdown.c:636
> J> #3 0x80ad2700 in trap_fatal (frame=0xc, eva=Variable "eva" is not
> available.
> J> )
> J> at /usr/src/sys/amd64/amd64/trap.c:857
> J> #4 0x80ad2a3d in trap_pfault (frame=0xff82e97a3500,
> usermode=0)
> J> at /usr/src/sys/amd64/amd64/trap.c:773
> J> #5 0x80ad305e in trap (frame=0xff82e97a3500)
> J> at /usr/src/sys/amd64/amd64/trap.c:456
> J> #6 0x80abd67f in calltrap ()
> J> at /usr/src/sys/amd64/amd64/exception.S:228
> J> #7 0x8085f597 in m_copym (m=0x0, off0=1500, len=1480, wait=1)
> J> at /usr/src/sys/kern/uipc_mbuf.c:542
> J> #8 0x8092f2c8 in ip_fragment (ip=0xfe00970e0580,
> J> m_frag=0xff82e97a3728, mtu=Variable "mtu" is not available.
> J> ) at /usr/src/sys/netinet/ip_output.c:822
> J> #9 0x8092fc17 in ip_output (m=0xfe00970e0500, opt=Variable
> "opt" is not available.
> J> )
> J> at /usr/src/sys/netinet/ip_output.c:653
> J> #10 0x80928713 in ip_forward (m=0xfe00970e0500, srcrt=Variable
> "srcrt" is not available.
> J> )
> J> at /usr/src/sys/netinet/ip_input.c:1494
> J> #11 0x80929dc8 in ip_input (m=0xfe00970e0500)
> J> at /usr/src/sys/netinet/ip_input.c:702
>
> I don't see that this is CARP related. Do you use any firewall: pf or ipfw?
>
> Can you please show the below session in gdb with discussed core file:
>
> gdb> fr 9
> gdb> p mtu
> gdb> fr 7
> gdb> p off
> gdb> fr 8
> gdb> p m0
> gdb> p *m0
>
> --
> Totus tuus, Glebius.
> ___
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Hi,
Thank you very much for your reply, we are using IPFW with several VLAN and
several CARP on intel igb and bce network cards on a dell poweredge servers.
When we stopped using the bce and using only the igb (with more vlans per
interface) we don't have any more panics.
Here is the output of the debugger as asked :
(kgdb) fr 9
#9 0x8092fc17 in ip_output (m=0xfe00941c8300, opt=Variable "opt"
is not available.
) at /usr/src/sys/netinet/ip_output.c:653
653 error = ip_fragment(ip, &m, mtu, ifp->if_hwassist, sw_csum);
(kgdb) p mtu
$1 = 1500
(kgdb) fr 7
#7 0x8085f597 in m_copym (m=0x0, off0=1500, len=1317, wait=1) at
/usr/src/sys/kern/uipc_mbuf.c:542
542 if (off < m->m_len)
(kgdb) p off
$2 = 1233
(kgdb) fr 8
#8 0x8092f2c8 in ip_fragment (ip=0xfe00941c8380,
m_frag=0xff834869e7f8, mtu=Variable "mtu" is not available.
)
at /usr/src/sys/netinet/ip_output.c:822
822 m->m_next = m_copym(m0, off, len, M_DONTWAIT);
(kgdb) p m0
$3 = (struct mbuf *) 0xfe00941c8300
(kgdb) p *m0
$4 = {m_hdr = {mh_next = 0xfe0081d51800, mh_nextpkt = 0x0, mh_data =
0xfe00941c8380 "E", mh_len = 40,
mh_flags = 2, mh_type = 1, pad = "\000\000\000\000\000"}, M_dat = {MH =
{MH_pkthdr = {
rcvif = 0xfe0003b53800, header = 0x0, len = 267, flowid = 0,
csum_flags = 0, csum_data = 65535,
tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0}, tags = {slh_first =
0x0}}, MH_dat = {MH_ext = {
ext_buf = 0x400092ae00400045 , ext_free = 0x16207,
ext_arg1 = 0x42011d, ext_arg2 = 0x601005e, ext_size =
2660147200,
ref_cnt = 0x40f7e20b010045, ext_type = -843971023},
MH_databuf =
"E\000@\000�\222\000@\ab\001\000\000\000\000\000\
RE: Kernel Panic on 9.0 and 9.1 with carp on BCE network interface
> Date: Fri, 7 Sep 2012 17:14:41 +0400
> From: gleb...@freebsd.org
> To: jl.dup...@outlook.com
> CC: freebsd-stable@FreeBSD.org
> Subject: Re: Kernel Panic on 9.0 and 9.1 with carp on BCE network interface
>
> On Thu, Aug 30, 2012 at 02:39:10PM +, Jean-Luc Dupont wrote:
> J> Sorry, it seems that I didn't put the right backtrace :
> J>
> J> #0 doadump (textdump=Variable "textdump" is not available.
> J> ) at /usr/src/sys/kern/kern_shutdown.c:271
> J> 271 dumpsys(&dumper);
> J> (kgdb) #0 doadump (textdump=Variable "textdump" is not available.
> J> ) at /usr/src/sys/kern/kern_shutdown.c:271
> J> #1 0x807fdf02 in kern_reboot (howto=260)
> J> at /usr/src/sys/kern/kern_shutdown.c:448
> J> #2 0x807fe3e3 in panic (fmt=0x104 )
> J> at /usr/src/sys/kern/kern_shutdown.c:636
> J> #3 0x80ad2700 in trap_fatal (frame=0xc, eva=Variable "eva" is not
> available.
> J> )
> J> at /usr/src/sys/amd64/amd64/trap.c:857
> J> #4 0x80ad2a3d in trap_pfault (frame=0xff82e97a3500,
> usermode=0)
> J> at /usr/src/sys/amd64/amd64/trap.c:773
> J> #5 0x80ad305e in trap (frame=0xff82e97a3500)
> J> at /usr/src/sys/amd64/amd64/trap.c:456
> J> #6 0x80abd67f in calltrap ()
> J> at /usr/src/sys/amd64/amd64/exception.S:228
> J> #7 0x8085f597 in m_copym (m=0x0, off0=1500, len=1480, wait=1)
> J> at /usr/src/sys/kern/uipc_mbuf.c:542
> J> #8 0x8092f2c8 in ip_fragment (ip=0xfe00970e0580,
> J> m_frag=0xff82e97a3728, mtu=Variable "mtu" is not available.
> J> ) at /usr/src/sys/netinet/ip_output.c:822
> J> #9 0x8092fc17 in ip_output (m=0xfe00970e0500, opt=Variable
> "opt" is not available.
> J> )
> J> at /usr/src/sys/netinet/ip_output.c:653
> J> #10 0x80928713 in ip_forward (m=0xfe00970e0500, srcrt=Variable
> "srcrt" is not available.
> J> )
> J> at /usr/src/sys/netinet/ip_input.c:1494
> J> #11 0x80929dc8 in ip_input (m=0xfe00970e0500)
> J> at /usr/src/sys/netinet/ip_input.c:702
>
> I don't see that this is CARP related. Do you use any firewall: pf or ipfw?
>
> Can you please show the below session in gdb with discussed core file:
>
> gdb> fr 9
> gdb> p mtu
> gdb> fr 7
> gdb> p off
> gdb> fr 8
> gdb> p m0
> gdb> p *m0
>
> --
> Totus tuus, Glebius.
> ___
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Hi,
Thank you very much for your reply, we are using IPFW with several VLAN and
several CARP on intel igb and bce network cards on a dell poweredge servers.
When we stopped using the bce and using only the igb (with more vlans per
interface) we don't have any more panics.
Here is the output of the debugger as asked :
(kgdb) fr 9
#9 0x8092fc17 in ip_output (m=0xfe00941c8300, opt=Variable "opt"
is not available.
) at /usr/src/sys/netinet/ip_output.c:653
653 error = ip_fragment(ip, &m, mtu, ifp->if_hwassist, sw_csum);
(kgdb) p mtu
$1 = 1500
(kgdb) fr 7
#7 0x8085f597 in m_copym (m=0x0, off0=1500, len=1317, wait=1) at
/usr/src/sys/kern/uipc_mbuf.c:542
542 if (off < m->m_len)
(kgdb) p off
$2 = 1233
(kgdb) fr 8
#8 0x8092f2c8 in ip_fragment (ip=0xfe00941c8380,
m_frag=0xff834869e7f8, mtu=Variable "mtu" is not available.
)
at /usr/src/sys/netinet/ip_output.c:822
822 m->m_next = m_copym(m0, off, len, M_DONTWAIT);
(kgdb) p m0
$3 = (struct mbuf *) 0xfe00941c8300
(kgdb) p *m0
$4 = {m_hdr = {mh_next = 0xfe0081d51800, mh_nextpkt = 0x0, mh_data =
0xfe00941c8380 "E", mh_len = 40,
mh_flags = 2, mh_type = 1, pad = "\000\000\000\000\000"}, M_dat = {MH =
{MH_pkthdr = {
rcvif = 0xfe0003b53800, header = 0x0, len = 267, flowid = 0,
csum_flags = 0, csum_data = 65535,
tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0}, tags = {slh_first =
0x0}}, MH_dat = {MH_ext = {
ext_buf = 0x400092ae00400045 , ext_free = 0x16207,
ext_arg1 = 0x42011d, ext_arg2 = 0x601005e, ext_size =
2660147200,
ref_cnt = 0x40f7e20b010045, ext_type = -843971023},
MH_databuf =
"E\000@\000�\222\000@\ab\001\000\000\000\000\000\000\000\035\001��B\000\000\000\000\000^\000\001\006\000�\216\236�\200\b\000E\000\001\v��@\0001\006��H\025T�\n\n\vK\000
Re: Kernel Panic on 9.0 and 9.1 with carp on BCE network interface
On Thu, Aug 30, 2012 at 02:39:10PM +, Jean-Luc Dupont wrote: J> Sorry, it seems that I didn't put the right backtrace : J> J> #0 doadump (textdump=Variable "textdump" is not available. J> ) at /usr/src/sys/kern/kern_shutdown.c:271 J> 271 dumpsys(&dumper); J> (kgdb) #0 doadump (textdump=Variable "textdump" is not available. J> ) at /usr/src/sys/kern/kern_shutdown.c:271 J> #1 0x807fdf02 in kern_reboot (howto=260) J> at /usr/src/sys/kern/kern_shutdown.c:448 J> #2 0x807fe3e3 in panic (fmt=0x104 ) J> at /usr/src/sys/kern/kern_shutdown.c:636 J> #3 0x80ad2700 in trap_fatal (frame=0xc, eva=Variable "eva" is not available. J> ) J> at /usr/src/sys/amd64/amd64/trap.c:857 J> #4 0x80ad2a3d in trap_pfault (frame=0xff82e97a3500, usermode=0) J> at /usr/src/sys/amd64/amd64/trap.c:773 J> #5 0x80ad305e in trap (frame=0xff82e97a3500) J> at /usr/src/sys/amd64/amd64/trap.c:456 J> #6 0x80abd67f in calltrap () J> at /usr/src/sys/amd64/amd64/exception.S:228 J> #7 0x8085f597 in m_copym (m=0x0, off0=1500, len=1480, wait=1) J> at /usr/src/sys/kern/uipc_mbuf.c:542 J> #8 0x8092f2c8 in ip_fragment (ip=0xfe00970e0580, J> m_frag=0xff82e97a3728, mtu=Variable "mtu" is not available. J> ) at /usr/src/sys/netinet/ip_output.c:822 J> #9 0x8092fc17 in ip_output (m=0xfe00970e0500, opt=Variable "opt" is not available. J> ) J> at /usr/src/sys/netinet/ip_output.c:653 J> #10 0x80928713 in ip_forward (m=0xfe00970e0500, srcrt=Variable "srcrt" is not available. J> ) J> at /usr/src/sys/netinet/ip_input.c:1494 J> #11 0x80929dc8 in ip_input (m=0xfe00970e0500) J> at /usr/src/sys/netinet/ip_input.c:702 I don't see that this is CARP related. Do you use any firewall: pf or ipfw? Can you please show the below session in gdb with discussed core file: gdb> fr 9 gdb> p mtu gdb> fr 7 gdb> p off gdb> fr 8 gdb> p m0 gdb> p *m0 -- Totus tuus, Glebius. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
RE: Kernel Panic on 9.0 and 9.1 with carp on BCE network interface
Sorry, it seems that I didn't put the right backtrace : #0 doadump (textdump=Variable "textdump" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:271 271 dumpsys(&dumper); (kgdb) #0 doadump (textdump=Variable "textdump" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:271 #1 0x807fdf02 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448 #2 0x807fe3e3 in panic (fmt=0x104 ) at /usr/src/sys/kern/kern_shutdown.c:636 #3 0x80ad2700 in trap_fatal (frame=0xc, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0x80ad2a3d in trap_pfault (frame=0xff82e97a3500, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:773 #5 0x80ad305e in trap (frame=0xff82e97a3500) at /usr/src/sys/amd64/amd64/trap.c:456 #6 0x80abd67f in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 #7 0x8085f597 in m_copym (m=0x0, off0=1500, len=1480, wait=1) at /usr/src/sys/kern/uipc_mbuf.c:542 #8 0x8092f2c8 in ip_fragment (ip=0xfe00970e0580, m_frag=0xff82e97a3728, mtu=Variable "mtu" is not available. ) at /usr/src/sys/netinet/ip_output.c:822 #9 0x8092fc17 in ip_output (m=0xfe00970e0500, opt=Variable "opt" is not available. ) at /usr/src/sys/netinet/ip_output.c:653 #10 0x80928713 in ip_forward (m=0xfe00970e0500, srcrt=Variable "srcrt" is not available. ) at /usr/src/sys/netinet/ip_input.c:1494 #11 0x80929dc8 in ip_input (m=0xfe00970e0500) at /usr/src/sys/netinet/ip_input.c:702 #12 0x808bfe17 in netisr_dispatch_src (proto=56, source=Variable "source" is not available. ) at /usr/src/sys/net/netisr.c:1013 #13 0x808b6d8c in ether_demux (ifp=0xfe0011056000, m=0xfe00970e0500) at /usr/src/sys/net/if_ethersubr.c:940 #14 0x808b7054 in ether_nh_input (m=Variable "m" is not available. ) at /usr/src/sys/net/if_ethersubr.c:759 #15 0x808bfe17 in netisr_dispatch_src (proto=504, source=Variable "source" is not available. ) at /usr/src/sys/net/netisr.c:1013 #16 0x808b6caf in ether_demux (ifp=0xfe0003b54000, m=0xfe00970e0500) at /usr/src/sys/net/if_ethersubr.c:849 #17 0x808b7054 in ether_nh_input (m=Variable "m" is not available. ) at /usr/src/sys/net/if_ethersubr.c:759 #18 0x808bfe17 in netisr_dispatch_src (proto=504, source=Variable "source" is not available. ) at /usr/src/sys/net/netisr.c:1013 #19 0x8040048a in bce_intr (xsc=Variable "xsc" is not available. ) at /usr/src/sys/dev/bce/if_bce.c:6578 #20 0x807d39e4 in intr_event_execute_handlers (p=Variable "p" is not available. ) at /usr/src/sys/kern/kern_intr.c:1260 #21 0x807d5176 in ithread_loop (arg=0xfe0003e5) at /usr/src/sys/kern/kern_intr.c:1273 #22 0x807d0b0f in fork_exit ( callout=0x807d50d0 , arg=0xfe0003e5, frame=0xff82e97a3c40) at /usr/src/sys/kern/kern_fork.c:992 #23 0x80abdbae in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:602 > From: jl.dup...@outlook.com > To: freebsd-stable@freebsd.org > Date: Thu, 30 Aug 2012 14:33:20 + > Subject: Kernel Panic on 9.0 and 9.1 with carp on BCE network interface > > Hello, > > We are using Freebsd 9.0 on dell poweredge servers with broadcom (bce) > integrated interfaces and intel pci-x cards. > > This server is used as gateway/firewall (IPFW) with several VLANs and each > interface and carp for redundancy with another identical server. > > We encounter a kernel panic and reboot when some users connect to an external > FTP (just after > accepting the password). > > We upgraded to 9.1-prerelease and 9.1-stable, without any change, still same > systematic panic. > > Here is the backtrace : > > cpuid = 3 > KDB: stack backtrace: > #0 0x808342b6 at kdb_backtrace+0x66 > #1 0x807fe40d at panic+0x1cd > #2 0x80a21c6f at ufs_dirbad+0x4f > #3 0x80a2334a at ufs_lookup_ino+0x68a > #4 0x8088077f at vfs_cache_lookup+0xff > #5 0x80b3b380 at VOP_LOOKUP_APV+0x40 > #6 0x80887e94 at lookup+0x464 > #7 0x80888fa9 at namei+0x4e9 > #8 0x808a307b at vn_open_cred+0x3bb > #9 0x808a2117 at kern_openat+0x1e7 > #10 0x80ad1fa6 at amd64_syscall+0x546 > #11 0x80abd967 at Xfast_syscall+0xf7 > Uptime: 6m4s > > (kgdb) #0 doadump (textdump=Variable "textdump" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:271 > #1 0x807fdf02 in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:448 > #2 0x807f
Kernel Panic on 9.0 and 9.1 with carp on BCE network interface
Hello, We are using Freebsd 9.0 on dell poweredge servers with broadcom (bce) integrated interfaces and intel pci-x cards. This server is used as gateway/firewall (IPFW) with several VLANs and each interface and carp for redundancy with another identical server. We encounter a kernel panic and reboot when some users connect to an external FTP (just after accepting the password). We upgraded to 9.1-prerelease and 9.1-stable, without any change, still same systematic panic. Here is the backtrace : cpuid = 3 KDB: stack backtrace: #0 0x808342b6 at kdb_backtrace+0x66 #1 0x807fe40d at panic+0x1cd #2 0x80a21c6f at ufs_dirbad+0x4f #3 0x80a2334a at ufs_lookup_ino+0x68a #4 0x8088077f at vfs_cache_lookup+0xff #5 0x80b3b380 at VOP_LOOKUP_APV+0x40 #6 0x80887e94 at lookup+0x464 #7 0x80888fa9 at namei+0x4e9 #8 0x808a307b at vn_open_cred+0x3bb #9 0x808a2117 at kern_openat+0x1e7 #10 0x80ad1fa6 at amd64_syscall+0x546 #11 0x80abd967 at Xfast_syscall+0xf7 Uptime: 6m4s (kgdb) #0 doadump (textdump=Variable "textdump" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:271 #1 0x807fdf02 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448 #2 0x807fe3e3 in panic (fmt=0x104 ) at /usr/src/sys/kern/kern_shutdown.c:636 #3 0x80a21c6f in ufs_dirbad (ip=Variable "ip" is not available. ) at /usr/src/sys/ufs/ufs/ufs_lookup.c:773 #4 0x80a2334a in ufs_lookup_ino (vdp=0xfe00b0aee5a0, vpp=0xff834bb30a10, cnp=0xff834bb30a38, dd_ino=0x0) at /usr/src/sys/ufs/ufs/ufs_lookup.c:393 #5 0x8088077f in vfs_cache_lookup (ap=Variable "ap" is not available. ) at vnode_if.h:80 #6 0x80b3b380 in VOP_LOOKUP_APV (vop=0x8104e960, a=0xff834bb306f0) at vnode_if.c:123 #7 0x80887e94 in lookup (ndp=0xff834bb309d0) at vnode_if.h:54 #8 0x80888fa9 in namei (ndp=0xff834bb309d0) at /usr/src/sys/kern/vfs_lookup.c:297 #9 0x808a307b in vn_open_cred (ndp=0xff834bb309d0, flagp=0xff834bb309cc, cmode=0, vn_open_flags=Variable "vn_open_flags" is not available. ) at /usr/src/sys/kern/vfs_vnops.c:195 #10 0x808a2117 in kern_openat (td=0xfe000ff75470, fd=-100, path=0x809ff80c0 , pathseg=UIO_USERSPACE, flags=1, mode=Variable "mode" is not available. ) at /usr/src/sys/kern/vfs_syscalls.c:1132 #11 0x80ad1fa6 in amd64_syscall (td=0xfe000ff75470, traced=0) at subr_syscall.c:135 #12 0x80abd967 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:387 #13 0x000801e5366c in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) We have the coredump and the kernel.debug, if there is anything we can do to help you identifying the origin of this issue. Thank you very much ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
