Hello.
A box of mine, which acts as firewall/bridge, is experiencing frequent panics. As said in the subject line, it's a 4.10-RELEASE-p4 with ipfw2 enabled in the kernel.
I've run through post mortem kernel analisys and found out that the crashes are always related to ipfw2; specifically I get:
panic: free: multiple frees
Here is the complete backtrack:
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1 0xc0150993 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
#2 0xc0150db8 in poweroff_wait (junk=0xc02354ac, howto=-1071427665)
at /usr/src/sys/kern/kern_shutdown.c:595
#3 0xc0208a3e in trap_fatal (frame=0xc023a3e4, eva=48)
at /usr/src/sys/i386/i386/trap.c:974
#4 0xc0208711 in trap_pfault (frame=0xc023a3e4, usermode=0, eva=48)
at /usr/src/sys/i386/i386/trap.c:867
#5 0xc02082fb in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi = 0,
tf_esi = 0, tf_ebp = -1071406036, tf_isp = -1071406064,
tf_ebx = -1071330820, tf_edx = 6864896, tf_ecx = -1054588914,
tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1071892584, tf_cs = 8,
tf_eflags = 66182, tf_esp = -967647568, tf_ss = 0})
at /usr/src/sys/i386/i386/trap.c:466
#6 0xc01c3798 in acquire_lock (lk=0xc024c9fc)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:266
#7 0xc01c8e7c in softdep_count_dependencies (bp=0xc652deb0, wantcount=0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:4792
#8 0xc01cc0d8 in ffs_fsync (ap=0xc023a4a0)
at /usr/src/sys/ufs/ffs/ffs_vnops.c:168
#9 0xc01cabab in ffs_sync (mp=0xc123fc00, waitfor=2, cred=0xc0a3e800,
p=0xc026dbe0) at vnode_if.h:558
#10 0xc0181737 in sync (p=0xc026dbe0, uap=0x0)
at /usr/src/sys/kern/vfs_syscalls.c:583
#11 0xc015072e in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
#12 0xc0150db8 in poweroff_wait (junk=0xc0218cff, howto=-1051816704)
at /usr/src/sys/kern/kern_shutdown.c:595
#13 0xc014c41f in free (addr=0xc18fc100, type=0xc0249420)
at /usr/src/sys/kern/kern_malloc.c:385
#14 0xc01a56ce in lookup_dyn_rule (pkt=0xc023a650, match_direction=0xc023a5c8,
tcp=0xc0b26b50) at /usr/src/sys/netinet/ip_fw2.c:784
#15 0xc01a6ae7 in ipfw_chk (args=0xc023a630)
at /usr/src/sys/netinet/ip_fw2.c:1900
#16 0xc01aa5f5 in ip_output (m0=0xc0b26b00, opt=0x0, ro=0xd0bfb0fc, flags=0,
imo=0x0, inp=0xd0bfb0c0) at /usr/src/sys/netinet/ip_output.c:733
#17 0xc01afc51 in tcp_output (tp=0xd0bfb180)
at /usr/src/sys/netinet/tcp_output.c:953
#18 0xc01ae977 in tcp_input (m=0xc0b26b00, off0=20, proto=6)
at /usr/src/sys/netinet/tcp_input.c:2229
#19 0xc01a8f1c in ip_input (m=0xc0b26b00)
at /usr/src/sys/netinet/ip_input.c:934
#20 0xc01a8f7b in ipintr () at /usr/src/sys/netinet/ip_input.c:955
#21 0xc01fbd89 in swi_net_next ()
#22 0xc0156a69 in softclock () at /usr/src/sys/kern/kern_timeout.c:131
#23 0xc01fbd43 in doreti_swi ()
So, free is called from the following fragment:
/** * lookup a dynamic rule. */ static ipfw_dyn_rule * lookup_dyn_rule(struct ipfw_flow_id *pkt, int *match_direction, struct tcphdr *tcp) { /* * stateful ipfw extensions. * Lookup into dynamic session queue */ #define MATCH_REVERSE 0 #define MATCH_FORWARD 1 #define MATCH_NONE 2 #define MATCH_UNKNOWN 3 int i, dir = MATCH_NONE; ipfw_dyn_rule *prev, *q=NULL;
if (ipfw_dyn_v == NULL) goto done; /* not found */ i = hash_packet( pkt ); for (prev=NULL, q = ipfw_dyn_v[i] ; q != NULL ; ) { if (q->dyn_type == O_LIMIT_PARENT && q->count) goto next; if (TIME_LEQ( q->expire, time_second)) { /* expire entry */ => UNLINK_DYN_RULE(prev, ipfw_dyn_v[i], q); continue; } if (pkt->proto == q->id.proto && q->dyn_type != O_LIMIT_PARENT) {
I'm no kernel expert, so take my observation for what they might be worth, but:
(kgdb) p *q
$24 = {next = 0xc18a2d00, rule = 0xc6523b3c, parent = 0xd0001,
pcnt = 13916504069872025600, bcnt = 11709303859986432, id = {dst_ip = 0,
src_ip = 0, dst_port = 15744, src_port = 49469, proto = 164 '\244',
flags = 129 '\201'}, expire = 0, bucket = 4294967295, state = 4294967295,
ack_fwd = 0, ack_rev = 0, dyn_type = 0, count = 0}
(kgdb)
These values do not make much sense to me... maybe the mess has already happened?
Any hint? Is ipfw2 known to be broken in 4_10? Should I upgrade to 4_STABLE? Or is it just a matter of finding a better configuration for all the relevant sysctl (which are all set to their default values)?
Really any help is appreciated!!!
bye & Thanks
av.
smime.p7s
Description: S/MIME Cryptographic Signature
