My apologies. Here's the text version. Thanks.
Hi,
I'm not sure if anyone's seen this before. I have an issue on my Solaris 9 box running ipf where the system panics when I unplumb a qfe interface. I just recently upgraded from ipf 3.x to 4.1.3. Here's the stack trace of the core dump from SUNWscat:
SolarisCAT(vmcore.3)> analyze PANIC: %s, lp=%p wwwh=%lx thread=%p
core file: /var/crash/alyssa/vmcore.3
user: Super-User (root:0)
release: 5.9 (64-bit)
version: Generic_117171-12
machine: sun4u
node name: alyssa
domain: cats
hw_provider: Sun_Microsystems
system type: SUNW,Ultra-60
hostid: 80c949b1
time of crash: Fri Dec 24 00:31:21 EST 2004
age of system: 2 minutes 18.63 seconds
panic cpu: 0 (ncpus: 2)
panic string: recursive rw_enter, lp=78020df0 wwwh=300043567e4 thread=30004356
7e0
2 cpus
==== printing for generic panic information ====
cpu 0 had the panic
==== panic thread: 0x300043567e0 ==== cpu: 0 ====
==== panic user thread: 0x300043567e0 pid: 668 on cpu: 0 (last_swtch: -2m18.63
s) ====
cmd: ifconfig qfe0 unplumb
t_stk: 0x2a1003cbaf0 sp: 0x1437c71 t_stkbase: 0x2a1003c8000
t_pri: 54(TS) pctcpu: 0.048828 t_lwp: 0x3000434eab0 machpcb: 0x2a1003cbaf0
t_procp: 0x300043f4ad0 p_as: 0x300041ed3a8 hat: 0x3000005c878 cnum: 0x1119
size: 2138112 rss: 1327104
last cpuid: 0 idle: 0 ticks (0 seconds)
start: Fri Dec 24 00:31:21 2004
age: 0 seconds (0 seconds)
stime: 13863 (0 seconds later)
syscall: ioctl (0x45c0e, 0x45998, 0x45aac)
tstate: TS_ONPROC - thread is being run on a processor
tflg: T_PANIC - thread initiated a system panic
tpflg: TP_TWAIT - wait to be freed by lwp_wait
tsched: TS_LOAD - thread is in memory
TS_DONT_SWAP - thread/LWP should not be swapped
pflag: SLOAD - in core
pc: 0x104bd3c unix:panicsys+0x44: call unix:setjmp
unix:panicsys+0x44(0x105d4e8, 0x2a1003cac68, 0x1438640, 0x1, , , 0x4480001603, ,
, , , , , , 0x105d4e8, 0x2a1003cac68)
unix:vpanic+0xcc(0x105d4e8, 0x2a1003cac68)
unix:panic+0x1c(, 0x1414960, 0x78020df0, 0x300043567e4, 0x300043567e0, 0x3000435
67ec)
unix:rw_enter - frame recycled
ipf:get_unit+0x48(0x30001129520, 0x4)
ipf:frsynclist+0xb8(0x30001129410)
ipf:frsync+0x60()
ipf:fr_qifsync+0x18(0x0, 0x0, 0x30000f69880, 0x1, 0x30000f69880, 0x0)
pfil:qif_delete+0x1d4(0x30000f69880, 0x300041d07c8)
pfil:pfilmodclose+0x84(0x300041d07c8, 0x3, 0x300004fff28)
genunix:qdetach+0xa8(0x300041d08b8?, 0x1, 0x3?, 0x300004fff28, 0x0)
genunix:strclose+0x3cc(0x30003dca698, 0x3, 0x300004fff28)
specfs:device_close+0x98(0x30003dca7b0, 0x3, 0x300004fff28)
specfs:spec_close+0x128(0x30003dca7b0, 0x3, 0x1, 0x0, 0x300004fff28)
genunix:closef+0x54(0x30003ea32d0)
genunix:munlink+0x340(0x30004352930, 0x30000c23da8?, , 0x300004ff3e8, 0x2a1003cb
aec)
genunix:strioctl+0x27c0(0x300042f68c8, 0x5317, 0x12, , 0x1, 0x300004ff3e8, 0x2a1
003cbaec)
specfs:spec_ioctl+0x94(0x300042f68c8, 0x5317, 0x12, 0x100003, , 0x2a1003cbaec)
genunix:ioctl+0x1f8()
unix:syscall_trap32+0xa8()
-- switch to user thread's user stack --
This only happens when I bring the interface to my cable modem down. I do this with a script that runs the following sequence of commands:
ifconfig qfe0 dhcp release ifconfig qfe0 unplumb ipf -Fa ipnat -C ipnat -F ndd -set /dev/ip ip_forwarding 0
I've tried flushing the ipfilter and NAT tables before unplumbing the interface, but the system still panics. I probably don't really need to unplumb the interface, but this probably shouldn't happen, and it didn't happen with 3.x. I assume the problem is with ipf since that's the last module it was in before the panic. Also, unplumbing an interface dissociates the streams modules from the interface, and 4.1.3 seems to be different than 3.x in that it inserts the pfil module into the stream. It looks like it doesn't like to be pulled out. Just speculation on my part, I'll defer to the experts.
Everything runs fine as long as I don't try to unplumb qfe0. Also, I updated my system with the latest patch bundle after the first time it happened. It didn't seem to help.
Here's the vital info for my machine. If you need a core dump, you can download one from this URL: http://68.192.99.127:90/files/ipf-core.tar.bz2 (11MB)
********************
uname -a
********************
SunOS alyssa 5.9 Generic_117171-12 sun4u sparc SUNW,Ultra-60
********************
isainfo -vk
********************
64-bit sparcv9 kernel modules
********************
ifconfig -a
********************
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 3
inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 192.168.3.254 netmask ffffff00 broadcast 192.168.3.255
ether 8:0:20:c9:49:b1
qfe1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5
inet 192.168.4.254 netmask ffffff00 broadcast 192.168.4.255
ether 8:0:20:c9:49:b1
qfe0: flags=100c843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,PRIVATE,IPv4> mtu 1500 index 6
inet 68.192.99.127 netmask fffff000 broadcast 68.192.111.255
ether 8:0:20:c9:49:b1
********************
netstat -rn
********************
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ------ ---------
192.168.3.0 192.168.3.254 U 1 75 hme0
192.168.4.0 192.168.4.254 U 1 0 qfe1
68.192.96.0 68.192.99.127 U 1 1 qfe0
224.0.0.0 192.168.3.254 U 1 0 hme0
default 68.192.96.1 UG 1 50 127.0.0.1 127.0.0.1 UH 4 85 lo0
********************
netstat -i
********************
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
lo0 8232 loopback localhost 1289 0 1289 0 0 0 hme0 1500 alyssa alyssa 2432 0 2302 0 0 0 qfe1 1500 alyssa-4 alyssa-4 0 0 57 0 0 0 qfe0 1500 ool-44c0637f.dyn.optonline.net ool-44c0637f.dyn.optonline.net 12931 0 1402 0 0 0
******************** netstat -s -P ip ********************
IPv4 ipForwarding = 1 ipDefaultTTL = 255
ipInReceives = 3687 ipInHdrErrors = 0
ipInAddrErrors = 0 ipInCksumErrs = 0
ipForwDatagrams = 1419 ipForwProhibits = 0
ipInUnknownProtos = 12 ipInDiscards = 0
ipInDelivers = 3444 ipOutRequests = 1915
ipOutDiscards = 0 ipOutNoRoutes = 9
ipReasmTimeout = 60 ipReasmReqds = 0
ipReasmOKs = 0 ipReasmFails = 0
ipReasmDuplicates = 0 ipReasmPartDups = 0
ipFragOKs = 0 ipFragFails = 0
ipFragCreates = 0 ipRoutingDiscards = 0
tcpInErrs = 0 udpNoPorts = 0
udpInCksumErrs = 0 udpInOverflows = 108
rawipInOverflows = 0 ipsecInSucceeded = 0
ipsecInFailed = 0 ipInIPv6 = 0
ipOutIPv6 = 0 ipOutSwitchIPv6 = 16********************
ipf -V
********************
ipf: IP Filter: v4.1.3 (592)
Kernel: IP Filter: v4.1.3 Running: yes
Log Flags: 0 = none set
Default: pass all, Logging: available
Active list: 1
Feature mask: 0x187
********************
ipfstat
********************
bad packets: in 0 out 0
IPv6 packets: in 0 out 0
input packets: blocked 2 passed 3539 nomatch 1390 counted 0 short 0
output packets: blocked 4 passed 3319 nomatch 263 counted 0 short 0
input packets logged: blocked 1 passed 0
output packets logged: blocked 0 passed 0
packets logged: input 0 output 0
log failures: input 0 output 0
fragment state(in): kept 0 lost 0 not fragmented 0
fragment state(out): kept 0 lost 0 not fragmented 0
packet state(in): kept 3 lost 0
packet state(out): kept 85 lost 4
ICMP replies: 0 TCP RSTs sent: 0
Invalid source(in): 0
Result cache hits(in): 312 (out): 944
IN Pullups succeeded: 0 failed: 0
OUT Pullups succeeded: 0 failed: 0
Fastroute successes: 0 failures: 0
TCP cksum fails(in): 0 (out): 0
IPF Ticks: 2171
Packet log flags set: (0)
none
********************
ipfstat -io
********************
pass out quick on qfe0 proto tcp from any to any flags S/FSRPAU keep state keep frags
pass out quick on qfe0 proto udp from any to any keep state
pass out quick on qfe0 proto icmp from any to any keep state
pass in quick on qfe0 proto tcp from any to any port = ssh flags S/FSRPAU keep state
pass in quick on qfe0 proto tcp from any to any port = 90 flags S/FSRPAU keep state
block in log quick on qfe0 proto tcp from any to 68.192.99.127/32
block in log quick on qfe0 proto udp from any to 68.192.99.127/32
block in quick on qfe0 all
********************
ipnat -slv
********************
mapped in 694 out 725
added 21 expired 0
no memory 0 bad nat 0
inuse 11
rules 6
wilds 0
table ffffffff7ffff998 list 30003180f40
List of active MAP/Redirect filters:
map qfe0 192.168.3.0/24 -> 68.192.99.127/32 proxy port ftp ftp/tcp
map qfe0 192.168.3.0/24 -> 68.192.99.127/32 portmap tcp/udp 40000:60000
map qfe0 192.168.3.0/24 -> 68.192.99.127/32
map qfe0 192.168.4.0/24 -> 68.192.99.127/32 proxy port ftp ftp/tcp
map qfe0 192.168.4.0/24 -> 68.192.99.127/32 portmap tcp/udp 40000:60000
map qfe0 192.168.4.0/24 -> 68.192.99.127/32
List of active sessions:
MAP 192.168.3.11 1674 <- -> 68.192.99.127 40020 [69.25.38.107 80]
age 2382 use 0 sumd 0x7a56/0x7a56 pr 6 bkt 1706/678 flags 1
ifp qfe0,qfe0 bytes 306/560 pkts 4/5 ipsumd e48b
MAP 192.168.3.11 1673 <- -> 68.192.99.127 40019 [64.14.128.200 80]
age 2214 use 0 sumd 0x7a56/0x7a56 pr 6 bkt 1769/741 flags 1
ifp qfe0,qfe0 bytes 5917/980 pkts 7/7 ipsumd e48b
MAP 192.168.3.11 1672 <- -> 68.192.99.127 40018 [216.73.87.58 80]
age 2213 use 0 sumd 0x7a56/0x7a56 pr 6 bkt 1012/1007 flags 1
ifp qfe0,qfe0 bytes 1355/741 pkts 3/5 ipsumd e48b
MAP 192.168.3.11 1671 <- -> 68.192.99.127 40017 [216.73.87.58 80]
age 2213 use 0 sumd 0x7a56/0x7a56 pr 6 bkt 1011/1006 flags 1
ifp qfe0,qfe0 bytes 708/751 pkts 3/5 ipsumd e48b
MAP 192.168.3.11 1670 <- -> 68.192.99.127 40016 [216.73.87.58 80]
age 2182 use 0 sumd 0x7a56/0x7a56 pr 6 bkt 1010/1005 flags 1
ifp qfe0,qfe0 bytes 686/744 pkts 3/5 ipsumd e48b
MAP 192.168.3.11 1669 <- -> 68.192.99.127 40015 [216.73.87.58 80]
age 2181 use 0 sumd 0x7a56/0x7a56 pr 6 bkt 1009/1004 flags 1
ifp qfe0,qfe0 bytes 708/754 pkts 3/5 ipsumd e48b
MAP 192.168.3.11 1668 <- -> 68.192.99.127 40014 [199.106.73.26 80]
age 2270 use 0 sumd 0x7a56/0x7a56 pr 6 bkt 920/915 flags 1
ifp qfe0,qfe0 bytes 94165/8125 pkts 76/55 ipsumd e48b
MAP 192.168.3.11 1667 <- -> 68.192.99.127 40013 [216.73.87.58 80]
age 2173 use 0 sumd 0x7a56/0x7a56 pr 6 bkt 1007/1002 flags 1
ifp qfe0,qfe0 bytes 651/676 pkts 3/5 ipsumd e48b
MAP 192.168.3.11 1666 <- -> 68.192.99.127 40012 [216.73.87.58 80]
age 2173 use 0 sumd 0x7a56/0x7a56 pr 6 bkt 1006/1001 flags 1
ifp qfe0,qfe0 bytes 607/686 pkts 3/5 ipsumd e48b
MAP 192.168.3.11 1660 <- -> 68.192.99.127 40006 [199.106.73.93 80]
age 2210 use 0 sumd 0x7a56/0x7a56 pr 6 bkt 979/974 flags 1
ifp qfe0,qfe0 bytes 28649/4218 pkts 26/20 ipsumd e48b
MAP 192.168.3.31 4500 <- -> 68.192.99.127 40002 [192.18.98.40 4500]
age 3370 use 0 sumd 0x6f26/0x6f26 pr 17 bkt 107/1332 flags 2
ifp qfe0,qfe0 bytes 368784/73222 pkts 530/560 ipsumd e477List of active host mappings: 192.168.3.11,199.106.73.26 -> 68.192.99.127 (use = 1 hv = 776) 192.168.3.11,199.106.73.93 -> 68.192.99.127 (use = 1 hv = 912) 192.168.3.11,216.73.87.58 -> 68.192.99.127 (use = 6 hv = 915) 192.168.3.11,64.14.128.200 -> 68.192.99.127 (use = 1 hv = 1340) 192.168.3.31,192.18.98.40 -> 68.192.99.127 (use = 1 hv = 1820) 192.168.3.11,69.25.38.107 -> 68.192.99.127 (use = 1 hv = 1869)
Let me know if there's anything else you need.
Thanks, John
-- John Cecere Sun Microsystems 732-302-3922 / [EMAIL PROTECTED]
