Re: [BUG][debian-2.6.20-1-686] bridging + vlans + "vconfig rem" == stuck kernel
On May 11, 2007, at 01:49:27, Kyle Moffett wrote:
On May 10, 2007, at 00:34:11, Kyle Moffett wrote:
On May 10, 2007, at 00:25:54, Ben Greear wrote:
Looks like a deadlock in the vlan code. Any chance you can run
this test with lockdep enabled?
You could also add a printk in vlan_device_event() to check which
event it is hanging on, and the netdevice that is passed in.
Ok, I'll try building a 2.6.21 kernel with lockdep and some
debugging printk()s in the vlan_device_event() function and get
back to you tomorrow. Thanks for the quick response!
[snip]
ifup -a brings up the interfaces in this order (See previous email
for configuration details):
lo net0 wfi0 world0 lan lan:0 world
ifdown -a appears to bring them down in the same order (at least,
until it gets stuck).
Hmm, turns out that it always hung downing this entry in my
interfaces file, independent of ordering:
iface world0 inet manual
mac-address 8b:8d:cb:91:e2:4c
minimally-up yes
vlan-dev net0
vlan-id 4094
By commenting out the MAC address line it worked. Yes, I realize the
MAC address specified there is bogus, I managed to {think,type}o that
one somehow. I had been intending to specify a locally-allocated
virtual MAC address on world0 but instead I managed to somehow assign
one with the MAC multicast bit set (01:00:00:00:00:00)
If I change the above garbage MAC to 02:00:00:00:00:01 (first 02 is
the locally-administrated bit) then it seems to work perfectly fine,
My guess that the bridging code doesn't properly drop all references
to world0 when it has that garbage MAC address on it (since the
problem only shows up when both the invalid mac-address is present
*AND* I start the "world" bridge). I suppose this isn't really a big
problem, but it would be nice if things didn't leak refcounts on
invalid input.
Cheers,
Kyle Moffett
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [BUG][debian-2.6.20-1-686] bridging + vlans + "vconfig rem" == stuck kernel
On May 10, 2007, at 00:34:11, Kyle Moffett wrote: On May 10, 2007, at 00:25:54, Ben Greear wrote: Looks like a deadlock in the vlan code. Any chance you can run this test with lockdep enabled? You could also add a printk in vlan_device_event() to check which event it is hanging on, and the netdevice that is passed in. Ok, I'll try building a 2.6.21 kernel with lockdep and some debugging printk()s in the vlan_device_event() function and get back to you tomorrow. Thanks for the quick response! Progress!!! I built a 2.6.21.1 kernel with a 1MB dmesg buffer, almost all of the locking debugging options on (as well as a few others just for kicks), a VLAN debug #define turned on in the net/ 8021q/vlan.h file, and lots of extra debugging messages added to the functions in vlan.c. My initial interpretation is that due to the funny order in which "ifdown -a" takes down interfaces, it tries to delete the VLAN interfaces before the bridges running atop them have been taken down. Ordinarily this seems to work, but when the underlying physical ethernet is down already, the last VLAN to be deleted seems to hang somehow. The full results are as follows: The lock dependency validator at startup passes all 218 testcases, indicating that all the locking crap is probably working correctly (those debug options chew up another meg of RAM). ifup -a brings up the interfaces in this order (See previous email for configuration details): lo net0 wfi0 world0 lan lan:0 world ifdown -a appears to bring them down in the same order (at least, until it gets stuck). Attached below is filtered debugging information. I cut out 90% of the crap in the syslog, but there's still a lot left over to sift through; sorry. If you want my .config or the full text of the log then email me privately and I'll send it to you, as it's kinda big. I appreciate any advice, thanks for all your help Cheers, Kyle Moffett This first bit is the "ifup -a -v -i interfaces": ADDRCONF(NETDEV_UP): net0: link is not ready vlan_ioctl_handler: args.cmd: 6 vlan_ioctl_handler: args.cmd: 0 register_vlan_device: if_name -:net0:-^Ivid: 2 About to allocate name, vlan_name_type: 3 Allocated new name -:net0.2:- About to go find the group for idx: 2 vlan_transfer_operstate: net0 state transition applies to net0.2 too: vlan_proc_add, device -:net0.2:- being added. Allocated new device successfully, returning. wfi0: add 33:33:00:00:00:01 mcast address to master interface wfi0: add 01:00:5e:00:00:01 mcast address to master interface ADDRCONF(NETDEV_UP): wfi0: link is not ready vlan_ioctl_handler: args.cmd: 6 vlan_ioctl_handler: args.cmd: 0 register_vlan_device: if_name -:net0:-^Ivid: 4094 About to allocate name, vlan_name_type: 3 Allocated new name -:net0.4094:- About to go find the group for idx: 2 vlan_transfer_operstate: net0 state transition applies to net0.4094 too: vlan_proc_add, device -:net0.4094:- being added. Allocated new device successfully, returning. world0: add 33:33:00:00:00:01 mcast address to master interface world0: add 01:00:5e:00:00:01 mcast address to master interface ADDRCONF(NETDEV_UP): world0: link is not ready tg3: net0: Link is up at 1000 Mbps, full duplex. tg3: net0: Flow control is on for TX and on for RX. ADDRCONF(NETDEV_CHANGE): net0: link becomes ready Propagating NETDEV_CHANGE for device net0... ... to wfi0 vlan_transfer_operstate: net0 state transition applies to wfi0 too: ...found a carrier, applying to VLAN device ... to world0 vlan_transfer_operstate: net0 state transition applies to world0 too: ...found a carrier, applying to VLAN device lan: port 1(net0) entering listening state ADDRCONF(NETDEV_CHANGE): wfi0: link becomes ready wfi0: dev_set_promiscuity(master, 1) wfi0: add 33:33:ff:5f:60:92 mcast address to master interface lan: port 2(wfi0) entering listening state ADDRCONF(NETDEV_CHANGE): world0: link becomes ready world0: add 33:33:ff:91:e2:4c mcast address to master interface lan: no IPv6 routers present world: no IPv6 routers present net0: no IPv6 routers present world0: no IPv6 routers present wfi0: no IPv6 routers present lan: port 1(net0) entering learning state lan: port 2(wfi0) entering learning state lan: topology change detected, propagating lan: port 1(net0) entering forwarding state lan: topology change detected, propagating lan: port 2(wfi0) entering forwarding state This bit is for "ifdown -a -v -i interfaces": Propagating NETDEV_DOWN for device net0... ... to wfi0 wfi0: del 33:33:ff:5f:60:92 mcast address from vlan interface wfi0: del 33:33:ff:5f:60:92 mcast address from master interface wfi0: del 01:00:5e:00:00:01 mcast address from vlan interface wfi0: del 01:00:5e:00:00:01 mcast address from master interface wfi0: del 33:33:00:00:00:01 mcast address from vlan interface wfi0: del 33:33:00:00:00:01 mcast address from master interface lan: port 2(wfi0) entering disabled state ... to world0 world
Re: [BUG][debian-2.6.20-1-686] bridging + vlans + "vconfig rem" == stuck kernel
On May 10, 2007, at 00:25:54, Ben Greear wrote: Kyle Moffett wrote: vconfig D 83CCD8CE 0 16564 16562 (NOTLB) efdd7e7c 0086 ee120afb 83ccd8ce 98f00788 b7083ffa 5384b49a c76c0b05 9ebaf791 0004 efdd7e4e 0007 f1468a90 2ab74174 0362 0326 f1468b9c c180e420 0001 0286 c012933c efdd7e8c df98a000 c180e468 Call Trace: [] lock_timer_base+0x15/0x2f [] __mod_timer+0x91/0x9b [] schedule_timeout+0x70/0x8d [] vlan_device_event+0x13/0xf8 [8021q] Looks like a deadlock in the vlan code. Any chance you can run this test with lockdep enabled? You could also add a printk in vlan_device_event() to check which event it is hanging on, and the netdevice that is passed in. Ok, I'll try building a 2.6.21 kernel with lockdep and some debugging printk()s in the vlan_device_event() function and get back to you tomorrow. Thanks for the quick response! Since the vlan code holds RTNL at this point, then most other network tasks will eventually hang as well. Well, it's less of an "eventually" and more of an "almost immediately". When that happens pretty close to everything more complicated than socket(), bind(), and connect() with straightforward UNIX or INET sockets tends to stick completely. Thanks again! Cheers, Kyle Moffett - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [BUG][debian-2.6.20-1-686] bridging + vlans + "vconfig rem" == stuck kernel
Kyle Moffett wrote: vconfig D 83CCD8CE 0 16564 16562 (NOTLB) efdd7e7c 0086 ee120afb 83ccd8ce 98f00788 b7083ffa 5384b49a c76c0b05 9ebaf791 0004 efdd7e4e 0007 f1468a90 2ab74174 0362 0326 f1468b9c c180e420 0001 0286 c012933c efdd7e8c df98a000 c180e468 Call Trace: [] lock_timer_base+0x15/0x2f [] __mod_timer+0x91/0x9b [] schedule_timeout+0x70/0x8d [] vlan_device_event+0x13/0xf8 [8021q] Looks like a deadlock in the vlan code. Any chance you can run this test with lockdep enabled? You could also add a printk in vlan_device_event() to check which event it is hanging on, and the netdevice that is passed in. Since the vlan code holds RTNL at this point, then most other network tasks will eventually hang as well. Thanks, Ben -- Ben Greear <[EMAIL PROTECTED]> Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[BUG][debian-2.6.20-1-686] bridging + vlans + "vconfig rem" == stuck kernel
I've managed to fairly reliably trigger a deadlock in some portion of
the linux networking code on my Debian test box (using the debian
kernel linux-image-2.6.20-1-686). I'm pretty sure that it's a race
condition of some sort as it doesn't trigger if I ifdown the
interfaces one by one, but if I run "ifdown -a" then it triggers
halfway through reliably (although not with the same reference count
numbers, once it was 6, this time it was 2).
The message I get is: "unregister_netdevice: waiting for world0 to
become free. Usage count = 2"
My /etc/network/interfaces file uses a couple custom-made if-pre-up.d
and if-post-down.d scripts to set up the bridges and VLANs a little
more cleanly than the standard debian scripts do, but the general
configuration is as follows:
net0: Tigon-3 onboard gigabit NIC, hooked to managed switch, untagged
packets
wfi0: (net0.2 before renaming) WAP-connected VLAN 2 on the managed
switch
world0: (net0.4094 before renaming) Internet connection, runs DHCP
lan: Local Area Network bridge of "net0" and "wfi0" (current box has
lowest STP priority) This will eventually also have another untagged-
only ethernet port attached to it.
lan:0: Alias for acting as primary nameserver
world: pseudo-bridge of "world0" for highly-available DHCP client.
Just for a bit of background on why this is so complex: When I get
this networking problem sorted out I'm going to set up heartbeat and
a dummy "world1" interface with a shared MAC which is added to the
"world" bridge when the current system is the DHCP-client master.
Hopefully that way I can have 3 systems act as a highly-available
router. Whichever one is currently the master will add the dummy
world1 interface with shared MAC address to its "world" bridge and
bring dhcp3-client up on the "world1" interface with the latest copy
of the leases file from the previous master (even if the previous
master mysteriously crashed. This should keep my public IP from
changing unnecessarily and should allow me to reboot any one of the 3
router/firewall/mailserver/fileserver/etc systems without adversely
affecting the internet connection or other critical services.
Eventually I plan to add ebtables to help filter traffic between wfi*
interfaces and untagged VLAN-1 net* interfaces, but for now there's
no filtering.
Anyways, after the unregister_netdevice messages start popping up all
sorts of networking related things start to hang hard in the kernel;
I can't "ip link set down" any interfaces, etc. I've captured sysrq
dumps of the process state on the system at the time with most
processes. See the attached syslog.bz2 file.
The important part is probably the stuck "vconfig" process, but I
don't really understand enough about the timer stuff to interpret
that backtrace.
vconfig D 83CCD8CE 0 16564 16562 (NOTLB)
efdd7e7c 0086 ee120afb 83ccd8ce 98f00788 b7083ffa
5384b49a c76c0b05
9ebaf791 0004 efdd7e4e 0007 f1468a90 2ab74174
0362 0326
f1468b9c c180e420 0001 0286 c012933c efdd7e8c
df98a000 c180e468
Call Trace:
[] lock_timer_base+0x15/0x2f
[] __mod_timer+0x91/0x9b
[] schedule_timeout+0x70/0x8d
[] vlan_device_event+0x13/0xf8 [8021q]
[] process_timeout+0x0/0x5
[] msleep+0x1a/0x1f
[] netdev_run_todo+0x10f/0x203
[] vlan_ioctl_handler+0x4dc/0x594 [8021q]
[] sock_ioctl+0x145/0x1be
[] sock_ioctl+0x0/0x1be
[] do_ioctl+0x1f/0x62
[] vfs_ioctl+0x244/0x256
[] sys_ioctl+0x4c/0x64
[] syscall_call+0x7/0xb
There's also a whole bunch of processes stuck in netdev_run_todo()
trying to lock a mutex. This even frustratingly affects things like
rndc which use netlink sockets:
rpc.mountdD C9F2BD14 0 4351 1 4425 4229 (NOTLB)
c9f2bd28 0086 0002 c9f2bd14 c9f2bd10
10ff 46422e36
0002 0202 0007 ed266030 495bcd12
03a5 00013461
ed26613c c180e420 0001 dffc8200 dfaeb580
10ff df99ce00
Call Trace:
[] __mutex_lock_slowpath+0x48/0x77
[] mutex_lock+0xa/0xb
[] netdev_run_todo+0x10/0x203
[] netlink_run_queue+0x9f/0xbe
[] rtnetlink_rcv_msg+0x0/0x1d9
[] rtnetlink_rcv+0x34/0x3d
[] netlink_data_ready+0x12/0x4c
[] netlink_sendskb+0x19/0x30
[] netlink_sendmsg+0x26a/0x276
[] sock_sendmsg+0xd0/0xeb
[] autoremove_wake_function+0x0/0x35
[] extract_entropy+0x45/0x89
[] __wake_up+0x32/0x43
[] netlink_insert+0x10f/0x119
[] sys_sendto+0x116/0x140
[] find_get_page+0x18/0x41
[] filemap_nopage+0x197/0x2f9
[] sys_getsockname+0x86/0xb0
[] __handle_mm_fault+0x2ee/0x7d4
[] sys_socketcall+0x15e/0x242
[] syscall_call+0x7/0xb
The "zz-km-vlan" script is the bash if-post-down.d script in charge
of disassembling VLAN interfaces. There is an equivalent "zz-km-
bridge" script for bridge interfaces, as well as if-pre-up.d scripts
called "00-km-vlan" and "00-km-bridge" to create the interfaces.
If anyone has any suggestions, p
