Bug#577640: linux-image-2.6.33-2-amd64: Kernel warnings in netns thread
On Thu, Apr 22, 2010 at 04:38, Eric W. Biederman ebied...@xmission.com wrote:
$ sudo ./startns bash
# ip l a type veth
# ip l s veth0 netns 1
# exit
Then I should ask what is startns?
That's just a simple C program that calls unshare(CLONE_NEWNET) and
execs a shell.
Either that is doing something different from my equivalent program, or I have
patches to fix this, that just haven't been merged yet.
I have just downloaded and compiled 2.6.32-2 and 2.6.34-rc5 from
kernel.org using the .config from the debian package, and the oops is
reproducible in both.
This small C file reproduces the error every time:
$ cat netnsoops.c
#include stdio.h
#include stdlib.h
#define _GNU_SOURCE
#include sched.h
int main(int argc, char *argv[])
{
int c;
unsigned long flags = CLONE_NEWNET;
if(unshare(flags) == -1) {
perror(unshare);
return 1;
}
system(ip link add name FOO type veth peer name BAR);
system(ip link set FOO netns 1);
system(ip link show);
return 0;
}
--
Martín Ferrari
--
To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/y2kb9800b71004220705s4db40c4csb3923b61f9ae8...@mail.gmail.com
Bug#577640: linux-image-2.6.33-2-amd64: Kernel warnings in netns thread
I'm not starting a new thread/bug, as this is probably related... I just discovered that in 2.6.33, if I create a veth inside a namespace and then move one of the halves into the main namespace, when I kill the namespace, I get one of these warnings followed by an oops. This does not happen if the veth is created from the main ns and then moved, nor in 2.6.32. This happens both in Qemu and on real hardware (both amd64) To reproduce: $ sudo ./startns bash # ip l a type veth # ip l s veth0 netns 1 # exit The full log: [ 280.655876] [ cut here ] [ 280.655916] WARNING: at /build/mattems-linux-2.6_2.6.33-1~experimental.4-amd64-ieqSsa/linux-2.6-2.6.33-1~experimental.4/debian/build/source_amd64_none/fs/sysfs/dir.c:487 sysfs_add_one+0xcc/0xe4() [ 280.655920] Hardware name: [ 280.655922] sysfs: cannot create duplicate filename '/devices/virtual/net/veth0/statistics' [ 280.655924] Modules linked in: veth loop parport_pc parport snd_pcm evdev snd_timer tpm_tis button serio_raw snd i2c_piix4 tpm soundcore tpm_bios processor snd_page_alloc psmouse i2c_core pcspkr ext3 jbd mbcache sg sr_mod cdrom sd_mod crc_t10dif ata_generic ata_piix libata thermal floppy thermal_sys 8139cp 8139too mii scsi_mod [last unloaded: scsi_wait_scan] [ 280.655954] Pid: 1276, comm: ip Not tainted 2.6.33-2-amd64 #1 [ 280.655956] Call Trace: [ 280.655961] [8113a399] ? sysfs_add_one+0xcc/0xe4 [ 280.655964] [8113a399] ? sysfs_add_one+0xcc/0xe4 [ 280.655987] [81046b81] ? warn_slowpath_common+0x77/0xa3 [ 280.655991] [81046c09] ? warn_slowpath_fmt+0x51/0x59 [ 280.655994] [8113a2c5] ? sysfs_pathname+0x35/0x3d [ 280.655997] [8113a2c5] ? sysfs_pathname+0x35/0x3d [ 280.656000] [8113a2c5] ? sysfs_pathname+0x35/0x3d [ 280.656007] [8113a2c5] ? sysfs_pathname+0x35/0x3d [ 280.656031] [8113a399] ? sysfs_add_one+0xcc/0xe4 [ 280.656035] [8113aac5] ? create_dir+0x4f/0x7c [ 280.656038] [8113baac] ? internal_create_group+0x51/0x15a [ 280.656062] [8120f3ba] ? device_add_groups+0x22/0x5d [ 280.656066] [8120faba] ? device_add+0x2d1/0x54a [ 280.656085] [81243d22] ? dev_change_net_namespace+0x1a1/0x1fc [ 280.656097] [8124b625] ? do_setlink+0x67/0x356 [ 280.656101] [8124c452] ? rtnl_newlink+0x2f4/0x4f3 [ 280.656104] [8124c203] ? rtnl_newlink+0xa5/0x4f3 [ 280.656107] [8123cdcf] ? __skb_recv_datagram+0x11a/0x24f [ 280.656116] [8123c040] ? memcpy_toiovec+0x3d/0x6d [ 280.656124] [8125d39d] ? netlink_sendmsg+0x15f/0x252 [ 280.656127] [8124bf69] ? rtnetlink_rcv_msg+0x0/0x1f5 [ 280.656130] [8125cf55] ? netlink_rcv_skb+0x34/0x7c [ 280.656133] [8124bf63] ? rtnetlink_rcv+0x1f/0x25 [ 280.656136] [8125cd49] ? netlink_unicast+0xe2/0x148 [ 280.656139] [8125d47d] ? netlink_sendmsg+0x23f/0x252 [ 280.656142] [81232ea3] ? sock_sendmsg+0x83/0x9b [ 280.656162] [810b5122] ? __alloc_pages_nodemask+0x10f/0x5e1 [ 280.656166] [8123bd03] ? copy_from_user+0x13/0x25 [ 280.656169] [8123c0b9] ? verify_iovec+0x49/0x84 [ 280.656172] [81233178] ? sys_sendmsg+0x225/0x2af [ 280.656176] [812343c7] ? sys_recvmsg+0x48/0x56 [ 280.656188] [81008ac2] ? system_call_fastpath+0x16/0x1b [ 280.656191] ---[ end trace f31191528d9325da ]--- [ 280.658197] [ cut here ] [ 280.658202] WARNING: at /build/mattems-linux-2.6_2.6.33-1~experimental.4-amd64-ieqSsa/linux-2.6-2.6.33-1~experimental.4/debian/build/source_amd64_none/net/core/dev.c:5597 dev_change_net_namespace+0x1b6/0x1fc() [ 280.658205] Hardware name: [ 280.658207] Modules linked in: veth loop parport_pc parport snd_pcm evdev snd_timer tpm_tis button serio_raw snd i2c_piix4 tpm soundcore tpm_bios processor snd_page_alloc psmouse i2c_core pcspkr ext3 jbd mbcache sg sr_mod cdrom sd_mod crc_t10dif ata_generic ata_piix libata thermal floppy thermal_sys 8139cp 8139too mii scsi_mod [last unloaded: scsi_wait_scan] [ 280.658229] Pid: 1276, comm: ip Tainted: GW 2.6.33-2-amd64 #1 [ 280.658231] Call Trace: [ 280.658235] [81243d37] ? dev_change_net_namespace+0x1b6/0x1fc [ 280.658238] [81243d37] ? dev_change_net_namespace+0x1b6/0x1fc [ 280.658241] [81046b81] ? warn_slowpath_common+0x77/0xa3 [ 280.658245] [81243d37] ? dev_change_net_namespace+0x1b6/0x1fc [ 280.658248] [8124b625] ? do_setlink+0x67/0x356 [ 280.658251] [8124c452] ? rtnl_newlink+0x2f4/0x4f3 [ 280.658254] [8124c203] ? rtnl_newlink+0xa5/0x4f3 [ 280.658257] [8123cdcf] ? __skb_recv_datagram+0x11a/0x24f [ 280.658261] [8123c040] ? memcpy_toiovec+0x3d/0x6d [ 280.658264] [8125d39d] ? netlink_sendmsg+0x15f/0x252 [ 280.658267] [8124bf69] ? rtnetlink_rcv_msg+0x0/0x1f5 [ 280.658270] [8125cf55] ? netlink_rcv_skb+0x34/0x7c [
Bug#577640: linux-image-2.6.33-2-amd64: Kernel warnings in netns thread
Martín Ferrari martin.ferr...@gmail.com writes: I'm not starting a new thread/bug, as this is probably related... I just discovered that in 2.6.33, if I create a veth inside a namespace and then move one of the halves into the main namespace, when I kill the namespace, I get one of these warnings followed by an oops. This does not happen if the veth is created from the main ns and then moved, nor in 2.6.32. This happens both in Qemu and on real hardware (both amd64) To reproduce: $ sudo ./startns bash # ip l a type veth # ip l s veth0 netns 1 # exit Nasty weird. I did a quick test here, and I'm not seeing that. Does the 2.6.33 experimental kernel have any patches applied? The sysfs_add_one path looks like someone we hit a duplicate name, which is a bug of an entirely different kind. From there it appears that we later destroy the device not realizing it isn't in sysfs. Which causes everything to explode. The sysctl issues appears to be that somewhere we have the ordering of creates and/or deletes wrong. It is just possible the changes for batch deletion might be exposing a bug under load. The sysctl error appears to be in the class of things that should never happen but that we should handle correctly anyway. Eric -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/m1ljcgzjh4@fess.ebiederm.org
Bug#577640: linux-image-2.6.33-2-amd64: Kernel warnings in netns thread
On Wed, 2010-04-21 at 12:36 -0700, Eric W. Biederman wrote: Martín Ferrari martin.ferr...@gmail.com writes: I'm not starting a new thread/bug, as this is probably related... I just discovered that in 2.6.33, if I create a veth inside a namespace and then move one of the halves into the main namespace, when I kill the namespace, I get one of these warnings followed by an oops. This does not happen if the veth is created from the main ns and then moved, nor in 2.6.32. This happens both in Qemu and on real hardware (both amd64) To reproduce: $ sudo ./startns bash # ip l a type veth # ip l s veth0 netns 1 # exit Nasty weird. I did a quick test here, and I'm not seeing that. Does the 2.6.33 experimental kernel have any patches applied? Yes, but not many beyond the stable updates, and nothing in this area. You can see the list at: http://svn.debian.org/wsvn/kernel/dists/trunk/linux-2.6/debian/patches/series/base Ben. -- Ben Hutchings Once a job is fouled up, anything done to improve it makes it worse. signature.asc Description: This is a digitally signed message part
Bug#577640: linux-image-2.6.33-2-amd64: Kernel warnings in netns thread
Ben Hutchings b...@decadent.org.uk writes: On Wed, 2010-04-21 at 12:36 -0700, Eric W. Biederman wrote: Martín Ferrari martin.ferr...@gmail.com writes: I'm not starting a new thread/bug, as this is probably related... I just discovered that in 2.6.33, if I create a veth inside a namespace and then move one of the halves into the main namespace, when I kill the namespace, I get one of these warnings followed by an oops. This does not happen if the veth is created from the main ns and then moved, nor in 2.6.32. This happens both in Qemu and on real hardware (both amd64) To reproduce: $ sudo ./startns bash # ip l a type veth # ip l s veth0 netns 1 # exit Nasty weird. I did a quick test here, and I'm not seeing that. Does the 2.6.33 experimental kernel have any patches applied? Yes, but not many beyond the stable updates, and nothing in this area. You can see the list at: http://svn.debian.org/wsvn/kernel/dists/trunk/linux-2.6/debian/patches/series/base Then I should ask what is startns? Either that is doing something different from my equivalent program, or I have patches to fix this, that just haven't been merged yet. Eric -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/m1sk6oky96@fess.ebiederm.org
Bug#577640: linux-image-2.6.33-2-amd64: Kernel warnings in netns thread
On Tue, 2010-04-13 at 13:20 +0200, Martin Ferrari wrote: Package: linux-2.6 Version: 2.6.33-1~experimental.4 Severity: normal Tags: experimental Firstly, please note that I'm running this inside a Qemu, but I imagine that it should not change things much. I installed 2.6.33 to try out the new improvements regarding network namespaces, and while creating and killing hundreds of them, I got many warnings from the kernel that might indicate a bug somewhere. Please see the log already included by reportbug. I'm forwarding this to the upstream maintainers. Ben. [...] [ 6696.035331] [ cut here ] [ 6696.035334] WARNING: at /build/mattems-linux-2.6_2.6.33-1~experimental.4-amd64-ieqSsa/linux-2.6-2.6.33-1~experimental.4/debian/build/source_amd64_none/kernel/sysctl.c:1894 unregister_sysctl_table+0xa6/0xd1() [ 6696.035336] Hardware name: [ 6696.035337] Modules linked in: veth loop parport_pc parport snd_pcm tpm_tis evdev snd_timer i2c_piix4 tpm tpm_bios button processor serio_raw i2c_core snd soundcore snd_page_alloc pcspkr psmouse ext3 jbd mbcache sg sr_mod cdrom sd_mod crc_t10dif ata_generic ata_piix libata floppy 8139cp thermal thermal_sys 8139too mii scsi_mod [last unloaded: veth] [ 6696.035354] Pid: 9, comm: netns Tainted: GW 2.6.33-2-amd64 #1 [ 6696.035355] Call Trace: [ 6696.035357] [8104e303] ? unregister_sysctl_table+0xa6/0xd1 [ 6696.035360] [8104e303] ? unregister_sysctl_table+0xa6/0xd1 [ 6696.035362] [81046b81] ? warn_slowpath_common+0x77/0xa3 [ 6696.035364] [8104e303] ? unregister_sysctl_table+0xa6/0xd1 [ 6696.035367] [812b1205] ? addrconf_ifdown+0x26f/0x2cc [ 6696.035369] [81247edc] ? neigh_sysctl_unregister+0x1a/0x31 [ 6696.035371] [812b1211] ? addrconf_ifdown+0x27b/0x2cc [ 6696.035374] [812b2b0c] ? addrconf_notify+0x714/0x7ea [ 6696.035376] [811eb2e7] ? extract_entropy+0x6a/0x125 [ 6696.035379] [81053aaf] ? lock_timer_base+0x26/0x4b [ 6696.035382] [8123951c] ? skb_dequeue+0x50/0x58 [ 6696.035384] [812482c8] ? pneigh_queue_purge+0x25/0x2f [ 6696.035386] [81249a91] ? neigh_ifdown+0xba/0xc9 [ 6696.035389] [81062f6d] ? notifier_call_chain+0x29/0x4c [ 6696.035392] [81243662] ? rollback_registered_many+0xed/0x19c [ 6696.035394] [8124371f] ? unregister_netdevice_many+0xe/0x57 [ 6696.035397] [812438a3] ? default_device_exit_batch+0x92/0xa3 [ 6696.035399] [8123e97a] ? cleanup_net+0xfd/0x1af [ 6696.035402] [8105bd0d] ? worker_thread+0x188/0x21d [ 6696.035404] [8123e87d] ? cleanup_net+0x0/0x1af [ 6696.035406] [8105f2d2] ? autoremove_wake_function+0x0/0x2e [ 6696.035409] [8105bb85] ? worker_thread+0x0/0x21d [ 6696.035411] [8105ee99] ? kthread+0x79/0x81 [ 6696.035414] [810098e4] ? kernel_thread_helper+0x4/0x10 [ 6696.035416] [8105ee20] ? kthread+0x0/0x81 [ 6696.035418] [810098e0] ? kernel_thread_helper+0x0/0x10 [ 6696.035419] ---[ end trace ef7b93cb006e989e ]--- [...] -- Ben Hutchings Once a job is fouled up, anything done to improve it makes it worse. signature.asc Description: This is a digitally signed message part
Bug#577640: linux-image-2.6.33-2-amd64: Kernel warnings in netns thread
Package: linux-2.6 Version: 2.6.33-1~experimental.4 Severity: normal Tags: experimental Firstly, please note that I'm running this inside a Qemu, but I imagine that it should not change things much. I installed 2.6.33 to try out the new improvements regarding network namespaces, and while creating and killing hundreds of them, I got many warnings from the kernel that might indicate a bug somewhere. Please see the log already included by reportbug. -- Package-specific info: ** Version: Linux version 2.6.33-2-amd64 (Debian 2.6.33-1~experimental.4) (m...@debian.org) (gcc version 4.3.4 (Debian 4.3.4-8) ) #1 SMP Wed Mar 17 18:41:49 UTC 2010 ** Command line: BOOT_IMAGE=/boot/vmlinuz-2.6.33-2-amd64 root=UUID=195c18a2-eda7-40b8-8748-381999d6c854 ro quiet ** Tainted: W (512) * Taint on warning. ** Kernel log: [ 6696.034566] ---[ end trace ef7b93cb006e989d ]--- [ 6696.035331] [ cut here ] [ 6696.035334] WARNING: at /build/mattems-linux-2.6_2.6.33-1~experimental.4-amd64-ieqSsa/linux-2.6-2.6.33-1~experimental.4/debian/build/source_amd64_none/kernel/sysctl.c:1894 unregister_sysctl_table+0xa6/0xd1() [ 6696.035336] Hardware name: [ 6696.035337] Modules linked in: veth loop parport_pc parport snd_pcm tpm_tis evdev snd_timer i2c_piix4 tpm tpm_bios button processor serio_raw i2c_core snd soundcore snd_page_alloc pcspkr psmouse ext3 jbd mbcache sg sr_mod cdrom sd_mod crc_t10dif ata_generic ata_piix libata floppy 8139cp thermal thermal_sys 8139too mii scsi_mod [last unloaded: veth] [ 6696.035354] Pid: 9, comm: netns Tainted: GW 2.6.33-2-amd64 #1 [ 6696.035355] Call Trace: [ 6696.035357] [8104e303] ? unregister_sysctl_table+0xa6/0xd1 [ 6696.035360] [8104e303] ? unregister_sysctl_table+0xa6/0xd1 [ 6696.035362] [81046b81] ? warn_slowpath_common+0x77/0xa3 [ 6696.035364] [8104e303] ? unregister_sysctl_table+0xa6/0xd1 [ 6696.035367] [812b1205] ? addrconf_ifdown+0x26f/0x2cc [ 6696.035369] [81247edc] ? neigh_sysctl_unregister+0x1a/0x31 [ 6696.035371] [812b1211] ? addrconf_ifdown+0x27b/0x2cc [ 6696.035374] [812b2b0c] ? addrconf_notify+0x714/0x7ea [ 6696.035376] [811eb2e7] ? extract_entropy+0x6a/0x125 [ 6696.035379] [81053aaf] ? lock_timer_base+0x26/0x4b [ 6696.035382] [8123951c] ? skb_dequeue+0x50/0x58 [ 6696.035384] [812482c8] ? pneigh_queue_purge+0x25/0x2f [ 6696.035386] [81249a91] ? neigh_ifdown+0xba/0xc9 [ 6696.035389] [81062f6d] ? notifier_call_chain+0x29/0x4c [ 6696.035392] [81243662] ? rollback_registered_many+0xed/0x19c [ 6696.035394] [8124371f] ? unregister_netdevice_many+0xe/0x57 [ 6696.035397] [812438a3] ? default_device_exit_batch+0x92/0xa3 [ 6696.035399] [8123e97a] ? cleanup_net+0xfd/0x1af [ 6696.035402] [8105bd0d] ? worker_thread+0x188/0x21d [ 6696.035404] [8123e87d] ? cleanup_net+0x0/0x1af [ 6696.035406] [8105f2d2] ? autoremove_wake_function+0x0/0x2e [ 6696.035409] [8105bb85] ? worker_thread+0x0/0x21d [ 6696.035411] [8105ee99] ? kthread+0x79/0x81 [ 6696.035414] [810098e4] ? kernel_thread_helper+0x4/0x10 [ 6696.035416] [8105ee20] ? kthread+0x0/0x81 [ 6696.035418] [810098e0] ? kernel_thread_helper+0x0/0x10 [ 6696.035419] ---[ end trace ef7b93cb006e989e ]--- [ 6696.037287] [ cut here ] [ 6696.037290] WARNING: at /build/mattems-linux-2.6_2.6.33-1~experimental.4-amd64-ieqSsa/linux-2.6-2.6.33-1~experimental.4/debian/build/source_amd64_none/kernel/sysctl.c:1894 unregister_sysctl_table+0xa6/0xd1() [ 6696.037293] Hardware name: [ 6696.037294] Modules linked in: veth loop parport_pc parport snd_pcm tpm_tis evdev snd_timer i2c_piix4 tpm tpm_bios button processor serio_raw i2c_core snd soundcore snd_page_alloc pcspkr psmouse ext3 jbd mbcache sg sr_mod cdrom sd_mod crc_t10dif ata_generic ata_piix libata floppy 8139cp thermal thermal_sys 8139too mii scsi_mod [last unloaded: veth] [ 6696.037311] Pid: 9, comm: netns Tainted: GW 2.6.33-2-amd64 #1 [ 6696.037312] Call Trace: [ 6696.037314] [8104e303] ? unregister_sysctl_table+0xa6/0xd1 [ 6696.037317] [8104e303] ? unregister_sysctl_table+0xa6/0xd1 [ 6696.037319] [81046b81] ? warn_slowpath_common+0x77/0xa3 [ 6696.037321] [8104e303] ? unregister_sysctl_table+0xa6/0xd1 [ 6696.037324] [812b1205] ? addrconf_ifdown+0x26f/0x2cc [ 6696.037326] [81247edc] ? neigh_sysctl_unregister+0x1a/0x31 [ 6696.037328] [812b1211] ? addrconf_ifdown+0x27b/0x2cc [ 6696.037331] [812b2b0c] ? addrconf_notify+0x714/0x7ea [ 6696.037333] [811eb2e7] ? extract_entropy+0x6a/0x125 [ 6696.037336] [81053aaf] ? lock_timer_base+0x26/0x4b [ 6696.037339] [8123951c] ? skb_dequeue+0x50/0x58 [ 6696.037341] [812482c8] ? pneigh_queue_purge+0x25/0x2f [ 6696.037343] [81249a91] ? neigh_ifdown+0xba/0xc9
