[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Hi Dan, the CPU usage is zero, there is nothing popping up. The tasks causing the high load value are in D state. If I do "echo w > /proc/sysrq- trigger", I will get for every leaked NS: [392149.562095] kworker/u81:40 D0 4290 2 0x8000 [392149.562112] Workqueue: netns cleanup_net [392149.562125] Call Trace: [392149.562135] __schedule+0x291/0x8a0 [392149.562146] schedule+0x2c/0x80 [392149.562157] schedule_timeout+0x182/0x360 [392149.562167] ? call_timer_fn+0x130/0x130 [392149.562177] msleep+0x2d/0x40 [392149.562188] ? msleep+0x2d/0x40 [392149.562199] netdev_run_todo+0x15e/0x5d0 [392149.562210] rtnl_unlock+0xe/0x10 [392149.562221] default_device_exit_batch+0x156/0x180 [392149.562233] ? do_wait_intr_irq+0x90/0x90 [392149.562245] cleanup_net+0x14e/0x2a0 [392149.562257] process_one_work+0x1ec/0x410 [392149.562270] worker_thread+0x32/0x410 [392149.562281] kthread+0x128/0x140 [392149.562292] ? process_one_work+0x410/0x410 [392149.562300] ? kthread_create_on_node+0x70/0x70 [392149.562305] ? do_syscall_64+0x67/0x130 [392149.562310] ? SyS_exit_group+0x14/0x20 [392149.562316] ret_from_fork+0x35/0x40 I am not sure how fine is to have e.g. thousands of such leaked namespaces - if I can't run out of some resources (kworker threads?). I am going to keep the machine running + generate even higher load there to see if I hit any such limit. Thanks JH -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: Won't Fix Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Hi Dan, so here are the information for the latest kernel (4.13.0-38.43+hf1711407v20180413b1): The bug got reproduced: [ 1085.626663] unregister_netdevice: possible namespace leak (netns 8d2dc3d34800), have been waiting for 60 seconds [ 1085.626696] (netns 8d2dc3d34800): dst 8d2dd006db00 expires 0 error 0 obsolete 2 flags 0 refcnt 1 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0x9c0 If I grep the netns: 8d2dc3d34800, it is visible that during first 10 minutes, it outputs more info, then until ~ 1 hour, it says the same message more frequently and then only outputs it ever hour. [ 901.854919] net_namespace: alloced net 8d2dc3d34800 [ 1085.626663] unregister_netdevice: possible namespace leak (netns 8d2dc3d34800), have been waiting for 60 seconds [ 1085.626696] (netns 8d2dc3d34800): dst 8d2dd006db00 expires 0 error 0 obsolete 2 flags 0 refcnt 1 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0x9c0 [ 1095.706659] unregister_netdevice: possible namespace leak (netns 8d2dc3d34800), have been waiting for 70 seconds [ 1095.706692] (netns 8d2dc3d34800): dst 8d2dd006db00 expires 0 error 0 obsolete 2 flags 0 refcnt 1 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0x9c0 [ 1105.786651] unregister_netdevice: possible namespace leak (netns 8d2dc3d34800), have been waiting for 80 seconds [ 1105.786669] (netns 8d2dc3d34800): dst 8d2dd006db00 expires 0 error 0 obsolete 2 flags 0 refcnt 1 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0x9c0 [ 1115.866654] unregister_netdevice: possible namespace leak (netns 8d2dc3d34800), have been waiting for 90 seconds [ 1631.003906] unregister_netdevice: likely namespace leak (netns 8d2dc3d34800), have been waiting for 605 seconds [ 1641.147950] unregister_netdevice: likely namespace leak (netns 8d2dc3d34800), have been waiting for 616 seconds [ 1651.291981] unregister_netdevice: likely namespace leak (netns 8d2dc3d34800), have been waiting for 626 seconds [ 1661.372025] unregister_netdevice: likely namespace leak (netns 8d2dc3d34800), have been waiting for 636 seconds [ 1671.452071] unregister_netdevice: likely namespace leak (netns 8d2dc3d34800), have been waiting for 646 seconds [ 1681.564110] unregister_netdevice: likely namespace leak (netns 8d2dc3d34800), have been waiting for 656 seconds ... [ 4628.301622] unregister_netdevice: namespace leak (netns 8d2dc3d34800), have been waiting for 3603 seconds [ 8252.383668] unregister_netdevice: namespace leak (netns 8d2dc3d34800), have been waiting for 7227 seconds [11877.333159] unregister_netdevice: namespace leak (netns 8d2dc3d34800), have been waiting for 10852 seconds [15502.281254] unregister_netdevice: namespace leak (netns 8d2dc3d34800), have been waiting for 14477 seconds [19127.230600] unregister_netdevice: namespace leak (netns 8d2dc3d34800), have been waiting for 18102 seconds [22752.179625] unregister_netdevice: namespace leak (netns 8d2dc3d34800), have been waiting for 21727 seconds [26377.129256] unregister_netdevice: namespace leak (netns 8d2dc3d34800), have been waiting for 25352 seconds [30002.078342] unregister_netdevice: namespace leak (netns 8d2dc3d34800), have been waiting for 28977 seconds The machine continue to run and the docker is still able to create new instances, i.e. there is no complete lock out. The machine continued to run further, and after an ~hour , 3 namespaces leaked. The machine now has load "3", which seems to be an artifact of this. Other than that, it seems to continue to run! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: Won't Fix Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker-
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Hi there, great to hear the progress! I have just reproduced it using the previous #43+hf1711407v20180412b2" kernel: Apr 13 19:12:14 prg41-004 kernel: [13621.398319] unregister_netdevice: likely namespace leak (netns 8c7a056f), have been waiting for 1973452 seconds Apr 13 19:12:24 prg41-004 kernel: [13631.478266] unregister_netdevice: likely namespace leak (netns 8c7a056f), have been waiting for 1983532 seconds The seconds seems to be rather milliseconds. Waiting here for the newest kernel and will try it right away. Jiri Horky -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: Won't Fix Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Hi Dan, how should I read that it got to "Won't fix" state? No more time to debug it? Thanks Jirka H. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: Won't Fix Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Hi Dan, any luck in preparing anothe kernel with more debugging info? Looking forward to try another one. Thanks for your effort! Jiri Horky -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: In Progress Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Hi Dan, thanks for commenting. I started to lose hope already ;) Looking forward for next kernel, will test it right away. Jiri Horky -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: In Progress Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Hi again, so for the second kernel (), was able to reproduce it in ~same time. The messages are: Feb 13 23:04:23 prg41-004 kernel: [ 650.285711] unregister_netdevice: waiting for lo (netns 943cfe8ce000) to become free. Usage count = 1 Feb 13 23:04:23 prg41-004 kernel: [ 650.285736](netns 943cfe8ce000): dst 943d18ac4200 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0xa20 root@prg41-004:~# grep 943d18ac4200 /var/log/syslog Feb 13 23:02:14 prg41-004 kernel: [ 521.458077] dst_init: dst 943d18ac4200 refcnt 1 from __dst_alloc+0x7e/0x100 Feb 13 23:02:14 prg41-004 kernel: [ 521.458085] dst_hold: dst 943d18ac4200 refcnt 2 from rt_cache_route+0x54/0xe0 Feb 13 23:02:14 prg41-004 kernel: [ 521.634088] dst_hold_safe: dst 943d18ac4200 refcnt 3 from ip_route_output_key_hash_rcu+0x8a1/0xa20 Feb 13 23:02:14 prg41-004 kernel: [ 521.778104] dst_release_immediate: dst 943d18ac4200 refcnt 2 from free_fib_info_rcu+0x146/0x1c0 Feb 13 23:04:23 prg41-004 kernel: [ 650.285736](netns 943cfe8ce000): dst 943d18ac4200 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0xa20 Feb 13 23:04:33 prg41-004 kernel: [ 660.525715](netns 943cfe8ce000): dst 943d18ac4200 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0xa20 Feb 13 23:04:43 prg41-004 kernel: [ 670.669641](netns 943cfe8ce000): dst 943d18ac4200 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0xa20 etc. I hope it helps! Regards Jirka H. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: In Progress Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Forgot to paste the second kernel version. The previous message was obtained using 4.13.13-00941-gf382397cf315 kernel (the one with ipsec/xfrm dst leak patch). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: In Progress Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Hi Dan, so for the first kernel (linux- image-4.13.13-00940-gf16e2bbbddee_4.13.13-00940-gf16e2bbbddee- 24_amd64.deb), it failed after some 10 minutes. The outputs are: Feb 13 22:35:54 prg41-004 kernel: [ 736.399342] unregister_netdevice: waiting for lo (netns 9adbfbebb000) to become free. Usage count = 1 Feb 13 22:35:54 prg41-004 kernel: [ 736.399376](netns 9adbfbebb000): dst 9ae3fee51f00 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0xa20 And then: root@prg41-004:~# grep 9ae3fee51f00 /var/log/syslog Feb 13 22:33:33 prg41-004 kernel: [ 595.493003] dst_init: dst 9ae3fee51f00 refcnt 1 from __dst_alloc+0x7e/0x100 Feb 13 22:33:33 prg41-004 kernel: [ 595.493009] dst_hold: dst 9ae3fee51f00 refcnt 2 from rt_cache_route+0x54/0xe0 Feb 13 22:33:33 prg41-004 kernel: [ 595.648996] dst_hold_safe: dst 9ae3fee51f00 refcnt 3 from ip_route_output_key_hash_rcu+0x8a1/0xa20 Feb 13 22:33:33 prg41-004 kernel: [ 595.689067] dst_release_immediate: dst 9ae3fee51f00 refcnt 2 from free_fib_info_rcu+0x146/0x1c0 Feb 13 22:35:54 prg41-004 kernel: [ 736.399376](netns 9adbfbebb000): dst 9ae3fee51f00 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0xa20 Feb 13 22:36:04 prg41-004 kernel: [ 746.447322](netns 9adbfbebb000): dst 9ae3fee51f00 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0xa20 Feb 13 22:36:14 prg41-004 kernel: [ 756.527237](netns 9adbfbebb000): dst 9ae3fee51f00 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0xa20 Feb 13 22:36:24 prg41-004 kernel: [ 766.639159](netns 9adbfbebb000): dst 9ae3fee51f00 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0xa20 Feb 13 22:36:34 prg41-004 kernel: [ 776.815056](netns 9adbfbebb000): dst 9ae3fee51f00 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0xa20 Feb 13 22:36:45 prg41-004 kernel: [ 786.958967](netns 9adbfbebb000): dst 9ae3fee51f00 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0xa20 Feb 13 22:36:55 prg41-004 kernel: [ 797.070875](netns 9adbfbebb000): dst 9ae3fee51f00 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0xa20 (and obviously more of such messages every 10 seconds) BTW from the timing it seems that the docker image who produced this error might be interrupted with timeout (normally, the dockers images we run on the machine live for few seconds only). Will post the second kernel soon. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: In Progress Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Hi Dan, Glad to help, I thank you for digging into that! Here is the latest output: [ 912.489822] unregister_netdevice: waiting for lo (netns 8de5c10f3000) to become free. Usage count = 1 [ 912.489927] (netns 8de5c10f3000): dst 8de5ba0ced00 expires 0 error 0 obsolete 2 flags 0 refcnt 2 ops ipv4_dst_ops+0x0/0xc0 creator ip_route_output_key_hash_rcu+0x355/0x9c0 Waiting for another image ;-) Regards Jiri Horky -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: In Progress Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Hi Dan, here is the output: Feb 2 18:50:14 prg41-004 kernel: [ 482.151773] unregister_netdevice: waiting for lo (netns 8de11803e000) to become free. Usage count = 1 Feb 2 18:50:14 prg41-004 kernel: [ 482.151876](netns 8de11803e000): dst 8dd905360300 expires 0 error 0 obsolete 2 refcnt 1 ops ipv4_dst_ops+0x0/0xc0 creator rt_dst_alloc+0x54/0xe0 It keeps to be output every 10s for over 500s now. I hope it helps. Jirka Horky -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: In Progress Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Hi Dan, to answer your questions: 1) Yes, it was full output before next "waiting for..." message 2) If I leave the box running (and stops generating the load), it actually does not recover itself at all. It keeps outputing the "waiting for..." forever with the same device: # dmesg | grep waiti [35684.449735] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35694.529742] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35704.609742] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35714.689748] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35724.769751] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35734.849759] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35744.929763] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35755.009758] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35765.089759] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35775.169756] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35785.249761] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35795.329763] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35805.409758] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35815.489753] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35825.569749] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35835.649747] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35845.729752] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35855.809741] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35865.889736] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35875.969733] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35886.049726] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 [35896.129717] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 I am happy to continue testing with a new kernel. Cheers Jirka H. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: In Progress Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Hi Dan, Here is the output of the kernel when the "unregister_netdevice" bug was hit for the first time on the reproducer machine. I hope it helps. Let me know if I can provide you more debug output or try to reproduce it once more. Thanks Jiri Horky Jan 31 08:42:26 prg41-004 kernel: [ 663.049461] unregister_netdevice: waiting for lo (netns 88cd026e) to become free. Usage count = 1 Jan 31 08:42:26 prg41-004 kernel: [ 663.049477](netns 88cd026e): open sk 88cd0120a000 family 16 state 7 dst_cache (null) rx_dst (null) creator __netlink_kernel_create+0x6c/0x260 Jan 31 08:42:26 prg41-004 kernel: [ 663.049481](netns 88cd026e): open sk 88cd0120d800 family 16 state 7 dst_cache (null) rx_dst (null) creator __netlink_kernel_create+0x6c/0x260 Jan 31 08:42:26 prg41-004 kernel: [ 663.049486](netns 88cd026e): open sk 88cd0120d000 family 16 state 7 dst_cache (null) rx_dst (null) creator __netlink_kernel_create+0x6c/0x260 Jan 31 08:42:26 prg41-004 kernel: [ 663.049490](netns 88cd026e): open sk 88cd0120c000 family 16 state 7 dst_cache (null) rx_dst (null) creator __netlink_kernel_create+0x6c/0x260 Jan 31 08:42:26 prg41-004 kernel: [ 663.049495](netns 88cd026e): open sk 88cd18dc2200 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049498](netns 88cd026e): open sk 88cd10e58f00 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049501](netns 88cd026e): open sk 88cd10e5da00 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049504](netns 88cd026e): open sk 88cd10e5b0c0 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049506](netns 88cd026e): open sk 88cd10e5c380 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049509](netns 88cd026e): open sk 88cd10e5b480 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049512](netns 88cd026e): open sk 88cd10e5a1c0 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049514](netns 88cd026e): open sk 88cd10e5b840 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049517](netns 88cd026e): open sk 88cd10e5cb00 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049520](netns 88cd026e): open sk 88cd10e5f440 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049523](netns 88cd026e): open sk 88cd10e58000 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049526](netns 88cd026e): open sk 88cd10e5a580 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049529](netns 88cd026e): open sk 88cd10e58b40 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049531](netns 88cd026e): open sk 88cd10e59e00 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049534](netns 88cd026e): open sk 88cd10e5bc00 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049537](netns 88cd026e): open sk 88cd10e5c740 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-004 kernel: [ 663.049539](netns 88cd026e): open sk 88cd10e5d280 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 Jan 31 08:42:26 prg41-
[Kernel-packages] [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free
Ok, waiting for the kernel with dbg messages. I should be able to test it right away. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: In Progress Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
