Your message dated Sat, 09 Jan 2021 22:33:33 +0000 with message-id <[email protected]> and subject line Bug#949235: fixed in iproute2 4.20.0-2+deb10u1 has caused the Debian Bug report #949235, regarding ip netns add race condition can wreak havoc in mount points to be marked as done.
This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact [email protected] immediately.) -- 949235: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=949235 Debian Bug Tracking System Contact [email protected] with problems
--- Begin Message ---Package: iproute2 Version: 4.20.0-2 Severity: important Control: found -1 5.4.0-1 The ipnetns.c:netns_add() function sets up the /var/run/netns mount point in a way that is fragile to race conditions if the routine is entered to by multiple processes at the same time. If the race condition is triggered, some kind of mount point recursion explosion seems to happen, messing up the entire system in "interesting" ways. For example, /proc/self/mountinfo ends up with tons of duplicate entries, and the mountinfo file itself becomes so large that the entire system tends to slow down. Also, subsequent netns add commands might fail with the following error message (note that this doesn't always happen): mount --bind /run/netns /run/netns failed: No space left on device Since it is a race condition, the issue is hard to reproduce on its own, but it is possible to force it to happen by using strace to inject an artificial delay in the mount() system call. See below. I observed this race condition happen multiple times on a real production system during the boot process. This is because this particular system sets up network namespaces using systemd units. Because systemd is designed to start units in parallel, and due to cold caches, multiple units running "netns add" end up synchronizing with each other, making it quite likely the race condition will be triggered. STEPS TO REPRODUCE Do NOT follow this procedure on a system you care about. This procedure WILL mess up your system and likely require you to reboot! 1. Start from a fresh system that never ran "netns add" since boot (or just unmount /var/run/netns manually). I can reproduce it on Debian Buster (iproute2 4.20.0-2) as well as latest Sid (5.4.0-1). 2. Run the following bash script: --- for i in {0..9} do strace -e trace=mount -e inject=mount:delay_exit=1000000 ip netns add "testnetns$i" 2>&1 | tee "$i.log" & done wait --- 3. Look at /proc/self/mountinfo. Hilarity ensues. If you increase the count in the script you might even get to see some "mount failed: No space left on device" errors. WORKAROUND Make sure that the first "netns add" command that runs after boot cannot run in parallel with any other "netns add" command. flock(1) might be useful here. I guess setting up the /var/run/netns point manually during boot, before any "netns add" command runs, might also work.
--- End Message ---
--- Begin Message ---Source: iproute2 Source-Version: 4.20.0-2+deb10u1 Done: Luca Boccassi <[email protected]> We believe that the bug you reported is fixed in the latest version of iproute2, which is due to be installed in the Debian FTP archive. A summary of the changes between this version and the previous one is attached. Thank you for reporting the bug, which will now be closed. If you have further comments please address them to [email protected], and the maintainer will reopen the bug report if appropriate. Debian distribution maintenance software pp. Luca Boccassi <[email protected]> (supplier of updated iproute2 package) (This message was generated automatically at their request; if you believe that there is a problem with it please contact the archive administrators by mailing [email protected]) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Format: 1.8 Date: Thu, 03 Dec 2020 18:42:49 +0000 Source: iproute2 Architecture: source Version: 4.20.0-2+deb10u1 Distribution: buster Urgency: medium Maintainer: Alexander Wirt <[email protected]> Changed-By: Luca Boccassi <[email protected]> Closes: 949235 961278 972784 Changes: iproute2 (4.20.0-2+deb10u1) buster; urgency=medium . * Backport ip-route-print-route-type-in-JSON-output.patch. Fixes bug in json output, backported from upstream. (Closes: #961278) * Backport tc-mqprio-json-ify-output.patch. Fixes bug in json output, backported from upstream. (Closes: #972784) * Backport ip-netns-use-flock-when-setting-up-run-netns.patch. Fixes race condition that DOSes the system when using ip netns add at boot. (Closes: #949235) Checksums-Sha1: 9495f2e399b9f57e6bd789d1ea07ad5b08d755cd 1929 iproute2_4.20.0-2+deb10u1.dsc 3713570a68fd31539737fcf639c15abef566b606 707016 iproute2_4.20.0.orig.tar.xz 72c40646da999d3d8f49367e2282574ce2cad4de 146688 iproute2_4.20.0-2+deb10u1.debian.tar.xz ccde9501131bffba65093d26881b3dd362884aab 6526 iproute2_4.20.0-2+deb10u1_source.buildinfo Checksums-Sha256: 5d7968a3a021bfdfcf546af5e8fc905aea66d6b5adc923e8d187baddffc4e91f 1929 iproute2_4.20.0-2+deb10u1.dsc c8adaa6a40f888476b23acb283cfa30c0dd55f07b5aa20663ed5ba2ef1f6fda8 707016 iproute2_4.20.0.orig.tar.xz d01f9c4b17519156cc0aadd5103cb38e928ceb3e86efb6b6e7479358794658e3 146688 iproute2_4.20.0-2+deb10u1.debian.tar.xz 45433b811e9a751de0e1a4e4d40d8fa55ddf0e6ddce785557ad1ed89df7061ae 6526 iproute2_4.20.0-2+deb10u1_source.buildinfo Files: 66baf33d93565998883648110f096280 1929 net optional iproute2_4.20.0-2+deb10u1.dsc f3dab4c812812bbb5873cb90f471bcbf 707016 net optional iproute2_4.20.0.orig.tar.xz 8dd7fe4ce79f658464244a3b762d64a7 146688 net optional iproute2_4.20.0-2+deb10u1.debian.tar.xz a249aba7dcd543e01a1ab32ac22b182b 6526 net optional iproute2_4.20.0-2+deb10u1_source.buildinfo -----BEGIN PGP SIGNATURE----- iQFFBAEBCgAvFiEE6g0RLAGYhL9yp9G8SylmgFB4UWIFAl/uBt4RHGJsdWNhQGRl Ymlhbi5vcmcACgkQSylmgFB4UWJ+DAf/WI2kqQSxdvA70LO3bfB5PfDGRONKl7LK uoQul0sHSeUMbclO6WX438iUniDqkMx8mZUiT6W3B4qE9XDk2FEm5sO7JJ70UMqR RezTmVtfV3OVXwiKX7F9Es0Kp+A8GjVTjJLknUUJSxbsEJq/PENGml80LpiJ46Ia IgvmaMOKZqgR4eCSQF2iBIEn2lFIbYjrv74XXiOAeytfZeREYT9/9dfC/4iitkNm mKRGnJNs9eOfchfkqJRv4E5rg5v1OqAwgzqiiI1jW48ssxiqK0f9KQGWGDSasvVc v0YuL9nCT+v6ShgXtCOxySSs/S66SOQgqPhxE6U8wnyp8BPSPPv1eA== =lZwX -----END PGP SIGNATURE-----
--- End Message ---
