Bug#754987: udevadm settle causes 30 sec delay under sysvinit
Am 02.11.2014 um 03:06 schrieb Ben Hutchings: On Sun, 2014-11-02 at 02:34 +0100, Christian Hofstaedtler wrote: * Ben Hutchings b...@decadent.org.uk [141101 14:02]: On Sat, 2014-11-01 at 13:22 +0100, Christian Hofstaedtler wrote: dash backs up any FDs it redirects, so it can restore them later on. bash just closes them outright in this situation (forked subshell + parent exits), causing udev's spawn_read to immediately, thereby marking the udev event as finished. That makes *much* more sense. [..] I'd suggest the following patch, as a stop gap for sysvinit users: Index: systemd-215/debian/extra/net.agent === --- systemd-215.orig/debian/extra/net.agent 2014-09-27 17:50:52.0 +0200 +++ systemd-215/debian/extra/net.agent 2014-11-02 02:33:40.970469131 +0100 @@ -1,4 +1,4 @@ -#!/bin/sh -e +#!/bin/bash -e [..] This seems to work under both bash and dash: exec /dev/null 2 /dev/null do_everything Thanks Ben and Christian for further debugging this. I suspected the problem to be related to file descriptors when this issue came up on debian-user [1] back in July. There I suggested to use ( do_everything ) /dev/null 2 /dev/null as a possible workaround. I don't particularly like using a subshell but I'm just posting it for completeness sake here. I admit, I don't quite understand Ben's patch, which runs exec twice: Once before do_everything() and later for the actual ifup/ifdown call. What exactly does the first exec do to fix this? Marco, do you have any preference? Michael [1] https://lists.debian.org/debian-user/2014/07/msg01509.html -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#754987: udevadm settle causes 30 sec delay under sysvinit
On Mon, 2014-11-03 at 13:03 +0100, Michael Biebl wrote: Am 02.11.2014 um 03:06 schrieb Ben Hutchings: On Sun, 2014-11-02 at 02:34 +0100, Christian Hofstaedtler wrote: * Ben Hutchings b...@decadent.org.uk [141101 14:02]: On Sat, 2014-11-01 at 13:22 +0100, Christian Hofstaedtler wrote: dash backs up any FDs it redirects, so it can restore them later on. bash just closes them outright in this situation (forked subshell + parent exits), causing udev's spawn_read to immediately, thereby marking the udev event as finished. That makes *much* more sense. [..] I'd suggest the following patch, as a stop gap for sysvinit users: Index: systemd-215/debian/extra/net.agent === --- systemd-215.orig/debian/extra/net.agent 2014-09-27 17:50:52.0 +0200 +++ systemd-215/debian/extra/net.agent 2014-11-02 02:33:40.970469131 +0100 @@ -1,4 +1,4 @@ -#!/bin/sh -e +#!/bin/bash -e [..] This seems to work under both bash and dash: exec /dev/null 2 /dev/null do_everything Thanks Ben and Christian for further debugging this. I suspected the problem to be related to file descriptors when this issue came up on debian-user [1] back in July. There I suggested to use ( do_everything ) /dev/null 2 /dev/null as a possible workaround. I don't particularly like using a subshell but I'm just posting it for completeness sake here. I admit, I don't quite understand Ben's patch, which runs exec twice: Once before do_everything() and later for the actual ifup/ifdown call. What exactly does the first exec do to fix this? The first exec does file redirection only, and this doesn't leave duplicate file descriptors. Ben. Marco, do you have any preference? Michael [1] https://lists.debian.org/debian-user/2014/07/msg01509.html -- Ben Hutchings Power corrupts. Absolute power is kind of neat. - John Lehman, Secretary of the US Navy 1981-1987 signature.asc Description: This is a digitally signed message part
Bug#754987: udevadm settle causes 30 sec delay under sysvinit
* Ben Hutchings [Sun Nov 02, 2014 at 02:06:29AM +]: On Sun, 2014-11-02 at 02:34 +0100, Christian Hofstaedtler wrote: dash backs up any FDs it redirects, so it can restore them later on. bash just closes them outright in this situation (forked subshell + parent exits), causing udev's spawn_read to immediately, thereby marking the udev event as finished. That makes *much* more sense. [...] This seems to work under both bash and dash: exec /dev/null 2 /dev/null do_everything I can confirm that also this works and solves the problem for non-systemd systems. Thanks, Ben! regards, -mika- signature.asc Description: Digital signature
Bug#754987: udevadm settle causes 30 sec delay under sysvinit
* Marco d'Itri [Sun Nov 02, 2014 at 02:35:42AM +0100]: On Nov 02, Christian Hofstaedtler z...@debian.org wrote: I'd suggest the following patch, as a stop gap for sysvinit users: If the others can confirm that it works then I have no objections. I can confirm that this works as expected. Thanks a ton for debugging this, Christian. regards, -mika- signature.asc Description: Digital signature
Bug#754987: udevadm settle causes 30 sec delay under sysvinit
* Ben Hutchings b...@decadent.org.uk [141101 06:26]: On Sat, 2014-11-01 at 04:44 +0100, Christian Hofstaedtler wrote: I've tried to debug this a bit more tonight, and after a good 4 hours there are two things I can report: 1. with the default ruleset, udev leaks an ethtool socket to net.agent (and by extension to ifup, dhclient, ...) This is a bug but probably harmless. SIOCETHTOOL doesn't depend on any socket state so it is normally used with an unconnected socket of an arbitrary address family. While it may be harmless from a security POV, the leaked FD causes the udev worker to hold on to the forked-away net.agent process. net.agent tries hard to close udev's logging sockets (see the last few lines of net.agent), but it doesn't know about any leaked sockets and therefore doesn't try to close them. -fd = socket(PF_INET, SOCK_DGRAM, 0); +fd = socket(PF_INET, SOCK_DGRAM|SOCK_CLOEXEC, 0); Ah indeed. 4 AM is not the best time for such things ;-) -- ,''`. Christian Hofstaedtler z...@debian.org : :' : Debian Developer `. `' 7D1A CFFA D9E0 806C 9C4C D392 5C13 D6DB 9305 2E03 `- pgpbQXeDgGjMY.pgp Description: PGP signature
Bug#754987: udevadm settle causes 30 sec delay under sysvinit
On Sat, 2014-11-01 at 13:22 +0100, Christian Hofstaedtler wrote: * Ben Hutchings b...@decadent.org.uk [141101 06:26]: On Sat, 2014-11-01 at 04:44 +0100, Christian Hofstaedtler wrote: I've tried to debug this a bit more tonight, and after a good 4 hours there are two things I can report: 1. with the default ruleset, udev leaks an ethtool socket to net.agent (and by extension to ifup, dhclient, ...) This is a bug but probably harmless. SIOCETHTOOL doesn't depend on any socket state so it is normally used with an unconnected socket of an arbitrary address family. While it may be harmless from a security POV, the leaked FD causes the udev worker to hold on to the forked-away net.agent process. How is that? Ben. net.agent tries hard to close udev's logging sockets (see the last few lines of net.agent), but it doesn't know about any leaked sockets and therefore doesn't try to close them. -fd = socket(PF_INET, SOCK_DGRAM, 0); +fd = socket(PF_INET, SOCK_DGRAM|SOCK_CLOEXEC, 0); Ah indeed. 4 AM is not the best time for such things ;-) -- Ben Hutchings Kids! Bringing about Armageddon can be dangerous. Do not attempt it in your own home. - Terry Pratchett and Neil Gaiman, `Good Omens' signature.asc Description: This is a digitally signed message part
Bug#754987: udevadm settle causes 30 sec delay under sysvinit
* Ben Hutchings b...@decadent.org.uk [141101 14:02]:
On Sat, 2014-11-01 at 13:22 +0100, Christian Hofstaedtler wrote:
* Ben Hutchings b...@decadent.org.uk [141101 06:26]:
On Sat, 2014-11-01 at 04:44 +0100, Christian Hofstaedtler wrote:
I've tried to debug this a bit more tonight, and after a good 4
hours there are two things I can report:
1. with the default ruleset, udev leaks an ethtool socket to
net.agent (and by extension to ifup, dhclient, ...)
This is a bug but probably harmless. SIOCETHTOOL doesn't depend on any
socket state so it is normally used with an unconnected socket of an
arbitrary address family.
While it may be harmless from a security POV, the leaked FD causes
the udev worker to hold on to the forked-away net.agent process.
How is that?
You're obviously correct; the extra FDs are red herrings.
dash backs up any FDs it redirects, so it can restore them
later on. bash just closes them outright in this situation (forked
subshell + parent exits), causing udev's spawn_read to immediately,
thereby marking the udev event as finished.
I've now looked at wheezy's udev, and the relevant functions are
virtually unchanged; IMVHO the redirection/fork hack likely never
worked the way it was meant to under dash.
Upstream commit 2004d23a0fcaa6e74631057a2ff75594a038d86e changed
udev's default log level from LOG_ERR to LOG_INFO (altough all older
versions always suggested that INFO was the default -- it wasn't).
That commit is part of systemd v209 and newer.
I'd suggest the following patch, as a stop gap for sysvinit users:
Index: systemd-215/debian/extra/net.agent
===
--- systemd-215.orig/debian/extra/net.agent 2014-09-27 17:50:52.0
+0200
+++ systemd-215/debian/extra/net.agent 2014-11-02 02:33:40.970469131 +0100
@@ -1,4 +1,4 @@
-#!/bin/sh -e
+#!/bin/bash -e
#
# run /sbin/{ifup,ifdown} with the --allow=hotplug option.
#
@@ -100,9 +100,12 @@ esac
}
-# When udev_log=debug stdout and stderr are pipes connected to udevd.
+# When udev_log=info (default) or debug, stdout and stderr are
+# pipes connected to udevd.
# They need to be closed or udevd will wait for this process which will
# deadlock with udevsettle until the timeout.
+# Note that this trick does not work under dash, which backs up any
+# redirected FDs to FD 10 and higher.
do_everything /dev/null 2 /dev/null
exit 0
--
,''`. Christian Hofstaedtler z...@debian.org
: :' : Debian Developer
`. `' 7D1A CFFA D9E0 806C 9C4C D392 5C13 D6DB 9305 2E03
`-
signature.asc
Description: Digital signature
Bug#754987: udevadm settle causes 30 sec delay under sysvinit
On Nov 02, Christian Hofstaedtler z...@debian.org wrote: I'd suggest the following patch, as a stop gap for sysvinit users: If the others can confirm that it works then I have no objections. -- ciao, Marco signature.asc Description: Digital signature
Bug#754987: udevadm settle causes 30 sec delay under sysvinit
On Sun, 2014-11-02 at 02:34 +0100, Christian Hofstaedtler wrote:
* Ben Hutchings b...@decadent.org.uk [141101 14:02]:
On Sat, 2014-11-01 at 13:22 +0100, Christian Hofstaedtler wrote:
* Ben Hutchings b...@decadent.org.uk [141101 06:26]:
On Sat, 2014-11-01 at 04:44 +0100, Christian Hofstaedtler wrote:
I've tried to debug this a bit more tonight, and after a good 4
hours there are two things I can report:
1. with the default ruleset, udev leaks an ethtool socket to
net.agent (and by extension to ifup, dhclient, ...)
This is a bug but probably harmless. SIOCETHTOOL doesn't depend on any
socket state so it is normally used with an unconnected socket of an
arbitrary address family.
While it may be harmless from a security POV, the leaked FD causes
the udev worker to hold on to the forked-away net.agent process.
How is that?
You're obviously correct; the extra FDs are red herrings.
dash backs up any FDs it redirects, so it can restore them
later on. bash just closes them outright in this situation (forked
subshell + parent exits), causing udev's spawn_read to immediately,
thereby marking the udev event as finished.
That makes *much* more sense.
[...]
I'd suggest the following patch, as a stop gap for sysvinit users:
Index: systemd-215/debian/extra/net.agent
===
--- systemd-215.orig/debian/extra/net.agent 2014-09-27 17:50:52.0
+0200
+++ systemd-215/debian/extra/net.agent 2014-11-02 02:33:40.970469131 +0100
@@ -1,4 +1,4 @@
-#!/bin/sh -e
+#!/bin/bash -e
#
# run /sbin/{ifup,ifdown} with the --allow=hotplug option.
#
@@ -100,9 +100,12 @@ esac
}
-# When udev_log=debug stdout and stderr are pipes connected to udevd.
+# When udev_log=info (default) or debug, stdout and stderr are
+# pipes connected to udevd.
# They need to be closed or udevd will wait for this process which will
# deadlock with udevsettle until the timeout.
+# Note that this trick does not work under dash, which backs up any
+# redirected FDs to FD 10 and higher.
do_everything /dev/null 2 /dev/null
This seems to work under both bash and dash:
exec /dev/null 2 /dev/null
do_everything
Ben.
exit 0
--
Ben Hutchings
A free society is one where it is safe to be unpopular. - Adlai Stevenson
signature.asc
Description: This is a digitally signed message part
Bug#754987: udevadm settle causes 30 sec delay under sysvinit
I've tried to debug this a bit more tonight, and after a good 4
hours there are two things I can report:
1. with the default ruleset, udev leaks an ethtool socket to
net.agent (and by extension to ifup, dhclient, ...)
Details: 80-net-setup-link.rules loads the udev builtin
net_setup_link, which in turn creates a static object of type
link_config_ctx, and then goes on to call link_config_apply,
which calls link_config_ctx_connect. This function opens both an
ethtool and a netlink socket. The netlink socket is correctly marked
as CLOEXEC, but the ethtool socket is not marked this way.
With net_setup_link loaded, the Debian-specific net rules run, and
call into the /lib/udev/net.agent shell script.
The end result is that dhclient (and many other programs!) inherit an
ethtool socket opened by udev.
2. Replacing #!/bin/sh with #!/bin/bash magically makes the problem
go away in my VM. I believe this is because bash closes all sorts of
things on startup, while dash doesn't do that.
Unfortunately, fixing the fdleak from #1 above doesn't fix the
overall problem -- likely there's another fdleak somewhere that I
didn't find.
A patch for #1 that totally ignores error branches, but could serve
as a starting point for somebody else:
--- systemd-215/src/udev/net/link-config.c 2014-11-01 04:40:50.0
+0100
+++ systemd-215.ch/src/udev/net/link-config.c 2014-11-01 04:41:50.603183984
+0100
@@ -130,6 +130,16 @@
}
}
+void link_config_ctx_disconnect(link_config_ctx *ctx) {
+if (!ctx)
+return;
+
+safe_close(ctx-ethtool_fd);
+ctx-ethtool_fd = -1;
+sd_rtnl_unref(ctx-rtnl);
+ctx-rtnl = NULL;
+}
+
void link_config_ctx_free(link_config_ctx *ctx) {
if (!ctx)
return;
@@ -410,6 +420,7 @@
return r;
}
+link_config_ctx_disconnect(ctx);
return 0;
}
@@ -430,6 +441,7 @@
if (r 0)
return r;
+link_config_ctx_disconnect(ctx);
*ret = driver;
return 0;
}
--
,''`. Christian Hofstaedtler z...@debian.org
: :' : Debian Developer
`. `' 7D1A CFFA D9E0 806C 9C4C D392 5C13 D6DB 9305 2E03
`-
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#754987: udevadm settle causes 30 sec delay under sysvinit
BTW, I meant to document how net.agent and the networking init script interact, so the next person doesn't have to figure this out. 1. /etc/init.d/networking start creates /run/network/ifstate. (it does nothing if it detects upstart is the running init.) 2. ifupdown does nothing when /run/network/ifstate is missing. 3. ifupdown -a sets up lo, even if lo is not configured in /etc/network/interfaces (see the man page of ifupdown). 4. ifupdown -a (no further args) only brings up interfaces marked as auto in /etc/network/interfaces. 5. /lib/udev/net.agent defers its work to ifup@.service if it detects systemd is the running init. With other inits, it waits for the lo device to become not-down (usually it will become unknown), and then calls ifup --allow=hotplug $INTERFACE. (net.agent exits if $INTERFACE is lo or some of the common 'virtual' interface names.) 6. ifup@.service pulls in networking.service (- networking init script), and then calls ifup --allow=hotplug %i. Therefore, under sysvinit, net.agent waits (by extension of waiting for the lo device) for the networking init script to start. If one forces 'lo' to become up before that, net.agent will immediately call ifupdown, which then does nothing (because /run/network/ifstate will be absent). -- ,''`. Christian Hofstaedtler z...@debian.org : :' : Debian Developer `. `' 7D1A CFFA D9E0 806C 9C4C D392 5C13 D6DB 9305 2E03 `- -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#754987: udevadm settle causes 30 sec delay under sysvinit
On Sat, 2014-11-01 at 04:44 +0100, Christian Hofstaedtler wrote:
I've tried to debug this a bit more tonight, and after a good 4
hours there are two things I can report:
1. with the default ruleset, udev leaks an ethtool socket to
net.agent (and by extension to ifup, dhclient, ...)
This is a bug but probably harmless. SIOCETHTOOL doesn't depend on any
socket state so it is normally used with an unconnected socket of an
arbitrary address family.
[...]
A patch for #1 that totally ignores error branches, but could serve
as a starting point for somebody else:
--- systemd-215/src/udev/net/link-config.c2014-11-01 04:40:50.0
+0100
+++ systemd-215.ch/src/udev/net/link-config.c 2014-11-01 04:41:50.603183984
+0100
@@ -130,6 +130,16 @@
}
}
+void link_config_ctx_disconnect(link_config_ctx *ctx) {
+if (!ctx)
+return;
+
+safe_close(ctx-ethtool_fd);
+ctx-ethtool_fd = -1;
+sd_rtnl_unref(ctx-rtnl);
+ctx-rtnl = NULL;
+}
+
[...]
I think the proper fix for #1 is:
--- a/src/udev/net/ethtool-util.c
+++ b/src/udev/net/ethtool-util.c
@@ -53,7 +53,7 @@ int ethtool_connect(int *ret) {
assert_return(ret, -EINVAL);
-fd = socket(PF_INET, SOCK_DGRAM, 0);
+fd = socket(PF_INET, SOCK_DGRAM|SOCK_CLOEXEC, 0);
if (fd 0) {
return -errno;
}
--- END ---
Ben.
--
Ben Hutchings
Kids! Bringing about Armageddon can be dangerous. Do not attempt it in
your own home. - Terry Pratchett and Neil Gaiman, `Good Omens'
signature.asc
Description: This is a digitally signed message part
Bug#754987: udevadm settle causes 30 sec delay under sysvinit
hello! i can confirm the same issue, and /lib/udev/net.agent is the reason. after trying the tips given above, i thought: why the hell udev should raise up the network when it starts? we have /etc/init.d/networking for this! and i've just added exit 0 at the very beginning of /lib/udev/net.agent folks, this speeded up boot process like noting else)) network is maintained fine by good old init script. PS. seems that it's not only thing, that is duplucating init scripts functions... -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
