Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Tue, 5 Mar 2019 23:39:00 +0100
Michael Biebl  wrote:

> Strictly speaking, we should only need either --notify-await when
> starting the daemon via s-s-d, or --wait-daemon when calling udevadm
> trigger. Using both probably doesn't hurt, but isn't really necessary.

correct, actually only the s-s-d --notify-await solves the issue, as it
was confirmed by Bill Brelsford in the messages #134 and #258


> Since I can't reproduce the original issue myself, I wonder if
> udevadm trigger --wait-daemon actually works. There was conflicting
> feedback regarding this matter.

udevadm --wait-daemon doesn't fix this specific issue, as it doesn't
retry if the connection was refused (udevadm-trigger.c:177)


> Would be cool if someone affected by this problem could try the
> following:

it was tested a similar configuration:

  udevadm trigger --type=subsystems --action=add --wait-daemon

ciao!

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

Am 05.03.19 um 22:40 schrieb Trek:
> On Tue, 5 Mar 2019 09:46:53 +0100
> Andreas Henriksson  wrote:
> 
>> You likely also want to include in your patch a debian/control
>> change that makes udev depend on a new enough dpkg which has the
>> new flags you're making use of.
> 
> right, a new patch is attached
> thank you!

Strictly speaking, we should only need either --notify-await when
starting the daemon via s-s-d, or --wait-daemon when calling udevadm
trigger. Using both probably doesn't hurt, but isn't really necessary.

Since I can't reproduce the original issue myself, I wonder if
udevadm trigger --wait-daemon actually works. There was conflicting
feedback regarding this matter.

Would be cool if someone affected by this problem could try the following:

1/ Comment out lines 180-186 (the udevadm trigger --type=subsystems
--action=add part)
2/ Verify that the problem occurs
3/ Use udevadm trigger --type=devices --action=add --wait-daemon
and check if that fixes the problem.




-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Tue, 5 Mar 2019 09:46:53 +0100
Andreas Henriksson  wrote:

> You likely also want to include in your patch a debian/control
> change that makes udev depend on a new enough dpkg which has the
> new flags you're making use of.

right, a new patch is attached
thank you!diff --git a/debian/control b/debian/control
index 8fbe346..f26a177 100644
--- a/debian/control
+++ b/debian/control
@@ -330,6 +330,7 @@ Pre-Depends: ${misc:Pre-Depends}
 Depends: ${shlibs:Depends},
  ${misc:Depends},
  adduser,
+ dpkg (>= 1.19.3),
  libudev1 (= ${binary:Version}),
  lsb-base (>= 3.0-6),
  util-linux (>= 2.27.1),
diff --git a/debian/udev.init b/debian/udev.init
index 6a3c9b3..cf93107 100644
--- a/debian/udev.init
+++ b/debian/udev.init
@@ -166,7 +166,8 @@ case "$1" in
 
 log_daemon_msg "Starting $DESC" "$NAME"
 if start-stop-daemon --start --name $NAME --user root --quiet \
---pidfile $PIDFILE --exec $DAEMON --background --make-pidfile; then
+--pidfile $PIDFILE --exec $DAEMON --background --make-pidfile \
+--notify-await; then
 # prevents udevd to be killed by sendsigs (see #791944)
 mkdir -p $OMITDIR
 ln -sf $PIDFILE $OMITDIR/$NAME
@@ -178,7 +179,7 @@ case "$1" in
 fi
 
 log_action_begin_msg "Synthesizing the initial hotplug events (subsystems)"
-if udevadm trigger --type=subsystems --action=add; then
+if udevadm trigger --type=subsystems --action=add --wait-daemon; then
 log_action_end_msg $?
 else
 log_action_end_msg $?
@@ -226,7 +227,8 @@ case "$1" in
 
 log_daemon_msg "Starting $DESC" "$NAME"
 if start-stop-daemon --start --name $NAME --user root --quiet \
---pidfile $PIDFILE --exec $DAEMON --background --make-pidfile; then
+--pidfile $PIDFILE --exec $DAEMON --background --make-pidfile \
+--notify-await; then
 # prevents udevd to be killed by sendsigs (see #791944)
 mkdir -p $OMITDIR
 ln -sf $PIDFILE $OMITDIR/$NAME

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Andreas Henriksson]

Hello Trek,

On Mon, Mar 04, 2019 at 08:47:41PM +0100, Trek wrote:
> Control: tag -1 patch
> 
> hello,
> 
> the new dpkg version finally landed to testing, so I send you a little
> patch to enable both --notify-await and --wait-daemon options
> 
> this patch was tested against dpkg 1.19.5 and udev 241-1 (unstable)
[...]

You likely also want to include in your patch a debian/control
change that makes udev depend on a new enough dpkg which has the
new flags you're making use of.

Regards,
Andreas Henriksson

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

Control: tag -1 patch

hello,

the new dpkg version finally landed to testing, so I send you a little
patch to enable both --notify-await and --wait-daemon options

this patch was tested against dpkg 1.19.5 and udev 241-1 (unstable)

all is running fine! (boot and shutdown)

I would say a big thank you to everyone involved :)

Trekdiff --git a/debian/udev.init b/debian/udev.init
index 6a3c9b3926..cf9310734c 100644
--- a/debian/udev.init
+++ b/debian/udev.init
@@ -166,7 +166,8 @@ case "$1" in
 
 log_daemon_msg "Starting $DESC" "$NAME"
 if start-stop-daemon --start --name $NAME --user root --quiet \
---pidfile $PIDFILE --exec $DAEMON --background --make-pidfile; then
+--pidfile $PIDFILE --exec $DAEMON --background --make-pidfile \
+--notify-await; then
 # prevents udevd to be killed by sendsigs (see #791944)
 mkdir -p $OMITDIR
 ln -sf $PIDFILE $OMITDIR/$NAME
@@ -178,7 +179,7 @@ case "$1" in
 fi
 
 log_action_begin_msg "Synthesizing the initial hotplug events (subsystems)"
-if udevadm trigger --type=subsystems --action=add; then
+if udevadm trigger --type=subsystems --action=add --wait-daemon; then
 log_action_end_msg $?
 else
 log_action_end_msg $?
@@ -226,7 +227,8 @@ case "$1" in
 
 log_daemon_msg "Starting $DESC" "$NAME"
 if start-stop-daemon --start --name $NAME --user root --quiet \
---pidfile $PIDFILE --exec $DAEMON --background --make-pidfile; then
+--pidfile $PIDFILE --exec $DAEMON --background --make-pidfile \
+--notify-await; then
 # prevents udevd to be killed by sendsigs (see #791944)
 mkdir -p $OMITDIR
 ln -sf $PIDFILE $OMITDIR/$NAME

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Fri, 25 Jan 2019 19:14:54 -0800
Bill Brelsford  wrote:

> It works with a 2-second sleep inserted before the udevadm call.
> Sounds familiar..

too much familiar.. :)
well, at least we found this patch is absolutely useless regarding this
issue

hopefully it seems dpkg version 1.19.3 will include the notify-await
parameter, just in time for buster release

thank you for all your time and efforts in testing all the possible
solutions!

ciao

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Bill Brelsford]

With the tests removed, it builds and installs successfully.  But
during boot,

udevadm trigger --type=subsystems --action=add --wait-daemon

fails with something like

Failed to connect to udev daemon (connection refused).

It works with a 2-second sleep inserted before the udevadm call.
Sounds familiar..

Bill

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Thu, 24 Jan 2019 19:02:54 -0800
Bill Brelsford  wrote:

> Thanks for the build instructions, Trek.  However, dpkg-buildpackage
> fails (at test-condition, line 4259 in attached trace file).

to workaround these @#!%^&* tests, you should go in the systemd-240
directory and type as user:

  sed -i 's/\(main(.*) {\)/\1 return 0;/' src/test/test-*.c

then try again to rebuild:

  dpkg-buildpackage -b -uc -us


I hope it will work fine this time!
ciao

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Tue, 22 Jan 2019 18:21:43 -0800
Bill Brelsford  wrote:

> I'm still using version 1.19.2+test2 of dpkg, but without udev's
> --wait-daemon argument.  The next dpkg (1.19.3) will handle
> --wait-daemon; I assume including it in udev will fix the problem..?

these are two independent ways to fix the same issue:
1. modified dpkg to use start-stop-daemon with --notify-await parameter
2. modified udev to use udevadm trigger with --wait-daemon parameter

actually I don't know if the dpkg will be released with these
modifications in time for buster release

thanks to the work done by Michael Biebl, udev developers added the
--wait-daemon parameter


> Unfortunately, I'm not set up to re-compile so can't test your
> patches.

I can provide you a binary package with these patches applied, but it's
not a good security practice to install a package, that runs as root, if
it does not come from the package maintainers


if you want try to patch and compile yourself, you should type as root:

  apt install build-essential fakeroot

  echo 'deb-src http://httpredir.debian.org/debian buster main' \
>>/etc/apt/sources.list

  apt-get update

  LANG=C apt-get -s build-dep systemd | \
sed '/^ /!bA;H;$bA;d;:A;x;/The following NEW/!d;s/^The .*:\n//' \
>>/root/build-packages.txt

  apt-get build-dep systemd


then as a normal user, download the attached patch into your home
directory and type:

  mkdir ~/debian

  cd ~/debian

  apt-get source systemd

  cd systemd-240

  patch -p1 <~/udev-wait-daemon-full.patch

  dpkg-buildpackage -b -uc -us


finally you can install the binary package and remove the development
packages, as root:

  dpkg -i /home/user/debian/udev_240-4_i386.deb

  apt-get purge $(cat /root/build-packages.txt)

  apt-get purge build-essential fakeroot


for more info see https://wiki.debian.org/BuildingTutorial

thanks for your time!
ciaodiff -urN systemd-240/debian/udev.init systemd-240-new/debian/udev.init
--- systemd-240/debian/udev.init	2019-01-12 21:49:44.0 +0100
+++ systemd-240-new/debian/udev.init	2019-01-24 19:17:47.727632051 +0100
@@ -178,7 +178,7 @@
 fi
 
 log_action_begin_msg "Synthesizing the initial hotplug events (subsystems)"
-if udevadm trigger --type=subsystems --action=add; then
+if udevadm trigger --type=subsystems --action=add --wait-daemon; then
 log_action_end_msg $?
 else
 log_action_end_msg $?
diff -urN systemd-240/man/udevadm.xml systemd-240-new/man/udevadm.xml
--- systemd-240/man/udevadm.xml	2018-12-21 19:53:33.0 +0100
+++ systemd-240-new/man/udevadm.xml	2019-01-24 19:17:47.687632051 +0100
@@ -319,6 +319,14 @@
 the same command to finish.
   
 
+
+  --wait-daemon[=SECONDS]
+  
+Before triggering uevents, wait for systemd-udevd daemon to be initialized.
+Optionally takes timeout value. Default timeout is 5 seconds. This is equivalent to invoke
+invoking udevadm control --ping before udevadm trigger.
+  
+
 
 
   
diff -urN systemd-240/src/udev/udev-ctrl.c systemd-240-new/src/udev/udev-ctrl.c
--- systemd-240/src/udev/udev-ctrl.c	2018-12-21 19:53:33.0 +0100
+++ systemd-240-new/src/udev/udev-ctrl.c	2019-01-24 19:20:06.647633460 +0100
@@ -213,7 +213,7 @@
 
 DEFINE_TRIVIAL_REF_UNREF_FUNC(struct udev_ctrl_connection, udev_ctrl_connection, udev_ctrl_connection_free);
 
-static int ctrl_send(struct udev_ctrl *uctrl, enum udev_ctrl_msg_type type, int intval, const char *buf, int timeout) {
+static int ctrl_send(struct udev_ctrl *uctrl, enum udev_ctrl_msg_type type, int intval, const char *buf, usec_t timeout) {
 struct udev_ctrl_msg_wire ctrl_msg_wire;
 int err = 0;
 
@@ -246,7 +246,7 @@
 
 pfd[0].fd = uctrl->sock;
 pfd[0].events = POLLIN;
-r = poll(pfd, 1, timeout * MSEC_PER_SEC);
+r = poll(pfd, 1, DIV_ROUND_UP(timeout, USEC_PER_MSEC));
 if (r < 0) {
 if (errno == EINTR)
 continue;
@@ -267,35 +267,35 @@
 return err;
 }
 
-int udev_ctrl_send_set_log_level(struct udev_ctrl *uctrl, int priority, int timeout) {
+int udev_ctrl_send_set_log_level(struct udev_ctrl *uctrl, int priority, usec_t timeout) {
 return ctrl_send(uctrl, UDEV_CTRL_SET_LOG_LEVEL, priority, NULL, timeout);
 }
 
-int udev_ctrl_send_stop_exec_queue(struct udev_ctrl *uctrl, int timeout) {
+int udev_ctrl_send_stop_exec_queue(struct udev_ctrl *uctrl, usec_t timeout) {
 return ctrl_send(uctrl, UDEV_CTRL_STOP_EXEC_QUEUE, 0, NULL, timeout);
 }
 
-int udev_ctrl_send_start_exec_queue(struct udev_ctrl *uctrl, int timeout) {
+int udev_ctrl_send_start_exec_queue(struct udev_ctrl *uctrl, usec_t timeout) {
 return ctrl_send(uctrl, UDEV_CTRL_START_EXEC_QUEUE, 0, NULL, timeout);
 }
 
-int udev_ctrl_send_reload(struct udev_ctrl *uctrl, int timeout) {
+int 

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Bill Brelsford]

On Mon Jan 21 2019 at 07:14 AM -0800, Bill Brelsford wrote:
> On Mon Jan 21 2019 at 08:36 AM +0100, Trek wrote:
> > @Bill Brelsford: is the problem gone with the 240-4 version? if not,
> > can you test udev with these patches applied? thanks!
> 
> Yes, 240-4 fixed the problem.

Well, not quite -- it failed once today.  It has worked every other
time, so it does indeed sound like a race condition.

I'm still using version 1.19.2+test2 of dpkg, but without udev's
--wait-daemon argument.  The next dpkg (1.19.3) will handle
--wait-daemon; I assume including it in udev will fix the problem..?

Unfortunately, I'm not set up to re-compile so can't test your patches.

Bill

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Bill Brelsford]

On Mon Jan 21 2019 at 08:36 AM +0100, Trek wrote:
> @Bill Brelsford: is the problem gone with the 240-4 version? if not,
> can you test udev with these patches applied? thanks!

Yes, 240-4 fixed the problem.  Thanks Trek, Michael and others who
helped resolve it!

Regards..  Bill

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Mon, 21 Jan 2019 08:36:51 +0100
Trek  wrote:

> and the one in the attach

missed, sorry
attached to this message--- udev.init.orig	2019-01-12 21:49:44.0 +0100
+++ udev.init	2019-01-21 01:54:51.047997585 +0100
@@ -178,7 +178,7 @@
 fi
 
 log_action_begin_msg "Synthesizing the initial hotplug events (subsystems)"
-if udevadm trigger --type=subsystems --action=add; then
+if udevadm trigger --type=subsystems --action=add --wait-daemon; then
 log_action_end_msg $?
 else
 log_action_end_msg $?

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Sun, 13 Jan 2019 20:11:04 +0100
Michael Biebl  wrote:

> Can you test https://github.com/systemd/systemd/pull/11349

I have built and tested udev 240-4 with these patches:

https://github.com/systemd/systemd/commit/3797776e11d2a242517c3a20a953b5d0e80384f8.patch

https://github.com/systemd/systemd/commit/2001622c58c1989f386086d11bd2a00d5fe00a30.patch

and the one in the attach

it works, however I can't reproduce the bug if I remove the
--wait-daemon argument, so I am not sure in practice, but at least in
theory it seems to be the correct approach to fix this issue

@Michael Biebl: thank you for your work with the upstream maintainers

@Bill Brelsford: is the problem gone with the 240-4 version? if not,
can you test udev with these patches applied? thanks!

ciao

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Sun, 13 Jan 2019 20:11:04 +0100
Michael Biebl  wrote:

> Can you test https://github.com/systemd/systemd/pull/11349

I've just figured out how to download the patch from github

I assume that you would apply this patch

https://github.com/systemd/systemd/pull/11349/commits/4c234f0e6c03dd6f8cae2bc47e49a5f2e9a23d4b

on top of the 240 version?


this weekend I'll try to compile and test the package

is it possible to compile only udev without rebuilding all the systemd
packages?


ciao!

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

On Wed, 9 Jan 2019 15:05:38 +0100 Trek  wrote:

> if dpkg will not be updated with the new sd-notify interface, I think
> it would be better to rollback the #791944 patch: the system will not
> shutdown correctly with cryptsetup, but at least it can boot (with and
> without cryptsetup)

Can you test https://github.com/systemd/systemd/pull/11349

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

Version: 240-2

after upgrading from 239-13 to 240-2, the VGA card is no more
recognized on my system, but adding a small sleep fixes the problem

if dpkg will not be updated with the new sd-notify interface, I think
it would be better to rollback the #791944 patch: the system will not
shutdown correctly with cryptsetup, but at least it can boot (with and
without cryptsetup)

I can prepare a patch if needed

ciao!

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

Hi,

is there any blocking to fix this bug? can I help in some way?

this is nearly a release critical bug, as it "makes unrelated software
on the system break" (cryptsetup) and "the whole system" (some system
don't boot)

if dpkg can't be upgraded before the freeze, can we add a workaround to
the cryptsetup package? the workaround would be to remove the udev
control socket before sendsigs


thank you!

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Tue, 16 Oct 2018 22:26:40 +0200
Michael Biebl  wrote:

> > to Guilhem Moulin: I made a little patch because the socket
> > permissions seems to be wrong when --chuid is specified
> I think you meant Guillem (our dear dpkg maintainer) here?

yes, I'm sorry, please Guillem Jover can you check this patch?

ciao!>From 67d080cc7c195f1a34cb6a0dc7ac7a5d9dbad28d Mon Sep 17 00:00:00 2001
From: Trek 
Date: Tue, 16 Oct 2018 21:45:42 +0200
Subject: [PATCH] Set the proper permissions to s-s-d notify socket and
 directory

If the --chuid parameter is specified, the notify socket is not
accessible by the client, because mkdtemp() creates a directory owned
by root with 0700 permission. Moreover fchown() on a socket does not
have effects, because a socket doesn't have an associated inode.

Change the directory owner to runas_uid and use chown() instead of
fchown() to change the socket owner. Drop unneeded fchmod().
---
 utils/start-stop-daemon.c | 14 ++
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/utils/start-stop-daemon.c b/utils/start-stop-daemon.c
index 476b31b..5f14931 100644
--- a/utils/start-stop-daemon.c
+++ b/utils/start-stop-daemon.c
@@ -548,6 +548,9 @@ setup_socket_name(const char *suffix)
 
 	atexit(cleanup_socket_dir);
 
+	if (chown(notify_sockdir, runas_uid, runas_gid))
+		fatal("cannot change socket directory ownership");
+
 	if (asprintf(_socket, "%s/notify", notify_sockdir) < 0)
 		fatal("cannot allocate socket name");
 
@@ -578,7 +581,7 @@ create_notify_socket(void)
 	if (fcntl(fd, F_SETFD, flags | FD_CLOEXEC) < 0)
 		fatal("cannot set close-on-exec flag for notification socket");
 
-	sockname = setup_socket_name(".s-s-d-notify");
+	sockname = setup_socket_name("start-stop-daemon");
 
 	/* Bind to a socket in a temporary directory, selected based on
 	 * the platform. */
@@ -590,12 +593,7 @@ create_notify_socket(void)
 	if (rc < 0)
 		fatal("cannot bind to notification socket");
 
-	rc = fchmod(fd, 0660);
-	if (rc < 0)
-		fatal("cannot change notification socket permissions");
-
-	rc = fchown(fd, runas_uid, runas_gid);
-	if (rc < 0)
+	if (chown(su.sun_path, runas_uid, runas_gid))
 		fatal("cannot change notification socket ownership");
 
 	// XXX: verify we are talking to an expected child?? not sure whether
@@ -1446,7 +1444,7 @@ parse_options(int argc, char * const *argv)
 		badusage("--remove-pidfile requires --pidfile");
 
 	if (pid_str && pidfile)
-		badusage("need either --pid of --pidfile, not both");
+		badusage("need either --pid or --pidfile, not both");
 
 	if (background && action != ACTION_START)
 		badusage("--background is only relevant with --start");
-- 
2.1.4

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

Am 16.10.18 um 22:24 schrieb Trek:
> good job!
> 
> with cryptsetup the new patches are running fine
> 
> thank you to every one!
> 
> to Guilhem Moulin: I made a little patch because the socket permissions
> seems to be wrong when --chuid is specified
I think you meant Guillem (our dear dpkg maintainer) here?


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

good job!

with cryptsetup the new patches are running fine

thank you to every one!

to Guilhem Moulin: I made a little patch because the socket permissions
seems to be wrong when --chuid is specified

ciao :)>From 67d080cc7c195f1a34cb6a0dc7ac7a5d9dbad28d Mon Sep 17 00:00:00 2001
From: Trek 
Date: Tue, 16 Oct 2018 21:45:42 +0200
Subject: [PATCH] Set the proper permissions to s-s-d notify socket and
 directory

If the --chuid parameter is specified, the notify socket is not
accessible by the client, because mkdtemp() creates a directory owned
by root with 0700 permission. Moreover fchown() on a socket does not
have effects, because a socket doesn't have an associated inode.

Change the directory owner to runas_uid and use chown() instead of
fchown() to change the socket owner. Drop unneeded fchmod().
---
 utils/start-stop-daemon.c | 14 ++
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/utils/start-stop-daemon.c b/utils/start-stop-daemon.c
index 476b31b..5f14931 100644
--- a/utils/start-stop-daemon.c
+++ b/utils/start-stop-daemon.c
@@ -548,6 +548,9 @@ setup_socket_name(const char *suffix)
 
 	atexit(cleanup_socket_dir);
 
+	if (chown(notify_sockdir, runas_uid, runas_gid))
+		fatal("cannot change socket directory ownership");
+
 	if (asprintf(_socket, "%s/notify", notify_sockdir) < 0)
 		fatal("cannot allocate socket name");
 
@@ -578,7 +581,7 @@ create_notify_socket(void)
 	if (fcntl(fd, F_SETFD, flags | FD_CLOEXEC) < 0)
 		fatal("cannot set close-on-exec flag for notification socket");
 
-	sockname = setup_socket_name(".s-s-d-notify");
+	sockname = setup_socket_name("start-stop-daemon");
 
 	/* Bind to a socket in a temporary directory, selected based on
 	 * the platform. */
@@ -590,12 +593,7 @@ create_notify_socket(void)
 	if (rc < 0)
 		fatal("cannot bind to notification socket");
 
-	rc = fchmod(fd, 0660);
-	if (rc < 0)
-		fatal("cannot change notification socket permissions");
-
-	rc = fchown(fd, runas_uid, runas_gid);
-	if (rc < 0)
+	if (chown(su.sun_path, runas_uid, runas_gid))
 		fatal("cannot change notification socket ownership");
 
 	// XXX: verify we are talking to an expected child?? not sure whether
@@ -1446,7 +1444,7 @@ parse_options(int argc, char * const *argv)
 		badusage("--remove-pidfile requires --pidfile");
 
 	if (pid_str && pidfile)
-		badusage("need either --pid of --pidfile, not both");
+		badusage("need either --pid or --pidfile, not both");
 
 	if (background && action != ACTION_START)
 		badusage("--background is only relevant with --start");
-- 
2.1.4

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Bill Brelsford]

On Mon Oct 15 2018 at 12:19 AM +0200, Michael Biebl wrote:
> I was a bit too eager. The sd-notify support in dpkg was not quite ready
> yet. The dpkg maintainer has improved the code.
> Could you test again with version 1.19.2+test2, 

This one fixes it (3 successful boots so far).  Thanks, Michael.

-- 
Bill Brelsford
w...@k2di.net

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

Am 11.10.18 um 01:00 schrieb Bill Brelsford:
> On Wed Oct 10 2018 at 09:03 PM +0200, Michael Biebl wrote:
>> I've compiled a dpkg test package from this branch [1].
>> Bill, it would be great if you can install this dpkg package and try it
>> along with the attached patch for the udev init script.
> 
> I installed your new dpkg and patched the udev init script, but it
> doesn't work.  S-s-d apparently doesn't wait for udev to be ready.

I was a bit too eager. The sd-notify support in dpkg was not quite ready
yet. The dpkg maintainer has improved the code.
Could you test again with version 1.19.2+test2, please

Regards,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Bill Brelsford]

On Wed Oct 10 2018 at 09:03 PM +0200, Michael Biebl wrote:
> I've compiled a dpkg test package from this branch [1].
> Bill, it would be great if you can install this dpkg package and try it
> along with the attached patch for the udev init script.

I installed your new dpkg and patched the udev init script, but it
doesn't work.  S-s-d apparently doesn't wait for udev to be ready.

Bill

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

Am 10.10.18 um 22:28 schrieb Bill Brelsford:
> On Wed Oct 10 2018 at 09:03 PM +0200, Michael Biebl wrote:
>> I've compiled a dpkg test package from this branch [1].
>> Bill, it would be great if you can install this dpkg package and try it
>> along with the attached patch for the udev init script.
> 
> I'd be glad to, but that machine is running i686.  Can you make a
> 32-bit version..?

Sure, no problem. You can find them at the same URL.



-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#910707: Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

Am 10.10.18 um 21:00 schrieb Michael Biebl:
> I quickly looked at the branch, and noticed a a few typos. Patch attached.
  ^^^
oh the irony :-)


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Bill Brelsford]

On Wed Oct 10 2018 at 09:03 PM +0200, Michael Biebl wrote:
> Am 10.10.18 um 09:37 schrieb Guillem Jover:
> > Hah, this is already almost implemented. Was planning on finishing the
> > couple of XXX (including setting the envvar) and docs, testing and
> > merging it for 1.19.3 or so (targeted at 1w or 2w from now). :)
> > 
> >   
> > 
> 
> 
> I've compiled a dpkg test package from this branch [1].
> Bill, it would be great if you can install this dpkg package and try it
> along with the attached patch for the udev init script.

I'd be glad to, but that machine is running i686.  Can you make a
32-bit version..?

Bill

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

Am 10.10.18 um 09:37 schrieb Guillem Jover:
> Hah, this is already almost implemented. Was planning on finishing the
> couple of XXX (including setting the envvar) and docs, testing and
> merging it for 1.19.3 or so (targeted at 1w or 2w from now). :)
> 
>   
> 


I've compiled a dpkg test package from this branch [1].
Bill, it would be great if you can install this dpkg package and try it
along with the attached patch for the udev init script.

Regards,
Michael


[1] https://people.debian.org/~biebl/dpkg/

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
diff --git a/debian/udev.init b/debian/udev.init
index 9c394bb..73a23f4 100644
--- a/debian/udev.init
+++ b/debian/udev.init
@@ -166,7 +166,7 @@ case "$1" in
 
 log_daemon_msg "Starting $DESC" "$NAME"
 if start-stop-daemon --start --name $NAME --user root --quiet \
---pidfile $PIDFILE --exec $DAEMON --background --make-pidfile; then
+--pidfile $PIDFILE --exec $DAEMON --background --make-pidfile --notify-await; then
 # prevents udevd to be killed by sendsigs (see #791944)
 mkdir -p $OMITDIR
 ln -sf $PIDFILE $OMITDIR/$NAME
@@ -220,7 +220,7 @@ case "$1" in
 
 log_daemon_msg "Starting $DESC" "$NAME"
 if start-stop-daemon --start --name $NAME --user root --quiet \
---pidfile $PIDFILE --exec $DAEMON --background --make-pidfile; then
+--pidfile $PIDFILE --exec $DAEMON --background --make-pidfile --notify-await; then
 # prevents udevd to be killed by sendsigs (see #791944)
 mkdir -p $OMITDIR
 ln -sf $PIDFILE $OMITDIR/$NAME


signature.asc
Description: OpenPGP digital signature

Bug#910707: Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

On Wed, 10 Oct 2018 11:10:38 -0300 Felipe Sateler 
wrote:

> Awesome. This should help a lot.

This is great indeed, thanks Guillem!
I quickly looked at the branch, and noticed a a few typos. Patch attached.

Regards,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
diff --git a/utils/start-stop-daemon.c b/utils/start-stop-daemon.c
index 07b886b10..0b1449402 100644
--- a/utils/start-stop-daemon.c
+++ b/utils/start-stop-daemon.c
@@ -621,7 +621,7 @@ daemonize(void)
 		fatal("unable to do first fork");
 	else if (pid) { /* First Parent. */
 		if (notify_await) {
-			/* Wait for a readinnes notification from the second
+			/* Wait for a readiness notification from the second
 			 * child, so that we can safely exit when the service
 			 * is up. */
 			wait_for_notify(notify_fd);
@@ -731,7 +731,7 @@ usage(void)
 "  scheduler (default prio is 4)\n"
 "  -k, --umask change the umask to  before starting\n"
 "  -b, --background  force the process to detach\n"
-"  --notify-awaitwait for a readinnes notification\n"
+"  --notify-awaitwait for a readiness notification\n"
 "  --notify-timeout timeout after  seconds of notify wait\n"
 "  -C, --no-closedo not close any file descriptor\n"
 "  -m, --make-pidfilecreate the pidfile before starting\n"
@@ -1124,7 +1124,7 @@ parse_options(int argc, char * const *argv)
 		{ "iosched",	  1, NULL, 'I'},
 		{ "umask",	  1, NULL, 'k'},
 		{ "background",	  0, NULL, 'b'},
-		{ "notify-away",  0, NULL, OPT_NOTIFY_AWAIT},
+		{ "notify-await", 0, NULL, OPT_NOTIFY_AWAIT},
 		{ "notify-timeout", 0, NULL, OPT_NOTIFY_TIMEOUT},
 		{ "no-close",	  0, NULL, 'C'},
 		{ "make-pidfile", 0, NULL, 'm'},


signature.asc
Description: OpenPGP digital signature

Bug#910707: Bug#908796: udev (with sysvinit) fails to find devices at boot

[Felipe Sateler]

(Dropping original bug, adding cloned bug, full quote for context)

On Wed, Oct 10, 2018 at 4:38 AM Guillem Jover  wrote:

> Hi!
>
> On Tue, 2018-10-09 at 22:01:21 -0300, Felipe Sateler wrote:
> > Control: clone -1 -2
> > Control: retitle -2 start-stop-daemon: please implement service
> readiness protocol
> > Control: severity -2 wishlist
>
> > On Sun, Sep 30, 2018 at 8:15 PM Michael Biebl  wrote:
> > > The only supported way in udev to signal readiness is the sd-notify
> API.
> > > My gut feeling is, that having an sd-notify wrapper for non-systemd
> > > systems (e.g. directly built into start-stop-daemon via say an
> > > --wait-for-sd-notify switch) would be the cleanest and most robust way.
> > > This might even benefit other daemons besides udevd.
> >
> > Indeed, it seems to me that start-stop-daemon supporting the same service
> > readiness protocol systemd implements[1] would make things easier for
> udev
> > and other services.
> >
> > Short description for those not familiar: the service manager (or ssd in
> > this case) sets up a AF_UNIX socket, and passes on the address to the
> > daemon via the NOTIFY_SOCKET environment variable. When the daemon sends
> a
> > message containing READY=1, then ssd should consider the service up and
> it
> > can exit. More details at the linked manpage.
>
> Hah, this is already almost implemented. Was planning on finishing the
> couple of XXX (including setting the envvar) and docs, testing and
> merging it for 1.19.3 or so (targeted at 1w or 2w from now). :)
>
>   <
> https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/commit/?h=pu/s-s-d-notify
> >
>


Awesome. This should help a lot.

-- 

Saludos,
Felipe Sateler

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Guillem Jover]

Hi!

On Tue, 2018-10-09 at 22:01:21 -0300, Felipe Sateler wrote:
> Control: clone -1 -2
> Control: retitle -2 start-stop-daemon: please implement service readiness 
> protocol
> Control: severity -2 wishlist

> On Sun, Sep 30, 2018 at 8:15 PM Michael Biebl  wrote:
> > The only supported way in udev to signal readiness is the sd-notify API.
> > My gut feeling is, that having an sd-notify wrapper for non-systemd
> > systems (e.g. directly built into start-stop-daemon via say an
> > --wait-for-sd-notify switch) would be the cleanest and most robust way.
> > This might even benefit other daemons besides udevd.
> 
> Indeed, it seems to me that start-stop-daemon supporting the same service
> readiness protocol systemd implements[1] would make things easier for udev
> and other services.
> 
> Short description for those not familiar: the service manager (or ssd in
> this case) sets up a AF_UNIX socket, and passes on the address to the
> daemon via the NOTIFY_SOCKET environment variable. When the daemon sends a
> message containing READY=1, then ssd should consider the service up and it
> can exit. More details at the linked manpage.

Hah, this is already almost implemented. Was planning on finishing the
couple of XXX (including setting the envvar) and docs, testing and
merging it for 1.19.3 or so (targeted at 1w or 2w from now). :)

  

Thanks,
Guillem

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Felipe Sateler]

Control: clone -1 -2
Control: retitle -2 start-stop-daemon: please implement service readiness
protocol
Control: severity -2 wishlist



On Sun, Sep 30, 2018 at 8:15 PM Michael Biebl  wrote:

>
> The only supported way in udev to signal readiness is the sd-notify API.
> My gut feeling is, that having an sd-notify wrapper for non-systemd
> systems (e.g. directly built into start-stop-daemon via say an
> --wait-for-sd-notify switch) would be the cleanest and most robust way.
> This might even benefit other daemons besides udevd.
>

Indeed, it seems to me that start-stop-daemon supporting the same service
readiness protocol systemd implements[1] would make things easier for udev
and other services.

Short description for those not familiar: the service manager (or ssd in
this case) sets up a AF_UNIX socket, and passes on the address to the
daemon via the NOTIFY_SOCKET environment variable. When the daemon sends a
message containing READY=1, then ssd should consider the service up and it
can exit. More details at the linked manpage.

[1] https://manpages.debian.org/stretch/libsystemd-dev/sd_notify.3.en.html
-- 

Saludos,
Felipe Sateler

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Bill Brelsford]

On Tue Oct 09 2018 at 08:29 PM +0200, Michael Biebl wrote:
> On Mon, 1 Oct 2018 13:20:47 -0700 Bill Brelsford  wrote:
> > On Mon Oct 01 2018 at 06:45 AM +0200, Trek wrote:
> > > to Bill Brelsford: please, can you try again if this new patch fixes the
> > > problem? thank you!
> > 
> > No -- $CTRLFILE must already be there as it doesn't sleep ($timeout
> > stays at 150).
> 
> Just so I understand correctly: This patch didn't work for you?
> When udevadm --trigger was run, /run/udev/control existed but udevd did
> not successfully process the udevadm --trigger request?

Yes, that is correct.  I assume the control file existed since the
sleep loop was skipped, and udevadm --trigger was not successfully
processed.

Bill

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

On Sun, 7 Oct 2018 23:49:57 +0200 Trek  wrote:
> On Sun, 7 Oct 2018 01:57:31 +0200
> Michael Biebl  wrote:
> 
> > > - We tweak the LSB headers to make sure the udev init script is
> > > called before sendsigs on shutdown. This is important!
> > 
> > I have to add, that this change has the potential to significantly
> > change the shutdown order or cause a conflicting ordering, in case
> > there are other init scripts which declare an explicit shutdown
> > dependency.
> 
> yes, this would create a circular dependency

[..]

> this change should be mentioned in the NEWS.Debian file, as it could
> break any script that depends on udev on shutdown and that is not
> distributed by debian (open/closed source packages and system
> administrator scripts)

Yeah, on second thought, I don't think this proposal was actually such a
good idea.
We not only would have to coordinate that change tightly with
cryptsetup, which e.g. would make backports more complicated.
But more importantly this change would have the potential to break 3rd
party init scripts.

So let's discard this idea again.

Sorry for making you waste time on this.

Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

On Mon, 1 Oct 2018 13:20:47 -0700 Bill Brelsford  wrote:
> On Mon Oct 01 2018 at 06:45 AM +0200, Trek wrote:
> > to Bill Brelsford: please, can you try again if this new patch fixes the
> > problem? thank you!
> 
> No -- $CTRLFILE must already be there as it doesn't sleep ($timeout
> stays at 150).

Just so I understand correctly: This patch didn't work for you?
When udevadm --trigger was run, /run/udev/control existed but udevd did
not successfully process the udevadm --trigger request?


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Sun, 7 Oct 2018 01:57:31 +0200
Michael Biebl  wrote:

> > - We tweak the LSB headers to make sure the udev init script is
> > called before sendsigs on shutdown. This is important!
> 
> I have to add, that this change has the potential to significantly
> change the shutdown order or cause a conflicting ordering, in case
> there are other init scripts which declare an explicit shutdown
> dependency.

yes, this would create a circular dependency

before the 791944 patch, the shutdown sequence was:
K03sendsigs (before umountnfs)
K05umountnfs(before umountfs)
K07umountfs (before umountroot)
K08cryptdisks   (after umountfs, before umountroot udev)
K10umountroot

if udev is stopped before sendsigs:
- cryptdisks should be stopped before udev
- umountfs should be stopped before cryptdisks
- umountnfs should be stopped before umountfs
- but sendsigs cannot be stopped after umountnfs

it can be easily resolved removing the udev entry in the Should-Stop
header of cryptdisks (and cryptdisks-early)


> Looking at https://codesearch.debian.net/ and searching for packages
> which declare a Should-Stop or Required-Stop on udev and checking if
> any late shutdown services are involved.

wow! I didn't know this site

with this simple search "Stop(|-After):.*udev", it returned these
packages that could be affected: multipath-tools, pcsc-lite,
network-manager, mouseemu, pulseaudio, cryptsetup, nut

it seems to me none of these packages would break (except cryptsetup)

this change should be mentioned in the NEWS.Debian file, as it could
break any script that depends on udev on shutdown and that is not
distributed by debian (open/closed source packages and system
administrator scripts)

the patches attached are tested on a system with swaps and filesystems
encrypted in plain mode (no luks, no cryptroot)

ciao!>From b8f8bcfa1c447bc0839a84c4f64a551278a33dec Mon Sep 17 00:00:00 2001
From: Trek 
Date: Sun, 7 Oct 2018 17:59:25 +0200
Subject: [PATCH] Stop udev before sendsigs to fix a regression of #791944

The start-stop-daemon command with the --background argument returns
immediately, too soon to trigger events on some systems. The #791944
patch is partly reverted and a dependency to sendsigs is added in the
SysV init script at shutdown.

Closes: #908796
---
 debian/udev.init | 19 +--
 1 file changed, 5 insertions(+), 14 deletions(-)

diff --git a/debian/udev.init b/debian/udev.init
index 9c394bb..794373d 100644
--- a/debian/udev.init
+++ b/debian/udev.init
@@ -3,6 +3,7 @@
 # Provides:  udev
 # Required-Start:mountkernfs
 # Required-Stop: umountroot
+# Should-Stop:   sendsigs
 # Default-Start: S
 # Default-Stop:  0 6
 # Short-Description: Start systemd-udevd, populate /dev and load drivers.
@@ -12,9 +13,7 @@ PATH="/sbin:/bin"
 NAME="systemd-udevd"
 DAEMON="/lib/systemd/systemd-udevd"
 DESC="hotplug events dispatcher"
-PIDFILE="/run/udev.pid"
 CTRLFILE="/run/udev/control"
-OMITDIR="/run/sendsigs.omit.d"
 
 # we need to unmount /dev/pts/ and remount it later over the devtmpfs
 unmount_devpts() {
@@ -165,11 +164,7 @@ case "$1" in
 [ -x /sbin/restorecon ] && /sbin/restorecon -R /dev
 
 log_daemon_msg "Starting $DESC" "$NAME"
-if start-stop-daemon --start --name $NAME --user root --quiet \
---pidfile $PIDFILE --exec $DAEMON --background --make-pidfile; then
-# prevents udevd to be killed by sendsigs (see #791944)
-mkdir -p $OMITDIR
-ln -sf $PIDFILE $OMITDIR/$NAME
+if $DAEMON --daemon; then
 log_end_msg $?
 else
 log_warning_msg $?
@@ -198,7 +193,7 @@ case "$1" in
 stop)
 log_daemon_msg "Stopping $DESC" "$NAME"
 if start-stop-daemon --stop --name $NAME --user root --quiet \
---pidfile $PIDFILE --remove-pidfile --oknodo --retry 5; then
+--oknodo --retry 5; then
 # prevents cryptsetup/dmsetup hangs (see #791944)
 rm -f $CTRLFILE
 log_end_msg $?
@@ -210,7 +205,7 @@ case "$1" in
 restart)
 log_daemon_msg "Stopping $DESC" "$NAME"
 if start-stop-daemon --stop --name $NAME --user root --quiet \
---pidfile $PIDFILE --remove-pidfile --oknodo --retry 5; then
+--oknodo --retry 5; then
 # prevents cryptsetup/dmsetup hangs (see #791944)
 rm -f $CTRLFILE
 log_end_msg $?
@@ -219,11 +214,7 @@ case "$1" in
 fi
 
 log_daemon_msg "Starting $DESC" "$NAME"
-if start-stop-daemon --start --name $NAME --user root --quiet \
---pidfile $PIDFILE --exec $DAEMON --background --make-pidfile; then
-# prevents udevd to be killed by sendsigs (see #791944)
-mkdir -p $OMITDIR
-ln -sf $PIDFILE $OMITDIR/$NAME
+if $DAEMON --daemon; then
 log_end_msg $?
 else
 log_end_msg $?
-- 
2.1.4

>From 46ef94eb04ea03b7ef53b156439eef22e3c0a054 Mon Sep 17 00:00:00 2001
From: Trek 
Date: Sun, 7 Oct 2018 17:57:54 +0200
Subject: [PATCH] Remove the shutdown dependency on udev to fix a regression 

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

Am 07.10.18 um 01:57 schrieb Michael Biebl:
> Am 07.10.18 um 01:06 schrieb Michael Biebl:
> 
>> We change the init script as follows:
>> - We revert most the changes from #791944, we only keep the stop
>> symlinks for rc0 and rc6 and the cleanup of /run/udev/control file on stop.
>> - We tweak the LSB headers to make sure the udev init script is called
>> before sendsigs on shutdown. This is important!
>>
>> This should ensure that udevd is properly killed and the control file
>> removed, i.e. there should be (almost) no time window where the udevd
>> daemon is not running but the control file exists.
>>
>> On the other hand, this would mean we again stop udevd earlier during
>> shutdown, but we mostly had this behaviour already in previous releases
>> where udevd is killed by sendsigs.
> 
> I have to add, that this change has the potential to significantly
> change the shutdown order or cause a conflicting ordering, in case there
> are other init scripts which declare an explicit shutdown dependency.

To answer my own question:
Adding "Should-Stop: sendsigs" to udev will not work as it creates a
dependency cycle with cryptdisks and cryptdisks-early (which have
Should-Stop: udev, so require udev to stop *after* themselves).

Trek, what was the problem with checking for /run/udev/control to
determine if the udev service is available or not?


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

Am 07.10.18 um 01:06 schrieb Michael Biebl:

> We change the init script as follows:
> - We revert most the changes from #791944, we only keep the stop
> symlinks for rc0 and rc6 and the cleanup of /run/udev/control file on stop.
> - We tweak the LSB headers to make sure the udev init script is called
> before sendsigs on shutdown. This is important!
> 
> This should ensure that udevd is properly killed and the control file
> removed, i.e. there should be (almost) no time window where the udevd
> daemon is not running but the control file exists.
> 
> On the other hand, this would mean we again stop udevd earlier during
> shutdown, but we mostly had this behaviour already in previous releases
> where udevd is killed by sendsigs.

I have to add, that this change has the potential to significantly
change the shutdown order or cause a conflicting ordering, in case there
are other init scripts which declare an explicit shutdown dependency.

This will need someone to further investigate and test it.

Looking at https://codesearch.debian.net/ and searching for packages
which declare a Should-Stop or Required-Stop on udev and checking if any
late shutdown services are involved.

Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

Am 05.10.18 um 10:25 schrieb Trek:
> On Thu, 4 Oct 2018 21:58:38 +0200
> Michael Biebl  wrote:
> 
>>> 1) revert the 791944 patch, create a new init.d/udev-clear script to
>>> remove the control file and run it just after sendsigs (this will
>>> restore the old well tested behavior) 
>>
>> The removal of the control file should be bound to the live time of
>> the udev service, so splitting it off into a separate init script is
>> not a good idea.
> 
> yes, to be sure, we should call udev-clear before sendsigs
> 
> this will mimic the old behavior, where the control file doesn't exists
> after sendsigs
> 
> I think this is the first step we should do to restore the
> functionality, then we have more time to find the proper solution, but
> at least we will not miss another stable release without a functioning
> udev + sysvinit + dmsetup/cryptdisks

Tbh, I'm not interested in a quick hack, which we later have to clean up
again and is likely to break in other slightly odd ways.
And a new, separate init script feels like this.

>> A proper solution might (emphasis on might, as I haven't checked this)
>> be to teach start-stop-daemon about daemons using the sd-notify
>> interface. This is a more elaborate fix, for sure, but everything else
>> feels like an incomplete hack.
> 
> yes, probably this is a good solution
> 
> as start-stop-daemon is called in parallel, it could be required to
> create a different notify socket per instance, but luckily the client
> library manage the NOTIFY_SOCKET environment variable

Would you be willing to talk to the dpkg/start-stop-daemon maintainer to
gather his input on this matter?
Someone else willing/interested in doing this?

> another possible solution is to add an option to udev to remove the
> control file on exit, as it was in the past

Changing systemd-udevd to unlink the control file itself would work.
This would require someone to talk to upstream though and convince them
to make this change. We've been tirelessly working on reducing the
amount of downstream patches and I don't want to add a downstream patch
which we'd never get rid off again.

So, aside from these two options (providing sd-notify support in s-s-d,
unlinking the control in systemd-udevd directly), what you said above
got me thinking:

We change the init script as follows:
- We revert most the changes from #791944, we only keep the stop
symlinks for rc0 and rc6 and the cleanup of /run/udev/control file on stop.
- We tweak the LSB headers to make sure the udev init script is called
before sendsigs on shutdown. This is important!

This should ensure that udevd is properly killed and the control file
removed, i.e. there should be (almost) no time window where the udevd
daemon is not running but the control file exists.

On the other hand, this would mean we again stop udevd earlier during
shutdown, but we mostly had this behaviour already in previous releases
where udevd is killed by sendsigs.

I hope the idea is clear, if not, please don't hesitate to ask.


Trek, would you be willing to work on an patch for this?


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Thu, 4 Oct 2018 21:58:38 +0200
Michael Biebl  wrote:

> > 1) revert the 791944 patch, create a new init.d/udev-clear script to
> > remove the control file and run it just after sendsigs (this will
> > restore the old well tested behavior) 
> 
> The removal of the control file should be bound to the live time of
> the udev service, so splitting it off into a separate init script is
> not a good idea.

yes, to be sure, we should call udev-clear before sendsigs

this will mimic the old behavior, where the control file doesn't exists
after sendsigs

I think this is the first step we should do to restore the
functionality, then we have more time to find the proper solution, but
at least we will not miss another stable release without a functioning
udev + sysvinit + dmsetup/cryptdisks


> Afaics this problem is unfortunately not really fixable with the
> limited facilities sysvinit/sysv-rc provides.

this is why I think we should consider to live with a workaround, that
at least allows the users to boot and shutdown the system


> A proper solution might (emphasis on might, as I haven't checked this)
> be to teach start-stop-daemon about daemons using the sd-notify
> interface. This is a more elaborate fix, for sure, but everything else
> feels like an incomplete hack.

yes, probably this is a good solution

as start-stop-daemon is called in parallel, it could be required to
create a different notify socket per instance, but luckily the client
library manage the NOTIFY_SOCKET environment variable

another possible solution is to add an option to udev to remove the
control file on exit, as it was in the past

thank you and ciao!

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

Am 03.10.18 um 13:20 schrieb Trek:
> 
> thanks to Bill Brelsford, the last test confirmed we are in the worst
> situation: udev is running, the control file is created, but udev is
> not ready to listen new events
> 
> so we must to rethink about the 791944 bug: it was caused because udev
> no longer removes the control file on stop
> 
> we have at least three options to workaround it:
> 
> 1) revert the 791944 patch, create a new init.d/udev-clear script to
> remove the control file and run it just after sendsigs (this will
> restore the old well tested behavior) 

The removal of the control file should be bound to the live time of the
udev service, so splitting it off into a separate init script is not a
good idea.

> 2) revert the 791944 patch, remove the control file on stop in
> init.d/udev, stop it after sendsigs and remove udev from the Should-Stop
> header in any script, probably cryptdisks, mdadm and lvm2 (this could
> break any script that depends on udev and not distributed by debian)

If you revert 791944, i.e. don't add the pid file to
/run/sendsigs.omit.d, the systemd-udevd process will be killed by the
sendsigs init script. Which means there is a time window where
/run/udev/control exists but no functional udev service. In other words,
not a proper solution either.

> 3) do not revert, but start with udevd --background and create the
> pidfile using pidof udevd (this could break if there are more than one
> process of udevd)

As you already mentioned, an approach using pidof is not going to work.
For one thing, containers are a thing nowadays (and some containers run
udev), and second, udevd will spawn off worker processes. Using pidof
you don't know which process you will actually get.

> what do you think about?

None of the above proposals does really solve the problem, unfortunately.

Afaics this problem is unfortunately not really fixable with the limited
facilities sysvinit/sysv-rc provides.
I'm of course happy if someone proves me wrong.

Regarding the changes that were made for #791944, I'm ok with reverting
them, if you want me to. Which means we will simply break the boot for
another set of (sysvinit) users.

A proper solution might (emphasis on might, as I haven't checked this)
be to teach start-stop-daemon about daemons using the sd-notify
interface. This is a more elaborate fix, for sure, but everything else
feels like an incomplete hack.

Michael



-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

thanks to Bill Brelsford, the last test confirmed we are in the worst
situation: udev is running, the control file is created, but udev is
not ready to listen new events

so we must to rethink about the 791944 bug: it was caused because udev
no longer removes the control file on stop

we have at least three options to workaround it:

1) revert the 791944 patch, create a new init.d/udev-clear script to
remove the control file and run it just after sendsigs (this will
restore the old well tested behavior) 

2) revert the 791944 patch, remove the control file on stop in
init.d/udev, stop it after sendsigs and remove udev from the Should-Stop
header in any script, probably cryptdisks, mdadm and lvm2 (this could
break any script that depends on udev and not distributed by debian)

3) do not revert, but start with udevd --background and create the
pidfile using pidof udevd (this could break if there are more than one
process of udevd)

what do you think about?

thanks for your time
Trek

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Bill Brelsford]

On Mon Oct 01 2018 at 06:45 AM +0200, Trek wrote:
> to Bill Brelsford: please, can you try again if this new patch fixes the
> problem? thank you!

No -- $CTRLFILE must already be there as it doesn't sleep ($timeout
stays at 150).

(Even so, it worked consistently for a number of boots, then began
to fail consistently.  Strange.  Apparently it doesn't need much of
a pause, on my hardware anyway..)

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Mon, 1 Oct 2018 01:11:30 +0200
Michael Biebl  wrote:

> The only supported way in udev to signal readiness is the sd-notify
> API. My gut feeling is, that having an sd-notify wrapper for
> non-systemd systems (e.g. directly built into start-stop-daemon via
> say an --wait-for-sd-notify switch) would be the cleanest and most
> robust way. This might even benefit other daemons besides udevd.

nice catch, but probably it cannot be done in time for buster freeze

after another code inspection, I wrote a simpler version of the patch,
as it only checks for the existence of the control file

it should be sufficient as udev --daemon forks and exits just after
calling listen_fds(), that creates the control file via bind()

to Bill Brelsford: please, can you try again if this new patch fixes the
problem? thank you!

ciao>From e1322be4d6416259a0e3e1e4370e8ff722cda146 Mon Sep 17 00:00:00 2001
From: Trek 
Date: Mon, 1 Oct 2018 06:38:35 +0200
Subject: [PATCH] Wait for udev to be ready before trigger under sysvinit

The start-stop-daemon command with the --background argument returns
immediately, too soon to trigger events on some systems. Update the
SysV init script to wait until udev is ready, checking the control file.

Closes: #908796
---
 debian/udev.init | 13 +
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/debian/udev.init b/debian/udev.init
index 9c394bb..374df4e 100644
--- a/debian/udev.init
+++ b/debian/udev.init
@@ -170,11 +170,16 @@ case "$1" in
 # prevents udevd to be killed by sendsigs (see #791944)
 mkdir -p $OMITDIR
 ln -sf $PIDFILE $OMITDIR/$NAME
-log_end_msg $?
+
+# wait for udev to be ready (see #908796)
+timeout=150
+until [ -S $CTRLFILE -o $timeout -le 0 ]; do
+timeout=$((timeout - 1))
+sleep 0.1
+done
+[ -S $CTRLFILE ] && log_success_msg || log_failure_msg
 else
-log_warning_msg $?
-log_warning_msg "Waiting 15 seconds and trying to continue anyway"
-sleep 15
+log_failure_msg
 fi
 
 log_action_begin_msg "Synthesizing the initial hotplug events"
-- 
2.1.4

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

Am 01.10.18 um 00:55 schrieb Trek:
> On Sat, 29 Sep 2018 18:41:45 +0200
> Michael Biebl  wrote:
> 
>>> +# wait for udev to be ready (see #908796)
>>> +timeout=15
>>> +until udevadm control -S || [ $timeout -le 0 ]; do
>>> +timeout=$((timeout-1))
>>> +sleep 1
>>> +done
>>> +udevadm control -S && log_success_msg || log_failure_msg
>>
>> This repeatedly tries to start the execution queue.
>> Besides having this side-effect, why is this a proper test to check if
>> udev is ready?
> 
> this probably shows my ignorance about udev (and lack of documentation),
> but it seems to me there isn't a proper way to test if udev is ready

The only supported way in udev to signal readiness is the sd-notify API.
My gut feeling is, that having an sd-notify wrapper for non-systemd
systems (e.g. directly built into start-stop-daemon via say an
--wait-for-sd-notify switch) would be the cleanest and most robust way.
This might even benefit other daemons besides udevd.

Similar to what apulse is for PulseAudio clients without a PulseAudio
daemon.
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Sat, 29 Sep 2018 18:41:45 +0200
Michael Biebl  wrote:

> > +# wait for udev to be ready (see #908796)
> > +timeout=15
> > +until udevadm control -S || [ $timeout -le 0 ]; do
> > +timeout=$((timeout-1))
> > +sleep 1
> > +done
> > +udevadm control -S && log_success_msg || log_failure_msg
> 
> This repeatedly tries to start the execution queue.
> Besides having this side-effect, why is this a proper test to check if
> udev is ready?

this probably shows my ignorance about udev (and lack of documentation),
but it seems to me there isn't a proper way to test if udev is ready

as the bug doesn't affect my system, I cannot know the state of udev
when the udevadm trigger is launched: it could be the control file is
missing or it's created but the daemon is not already accepting events

reading the source, there is a ping message that is recognized by the
control socket, sent internally by udevadm settle, but there isn't a
command line for that message in particular

it seems to me the least dangerous command is to start the execution
queue: in fact when udev is just started, the stop_exec_queue variable
is set to false, then the start of exec queue command will set again
stop_exec_queue to false and it should results in a no-op


we could simply rerun udevadm trigger until it's ok, but it seems not a
good idea https://github.com/kubernetes/kubernetes/issues/55007

may be their fix (calling settle with timeout before trigger) will work
for us? I don't know, as there are many possible side effects
depending on the hardware


it could be a good idea to ask the developers, but reading
https://github.com/systemd/systemd/issues/2477 it seems they are not
really interested about this type of issue and in forward compatibility
(it was running fine until they stopped to remove the control socked on
exit)


I hope I have answered your question, anyway tell me if I can do
something more

ciao!

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Michael Biebl]

Hi Trek,

thanks for the patch

Am 15.09.18 um 06:28 schrieb Trek:
> +
> +# wait for udev to be ready (see #908796)
> +timeout=15
> +until udevadm control -S || [ $timeout -le 0 ]; do
> +timeout=$((timeout-1))
> +sleep 1
> +done
> +udevadm control -S && log_success_msg || log_failure_msg

This repeatedly tries to start the execution queue.
Besides having this side-effect, why is this a proper test to check if
udev is ready?


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Trek]

On Thu, 13 Sep 2018 19:02:26 -0700
Bill Brelsford  wrote:

> With the --background argument, a race condition exists and
> "udevadm trigger" starts too soon.
> A workaround is to add a short sleep:

can you try if this patch fixes the problem?
my system is not affected so I can't check

thank you!


the sleep timeout could be different depending on the hardware, so it
checks every second if udev is ready, up to 15 seconds

it seems to be there is no reliable method to know if udev is running
fine, so it calls udevadm control -S, that it's essentially a no-op
when udev is just started, but at least it checks if the control socket
is listening

ciao
>From 1c9dd060a25d904aa200c9fdc33e34dc003dd1d8 Mon Sep 17 00:00:00 2001
From: Trek 
Date: Sat, 15 Sep 2018 05:50:03 +0200
Subject: [PATCH] Wait for udev to be ready before trigger under sysvinit

The start-stop-daemon command with the --background argument returns
immediately, too soon to trigger events on some systems. Update the
SysV init script to wait until udev is ready.

Closes: #908796
---
 debian/udev.init | 13 +
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/debian/udev.init b/debian/udev.init
index 9c394bb..4b30297 100644
--- a/debian/udev.init
+++ b/debian/udev.init
@@ -170,11 +170,16 @@ case "$1" in
 # prevents udevd to be killed by sendsigs (see #791944)
 mkdir -p $OMITDIR
 ln -sf $PIDFILE $OMITDIR/$NAME
-log_end_msg $?
+
+# wait for udev to be ready (see #908796)
+timeout=15
+until udevadm control -S || [ $timeout -le 0 ]; do
+timeout=$((timeout-1))
+sleep 1
+done
+udevadm control -S && log_success_msg || log_failure_msg
 else
-log_warning_msg $?
-log_warning_msg "Waiting 15 seconds and trying to continue anyway"
-sleep 15
+log_failure_msg
 fi
 
 log_action_begin_msg "Synthesizing the initial hotplug events"
-- 
2.1.4

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Bill Brelsford]

On Sat Sep 15 2018 at 06:28 AM +0200, Trek wrote:
> On Thu, 13 Sep 2018 19:02:26 -0700
> Bill Brelsford  wrote:
> 
> > With the --background argument, a race condition exists and
> > "udevadm trigger" starts too soon.
> > A workaround is to add a short sleep:
> 
> can you try if this patch fixes the problem?
> my system is not affected so I can't check

That fixes it (worked twice, anyway).  Thanks!

Bill

Bug#908796: udev (with sysvinit) fails to find devices at boot

[Bill Brelsford]

Package: udev
Version: 239-9
Severity: important

After upgrading udev to 239-8, most devices are no longer detected
during boot (e.g. usb, network card, video, ..).  The problem
appears to be caused by /etc/init.d/udev now using
start-stop-daemon (thanks Sven Joachim for pointing me there).
With the --background argument, a race condition exists and
"udevadm trigger" starts too soon.

A workaround is to add a short sleep:

if start-stop-daemon --start --name $NAME --user root --quiet \
--pidfile $PIDFILE --exec $DAEMON --background --make-pidfile; then
sleep 2

Regards..  Bill


-- Package-specific info:

-- System Information:
Debian Release: buster/sid
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: i386 (i686)

Kernel: Linux 4.18.0-1-686-pae (SMP w/2 CPU cores)
Locale: LANG=en_US, LC_CTYPE=en_US (charmap=ISO-8859-1), LANGUAGE=en_US 
(charmap=ISO-8859-1)
Shell: /bin/sh linked to /bin/bash
Init: sysvinit (via /sbin/init)

Versions of packages udev depends on:
ii  adduser  3.117
ii  libacl1  2.2.52-3+b1
ii  libblkid12.32.1-0.1
ii  libc62.27-6
ii  libkmod2 25-1
ii  libselinux1  2.8-1+b1
ii  libudev1 239-9
ii  lsb-base 9.20170808
ii  procps   2:3.3.15-2
ii  util-linux   2.32.1-0.1

udev recommends no packages.

udev suggests no packages.

Versions of packages udev is related to:
pn  systemd  

-- Configuration Files:
/etc/init.d/udev changed [not included]

-- debconf information:
  udev/new_kernel_needed: false
  udev/sysfs_deprecated_incompatibility:
  udev/title/upgrade:
  udev/reboot_needed:

-- 
Bill Brelsford
P: /devices/LNXSYSTM:00
E: DEVPATH=/devices/LNXSYSTM:00
E: ID_VENDOR_FROM_DATABASE=The Linux Foundation
E: MODALIAS=acpi:LNXSYSTM:
E: SUBSYSTEM=acpi
E: USEC_INITIALIZED=12650789

P: /devices/LNXSYSTM:00/LNXCPU:00
E: DEVPATH=/devices/LNXSYSTM:00/LNXCPU:00
E: ID_VENDOR_FROM_DATABASE=The Linux Foundation
E: MODALIAS=acpi:LNXCPU:
E: SUBSYSTEM=acpi
E: USEC_INITIALIZED=12678811

P: /devices/LNXSYSTM:00/LNXCPU:01
E: DEVPATH=/devices/LNXSYSTM:00/LNXCPU:01
E: ID_VENDOR_FROM_DATABASE=The Linux Foundation
E: MODALIAS=acpi:LNXCPU:
E: SUBSYSTEM=acpi
E: USEC_INITIALIZED=12924018

P: /devices/LNXSYSTM:00/LNXPWRBN:00
E: DEVPATH=/devices/LNXSYSTM:00/LNXPWRBN:00
E: DRIVER=button
E: ID_VENDOR_FROM_DATABASE=The Linux Foundation
E: MODALIAS=acpi:LNXPWRBN:
E: SUBSYSTEM=acpi
E: USEC_INITIALIZED=12924015

P: /devices/LNXSYSTM:00/LNXPWRBN:00/input/input12
E: DEVPATH=/devices/LNXSYSTM:00/LNXPWRBN:00/input/input12
E: EV=3
E: ID_INPUT=1
E: ID_INPUT_KEY=1
E: ID_PATH=acpi-LNXPWRBN:00
E: ID_PATH_TAG=acpi-LNXPWRBN_00
E: KEY=10 0 0 0
E: MODALIAS=input:b0019vp0001e-e0,1,k74,ramlsfw
E: NAME="Power Button"
E: PHYS="LNXPWRBN/button/input0"
E: PRODUCT=19/0/1/0
E: PROP=0
E: SUBSYSTEM=input
E: USEC_INITIALIZED=15647854

P: /devices/LNXSYSTM:00/LNXPWRBN:00/input/input12/event5
N: input/event5
E: BACKSPACE=guess
E: DEVNAME=/dev/input/event5
E: DEVPATH=/devices/LNXSYSTM:00/LNXPWRBN:00/input/input12/event5
E: ID_INPUT=1
E: ID_INPUT_KEY=1
E: ID_PATH=acpi-LNXPWRBN:00
E: ID_PATH_TAG=acpi-LNXPWRBN_00
E: LIBINPUT_DEVICE_GROUP=19/0/1:LNXPWRBN/button
E: MAJOR=13
E: MINOR=69
E: SUBSYSTEM=input
E: TAGS=:power-switch:
E: USEC_INITIALIZED=17837919
E: XKBLAYOUT=us
E: XKBMODEL=pc104
E: XKBOPTIONS=terminate:ctrl_alt_bksp

P: /devices/LNXSYSTM:00/LNXSYBUS:00
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00
E: ID_VENDOR_FROM_DATABASE=The Linux Foundation
E: MODALIAS=acpi:LNXSYBUS:
E: SUBSYSTEM=acpi
E: USEC_INITIALIZED=12924657

P: /devices/LNXSYSTM:00/LNXSYBUS:00/ATK0100:00
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/ATK0100:00
E: DRIVER=Asus Laptop Support
E: ID_VENDOR_FROM_DATABASE=Allied Telesyn Int'l
E: MODALIAS=acpi:ATK0100:
E: SUBSYSTEM=acpi
E: USEC_INITIALIZED=12967738

P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00
E: ID_VENDOR_FROM_DATABASE=The Linux Foundation
E: MODALIAS=acpi:PNP0A08:PNP0A03:
E: SUBSYSTEM=acpi
E: USEC_INITIALIZED=12926635

P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/ACPI0003:00
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/ACPI0003:00
E: DRIVER=ac
E: ID_VENDOR_FROM_DATABASE=The Linux Foundation
E: MODALIAS=acpi:ACPI0003:
E: SUBSYSTEM=acpi
E: USEC_INITIALIZED=12967880

P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/ACPI0003:00/power_supply/AC0
E: 
DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/ACPI0003:00/power_supply/AC0
E: POWER_SUPPLY_NAME=AC0
E: POWER_SUPPLY_ONLINE=1
E: SUBSYSTEM=power_supply

P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00
E: DRIVER=video
E: ID_VENDOR_FROM_DATABASE=The Linux Foundation
E: MODALIAS=acpi:LNXVIDEO:
E: SUBSYSTEM=acpi
E: USEC_INITIALIZED=12928294

P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/device:1e
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/device:1e
E: SUBSYSTEM=acpi

P: