Package: sysvinit-utils Version: 2.88dsf-58 Severity: important Tags: patch
Hello all, Due to the way systemd job transactions work, we had and still have several occasions where DHCP/ifupdown and similar hooks try to reload or restart a service and cause a deadlock. That's because especially during boot and shutdown there is a big transaction of services and their dependencies which want to start or stop, and if within that transaction you synchronously request another one you get a circular dependency loop which causes such a deadlock. Examples are https://bugs.debian.org/652942 (still open and reproducible), and https://bugs.debian.org/635777 and https://bugs.debian.org/624599. For the latter two the systemd package currently has workarounds, but they don't always apply. This is an emergent problem as we try to combine two completely different paradigms: * Under SysV init, commands like "service foo restart" are always synchronous, i. e. they wait for the operation to finish. But they don't take any dependencies into account, i. e. restarting "foo" won't ensure that foo's dependencies are already started. * Under systemd, starting a unit always ensures that its dependencies (and ordering constraints) are fulfilled. But in that world, things like DHCP hooks should reload services asynchronously. (E. g. with systemctl --no-block reload foo) Both are valid, the trouble comes with trying to combine them. Our "service" and "invoke-rc.d" hooks work under all init systems and don't offer async operations. This was discussed several times on the systemd ML, like on http://lists.freedesktop.org/archives/systemd-devel/2014-July/021457.html (continuation: http://lists.freedesktop.org/archives/systemd-devel/2014-August/022048.html) http://lists.freedesktop.org/archives/systemd-devel/2015-February/027966.html But in the end all of those are just hacks which break systemd's semantics. There is fundamentally no good way how we can fulfill both the SysV and the systemd semantics, so we need some heuristics/workarounds. One such thing would be that service/invoke-rc.d always call systemctl with --job-mode=ignore-dependencies which would make it do what SysV does: only operate on the given job but not its dependencies. But often this behaviour is actually handy, and the lockups realistically only occur on boot and shutdown, so we can restrict it to these times. That's what the attached patch does. After long discussions with both systemd upstream as well as Michael Biebl I am now convinced that this is the most sensible approach, doesn't break either SysV or systemctl semantics, and is a good enough heuristics to avoid deadlocks. Such deadlocks in principle happen during runtime as well, but there's no evidence of that so far; if it does happen, we need to adjust the scope of the "ignore-dependencies", but I don't think it will be necessary. With this I verified that all of the bugs above (plus https://launchpad.net/bugs/1417010, which is the Ubuntu equivalent of https://bugs.debian.org/652942) are fixed with a pristine upstream systemd, i. e. we can drop our insufficient patches. Thanks for considering, Martin -- Martin Pitt | http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
diff -Nru sysvinit-2.88dsf/debian/changelog sysvinit-2.88dsf/debian/changelog --- sysvinit-2.88dsf/debian/changelog 2014-11-11 20:34:28.000000000 +0100 +++ sysvinit-2.88dsf/debian/changelog 2015-02-05 09:51:50.000000000 +0100 @@ -1,3 +1,16 @@ +sysvinit (2.88dsf-59) UNRELEASED; urgency=medium + + * service, invoke-rc.d: Avoid deadlocks during bootup and shutdown from + units/hooks which call "invoke-rc.d service reload" and similar, since the + synchronous wait plus systemd's normal behaviour of transactionally + processing all dependencies first easily causes dependency loops. Thus + during boot/shutdown operate only on the unit and not on its dependencies, + just like SysV behaves. + * Make sysvinit-utils and sysv-rc break systemd << 215 to ensure we have the + "systemctl is-system-running" command. + + -- Martin Pitt <mp...@debian.org> Thu, 05 Feb 2015 09:48:40 +0100 + sysvinit (2.88dsf-58) unstable; urgency=low * Fix typo in invoke-rc.d breaking upstart installations (Closes: diff -Nru sysvinit-2.88dsf/debian/control sysvinit-2.88dsf/debian/control --- sysvinit-2.88dsf/debian/control 2014-10-25 23:24:19.000000000 +0200 +++ sysvinit-2.88dsf/debian/control 2015-02-05 09:51:15.000000000 +0100 @@ -65,7 +65,7 @@ Replaces: last, sysvinit (<= 2.86.ds1-65) Depends: ${shlibs:Depends}, ${misc:Depends} , startpar -Breaks: upstart (<< 1.5-0ubuntu5) +Breaks: upstart (<< 1.5-0ubuntu5), systemd (<< 215) Suggests: bootlogd, sash Description: System-V-like utilities This package contains the important System-V-like utilities. @@ -85,7 +85,7 @@ sysvinit-utils (>= 2.86.ds1-62), insserv (>> 1.12.0-10) , startpar -Breaks: initscripts (<< 2.86.ds1-63) +Breaks: initscripts (<< 2.86.ds1-63), systemd (<< 215) Description: System-V-like runlevel change mechanism This package provides support for the System-V like system for booting, changing runlevels, and shutting down, diff -Nru sysvinit-2.88dsf/debian/service/service sysvinit-2.88dsf/debian/service/service --- sysvinit-2.88dsf/debian/service/service 2014-10-25 23:20:33.000000000 +0200 +++ sysvinit-2.88dsf/debian/service/service 2015-02-05 09:31:14.000000000 +0100 @@ -184,6 +184,15 @@ if [ -n "$is_systemd" ] then UNIT="${SERVICE%.sh}.service" + # avoid deadlocks during bootup and shutdown from units/hooks + # which call "invoke-rc.d service reload" and similar, since + # the synchronous wait plus systemd's normal behaviour of + # transactionally processing all dependencies first easily + # causes dependency loops + if ! systemctl is-system-running; then + sctl_args="--job-mode=ignore-dependencies" + fi + case "${ACTION}" in restart|status) exec systemctl ${ACTION} ${UNIT} @@ -195,10 +204,10 @@ # Users who need more control will use systemctl directly. for unit in $(systemctl list-unit-files --full --type=socket 2>/dev/null | sed -ne 's/\.socket\s*[a-z]*\s*$/.socket/p'); do if [ "$(systemctl -p Triggers show $unit)" = "Triggers=${UNIT}" ]; then - systemctl ${ACTION} $unit + systemctl $sctl_args ${ACTION} $unit fi done - exec systemctl ${ACTION} ${UNIT} + exec systemctl $sctl_args ${ACTION} ${UNIT} ;; reload) _canreload="$(systemctl -p CanReload show ${UNIT} 2>/dev/null)" @@ -208,7 +217,7 @@ # specific service. run_via_sysvinit else - exec systemctl reload "${UNIT}" + exec systemctl $sctl_args reload "${UNIT}" fi ;; force-stop) @@ -217,9 +226,9 @@ force-reload) _canreload="$(systemctl -p CanReload show ${UNIT} 2>/dev/null)" if [ "$_canreload" = "CanReload=no" ]; then - exec systemctl restart "${UNIT}" + exec systemctl $sctl_args restart "${UNIT}" else - exec systemctl reload "${UNIT}" + exec systemctl $sctl_args reload "${UNIT}" fi ;; *) diff -Nru sysvinit-2.88dsf/debian/src/sysv-rc/sbin/invoke-rc.d sysvinit-2.88dsf/debian/src/sysv-rc/sbin/invoke-rc.d --- sysvinit-2.88dsf/debian/src/sysv-rc/sbin/invoke-rc.d 2014-11-08 21:38:18.000000000 +0100 +++ sysvinit-2.88dsf/debian/src/sysv-rc/sbin/invoke-rc.d 2015-02-05 09:32:35.000000000 +0100 @@ -519,16 +519,24 @@ # pick up any changes. systemctl daemon-reload fi + # avoid deadlocks during bootup and shutdown from units/hooks + # which call "invoke-rc.d service reload" and similar, since + # the synchronous wait plus systemd's normal behaviour of + # transactionally processing all dependencies first easily + # causes dependency loops + if ! systemctl is-system-running; then + sctl_args="--job-mode=ignore-dependencies" + fi case $saction in start|stop|restart|status) - systemctl "${saction}" "${UNIT}" && exit 0 + systemctl $sctl_args "${saction}" "${UNIT}" && exit 0 ;; reload) _canreload="$(systemctl -p CanReload show ${UNIT} 2>/dev/null)" if [ "$_canreload" = "CanReload=no" ]; then "${INITDPREFIX}${INITSCRIPTID}" "${saction}" "$@" && exit 0 else - systemctl reload "${UNIT}" && exit 0 + systemctl $sctl_args reload "${UNIT}" && exit 0 fi ;; force-stop) @@ -537,9 +545,9 @@ force-reload) _canreload="$(systemctl -p CanReload show ${UNIT} 2>/dev/null)" if [ "$_canreload" = "CanReload=no" ]; then - systemctl restart "${UNIT}" && exit 0 + systemctl $sctl_args restart "${UNIT}" && exit 0 else - systemctl reload "${UNIT}" && exit 0 + systemctl $sctl_args reload "${UNIT}" && exit 0 fi ;; *)
signature.asc
Description: Digital signature