On Fri, Sep 11, 2015 at 09:41:34AM +0200, Martin Pitt wrote: > Josh Triplett [2015-09-10 23:54 -0700]: > > * Make networkd call if-up.d/ scripts when it brings up interfaces, to > > become compatible with ifupdown and NetworkManager for packages shipping > > hooks. (LP: #1492129) > > > > (Along with various other changes related to these hooks.) > > > > This is an *extremely* bad idea; please revert it before any package > > incorrectly starts to rely on it. And this should have been discussed > > on at least pkg-systemd-maintainers, if not systemd-devel, before being > > implemented. > > FTR, this was discussed a few months ago with Tom Gundersen (the > author of networkd) at the UOS discussion for this: > > > https://blueprints.launchpad.net/ubuntu/+spec/foundations-w-networkd-vs-ifupdown
What was his response? I'd be quite surprised to hear any upstream systemd or networkd developers favoring this approach. In any case, I'd really appreciate this being discussed more broadly, ideally on systemd-devel, to get some feedback there. The proposal there also seems somewhat different than the one implemented here. For that matter, it seems like there's a race condition here: if resolveconf is going to pull in systemd-resolved's resolv.conf, networkd hooks can't guarantee that systemd-resolved actually creates that *before* those hooks run. Perhaps resolveconf could start using inotify (along with a list of places to look for generated resolv.conf files)? > > Several reasons why this is a bad idea: > > > > - networkd is intended to bring up interfaces *quickly*, on the order of > > microseconds (not milliseconds) even with DHCP, let alone without. > > Spawning arbitrary processes, and especially shell scripts, is not and > > will never be compatible with networkd's performance requirements. > > The hooks are run asynchronously and don't block networkd. Interesting; as far as I know, that's a change compared to how other as far as I know those hooks are normally run synchronously by other software. The changelog didn't give any indication about asynchronous invocation. I'd still be concerned about whether this adds any delay to the fast-path, though a bit less so. Moving this to use networkd's existing event mechanism seems preferable to spawning a process from networkd. > > - These hooks don't exist upstream. Packages shipping if-up.d hooks are > > thus still broken anywhere other than Debian, and even *in* Debian > > they're broken with dynamic network configuration. Those package need > > fixing (upstream) to handle dynamic network configuration, and once > > they do, the Debian-specific hooks become obsolete. Allowing these > > hooks makes it less obvious that the packages themselves need fixing. > > I do agree that it would be better to fix openssh, postfix, ntpdate, > avahi-autoipd, and the zillion other packages that ship if-up.d hooks > (and other if-*.d/ hooks, but these are much less important). > However, IMHO this shouldn't be a blocker for adopting networkd -- I'd > like networkd to work in Debian as well as ifupdown and > NetworkManager; providing if* hook and resolvconf integration should > bring it up to par with NetworkManager, and thus I believe people can > now actually use this. Without these, a lot of things would suddenly > not work as expected any more, like openssh not listening to newly > brought up interfaces or postfix not flushing the mail queue when the > system gets online (in a hotplug environment). Hardly a zillion packages; here's a complete list of Debian packages: $ zgrep '^etc/network/if-up\.d' Contents-amd64.gz etc/network/if-up.d/000resolvconf net/openresolv,net/resolvconf etc/network/if-up.d/00README.linkdir net/ifupdown-scripts-zg2 etc/network/if-up.d/00check-network-cable admin/ifupdown-extra etc/network/if-up.d/10check-duplicate-ip admin/ifupdown-extra etc/network/if-up.d/20static-routes admin/ifupdown-extra etc/network/if-up.d/30check-gateway admin/ifupdown-extra etc/network/if-up.d/51guidedog net/guidedog etc/network/if-up.d/arping net/arping etc/network/if-up.d/avahi-autoipd net/avahi-autoipd etc/network/if-up.d/avahi-daemon net/avahi-daemon etc/network/if-up.d/bind9 net/bind9 etc/network/if-up.d/clamav-freshclam-ifupdown utils/clamav-freshclam etc/network/if-up.d/controlaula.ifup admin/controlaula etc/network/if-up.d/epoptes-client admin/epoptes-client etc/network/if-up.d/ethtool net/ethtool etc/network/if-up.d/gogoc net/gogoc etc/network/if-up.d/htpdate net/htpdate etc/network/if-up.d/ifenslave net/ifenslave etc/network/if-up.d/ifmetric net/ifmetric etc/network/if-up.d/ifupdown-multi admin/ifupdown-multi etc/network/if-up.d/ip misc/vlan etc/network/if-up.d/isatapd net/isatapd etc/network/if-up.d/lprng net/lprng etc/network/if-up.d/miredo net/miredo etc/network/if-up.d/mountnfs admin/initscripts etc/network/if-up.d/nslcd admin/nslcd etc/network/if-up.d/ntpdate net/ntpdate etc/network/if-up.d/openntpd net/openntpd etc/network/if-up.d/openssh-server net/openssh-server etc/network/if-up.d/openvpn net/openvpn etc/network/if-up.d/postfix mail/postfix etc/network/if-up.d/sendmail mail/sendmail-base etc/network/if-up.d/shorewall net/shorewall-init etc/network/if-up.d/slrn news/slrn etc/network/if-up.d/slrnpull news/slrnpull etc/network/if-up.d/tinc net/tinc etc/network/if-up.d/ucarp net/ucarp etc/network/if-up.d/uml-utilities otherosfs/uml-utilities etc/network/if-up.d/upstart admin/ifupdown etc/network/if-up.d/uruk net/uruk etc/network/if-up.d/vzifup-post admin/vzctl etc/network/if-up.d/wide-dhcpv6-client net/wide-dhcpv6-client etc/network/if-up.d/wpasupplicant net/wpasupplicant This seems quite manageable, especially once you rule out the ones that make no sense at all with networkd (or with anything other than ifupdown). I'd be happy to help evaluate these further. In the absence of the wireless support and desktop integration that NetworkManager has, networkd seems unlikely to get used in non-server environments anytime soon. As far as I know, wireless support is being worked on, and desktop integration will likely wait for that. I'd like to see networkd more widely used too, but not at the expense of diverging this drastically from upstream in an area where doing so creates an API and an expectation for future support of that API. If we start doing this, trying to remove it in the future will *hurt*. (As will dealing with portability between distributions.) For OpenSSH, and for any software that just re-listens when new interfaces come up, they should just listen to INADDR_ANY, rather than caring what specific interfaces are up or down. Any software that listens on INADDR_ANY shouldn't need any changes. And even with specific ListenAddress lines (or equivalent for other software), OpenSSH and similar daemons could use IP_FREEBIND (see "man 7 ip") to start listening before any interface has that address, and it would again Just Work without ever reloading the daemon. (Preempting the obvious issue: IP_FREEBIND may not work on arbitrary non-Linux systems, but then neither will networkd ;) On any system networkd can run on, dynamically reloading OpenSSH and other listening daemons shouldn't be needed.) ntpd handles dynamic network configurations just fine, as does timesyncd; ntpdate *never* handled dynamic network configurations, and I don't think it makes sense to work around that with scripts when real NTP daemons do the right thing automatically. ntpdate (and other software like tlsdate) makes sense as a standalone tool, but invoking it automatically seems less sensible. (Plus, it makes more sense to continuously keep time up to date, rather than only at boot or network change.) As far as I know, avahi-autoipd is entirely obsoleted by networkd, which assigns link-local addresses itself; see the "LinkLocalAddressing" option in "man systemd.network". > > - Network configuration can change at any time, and networkd is not > > stateful; state lives in the kernel, not in networkd. These hooks > > break that assumption. > > Can you please explain this? How is this different from ifupdown and NM? ifupdown is stateful; it remembers whether it thinks the state is up or down, independently from the hardware. (That sometimes causes prolems.) NetworkManager, similarly, has quite a bit of state remembered in the NetworkManager daemon; to the extent the hardware changes "out from under it", NetworkManager reacts by updating its state and possibly taking other actions, but fundamentally NetworkManager considers itself the canonical location of network state, rather than just something that manipulates network state. networkd intentionally *doesn't* maintain such state. In particular, if networkd has no ongoing work to do, it can exit. And on systems that need networking in the initramfs (such as systems that need the network for the root filesystem), networkd can run in the initramfs, configure those interfaces, *exit*, and then re-run after switching to the root filesystem. Finally, networkd has never had the problem that NetworkManager has of assuming it should manage every interface on the system; it manages what you tell it to.o Any daemon or other software that has /etc/network/if-*.d/* hooks will also break if someone manually brings an interface up or down, yanks out a network cable, etc. Sysadmins shouldn't have to run such scripts. Network software should just say "oh, look, stuff changed, I'll adapt". > > (This will also likely break with future > > changes to networkd and other packages integrating with it, as well as > > with other types of interfaces or virtual networks networkd can > > configure.) Among other things, as the systemd-networkd manpage > > documents, "Network configurations applied before networkd is started > > are not removed, and static configuration applied by networkd is not > > removed when networkd exits. Dynamic configuration applied by networkd > > may also optionally be left in place on shutdown. This ensures > > restarting networkd does not cut the network connection, and, in > > particular, that it is safe to transition between the initrd and the > > real root, and back." > > Right, and consequently it won't call if-post-down.d/ when you stop > networkd. And what happens if you then re-start networkd? Will it call if-up.d/* ? How confident are you that every such script in Debian is idempotent? Or if it doesn't run those scripts when re-started, then those scripts won't run at all if the network was already set up (such as in virtual environments, when networkd ran in the initramfs, or if some other software configured the network interface and handed it to networkd, which is currently the plan for how some other types of software will extend networkd). Will those scripts work if networkd brings up a bridge? Or a bond? Or a virtual network interface? Or a network with no non-link-local IP? > > - Several of the existing if-up.d and if-post-down.d hooks should not > > run under networkd. Among others: wpasupplicant's hooks shouldn't run > > at all under anything but ifupdown > > Why not? If it's not meant to run under NM or other mechanisms, it > should check $METHOD. But right now the hook doesn't do anything under > !ifupdown anyway as none of the $WPA_* and $IF_* vars is supplied. Sure it does: it runs, and checks environment variables. :) But that's a convenient way to filter out scripts from the above list that we don't need to find replacements for. > That said, as networkd doesn't have wifi support by itself, it would > certainly be *nice* to make wpa_supplicant work OOTB with networkd :-) Not that way, though; not least of which because the ordering would be backward, since the wifi link layer needs to run (and complete) *before* networkd can set up an interface. There's a plan to do that upstream; as I understand it, the plan involves using a wifi configuration daemon that will set up the link layer and then hand the interface to networkd to configure at the IP layer. Hooks aren't going to suffice there. > > , mountnfs's hooks shouldn't run (because they conflict with several > > other approaches to nfs handling that integrate properly with > > systemd) > > That sounds like a real issue indeed, do you have some details? Again, > how is that not affecting NetworkManager? See some of the comments at the top of mountnfs. mountnfs calls into various /etc/init.d/* scripts, and mounts filesystems from /etc/fstab. systemd already handles mounting filesystems, and has its own logic for making sure network filesystems don't get mounted before bringing up the network. Because of NetworkManager's target use case and system type, I doubt many people run NetworkManager on systems that have NFS filesystems listed in /etc/fstab, let alone critical ones. :) Also see bugs filed against systemd and nfs about the integration between the two and various failure modes. We don't need to throw *another* complex mechanism into the mix when people are already working on properly integrated solutions. > > , avahi-daemon's hook is responsible for numerous problems and > > slowdowns even under ifupdown > > This just disables avahi if there is a "real" .local domain. This > seems rather independent of the mechanism used to bring up networking? It can cause *long* delays bringing up and configuring interfaces (including under NetworkManager). Also see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785325 . > > and wireless-tools' hook shouldn't run under anything but > > ifupdown. > > Same question/issues as for wpasupplicant? Same answer. > > - Calling if-up.d and if-post-down.d, but not calling if-down.d or > > if-pre-up.d, may well break assumptions that a family of scripts in > > those directories have about when they'll be called and what state > > machine they'll go through. > > NetworkManager has done just that for many years. if-pre-up.d/ is > rather specific to ifupdown indeed (configuring device drivers and > stuff), and if-down.d/ has been a promise which we've never been able > to keep anyway since we have hotplugging and wifi. NetworkManager doing it was a bad idea too. :) > (FTR, I consider NM now calling if-pre-up.d/ and down.d/ not an > improvement; but that's another story) It's an argument that networkd shouldn't go down the same road, though. :) > > Packages shipping if-up.d or if-post-down.d scripts are not compatible > > with networkd. Primarily because they aren't compatible with > > dynamically changing network configurations, and secondarily because > > they tend to do the kind of really silly things that happen with > > arbitrary shell-script hooks available. This is not the right way to > > fix that problem. > > Well, we have two options: Make networkd work with Debian packages > today, or wait an infinite time until all packages listen to dynamic > network changes by themselves. For some this totally should be done > (openssh), for others it's impractical as they don't have a > permanently running daemon. There's a third option: Fix critical packages like OpenSSH (which I'd be happy to help with) and file bugs on the rest. > > What specific problem is this trying to solve? > > See above -- making use of Debian's integration logic for a lot of > packages, to be on par with ifupdown and NetworkManager. I'm asking to find out if there's some specific package or packages whose lack of such integration motivated the change, primarily to scope the problem of fixing those packages. This problem is not unbounded; I provided a complete list of Debian packages above, and the list of Ubuntu universe packages is likely similar, with the list of non-universe packages even shorter. > That said: If Michael, or a majority on debian-devel@ or the TC or > whatever disagree with this, I'm okay with dropping this from Debian > again (and just keep it in the Ubuntu package for now). I don't feel > like having a big fight over this. :-) I really don't want this to be acrimonious, and I certainly have no intention of invoking the TC. Let's talk this through and work out a solution. And I'm happy to help. What I'm asking, though, is that before any packages start *relying* on networkd invoking these scripts (which thus becomes an API with potential backward-compatibility commitments), this change be reverted (or at the very least stuck in experimental or hidden behind a disabled-by-default configuration option), and we figure out how much work it would be to fix the critical packages that ship these scripts to do the right thing on *all* systems. - Josh Triplett