Bug#727708: init system thoughts
]] Russ Allbery That, however, is also a good point. This specific case is the place where an event model does have a clear advantage. It looks like the preferred strategy in the systemd model is to teach daemons to watch for this themselves, which while certainly a good idea (most high-quality UDP daemons I know of that care about binding to specific interfaces do in fact do this), is decidedly non-trivial and hard to do portably if the daemon doesn't already support it. Freebind sockets get one out of a lot of the places where this is needed, but not all of them. I believe you can do this fairly easily. A is the service that needs to be reloaded when a network device shows up. In A's service file, have ReloadPropagatedFrom=network.target and then make your ifup@.service include an ExecStart=systemctl reload network.target. You probably want the same for ExecStop too, to handle interfaces going away. Another alternative is having multiple instance services and using BindTo=sys-subsystem-net-devices-%i.device (like ifup@.service is doing). -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
Anthony Towns a...@erisian.com.au writes: I think this would be most analogous to the complex conditions bit, where you'd say start on Y and Q so that it will only be started when event Q happens if Y has also already happened. I don't see how you'd prevent it from being manually started without a prescript checking for Y though, but I'm not entirely sure if that bothers me -- if I'm manually telling it to start a daemon, that's configured not to automatically start some thing it needs, do I care that much if I get an error from upstart or the daemon itself? That depends on whether you think you'll ever want to have a job that won't be able to detect its own prerequisites and might just do something harmful if started without its prerequisites instead of exiting with an error. But agreed, that's probably relatively rare, and upstart *does* offer a way to represent that via a pre-start shell condition. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
Tollef Fog Heen tfh...@err.no writes: I believe you can do this fairly easily. A is the service that needs to be reloaded when a network device shows up. In A's service file, have ReloadPropagatedFrom=network.target and then make your ifup@.service include an ExecStart=systemctl reload network.target. You probably want the same for ExecStop too, to handle interfaces going away. Oh, thank you! I didn't know about ReloadPropagatedFrom= and PropagatesReloadTo=. That looks quite a bit like an event. :) -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On 31 December 2013 12:55, Colin Watson cjwat...@debian.org wrote: The criticisms of Upstart's event model in the systemd position statement simply do not make sense to me. Events model how things actually happen in reality; dependencies are artificial constructions on top of them, and making them work requires the plethora of different directives in systemd (e.g. Wants, which is sort of a non-depending dependency) to avoid blocking the boot process on a single failing service. Riffing off this more than replying to it. I tend to think dependencies and events are both useful ways of describing when to start up parts of the system. In particular, it seems like: - when a network is connected, start web server - when a usb disk is connected, mount it - when a VPN is started, sync various things are best described by an event model, while: - in order to run GNOME, logind must be started - in order to run logind, dbus must be available - as part of making the system ready for a user, network-manager should be running make the most sense when described by dependencies. In particular, in many of those cases, the reverse might not be true: For debugging, I might want to start the web server manually without connecting the network; or I might want logind running without GNOME, or network-manager running without the other parts of my desktop environment. Events and dependencies aren't that different; an event essentially lets a service X say that: whenever Y happens, X happens whenever Y happens, X stops happening while a (systemd'ish) dependency says either: when X happens, Y happens as well[X Requires: Y] before X happens, Y happens as well [X Requires: Y, After: Y] after X happens, Y happens as well [X Requires: Y, Before: Y] (with Wants and Requisite and Overridable variants as well; also RequiredBy and WantedBy variants) If you look at Y, there are a few phases it could go through: no-Y Y-starts-starting Y-started Y-begins-ending no-Y If you wanted to emulate upstart events with dependencies, you'd need to do four things, I think: * create a dummy Y-started-event unit [network-is-available, usb-is-available] * invoke systemctl start Y-started-event when Y is finished starting * invoke systemctl stop Y-started-event when Y begins ending * add RequiredBy: Y-started-event and PartOf: Y-started-event to X's unit file That seems reasonably straight forward to me? If the event is something systemd already knows about, you might only need to do something equivalent to the last step. I don't think invoking systemctl start/stop is any better or worse than whatever would be needed to notify upstart of the same events. To emulate systemd dependencies in an event model (ie, X depends on Y), you'd need to do either: * change Y's job to say start on starting X * add stop on stopping Y to X's job description or * add a pre-start script to X in order to start Y first * add stop on stopping Y to X's job description The latter looks like it's the documented way of doing things. Neither of those seem particularly great -- I think that's due to upstart not letting you reverse event descriptions in the same way that systemd has Requires/RequiredBy statements. If you could say: * on starting start Y * stop on stopping Y in X's job description, by contrast, I think that would be a fine way of declaring a dependency from X to Y without leaving the event model. Not having a simple way of specifying this sort of dependency seems pretty weak on upstart's behalf. It would probably also be nice to have a way of saying when a new network comes up, reload/refresh service X -- so that it could bind to new ports or whatever even if it was already running; that seems like the sort of thing that would be easier to specify in an event model (on new-network-interface-started reload or start), than in a dependency model. Cheers, aj -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On Fri, Jan 17, 2014 at 12:50 AM, Anthony Towns a...@erisian.com.au wrote: On 31 December 2013 12:55, Colin Watson cjwat...@debian.org wrote: The criticisms of Upstart's event model in the systemd position statement simply do not make sense to me. Events model how things actually happen in reality; dependencies are artificial constructions on top of them, and making them work requires the plethora of different directives in systemd (e.g. Wants, which is sort of a non-depending dependency) to avoid blocking the boot process on a single failing service. Riffing off this more than replying to it. I tend to think dependencies and events are both useful ways of describing when to start up parts of the system. In particular, it seems like: - when a network is connected, start web server - when a usb disk is connected, mount it - when a VPN is started, sync various things are best described by an event model, while: - in order to run GNOME, logind must be started - in order to run logind, dbus must be available - as part of making the system ready for a user, network-manager should be running You could express that as an event because GNOME and logind communicate with logind and dbus (respectively) through IPC. So you can say when GNOME tries to use logind's dbus interface, start logind, or you can say when //anything// tries to use logind's dbus interface, start logind and have that done for. Same for starting dbus, just change a dbus interface to dbus's port. make the most sense when described by dependencies. In particular, in many of those cases, the reverse might not be true: For debugging, I might want to start the web server manually without connecting the network; or I might want logind running without GNOME, or network-manager running without the other parts of my desktop environment. With the above method, this problem is avoided because GNOME does not start when logind starts, it just starts whenever the runlevel is right and then logind is started automatically. So if GNOME is stopped/waiting, you can start logind without GNOME starting. -- Cameron Norman
Bug#727708: init system thoughts
Anthony Towns a...@erisian.com.au writes: To emulate systemd dependencies in an event model (ie, X depends on Y), you'd need to do either: * change Y's job to say start on starting X * add stop on stopping Y to X's job description or * add a pre-start script to X in order to start Y first * add stop on stopping Y to X's job description The latter looks like it's the documented way of doing things. But maybe not the working and/or recommend one: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=727708#3150 (the quoted parts are important). Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C »Time flies like an arrow, fruit flies like a Banana.« -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
Anthony Towns a...@erisian.com.au writes: To emulate systemd dependencies in an event model (ie, X depends on Y), you'd need to do either: * change Y's job to say start on starting X * add stop on stopping Y to X's job description or * add a pre-start script to X in order to start Y first * add stop on stopping Y to X's job description The latter looks like it's the documented way of doing things. Neither of those seem particularly great -- I think that's due to upstart not letting you reverse event descriptions in the same way that systemd has Requires/RequiredBy statements. If you could say: * on starting start Y * stop on stopping Y in X's job description, by contrast, I think that would be a fine way of declaring a dependency from X to Y without leaving the event model. Not having a simple way of specifying this sort of dependency seems pretty weak on upstart's behalf. It's worth noting that even the second solution above does not allow simulation of systemd's Requisite=, only Requires=. Now, normally Requires= (when starting X, start Y if not already started) is going to be fine, but I can certainly imagine cases where Requisite= (if Y is not started, just immediately fail startup of X) is the behavior one actually wants. I'm not sure how you would pry that behavior out of upstart's event model short of bypassing the event structure entirely and just writing an explicit check in shell in pre-start to ask whether Y is running. Which, of course, one can certainly do, but it means that information is now stored outside the dependency graph in difficult-to-parse shell and doesn't show up in analysis tools, etc. It took me a long time to wrap my mind around the objections to upstart's event model, but now that I'm starting to understand the exception cases, I keep seeing more things that feel like they should be straightforward but end up oddly convoluted when expressed in upstart's event model. It would probably also be nice to have a way of saying when a new network comes up, reload/refresh service X -- so that it could bind to new ports or whatever even if it was already running; that seems like the sort of thing that would be easier to specify in an event model (on new-network-interface-started reload or start), than in a dependency model. That, however, is also a good point. This specific case is the place where an event model does have a clear advantage. It looks like the preferred strategy in the systemd model is to teach daemons to watch for this themselves, which while certainly a good idea (most high-quality UDP daemons I know of that care about binding to specific interfaces do in fact do this), is decidedly non-trivial and hard to do portably if the daemon doesn't already support it. Freebind sockets get one out of a lot of the places where this is needed, but not all of them. In general, it would be nice to move this sort of thing out of the daemon into the init system. Given that there's already a system service that knows when things like this happen, duplicating that listening code in all the daemons isn't as appealing as just letting the system service inform the daemons when it happens. That said, I think the dependency model is a lot clearer in the common cases, and the places where the event model is superior are fairly unusual. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On 18 January 2014 17:19, Russ Allbery r...@debian.org wrote: It's worth noting that even the second solution above does not allow simulation of systemd's Requisite=, only Requires=. Now, normally Requires= (when starting X, start Y if not already started) is going to be fine, but I can certainly imagine cases where Requisite= (if Y is not started, just immediately fail startup of X) is the behavior one actually wants. I'm not sure how you would pry that behavior out of upstart's event model short of bypassing the event structure entirely and just writing an explicit check in shell in pre-start to ask whether Y is running. I think this would be most analogous to the complex conditions bit, where you'd say start on Y and Q so that it will only be started when event Q happens if Y has also already happened. I don't see how you'd prevent it from being manually started without a prescript checking for Y though, but I'm not entirely sure if that bothers me -- if I'm manually telling it to start a daemon, that's configured not to automatically start some thing it needs, do I care that much if I get an error from upstart or the daemon itself? Of course, upstart's complex conditions are documented not to work the way you'd expect, so that's not entirely a solution. I could easily imagine the complex condition support being enhanced to work to fix the behaviour described in [0] and to also block manual starts of the service prior to events happening, but that's not something that exists now afaics. And the CLA's essentially the opposite of saying patches welcome... (Personally, I think I'm convincing myself that in an ideal world I'd prefer the event model. upstart doesn't seem to implement that sufficiently though at this point. Sigh, now I'm imagining an event based init system written in haskell...) Cheers, aj [0] http://upstart.ubuntu.com/cookbook/#restarting-jobs-with-complex-conditions -- Anthony Towns a...@erisian.com.au -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
Hello, I am aware that this bug already has a lot of emails in it, but I think the issue below is important enough to warrant a *ping* to the upstart developers. It would be great if someone could comment on this. Best Nikolaus Nikolaus Rath nikol...@rath.org writes: Cameron Norman camerontnor...@gmail.com writes: On Wed, Jan 1, 2014 at 4:00 PM, Nikolaus Rath nikol...@rath.org wrote: Colin Watson cjwat...@debian.org writes: The criticisms of Upstart's event model in the systemd position statement simply do not make sense to me. Events model how things actually happen in reality; dependencies are artificial constructions on top of them, and making them work requires the plethora of different directives in systemd (e.g. Wants, which is sort of a non-depending dependency) to avoid blocking the boot process on a single failing service. Yes, there are some bugs possible in the Upstart design, but I don't agree that those are world-ending fundamental issues and I think they're generally incrementally fixable. The systemd design appears to me to expose far more complexity to the user writing unit files, and I think it's important that their mental model be as straightforward as possible. (Now, I've been working with Upstart's model for years, and it's now a pretty fundamental way of how I think of system operation; so if people who are new to *both* models rather than partisans of one side or the other consistently tell me that the systemd model is easier to grasp, then I'll listen.) For what it's worth, I consider myself new to both the upstart and the systemd model, and for me the upstart event model has been pretty much the only reason to prefer systemd over upstart (though after reading Russ' opinion, that has changed a bit). For me, upstart was actually the reason for me switching from Debian to Ubuntu for a while. I am less familiar with systemd, so it may well be that I underestimate the complexities of the systemd model. However, I am very certain in my dislike for the upstart approach. My first point of criticism has already been described by Russ, but maybe I can make it clearer by giving an example. In my opinion, the following job looks completely harmless, and should not result in an unbootable system (but at least on Ubuntu 13.10 it does, you have to reboot with init=/bin/sh and remove/fix the evilJob.conf file): $ cat evilJob.conf start on (mounted MOUNTPOINT=/var and started lightdm) stop on runleves [016] respawn script exec /bin/sleep 60 end script I believe that the equivalent systemd unit (where dependencies are specified with Requires=) works just fine. It is not clear to me how this can be incrementally fixed in upstart without fundamentally changing the design. My second point is that by treating dependencies as events, upstart does not seem to truly recognize dependencies as such and is then unable to resolve them. For example, with the following two job files (created according to the upstart cookbook, 6.32.2): $ cat jobOne.conf start on (starting jobTwo and runlevel stop on runlevel [016]) stop on runlevel [016] respawn script exec /bin/sleep 60 end script $ cat jobTwo.conf start on runlevel [2345] stop on runlevel [016] respawn script exec /bin/sleep 60 end script I can type service start jobOne, and upstart will willingly start jobOne without starting jobTwo, or doing anything about the unfulfilled dependency (not even a warning): root@ubuntu-kvm:/etc/init# service jobOne status jobOne stop/waiting root@ubuntu-kvm:/etc/init# service jobTwo status jobTwo stop/waiting root@ubuntu-kvm:/etc/init# service jobOne start jobOne start/running, process 3177 root@ubuntu-kvm:/etc/init# service jobTwo status jobTwo stop/waiting (on Ubuntu 13.10). I think you raise a lot of good points in this email, but here you are saying something which may demonstrate your (understandable) confusion about the Upstart event model. Upstart does not treat dependencies as events. Often times, Upstart //jobs// treat dependencies as events (and the ones you wrote below do), but events do not signal a dependency. Just because you said that jobOne starts with jobTwo does not mean that jobOne needs jobTwo, just that during boot up jobOne will start with jobTwo. To express a dependency, jobTwo needs to have a start on (event where I am needed). If, for example, jobOne depends on a dbus interface of jobTwo, then jobTwo could have a start on dbus ... to show that dependency. I think I understand what you're saying, thanks for the explanations! However, I can't say that this improved understanding has improved my opinion about upstart. If I understand correctly, this means that either a) every upstart job definition needs to explicitly list every possible way in which another service may depend on it (and, furthermore, every possible such dependency needs to have
Bug#727708: init system thoughts
On Thu, 02 Jan 2014, Steve Langasek wrote: our users. If we decide for systemd, what do you think is the right way to mitigate such problems for jessie? Some of these issues are only going to be seen once people start making use of systemd in anger with their existing server configs (e.g., an ec2 VM with a simple disk and network config is not going to expose these problems), and I don't really think we want to do this by way of switching the default in unstable and waiting for the bugs to roll in. Why not? I have been happily using systemd on my laptops for multiple months now and while this probably doesn't cover some of the problems you expect in complex server environment, I fail to see why unstable integration would not be the right path forward. Perhaps you're right that there is such a night and day difference between systemd and upstart that it warrants us redoing the integration work on top of systemd that has already been done on top of upstart in Ubuntu. But in that case I would still want to know that, while redoing that integration, we aren't leaving our users in the lurch. What users? It seems to me that the desktop case is not so problematic as to warrant special measure. The server case is not really concerned with unstable (most servers run stable). The real question is how to discover those potential bugs affecting servers precisely when those users are not using unstable? Cheers, -- Raphaël Hertzog ◈ Debian Developer Discover the Debian Administrator's Handbook: → http://debian-handbook.info/get/ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
Steve Langasek vor...@debian.org writes: The purpose of failsafe.conf is to ensure that services which have not been converted to the native format, but instead provide initscripts that are called upon reaching runlevel 2, are started at the right time - so that they aren't unreliable due to racing the network stack. This is an existing bug on sysvinit systems, but the race is hit much less frequently there because sysvinit is slower. Okay, thanks, that's pretty much what I'd thought. Yes, that's what in systemd one should address via network-online.target and some sort of local integration that implements whatever network is up policy that you want to enforce. Given that ifupdown is still by far the best way to manage networks on servers, and most of these init issues are most likely to happen on servers, I think we should add some sort of ifupdown integration with the network-online.target in the Debian systemd package that matches Debian's current definition of the LSB $network target. systemd's upstream is entirely correct that $network is rather underspecified from an LSB perspective, but Debian *does* have a definition, and the principle of least surprise says that we should duplicate that definition in a new init system. I assume that's what failsafe.conf is effectively doing for upstart. I am left with the concern that I seem to be the first person to ask this question, in discussions with the TC, six months after AIUI the systemd maintainers considered systemd ready to be made the default. Well, one, that's why we have these discussions. More eyes on things like this are going to find issues that we need to deal with. And your expertise in the sorts of issues Ubuntu encountered is very helpful. And, second, we're talking about problems that will happen with badly written local init scripts and are less likely to happen with packages in the archive (which are more likely to be well-written). I'm not particularly surprised that systemd early adopters don't have a lot of badly-written local init scripts that they continue to use. So I fear that switching to systemd by default is going to result in easier package maintenance for early adopters, but a much buggier experience for our users. If we decide for systemd, what do you think is the right way to mitigate such problems for jessie? Some of these issues are only going to be seen once people start making use of systemd in anger with their existing server configs (e.g., an ec2 VM with a simple disk and network config is not going to expose these problems), and I don't really think we want to do this by way of switching the default in unstable and waiting for the bugs to roll in. I think there are multiple tiers of answers to this question. Changing init systems is going to be disruptive. There's simply no way around that. It was disruptive when we switched to dependency-based boot, which also surfaced tons of problems with local init scripts and caused a lot of users to complain. It's going to be disruptive when we switch to any other new init system. That's just the nature of the beast. This is one of the reasons why I think we should support booting jessie with sysvinit. This parallels the migration path that we took for dependency-based boot. We make it clear what the new default is, but if people run into trouble, they can always fall back to sysvinit to get their stuff working again. It gives people a release cycle of leeway before they *have* to make sure their systems work with the new init system since, indeed, problems with local hacks are unlikely to start showing up until we release the new init system in stable. I therefore think we should use a very similar approach to what we did with dependency-based boot. We're already in the first stage of that: systemd is available as an option in unstable. A bunch of people are using it, and have been using it for a while, and are reporting problems. The next step will be to start pushing for broader adoption, and possibly, if we can figure out a good way to do it so that people can switch back, have dist-upgrade switch systems to systemd. (Of course, we would do this after we've hammered out the Policy work.) Then, when we release, there will obviously need to be a discussion of this in the release notes, as well as instructions on how to fall back to sysvinit, and possibly additional notes about common problems based on what we uncover from early upgrade reports. So, in other words, I do think a large component of the solution is to, indeed, switch in unstable and let the bugs roll in, which is how Debian tests everything. We can stage things somewhat more (for example, I think we should actively encourage Debian developers to switch to the new default in advance and report problems), but at the end of the day that's going to be a large part of the testing process, just like it was with dependency-based boot. Now, you are entirely correct that
Bug#727708: init system thoughts
Russ Allbery r...@debian.org writes: However, that said, I believe the integration of systemd will actually be easier in the long run because upstart is rather... weird. On that front, I also wanted to ask about: https://bugs.launchpad.net/upstart/+bug/447654 If I'm understanding this bug correctly, it's another case where upstart's dependencies work (at least currently) in rather surprising ways that you have to understand fairly well to work around. But I wasn't sure if this was just a stray left-over bug for an issue that hadn't yet been resolved, so I wanted to keep it separate from the broader argument. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
Hi Colin, Le mercredi 01 janvier 2014 à 17:17 +, Colin Watson a écrit : Basically, systemd would be more compelling to me if it tried to do less. I don't expect to persuade systemd advocates of this, as I think it amounts to different basic views of the world, but the basic approach here is probably the single biggest factor influencing my position. I'm referring to features that I don't think we'll need, not to logind et al. Could you list the features you don’t think we’ll need in the list from the wiki? * Reliable service management through cgroups * Logging features * Multi-seat * Defense-in-depth security features * Centralized service startup and monitoring * timedated/hostnamed/… * D-Bus API for service control * Transparent virtual environment support * Watchdog * Initramfs support * Introspection and debugging * Configuration-less system * User sessions I’m curious as to what features you consider not relevant for Debian. Thanks, -- .''`.Josselin Mouette : :' : `. `' `- -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On Wed, Jan 01, 2014 at 08:15:46PM +0200, Uoti Urpala wrote: On Wed, 2014-01-01 at 17:17 +, Colin Watson wrote: On Wed, Jan 01, 2014 at 05:52:03PM +0100, Ansgar Burchardt wrote: On the other hand even when using upstart as an init replacement, we'll continue to use large chunks of systemd (logind, other dbus services). I personally think less is more would only be a convincing argument if we actually would not need the aditional features. I think this counterargument, while valid, misses the major flaw in the would be more compelling if it tried to do less claim: You can simply not install any of these additional services if you don't want them. This is completely trivial to do. It is indeed technically trivial, but I invite you to review your own responses to the binfmt-support thread to see why it isn't socially trivial. To put it another way, it will help to be careful with pronouns. The you in your sentence necessarily doesn't refer to me (as the person objecting to the presence of certain features in this case), but to the Debian systemd maintainers. It's also not, in general, trivial to do something if it involves a massive argument, even if the patch would end up being a one-liner. Social costs are costs too. I'm referring to features that I don't think we'll need, not to logind et al. So far I feel that the debates about those seem to be a bit circular and go something like this: A: This feature of systemd conflicts with something else; I'd rather we didn't use it. B: You can disable that, you know. A: OK, let's disable it. B: But you shouldn't disable it because that would make Debian systemd less compatible with systemd on other distributions. A: ... Here B first correctly points out that the feature can be disabled if desired, and thus the situation cannot be worse than with upstart - you can do at least as well with systemd as you could with upstart. Then you (A) have a disagreement with B about whether disabling the feature is the _best_ way to handle the situation: B thinks that gaining compatibility with other distributions is a bigger plus, and that changing to the systemd way is an actual win over current situation, rather than just neutral like the the upstart and disabling with systemd cases. Perhaps this is the fundamental disagreement. I do not necessarily consider compatibility as an end in itself. Where Debian is already better than other distributions, we should remain better, not stick to a lowest common denominator for the sake of compatibility. I'm specifically referring here to cases where Debian already has a superior implementation of a given feature. And to answer the strawman that several people have directed my way, I obviously have no objection to enabling features in the case where they're the best implementation available. But it looks as though the default is to rank compatibility above quality of behaviour; that is certainly the impression I get from the e-mails I've received. -- Colin Watson [cjwat...@debian.org] -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On Thu, 2014-01-02 at 12:31 +, Colin Watson wrote: On Wed, Jan 01, 2014 at 08:15:46PM +0200, Uoti Urpala wrote: You can simply not install any of these additional services if you don't want them. This is completely trivial to do. It is indeed technically trivial, but I invite you to review your own It's also not, in general, trivial to do something if it involves a massive argument, even if the patch would end up being a one-liner. Social costs are costs too. So your position is essentially systemd may be a technically better init, but it would allow maintainers to choose services I do not like, so I'll try to pre-empt that by choosing upstart? The obvious alternative, and IMO correct choice, would be to decide the init on technical merits and handle choice of services separately (if needed). the _best_ way to handle the situation: B thinks that gaining compatibility with other distributions is a bigger plus, and that Perhaps this is the fundamental disagreement. I do not necessarily consider compatibility as an end in itself. Where Debian is already Maybe that's a basic disagreement in discussions concerning specific services, but certainly not for init system choice. What I view as the basic reasons I find your given rationale for choosing upstart completely unsatisfactory: * I don't think it's appropriate to use the init decision to push your views on the service choice question, when the latter could be decided independently regardless. * Your initial posting of the rationale represented it as a technical issue. Now you say it's a social one (pre-empting future arguments), but you still haven't given any real analysis to support your view. No mention of the obvious alternatives, no estimation of how common service disagreements would be, almost nothing at all. Basically you've only mentioned two discussions you didn't like; there's a gap from that to and therefore this should be resolved by choosing a different init. I also do not consider it plausible that disagreements over which services to use could be so bad that they would override all other concerns in choosing an init system. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
]] Colin Watson Perhaps this is the fundamental disagreement. I do not necessarily consider compatibility as an end in itself. Where Debian is already better than other distributions, we should remain better, not stick to a lowest common denominator for the sake of compatibility. I think it might well be the fundamental disagreement, since I believe there is value by us helping other distributions catch up and accomodating their use cases where we reasonably can. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On Mon, Dec 30, 2013 at 09:52:04PM -0800, Russ Allbery wrote: Steve Langasek vor...@debian.org writes: Upstart (as implemented in Ubuntu) restores this guarantee (with provisions for failsafe booting in the case of a wrong network config), and it takes advantage of upstart's capability of sending arbitrary signals to do so. I can see how one could implement the equivalent of upstart's static-network-up event on systemd, using generators. But what would the equivalent to /etc/init/failsafe.conf look like? I think this would be very difficult to express in systemd language, yet it's altogether vital for providing a boot that is both reliably ordered, and recoverable in the event of problems. I'm not sure I completely understand what failsafe.conf actually does, The purpose of failsafe.conf is to ensure that services which have not been converted to the native format, but instead provide initscripts that are called upon reaching runlevel 2, are started at the right time - so that they aren't unreliable due to racing the network stack. This is an existing bug on sysvinit systems, but the race is hit much less frequently there because sysvinit is slower. The failsafe.conf strikes a balance between never waiting for networks (breaking services that assume the network is up) and always waiting for all networks (breaking systems that have stale network configs in /etc/network/interfaces), by ensuring services will start as soon as all the static networks come up *if* they are present, and falling back to a reasonable timed delay if they're not. Without an equivalent to failsafe.conf, server systems converted from sysvinit to systemd will find some of their (poorly-coded, but nevertheless common and supported in Debian) services randomly failing to start on boot where they started reliably before. So, in other words, assuming that I understand this correctly, the way that you implement this in systemd is that you add a Before= dependency to the network.target (hm, which implies that my assumption about things meddling with dependencies of core services being less likely is not as correct as I thought it was, although I still think it's less likely to be done by accident) that waits for the network to be configured, but implements a timeout to ensure that you don't stall forever. From your answer and Tollef's, I'm satisfied that this requirement can be represented in a reasonable fashion on top of systemd, probably with a combination of an if-up.d hook (like for upstart) and a systemd unit that wraps a script much like /etc/init/failsafe.conf to set a timeout. I am left with the concern that I seem to be the first person to ask this question, in discussions with the TC, six months after AIUI the systemd maintainers considered systemd ready to be made the default. The kinds of race conditions that I'm highlighting here are ones that Ubuntu identified and resolved over the course of two years of experience with Upstart in the wild, at the cost of quite a bit of pain for Ubuntu's users in the meantime. I fear that switching Debian to systemd by default would inflict the same kind of pain on Debian's users, because the fixes available in Ubuntu will not translate directly to systemd and no other distribution that has adopted systemd has dealt with these issues yet. Russ, the feature gaps that you rightly point out between systemd and upstart, while they represent a significant amount of work, are all IMHO solvable for jessie. The integration work, in contrast, feels very open-ended to me; I'm not worried that the work will be insurmountable, but that we will fail to identify the issues in a timely fashion before the jessie release. I'm not saying this to deliberately engage in FUD by pointing to an unquantifiable risk; I genuinely fear that this *is* a risk, and the full extent of the risk is only becoming clear to me as a result of these discussions. Ifupdown integration was one of the very first things I addressed after adopting the upstart package in Debian, and I would never have proposed people run it on their systems without this in place. So I fear that switching to systemd by default is going to result in easier package maintenance for early adopters, but a much buggier experience for our users. If we decide for systemd, what do you think is the right way to mitigate such problems for jessie? Some of these issues are only going to be seen once people start making use of systemd in anger with their existing server configs (e.g., an ec2 VM with a simple disk and network config is not going to expose these problems), and I don't really think we want to do this by way of switching the default in unstable and waiting for the bugs to roll in. Perhaps you're right that there is such a night and day difference between systemd and upstart that it warrants us redoing the integration work on top of systemd that has already been done on top of upstart in Ubuntu. But in that
Bug#727708: init system thoughts
Hi, first, thank you for describing this problem in details. I have never met it while using systemd on Debian Wheezy and sid for 18 months. Anyhow, with your indications at hand, I realize that my use cases probably fall into the category that does not expose this problem. Steve Langasek wrote (03 Jan 2014 01:19:40 GMT) : Without an equivalent to failsafe.conf, server systems converted from sysvinit to systemd will find some of their (poorly-coded, but nevertheless common and supported in Debian) services randomly failing to start on boot where they started reliably before. Just to check that I understood you correctly: this will be the case only for services that do not provide systemd integration, right? I'm obviously not saying we should disregard these, but I think it is useful for this discussion to clearly state which packages would be affected, and which would not. The kinds of race conditions that I'm highlighting here are ones that Ubuntu identified and resolved over the course of two years of experience with Upstart in the wild, at the cost of quite a bit of pain for Ubuntu's users in the meantime. Trying to better evaluate the required effort, I am wondering to what extend Debian would benefit from the Ubuntu identified part, even if the fixes cannot be directly translated to systemd. So, I have a few questions for you. First, do you expect the list of race conditions identified and resolved in Ubuntu to cover most of those that could hit Debian if we were to use systemd? Second, if we don't want to simply wait for the bugs to roll in, as you put it, regardless of whether we go with systemd or upstart, we will need to build this list at some point (to measure progress, to start with the most critical issues, and in last resort to document remaining ones in the release notes). The earlier, the better. I wonder how spread out the information is. Does the resolution of these problems live primarily in upstart jobs (easily gathered), or in other kinds of Ubuntu-specific patches (flagged in a special way to make upstreaming easier, I would hope), or elsewhere? If the answer is meld in various kinds of Ubuntu-specific patches, how do we identify the relevant bits? It seems to me that the answer to this last set of questions is not only relevant in case we pick systemd, but also if we choose upstart: if *gathering* this list of problems and (upstart-specific) solutions is hard for Debian+systemd, then I fail to see why it would be easy for Debian+upstart. If we agree that compiling this list will be needed at some point anyway, I'm now wondering if it would perhaps be useful input for the current decision making process: not only it would help us assert the scope of the problem, but it would avoid a potential outcome I am personally scared about: after picking upstart in part because integrating it would be so much easier, realizing later that, oh well, the best we can do is often to go dig stuff from https://patches.ubuntu.com/ as we see bug reports roll in. [...] no other distribution that has adopted systemd has dealt with these issues yet. If I were among the ones who make the decision, I would want to see this statement supported by a list of bugs caused by this fact, that were reported against distributions that ship systemd by default. Anyway, I'm curious, but if I'm the only one who cares, please don't waste your time on it :) Also, I suppose that Red Hat is actively tackling these issues, with RHEL7 now in beta. It is not clear to me what amount of their fixes we can (find and) take if we pick systemd. Same here, depends how broadly the fixes are gathered, and how deeply they are meld in with changes we do not care about, I guess. If we decide for systemd, what do you think is the right way to mitigate such problems for jessie? Some of these issues are only going to be seen once people start making use of systemd in anger with their existing server configs (e.g., an ec2 VM with a simple disk and network config is not going to expose these problems), and I don't really think we want to do this by way of switching the default in unstable and waiting for the bugs to roll in. I think that designing a good plan depends quite a bit on the answer to questions above, wrt. reusing the list of race conditions identified by Ubuntu, and the corresponding list of fixes. Cheers, -- intrigeri | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
Hi, Colin Watson cjwat...@debian.org writes: Reservations with systemd - [...] Basically, systemd would be more compelling to me if it tried to do less. I don't expect to persuade systemd advocates of this, as I think it amounts to different basic views of the world, but the basic approach here is probably the single biggest factor influencing my position. On the other hand even when using upstart as an init replacement, we'll continue to use large chunks of systemd (logind, other dbus services). I personally think less is more would only be a convincing argument if we actually would not need the aditional features. I also have one question: your mail doesn't mention the integration problems with logind into a system that uses upstart and not systemd as init. Do you think this will not be an issue? Given it means ongoing work instead of a one-time investment, this is one of my main gripes with upstart. I feel that minor technical differences between the init replacements are not work committing to long-time maintaince of a systemd-logind branch that works outside of systemd. There are more interesting areas we can invest our resources into. Note that this might also include session management functions in the future. As you mentioned yourself in [1], DEs are looking into using advanced session supervision. So far both kwin and GNOME seem to target systemd for this. So this would be another area that we would need to invest resources into to maintain an upstart replacement. [1] https://lists.debian.org/debian-ctte/2014/01/msg00017.html Ansgar -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On Wed, Jan 01, 2014 at 05:52:03PM +0100, Ansgar Burchardt wrote: Colin Watson cjwat...@debian.org writes: Reservations with systemd - [...] Basically, systemd would be more compelling to me if it tried to do less. I don't expect to persuade systemd advocates of this, as I think it amounts to different basic views of the world, but the basic approach here is probably the single biggest factor influencing my position. On the other hand even when using upstart as an init replacement, we'll continue to use large chunks of systemd (logind, other dbus services). I personally think less is more would only be a convincing argument if we actually would not need the aditional features. I'm referring to features that I don't think we'll need, not to logind et al. So far I feel that the debates about those seem to be a bit circular and go something like this: A: This feature of systemd conflicts with something else; I'd rather we didn't use it. B: You can disable that, you know. A: OK, let's disable it. B: But you shouldn't disable it because that would make Debian systemd less compatible with systemd on other distributions. A: ... I also have one question: your mail doesn't mention the integration problems with logind into a system that uses upstart and not systemd as init. Do you think this will not be an issue? I think the amount of time spent arguing about it has already exceeded the total amount of effort it would ever take. That said, I rather suspect that it was a tactical error for Upstart people to choose to use the logind code from systemd, even though that involved less reimplementation of wheels. In the long term it would probably be better to write an independent implementation of the same interfaces or fork the existing code to a different source package, partly because that would help to dispose of the idea that Upstart is really dependent on systemd anyway, and partly because it would be friendlier to the Debian systemd maintainers. No doubt that could still be done. It's unfortunate that it's not one of the components listed as Reimplementable Independently, but I guess that just means more manual effort to keep track of the interface; and this would move the grunt work from the shoulders of the Debian systemd maintainers to those of the Upstart maintainers, which is probably the right place for it to lie. -- Colin Watson [cjwat...@debian.org] -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On Wed, 2014-01-01 at 17:17 +, Colin Watson wrote: On Wed, Jan 01, 2014 at 05:52:03PM +0100, Ansgar Burchardt wrote: Colin Watson cjwat...@debian.org writes: Basically, systemd would be more compelling to me if it tried to do less. I don't expect to persuade systemd advocates of this, as I think it amounts to different basic views of the world, but the basic approach here is probably the single biggest factor influencing my position. On the other hand even when using upstart as an init replacement, we'll continue to use large chunks of systemd (logind, other dbus services). I personally think less is more would only be a convincing argument if we actually would not need the aditional features. I think this counterargument, while valid, misses the major flaw in the would be more compelling if it tried to do less claim: You can simply not install any of these additional services if you don't want them. This is completely trivial to do. Using systemd as init in no way requires using them. Thus their existence cannot cause any technical problems for the use of systemd in Debian (beyond possibly needing to do the trivial disabling). If some other components that Debian does want to use start to depend on those services, such that disabling them is not easy, then this problem would exist regardless of the chosen init, and is again not a reason for favor upstart. I'm referring to features that I don't think we'll need, not to logind et al. So far I feel that the debates about those seem to be a bit circular and go something like this: A: This feature of systemd conflicts with something else; I'd rather we didn't use it. B: You can disable that, you know. A: OK, let's disable it. B: But you shouldn't disable it because that would make Debian systemd less compatible with systemd on other distributions. A: ... Here B first correctly points out that the feature can be disabled if desired, and thus the situation cannot be worse than with upstart - you can do at least as well with systemd as you could with upstart. Then you (A) have a disagreement with B about whether disabling the feature is the _best_ way to handle the situation: B thinks that gaining compatibility with other distributions is a bigger plus, and that changing to the systemd way is an actual win over current situation, rather than just neutral like the the upstart and disabling with systemd cases. There is no technical reason to prefer upstart here. Given your preferences, systemd can do at least as well (after disabling the service). Given B's preferences, systemd can do better (after enabling the service). The only benefit that choosing upstart would give you here is that it'd let you automatically win your argument with B: we already chose upstart, so enabling the systemd service is not an available choice any more. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
Colin Watson cjwat...@debian.org writes: The criticisms of Upstart's event model in the systemd position statement simply do not make sense to me. Events model how things actually happen in reality; dependencies are artificial constructions on top of them, and making them work requires the plethora of different directives in systemd (e.g. Wants, which is sort of a non-depending dependency) to avoid blocking the boot process on a single failing service. Yes, there are some bugs possible in the Upstart design, but I don't agree that those are world-ending fundamental issues and I think they're generally incrementally fixable. The systemd design appears to me to expose far more complexity to the user writing unit files, and I think it's important that their mental model be as straightforward as possible. (Now, I've been working with Upstart's model for years, and it's now a pretty fundamental way of how I think of system operation; so if people who are new to *both* models rather than partisans of one side or the other consistently tell me that the systemd model is easier to grasp, then I'll listen.) For what it's worth, I consider myself new to both the upstart and the systemd model, and for me the upstart event model has been pretty much the only reason to prefer systemd over upstart (though after reading Russ' opinion, that has changed a bit). For me, upstart was actually the reason for me switching from Debian to Ubuntu for a while. I am less familiar with systemd, so it may well be that I underestimate the complexities of the systemd model. However, I am very certain in my dislike for the upstart approach. My first point of criticism has already been described by Russ, but maybe I can make it clearer by giving an example. In my opinion, the following job looks completely harmless, and should not result in an unbootable system (but at least on Ubuntu 13.10 it does, you have to reboot with init=/bin/sh and remove/fix the evilJob.conf file): $ cat evilJob.conf start on (mounted MOUNTPOINT=/var and started lightdm) stop on runleves [016] respawn script exec /bin/sleep 60 end script I believe that the equivalent systemd unit (where dependencies are specified with Requires=) works just fine. It is not clear to me how this can be incrementally fixed in upstart without fundamentally changing the design. My second point is that by treating dependencies as events, upstart does not seem to truly recognize dependencies as such and is then unable to resolve them. For example, with the following two job files (created according to the upstart cookbook, 6.32.2): $ cat jobOne.conf start on (starting jobTwo and runlevel stop on runlevel [016]) stop on runlevel [016] respawn script exec /bin/sleep 60 end script $ cat jobTwo.conf start on runlevel [2345] stop on runlevel [016] respawn script exec /bin/sleep 60 end script I can type service start jobOne, and upstart will willingly start jobOne without starting jobTwo, or doing anything about the unfulfilled dependency (not even a warning): root@ubuntu-kvm:/etc/init# service jobOne status jobOne stop/waiting root@ubuntu-kvm:/etc/init# service jobTwo status jobTwo stop/waiting root@ubuntu-kvm:/etc/init# service jobOne start jobOne start/running, process 3177 root@ubuntu-kvm:/etc/init# service jobTwo status jobTwo stop/waiting (on Ubuntu 13.10). As I understand, a systemd unit with Requires=jobTwo will not start without jobTwo running. I guess this could be fixed by hardcoding a special treatment of the start on starting event, but that would effectively be equivalent to introducing a new kind of depends on stanza, and thus acknowledge that the everything is an event model doesn't quite work. Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C »Time flies like an arrow, fruit flies like a Banana.« -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On Wed, Jan 1, 2014 at 4:00 PM, Nikolaus Rath nikol...@rath.org wrote: Colin Watson cjwat...@debian.org writes: The criticisms of Upstart's event model in the systemd position statement simply do not make sense to me. Events model how things actually happen in reality; dependencies are artificial constructions on top of them, and making them work requires the plethora of different directives in systemd (e.g. Wants, which is sort of a non-depending dependency) to avoid blocking the boot process on a single failing service. Yes, there are some bugs possible in the Upstart design, but I don't agree that those are world-ending fundamental issues and I think they're generally incrementally fixable. The systemd design appears to me to expose far more complexity to the user writing unit files, and I think it's important that their mental model be as straightforward as possible. (Now, I've been working with Upstart's model for years, and it's now a pretty fundamental way of how I think of system operation; so if people who are new to *both* models rather than partisans of one side or the other consistently tell me that the systemd model is easier to grasp, then I'll listen.) For what it's worth, I consider myself new to both the upstart and the systemd model, and for me the upstart event model has been pretty much the only reason to prefer systemd over upstart (though after reading Russ' opinion, that has changed a bit). For me, upstart was actually the reason for me switching from Debian to Ubuntu for a while. I am less familiar with systemd, so it may well be that I underestimate the complexities of the systemd model. However, I am very certain in my dislike for the upstart approach. My first point of criticism has already been described by Russ, but maybe I can make it clearer by giving an example. In my opinion, the following job looks completely harmless, and should not result in an unbootable system (but at least on Ubuntu 13.10 it does, you have to reboot with init=/bin/sh and remove/fix the evilJob.conf file): $ cat evilJob.conf start on (mounted MOUNTPOINT=/var and started lightdm) stop on runleves [016] respawn script exec /bin/sleep 60 end script I believe that the equivalent systemd unit (where dependencies are specified with Requires=) works just fine. It is not clear to me how this can be incrementally fixed in upstart without fundamentally changing the design. My second point is that by treating dependencies as events, upstart does not seem to truly recognize dependencies as such and is then unable to resolve them. For example, with the following two job files (created according to the upstart cookbook, 6.32.2): I think you raise a lot of good points in this email, but here you are saying something which may demonstrate your (understandable) confusion about the Upstart event model. Upstart does not treat dependencies as events. Often times, Upstart //jobs// treat dependencies as events (and the ones you wrote below do), but events do not signal a dependency. Just because you said that jobOne starts with jobTwo does not mean that jobOne needs jobTwo, just that during boot up jobOne will start with jobTwo. To express a dependency, jobTwo needs to have a start on (event where I am needed). If, for example, jobOne depends on a dbus interface of jobTwo, then jobTwo could have a start on dbus ... to show that dependency. Basically, to express dependencies in Upstart you express all the ways a job is needed inside of that job, not inside others. That said, Upstart developers and Ubuntu developers do not use Upstart this way and write jobs like you did below, so an Upstart system will most likely not act like I explained above (however this is not an inherent flaw in Upstart). I would also like to note that launchd acts the same exact way as I have described and had similar limitations. Furthermore, SystemStarter (Apple's previous init) acted in a very similar way to systemd: The hardest part to manage during a launchd boot is dependencies. SystemStarter had a very simple system of dependencies that used the Uses, Requires, and Provides keys in the plist of a startup item. There are two main strategies when creating launch dependencies on [OS X] . Using IPC will allow the daemons to talk amongst themselves to work it out, or you can watch files or paths for changes. Using IPC is much more subtle than the SystemStarter's keys and requires more work from the developer, but it may* https://en.wikipedia.org/wiki/Wikipedia:Citation_needed*lead to cleaner and quicker startups. -- https://en.wikipedia.org/wiki/Launchd#launchd I still think your points are relevant, however, because Upstart in Ubuntu acts in the same way as the below jobs, and Debian will most likely inherit that behavior if it uses Upstart. Happy New Year, Cameron Norman $ cat jobOne.conf start on (starting jobTwo and runlevel stop on runlevel [016]) stop
Bug#727708: init system thoughts
On 01/01/2014 04:00 PM, Nikolaus Rath wrote: My second point is that by treating dependencies as events, upstart does not seem to truly recognize dependencies as such and is then unable to resolve them. For example, with the following two job files (created according to the upstart cookbook, 6.32.2): $ cat jobOne.conf start on (starting jobTwo and runlevel stop on runlevel [016]) stop on runlevel [016] respawn script exec /bin/sleep 60 end script Wops, something went wrong with copy and paste here. This should be: $ cat jobOne.conf start on (starting jobTwo and runlevel [2345]) stop on runlevel [016] respawn script exec /bin/sleep 60 end script Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C »Time flies like an arrow, fruit flies like a Banana.« -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
Cameron Norman camerontnor...@gmail.com writes: On Wed, Jan 1, 2014 at 4:00 PM, Nikolaus Rath nikol...@rath.org wrote: Colin Watson cjwat...@debian.org writes: The criticisms of Upstart's event model in the systemd position statement simply do not make sense to me. Events model how things actually happen in reality; dependencies are artificial constructions on top of them, and making them work requires the plethora of different directives in systemd (e.g. Wants, which is sort of a non-depending dependency) to avoid blocking the boot process on a single failing service. Yes, there are some bugs possible in the Upstart design, but I don't agree that those are world-ending fundamental issues and I think they're generally incrementally fixable. The systemd design appears to me to expose far more complexity to the user writing unit files, and I think it's important that their mental model be as straightforward as possible. (Now, I've been working with Upstart's model for years, and it's now a pretty fundamental way of how I think of system operation; so if people who are new to *both* models rather than partisans of one side or the other consistently tell me that the systemd model is easier to grasp, then I'll listen.) For what it's worth, I consider myself new to both the upstart and the systemd model, and for me the upstart event model has been pretty much the only reason to prefer systemd over upstart (though after reading Russ' opinion, that has changed a bit). For me, upstart was actually the reason for me switching from Debian to Ubuntu for a while. I am less familiar with systemd, so it may well be that I underestimate the complexities of the systemd model. However, I am very certain in my dislike for the upstart approach. My first point of criticism has already been described by Russ, but maybe I can make it clearer by giving an example. In my opinion, the following job looks completely harmless, and should not result in an unbootable system (but at least on Ubuntu 13.10 it does, you have to reboot with init=/bin/sh and remove/fix the evilJob.conf file): $ cat evilJob.conf start on (mounted MOUNTPOINT=/var and started lightdm) stop on runleves [016] respawn script exec /bin/sleep 60 end script I believe that the equivalent systemd unit (where dependencies are specified with Requires=) works just fine. It is not clear to me how this can be incrementally fixed in upstart without fundamentally changing the design. My second point is that by treating dependencies as events, upstart does not seem to truly recognize dependencies as such and is then unable to resolve them. For example, with the following two job files (created according to the upstart cookbook, 6.32.2): I think you raise a lot of good points in this email, but here you are saying something which may demonstrate your (understandable) confusion about the Upstart event model. Upstart does not treat dependencies as events. Often times, Upstart //jobs// treat dependencies as events (and the ones you wrote below do), but events do not signal a dependency. Just because you said that jobOne starts with jobTwo does not mean that jobOne needs jobTwo, just that during boot up jobOne will start with jobTwo. To express a dependency, jobTwo needs to have a start on (event where I am needed). If, for example, jobOne depends on a dbus interface of jobTwo, then jobTwo could have a start on dbus ... to show that dependency. I think I understand what you're saying, thanks for the explanations! However, I can't say that this improved understanding has improved my opinion about upstart. If I understand correctly, this means that either a) every upstart job definition needs to explicitly list every possible way in which another service may depend on it (and, furthermore, every possible such dependency needs to have upstart integration so that it can actually be used as an event) or b) a package providing jobOne that depends on jobTwo from another package needs to patch the *other* package's configuration to add the dependency information to jobTwo's definition. Neither of this sounds appealing to me at all, especially compared to systemd, where all you have to do is drop a Requires= line into jobOne's configuration. Basically, to express dependencies in Upstart you express all the ways a job is needed inside of that job, not inside others. That said, Upstart developers and Ubuntu developers do not use Upstart this way and write jobs like you did below, so an Upstart system will most likely not act like I explained above (however this is not an inherent flaw in Upstart). Well, so what is the proper way to encode a dependency in an upstart job for Ubuntu (and presumable Debian) then? Apparently the proper way is extremly tedious and not used by anyone, and the other way isn't able to satisfy dependencies. Also, I would think that your
Bug#727708: init system thoughts
On Wed, Jan 1, 2014 at 5:09 PM, Nikolaus Rath nikol...@rath.org wrote: Cameron Norman camerontnor...@gmail.com writes: On Wed, Jan 1, 2014 at 4:00 PM, Nikolaus Rath nikol...@rath.org wrote: Colin Watson cjwat...@debian.org writes: The criticisms of Upstart's event model in the systemd position statement simply do not make sense to me. Events model how things actually happen in reality; dependencies are artificial constructions on top of them, and making them work requires the plethora of different directives in systemd (e.g. Wants, which is sort of a non-depending dependency) to avoid blocking the boot process on a single failing service. Yes, there are some bugs possible in the Upstart design, but I don't agree that those are world-ending fundamental issues and I think they're generally incrementally fixable. The systemd design appears to me to expose far more complexity to the user writing unit files, and I think it's important that their mental model be as straightforward as possible. (Now, I've been working with Upstart's model for years, and it's now a pretty fundamental way of how I think of system operation; so if people who are new to *both* models rather than partisans of one side or the other consistently tell me that the systemd model is easier to grasp, then I'll listen.) For what it's worth, I consider myself new to both the upstart and the systemd model, and for me the upstart event model has been pretty much the only reason to prefer systemd over upstart (though after reading Russ' opinion, that has changed a bit). For me, upstart was actually the reason for me switching from Debian to Ubuntu for a while. I am less familiar with systemd, so it may well be that I underestimate the complexities of the systemd model. However, I am very certain in my dislike for the upstart approach. My first point of criticism has already been described by Russ, but maybe I can make it clearer by giving an example. In my opinion, the following job looks completely harmless, and should not result in an unbootable system (but at least on Ubuntu 13.10 it does, you have to reboot with init=/bin/sh and remove/fix the evilJob.conf file): $ cat evilJob.conf start on (mounted MOUNTPOINT=/var and started lightdm) stop on runleves [016] respawn script exec /bin/sleep 60 end script I believe that the equivalent systemd unit (where dependencies are specified with Requires=) works just fine. It is not clear to me how this can be incrementally fixed in upstart without fundamentally changing the design. My second point is that by treating dependencies as events, upstart does not seem to truly recognize dependencies as such and is then unable to resolve them. For example, with the following two job files (created according to the upstart cookbook, 6.32.2): I think you raise a lot of good points in this email, but here you are saying something which may demonstrate your (understandable) confusion about the Upstart event model. Upstart does not treat dependencies as events. Often times, Upstart //jobs// treat dependencies as events (and the ones you wrote below do), but events do not signal a dependency. Just because you said that jobOne starts with jobTwo does not mean that jobOne needs jobTwo, just that during boot up jobOne will start with jobTwo. To express a dependency, jobTwo needs to have a start on (event where I am needed). If, for example, jobOne depends on a dbus interface of jobTwo, then jobTwo could have a start on dbus ... to show that dependency. I think I understand what you're saying, thanks for the explanations! However, I can't say that this improved understanding has improved my opinion about upstart. If I understand correctly, this means that either a) every upstart job definition needs to explicitly list every possible way in which another service may depend on it (and, furthermore, every possible such dependency needs to have upstart integration so that it can actually be used as an event) Pretty much all IPC mechanisms have integration into Upstart, so a lot of dependencies are covered. or b) a package providing jobOne that depends on jobTwo from another package needs to patch the *other* package's configuration to add the dependency information to jobTwo's definition. Yes. The patch would, however, be limited to a or (...) in the start on section. So it is not like the patch is going to change a ton. Neither of this sounds appealing to me at all, especially compared to systemd, where all you have to do is drop a Requires= line into jobOne's configuration. I agree, especially when one can have a Requires=*.socket or Requires=*.bus and get the same benefits. Basically, to express dependencies in Upstart you express all the ways a job is needed inside of that job, not inside others. That
Bug#727708: init system thoughts
Cameron Norman camerontnor...@gmail.com writes: I think you raise a lot of good points in this email, but here you are saying something which may demonstrate your (understandable) confusion about the Upstart event model. Upstart does not treat dependencies as events. Often times, Upstart //jobs// treat dependencies as events (and the ones you wrote below do), but events do not signal a dependency. Just because you said that jobOne starts with jobTwo does not mean that jobOne needs jobTwo, just that during boot up jobOne will start with jobTwo. To express a dependency, jobTwo needs to have a start on (event where I am needed). If, for example, jobOne depends on a dbus interface of jobTwo, then jobTwo could have a start on dbus ... to show that dependency. I think I understand what you're saying, thanks for the explanations! However, I can't say that this improved understanding has improved my opinion about upstart. If I understand correctly, this means that either [...] or b) a package providing jobOne that depends on jobTwo from another package needs to patch the *other* package's configuration to add the dependency information to jobTwo's definition. Yes. The patch would, however, be limited to a or (...) in the start on section. So it is not like the patch is going to change a ton. No it's not a difficult change, but you'd be patching a *different packages* configuration file. I am not a dpkg expert, but I'm pretty sure this is not something a maintainer script should do. Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C »Time flies like an arrow, fruit flies like a Banana.« -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
]] Steve Langasek In any case, systemd does indeed support this case; simply make your service depend on network-online.target, which will block for a reasonable amount of time to see if a network interface comes online, and eventually time out if that doesn't occur. This will actually work even if your primary network is a wireless network managed by NetworkManager, and even if that network doesn't actually work until the user has logged in, assuming your service is not actually in the dependency path of the user logging in. And what makes this work in the case where you *aren't* using NetworkManager? I see no integration with ifupdown in the systemd package. There is none, and it would not be ifupdown-specific. We could easily enough add a «wait for a default ipv4 and ipv6 default route to appear» unit if somebody desired that, which would be pulled in by network-online.target. It's a pretty trivial piece of code. Alternatively, just put systemctl start network-online.target into a fragment in if-up.d if you consider ifup considering a network interface up to be enough. (I don't, but as pointed out on the systemd wiki page referenced, people have different ideas about what «network online» means.) -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On Mon, Dec 30, 2013 at 6:55 PM, Colin Watson cjwat...@debian.org wrote: The criticisms of Upstart's event model in the systemd position statement simply do not make sense to me. Events model how things actually happen in reality; dependencies are artificial constructions on top of them, and making them work requires the plethora of different directives in systemd (e.g. Wants, which is sort of a non-depending dependency) to avoid blocking the boot process on a single failing service. Yes, there are some bugs possible in the Upstart design, but I don't agree that those are world-ending fundamental issues and I think they're generally incrementally fixable. The systemd design appears to me to expose far more complexity to the user writing unit files, and I think it's important that their mental model be as straightforward as possible. (Now, I've been working with Upstart's model for years, and it's now a pretty fundamental way of how I think of system operation; so if people who are new to *both* models rather than partisans of one side or the other consistently tell me that the systemd model is easier to grasp, then I'll listen.) I would like to give my view of the event vs. dependency model. I will declare my conclusion up front: Upstart's event model is, in my opinion, more flexible, and much more compatible with socket activation than systemd's dependency model (which is ironic, since Upstart does not stress socket activation, and systemd does). I will start with an example including syslog, dbus, and NetworkManager. One way a distribution developer would write these job files is by saying start on syslog-started and stop on syslog-stopped for dbus. Although this may seem like the obvious way to write the job, it is not the most efficient. This is what I will call an always on job. Whenever it is possible for this job to be on (its dependencies have started, and the job is enabled) it will be on. This can cause slow boot, because a ton of jobs are being started that do not need to be. This is the fault of //the distribution developer//, not Upstart itself. Now, lets imagine this developer stopped for a second to think before denouncing Upstart as inefficient crap. He knows that his dbus job was probably not efficient, and he wants to try to make a more efficient NetworkManager job. So, the developer crafts a start on dbus-started and /* dbus signal received */ stop on dbus-stopped or /* dbus signal not received */ So this is cool. NetworkManager is started not simply when it is able to start, but also when it needs to start. It stops when dbus stops (its dependencies are unavailable) or when it is not needed by anyone. What is great about Upstart's model is its flexibility. Example: start on dbus-started and /* dbus address accessed */ stop on dbus-stopped This starts NetworkManager when its services are required, but then keeps it running even after they are not, until it can no longer provide its services. This may be desirable in some situations (maybe starting and stopping nm a lot is unwanted), and shows how this event model puts more control in the job, rather than a config file. Now lets say that developer realizes how powerful the event model is, and goes back to reimplement his dbus job. He/she wants dbus to be running only when its services are needed. start on syslog-started and /* socket event aimed at dbus */ stop on syslog-started or /* no socket events toward dbus */ Now, this changes things around, and the NetworkManager job needs to modified to not wait for dbus to start, but just access the socket and let Upstart automatically start dbus following that event. start on dbus SIGNAL= stop on dbus-stopped I think this usage of Upstart's event model is incredibly superb, and much more understandable and usable than systemd's socket and bus activation model. Although systemd's declarative syntax is simple, I think it offers too little flexibility and does not reflect the realities of the system to the unit, which makes the declarative syntax an abstraction that needs to be understood by the admin, or it will be misused. Furthermore, with a little work put into Upstart's socket event bridge and socket handling (which should not be a problem since the infrastructure is already there), the boot time speed ups and socket based activation model of systemd can be achieved with only a little effort, and considerably more flexibility. Good night, Cameron Norman
Bug#727708: init system thoughts
Steve Langasek vor...@debian.org writes: Upstart (as implemented in Ubuntu) restores this guarantee (with provisions for failsafe booting in the case of a wrong network config), and it takes advantage of upstart's capability of sending arbitrary signals to do so. I can see how one could implement the equivalent of upstart's static-network-up event on systemd, using generators. But what would the equivalent to /etc/init/failsafe.conf look like? I think this would be very difficult to express in systemd language, yet it's altogether vital for providing a boot that is both reliably ordered, and recoverable in the event of problems. I'm not sure I completely understand what failsafe.conf actually does, but I think the systemd answer to this is partly discussed here: http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ A systemd integration in Debian may want to implement something akin to upstart's if-up.d hook as a service that waits for ifup notification with a timeout and then signals network.service to duplicate that upstart functionality. (I'm not sure I completely understand all of the issues involved and whether that would always make sense.) When using NetworkManager, as mentioned in that discussion, you may want to enable NetworkManager-wait-online.service instead or in addition. That has a failsafe to not wait too long for network before continuing the boot, including network-dependent services, anyway. So, in other words, assuming that I understand this correctly, the way that you implement this in systemd is that you add a Before= dependency to the network.target (hm, which implies that my assumption about things meddling with dependencies of core services being less likely is not as correct as I thought it was, although I still think it's less likely to be done by accident) that waits for the network to be configured, but implements a timeout to ensure that you don't stall forever. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
Oh, sorry, I forgot to respond to this part. Steve Langasek vor...@debian.org writes: Of course if we were writing all our services according to best practices, we wouldn't have to worry about this, as the service would just handle this gracefully (or maybe hand the complexity off to the init system for socket-based activation - but then what does init do with a request for a socket address that's not available? This is what IP_FREEBIND is for, which is why it needs to be supported by the socket activation configuration. It's been considered best practice for some time for IPv6 services binding to particular configured IP addresses to use IP_FREEBIND because IPv6 network setup can take an unpredictable amount of time. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On Mon, Dec 30, 2013 at 10:04:09PM -0800, Russ Allbery wrote: Oh, sorry, I forgot to respond to this part. Steve Langasek vor...@debian.org writes: Of course if we were writing all our services according to best practices, we wouldn't have to worry about this, as the service would just handle this gracefully (or maybe hand the complexity off to the init system for socket-based activation - but then what does init do with a request for a socket address that's not available? This is what IP_FREEBIND is for, which is why it needs to be supported by the socket activation configuration. It's been considered best practice for some time for IPv6 services binding to particular configured IP addresses to use IP_FREEBIND because IPv6 network setup can take an unpredictable amount of time. Ah, thanks for the pointer. I saw your previous mention of IP_FREEBIND but hadn't looked it up yet - that certainly does sound like an important feature to take advantage of in socket activation. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Bug#727708: init system thoughts
I see that the LWN commentariat already has my decision selected in advance, so I might as well write up some more detailed thoughts before any other words are put into my mouth! Choice of default - Firstly, I've tried to keep my employer affiliation out of this as much as possible, and I have come under no pressure from that quarter. I'll reiterate my previous comments on debian-devel: http://lists.debian.org/debian-devel/2013/10/msg00747.html http://lists.debian.org/debian-devel/2013/10/msg00777.html http://lists.debian.org/debian-devel/2013/10/msg00818.html It's apparently no surprise that my preference for a default lies with Upstart. To the extent that I have a pre-existing bias it has more to do with the fact that I have substantially more real-world deployment experience with Upstart in an environment structurally very similar to Debian (Ubuntu, obviously), which has led me to believe that it would be a fine choice for deployment in Debian which could be put in place with a minimum of disruption, and which would close as few doors as possible to experimentation and innovation. That said, I will add that I have been impressed with the technical aspects of systemd that I have seen while working on adding support for it to my packages by way of research for this debate. Its documentation is comprehensive, a great deal of work has clearly gone into the range of options available in unit files, and it has admirable facilities available for system administrators. I am quite happy for it to be my second choice (which is by no means irrelevant given our voting protocol); I think it would be a more than worthwhile step forward from sysvinit, just in a somewhat different direction that does not appeal quite as much to me. Reservations with systemd - My main concerns with systemd relate to its broad scope regarding units it provides for system initialisation tasks currently performed by other packages, and the potential for that to interfere with past and future work elsewhere in Debian. I can understand why the systemd authors felt it valuable to expand its scope in this way, to provide a more consistent experience across the board. Nevertheless, it seems to significantly complicate the job of integrating it with some excellent features that already exist in Debian and which are superior to those currently in most other distributions. I realise that the Debian systemd maintainers have generally expressed willingness to deal with this kind of integration problem where appropriate, but I find the impedance mismatch to be much higher than it is with Upstart which leaves these tasks up to the distribution. The two examples which I've run across so far, both ones I was inclined to look at since I'm familiar with the territories, are: #716812 (binfmt-support) (I'm the author and maintainer of binfmt-support.) The discussion on this (archived earlier in #727708) does now seem to have been resolved reasonably amicably so far; I've turned binfmt-support into an independent upstream package hosted on nongnu.org, given it a systemd unit file while I was there, and will work on getting that into more distributions. However, I maintain that this is really something that has no particular reason to be integrated with the init system, that it has much more interesting interactions with packaging than it does with other system events, and that it is busywork to convert things away from a system that works measurably better in Debian and derivatives than (TTBOMK) any other GNU/Linux distribution. #608457 (console-setup) (I've contributed to console-setup in some small ways, although I'm not its primary maintainer by any stretch of the imagination.) console-setup is a fantastic piece of innovation in Debian; instead of shipping two entirely independent sets of keymap files for virtual consoles and for X, the former of which is mostly unloved, let's generate virtual console mappings dynamically from XKB! It's naturally not an easy task since the console is less capable in various important ways, but it's demonstrably possible to do a pretty accurate job. Like binfmt-support, it probably ought to be turned into an independent upstream package to make it easier for other distributions to adopt, but in the meantime it's tremendously useful for Debian. This seems a bit conceptually tricky to integrate with systemd, because localectl(1) exposes the console-data style of keymap naming in its interface, and with console-setup you only use the X style. The Debian systemd maintainers haven't yet offered a suggestion in the bug, but perhaps they'll come up with something appropriate (maybe just disabling set-keymap/list-keymaps, or converting both ways and accepting the inevitable round-trip errors). But my point is that this is a
Bug#727708: init system thoughts
Thanks for this write-up, Colin. This was very interesting to me, particularly in the concrete examples of where you've felt like systemd has stepped into areas that will pose integration problems for us. This is something that I've seen referred to in the abstract frequently, but without a lot of specifics that made sense to me. Both of those examples make sense to me. Just a couple of specific comments Colin Watson cjwat...@debian.org writes: (Now, I've been working with Upstart's model for years, and it's now a pretty fundamental way of how I think of system operation; so if people who are new to *both* models rather than partisans of one side or the other consistently tell me that the systemd model is easier to grasp, then I'll listen.) I am new to both models. I'm not very fond of the upstart model. Lennart had a comment on this point that I thought phrased the problem in an interesting way: both systemd and upstart deal with the same data, but with systemd, the full dependency tree is precalculated and exists as state within the init system. With upstart, behavior is dynamic and to some degree emergent: you know who is listening to what signals, but you can arbitrarily introduce signals, and you're dealing with a message bus rather than a dependency tree, so you're reacting to things as they happen on the bus. In systemd, you can dump everything that can possibly happen and work through the state transitions by hand if needed, with the possible exception of unexpected device events (which are horribly important in some cases -- don't get me wrong -- but which are uninteresting in a lot of common debugging scenarios). I think it's somewhat harder to do that with upstart, where events are generated more dynamically. Basically, the way I'd put it is that I think the upstart event model is the sort of elegant, minimal model that someone who has thought really hard about the space comes up with, but which is hard to understand for people who have *not* thought really hard about the space. It's elegant and clever; for me personally, I find it *too* clever. I understand what it's trying to do, and it has a certain grace, but I find a dependency graph more robust and predictable, and even the levels of dependencies make sense to me given packaging background and familiarity with the Depends vs. Recommends scenarios. I can build a relatively complete mental map more easily than I can with the message bus approach. I should probably note here that I'm a notorious skeptic about message buses in general. People I work with have probably gotten rather tired of me going on about my feeling that message bus integrations are overly complex to reason about, hard to guarantee correctness of, and are overused in system design. :) There is of course also the non-Linux porting issue, which Ian, Russ, and Steve have discussed at more length elsewhere, and I won't repeat it at length. I will just add in response to Russ that there is a great difference between one project whose lead maintainer has explicitly said that he will reject patches for porting to non-Linux architectures because they fundamentally do not make sense with its architecture, and another project one of whose developers has been working on such a port with some degree of success so far. This is a very valid point, and you're right, I probably understated that in my writeup. One of the things that I may turn out to be wrong about (and would be happy to be wrong about, for the record) is the likelihood of completing a working port of upstart to Hurd and kFreeBSD. As I mentioned but probably not explicitly enough, the existence of those ports would, for me, turn the whole second part of my analysis into a dead heat between systemd and upstart as opposed to a general advantage for upstart in terms of ecosystem integration. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
Colin Watson wrote: (Now, I've been working with Upstart's model for years, and it's now a pretty fundamental way of how I think of system operation; so if people who are new to *both* models rather than partisans of one side or the other consistently tell me that the systemd model is easier to grasp, then I'll listen.) I'd like to respond to this point, since I have a clear memory of dealing with both upstart's model and systemd's model for the first time. I also work professionally with a system (ChromeOS) that uses upstart, and I've dealt with systemd quite a bit as well. For the sake of full disclosure, my position is that I'd prefer systemd over upstart; I'd be OK with upstart if and only if systemd remained a viable alternative (in other words, packages maintaining both upstart jobs and systemd units but not necessarily maintaining init scripts seems pretty reasonable). So, at this point I'm squarely in the systemd camp, but not for any particular reasons of partisanship; I simply find the technology and model better. I started out dealing with the sysvinit model, pre-startpar; I dealt quite frequently with the magic ordering numbers (S20, K80, etc). My first exposure to dependencies was through startpar and LSB, and they made sense, though I was never a fan of the magic 'S' runlevel. I fairly quickly saw that mapping those dependencies to script ordering numbers was a hack, albeit a nicely done hack for compatibility; using the dependencies natively made a lot more sense, though. I read about upstart when it was first announced, and I dove into the event model quite extensively, thinking that it might help improve things. I never found it particularly intuitive, and in particular the whole start on starting, start on stopping etc model never really made much sense to me; it didn't seem like a natural mapping of how the system worked. I also, from the beginning, disliked the model of starting everything that was possible to start, though I did not at the time see what the alternative would be; it simply never really seemed like the right model for booting quickly. (This was reinforced when I started paying attention to boot time, inspired by the original boot in 5 seconds presentation; upstart's model seemed entirely unsuited for that exercise.) I also found the requirement to know how many times your daemon forked quite obnoxious (even more so the failure mode if you get it wrong on even one daemon). These days, dealing with upstart professionally, it still seems entirely too easy to get upstart confused and in a difficult to recover state via a single incorrect upstart job, and I find it one of the most annoying subsystems to deal with; there's a general feeling of oh, joy, that's probably an upstart issue every time any issue relating to booting comes up. By contrast, my initial exposure to systemd (via the systemd for system administrators blog series and the systemd documentation) was actually via socket activation, with the dependency mechanism coming later. The socket activation model, as a replacement for dependencies, was one of those rare ideas that seemed immediately obvious in hindsight. My introduction to systemd's dependencies was more along the lines of and explicit dependencies exist for when you can't fix your daemon and unit file to do things the *right* way. Similarly, the use of autofs and other techniques to allow starting things in parallel even when dependencies exist made immediate sense. And the entire model, including its handling of asynchronous events (notably the integration with udev), simply felt like a closer fit to how the kernel works and how reality works. Finally, the concept of factoring out common elements from daemons seemed quite nice, having dealt with quite a bit of ugly boilerplate code. In both cases, I had no particular reason to like or dislike either model; in each case I very much *wanted* them to work out, since my exposure to sysvinit versus startpar gave me a very early reason to want something better than sysvinit. I had no other reason to prefer one model over the other. I'm not attached to either init system, nor do I have any particular bias or reason to favor one over the other for any reason other than superior technology or a model I find clearer and more natural. So, in summary, upon initial introduction, I found systemd's model far more intuitive, both innately (in its use of socket activation and similar mechanisms to *avoid* dependencies) and as a more natural mapping to how the system, kernel, and hardware work. I hope this writeup of my first impressions of both systems proves useful in some way. - Josh Triplett -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On Tue, 2013-12-31 at 02:55 +, Colin Watson wrote: My main concerns with systemd relate to its broad scope regarding units it provides for system initialisation tasks currently performed by other packages, and the potential for that to interfere with past and future work elsewhere in Debian. I can understand why the systemd authors felt The two examples which I've run across so far, both ones I was inclined to look at since I'm familiar with the territories, are: #716812 (binfmt-support) #608457 (console-setup) Basically, systemd would be more compelling to me if it tried to do less. I don't expect to persuade systemd advocates of this, as I think it amounts to different basic views of the world, but the basic approach here is probably the single biggest factor influencing my position. I think this is absolutely ridiculous as a rationale for choosing upstart. If you have done any investigation, you should know that it's trivial to disable any of those components if Debian decides it's better off without them. Yet you really seriously present their existence as the biggest factor influencing your position? In what kind of scenario could their existence possibly cause noticeable problems for Debian systemd use? I can imagine some kind of extrapolated arguments about project scope issues that would not be completely laughable, but you haven't really presented any of those. As is, what you're saying just does not form an argument that could be taken seriously at all. The criticisms of Upstart's event model in the systemd position statement simply do not make sense to me. Events model how things actually happen in reality; dependencies are artificial constructions on top of them, and making them work requires the plethora of different directives in systemd (e.g. Wants, which is sort of a non-depending dependency) to avoid blocking the boot process on a single failing service. Dependencies are the natural way to express what people know about the system and what they need. Events are the internal implementation details of what the machine does to make it happen. I want task X started, and it needs task Y is what people express. Start Y, and when Y is ready, start X are the compiled implementation instructions to achieve the higher-level goal. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727708: init system thoughts
On Mon, Dec 30, 2013 at 6:55 PM, Colin Watson cjwat...@debian.org wrote: The ptrace arrangements used for expect fork and expect daemon have been rather flaky in practice, especially when Upstart jobs are written by people not experts in doing so, and they are an obstacle to portability. I think we should strongly discourage use of these in Debian in favour of some other readiness protocol. My personal aesthetic preference is for the expect stop scheme or something close to it, but I really don't care that deeply and the sd_notify scheme or similar would work too. Indeed, I might well be inclined to support disabling the ptrace-requiring features entirely in Debian, and working to disable them in Ubuntu in time as well (although that would require a transition plan). For reasons mentioned by Lennart Poettering in his G+ post, I disagree that expect stop is a good scheme for a readiness protocol. Sure, it is easy for the daemon and init system, but it is not extensible and can be harmful when debugging a process (send it a SIGSTOP, init thinks it is successful and gives it a SIGCONT). NOTIFY_SOCKET is a good scheme because it is very explicit and can not be misinterpreted by the init system. In addition, one init system already has implemented it :) I think Russ has sold me on systemd's journal being a win, at least in terms of how it enables much better status output for services, and it's a shame that something equivalent isn't present in Upstart. My practical experience of late has been that the per-job log files you get by default are good enough for most purposes where I need to debug service operations, but it certainly isn't all packaged up as neatly. Now, since two people agree on this, why do we not file a bug in launchpad asking for the per-job logs to be able to be shown by initctl (possibly via a --show-logs option)? In fact, I just did it: https://bugs.launchpad.net/upstart/+bug/1265123. (P.S. sorry for the sass, it is just so fun :) Nighty night, Cameron Norman
Bug#727708: init system thoughts
On Mon, Dec 30, 2013 at 07:26:23PM -0800, Russ Allbery wrote: (Now, I've been working with Upstart's model for years, and it's now a pretty fundamental way of how I think of system operation; so if people who are new to *both* models rather than partisans of one side or the other consistently tell me that the systemd model is easier to grasp, then I'll listen.) I am new to both models. I'm not very fond of the upstart model. Lennart had a comment on this point that I thought phrased the problem in an interesting way: both systemd and upstart deal with the same data, but with systemd, the full dependency tree is precalculated and exists as state within the init system. With upstart, behavior is dynamic and to some degree emergent: you know who is listening to what signals, but you can arbitrarily introduce signals, and you're dealing with a message bus rather than a dependency tree, so you're reacting to things as they happen on the bus. In systemd, you can dump everything that can possibly happen and work through the state transitions by hand if needed, with the possible exception of unexpected device events (which are horribly important in some cases -- don't get me wrong -- but which are uninteresting in a lot of common debugging scenarios). I think it's somewhat harder to do that with upstart, where events are generated more dynamically. So for a concrete example of where I think upstart's model lets us get things right at boot that systemd's dependency model does not (or at least, this hasn't been done yet in Debian), I'd like to direct your attention to /etc/network/if-up.d/upstart . Conceptually, there are certain kinds of services in a distro context that we don't want to start by default until the network is up - because they aren't set up for socket-based activation, or might need to bind to particular addresses/interfaces and not fall back gracefully, etc. Of course if we were writing all our services according to best practices, we wouldn't have to worry about this, as the service would just handle this gracefully (or maybe hand the complexity off to the init system for socket-based activation - but then what does init do with a request for a socket address that's not available? cycles within cycles, etc). But in the real world, we have a lot of services that we just want to start in runlevel 2 and be able to trust that the network and disk are sorted. This is the classic guarantee that sysvinit gave us pre-udev, but it's fallen apart since then because a fast system can get all the way through rcS before the kernel has even managed to enumerate all the network devices. Upstart (as implemented in Ubuntu) restores this guarantee (with provisions for failsafe booting in the case of a wrong network config), and it takes advantage of upstart's capability of sending arbitrary signals to do so. I can see how one could implement the equivalent of upstart's static-network-up event on systemd, using generators. But what would the equivalent to /etc/init/failsafe.conf look like? I think this would be very difficult to express in systemd language, yet it's altogether vital for providing a boot that is both reliably ordered, and recoverable in the event of problems. Cheers, -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
