Re: using upstart in Debian [was, Re: Debian systemd survey]
❦ 24 mai 2013 12:29 CEST, Dmitrijs Ledkovs x...@debian.org : The best way to run daemons under upstart is in foreground, then correct PID is tracked and the complete stdout/stderr is properly collected and stored in /var/log/upstart/$job.log (even early boot output). The best way to run a daemon under upstart is by using expect stop. This is the one thing I think upstart is better than systemd. This is really easy to implement in most daemons (one line of code). Notifying systemd is not really complex but this cannot be done in one line or without an external dependency. -- Keep it right when you make it faster. - The Elements of Programming Style (Kernighan Plauger) pgpYq0WcJ9yDs.pgp Description: PGP signature
Re: using upstart in Debian [was, Re: Debian systemd survey]
Steve Langasek vorlon at debian.org writes: Sorry you ran into trouble with upstart. Not a DD, just a happy Debian user, hope you'll excuse me, but on the topic of Upstart, I have some technical comments on why, surprisingly, I think it may not be mature enough yet. A couple of years ago I was doing emergency consultancy work for a company using Red Hat and upstart. The dev dude there was really on top of new tech and had made Upstart scripts for starting his Python web apps (which I thought was a great idea, this was back when Upstart looked like THE alternative to sysvinit). However, when debugging it, we had some weird lock-ups from Upstart, basically you'd ask Upstart for the status of the job and it would lock up really hard (IIRC Ctrl-C wouldn't work). After much swearing, it wasn’t immediately obvious why Upstart would be the culprit, I found this (at the time) old bug: https://bugs.launchpad.net/upstart/+bug/406397 Upstart couldn't track the forks going on inside a started process reliably. As one commenter notes: “I’m wondering why this bug has a importance of “low”, as it renders using upstart for many daemons (including apache, postfix and others) as impossible.” This bug doesn't appear to be fixed yet. So unfortunately, I don't think Upstart is ready for handling a server boot with native job files. I had a look at the apache2 packages for Ubuntu raring, and there’s a sysvinit file, but no Upstart job. I'm telling you the whole story here to explain that this isn't just a minor problem for packages shipped with the distribution, it's also a potentially big problem for ISVs. Also on technical merits although more philosophically, with Upstart you're expressing yourself in an event-based DSL rather than writing configuration files. It's pretty generic. But unfortunately, that means it's also not entirely straightforward, and, I believe, easier to get wrong. Scott had some explaining blog posts before he left Canonical that I still find confusing (from the POV of just getting a job file done): http://netsplit.com/2010/12/20/events-are-like-methods/ http://netsplit.com/2010/12/03/event-matching-in-upstart/ Lennart Poettering wrote in his initial blog posts on systemd about why he thought this fundamental model of Upstart wasn't entirely spot on, and while I thought this might have been NIH BS at the time I've later come to the conclusion that he's probably right. Taking as much confusing logic as possible out of the job files does seem like a win. Ole -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/loom.20130523t110151-...@post.gmane.org
Re: using upstart in Debian [was, Re: Debian systemd survey]
On 05/22/2013 06:19 PM, Steve Langasek wrote: On Wed, May 22, 2013 at 03:39:00PM +0200, Bernd Schubert wrote: On 05/22/2013 04:50 AM, Uoti Urpala wrote: Lucas Nussbaum wrote: I went through the various init systems threads again during the last few days. My understanding of the consensus so far is the following: - Both systemd and upstart bring many useful features, and are a clear improvement over sysvinit. Yes, both are an improvement over sysvinit. Hrmm, I have not tested systemd yet, but personally I'm not conviced about the advantages of upstart: - Stops booting *somtimes*, does not provide any information why. I didn't report a bug yet as an almost black screen won't help in any way how to figure out why it stopped. Already that stops without any further information why and where is a sufficiently big design issue, imho. (Btw, in the mean time I belive this issue is related to /etc/mtab, but I'm not sure yet.). Sorry you ran into trouble with upstart. Probably related to /etc/fstab, rather than /etc/mtab... Due to incomplete upstart integration of plymouth in Debian at present, if there are problems in /etc/fstab, mountall won't Well, actually this happens on 2 ubuntu systems. always be able to display any prompt to the user asking what to do (cf. http://web.dodds.net/~vorlon/wiki/blog/Plymouth_is_not_a_bootsplash/). I really meant mtab. After the issue appeared last time again and didn't go away on reboots, I booted into recovery shell and noticed that 'mount -a' didn't do anything. Remouting / rw, deleting /etc/mtab solved it, also on reboots. I now linked /proc/self/mtab to /etc/mtab and that solved it. Might not be an upstart bug at all, but I'm mainly complaining here that upstart does not provide any way to figure out what it is currently waiting for. Just opening a shell after some time and writing something like waiting on XXX to proceed with ... probably would be sufficient. Without any logs, screen output and no way to investigate it just gives the feeling that one now needs to re-install the system, similar to another famous OS . Whereas sysvinit would eventually give up on trying to mount filesystems in /etc/fstab if they were missing at boot and continue booting without them, upstart takes the design decision that this is an error that the admin needs to deal with directly. This ensures that the system always boots reliably in the face of slow disks, which become more of a problem now that your init system is so fast that it can boot before your hardware has been probed. Feel free to file a bug report against the upstart package (with a copy of your /etc/fstab attached) so we can look at this further. - Updating/install programs in a chroot fails with weird messages that those programs (afaik for example X) cannot connect to upstart. Well, it is a chroot, what does it expect? This is bug #685779, which will be fixed in the next upload of sysvinit to unstable. - Personally I'm using unionfs-fuse as very first init script to mount /etc and /var and others on my development systems. So no need to modify an initrams, but a very simple approach and controllable using a boot script. But specifying a script to be run *before* anything else is not possible (yet?). I'm skeptical of the value of such a design in place of just using an initramfs, but the 'friendly-recovery' package in Ubuntu gives an example of to do this. You create an upstart job that's 'start on real-startup-event', runs your necessary pre-setup, and at the end calls 'initctl emit startup' to call into the rest of the boot sequence; then you pass '--startup-event=real-startup-event' to init on the kernel commandline. Thanks, going to look into this. I also need to work on unionfs-fuse to properly work from an initramfs. The issue is just missing time for either of both... I also just wanted to point out that upstart is some way a regression. Thanks, Bernd -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/519f2d5e.2010...@fastmail.fm
Re: using upstart in Debian [was, Re: Debian systemd survey]
On 23 May 2013 10:37, Ole Laursen o...@hardworking.dk wrote: Steve Langasek vorlon at debian.org writes: Sorry you ran into trouble with upstart. Not a DD, just a happy Debian user, hope you'll excuse me, but on the topic of Upstart, I have some technical comments on why, surprisingly, I think it may not be mature enough yet. A couple of years ago I was doing emergency consultancy work for a company using Red Hat and upstart. The dev dude there was really on top of new tech and had made Upstart scripts for starting his Python web apps (which I thought was a great idea, this was back when Upstart looked like THE alternative to sysvinit). However, when debugging it, we had some weird lock-ups from Upstart, basically you'd ask Upstart for the status of the job and it would lock up really hard (IIRC Ctrl-C wouldn't work). After much swearing, it wasn’t immediately obvious why Upstart would be the culprit, I found this (at the time) old bug: https://bugs.launchpad.net/upstart/+bug/406397 Upstart couldn't track the forks going on inside a started process reliably. As one commenter notes: “I’m wondering why this bug has a importance of “low”, as it renders using upstart for many daemons (including apache, postfix and others) as impossible.” This bug doesn't appear to be fixed yet. So unfortunately, I don't think Upstart is ready for handling a server boot with native job files. I had a look at the apache2 packages for Ubuntu raring, and there’s a sysvinit file, but no Upstart job. I'm telling you the whole story here to explain that this isn't just a minor problem for packages shipped with the distribution, it's also a potentially big problem for ISVs. Yes, it's a real bug and as clearly stated in the bug report comments above it is due to using ptrace to count/track forks. The best way to run daemons under upstart is in foreground, then correct PID is tracked and the complete stdout/stderr is properly collected and stored in /var/log/upstart/$job.log (even early boot output). How to establish fork counts the consequences of getting it wrong are well documented in the upstart cookbook (see below). As far as I understand (correct me if I am wrong), systemd instead of counting/tracking forks uses cgroups to keep track of the started processes. Also on technical merits although more philosophically, with Upstart you're expressing yourself in an event-based DSL rather than writing configuration files. It's pretty generic. But unfortunately, that means it's also not entirely straightforward, and, I believe, easier to get wrong. Scott had some explaining blog posts before he left Canonical that I still find confusing (from the POV of just getting a job file done): http://netsplit.com/2010/12/20/events-are-like-methods/ http://netsplit.com/2010/12/03/event-matching-in-upstart/ Since those blog posts were published a coprehensive documentation book was written and is constantly kept up to date. http://upstart.ubuntu.com/cookbook/ It actually guides explains upstart concepts in a coherent and straight-forward manner with a very large section of examples on how to achieve various goals tasks. Lennart Poettering wrote in his initial blog posts on systemd about why he thought this fundamental model of Upstart wasn't entirely spot on, and while I thought this might have been NIH BS at the time I've later come to the conclusion that he's probably right. Taking as much confusing logic as possible out of the job files does seem like a win. Blog posts are interesting to read, but at times I'd like to look up reference manuals which are more than bear minimal man pages. Whilst systemd ships manpages, the website has either incorrectly formatted wiki-pages and/or eventual links to lennart's blog posts. Where is systemd's documentation? Regards, Dmitrijs. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/canbhluhbgvwkp8tsgicgmq2zermb61idbjgwd6261euohqn...@mail.gmail.com
Re: using upstart in Debian [was, Re: Debian systemd survey]
On Fri, May 24, 2013 at 11:29:27AM +0100, Dmitrijs Ledkovs wrote: Blog posts are interesting to read, but at times I'd like to look up reference manuals which are more than bear minimal man pages. Whilst systemd ships manpages, the website has either incorrectly formatted wiki-pages and/or eventual links to lennart's blog posts. Where is systemd's documentation? Could you point out where the systemd documentation or website is lacking? Then I'll send the information to systemd-devel. IMO the documentation of systemd is really good. As said by someone else, the systemd version in Debian is not the latest so please look at the latest version. In any case, looking forward to your reply. -- Regards, Olav -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130524110640.gc24...@bkor.dhs.org
Re: using upstart in Debian [was, Re: Debian systemd survey]
On 24/05/13 11:29, Dmitrijs Ledkovs wrote: As far as I understand (correct me if I am wrong), systemd instead of counting/tracking forks uses cgroups to keep track of the started processes. systemd uses cgroups to track which processes are part of this service?, which means the services may be configured to do traditional Unix daemonization (double-fork), or remain in the foreground, whichever is easier. One advantage of leaving the services forking is that some daemons effectively use the fork() as a signal that they are ready for use (because the point at which they fork and go to the background is the point at which their sysvinit-style init script would exit). For a simple daemon that doesn't fork, init can't necessarily tell when it's actually ready, which can be a problem when doing aggressive parallelization. systemd has several ways to tell that a service is ready, configured with the [Service] Type key in the .service file: the service could fork when it's ready (Type=forking), it could take a D-Bus name when it's ready (Type=dbus), or it could use the systemd-specific sd_notify(3) to tell systemd explicitly (Type=notify). For services that don't do any of those (Type=simple), it can only assume that the service is instantly ready: that's OK when using socket activation, but usually not otherwise. S -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/519f4f03.8060...@debian.org
Re: using upstart in Debian [was, Re: Debian systemd survey]
Dmitrijs Ledkovs xnox at debian.org writes: Also on technical merits although more philosophically, with Upstart you're expressing yourself in an event-based DSL rather than writing configuration files. It's pretty generic. But unfortunately, that means it's also not entirely straightforward, and, I believe, easier to get wrong. Scott had some explaining blog posts before he left Canonical that I still find confusing (from the POV of just getting a job file done): http://netsplit.com/2010/12/20/events-are-like-methods/ http://netsplit.com/2010/12/03/event-matching-in-upstart/ Since those blog posts were published a coprehensive documentation book was written and is constantly kept up to date. I think you misunderstood me. I was not questioning the documentation here, I was questioning some of the fundamental design choices that Scott's blog posts exposed. While the Upstart team has obviously done good work on documentation, I guess fixing mistakes in the architecture after the fact is going to be hard, if it's even on the radar. Ole -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/loom.20130524t181806-...@post.gmane.org
Re: using upstart in Debian [was, Re: Debian systemd survey]
On Fri, May 24, 2013 at 12:29:07PM +0100, Simon McVittie wrote: On 24/05/13 11:29, Dmitrijs Ledkovs wrote: As far as I understand (correct me if I am wrong), systemd instead of counting/tracking forks uses cgroups to keep track of the started processes. systemd uses cgroups to track which processes are part of this service?, which means the services may be configured to do traditional Unix daemonization (double-fork), or remain in the foreground, whichever is easier. One advantage of leaving the services forking is that some daemons effectively use the fork() as a signal that they are ready for use (because the point at which they fork and go to the background is the point at which their sysvinit-style init script would exit). For a simple daemon that doesn't fork, init can't necessarily tell when it's actually ready, which can be a problem when doing aggressive parallelization. Definitely. This is why upstart offers the 'expect fork' and 'expect daemon' options in the first place, and why this limitation in using fork/daemon is important for us to fix. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
using upstart in Debian [was, Re: Debian systemd survey]
On Wed, May 22, 2013 at 03:39:00PM +0200, Bernd Schubert wrote: On 05/22/2013 04:50 AM, Uoti Urpala wrote: Lucas Nussbaum wrote: I went through the various init systems threads again during the last few days. My understanding of the consensus so far is the following: - Both systemd and upstart bring many useful features, and are a clear improvement over sysvinit. Yes, both are an improvement over sysvinit. Hrmm, I have not tested systemd yet, but personally I'm not conviced about the advantages of upstart: - Stops booting *somtimes*, does not provide any information why. I didn't report a bug yet as an almost black screen won't help in any way how to figure out why it stopped. Already that stops without any further information why and where is a sufficiently big design issue, imho. (Btw, in the mean time I belive this issue is related to /etc/mtab, but I'm not sure yet.). Sorry you ran into trouble with upstart. Probably related to /etc/fstab, rather than /etc/mtab... Due to incomplete upstart integration of plymouth in Debian at present, if there are problems in /etc/fstab, mountall won't always be able to display any prompt to the user asking what to do (cf. http://web.dodds.net/~vorlon/wiki/blog/Plymouth_is_not_a_bootsplash/). Whereas sysvinit would eventually give up on trying to mount filesystems in /etc/fstab if they were missing at boot and continue booting without them, upstart takes the design decision that this is an error that the admin needs to deal with directly. This ensures that the system always boots reliably in the face of slow disks, which become more of a problem now that your init system is so fast that it can boot before your hardware has been probed. Feel free to file a bug report against the upstart package (with a copy of your /etc/fstab attached) so we can look at this further. - Updating/install programs in a chroot fails with weird messages that those programs (afaik for example X) cannot connect to upstart. Well, it is a chroot, what does it expect? This is bug #685779, which will be fixed in the next upload of sysvinit to unstable. - Personally I'm using unionfs-fuse as very first init script to mount /etc and /var and others on my development systems. So no need to modify an initrams, but a very simple approach and controllable using a boot script. But specifying a script to be run *before* anything else is not possible (yet?). I'm skeptical of the value of such a design in place of just using an initramfs, but the 'friendly-recovery' package in Ubuntu gives an example of to do this. You create an upstart job that's 'start on real-startup-event', runs your necessary pre-setup, and at the end calls 'initctl emit startup' to call into the rest of the boot sequence; then you pass '--startup-event=real-startup-event' to init on the kernel commandline. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Re: using upstart in Debian [was, Re: Debian systemd survey]
On 05/22/2013 06:19 PM, Steve Langasek wrote: I'm skeptical of the value of such a design in place of just using an initramfs, but the 'friendly-recovery' package in Ubuntu gives an example of to do this. live-config-upstart needs the same to be useful. personally i have no experience with upstart at all and would therefore welcome a patch to implement this properly in live-config, otherwise upstart support will be dropped with one of the next uploads. -- Address:Daniel Baumann, Donnerbuehlweg 3, CH-3012 Bern Email: daniel.baum...@progress-technologies.net Internet: http://people.progress-technologies.net/~daniel.bauman -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/519d2bc5.7050...@progress-technologies.net
