Re: Fedora 16 beta vice Knoppix
Camilo Mesias camilo at mesias.co.uk writes: Hi, I tried some of these changes and they seemed to work reasonably well apart from the grub2 infrastructure is still a bit immature at running without initrd... specifically ... I'm not sure where to report this? Bugs against grub2 or something else? Is there a specific forum for initrdless working? -Cam ... With regard to initrdless boots: https://bugzilla.redhat.com/show_bug.cgi?id=734274 JB -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
Hi, I tried some of these changes and they seemed to work reasonably well apart from the grub2 infrastructure is still a bit immature at running without initrd... specifically * I couldn't find a way to tell the grub2 scripts in /etc/grub.d (10_linux) that I didn't want initrd; I can edit out the initrd line at boot time * new kernels from yum stopped being installed because the grub2-mkconfig script chain relies on /dev/root which is missing in my initrdless boot; if I make ln -s /dev/sda3 /dev/root then the script starts working again * I suspect other stuff in grub2.cfg isn't needed for initrdless (eg. UUID is still mentioned) I'm not sure where to report this? Bugs against grub2 or something else? Is there a specific forum for initrdless working? -Cam On Tue, Oct 4, 2011 at 10:45 PM, Lennart Poettering mzerq...@0pointer.de wrote: On Tue, 04.10.11 21:01, JB (jb.1234a...@gmail.com) wrote: Results interpretation. --- Knoppix won by a wide margin, while: - Knoppix having microknoppix fast-parallel boot (based on SysV/LSB scripts) and DE with low resources usage and tailored for desktops - Fedora having systemd parallel boot and DE tailored for small and simple devices ^ huh? Fedora is not tailored for that. Would be great of it it was, but that's simply not the case. We install LVM and iSCSI and all kinds of other enterprisey stuff on even the smallest netbook. And LVM is a major source of slowness, since it requires all devices to be synchronously settled, before vgscan can be called. Also, we use SELinux and stuff which doesn't speed things up either. SELinux has become a lot faster at boot in F16, so that's good, but there's still a price to pay for it, which is more noticable the weaker your machine is. That said, I do believe that SELinux is a good thing and should definitely be part of the default install. Another bigger source of slowness at boot is currently Plymouth which also requires synchronous settling of devices (tough it's not as bad as LVM in that regard though, but costs too since EDID probing is apparently quite slow, and has every right to, but right now we delay the boot processes for that but we shoudl really do that in the background). I have been asking for the removal of LVM from the default install since a long time, and I am still firmly of the opinion that LVM needs to be something that folks who want it enable but not something that slows down everybody else's boot. If you want a quick boot on a netbook, then remove LVM, iscsi and the other enterprisey storage stuff. Then run systemctl mask fedora-wait-storage.service fedora-storage-init-late.service fedora-readonly.service fedora-storage-init.service fedora-loadmodules.service fedora-autoswap.service fedora-configure.service rc-local.service to mask a couple of always-on services, that are needed for enterprisey and legacy stuff. Also consider disabling stuff like abrtd, or even rsyslog (if you do all log output goes to kmsg, which reduces disk acesses and is often good enough), and audit, cpupower, iptables, lldapd, mcelog, multipathd, lvm2-monitor, mdmonitor, fcoe, dm-event. Check with systemctl list-unit-files what's still left. Then shortcut the initrd by adding rootfstype=ext4 to your kernel cmdline amd replacing root=UUID=X by root=/dev/sda6 (or whatever your harddisk is named in the kernel; what's important here is that the kernel can't look for harddisks by uuid on its own, that's only done by the initrd). Bypassing the initrd is well supported on F16 again, with one exception: plymouth breaks, so disable that: plymouth.disable=0 on the kernel cmdline. On my netbook this gives me a bios-to-gdm bootup time of around 10s, on my laptop of 5s, and Kay's newer laptop of 3s. And it's still an awesomely complete system, including SELinux and everything. And if you compare that with Knoppix then you will still be comparing apples and oranges, but we should be much more in the area of what Knoppix provides as boot times. I'd really like to see Fedora default to some more light-weight choices. Not only for netbooks and suchlike having LVM and all the enterprise stuff in the default is a bad choice, but for server VMs which tend to more lightweight that's the case too. The goals of what is needed to cope with netbooks and what is needed to cope with lightweighter VMs are actually much closer then people might think, and I'd love to see Fedora focus more on both. Lennart -- Lennart Poettering - Red Hat, Inc. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On 10/05/2011 02:41 AM, Adam Williamson wrote: You can try rebuilding your live image with this patch to spin-kickstarts: https://bugzilla.redhat.com/show_bug.cgi?id=739446 to see if it makes any difference. it migrates the livesys stuff to systemd, at least to an extent. -- Migrating livesys itself is just part of it as I found out when I looked at converting the livesys service in what middle of july but due to lack of time/interest from spin SIG we did not make that convertion in time for F16 anyway we are running a bunch of service that dont belong on a livecd/usb ( and on a desktop as well ) as has already be mentioned. If I can recollect then the livecd/usb tools needed to be looked at and patched for proper systemd integration and they seemed to depended on /dev/root which is a pretty broken concept today amongst other things which was amongst few of the gotcha as I ran into while migrating that stuff. JBG -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, 04.10.11 19:38, Adam Williamson (awill...@redhat.com) wrote: On Tue, 2011-10-04 at 15:53 -0800, Jef Spaleta wrote: On Tue, Oct 4, 2011 at 3:32 PM, JB jb.1234a...@gmail.com wrote: Let me append The Blame Game. # systemd-analyze blame 32983ms livesys.service 22828ms NetworkManager.service That timing for NM is so vastly different than what I'm seeing on my installed F15 system. I am intrigued. His numbers are all huge as he's booting live, either from an actual rotating shiny disc thing (the antiquity of it all!) or a USB stick. Either of which is going to be slower than an HD or SSD. If this is indeed a boot from CD then it's not really surprising our numbers are bad since seek times on CDs are awful. If people want to spend optimizing the boot time here it should be possible to reorder the files on the squashfs image to minimize seeking. mksquashfs has the -sort option for that. The data would have to be generated in a two-pass way: first burn and boot the unordered image, use it to determine the access order, then pass that to mksquashfs and generate a second, ordered image. You could use systemd-readahead-collect to collect that access order information, but you'd need to write a tool to convert systemd's format to something readable by mksquashfs. Optimizations like this are always thinkable, but then again spending the time on optimizing CD boots sounds like a lot of time wasted on yesterday's technology. Lennart -- Lennart Poettering - Red Hat, Inc. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Wed, Oct 5, 2011 at 15:09, Lennart Poettering mzerq...@0pointer.de wrote: On Tue, 04.10.11 19:38, Adam Williamson (awill...@redhat.com) wrote: On Tue, 2011-10-04 at 15:53 -0800, Jef Spaleta wrote: On Tue, Oct 4, 2011 at 3:32 PM, JB jb.1234a...@gmail.com wrote: Let me append The Blame Game. # systemd-analyze blame 32983ms livesys.service 22828ms NetworkManager.service That timing for NM is so vastly different than what I'm seeing on my installed F15 system. I am intrigued. His numbers are all huge as he's booting live, either from an actual rotating shiny disc thing (the antiquity of it all!) or a USB stick. Either of which is going to be slower than an HD or SSD. If this is indeed a boot from CD then it's not really surprising our numbers are bad since seek times on CDs are awful. If people want to spend optimizing the boot time here it should be possible to reorder the files on the squashfs image to minimize seeking. mksquashfs has the -sort option for that. The data would have to be generated in a two-pass way: first burn and boot the unordered image, use it to determine the access order, then pass that to mksquashfs and generate a second, ordered image. You could use systemd-readahead-collect to collect that access order information, but you'd need to write a tool to convert systemd's format to something readable by mksquashfs. Optimizations like this are always thinkable, but then again spending the time on optimizing CD boots sounds like a lot of time wasted on yesterday's technology. Might be interesting to just compare the CD boot speed with the same image on a USB stick. That should giva an idea. Kay -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
Lennart Poettering mzerq...@0pointer.de wrote: On Tue, 04.10.11 21:01, JB (jb.1234a...@gmail.com) wrote: Results interpretation. --- Knoppix won by a wide margin, while: - Knoppix having microknoppix fast-parallel boot (based on SysV/LSB scripts) and DE with low resources usage and tailored for desktops - Fedora having systemd parallel boot and DE tailored for small and simple devices ^ huh? Fedora is not tailored for that. Would be great of it it was, but that's simply not the case. We install LVM and iSCSI and all kinds of other enterprisey stuff on even the smallest netbook. [...] This is a great writeup. Why don't you add it to your systemd for sysadmins series? And/or some page on boot speedup in the Fedora wiki? Thanks for the pointers! -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de InformaticaFono: +56 32 2654431 Universidad Tecnica Federico Santa Maria +56 32 2654239 Casilla 110-V, Valparaiso, Chile 234 Fax: +56 32 2797513 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, 04.10.11 19:40, Adam Williamson (awill...@redhat.com) wrote: On Tue, 2011-10-04 at 16:55 -0800, Jef Spaleta wrote: On Tue, Oct 4, 2011 at 3:32 PM, JB jb.1234a...@gmail.com wrote: 13837ms udev-settle.service 11392ms plymouth-start.service if you use the plot option instead of blame option and produce the svg of the service timing you get a better feel for what Lennart was talking about with regard to the udev settle being problematic. I'll try to break it down for you. Keep the following in mind when you look over the svgs produced in susequent testing. udev-settle.service essentially calls udevadm settle and that's all it does. udevadm settle takes FOREVER (15 seconds) to return during boot up on my live media run But its returns more quickly on on F15 install (3 seconds). I'll check a full F16 beta install soonish. And remember that all udevadm settle does is wait for the udev event queue to empty. So essentially all that's going on here is 'wait for udev to be done', which is a fairly sensible prerequisite for all manner of other bits of boot. Nah, this is not a sensible prerequisite. User code should *not* have to wait for udev to be settled. They key message Kay and I and everybody else involved in udev/systemd and related technologies want to get into everybodies head is that applications should never ever expect that everything is settled, since that is simply not possible (i.e. USB init times are unbounded -- so how do you know that the usb disk fully initialized when you settled the udev queue?) and all attempts to fake that are major sources of slowness at boot (to deal with USB and stuff people basically just wait a couple of seconds, which doesn't fix the problem, just tapes over it). Or in other words: udev settle is a hack and is not part of our boot anymore -- unless LVM pulls it in. And the fact that it pulls it in is sad, and has been a constant source of complaining from us to the LVM folks over the years. They major point to make here is that all components of the system should wait exactly as long as they have to and not longer. More specifically: they should wait for the specific hardware they are needing but not any longer. Example: when mounting the file systems systemd will wait exactly until the point all devices listed in /etc/fstab have shown up -- but not any longer before continuing. And again, in short words: udev settle is a hack. Only broken code needs it. It has no place in a modern system. The reasons why udev takes a while to be 'done' are more interesting and Lennart went into some of them. It is completely fine if some probing done by udev rules takes a long time. It's not just fine, it's even expected. For example, I have a 3G card in my laptop I don't use. Since it has no SIM card it takes about 8s seconds to probe (i.e. the firmware finds it funny to reply with an 8s delay to AT commands if no SIM card is in the card). Now, there's no sensible way around this, since the the hw just takes that long to probe. As long as these 8s are spent in the background they shouldn't matter at all. Except that LVM requires settling of all devices, and hence simply enabling LVM means my boot is delayed for a whole 8s. Now thankfully, I opted out of LVM when I installed Fedora on my machine. That way the 8s probing of the modem continues in the background long after gdm is already up. That's why I mentioned in that earlier mail to ajax that I am not concerned that EDID takes so long: because it is OK. What isn't OK is that LVM has to wait for EDID and for my 3G modem probing to complete, and thus delays our entire boot. LVM needs to be ported to listen to hotplug events, and make use of devices as they show up, instead of expecting that all hardware has already shown up and has been probed before LVM is started. For a number of reasons: to not slow down the boot artificially, to fix the enumeration race and become fully compatible with today's storage technology that is much more dynamic than 10 years ago, and to become robust. Please, don't claim that udev settle was a sensible prerequisite. It isn't. It has no place in today's dynamic hardware. Lennart -- Lennart Poettering - Red Hat, Inc. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Wed, 05.10.11 10:17, Horst H. von Brand (vonbr...@inf.utfsm.cl) wrote: Lennart Poettering mzerq...@0pointer.de wrote: On Tue, 04.10.11 21:01, JB (jb.1234a...@gmail.com) wrote: Results interpretation. --- Knoppix won by a wide margin, while: - Knoppix having microknoppix fast-parallel boot (based on SysV/LSB scripts) and DE with low resources usage and tailored for desktops - Fedora having systemd parallel boot and DE tailored for small and simple devices ^ huh? Fedora is not tailored for that. Would be great of it it was, but that's simply not the case. We install LVM and iSCSI and all kinds of other enterprisey stuff on even the smallest netbook. [...] This is a great writeup. Why don't you add it to your systemd for sysadmins series? And/or some page on boot speedup in the Fedora wiki? Thanks for the pointers! Harald posted a blog story about this a while back: http://www.harald-hoyer.de/personal/blog/fedora-15-8-services-you-can-most-likely-disable http://www.harald-hoyer.de/personal/blog/fedora-15-boot-optimization I guess we could do an updated version for F16 of that since a couple of thing changed since then. For example the SELinux policy got updated now so that initrd-less boots do not trigger selinux faults anymore. In fact, initrd-less boots work really nicely nowadays, the only two missing bit to maybe make it a default is basically that a) the kernel cannot boot into a partition by UUID, only by logic name which is a bit fragile [1] and b) Plymouth currently doesn't cope nicely with it since it's called before the video devices are probed if used without an initrd. Lennart [1] What the kernel can do nowadays is mount a GUID partition by the GUID, which might be a pretty good replacement -- at least on machines which have a GUID partition table. -- Lennart Poettering - Red Hat, Inc. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Wed, Oct 5, 2011 at 15:28, Lennart Poettering mzerq...@0pointer.de wrote: On Tue, 04.10.11 19:40, Adam Williamson (awill...@redhat.com) wrote: On Tue, 2011-10-04 at 16:55 -0800, Jef Spaleta wrote: On Tue, Oct 4, 2011 at 3:32 PM, JB jb.1234a...@gmail.com wrote: 13837ms udev-settle.service 11392ms plymouth-start.service if you use the plot option instead of blame option and produce the svg of the service timing you get a better feel for what Lennart was talking about with regard to the udev settle being problematic. I'll try to break it down for you. Keep the following in mind when you look over the svgs produced in susequent testing. udev-settle.service essentially calls udevadm settle and that's all it does. udevadm settle takes FOREVER (15 seconds) to return during boot up on my live media run But its returns more quickly on on F15 install (3 seconds). I'll check a full F16 beta install soonish. And remember that all udevadm settle does is wait for the udev event queue to empty. So essentially all that's going on here is 'wait for udev to be done', which is a fairly sensible prerequisite for all manner of other bits of boot. Nah, this is not a sensible prerequisite. User code should *not* have to wait for udev to be settled. They key message Kay and I and everybody else involved in udev/systemd and related technologies want to get into everybodies head is that applications should never ever expect that everything is settled, since that is simply not possible (i.e. USB init times are unbounded -- so how do you know that the usb disk fully initialized when you settled the udev queue?) and all attempts to fake that are major sources of slowness at boot (to deal with USB and stuff people basically just wait a couple of seconds, which doesn't fix the problem, just tapes over it). Or in other words: udev settle is a hack and is not part of our boot anymore -- unless LVM pulls it in. And the fact that it pulls it in is sad, and has been a constant source of complaining from us to the LVM folks over the years. They major point to make here is that all components of the system should wait exactly as long as they have to and not longer. More specifically: they should wait for the specific hardware they are needing but not any longer. Example: when mounting the file systems systemd will wait exactly until the point all devices listed in /etc/fstab have shown up -- but not any longer before continuing. And again, in short words: udev settle is a hack. Only broken code needs it. It has no place in a modern system. The reasons why udev takes a while to be 'done' are more interesting and Lennart went into some of them. It is completely fine if some probing done by udev rules takes a long time. It's not just fine, it's even expected. For example, I have a 3G card in my laptop I don't use. Since it has no SIM card it takes about 8s seconds to probe (i.e. the firmware finds it funny to reply with an 8s delay to AT commands if no SIM card is in the card). Now, there's no sensible way around this, since the the hw just takes that long to probe. As long as these 8s are spent in the background they shouldn't matter at all. Except that LVM requires settling of all devices, and hence simply enabling LVM means my boot is delayed for a whole 8s. Now thankfully, I opted out of LVM when I installed Fedora on my machine. That way the 8s probing of the modem continues in the background long after gdm is already up. That's why I mentioned in that earlier mail to ajax that I am not concerned that EDID takes so long: because it is OK. What isn't OK is that LVM has to wait for EDID and for my 3G modem probing to complete, and thus delays our entire boot. LVM needs to be ported to listen to hotplug events, and make use of devices as they show up, instead of expecting that all hardware has already shown up and has been probed before LVM is started. For a number of reasons: to not slow down the boot artificially, to fix the enumeration race and become fully compatible with today's storage technology that is much more dynamic than 10 years ago, and to become robust. Please, don't claim that udev settle was a sensible prerequisite. It isn't. It has no place in today's dynamic hardware. Just to make sure that the message is clearly understood and there is nothing sensible in making any assumptions ever, like: 'all devices are there / we have settled'. That can never be true on today's systems. Any system service that today relies in its core on 'udevadm settle' or scsi-wait-scan module, or any of the other bad hacks in that category, anything that uses these barriers as a checkpoint to block on, to do its synchronous actions, should be considered non-hotplug capable, need to be fixed or legacy. The Fedora storage assembly shell scripts really need to be replaced by something that fits into today's reality. There are a few valid
Re: Fedora 16 beta vice Knoppix
Lennart Poettering mzerq...@0pointer.de wrote: [Optimize boot on CD] Optimizations like this are always thinkable, but then again spending the time on optimizing CD boots sounds like a lot of time wasted on yesterday's technology. Humm... for a LiveCD for forensic work (at least) it should be worthwhile, and as long as installation DVDs are distributed, it should be considered for them too. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de InformaticaFono: +56 32 2654431 Universidad Tecnica Federico Santa Maria +56 32 2654239 Casilla 110-V, Valparaiso, Chile 234 Fax: +56 32 2797513 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Wed, 05.10.11 11:40, Horst H. von Brand (vonbr...@inf.utfsm.cl) wrote: Lennart Poettering mzerq...@0pointer.de wrote: [Optimize boot on CD] Optimizations like this are always thinkable, but then again spending the time on optimizing CD boots sounds like a lot of time wasted on yesterday's technology. Humm... for a LiveCD for forensic work (at least) it should be worthwhile, and as long as installation DVDs are distributed, it should be considered for them too. If you think that way, then please start working on it. I am sure the Fedora LiveCD folks would be delighted to get patches for this! Lennart -- Lennart Poettering - Red Hat, Inc. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
JB jb.1234abcd at gmail.com writes: ... The only difference to previous run is that ethernet cable (with good ISP service) was plugged in during boot time. You can see userspace time, and thus total time reduced by more than 300%. # less -i /var/log/messages ... Oct 5 05:33:51 localhost systemd[1]: Startup finished in 1s 456ms 73us (kernel) + 12s 580ms 860us (initrd) + 1min 6s 747ms 352us (userspace) = 1min 20s 784ms 285us. ... # systemd-analyze blame 35513ms livesys.service 25004ms NetworkManager.service 20153ms bluetooth.service 20103ms ip6tables.service 20101ms iptables.service 19726ms sshd-keygen.service 19710ms abrtd.service 17752ms systemd-logind.service 17708ms avahi-daemon.service 13914ms udev-settle.service 8484ms dbus.service 6019ms fedora-loadmodules.service 4972ms mcelog.service 4782ms fcoe.service 4659ms auditd.service 4494ms multipathd.service 4330ms systemd-vconsole-setup.service 3862ms irqbalance.service 3248ms media.mount 3112ms udev-trigger.service 3102ms rsyslog.service 3046ms abrt-ccpp.service 3022ms mdmonitor-takeover.service 2987ms fedora-readonly.service 2490ms console-kit-log-system-start.service 2446ms systemd-remount-api-vfs.service 2335ms udev.service 1391ms systemd-tmpfiles-setup.service 1225ms fedora-storage-init.service 997ms systemd-sysctl.service 759ms systemd-readahead-collect.service 731ms remount-rootfs.service 577ms sm-client.service 518ms netfs.service 218ms fedora-wait-storage.service 131ms fedora-storage-init-late.service 122ms sendmail.service 92ms lvm2-monitor.service 60ms livesys-late.service 54ms console-kit-daemon.service 37ms sandbox.service 34ms iscsi.service 22ms systemd-user-sessions.service 21ms rtkit-daemon.service 14ms accounts-daemon.service Full outputs to analyze and plot are available below (valid for 30 days). Full /var/log/messages: http://pastebin.com/7jh2rHVk Full 'systemd-analyze plot plot.svg': http://pastebin.com/VY8wFUkt JB -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, Oct 4, 2011 at 6:40 PM, Adam Williamson awill...@redhat.com wrote: So essentially all that's going on here is 'wait for udev to be done', which is a fairly sensible prerequisite for all manner of other bits of boot. The reasons why udev takes a while to be 'done' are more interesting and Lennart went into some of them. Right, and as I've said..in the context of the comparison with Knoppix specifically I found evidence that udev settle use to be a long boot up blocker in previous Knoppix releases. I wouldn't be surprised at all if Knoppix init had been changed in the newest release that JB tried to no longer call the settle function (or call it with a very short timeout) But I'm not going to be downloading Knoppix and dissecting its init to prove that to myself. Its obvious from my testing that settle is one of the big blockers, a blocker multiple live distributions have hit in the last year actually. What I'm trying to do is wrap my head around is even if we defaulted to a no LVM install scenario how could we reconstitute the logic associated with fedora-local-fs so the lvm based need for udev settle was optional. It's seems like digging ourselves out of the hole while still supporting lvm as a non-default option could be complicated. -jef -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
Jef Spaleta jspaleta at gmail.com writes: On Tue, Oct 4, 2011 at 6:40 PM, Adam Williamson awilliam at redhat.com wrote: So essentially all that's going on here is 'wait for udev to be done', which is a fairly sensible prerequisite for all manner of other bits of boot. The reasons why udev takes a while to be 'done' are more interesting and Lennart went into some of them. Right, and as I've said..in the context of the comparison with Knoppix specifically I found evidence that udev settle use to be a long boot up blocker in previous Knoppix releases. I wouldn't be surprised at all if Knoppix init had been changed in the newest release that JB tried to no longer call the settle function (or call it with a very short timeout) But I'm not going to be downloading Knoppix and dissecting its init to prove that to myself. Its obvious from my testing that settle is one of the big blockers, a blocker multiple live distributions have hit in the last year actually. ... Here it is. # grep -ir udevadm /etc/ ... /etc/init.d/knoppix-autoconfig: /sbin/udevadm settle /etc/init.d/knoppix-autoconfig: udevadm settle /etc/init.d/knoppix-autoconfig: /sbin/udevadm settle /etc/init.d/open-iscsi: udevadm settle /etc/init.d/udev:if udevadm settle; then ... All references to 'udevadm settle' are without parameters, so: $ man udevadm ... udevadm settle [options] Watches the udev event queue, and exits if all current events are handled. --timeout=seconds Maximum number of seconds to wait for the event queue to become empty. The default value is 180 seconds. A value of 0 will check if the queue is empty and always return immediately. You can see knoppix-autoconfig http://pastebin.com/uU5Av6Pf You can see open-iscsi http://pastebin.com/9nRp5JGh You can see udev http://pastebin.com/aGgghx0s JB -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Wed, Oct 5, 2011 at 9:22 AM, JB jb.1234a...@gmail.com wrote: Here it is. No..that's not it.. that is the starting point necessary to understand the udev differences between the two systems. It is not a dissection. To understand what is happening with udev across those systems you have to look really close at the udev rules and the scripts being fired by the rules to make a comparison about the number of udev events which udevd is trying to deal with before settle returns. For all I know knoppix delibrately ships with a very different udev rule set to keep udev settle fast. I'm sure someone will correct me if I'm wrong but I think a udev with no rules to parse will settle faster than a udev with many rules that need to be parsed, and the details of the rule construction will also matter, as will which modules are loaded in initrd I believe. How many udev events is Knoppix actually processing on your hardware? How many events is Fedora Live processing on the same hardware? I'm pretty sure the number of events to process depends a lot of the ruleset. And how much of the time parsing rules is bound up with slow seek performance and unoptimized file layout? Are you doing spinning media for your comparisons or usb? The fact that the udev init script that you pasted calls /lib/udev/write_dev_root_rule script, which we don't provide at all is already an indication that things are very different in terms of udev configuration across the systems.What other differences are lurking _just_ in the udev config in Knoppix? My point is, this is not an apples to apples comparison of sysinit versus systemd...not by a long shot. To understand why udevadm settle is taking longer in our stack you have to ask some very detailed questions about the udev configuration differences...including ruleset details such as timeouts in each rule being parsed..and not just the master overriding timeout that can be given for the settle command. -jef -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Wed, 2011-10-05 at 15:28 +0200, Lennart Poettering wrote: Please, don't claim that udev settle was a sensible prerequisite. It isn't. It has no place in today's dynamic hardware. Thanks for the correction. (you might want to talk to the anaconda team, then, because liveinst runs 'udevadm settle' when it starts up, and refuses to proceed until udevadm settle is happy.) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
Kay Sievers (kay.siev...@vrfy.org) said: Any system service that today relies in its core on 'udevadm settle' or scsi-wait-scan module, or any of the other bad hacks in that category, anything that uses these barriers as a checkpoint to block on, to do its synchronous actions, should be considered non-hotplug capable, need to be fixed or legacy. The Fedora storage assembly shell scripts really need to be replaced by something that fits into today's reality. ... which has been the plan for a few years now, but is waiting on assorted LVM infrastructure work. Bill -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Wed, Oct 5, 2011 at 5:56 AM, Kay Sievers kay.siev...@vrfy.org wrote: There is no general rule, but anything that calls 'udevadm settle' is suspicious and should be carefully checked if it does not rely on assumptions which just bet on luck and can't reliably work in hotplug setups. Kay, Is there a general purpose way to get an accounting of the number of udev events or the maximum length of the unhandled event que since boot( or since some point in time)? Basically a way to ask how many events udev has been asked to handle prior to the existing call to settle as it exists now in the boot process of Fedora. Not all events are equal, but I'd like to make sure that when I'm looking at settle timing in different situations I have a rough idea of how much work udev has been asked to do ahead of that settle call. -jef -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora 16 beta vice Knoppix
Hi, I performed a simple home test, a comparison of startup and shutdown times of: - Live-CD Fedora 16 beta - systemd parallel boot, GNOME 3 - Live-CD Knoppix 6.7.1 - microknoppix-fast-parallel-boot (based on SysV/LSB scripts), LXDE; note that Knoppix does decompression while executing The times measured were: t1 - time between machine turned ON and showing of live system DE menu t2 - time between machine Shutdown from DE menu and actual machine shutdown Note: no custom configuration or other user activities were performed. Notebook 1: --- Lenovo TP R61i, Intel Pentium Core 2 Duo 1.86 GHZ, Intel Mobile 965GM, 2 GB RAM, HD, CD-RW, sound, internal ethernet and wireless. F16 beta average t1=3m8s average t2=10s Knoppix average t1=1m37s average t2=20s Notebook 2: --- HP Nx6110, Intel Celeron M 1.3 GHZ, Intel Mobile 915GM, 768 MB RAM, HD, CD-RW, sound, internal ethernet. F16 beta average t1=3m42s average t2=26s Knoppix average t1=2m38s average t2=20s Results interpretation. --- Knoppix won by a wide margin, while: - Knoppix having microknoppix fast-parallel boot (based on SysV/LSB scripts) and DE with low resources usage and tailored for desktops - Fedora having systemd parallel boot and DE tailored for small and simple devices JB -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, 04.10.11 21:01, JB (jb.1234a...@gmail.com) wrote: Results interpretation. --- Knoppix won by a wide margin, while: - Knoppix having microknoppix fast-parallel boot (based on SysV/LSB scripts) and DE with low resources usage and tailored for desktops - Fedora having systemd parallel boot and DE tailored for small and simple devices ^ huh? Fedora is not tailored for that. Would be great of it it was, but that's simply not the case. We install LVM and iSCSI and all kinds of other enterprisey stuff on even the smallest netbook. And LVM is a major source of slowness, since it requires all devices to be synchronously settled, before vgscan can be called. Also, we use SELinux and stuff which doesn't speed things up either. SELinux has become a lot faster at boot in F16, so that's good, but there's still a price to pay for it, which is more noticable the weaker your machine is. That said, I do believe that SELinux is a good thing and should definitely be part of the default install. Another bigger source of slowness at boot is currently Plymouth which also requires synchronous settling of devices (tough it's not as bad as LVM in that regard though, but costs too since EDID probing is apparently quite slow, and has every right to, but right now we delay the boot processes for that but we shoudl really do that in the background). I have been asking for the removal of LVM from the default install since a long time, and I am still firmly of the opinion that LVM needs to be something that folks who want it enable but not something that slows down everybody else's boot. If you want a quick boot on a netbook, then remove LVM, iscsi and the other enterprisey storage stuff. Then run systemctl mask fedora-wait-storage.service fedora-storage-init-late.service fedora-readonly.service fedora-storage-init.service fedora-loadmodules.service fedora-autoswap.service fedora-configure.service rc-local.service to mask a couple of always-on services, that are needed for enterprisey and legacy stuff. Also consider disabling stuff like abrtd, or even rsyslog (if you do all log output goes to kmsg, which reduces disk acesses and is often good enough), and audit, cpupower, iptables, lldapd, mcelog, multipathd, lvm2-monitor, mdmonitor, fcoe, dm-event. Check with systemctl list-unit-files what's still left. Then shortcut the initrd by adding rootfstype=ext4 to your kernel cmdline amd replacing root=UUID=X by root=/dev/sda6 (or whatever your harddisk is named in the kernel; what's important here is that the kernel can't look for harddisks by uuid on its own, that's only done by the initrd). Bypassing the initrd is well supported on F16 again, with one exception: plymouth breaks, so disable that: plymouth.disable=0 on the kernel cmdline. On my netbook this gives me a bios-to-gdm bootup time of around 10s, on my laptop of 5s, and Kay's newer laptop of 3s. And it's still an awesomely complete system, including SELinux and everything. And if you compare that with Knoppix then you will still be comparing apples and oranges, but we should be much more in the area of what Knoppix provides as boot times. I'd really like to see Fedora default to some more light-weight choices. Not only for netbooks and suchlike having LVM and all the enterprise stuff in the default is a bad choice, but for server VMs which tend to more lightweight that's the case too. The goals of what is needed to cope with netbooks and what is needed to cope with lightweighter VMs are actually much closer then people might think, and I'd love to see Fedora focus more on both. Lennart -- Lennart Poettering - Red Hat, Inc. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, 2011-10-04 at 23:45 +0200, Lennart Poettering wrote: Another bigger source of slowness at boot is currently Plymouth which also requires synchronous settling of devices (tough it's not as bad as LVM in that regard though, but costs too since EDID probing is apparently quite slow, and has every right to, but right now we delay the boot processes for that but we shoudl really do that in the background). It's much slower than it needs to be. As of last time I looked there's still cases where a) we're using full EDID fetch as a proxy for connected-or-not, instead of just a zero-length I2C read, and b) we're not prefetching and caching EDID on hotplug interrupts, instead fetching it every time it's asked for. Even given all that, we should try the faster I2C speeds first and fall back to slower. And we're not. Might or might not be able to fix all that up for F16 gold, but it's on the todo list somewhere. - ajax signature.asc Description: This is a digitally signed message part -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, Oct 4, 2011 at 11:45 PM, Lennart Poettering mzerq...@0pointer.de wrote: On Tue, 04.10.11 21:01, JB (jb.1234a...@gmail.com) wrote: Results interpretation. --- Knoppix won by a wide margin, while: - Knoppix having microknoppix fast-parallel boot (based on SysV/LSB scripts) and DE with low resources usage and tailored for desktops - Fedora having systemd parallel boot and DE tailored for small and simple devices ^ huh? Fedora is not tailored for that. Would be great of it it was, but that's simply not the case. We install LVM and iSCSI and all kinds of other enterprisey stuff on even the smallest netbook. And LVM is a major source of slowness, since it requires all devices to be synchronously settled, before vgscan can be called. Also, we use SELinux and stuff which doesn't speed things up either. SELinux has become a lot faster at boot in F16, so that's good, but there's still a price to pay for it, which is more noticable the weaker your machine is. That said, I do believe that SELinux is a good thing and should definitely be part of the default install. Another bigger source of slowness at boot is currently Plymouth which also requires synchronous settling of devices (tough it's not as bad as LVM in that regard though, but costs too since EDID probing is apparently quite slow, and has every right to, but right now we delay the boot processes for that but we shoudl really do that in the background). I have been asking for the removal of LVM from the default install since a long time, and I am still firmly of the opinion that LVM needs to be something that folks who want it enable but not something that slows down everybody else's boot. If you want a quick boot on a netbook, then remove LVM, iscsi and the other enterprisey storage stuff. Then run systemctl mask fedora-wait-storage.service fedora-storage-init-late.service fedora-readonly.service fedora-storage-init.service fedora-loadmodules.service fedora-autoswap.service fedora-configure.service rc-local.service to mask a couple of always-on services, that are needed for enterprisey and legacy stuff. Also consider disabling stuff like abrtd, or even rsyslog (if you do all log output goes to kmsg, which reduces disk acesses and is often good enough), and audit, cpupower, iptables, lldapd, mcelog, multipathd, lvm2-monitor, mdmonitor, fcoe, dm-event. And *sendmail* (in my vms it takes up to 60s to start even though I never use it; and I it does not really make much sense on desktops). -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, Oct 04, 2011 at 11:59:09PM +0200, drago01 wrote: And *sendmail* (in my vms it takes up to 60s to start even though I never use it; and I it does not really make much sense on desktops). https://fedoraproject.org/wiki/Features/NoMTA Yeah ... I was technically the owner on that one, suppose I did a massive fail there. I suppose we could try and bring this back for F17? (I would like to note that I did zero of the heavy lifting on the feature and would like to thank Will Woods for his awesome patches that went into the NoMTA bits) -AdamM -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On 10/04/2011 11:01 PM, JB wrote: I performed a simple home test, a comparison of startup and shutdown times of: - Live-CD Fedora 16 beta - systemd parallel boot, GNOME 3 - Live-CD Knoppix 6.7.1 - microknoppix-fast-parallel-boot (based on SysV/LSB scripts), LXDE; This is roughly analogous to: I performed a simple track test, a comparison of lap times of: - A family station wagon, tuned and optimized for everyday driving - A Formula 1 race car, tuned and optimized for track racing Ignoring the general irrelevancy of such an apples to oranges comparison for a moment, the conclusion that I draw is this: For a family station wagon, it isn't that slow. That's not to say that there are no places that we can optimize Fedora livecd performance, or that we should just be happy with how things are, but if you want to be taken seriously, you cannot compare it to Knoppix, whose entire focus and efforts are focused around generating an extremely minimal and fast Live Linux experience, and you cannot choose to compare different DEs. I know I shouldn't feed the anonymous trolls. I guess the jetlag is making me punchy, and I'm avoiding Chromium NativeClient. ~tom == Fedora Project -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, 04.10.11 17:54, Adam Jackson (a...@redhat.com) wrote: On Tue, 2011-10-04 at 23:45 +0200, Lennart Poettering wrote: Another bigger source of slowness at boot is currently Plymouth which also requires synchronous settling of devices (tough it's not as bad as LVM in that regard though, but costs too since EDID probing is apparently quite slow, and has every right to, but right now we delay the boot processes for that but we shoudl really do that in the background). It's much slower than it needs to be. As of last time I looked there's still cases where a) we're using full EDID fetch as a proxy for connected-or-not, instead of just a zero-length I2C read, and b) we're not prefetching and caching EDID on hotplug interrupts, instead fetching it every time it's asked for. Even given all that, we should try the faster I2C speeds first and fall back to slower. And we're not. Might or might not be able to fix all that up for F16 gold, but it's on the todo list somewhere. Well, tbh I am not too concerned about the initialization speed of specific drivers like the DRI stuff. What matters more is that we can initialize them in parallel with other stuff, and don't have to synchronously stall the entire boot for it, which Plymouth currently makes us to for the DRI drivers. Or in other words: I believe the priority should be to fix Plymouth. Fixing the EDID stuff matters only if we manage to pull gdm so early into the boot that EDID would become a stumbling block since gdm/X11 actually really needs the EDID stuff to be fully probed. Lennart -- Lennart Poettering - Red Hat, Inc. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
JB jb.1234abcd at gmail.com writes: ... Notebook 1: --- Lenovo TP R61i, Intel Pentium Core 2 Duo 1.86 GHZ, Intel Mobile 965GM, 2 GB RAM, HD, CD-RW, sound, internal ethernet and wireless. F16 beta average t1=3m8s average t2=10s ... Let me append The Blame Game. # less -i /var/log/messages ... Oct 4 20:40:40 localhost systemd[1]: Startup finished in 1s 438ms 413us (kernel) + 12s 445ms 772us (initrd) + 3min 58s 333ms 952us (userspace) = 4min 12s 218ms 137us. ... # systemd-analyze blame 32983ms livesys.service 22828ms NetworkManager.service 19438ms bluetooth.service 19247ms iptables.service 19245ms ip6tables.service 18837ms abrtd.service 18672ms mcelog.service 18657ms auditd.service 18035ms irqbalance.service 16885ms rsyslog.service 16814ms systemd-logind.service 16766ms avahi-daemon.service 16696ms abrt-ccpp.service 16659ms mdmonitor-takeover.service 13837ms udev-settle.service 11392ms plymouth-start.service 6264ms fedora-loadmodules.service 4306ms systemd-vconsole-setup.service 4258ms multipathd.service 3850ms fedora-storage-init.service 3744ms fcoe.service 3453ms fedora-readonly.service 3270ms media.mount 3121ms udev-trigger.service 2728ms console-kit-log-system-start.service 2483ms systemd-remount-api-vfs.service 2283ms udev.service 2189ms dbus.service 1994ms systemd-tmpfiles-setup.service 1332ms fedora-storage-init-late.service 1061ms systemd-sysctl.service 790ms remount-rootfs.service 456ms sm-client.service 404ms netfs.service 385ms fedora-wait-storage.service 341ms lvm2-monitor.service 279ms systemd-readahead-collect.service 253ms rtkit-daemon.service 77ms sendmail.service 57ms console-kit-daemon.service 45ms livesys-late.service 44ms iscsi.service 31ms sandbox.service 15ms accounts-daemon.service 12ms systemd-user-sessions.service JB -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, Oct 4, 2011 at 3:32 PM, JB jb.1234a...@gmail.com wrote: Let me append The Blame Game. # systemd-analyze blame 32983ms livesys.service 22828ms NetworkManager.service That timing for NM is so vastly different than what I'm seeing on my installed F15 system. I am intrigued. -jef -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, Oct 4, 2011 at 3:32 PM, JB jb.1234a...@gmail.com wrote: 13837ms udev-settle.service 11392ms plymouth-start.service if you use the plot option instead of blame option and produce the svg of the service timing you get a better feel for what Lennart was talking about with regard to the udev settle being problematic. I'll try to break it down for you. Keep the following in mind when you look over the svgs produced in susequent testing. udev-settle.service essentially calls udevadm settle and that's all it does. udevadm settle takes FOREVER (15 seconds) to return during boot up on my live media run But its returns more quickly on on F15 install (3 seconds). I'll check a full F16 beta install soonish. as a result udev-settle.service its essentially a hard blocking dependency in the dependency ordering through the fedora-wait-storage.service which waits for udev-settle and completes before local-fs.service starts which in terms comes before a lot of other things via the dependency ordering (from visual inspection of the Before After logic in the service files). It's not just plymouth-start service that waits for udev-settle. So my question to use is. Is Knoppix at any point running udevadm settle in its init in the knoppix you tried? In the past Knoppix releases have also been impacted by udevadm settle taking a long time as well. I let you google for the forum references. It's not just Knoppix that's been impacted, other thin distros seem to have run afoul of long udevadm settle init waits according to multiple 2011 google hits ive found. Times are tough all over it seems. So my question to you is this, is the version of knoppix you are using still calling udevadm settle as part of its init or is it delibrately short circuited with a short duration timeout? -jef -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
Jef Spaleta jspaleta at gmail.com writes: On Tue, Oct 4, 2011 at 3:32 PM, JB jb.1234abcd at gmail.com wrote: Let me append The Blame Game. # systemd-analyze blame 32983ms livesys.service 22828ms NetworkManager.service That timing for NM is so vastly different than what I'm seeing on my installed F15 system. I am intrigued. -jef The ethernet cable was not plugged in, so it tried too hard ? If so, it should be looked into (I noticed on F15 a few times that when I lost my ISP service and checked for it again by restarting NM that it tried I believe 3 times by many seconds before finally giving up). The wireless and some 5 AP signals were identified. JB -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, 2011-10-04 at 15:53 -0800, Jef Spaleta wrote: On Tue, Oct 4, 2011 at 3:32 PM, JB jb.1234a...@gmail.com wrote: Let me append The Blame Game. # systemd-analyze blame 32983ms livesys.service 22828ms NetworkManager.service That timing for NM is so vastly different than what I'm seeing on my installed F15 system. I am intrigued. His numbers are all huge as he's booting live, either from an actual rotating shiny disc thing (the antiquity of it all!) or a USB stick. Either of which is going to be slower than an HD or SSD. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, 2011-10-04 at 16:55 -0800, Jef Spaleta wrote: On Tue, Oct 4, 2011 at 3:32 PM, JB jb.1234a...@gmail.com wrote: 13837ms udev-settle.service 11392ms plymouth-start.service if you use the plot option instead of blame option and produce the svg of the service timing you get a better feel for what Lennart was talking about with regard to the udev settle being problematic. I'll try to break it down for you. Keep the following in mind when you look over the svgs produced in susequent testing. udev-settle.service essentially calls udevadm settle and that's all it does. udevadm settle takes FOREVER (15 seconds) to return during boot up on my live media run But its returns more quickly on on F15 install (3 seconds). I'll check a full F16 beta install soonish. And remember that all udevadm settle does is wait for the udev event queue to empty. So essentially all that's going on here is 'wait for udev to be done', which is a fairly sensible prerequisite for all manner of other bits of boot. The reasons why udev takes a while to be 'done' are more interesting and Lennart went into some of them. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedora 16 beta vice Knoppix
On Tue, 2011-10-04 at 23:32 +, JB wrote: JB jb.1234abcd at gmail.com writes: ... Notebook 1: --- Lenovo TP R61i, Intel Pentium Core 2 Duo 1.86 GHZ, Intel Mobile 965GM, 2 GB RAM, HD, CD-RW, sound, internal ethernet and wireless. F16 beta average t1=3m8s average t2=10s ... Let me append The Blame Game. # less -i /var/log/messages ... Oct 4 20:40:40 localhost systemd[1]: Startup finished in 1s 438ms 413us (kernel) + 12s 445ms 772us (initrd) + 3min 58s 333ms 952us (userspace) = 4min 12s 218ms 137us. ... # systemd-analyze blame 32983ms livesys.service You can try rebuilding your live image with this patch to spin-kickstarts: https://bugzilla.redhat.com/show_bug.cgi?id=739446 to see if it makes any difference. it migrates the livesys stuff to systemd, at least to an extent. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
