Bug#722898: benchmarks
Hi Thiemo Thanks for working on this. Thiemo Nagel thiemo.na...@gmail.com writes: Hello Ben, thanks for your input! I'm attaching a series of patches to wrap up what we've discussed so far, more details are in the commit messages quoted below. I've tested the patches by running blockdev-wipe, they are looking good. I haven't tried to build the installer with the new block-dev wipe, though, and therefore would appreciate further testing and/or code review. Even with the latest version of blockdev-wipe, I'm still seeing ~20% improvement by setting min_speed to zero. Yet, I'd suggest to hold back the speedlimit patch because I'd expect similar gains for the package installation phase. Thus I believe that it makes more sense to set speed_limit_min to zero during the startup of the debian installer. Could someone please advise me as to where that would fit in best? The best place to enable this is probably partman-md after creating the RAID device. #5blockdev-wipe: Set blocksize to 512k This should be large enough to avoid excessive system call overhead and small enough to prevent problems on systems with very little RAM. Any reason why you choose 512k? If I understand your benchmarks right, doubling this to 1M yelds about another 27% gain. Does increasing the buffer to 1M just increase the memory requirement by 512k or is there a hidden penalty to it? If it's just 512k I don't think we should care as d-i needs in the order of 70-100M overall. And if it turns out to be a problem wiping could be disabled in lowmem situations. #4blockdev-wipe: Sync at most once per second Don't open output devices with O_SYNC, instead sync manually every time the progress indicator is updated, but not more often than once per second. This yields performance gains of up to factor 10 in setups with dm-crypt on dm-raid. Note: Seven years ago, O_SYNC was added to fix OOM issues (cf. bug #381135), however it is believed that this problem has been addressed in the kernel by now. #3blockdev-wipe: Allocate buffer from heap instead of stack This allows to use buffers larger than 8M, also it fails more gracefully in case the memory can't be allocated. #2blockdev-wipe: Reduce progress indicator granularity to 1/1000 This still sounds like a lot of granularity. IMO this could be reduced to 1/100. Do we really need progress updates for less than 1%? Gaudenz -- Ever tried. Ever failed. No matter. Try again. Fail again. Fail better. ~ Samuel Beckett ~ -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87txhkhr4n@meteor.durcheinandertal.bofh
Bug#723216: arcboot-installer link with -L/usr/lib
Package: arcboot-installer Version: 1.21 X-Debbugs-CC: wzss...@gmail.com This package has one or more -L/usr/lib in its build system, which will make it ftbfs if there is libraries under /usr/lib, while is not the default architecture, mips* for example. On mips* systems, /usr/lib is defined as place to hold O32 libraries, and /usr/lib32 for N32, and /usr/lib64 is for N64. Beside the way, on the multiarch system like Debian, user may install libraries under /usr/lib by hand. Please use the default search path if you can, and please consider fix this. I will try to fix this bug, while if you can help to fix it, It will be very appreciative. The attachement is the buildlog of this package on mips64el platform. arcboot-installer_1.21_mips64el.build.xz Description: Binary data
Bug#723263: colo-installer link with -L/usr/lib
Package: colo-installer Version: 1.25 X-Debbugs-CC: wzss...@gmail.com This package has one or more -L/usr/lib in its build system, which will make it ftbfs if there is libraries under /usr/lib, while is not the default architecture, mips* for example. On mips* systems, /usr/lib is defined as place to hold O32 libraries, and /usr/lib32 for N32, and /usr/lib64 is for N64. Beside the way, on the multiarch system like Debian, user may install libraries under /usr/lib by hand. Please use the default search path if you can, and please consider fix this. I will try to fix this bug, while if you can help to fix it, It will be very appreciative. The attachement is the buildlog of this package on mips64el platform. colo-installer_1.25_mips64el.build.xz Description: Binary data
Bug#723297: elilo-installer link with -L/usr/lib
Package: elilo-installer Version: 1.23 X-Debbugs-CC: wzss...@gmail.com This package has one or more -L/usr/lib in its build system, which will make it ftbfs if there is libraries under /usr/lib, while is not the default architecture, mips* for example. On mips* systems, /usr/lib is defined as place to hold O32 libraries, and /usr/lib32 for N32, and /usr/lib64 is for N64. Beside the way, on the multiarch system like Debian, user may install libraries under /usr/lib by hand. Please use the default search path if you can, and please consider fix this. I will try to fix this bug, while if you can help to fix it, It will be very appreciative. The attachement is the buildlog of this package on mips64el platform. elilo-installer_1.23_mips64el.build.xz Description: Binary data
Bug#723307: flash-kernel link with -L/usr/lib
Package: flash-kernel Version: 3.11 X-Debbugs-CC: wzss...@gmail.com This package has one or more -L/usr/lib in its build system, which will make it ftbfs if there is libraries under /usr/lib, while is not the default architecture, mips* for example. On mips* systems, /usr/lib is defined as place to hold O32 libraries, and /usr/lib32 for N32, and /usr/lib64 is for N64. Beside the way, on the multiarch system like Debian, user may install libraries under /usr/lib by hand. Please use the default search path if you can, and please consider fix this. I will try to fix this bug, while if you can help to fix it, It will be very appreciative. The attachement is the buildlog of this package on mips64el platform. flash-kernel_3.11_mips64el.build.xz Description: Binary data
Re: Debian installer build: failed or old builds
On 17/09/13 00:15, Cyril Brulebois wrote: The Err issue while downloading some files (21: Is a directory) only appears on kfreebsd AFAICT, so might be an arch-specific issue. I saw it happen on Linux i386 too. I now realise the buildd log URIs are not valid for very long (could /daily/ become a 302 redirect to something more permanent?), but fortunately I quoted this in an email: http://d-i.debian.org/daily-images/i386/daily/build_cdrom_gtk.log Err http://ftp.gr.debian.org/debian/ unstable/main/debian-installer xserver-xorg-video-fbdev-udeb i386 1:0.4.3-2 Could not open file /build/installer-yUDIZW/build/apt.udeb/cache/archives/partial/ - open (21: Is a directory) Err http://incoming.debian.org/debian/ unstable/main/debian-installer xserver-xorg-video-fbdev-udeb i386 1:0.4.3-2 Could not open file /build/installer-yUDIZW/build/apt.udeb/cache/archives/partial/ - open (21: Is a directory) Failed to fetch http://incoming.debian.org/debian/pool/main/x/xserver-xorg-video-fbdev/xserver-xorg-video-fbdev-udeb_0.4.3-2_i386.udeb Could not open file /build/installer-yUDIZW/build/apt.udeb/cache/archives/partial/ - open (21: Is a directory) And in the case of kfreebsd it was seen with a different mirror server (mirror-ubc). Regards, -- Steven Chamberlain ste...@pyro.eu.org -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/52383a4a.20...@pyro.eu.org
Bug#722898: benchmarks
Hello Gaudenz, thank you for your email! Any reason why you choose 512k? If I understand your benchmarks right, doubling this to 1M yelds about another 27% gain. I'm sorry, I forgot to mention that I've re-run the benchmarks. After removing O_SYNC, the performance was identical for block sizes in the range of 32k to 16M. I chose 512k (16 times larger than the lowest value that I've tested) with the intent to exclude a block size penalty for devices up to 16x faster than my md raid1 setup, which comes in at around 80MB/s. Except for low-memory installs, I'm not aware of any obstacle to increasing the buffer even more. (And of course, there's always the option to test for available memory and chose the buffer size depending on that.) #2blockdev-wipe: Reduce progress indicator granularity to 1/1000 This still sounds like a lot of granularity. IMO this could be reduced to 1/100. Do we really need progress updates for less than 1%? For a large device, wipe times still can be many hours. At a granularity of 1/1000, the progress indicator would advance every 10-50 seconds (order of magnitude), which I don't consider excessive. (Of course, this only holds true if the graphical frontend supports this kind of granularity, which I don't know.) Cheers, Thiemo
Bug#723566: mdcfg: Set guaranteed resync speed to zero to accelerate install
Package: mdcfg Severity: wishlist Tags: d-i patch Hello, from benchmarks in bug #722898 it has emerged that blockdev-wipe is approx. 20% faster when the guaranteed md resync speed [*] is zero, meaning that userspace I/O will always be scheduled ahead of resync I/O. The default setting enforces 1000kB/s resync I/O before userspace requests are honoured. Although setting aside 1MB/s for resync doesn't sound like much, this can translate to a loss of 10-20MB/s for userspace requests, presumably due to competition for disk read/write heads. I assume that changing this setting will also bring speed gains for the package installation phase, that's why I'm proposing a patch (cf. attachment) to alter the setting whenever md devices are detected. Bottom line: The proposed patch gives userspace full I/O priority, but md raid resync will use whatever I/O bandwith that is not required by userspace. Cheers, Thiemo [*] set via /proc/sys/dev/raid/speed_limit_min -- System Information: Debian Release: 7.1 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'proposed-updates'), (500, 'stable') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 3.2.0-4-amd64 (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash From 086c51a14dbbc8868461c41820d712b9158994ae Mon Sep 17 00:00:00 2001 From: Thiemo Nagel thiemo.na...@gmail.com Date: Tue, 17 Sep 2013 14:51:59 +0200 Subject: [PATCH] Set guaranteed resync speed to zero to accelerate install blockdev-wipe is approx. 20% faster that way (cf. bug #722898) and I'd expect that there'll be similar improvements in the package installation phase. --- mdcfg.sh |5 + 1 file changed, 5 insertions(+) diff --git a/mdcfg.sh b/mdcfg.sh index 506e58c..2e3c9f7 100755 --- a/mdcfg.sh +++ b/mdcfg.sh @@ -430,6 +430,11 @@ if ! [ -e /proc/mdstat ] || ! [ -d /dev/md ]; then exit 0 fi + # Speed up installation by de-priorizing (not stopping) array resync + if [ -w /proc/sys/dev/raid/speed_limit_min ]; then + echo 0 /proc/sys/dev/raid/speed_limit_min + fi + # Try to detect MD devices, and start them # mdadm will fail if /dev/md does not already exist mkdir -p /dev/md -- 1.7.10.4
Bug#722898: separate bug for raid speed limit
The issue of the raid speed limit is now tracked in bug #723566. -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAOGcq_58haAa82Lpp+AEYH=pwhtx81mlh65gpmjjxunyufr...@mail.gmail.com
Bug#723141: installation-reports: wheezy /w desktop installs ok, gdm starts on reboot, X fails on login
On Sep 16, 2013, at 5:18 PM, Cyril Brulebois k...@debian.org wrote: The default installation should have left you using nouveau already. What did you change when you configured X with the nouveau driver? I added the attached files to /usr/share/X11. 50-layout.conf Description: Binary data 30-monitor.conf Description: Binary data 40-screen.conf Description: Binary data 60-dri.conf Description: Binary data 20-device.conf Description: Binary data
Bug#722898: benchmarks
Hi
Thiemo Nagel thiemo.na...@gmail.com writes:
Hello Gaudenz,
thank you for your email!
Any reason why you choose 512k? If I understand your benchmarks right,
doubling this to 1M yelds about another 27% gain.
I'm sorry, I forgot to mention that I've re-run the benchmarks. After
removing O_SYNC, the performance was identical for block sizes in the range
of 32k to 16M. I chose 512k (16 times larger than the lowest value that
I've tested) with the intent to exclude a block size penalty for devices up
to 16x faster than my md raid1 setup, which comes in at around 80MB/s.
Except for low-memory installs, I'm not aware of any obstacle to increasing
the buffer even more. (And of course, there's always the option to test for
available memory and chose the buffer size depending on that.)
#2blockdev-wipe: Reduce progress indicator granularity to 1/1000
This still sounds like a lot of granularity. IMO this could be reduced
to 1/100. Do we really need progress updates for less than 1%?
For a large device, wipe times still can be many hours. At a granularity of
1/1000, the progress indicator would advance every 10-50 seconds (order of
magnitude), which I don't consider excessive. (Of course, this only holds
true if the graphical frontend supports this kind of granularity, which I
don't know.)
I don't know about the graphical frontend, but I'm pretty sure the console
based frontend is not able to display a finer granularity than 1/100.
If we are changing this anyway, maybe it's a good time to also make the
template partman-crypto/progress/erase a bit more explicit about
canceling.
It currently reads: Erasing data on ${DEVICE}. Maybe something like
Erasing data on ${DEVICE}. To continue without ereasing press
'Cancel'. How do others feel about this? Several bug reports show that
it's apparently not clear to many users that they can cancel the
operation and what happens if they select cancel.
Gaudenz
--
Ever tried. Ever failed. No matter.
Try again. Fail again. Fail better.
~ Samuel Beckett ~
--
To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87li2vi5tt@meteor.durcheinandertal.bofh
Bug#722898: benchmarks
If we are changing this anyway, maybe it's a good time to also make the
template partman-crypto/progress/erase a bit more explicit about
canceling.
I fully agree!
It currently reads: Erasing data on ${DEVICE}. Maybe something like
Erasing data on ${DEVICE}. To continue without ereasing press
'Cancel'. How do others feel about this?
Maybe Skip would be more precise than Cancel?
Also I think erasing doesn't quite hit the mark because it leads
users to believe that the step isn't necessary for a new drive or if
they don't care about securely deleting its previous contents. How
about something along the lines of: Overwriting ${DEVICE} with random
data to prevent meta-information leaks from the encrypted volume. This
step may be skipped at the expense of a slight reduction of the
quality of the encryption.
Several bug reports show that
it's apparently not clear to many users that they can cancel the
operation and what happens if they select cancel.
I can back this up with my own experience.
--
To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/caogcq_7xkefocvk0-nzwtg0hwzy5ycvp+g5ps6u66b_ruq2...@mail.gmail.com
Re: Weekly build status
On Mon, Sep 16, 2013 at 07:02:48PM +0200, Andrey Gursky wrote: Hi! Are you aware the automatic building of the weekly build is failing for some architectures (amd64, i386, kFreeBSD/i386, kFreeBSD/amd64) from three weeks ago? http://cdimage.debian.org/cdimage/weekly-builds/amd64/ I'm also wondering, it's happened today already the 4th time consecutively. Hi guys, Apologies for not responding to these build failures sooner. The CD builds are failing due to debian-installer build failures. If you check the logs linked from the URL above, you'll find a consistent set of messages about cdrom/initrd.gz not existing. CCing the debian-boot list where people are likely to know more. I *think* this is a know bug in the build scripts that has been causing a lot of failing builds lately. KiBi? -- Steve McIntyre, Cambridge, UK.st...@einval.com Welcome my son, welcome to the machine. -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130917222028.gs14...@einval.com
Re: How to make a custom installer CD/DVD/BD/USB-stick?
[ Just seen this thread, apologies for delayed responses ] On Fri, Jul 05, 2013 at 11:40:54PM -0700, Rick Thomas wrote: On Jul 5, 2013, at 1:25 PM, Brian wrote: Well wouldn't documentation for Jigdo template be in Jigdo package? The jigdo template is a lot lower level than I was hoping for. It's kinda like the object code, where I'm looking for the high-level language source code and a compiler to produce the object code from it. Correct. It's theoretically possible to edit the jigdo template, and all the binary parts it contains, just as it's theoretically possible to edit the java byte-code for a subroutine that almost does what you want -- but it's much more convenient (and comprehensible) to edit the java source code. What I'm hoping for is that there's something like a compiler that takes a human understandable specification, in the form of list of packages and other configurable components of an installer CD/DVD/BD/etc, and outputs a jigdo template (or such) that implements that specification. I'm hoping that's exactly what simple-cdd is. simple-cdd is meant to do that kind of thing AFAIK, yes. Alternatively, debian-cd is the core package that we use for creating the main Debian images. It's very flexible and (sorry!) not particularly well documented... -- Steve McIntyre, Cambridge, UK.st...@einval.com ...In the UNIX world, people tend to interpret `non-technical user' as meaning someone who's only ever written one device driver. -- Daniel Pead -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130917231048.gt14...@einval.com
Debian installer build: failed or old builds
Debian installer build overview --- Failed or old builds: * FAILED BUILD: amd64 Sep 18 00:02 buildd@barber build_cdrom_isolinux http://d-i.debian.org/daily-images/amd64/daily/build_cdrom_isolinux.log * FAILED BUILD: amd64 Sep 18 00:03 buildd@barber build_cdrom_gtk http://d-i.debian.org/daily-images/amd64/daily/build_cdrom_gtk.log * FAILED BUILD: amd64 Sep 18 00:03 buildd@barber build_cdrom-xen http://d-i.debian.org/daily-images/amd64/daily/build_cdrom-xen.log * FAILED BUILD: amd64 Sep 18 00:03 buildd@barber build_netboot http://d-i.debian.org/daily-images/amd64/daily/build_netboot.log * FAILED BUILD: amd64 Sep 18 00:03 buildd@barber build_netboot-gtk http://d-i.debian.org/daily-images/amd64/daily/build_netboot-gtk.log * FAILED BUILD: amd64 Sep 18 00:03 buildd@barber build_netboot-xen http://d-i.debian.org/daily-images/amd64/daily/build_netboot-xen.log * FAILED BUILD: amd64 Sep 18 00:04 buildd@barber build_hd-media http://d-i.debian.org/daily-images/amd64/daily/build_hd-media.log * FAILED BUILD: amd64 Sep 18 00:04 buildd@barber build_hd-media_gtk http://d-i.debian.org/daily-images/amd64/daily/build_hd-media_gtk.log * FAILED BUILD: armel Sep 17 08:10 buildd@ancina build_iop32x_netboot http://d-i.debian.org/daily-images/armel/daily/build_iop32x_netboot.log * FAILED BUILD: armel Sep 17 08:11 buildd@ancina build_iop32x_network-console_glantank http://d-i.debian.org/daily-images/armel/daily/build_iop32x_network-console_glantank.log * FAILED BUILD: armel Sep 17 08:11 buildd@ancina build_iop32x_network-console_n2100 http://d-i.debian.org/daily-images/armel/daily/build_iop32x_network-console_n2100.log * FAILED BUILD: armel Sep 17 08:12 buildd@ancina build_iop32x_network-console_ss4000e http://d-i.debian.org/daily-images/armel/daily/build_iop32x_network-console_ss4000e.log * FAILED BUILD: armel Sep 17 08:12 buildd@ancina build_kirkwood_netboot http://d-i.debian.org/daily-images/armel/daily/build_kirkwood_netboot.log * FAILED BUILD: armel Sep 17 08:13 buildd@ancina build_kirkwood_netboot-gtk http://d-i.debian.org/daily-images/armel/daily/build_kirkwood_netboot-gtk.log * FAILED BUILD: armel Sep 17 08:14 buildd@ancina build_kirkwood_network-console http://d-i.debian.org/daily-images/armel/daily/build_kirkwood_network-console.log * FAILED BUILD: armel Sep 17 08:15 buildd@ancina build_orion5x_network-console http://d-i.debian.org/daily-images/armel/daily/build_orion5x_network-console.log * FAILED BUILD: armel Sep 17 08:15 buildd@ancina build_versatile_netboot http://d-i.debian.org/daily-images/armel/daily/build_versatile_netboot.log * FAILED BUILD: armhf Sep 17 09:39 buildd@hasse build_mx5_netboot http://d-i.debian.org/daily-images/armhf/daily/build_mx5_netboot.log * FAILED BUILD: armhf Sep 17 09:39 buildd@hasse build_mx5_network-console http://d-i.debian.org/daily-images/armhf/daily/build_mx5_network-console.log * FAILED BUILD: armhf Sep 17 09:40 buildd@hasse build_mx5_netboot-gtk http://d-i.debian.org/daily-images/armhf/daily/build_mx5_netboot-gtk.log * FAILED BUILD: armhf Sep 17 09:41 buildd@hasse build_vexpress_netboot http://d-i.debian.org/daily-images/armhf/daily/build_vexpress_netboot.log * FAILED BUILD: i386 Sep 18 00:03 buildd@biber build_cdrom_isolinux http://d-i.debian.org/daily-images/i386/daily/build_cdrom_isolinux.log * FAILED BUILD: i386 Sep 18 00:03 buildd@biber build_cdrom_gtk http://d-i.debian.org/daily-images/i386/daily/build_cdrom_gtk.log * FAILED BUILD: i386 Sep 18 00:03 buildd@biber build_cdrom-xen http://d-i.debian.org/daily-images/i386/daily/build_cdrom-xen.log * FAILED BUILD: i386 Sep 18 00:03 buildd@biber build_netboot http://d-i.debian.org/daily-images/i386/daily/build_netboot.log * FAILED BUILD: i386 Sep 18 00:04 buildd@biber build_netboot-gtk http://d-i.debian.org/daily-images/i386/daily/build_netboot-gtk.log * FAILED BUILD: i386 Sep 18 00:04 buildd@biber build_netboot-xen http://d-i.debian.org/daily-images/i386/daily/build_netboot-xen.log * FAILED BUILD: i386 Sep 18 00:04 buildd@biber build_hd-media http://d-i.debian.org/daily-images/i386/daily/build_hd-media.log * FAILED BUILD: i386 Sep 18 00:04 buildd@biber build_hd-media_gtk http://d-i.debian.org/daily-images/i386/daily/build_hd-media_gtk.log * OLD BUILD:ia64 May 26 00:12 buildd@alkman build_cdrom http://d-i.debian.org/daily-images/ia64/daily/build_cdrom.log * OLD BUILD:ia64 May 26 00:16 buildd@alkman build_netboot
Re: Weekly build status
Quoting Steve McIntyre (st...@einval.com): CCing the debian-boot list where people are likely to know more. I *think* this is a know bug in the build scripts that has been causing a lot of failing builds lately. KiBi? Yes, there is a thread in debian-boot about the daily builds failures. Cyril is trying to investigate the issue or at least to work it around. signature.asc Description: Digital signature
Bug#722898: benchmarks
(let's drop CC: I believe that all participants to this discussion are
subscribed to debian-boot, that receves the bug contributions for D-I packages)
Quoting Thiemo Nagel (thiemo.na...@gmail.com):
It currently reads: Erasing data on ${DEVICE}. Maybe something like
Erasing data on ${DEVICE}. To continue without ereasing press
'Cancel'. How do others feel about this?
Maybe Skip would be more precise than Cancel?
You probably can't change this as the 'Cancel' button comes from the
cdebconf interface.
Also I think erasing doesn't quite hit the mark because it leads
users to believe that the step isn't necessary for a new drive or if
they don't care about securely deleting its previous contents. How
about something along the lines of: Overwriting ${DEVICE} with random
data to prevent meta-information leaks from the encrypted volume. This
step may be skipped at the expense of a slight reduction of the
quality of the encryption.
Though I'm never enthusiast when rewriting debconf messages (because
it needs some translation updates, which is always a PITA with more
than 60 translations), I'm OK with that wording.
signature.asc
Description: Digital signature
