Re: Making linaro-nano smaller or Give Up Disk Space for Lent
+++ Dave Martin [2011-03-11 11:20 +]: Although it's not directly related to nano (which is useful in itself as a miminal usable system) it could make sense to be able to generate images with no built-in packager support - i.e., the packer must effectively be run offline to generate the filesystem. Yes. Emdebian calls this 'baked', and has some support for it - mostly in postprocessing the packages to remove all the stuff which is only needed for package-management. http://www.emdebian.org/baked/ There's also the possibility of keeping the packager information outside the main tarball, so the filesystem can be modified/updated offline after creation. Yep - and so long as you have cross-installing working. The problem is that whilst apt has always had excellent suppport for 'install things over there, and keep the database somewhere else', maintainer scripts have no way of being pointed at a sub-dir, so have to be chrooted. I don't think we currently have support for keeping the dpkg database outside the chroot but being able to run all the maintiner scripts in the chroot (also needs qemu, so no use where that is not available). We did show (Ed Bartosh did the work) back in 2005 that's it dead simple to add $(ROOTFS) all over all the scripts and have this work from outside (for most things - there are a few things that go wrong due to useing host config instead of target config), but it's an intrusive solution and we didn't try to push it further. It's nice because it avoids the need for qemu so you get genuine cross-installing. This issue highlights the way that we don't distinguish between install-dependencies and runtime dependencies in packages. A lot of package deps are actually only needed by the install scripts, so on an externally-maintained system are not needed - again making for much smaller images. I'd love to see support for these things in Debian, but both are intrusive - especially the former, so we'd need good justification for it. Perhaps Linaro can provide that... As I understand it, debootstrap or germinate basically do the right thing. All we would need would be to document the use of the existing tools, and provide suitable ultra-minimal seeds (at the level of busybox+libc only) and/or an ultra-minimal --variant for debootstrap. I don't think it's that simple - as explained above. But perhaps there is stuff to deal with this. I haven't been keeping up with all the new chroot and image-making tools. If that's seen to be interesting, we should probably discuss it with the emdebian folks-- there's a risk of reinventing what they do; plus they certainly have tools which are useful for this. Indeed. Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Making linaro-nano smaller or Give Up Disk Space for Lent
Hi, Installed image is 125 Megs. (Down from 290 Meg) We're on the cusp of being able to fit into 128 megs of flash. If that's seen to be interesting, we should probably discuss it with the emdebian folks-- there's a risk of reinventing what they do; plus they certainly have tools which are useful for this. It is interesting indeed... The old style ALIP from ARM's PDSW, in it's minimal configuration, used to take 26MB of space (uncompressed!) providing very simple, yet still functional environment. Frankly speaking, when the nano idea was born, I was hoping that it would be comparable size-wise. Well, at most 50 Megs? (it's already twice bigger ;-) Cheers! Paweł ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Making linaro-nano smaller or Give Up Disk Space for Lent
On Fri, Mar 11, 2011 at 01:01:46PM +0100, Loïc Minier wrote: On Fri, Mar 11, 2011, Dave Martin wrote: As I understand it, debootstrap or germinate basically do the right thing. All we would need would be to document the use of the existing tools, and provide suitable ultra-minimal seeds (at the level of busybox+libc only) and/or an ultra-minimal --variant for debootstrap. So far, the two approaches which had been proposed were: * an initramfs-tools based initrd which would copy selected binaries manually; this probably gives a very minimal root image, but it's a bit cumbersome to manage for us * a classic seed based image; this is convenient to generate, but it's not particularly small The custom debootstrap script you're proposing is one way; I would also think we could consider the udeb route: udebs are meant to be small and used in Debian Installer which offers a rescue system. D-I also has fancy things like openssh, and can retrieve additional components from the network -- as long as they are udeb-ified. D-I images already exist as initrds today, with very small sizes; you can browse random image types under http://ftp.debian.org/debian/dists/stable/main/installer-armel/current/images/ I'm not keen on this route because we are limited then to precisely that set of components that currently build udebs for inclusion in our image. We don't want to be in the position of having to add udebs to the archive in order to make changes to our nano image, that's just too high a barrier. We should be able to get an equivalent effect with an initramfs, which can reuse the existing .debs and extract contents as appropriate. The main thing this won't get us is building with -Os by default; I'm not sure how much that helps on armel, but I wouldn't expect it to offset the maintenance cost? -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Making linaro-nano smaller or Give Up Disk Space for Lent
On Wed, Mar 9, 2011 at 8:15 PM, Tom Gall tom.g...@linaro.org wrote: I don't know why we're installed the firmware deb, does any of the hardware we're supporting even use that? fwiw... TI wlan chipset firmware (wl127x) is in this package. so on pandaboard with current mainline + linux-firmware you can get wifi out-of-the-box. not sure you are interested in wifi on -nano though... ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Making linaro-nano smaller or Give Up Disk Space for Lent
On 9 March 2011 19:15, Tom Gall tom.g...@linaro.org wrote: Going deeper it's pretty easy to spot low hanging fruit: From fs - Do we need afs, jfs, code, minix, hpfs, xfs, hfs, hfsplus, gfs2, reiserfs... I'm thinking no. From drivers - net and media make about about 1/3rd of the 28 meg in use, I'm sure there's a number of drivers in here that can just be turned off. From sound - ac97, are there arm boards that use that? From net - x25, decnet, appletalk ? So that said, what is the best way to proceed? 1) Is there agreement that for all the kernels we supply that we should change the policy for kernel configs to not default to everything on? (Maybe we should be using the upstream config with minimal modifications?) Pro: everyone benefits from the diet Con: Our kernel would be build slightly different than ubuntu's others: Can't we keep stuff configured as modules but move some of the modules out into other binary packages? That way everyone benefits from the diet, but it wouldn't stop anyone installing it if they had weird needs. I also don't think it would be a bad idea to propose making the same change to the Ubuntu kernels if you want to keep the differences down. I'd support turning the really obscure stuff off though, 2) Linaro-media-create shouldn't install linux-firmware_1.47_all.deb ? Do we have any any hardware that needs it? If so could there be a --nano option to not install it? Aren't there a few boards with PCI which could take a whole variety of boards some of which will need firmware? It's surprising just how many things need it, and if it was your ethernet adapter it's really nasty to fix. But I think I agree in generally it could be off by default. One thing that might help is to make it easy to specify on l-m-c that you want to add extra packages to the image; that way if you decide to take some packages out it's not that hard for someone to add them during the creation. Dave ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Making linaro-nano smaller or Give Up Disk Space for Lent
On Wed, 9 Mar 2011 13:15:25 -0600, Tom Gall tom.g...@linaro.org wrote: 2) Linaro-media-create shouldn't install linux-firmware_1.47_all.deb ? Do we have any any hardware that needs it? If so could there be a --nano option to not install it? This is currently in the hwpacks as linux-image-* depend on it. However, they are just metapackages, and the actual kernels do not depend on it. If that dependency isn't going to be removed, but we don't want to always install it then we can add some code to linaro-hwpack-create to avoid it. If though you want it to be optional we would then have to add some code to linaro-media-create to install it again. Another alternative would be the tags idea from when we first discussed hwpacks. This would mean that you could say things like --with firmware at install time, but there will be quite a bit of work involved in this. Plus, in all of this we still have that dependency, so if we aren't installing the firmware package we can't install the kernel metapackage, which is not ideal for upgrades, or do some trick like equivs, or even have a package in the archive (linux-firmware-none) that Provides: linux-firmware, which expresses the intent pretty well, but there would still be thought required of how to select that at linaro-media-create time (and without network access.) 3) linaro-media-create should have some kind of option (--nano) to clear out apt caches (saves ~40 meg of space) If you want this it should be an easy change to make. Thanks, James ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Making linaro-nano smaller or Give Up Disk Space for Lent
On 10/03/11 at 09:20am, James Westby wrote: On Wed, 9 Mar 2011 13:15:25 -0600, Tom Gall tom.g...@linaro.org wrote: 3) linaro-media-create should have some kind of option (--nano) to clear out apt caches (saves ~40 meg of space) If you want this it should be an easy change to make. If its an easy change it seems like a good option to support. I would go with --clear-apt-caches or something similar instead of --nano though unless we are tagging other behaviour on to this option too. Thanks, James Regards, Jamie. ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
RE: Making linaro-nano smaller or Give Up Disk Space for Lent
Hi In the developer platforms team we're working on getting the linaro-nano image so that it is considerably smaller. Brilliant! Some highlights to nano: * The linaro-image boots just as our linaro-headless image did (upstart and friends) * it can be updated, or additional pkgs installed via apt-get * networking works * busybox is in use tho not necessarily universally * ureadahead, python, have been removed * docs have been removed * linux-firmware has been removed (binary kernel firmware blobs) * locales is remove Installed image is 125 Megs. (Down from 290 Meg) We're on the cusp of being able to fit into 128 megs of flash. Isn't it quite big for a Nano image? It looks like a size for a 'developer' version. Do we know why it's so big? In comparison, ARM developed AEL/ALIP (http://www.linux-arm.org/LinuxFilesystem/AELFileSystemPage) and we were hoping to use Linaro's Nano image from now on. The size of the AEL/ALIP minimal version (busybox, ssh) was only 25MB (compressed - Cramfs). And the version with X11 and a desktop was 55MB. Those images are useful for development boards with 64MB of flash or when using CPU models. 3) linaro-media-create should have some kind of option (--nano) to clear out apt caches (saves ~40 meg of space) If you want this it should be an easy change to make. This looks good if it's an easy change. Does it address the problem with Modules and firmware? Is it possible to get an image without Hwpacks? Regards, Guillaume ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Making linaro-nano smaller or Give Up Disk Space for Lent
On Wed, Mar 09, 2011, Tom Gall wrote: Specifically from the installed image after the hwpack deps are installed get rid of the following: rm -f ./var/lib/apt/lists/*Packages rm -f ./var/lib/apt/lists/*Sources rm -f ./var/lib/apt/lists/*Release rm -f ./var/lib/apt/lists/*Release.gpg rm -f ./var/cache/apt/pkgcache.bin rm -f ./var/cache/apt/srcpkgcache.bin I think there is a way for APT to keep compressed versions of these files; it's the Acquire::GzipIndexes option -- Loïc Minier ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Making linaro-nano smaller or Give Up Disk Space for Lent
Dnia 2011-03-10, czw o godzinie 17:43 +0100, Loïc Minier pisze: I think there is a way for APT to keep compressed versions of these files; it's the Acquire::GzipIndexes option And all those files can be recreated by APT when needed so there is no need to keep them in image which has to be really minimal. ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Making linaro-nano smaller or Give Up Disk Space for Lent
On Thu, Mar 10, 2011 at 09:56:12AM +, David Gilbert wrote: 1) Is there agreement that for all the kernels we supply that we should change the policy for kernel configs to not default to everything on? (Maybe we should be using the upstream config with minimal modifications?) Pro: everyone benefits from the diet Con: Our kernel would be build slightly different than ubuntu's others: Can't we keep stuff configured as modules but move some of the modules out into other binary packages? That way everyone benefits from the diet, but it wouldn't stop anyone installing it if they had weird needs. I also don't think it would be a bad idea to propose making the same change to the Ubuntu kernels if you want to keep the differences down. I'd support turning the really obscure stuff off though, That involves a non-trivial amount of build engineering that isn't going to get done this cycle (and definitely not in the Ubuntu kernels). If we want to exclude a module completely, we can turn it off in the config; but if we want to build it and simply package it separately, that's a whole new build system layer that we have to deal with. It could probably reuse some of kernel-wedge's existing handling of module dependencies and so forth; but even so it's not achievable for this cycle. 2) Linaro-media-create shouldn't install linux-firmware_1.47_all.deb ? Do we have any any hardware that needs it? If so could there be a --nano option to not install it? Aren't there a few boards with PCI which could take a whole variety of boards some of which will need firmware? It's surprising just how many things need it, and if it was your ethernet adapter it's really nasty to fix. But I think I agree in generally it could be off by default. My thought here was USB rather than PCI. I think many of the boards have USB interfaces, and I think there's a non-zero number of USB devices that require externally-loaded firmware. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Making linaro-nano smaller or Give Up Disk Space for Lent
+++ Tom Gall [2011-03-09 13:15 -0600]: From sound - ac97, are there arm boards that use that? Some do. I know pxa270-based boards do. I don't know about new, shiny linaro v7-vintage stuff. So that said, what is the best way to proceed? 1) Is there agreement that for all the kernels we supply that we should change the policy for kernel configs to not default to everything on? (Maybe we should be using the upstream config with minimal modifications?) Pro: everyone benefits from the diet Con: Our kernel would be build slightly different than ubuntu's others: Smaller kernel packages would be useful. USB devices are awkward because you can't trim the list according to the target hardware set. In principle people can plug absolutely anything in and you want the drivers to hand. Not sure how much space that involves, but I do know it's a tedious amount of build-time. 2) Linaro-media-create shouldn't install linux-firmware_1.47_all.deb ? Do we have any any hardware that needs it? PLenty of USB devices people might plug in that probably need it. Plenty of USB devices want a firmware blob before they'll do anything. I notice this more with the ones that need a proprietary blob, but expect there are ones that are in linux-firmware too. I see a USB DAB device, a couple of USB tuners and 3 USB modems in the debian firmware-linux-nonfree package. Not sure what's in the ubuntu one. 3) linaro-media-create should have some kind of option (--nano) to clear out apt caches (saves ~40 meg of space) We shouldn't be shipping those in images anyway IMHO. They get re-created automatically on first use. Pro: if you never run apt* you won't mind the cold caches Con: if you're running out of nand and you're going to apt-get upgrade your system, you're probably going to run out of space (Probably a problem to solve in a future cycle) You mean if the original fs is a compressed one, or just where the image only just fit on and there is no room to generate an apt-cache? One thing that has been missing from apt for a long time is ability to split upgrades into coherent chunks so an upgrade requiring downloads larger than the space available can proceed. Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
