Re: building Rootfs
On Mon, Jul 07, 2008 at 06:24:48PM -0500, Haller, John H (John) wrote: There are three issues in building an embedded cross-filesystem: 1. Identification of the files to put into the filesystem. 2. Stripping or extracting debug symbols from the filesystem. 3. Getting files into the filesystem when the development system does not allow root access. Ack. In PTXdist, we also cover 4. what to put into the initial image on deployment (we had what you describe in 3 above but split that up into what to build vs. what to install in ptxdist-2.x). 5. how to push the image into the device (managing and writing complicated u-boot environments and pushing it into the device automatically, via serial+ethernet) 6. automatic test suites against embedded targets The first item is made a little easier by RPM, as it groups files together into packages, and package dependencies are quite often properly declared. However, packages sometimes have dependencies on another package when only part of the package may be required. Another problem is with documentation files (usually unwanted on an embedded system), and locales which are huge and generally unwanted. That was the reason why we went with ipkg; the meta data of rpm and deb are too large to be hold on an embedded system, but we want software updatability. However, if someone needs any sort of Carrier Grade Blubberblabber, we could easily add rpm and/or deb :-) The second item can be tricky, as one doesn't want to strip debug symbols every time a filesystem is built, but the debug symbols are generally needed for cross debugging. Yup; ptxdist solves this by having root/ and root-debug/, where the first one can be booted via NFS or flash, and the second one contains all the debug symbols. Cross debugging works that way, even with Eclipse (for the people who deny seeing the pure elegance and shinyness of vim and commandline gdb). The third item is easy for an initramfs, but not well handled by existing tools for an initrd (although I haven't looked recently). One possibility is to modify a mkfs program, such as mke2fs, for the filesystem of choice to accept proto files, like the traditional Unix mkfs does. [Google mkfs proto for some examples]. I'm not sure if ptxdist supports both initramfs and initrd. If you do have root access, this is all much simpler, as an initrd can be locally mounted and populated. I'm not sure as well; the parts for initramfs are there, but we don't use it here at Pengutronix, because all of our current customer systems have normal flash root filesystems; we once had a system with IXP425 that needed a userspace firmware for nfsroot, but that's the only case I can remember. However, if somebody needs improvement, he is free to send patches... One other item that pops up sometimes is post-install scripts that are associated with some packages. These scripts run in the target filesystem, and there is no good way to run them in the host environment. This is already solved :-) We have pre/post scripts, this way: rules/foobar.make rules/foobar.preinst rules/foobar.postinst ... In the scripts you can check if DESTDIR is set; if yes, you are in the host environment (development system tries to build a root fs from the ipkgs); if not, you are on the target (something does 'ipkg install'). With that mechanics, you can for example decide that on the target you need to stop a service first, then update, then start it again. Everything but update isn't required if only an image is being made. This doesn't happen too often, so it can usually be handled by hand as an exception. One example of this type of item is installing the symbolic links in the right place for init scripts. Yup, that can be done with the mechanics above. Using the tools from an embedded vendor has two problems - they can change from release to release, We have a stable ptxdist-1.x series with long term bugfix maintenance for that ... even this stable series is completely open source, for everyone's download pleasure. and they are proprietary tools that lock you into the vendor. No need to do so. Everything in ptxdist is GPL. If somebody wants to be locked to Pengutronix, we feel better if he does so because he has the warm and cuddly feeling that we solve his problems best ... which, by the way, doesn't work too bad. But, they may provide a GUI, if you are into graphical presentations. Well, if someone absolutely needs a GUI, he can use gvim¹ :-) rsc ¹ Or even Eclipse vmware, if your software department has a better feeling while using buzzword technology. -- Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de Pengutronix - Linux Solutions for Science and Industry Handelsregister: Amtsgericht Hildesheim, HRA 2686 Hannoversche Str. 2, 31134 Hildesheim, Germany Phone: +49-5121-206917-0 | Fax: +49-5121-206917-9 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a
Re: building Rootfs
In message [EMAIL PROTECTED] you wrote: There are three issues in building an embedded cross-filesystem: 1. Identification of the files to put into the filesystem. 2. Stripping or extracting debug symbols from the filesystem. 3. Getting files into the filesystem when the development system does not allow root access. 3. is a non-issue for most common file systems. The third item is easy for an initramfs, but not well handled by existing tools for an initrd (although I haven't looked recently). One possibility is to modify a mkfs program, such genext2fs has been around for a long, long time. And there are similar tools for other file system types like cramfs or jffs2 ... Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [EMAIL PROTECTED] The human race is a race of cowards; and I am not only marching in that procession but carrying a banner. - Mark Twain -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: building Rootfs
2008/7/8 Wolfgang Denk [EMAIL PROTECTED]: 3. Getting files into the filesystem when the development system does not allow root access. 3. is a non-issue for most common file systems. The only one thing I ever ran into trouble with was device nodes, these cannot be reproduced any way, not even with fakeroot environments, just in scratchbox, which in turn needs you to be root. initramfs etc can handle it with special description files. But if you want to cook up say a .tar file of your rootfs, you're pretty much lost AFAIK. The good thing is that you don't need the device nodes if you have udev, I think the kernel wants /dev/console and a few more at boot but actually it survives just fine without them. The rest is possible to create with fakeroot and clever scripting if I'm not mistaken. If you know some way of sneaking a device node into a .tar file created ENTIRELY running as a regular user, tell me! Linus -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: building Rootfs
2008/7/7 Robert Schwebel [EMAIL PROTECTED]: On Mon, Jul 07, 2008 at 10:34:11PM +0200, Linus Walleij wrote: Robert, can you brief us of how ptxdist fits together with OpenEmbedded? What does these two projects actually share? Where do they do similar things in parallel for example? They are both build systems for userlands, or whole embedded linux systems. I cannot speak for OE as I don't really know it in detail. Anyone have experience with both? I'd really like to have them contrasted and, well you know, cross-firtualized. The idea behind ptxdist is executable documentation. I really like ptxdist, especially it is beautiful that you have based it around just bash and make plus some crosscompiler of choice. I just have some trouble getting it to do what I want. I'll hopefully get there. This is not meant as any generic complaining. I'm trying to figure out if ptxdist or openembedded is the best starting point. I have a situation where the build system would be on top of some SCM like git, CVS or whatever and packages being checked out and worked on by several people daily. This seems to clash with ptxdist design goals, for example: ptxdist seems * hardcoded to run in a users home directory with projects in $HOME/foo (stopped me right now, perhaps it can easily be fixed or is already part of ptxdist 2?) * assume only one person builds rootfs. So it doesn't really want the project tree to be used or modified by more than one person at the time. (Same as first more or less.) * not store source code, patches and config plus my local additions in one place. For most other package systems e.g. RPM there is /foo/foo.spec /foo/foo-1.0.tar.gz /foo/foo-patch1.diff /foo/foo-patch2.diff /foo/my-foo-config.file In ptxdist these seem to be spread out, some in the installation of ptxdist itself in say /usr/lib/ptxdist/patches, /usr/lib/ptxdist/... then in my project space $HOME/project/... etc. I believe at archivicing time everything including ptxdist itself is zipped up and archived on one storage media. But I need to keep all files relating to a package in one place while developing, and living like that for a long time. Since /usr/lib/ptxdist isn't even being backed up normally, only NFS mounts like /home, I then need to move the entire ptxdist into the SCM, which it clearly doesn't like without a lot of hacking, plus, how would I upgrade it? OK this is a bit picky. And perhaps I can fix it all and merge back. Just trying to decide. Linus -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: building Rootfs
Linus Walleij wrote: If you know some way of sneaking a device node into a .tar file created ENTIRELY running as a regular user, tell me! We do this all the time. We have a very minimal root filesystem and aren't currently using any of the frameworks for building root filesystems, so everything is done with make. In general, we create tar files for each piece. The piece that has device nodes has a make file with something like: all: fakeroot make fakeroot-package fakeroot-package: mkdir -p image mknod image/dev/null c 1 3 mknod image/dev/random c 1 8 tar -C image -czf image.tgz . We ran into some issues when we separated running mknod and tar into separate fakeroot invocations because there is a bug in fakeroot with exporting and importing its state. Fortunately, doing all of this under a single fakeroot command works just fine. When we untar the tar files for all of the pieces to create a root filesystem image and then run mkcramfs to create a CRAMFS image, we do all commands under a single invocation of fakeroot. No stage of the build requires running as root. -- David VomLehn, [EMAIL PROTECTED] - - - - - Cisco- - - - - This e-mail and any attachments may contain information which is confidential, proprietary, privileged or otherwise protected by law. The information is solely intended for the named addressee (or a person responsible for delivering it to the addressee). If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer. -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: building Rootfs
OK Robert wrote this on ARM-Linux mailinglist but the discussion belongs on linux-embedded, so I'm moving it there, the suggested ptxdist ML seems inapropriate for the entire embedded community. 2008/7/7 Robert Schwebel [EMAIL PROTECTED]: On Mon, Jul 07, 2008 at 06:01:51PM +0800, Eric Miao wrote: or start from buildroot, scratchbox and openembedded as a tool, you've really got a lot to learn :-) Or ptxdist, or course :-) Robert, can you brief us of how ptxdist fits together with OpenEmbedded? What does these two projects actually share? Where do they do similar things in parallel for example? I'm trying - real hard - to get an idea of how people out there prefer to build their root filesystems in cross-compiled environments. Rob Landley recently wrote up a small blurb on why native compilation is the way to go, and a small roadmap on how he intended to get there using e.g. Qemu and Firmware Linux, c.f: http://www.landley.net/code/firmware/about.html which finally won me over to that line of thinking. Debian and friends obviously go this way now. However, when it comes to the widespread and much fragmented ways of cross-compiling a rootfs, including the stuff put together by MontaVista, WR and all those animals in the forest, obviously based on RPM (build systems I haven't put my hands inside, since they are proprietary) there seems to be very little consensus. ptxdist stands out but do you get a lot of outside contributions for it? As it looks it seems you're running it yourself. (Beware: I haven't looked close.) What else is there out there for rootfs, really? A hack from every embedded company there is? I'm more after what people actually *use* and what is community driven here, not so much opinions on what is best (which will probably be the unwanted side effect of this mail anyway...) Linus -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: building Rootfs
On Mon, Jul 07, 2008 at 10:34:11PM +0200, Linus Walleij wrote: Robert, can you brief us of how ptxdist fits together with OpenEmbedded? What does these two projects actually share? Where do they do similar things in parallel for example? They are both build systems for userlands, or whole embedded linux systems. I cannot speak for OE as I don't really know it in detail. The idea behind ptxdist is executable documentation. If you use Open Source software for business critical industrial applications, having control over the source is an important thing. So our customers are usually able to reproduce their root filesystems with one ptxdist go directly from the sources. Our design criteria are - things are highly configurable - you have to find out what was done why - going the mainline way where ever possible - use proven technology (ptxdist is written in bash gnu make) I'm trying - real hard - to get an idea of how people out there prefer to build their root filesystems in cross-compiled environments. Well, I still havn't seen a real argument against it. At least not for our usecases. Rob Landley recently wrote up a small blurb on why native compilation is the way to go, and a small roadmap on how he intended to get there using e.g. Qemu and Firmware Linux, c.f: http://www.landley.net/code/firmware/about.html which finally won me over to that line of thinking. Debian and friends obviously go this way now. Cool, so what do we do for platforms like Blackfin, or any of the random ARM processors out there, write qemu support first? Sorry, you cannot really suggest this. Not with Qemu being in such a crude state that it can not even be built with modern compilers. Being able to be cross compiled is actually a *feature* of unix software, be it autotoolized or not. And, in reality, the problems which usually come up in these 'build native' discussions are very often academic. However, when it comes to the widespread and much fragmented ways of cross-compiling a rootfs, including the stuff put together by MontaVista, WR and all those animals in the forest, obviously based on RPM (build systems I haven't put my hands inside, since they are proprietary) there seems to be very little consensus. Well, the consensus is usually configure make make install. If people just wouldn't assume that they know it better than autotools ... But don't let us start a flamewar in this direction here. It doesn't bring us anywhere. ptxdist stands out but do you get a lot of outside contributions for it? As it looks it seems you're running it yourself. (Beware: I haven't looked close.) Oh, we have quite a bunch of contributions from the outside, and almost 200 people on the mailing list. It goes up and down, there are times where we get more contributions from the customers, then from the community, but in general it works really good. Note that it is a tool to solve *our* problems (which consist of making reproducable embedded linux systems for our customers, while being able to fix problems as they arise). And it solves them very, very well. What else is there out there for rootfs, really? A hack from every embedded company there is? PTXdist doesn't invent something new, it just exercises the usual configure make make install canon, in a reproducable way. Nothing more and nothing less. It is completely open, being GPL, so you can freely choose if you accept realizing it being on the way to world domination or not :-) It's not even focussed on building root filesystems. We also use it to build cross compilers and well defined Eclipse installations. I'm more after what people actually *use* and what is community driven here, not so much opinions on what is best (which will probably be the unwanted side effect of this mail anyway...) PTXdist is in use and community driven, so it migth be worth a look :-) rsc -- Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de Pengutronix - Linux Solutions for Science and Industry Handelsregister: Amtsgericht Hildesheim, HRA 2686 Hannoversche Str. 2, 31134 Hildesheim, Germany Phone: +49-5121-206917-0 | Fax: +49-5121-206917-9 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: building Rootfs
What else is there out there for rootfs, really? A hack from every embedded company there is? I'm more after what people actually *use* and what is community driven here, not so much opinions on what is best (which will probably be the unwanted side effect of this mail anyway...) Linus There are three issues in building an embedded cross-filesystem: 1. Identification of the files to put into the filesystem. 2. Stripping or extracting debug symbols from the filesystem. 3. Getting files into the filesystem when the development system does not allow root access. The first item is made a little easier by RPM, as it groups files together into packages, and package dependencies are quite often properly declared. However, packages sometimes have dependencies on another package when only part of the package may be required. Another problem is with documentation files (usually unwanted on an embedded system), and locales which are huge and generally unwanted. The second item can be tricky, as one doesn't want to strip debug symbols every time a filesystem is built, but the debug symbols are generally needed for cross debugging. debugedit from the RPM package does a nice job of this, separating the debug symbols from the executable, but it is best to do this when an cross executable is installed on the host system, rather than at filesystem build time. The third item is easy for an initramfs, but not well handled by existing tools for an initrd (although I haven't looked recently). One possibility is to modify a mkfs program, such as mke2fs, for the filesystem of choice to accept proto files, like the traditional Unix mkfs does. [Google mkfs proto for some examples]. I'm not sure if ptxdist supports both initramfs and initrd. If you do have root access, this is all much simpler, as an initrd can be locally mounted and populated. One other item that pops up sometimes is post-install scripts that are associated with some packages. These scripts run in the target filesystem, and there is no good way to run them in the host environment. This doesn't happen too often, so it can usually be handled by hand as an exception. One example of this type of item is installing the symbolic links in the right place for init scripts. Another option would be to capture the scripts and run them during boot, but this would slow boot times. Using the tools from an embedded vendor has two problems - they can change from release to release, and they are proprietary tools that lock you into the vendor. But, they may provide a GUI, if you are into graphical presentations. John Haller -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
