Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
On 12/21/09 23:19, Somebody in the thread at some point said: Hi Robert - Thanks for your reply. mode, so you can re-flash as often as you like. However, our use cases are probably different than yours (deeply embedded systems, which often don't even have removable stuff like SD or USB sticks). Right, some of what Qi proposes won't work on all systems, like SD boot where there is no SD card. But the core just load and boot heuristic should work almost anywhere. - special update mechanisms What do you mean with special? Hmm, there have been interesting items in the openmoko trees. For barebox, we took the DFU support, which was done in a device specific way, cleaned that up and made a generic command out of it: DFU is a special update mechanism which I believe is a bad idea. I know a lot of people are still putting out full rootfs images as updates, and for some platforms that are too resource-constrained that's all people can do. But for modern devices like ARM11+ and the kind of board they typically find themselves on with a network connection, these are fundamentally at the level of PC from a few years ago. Linux PCs then and now use packaged update systems to manage the software on the device. And they package both the kernel and the bootloader and track and update it like any other package, apply packagesets as transactions, etc. The correct approach I believe is to unify the bootloader (and kernel) update path with the rest of the system, all done from Linux alone. (Personally I used Fedora ARM port and RPM, but any distro and packagesystem like Debian workable on ARM would be fine). dfu /dev/self0(bootloader)sr,/dev/nand0.root.bb(root) You can specify the slots on the command line, not hardcoded. Whereas we reworked the interfaces, the core code was pretty interesting. So I think some items it would have been worth to be pushed into u-boot at the time it was written. Bearing in mind they could only update by DFU and with GTA01, there was no bootloader recovery mechanism if it failed, Our DFU scenario goes like press a button while booting goes into DFU mode, so you can re-flash as often as you like. However, our use cases are probably different than yours (deeply embedded systems, which often don't even have removable stuff like SD or USB sticks). The issue GTA01 faced was that you are updating the thing the button takes you to. If that goes south you have to bust out JTAG / OpenOCD and that is definitely not an end-user tool for a consumer product. In GTA02 a separate NOR was added to contain the bootloader behind the button which was not updatable in the field, that then caused trouble since the updatable NAND bootloaders moved on but that never did. It also acted as the third pole in the love triangle betweeen NAND U-Boot and Linux in the NAND ECC / BBT differences since it could only recover the NAND bootloader only with the NOR bootloader's fixed idea of what ECC and BBT looked like, no matter what we had done with updates to the NAND bootloader in the meanwhile (eg, move from soft to incompatible but faster hard ECC in Linux). So we were actually unable to migrate to hard ECC in Linux, which is an insane outcome of a broken system. In contrast if your chip supports it (iMX31 and s3c6410 do and Qi works with those) having your bootloader on some sectors of SD card is wonderfully simple and easy to dd in on a postinstall scriptlet of your bootloader package. In general, I like in-system techniques much better than card juggeling, because it fits better into automated environments like our RemoteLab, which does our automatic nightly tests. But that's surely a matter of the use case you have. Agreed. But consider this: if your bootloader is on SD, and your bootloader completely rejects to hold private state on the board (other than onetime individualization, eg MAC address), something awesome happens when you pop your SD card and put it in another board, it comes up like the previous board did, no ifs or buts. You can imagine the effect that has on production / test virgin board bringup. When you have seen this, you do not want to return to raw onboard NAND. The main lessons I took from that was the dollar and time value of removing the unnecessary features in U-Boot and especially the Openmoko tree of it: In barebox, we use Kconfig to configure things away; so removing unnecessary features is just a matter of 'make menuconfig'. That is good, but what I am suggesting is that - these things are definitively unnecessary, ie, they deserve permanent deselection - the config system leads to bootloader-binary-per-variant Hell Because Qi burns off all the peripheral support and leaves it to Linux, actually building in support for multiple boards and multiple variants is pretty lightweight. The CPU bringup is always the same, SDRAM bringup may vary slightly and kernel commandlines and paths, amount and
Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
Hi Andy, [is this the right set of lists to discuss these issues? It's not directly CELF related, but I don't know a better place for general project independend bootloader discussions] On Tue, Dec 22, 2009 at 08:22:27AM +, Andy Green wrote: DFU is a special update mechanism which I believe is a bad idea. For dedicated embedded system in the non-phone league, with small root filesystems, it works pretty well. I know a lot of people are still putting out full rootfs images as updates, and for some platforms that are too resource-constrained that's all people can do. Ah, ok, we don't do that. But for modern devices like ARM11+ and the kind of board they typically find themselves on with a network connection, these are fundamentally at the level of PC from a few years ago. Linux PCs then and now use packaged update systems to manage the software on the device. And they package both the kernel and the bootloader and track and update it like any other package, apply packagesets as transactions, etc. The correct approach I believe is to unify the bootloader (and kernel) update path with the rest of the system, all done from Linux alone. (Personally I used Fedora ARM port and RPM, but any distro and packagesystem like Debian workable on ARM would be fine). Until now, we are using the build it yourself approach with ptxdist, basically because of these reasons: - If something goes wrong, we want to be able to fix it, which means that we must be able to recompile everything. Having the source is no value by itself, if you are not able to build it. - Root filesystems are small; a complete rootfs for a typical industrial application with Linux, glibc, dbus and qt is about 20...30 MiB. - People don't change software that often, and if they do, it has to be made sure that it is absolutely well tested. Nobody wants to reboot their deeply embedded machine controller at the other end of the world if somehting goes wrong. We usually don't have an administrator who can interoperate if something goes wrong. - Customizability. We recently tried Debian on the Neo, and it is an absolute mess. About 2.5 minutes of boot time, a lot of flicker and almost no reactivity of the system. So for us, the question remains how to customize standard distributions in a reproducable way. So at least at the moment, I prefer ptxdist over a customized debian. But in general, I respect the argument why people want to use standard distributions (I know the pain to fix all the cross compiling issues). I just don't think that today's distributions are there yet. Most embedded systems I've seen so far which follow the strip-down-standard-distro pattern have been unreproducable for anyone but the original developer. So we were actually unable to migrate to hard ECC in Linux, which is an insane outcome of a broken system. In contrast if your chip supports it (iMX31 and s3c6410 do and Qi works with those) having your bootloader on some sectors of SD card is wonderfully simple and easy to dd in on a postinstall scriptlet of your bootloader package. Agreed. In general, I like in-system techniques much better than card juggeling, because it fits better into automated environments like our RemoteLab, which does our automatic nightly tests. But that's surely a matter of the use case you have. Agreed. But consider this: if your bootloader is on SD, and your bootloader completely rejects to hold private state on the board (other than onetime individualization, eg MAC address), something awesome happens when you pop your SD card and put it in another board, it comes up like the previous board did, no ifs or buts. You can imagine the effect that has on production / test virgin board bringup. When you have seen this, you do not want to return to raw onboard NAND. In general, I agree with you here (although I think the MAC address should be glued to the hardware and not change if you change SD cards - people will then copy it and you have the same MAC address twice). However, I think it's more developer friendly to have that no changable state as a policy than a design decision: during development, we quite often change for example the kernel command line (adding quiet or debug switches, boot from net/disk...). For delivery, we just make barebox + it's scripting environment one image and change that to r/o, if necessary. So you can get best of both worlds. In barebox, we use Kconfig to configure things away; so removing unnecessary features is just a matter of 'make menuconfig'. That is good, but what I am suggesting is that - these things are definitively unnecessary, ie, they deserve permanent deselection - the config system leads to bootloader-binary-per-variant Hell For us, the bootloader is not only something which is delivered with the product - that's one use case. But there is also quite a long time where lots of developers work with the board - and in that use
Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
On 12/22/09 11:12, Somebody in the thread at some point said: Hi Robert - (Personally I used Fedora ARM port and RPM, but any distro and packagesystem like Debian workable on ARM would be fine). Until now, we are using the build it yourself approach with ptxdist, basically because of these reasons: - If something goes wrong, we want to be able to fix it, which means that we must be able to recompile everything. Having the source is no value by itself, if you are not able to build it. Fedora provides a whole solution there, with the restriction it's designed for native build, not cross. That puts a limit on what you can target with it, which is why ARM11+. You can just go here http://fedoraproject.org/wiki/Architectures/ARM#Getting_Started_with_the_Fedora_ARM_Port Get a rootfs tarball, bring it up and build packages with the compiler and build stuff in there already, yum install other stuff, etc. Again SD rootfs makes that simple to consider and do, you can untar it from your PC them replug to your device and run it. - Root filesystems are small; a complete rootfs for a typical industrial application with Linux, glibc, dbus and qt is about 20...30 MiB. True, typical Fedora footprint for X and so on will be 200MB. However. The developer cost in using these distro packages is next to nothing, both acquiring them and their ongoing uplevelling when fc13 comes. Quality of the packages is better than I can make :-) and it's pretty much anything there is to want http://ftp.linux.org.uk/pub/linux/arm/fedora/pub/fedora/linux/releases/12/Everything/arm/os/Packages/ And again, if you have SD Card rootfs basis, smallest one of those now you can get in a shop is 512MB anyway. - People don't change software that often, and if they do, it has to be made sure that it is absolutely well tested. Nobody wants to reboot their deeply embedded machine controller at the other end of the world if somehting goes wrong. We usually don't have an administrator who can interoperate if something goes wrong. Packages only help any QA effort. You don't have to release every package build to your stable repo, a staged development / testing / stable repo scheme is simple. - Customizability. We recently tried Debian on the Neo, and it is an absolute mess. About 2.5 minutes of boot time, a lot of flicker and :-| Yeah. But I admired the effort they put in there. The CPU is too weak for that game though and the Glamo in there is the final insult. (Actually there is a good rage comic: Problem, GTA02?) almost no reactivity of the system. So for us, the question remains how to customize standard distributions in a reproducable way. Well, I can explain what I did with Fedora, we were up in Bash with ethernet usb gadget and sshd ready for action in 5s from cold on iMX31 SD Card boot. That's the stock Fedora bash, sshd, etc from their tarball. The main action is not to use the stock /sbin/init. (You can still use sysV init / upstart to bring services up and down even if init never got started since it's basically all sat there in /etc/init.d.) So at least at the moment, I prefer ptxdist over a customized debian. But in general, I respect the argument why people want to use standard distributions (I know the pain to fix all the cross compiling issues). I just don't think that today's distributions are there yet. Most embedded systems I've seen so far which follow the strip-down-standard-distro pattern have been unreproducable for anyone but the original developer. Right. Fedora is different though, there are no cross-built packages (although they do provide cross compilers, I use them for kernel and Qi builds) and if storage is sufficient, there's no need to strip anything out. Just nobble init. But consider this: if your bootloader is on SD, and your bootloader completely rejects to hold private state on the board (other than onetime individualization, eg MAC address), something awesome happens In general, I agree with you here (although I think the MAC address should be glued to the hardware and not change if you change SD cards - people will then copy it and you have the same MAC address twice). (That's what I meant by other than onetime individualization eg MAC address -- that stuff (only) should live on the board not the card) However, I think it's more developer friendly to have that no changable state as a policy than a design decision: during development, we quite often change for example the kernel command line (adding quiet or debug switches, boot from net/disk...). For delivery, we just make barebox + it's scripting environment one image and change that to r/o, if necessary. So you can get best of both worlds. Right. Qi supports this by allowing the boot source filesystem that holds the kernel to also contain an optional append text file. The text in this file, if present, is appended to the generated kernel
Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
Rob Landley wrote: However, if that's your minimum then you can't use the bootloader to re-flash the device, which is kind of handy. (It gives you an un-bricking fallback short of pulling out a jtag.) But doing that requires things like a network driver, TCP/IP stack, tftp implementation, serial driver, command line interpreter, and so on. And of course code to erase and write flash blocks for your flash chip du jour, plus knowledge of the flash layout. (In theory, said knowledge comes from parsing a device tree.) What a lot of bloat you added for such a basic requirement. :-) You don't need all that to unbrick. It's enough to have a serial/USB/network driver (choose one), obediently listen for a kernel to be sent for booting from RAM, and boot it. In the network case it can be a simple UDP protocol, or even raw ethernet. No TCP/IP, no TFTP, not even BOOTP (but it's a nice bonus), no command line interpreter (just a GPIO on board to boot into unbrick me mode :-), and most strikingly _no_ flash driver for flash chip du jour. To flash it you send a kernel to boot from RAM which is capable of flashing it. http://wiki.openmoko.org/wiki/Qi Looking at the screen shot there, you've got code to parse ext2 filesystems. What is your definition of minimal? Ew, ext2 doesn't even satisfy powerfail-during-kernel-upgrade safety. I agree it does beg the question of what is minimal. The proposal did explain quite well what Qi aims for: not duplicating lots of kernel drivers badly. If it succeeds in the area of flash writing, network drivers, network protocols and so on it would be no bad thing. One area for potential common ground among bootloaders could be to share the code for parsing filesystems. It'd be great to see that in a library shared by GRUB, Qi, U-boot and so on as it's not device specific at all and not particularly large, but not so trivial that it's good to have lots of clones. It's possible to boot without parsing filesystems, but that is one rather nice feature, and with the right filesystems it can make system updates powerfail-safe. Rationale for not providing a boot menu is you don't want to mess with video init. I don't think I've actually seen an embedded bootloader that messes with video, they do serial console instead, and you have a screen shot of serial console messages so apparently the serial driver part is there... In perspective, serial is usually quite simple. Output only serial is even simpler, though :-) -- Jamie -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
