Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Hello, On Sat, 20 Jun 2009 20:39:05 +0200 Jo-Philipp Wich x...@subsignal.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Malte. There's no reason why the unified format couldn't be a tgz (or some other, maybe uncompressed, format which carries a magic number) with some meta informations :) Daniel's proposal sounded quite complicated to me. So I finally took some time to try the .tgz approach on a classic Fonera. As it turned out, the space in /tmp is not enough to store both the tgz and the unpacked images at the same time. To solve that I basically invented some simple image format with a two byte magic value in the beginning, followed by a 8 byte hex sequence descriping the length of the kernel image, a 16 byte space-padded string containing the name of the kernel mtd partition, a 8 byte hex-string containing the length of the rootfs image and a space-padded 16 byte string containing the name of the rootfs partition. the following 65486 bytes are space-padded to make the initial header stuff 64KB aligned. This initial 64KB block is concatenated together with the kernel lzma image and the rootfs image. The reason for doing such an awkward format is, that I wanted to be able to extract the sub-images with busybox' dd utility. Second step was to implement a platform.sh backend for sysupgrade which uses dd to extract name and length information from the first 64KB of the combined image. After determining the length information, dd is used again to extract the sub-images from the correct offsets in the combined image. The output is piped to the mtd utility. The configuration keeping is implemented with mtd's -j option when writing the rootfs. So far it seemed to work but I have only tested refleshing with the same images, need to try images of different size later. I've attached my platform.sh script and another script to generate a suitable combined image. I have no idea whether they will work with some other device than the Fonera, maybe someone want to try this out. It works for me on a Nanostation5. I attach again the mkimg.sh script with the possibility to override partition names. Since the kernel partition name is kernel for Nanostations, I used : KERNEL_PART=kernel ./mkimg.sh openwrt-atheros-vmlinux.lzma openwrt-atheros-root.squashfs I tested it also with an image with different rootfs size, it seems ok. I think it will work with flash partitions as long as the new kernel is not one block bigger than the old one. If one wants to flash a bigger kernel he has to get rid of the dependency on the bootloader partition table information or be able to edit it (like Ubiquity appears to do with fwupdate). I think Felix's plan is welcome :) Thank you for the solution. fred mkimg.sh Description: application/shellscript ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
On Wed, 17 Jun 2009 23:08:17 +0200 Felix Fietkau n...@openwrt.org wrote: My plan for the unified format is to keep the on-flash format the same as the input format. In the long run, I don't want any unpacking to be done in sysupgrade or the web interface. The structure is roughly like this: [second stage loader][environment][kernel][rootfs] Ok, here's my question. The brcm-63xx *require* a Broadcom format imagetag at the start of the flash image (on flash and for CFE upgrades), and won't load without it. Do you mean to keep this imagetag and have the [second stage loader][environment][kernel][rootfs] all in the kernel partition (as far as the CFE is concerned; for the OpenWRT obviously the partitions would be determined by the environment) ? Also, how does the second stage loader compare to the kernel? I ask because I have run across a router for which the CFE verifies the CRC of the kernel after it has been decompressed. Would the unified format be a problem for this router? Also, when it comes to decompression, do the 63xx routers have the CFE do the decompression or is it a loader? I ask because I know that the kernel has to be compressed with specific lzma parameters for it to work with the CFE and that indicates to me that it's not an arbitrary loader that's doing the unpacking, but rather the CFE. Or do you run the loader *after* un-lzma by the CFE? -- And that's my crabbing done for the day. Got it out of the way early, now I have the rest of the afternoon to sniff fragrant tea-roses or strangle cute bunnies or something. -- Michael Devore GnuPG Key Fingerprint 86 F5 81 A5 D4 2E 1F 1C http://gnupg.org The C Shore (Daniel Dickinson's Website) http://www.bmts.com/~cshore signature.asc Description: PGP signature ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Hi All, You have some very good ideas. I would like to suggest using [requiring] BOTH md5sum and sha1. This is much harder to fake. I have already seen TWO DIFFERENT VERSIONS of the same wireless driver in the wild ??!! It seems that those of us who are successful [openwrt] attract a lot of vultures! Real time monitoring of the executable portion of running code would probably also be wise. In this way the processor can hopefully eventually notice that code has been patched or otherwise changed and cause a reboot. Write once devices look better and better. Comment: There are lots of smart people who want to join our party without a proper invitation :)). warm regards to all, Wiz (pen name) On Tue, 16 Jun 2009, Daniel Dickinson wrote: [NON-Text Body part not included] ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Hi, On Wednesday 17 June 2009 03:38:01 Daniel Dickinson wrote: No one seems to have responded to this, and I'd like to start working on it next week. If developers could think about this and comment, it would be appreciated. I'm not a core OpenWrt developer but was thinking about implementing sysupgrade for the RouterBoard 411. That board has two mtd partitions, one for the kernel and one for the rootfs, so the default way of flashing it with sysupgrade doesn't work. I think most of what you wrote isn't needed, especially the logic on how to write the image. Because that stuff is already included in the platform specific part of sysupgrade and is already included in the old firmware. But if it is neded (like for a new platform), it could be included in my idea as well. The ideas I had for implementing it for the RB411 were quite simple: The firmware image can be a standard .tar.gz; nothing fancy, no own format. A tar would be enough but that one doesn't have a recognizable magic number. The tarball contains a folder named firmware with a bunch of files. Each file is an image which has the same name as the corresponding mtd partition. To introduce md5sums, each file could be accompanied by an optional .md5 file, ie. on the RB411 we'd have /firmware/kernel /firmware/kernel.md5 /firmware/rootfs /firmware/rootfs.md5 A broken tgz header will be recognized by these tools already. If we need any logic for flashing, we can also introduce a sysupgrade dir. Anything in there will go to /lib/upgrade before flashing. For even more flexibility, the root could contain an optional (executable) sysupgrade.sh which called instead of the system sysupgrade. Those were the ideas I had. Cheers, Malte -- ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Malte. I like that .tar.gz idea, it would also cover devices like the Fonera where you have two separate images too. However, it's planned to introduce a unified image format for OpenWrt which would solve most of this issues so I haven't started working on it. But imho it would be great to have such a capability for the stable 8.09 series since it can be added as package on top of the normal code. The question is whether config-keeping is doable in such a way, have you already tried this (with mtd's -j option) ? Regards, JoW -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAko4+kwACgkQdputYINPTPP7bACfdmB2t1AVeYc7vqJ6qMZj3Hql 0nAAnirtOHblYeIrYFeQ4S/Ids7tj92w =qQPA -END PGP SIGNATURE- ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Hi JoW, On Wednesday 17 June 2009 16:14:37 Jo-Philipp Wich wrote: I like that .tar.gz idea, it would also cover devices like the Fonera where you have two separate images too. However, it's planned to introduce a unified image format for OpenWrt which would solve most of this issues so I haven't started working on it. Is this planned for some time or do you mean Daniel's proposal (which sounds like a new idea but I'm not subscribed to this list very long). There's no reason why the unified format couldn't be a tgz (or some other, maybe uncompressed, format which carries a magic number) with some meta informations :) Daniel's proposal sounded quite complicated to me. But imho it would be great to have such a capability for the stable 8.09 series since it can be added as package on top of the normal code. The question is whether config-keeping is doable in such a way, have you already tried this (with mtd's -j option) ? No, I haven't. The RB411 is just a testbed for my hacks so I don't care about recreating the config (and switched to 8.09 with backported packages anyway). Also, the wget2nand script from the ar71xx platform (which I used for reflashing) uses the default yffs formatted partition, so I don't think the mtd -j switch would work anyway (haven't tried it though). But if the rootfs is a jffs, I don't see while config-keeping shouldn't work (ok, so the tarball would need a symlink or something pointing to the rootfs). Cheers, Malte -- ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Malte == Malte S Stretz m...@apache.org writes: Hi, Malte A broken tgz header will be recognized by these tools already. Malte If we need any logic for flashing, we can also introduce a Malte sysupgrade dir. Anything in there will go to /lib/upgrade Malte before flashing. For even more flexibility, the root could Malte contain an optional (executable) sysupgrade.sh which called Malte instead of the system sysupgrade. Why not just use a standard format like the Debian .deb instead of inventing a custom format? A Debian .deb is nothing more than a tar.gz with some meta data (version number/description/dependencies/conflicts) which can be used to ensure you can only install the upgrade on compatible hw, and 4 scripts: - preinst, which runs before installation - posting, which runs after installation - prerm, which runs before removal (/upgrade) - postrm, which runs after removal (/upgrade) Busybox contains a dpkg implementation. I'm using such a setup, where the .deb will extract into something like /tmp/upgrade and the postinst script will copy the individual files to the correct mtd partitions. Alternatively you could use ipkg/opkg. -- Bye, Peter Korsgaard ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Hi, On Wednesday 17 June 2009 22:01:24 Peter Korsgaard wrote: Malte A broken tgz header will be recognized by these tools already. Malte If we need any logic for flashing, we can also introduce a Malte sysupgrade dir. Anything in there will go to /lib/upgrade Malte before flashing. For even more flexibility, the root could Malte contain an optional (executable) sysupgrade.sh which called Malte instead of the system sysupgrade. Why not just use a standard format like the Debian .deb instead of inventing a custom format? [...] Alternatively you could use ipkg/opkg. Indeed, I think this is an even better idea. M -- ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
Malte S. Stretz wrote: Hi JoW, On Wednesday 17 June 2009 16:14:37 Jo-Philipp Wich wrote: I like that .tar.gz idea, it would also cover devices like the Fonera where you have two separate images too. However, it's planned to introduce a unified image format for OpenWrt which would solve most of this issues so I haven't started working on it. Is this planned for some time or do you mean Daniel's proposal (which sounds like a new idea but I'm not subscribed to this list very long). There's no reason why the unified format couldn't be a tgz (or some other, maybe uncompressed, format which carries a magic number) with some meta informations :) Daniel's proposal sounded quite complicated to me. My plan for the unified format is to keep the on-flash format the same as the input format. In the long run, I don't want any unpacking to be done in sysupgrade or the web interface. The structure is roughly like this: [second stage loader][environment][kernel][rootfs] The second stage loader unpacks the kernel part using lzma, and the environment contains information about the split between kernel and rootfs, so that it no longer has to be reflected in the boot loader's partition split. Another useful aspect of the environment is that it can contain preloaded uci config changes for the image, so you can build preconfigured images for lots of nodes without repacking the kernel or the rootfs. I don't know when I'll be able to finish this, but I did start working on the loader part already (which is the biggest part of this) - Felix ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
[OpenWrt-Devel] Ping Re: RFC: LuCI Web Interface Image Format
No one seems to have responded to this, and I'd like to start working on it next week. If developers could think about this and comment, it would be appreciated. On Fri, 12 Jun 2009 18:37:42 -0400 Daniel Dickinson crazycsh...@gmail.com (by way of Daniel Dickinson crazycsh...@gmail.com wrote: I would like to suggest an improvement I would implement for the LuCI firmware update. Current the LuCI update is pretty basic; it calculates the md5sum of the image and if you want to make sure it transferred correclty you have to find the md5sums file and manual verify the sum. This is awkward and not very effective, and it also misses the class of error in which the image is corrupt on the host machine. I would therefore like to propose that all targets implement the creation of an additional firmware image that is targeted at upgrades of OpenWRT from OpenWRT (using LuCI). Currently the firmware images are all designed to be flashed using vendor methods, and upgrading from within OpenWRT is a poor cousin. I think we need a firmware image that can be used to verify the validity of the image before flashing it, that is unified for all routers supported by OpenWRT. We need: * An MD5SUM or SHA1SUM of the image (that is, the portion that will be writen to the flash or other target device), so that the validity of the image can be verified. * An MD5SUM or SHA1SUM of the header (to ensure none of the header information has been corrupted, since it will determine how we flash the device) * An indicator the the method and/or parameters needed to flash the image (since different routers probably have different requirements); for example 'mtd linux' * For boards that need it; the starting offset of the kernel, the length of the kernel, the starting offset of the rootfs, and the length of the rootfs * Immediately before the start of the rootfs (in the padding between kernel and rootfs) (i.e. at the tail end of the kernel mtd partition for flash devices), there should be a space for vendors to place data, that, if non-null, must match what is already on the router. This allows a vendor who is a service provider to create firmware images for specific clients, and to ensure they can't accidentally flash the wrong router with the wrong image. * In the same location should probably also be an indication of which router this is, so that flashing the wrong router can be prevented. * Some vendors may want a signed header (with a public/private key) so that web interface flashing can only be done with approved images (serial console/tftp flashing is of course still open for anything, but in the service provider scenario the case would never be opened so serial console flashing is not an issue). This header would not be present on the router after flashing (but the flash safety features between kernel and rootfs would), it is only to ensure that the image flashed is not corrupt and is valid for this router. Ideally the between kernel+rootfs portion of the image would also be generated for the vendor flash methods (e.g. CFE, TFTP, Vendor web interface) so that upgrades via LuCI have the safety features available before the first upgrade (i.e. once OpenWRT is on the router). I'd like to implement this for brcm63xx soon, so any comments would be appreciated. Thank you, Daniel -- And that's my crabbing done for the day. Got it out of the way early, now I have the rest of the afternoon to sniff fragrant tea-roses or strangle cute bunnies or something. -- Michael Devore GnuPG Key Fingerprint 86 F5 81 A5 D4 2E 1F 1C http://gnupg.org The C Shore (Daniel Dickinson's Website) http://www.bmts.com/~cshore signature.asc Description: PGP signature ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
