Upgrading your system with updatinator
I've set up an updatinator repository at http://olpc.download.redhat.com/updatinator/ which contains all devel_ext3 and devel_jffs2 from build 400 up to build 499. (The total repository size for this is about 2 gig.) It also has binary diffs for changed files between all consecutive releases, which should get faster downloads when going to the next version. I've tried this by doing live updates from/to various versions on a virtualized system and a single update (406-410) on a real XO. I will do more testing on the XO, but it seems to work well. I've created a tarball with everything you need to get this going on your laptop, including a script that drives it all. http://people.redhat.com/alexl/olpc/update-xo-070712.tar.gz Just untar this and run the upgrade-xo.sh script. Its interactive and will ask you for some extra info on the first run, and then you just type in the version you want to upgrade (or downgrade) to and it'll pull things down and upgrade. This should make it pretty easy to keep your machine uptodate. Just re-run update-xo.sh when there is a new release out. NOTE: The peer-to-peer part of updatinator (autoupdatinator) doesn't work at the moment, as I've done some changes to the repository layout and it hasn't been updated to handle that yet. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
updatinator benchmarking (Was: rsync benchmarking)
On Wed, 2007-06-27 at 16:02 -0400, C. Scott Ananian wrote: Anyway, here are some quick stats, for upgrades from build 464 to 465 to 466: I've been doing some more work on updatinator. Now it stores the release info as a simple key-value file that contains the manifest sha1 value, and the actual manifest is just a blob like the other. This way we automatically get gzip and bsdiff support for the manifest files (which tend to be around 2 meg for a full olpc devel image). Here are some stats using bsdiffs and gzip. I don't have any byte counter in the app, so i just looked at interface RX/TX stats. It counts the full packet size instead of the tcp stream byte count, and it might have data from other apps running on the system, but it should give a somewhat correct maximum limit at least. 464-to-465: Full contents of tree: 429,302,610 bytes rsync --whole-file: sent 32,034 bytes; received 13,153,813 bytes. Standard rsync: sent 96,720 bytes; received 10,047,524 bytes. 9.0s user, 64.9s real Rsync-with-manifest: sent 79,192 bytes; received 9,139,515 bytes. 4.3 user, 11.5s real. Updatinator: sent 596,681 bytes recieved 7,545,281 bytes 465-to-466: Full contents of tree: 458,828,206 bytes rsync --whole-file: sent 21,990 bytes; received 25,826,516 bytes. Standard rsync: sent 141,948 bytes; received 23,981,423 bytes. 14.0s user, 68.5s real Rsync-with-manifest: sent 125,158 bytes; received 23,171,430 bytes. 10.4 user, 28.5s real. Updatinator: sent 1,263,885 bytes recieved 20,718,744 bytes I haven't looked into why we're sending so much data, perhaps its due to other traffic on my box, or perhaps the http accesses are just too chatty. But even taking that into consideration we're doing 1.2 meg less total network traffic in the 465-466 case than the best rsync case (and we're not doing a lot of local disk i/o). I'm also creating an updatinator repository at http://olpc1.redhat.com/updatinator/ so that people can start test it, and eventually use it to easily update their XOs. Another interesting piece of stats here is the size of this repository. A repository with all the ext3 and jffs2 developer images from version 400 to 450 has a blobs directory of 1.3 gigs. The estimated size for just the tarballs of these revisions is around 12 gigs. This is a pretty good compression ratio. This is good for the school server disk use and means it can easily store all old releases. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Fri, 2007-06-29 at 08:22 -0400, Dan Williams wrote: Two questions here: 1) what does the scheme do in the case where the file it's about to replace on the local machine isn't the same as what the manifest on the local machine says? ie, local changes have changed the sha1 hash of the local file. That's essentially a policy decision, do we replace when there are local changes or don't we. A %config-type thing could work here to mark files that you may/may not always want to replace. At the moment it only touches the files that are affected by the diff in the manifest. If a file wasn't changed in the manifests we don't even look at it. However, if the file we're replacing has changed locally, we currently throw that away. 2) After downloading a blob, I assume the tool sha1s the downloaded file to ensure that it's content matches the name, right? Yeah, or rather, while downloading. We need a library to do http downloads. I see we're shipping libcurl on the laptop. Is it ok to use this library? Yeah, though the ironic thing just occurred to me that libcurl is (along with Mozilla) one of the more frequently updated pieces of a linux distro :) I ended up using urllib2 in python. (We also need a very simple http sever that maps sha1 - file in system image for laptop-to-laptop upgrades, but I think we can do that without using a library.) simple means lighttpd in my mind, but apache has a longer track record. simple means 300 lines of C code in serve-manifest.c... ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Fri, 2007-06-29 at 10:21 -0400, Dan Williams wrote: On Tue, 2007-06-19 at 14:39 +0200, Alexander Larsson wrote: The update daemon must provide a fair amount read-only status information before and during the update process to allow the GUI bits the flexibility to present that information to the user in the way it sees fit. auto-updatinator (the mDNS background downloader) emits dbus signals when there is a new version availible, and when it downloaded a new blob. The actual applying of the update once its done downloading should be triggered by something listening to that signal. Oh, and we should probably rate-limit or connection-limit the local HTTP server that's serving updates so we don't hammer one particular XO. This could mean putting an _advisory_ try-me=true/false flag in the mDNS advertisement that we switch depending on the current connection load. The server would obviously still boot users if there were too many connections, but at least using the 'try-me' gives other laptops more information about whether they should try to connect right away, or try a different XO that may advertise the same blob. Yeah. We currently only handle one file at a time in the local http server. However, some rate limiting on that sounds fair. At the moment we pick a new server to download each blob for randomly from the set of availible servers that has the version we want. Have you thought about what information the mDNS record would advertise? In the best case, it's as large of a UDP packet as we can push out. And since we can probably deal easily with packets up to the wireless MTU (which is something 2000 bytes), there's more room than the traditional 512 byte UDP packet. However, the space in the mDNS announcement is not unlimited, so I don't think we could put the entire blob list in there :) There's certainly enough room to put a build number, HTTP server port number, manifest hash, etc. At the moment we have image id, a list of image version ranges availible, a base directory for the http uri, a hint that the server has diffs and a prio hint which is unused atm. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Fri, 2007-06-29 at 15:49 +0200, Alexander Larsson wrote: I've got the code mostly working now, and I managed to update a qemu instance of build 406 to the devel build 406 using something like: ./updatinator.py -u http://10.x.y.z/updates --get-manifest olpc-ext3 406 ./updatinator.py -u http://10.x.y.z/updates -i -m manifests/olpc-ext3.406 --download olpc-devel-ext3 406 ./upgrade-manifest -b blobs/ -d / manifests/olpc-ext3.406 manifests/olpc-devel-ext3.406 Where http://10.x.y.z/updates is an apache server with manifests and blobs generated with generate-manifests from the -tree OLPC images. However, then things failed when I upgraded to olpc-devel-ext3.445, due to out of diskspace. I need to look into using gzip of blobs and removing blobs once they have been applied (this is harder than it sounds because blobs can be used multiple times in one manifest). I added --delete to upgrade-manifest that does delete-behind (handling multiple uses of the same hash in one upgrade). So now it should be possible to do the upgrade using less space. It worked when running on my local machine, but I haven't tried it in a qemu instance yet. However, I gotta leave now, and I won't be able to do any OLPC work at all next week. It would be cool if someone else could play a bit with this, test it a bit more, get the blobs on a public service, test on a real XO, etc. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: System update spec proposal
On Wed, 2007-06-27 at 17:43 -0400, Polychronis Ypodimatopoulos wrote: I wrote some code to deal with both problems, so that you have a qualitative hop count (offering real numbers instead of just integers for hop count) for all nodes in the mesh without broadcasting/multicasting any frames. The following snapshot only shows one neighbor with perfect reception. http://web.media.mit.edu/~ypod/teleporter/dump.png I can clean up the code and upload it somewhere. I would very much appreciate that. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Tue, 2007-06-26 at 17:47 -0400, Mike C. Fletcher wrote: I may be missing an operation or two in there, but I *think* that's the intention. That is, there's an ultimate loading level that sits outside the standard root to make it all work. That loading level could be in the BIOS or on an initrd, but there does need to be a level to manage the ultimate overlay-based structure AFAIK. And at normal boot, something has to set up the overlays, regardless of what overlay set is loaded. That level of access is where the update manager has to sit to be able to do the updates AFAICS. That is to accomplish a merge or remount of the core fs we need a way to ask the overlay manager to do some work for us at the update-manager level. Oh I see. So, you update the root, but you can't update all the system software (like the part that does the switching between roots). In such a setup, how do you update e.g. the kernel? (how does that work for soft links, incidentally, I gather you recreate them rather than hard-linking?) They are recreated, same with fifos, directories, and device nodes. At least this is how update-manifest in updatinator does it, but I don't think there is any other way really. With update transaction on the filesystem level (like my jffs2 proposal) this kind of outer manager is not needed. However, with an outer controller I can see this working. One could even use symlinks to make this pretty simple: /rootfs/olpc.5 [contains image version 5] /rootfs/olpc.6 [contains image version 6, shared hardlinks with olpc.5] /rootfs/current - olpc.6 [a symlink] /rootfs/previous - olpc.5 /usr - /rootfs/current/usr /var - /rootfs/current/var /etc - /rootfs/current/etc Then, to upgrade almost atomically, one just does: clone_w_hardlinks (/rootfs/olpc.6, /rootfs/olpc.7) apply_changes (/rootfs/olpc.7) ln -sf /rootfs/olpc.6 /rootfs/previous [1] ln -sf /rootfs/olpc.7 /rootfs/current [2] rm -rf /rootfs/olpc.5 A power failure during [1], or between 1 and 2 can mess up the previous link, and since symlink isn't an atomic operation if the file exists a failure during [2] can cause current to disappear. However, we will never have both previous and current missing. Using symlinks like this means the jffs2 parser in the firmware (if it supports symlinks) will even be able to pick up the right kernel. (Although it will always pick the current kernel.) There is one tricky area with hardlinks though. All hard links of the same inode share the permission and other metadata with each other. What if the trees have different metadata for the same file? Or what if running the current image changes metadata for a file shared between the images? Hmm, i guess any kind of in-place change to the hardlinked files while running the image is also mirrored in the other image. I guess this is where the COW stuff is needed... I guess this means the /usr - /rootfs/current/usr symlinks don't cut it, but one needs something heavier, like overlays or vserver+COW. Sad, it was an interesting idea. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: System update spec proposal
On Tue, 2007-06-26 at 13:55 -0400, Ivan Krstić wrote: Software updates on the One Laptop per Child's XO laptop First some stray comments: 1.4. Design note: rsync scalability --- rsync is a known CPU hog on the server side. It would be absolutely infeasible to support a very large number of users from a single rsync server. This is far less of a problem in our scenario for three reasons: What about CPU hogging on the school server? That seems likely to be far less beefy than the centralized server. The most up-to-date bundle for each activity in the set is accessed, and the first several kilobytes downloaded. Since bundles are simple ZIP files, the downloaded data will contain the ZIP file index which stores byte offsets for the constituent compressed files. The updater then locates the bundle manifest in each index and makes a HTTP request with the respective byte range to each bundle origin. At the end of this process, the updater has cheaply obtained a set of manifests of the files in all available activity updates. Zip files have the file index at the end of the file. Now for comments on the general approach: First of all, there seems to be exceptional amounts of confusion as to exactly how some form of atomic updating of the system will happen. Some people talk about overlays, others about vserver, I myself has thrown in the filesystem transaction idea. I must say this area seems very uncertain, and I worry that this will result in the implementation of none of these options... But anyway, the exact way these updates are applied is quite orthogonal to how you download the bits required for the update, or how to discover new updates. So far I've been mainly working on this part in order to avoid blocking on the confusion I mentioned above. As to using rsync for the file transfers. This seems worse than the trivial manifest + sha1-named files on http approach I've been working on, especially with the optional usage of bsdiff I just commited. We already know (have to know in fact, so we can strongly verify them) the contents on both the laptop and the target image. To drop all this knowledge and have rsync reconstruct it at runtime seems both a waste and a possible performance problem (e.g. cpu and memory overload on the school server, and rsync re-hashing files on the laptop using up battery). You talk about the time it takes to implement another approach, but its really quite simple, and I have most of it done already. The only hard part is the atomic applying of the bits. Also, there seems to be development needed for the rsync approach too, as there is e.g. no support for xattrs in the current protocol. I've got the code for discovering local instances of updgrades and downloading it already working. I'll try to make it do an actual (non-atomic, unsafe) upgrade of a XO this week. I have a general question on how this vserver/overlay/whatever system is supposed to handle system files that are not part of the system image, but still exist in the root file system. For instance, take /var/log/messages or /dev/log? Where are they stored? Are they mixed in with the other system files? If so, then rolling back to an older version will give you e.g. your old log files back. Also, that could be complicating the usage of rsync. If you use --delete then it would delete these files (as they are not on the server). Also, your document contains a lot of comments about what will be in FRS and not. Does this mean you're working on actually developing this system for FRS? ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: System update spec proposal
On Tue, 2007-06-26 at 15:03 -0400, Mike C. Fletcher wrote: My understanding here is that Alex's system is currently point-to-point, but that we don't yet have any way to distribute it on the mesh? That is, that we are looking at a research project to determine how to make it work in-the-field using some form of mesh-based discovery protocol that's going to try to optimise for connecting to local laptops. I personally don't care which way we distribute, but I'm wary of having to have some mesh-network-level hacking implemented to provide discovery of an update server. I'm not sure what you mean here exactly. Discovery is done using avahi, a well known protocol which we are already using in many places on the laptop. The actual downloading of file uses http, which is a well known protocol with many implementations. The only thing i'm missing atm is a way to tell which ip addresses to prefer downloading from since they are close. This information is already availible in the mesh routing tables in the network driver (and possibly the arp cache), and its just a question of getting this info and using it to drive what servers to pick for downloading. Basically aside from the vserver bits, which no one has seen, I don't see a particular advantage to using rsync. In fact, I see serious downsides since it misses some of the key critical advantages of using our own tool not the least of which is that we can make our tool do what we want and with rsync you're talking about changing the protocols. Hmm, interestingly I see using our own tool as a disadvantage, not a huge one, but a disadvantage nonetheless, in that we have more untested code on the system (and we already have a lot), and in this case, in a critical must-never-fail system. For instance, what happens if the user is never connected to another XO or school server, but only connects to a (non-mesh) WiFi network? Does the mesh-broadcast upgrade discovery protocol work in that case? Avahi works fine for these cases too. Of course, since it was originally created for normal networks. However, if you never come close to another OLPC machine, then we won't find a machine to upgrade against. Its quite trivial to make it pull from any http server on the net, but that has to be either polled (which I don't like) or initiated manually (which might be fine). ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Sun, 2007-06-24 at 13:24 -0400, C. Scott Ananian wrote: On 6/24/07, Ivan Krstić [EMAIL PROTECTED] wrote: I should have a concrete spec ready for discussion later today. I will wait with bated breath. =) Some concrete concerns -- I've got some answers to these, but I'll try to just present the questions at this point: a) Robustness -- what can break our upgrade mechanism? Do we have a fallback? What dependencies does the upgrade mechanism have? b) Current vserver code requires restarting the containers when files inside the container are modified by the root context. There is also a relinking process necessary. Have we thought through these interactions? I haven't really seen a viable way to use vserver for updating the root filesystem. Is there a proposal for how this would work? I had a different idea to do a very robust root filesystem update. Given the layout and format of jffs2 (the filesystem we're using) it is very easy to add filessystem transaction support. That is, we'd have a way to trigger a transaction at the start of an update, then we just do all the changes we want (including just updating the kernel image file). Then when the update is done, we commit the transaction. If at any time we lose power or anything like that, none of the changes after the transaction start will be visible on the next boot. And also, if we have any problems during update (e.g. out of flash) we an rollback and abort. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Upgrades and image manifests
Hi, My name is Alexander Larsson, and I just started working on the field upgrade system of the olpc laptops. I have some ideas I'd like to explain and get feedback on. The olpc uses a full-image system, as opposed to the per-package versioning scheme of deb or rpms. So, an upgrade consists of replacing your system files with a full copy of a later (or earlier) version of the system image. We want to support both upgrading from a central server or from another laptop, and we want to minimize the data we have to download for an upgrade. To support this we create a manifest that describes each image, and we save a copy of the current image manifest installed on the laptop. The manifest format is very similar to the git tree object, and describes each file by listing name, metadata and sha1 hash value of the contents. Here is a small example: - link 777 500:500 foo bar blob 664 500:500 4e1243bd22c66e76c2ba9eddc1f91394e57f9f83 changed.txt blob 664 500:500 65ef9261ad6cfaa8403f2fa66d8edfada1445e5a common blob 664 500:500 f660633c832712eba3c6edd39151987e46bc3f87 image1.txt blob 711 500:500 empty logfs.pdf dir 775 500:500 subdir1 blob 664 500:500 empty subdir1/file1.txt blob 664 500:500 7448d8798a4380162d4b56f9b452e2f6f9e24e7a subdir1/file2.txt --- Given a manifest for the current image and a manifest for the image to upgrade to it is very simple and cheap to calculate what changes you need to make to transform the image (on the client), and what sha1 blobs you need to download. You can then download the file blobs from whatever source you have (they are self-verifying, since they are named by the sha1 hash of the content) and do the upgrade. It is very simple to host the blobs. We just put them all in a directory named by sha1 and export that with a http server. One can also host multiple versions of an image in the same directory, while minimizing the space used for common data. (Possibly one might want to gzip the blobs too.) I've written some code (attached) to generate and manipulate manifests like these. There are three tools: * generate-manifest: This generates a manifest files given a path to a directory containing the image tree. You can also make it populate a directory of sha1-named blobs by giving it a blob directory with -b. * diff-manifest: Gives a simple visual diff between two manifests. If you pass it -b it will instead give a list of all blobs required to update between the two manifests. * upgrade-manifest: Updates an image from one manifest to another, given a path of a directory with the required blobs. Using these tools I upgraded from olpc-redhat-stream-development-build-406-20070507_2141-devel_ext3-tree to olpc-redhat-stream-development-build-406-20070507_2157-ext3-tree in a directory on my development machine, so it seems to work so far. For finding and downloading updates I was thinking of using avahi to publish the image id + version, and then downloading the data and manifests using http. That way that clients can detect machines around them with later versions and auto-download (and apply automatically or later). The individual laptops can publish the version of the image they are using, and a school server can publish several versions. From the point of view of the individual laptop searching for upgrades they will look the same. Of course, we'd have to sign the manifests with some key to make sure you can't just auto-upgrade any machine to a rouge image. Now over to the questions I have: I'd like to put this code in a repository somewhere. Where should I put it? Does OLPC use selinux or xattrs? Because if so we have to extend the manifest format. We need a library to do http downloads. I see we're shipping libcurl on the laptop. Is it ok to use this library? (We also need a very simple http sever that maps sha1 - file in system image for laptop-to-laptop upgrades, but I think we can do that without using a library.) Is using avahi/mDNS on the mesh network ok? manifest-0.1.tar.gz Description: application/compressed-tar ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
