Re: XO in-field upgrades
On Tue, 2007-06-26 at 17:47 -0400, Mike C. Fletcher wrote: I may be missing an operation or two in there, but I *think* that's the intention. That is, there's an ultimate loading level that sits outside the standard root to make it all work. That loading level could be in the BIOS or on an initrd, but there does need to be a level to manage the ultimate overlay-based structure AFAIK. And at normal boot, something has to set up the overlays, regardless of what overlay set is loaded. That level of access is where the update manager has to sit to be able to do the updates AFAICS. That is to accomplish a merge or remount of the core fs we need a way to ask the overlay manager to do some work for us at the update-manager level. Oh I see. So, you update the root, but you can't update all the system software (like the part that does the switching between roots). In such a setup, how do you update e.g. the kernel? (how does that work for soft links, incidentally, I gather you recreate them rather than hard-linking?) They are recreated, same with fifos, directories, and device nodes. At least this is how update-manifest in updatinator does it, but I don't think there is any other way really. With update transaction on the filesystem level (like my jffs2 proposal) this kind of outer manager is not needed. However, with an outer controller I can see this working. One could even use symlinks to make this pretty simple: /rootfs/olpc.5 [contains image version 5] /rootfs/olpc.6 [contains image version 6, shared hardlinks with olpc.5] /rootfs/current - olpc.6 [a symlink] /rootfs/previous - olpc.5 /usr - /rootfs/current/usr /var - /rootfs/current/var /etc - /rootfs/current/etc Then, to upgrade almost atomically, one just does: clone_w_hardlinks (/rootfs/olpc.6, /rootfs/olpc.7) apply_changes (/rootfs/olpc.7) ln -sf /rootfs/olpc.6 /rootfs/previous [1] ln -sf /rootfs/olpc.7 /rootfs/current [2] rm -rf /rootfs/olpc.5 A power failure during [1], or between 1 and 2 can mess up the previous link, and since symlink isn't an atomic operation if the file exists a failure during [2] can cause current to disappear. However, we will never have both previous and current missing. Using symlinks like this means the jffs2 parser in the firmware (if it supports symlinks) will even be able to pick up the right kernel. (Although it will always pick the current kernel.) There is one tricky area with hardlinks though. All hard links of the same inode share the permission and other metadata with each other. What if the trees have different metadata for the same file? Or what if running the current image changes metadata for a file shared between the images? Hmm, i guess any kind of in-place change to the hardlinked files while running the image is also mirrored in the other image. I guess this is where the COW stuff is needed... I guess this means the /usr - /rootfs/current/usr symlinks don't cut it, but one needs something heavier, like overlays or vserver+COW. Sad, it was an interesting idea. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Sun, 2007-06-24 at 13:24 -0400, C. Scott Ananian wrote: On 6/24/07, Ivan Krstić [EMAIL PROTECTED] wrote: I should have a concrete spec ready for discussion later today. I will wait with bated breath. =) Some concrete concerns -- I've got some answers to these, but I'll try to just present the questions at this point: a) Robustness -- what can break our upgrade mechanism? Do we have a fallback? What dependencies does the upgrade mechanism have? b) Current vserver code requires restarting the containers when files inside the container are modified by the root context. There is also a relinking process necessary. Have we thought through these interactions? I haven't really seen a viable way to use vserver for updating the root filesystem. Is there a proposal for how this would work? I had a different idea to do a very robust root filesystem update. Given the layout and format of jffs2 (the filesystem we're using) it is very easy to add filessystem transaction support. That is, we'd have a way to trigger a transaction at the start of an update, then we just do all the changes we want (including just updating the kernel image file). Then when the update is done, we commit the transaction. If at any time we lose power or anything like that, none of the changes after the transaction start will be visible on the next boot. And also, if we have any problems during update (e.g. out of flash) we an rollback and abort. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On 6/25/07, Christopher Blizzard [EMAIL PROTECTED] wrote: Most of the stuff that Alex has done is (carefully) independent of any vserver or container discussion. Specifically, the update system in question could be applied inside of a container as easy as it would be outside of the container. I not sure I agree that the filesystem portion of the implementation ought to be completely independent of the network portion. I think that networking desiderata are going to impact our choices among the various upgrade schemes, and I'd prefer to have a high level design for the whole thing before we get too attached to particular bits of code. As one example, since network reliability goes down quickly as message size increases, it seems (to me, at least) that our upgrade data messages should be made as small as possible. It's not clear to me that our current design minimizes upgrade transfer size. Re: broadcast, that's basically the same as any laptop exposing presence information. For _transmission_ of an update, it's an interesting question as to whether or not to use a multicast update kind of thing. Do laptops usually wake each other up to process presence information? [Hopefully not.] Should they do so for urgent security upgrades? [Hopefully.] Here's a draft proposal: We listen to multicast address foo:bar:xxx:xxx:n + 1 if we currently have version # n installed on the machine. Then we won't be woken up by our friends announcing that they have version n like we do, but we will be woken up as soon as someone gets version n+1. On 6/25 Alex L wrote: mDNS *is* multicast. But the blobs won't be exposed over mDNS, that is far to much data for a protocol like that. Really? Do we know that? What's a typical 0-day patch look like? Have we tried to see how few bits it could be squashed into? Binary diffs seem much less useful. They enforce a specific base version that you have to have around, and they enforce the direction of upgrade. This is *exactly* why we need to have the big picture view. My understanding is that we *are* expecting all laptops to have identical bases, that the upgrade propagation rate and upgrade (in)frequency will be sufficient that all laptops will be running the same version of the software (save for a few stragglers who go on a long trip for a few weeks), and that the vserver copy-on-write mechanism will be used to perform rollback (so that we only need to worry about the forward direction). I'm not saying that my assumptions are correct. But I feel that deciding file formats before we've come to a big picture consensus may be premature. If you can cheaply generate at runtime (on the client) a minimal diff between any two versions you can avoid storing unnecessary information, Not sure exactly what you're getting at here: that we transfer complete blobs over the network but store them on the XO as binary diffs? My working assumption is that network and storage costs on the XO should be minimized as much as possible. Transferring complete blobs fails on these grounds. --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Mon, 2007-06-25 at 12:25 -0400, C. Scott Ananian wrote: mDNS *is* multicast. But the blobs won't be exposed over mDNS, that is far to much data for a protocol like that. Really? Do we know that? What's a typical 0-day patch look like? Have we tried to see how few bits it could be squashed into? The broadcast just contains a product/version ID - doesn't have to include the entire update. No more expensive than the presence stuff we have today. Binary diffs seem much less useful. They enforce a specific base version that you have to have around, and they enforce the direction of upgrade. This is *exactly* why we need to have the big picture view. My understanding is that we *are* expecting all laptops to have identical bases, that the upgrade propagation rate and upgrade (in)frequency will be sufficient that all laptops will be running the same version of the software (save for a few stragglers who go on a long trip for a few weeks), and that the vserver copy-on-write mechanism will be used to perform rollback (so that we only need to worry about the forward direction). I think that what Alex here can do it either way. You could use the vserver copy on write stuff if you want to use the hammer of a filesystem or you could use the image code he has to move it back whether or not the copy on write stuff is there. Or even after you have committed via vserver. I strongly suggest that we make sure that we keep the ability to go both directions. It gives us a lot more flexibility down the road, and for the countries and our users as well. I'm not saying that my assumptions are correct. But I feel that deciding file formats before we've come to a big picture consensus may be premature. If you can cheaply generate at runtime (on the client) a minimal diff between any two versions you can avoid storing unnecessary information, Not sure exactly what you're getting at here: that we transfer complete blobs over the network but store them on the XO as binary diffs? My working assumption is that network and storage costs on the XO should be minimized as much as possible. Transferring complete blobs fails on these grounds. When you hear complete blobs can you describe what you mean? I suspect that you're thinking of something different than what Alex has actually implemented. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On 6/25/07, Christopher Blizzard [EMAIL PROTECTED] wrote: The broadcast just contains a product/version ID - doesn't have to include the entire update. No more expensive than the presence stuff we have today. You're misunderstanding me. My concern is with waking machines up by broadcasting this information. We don't wake up on presence, but we do want to wake up on (some, urgent) upgrades. I think that what Alex here can do it either way. You could use the vserver copy on write stuff if you want to use the hammer of a Again, you're misunderstanding me slightly. Vserver has *very odd* semantics for its copy-on-write and for writing inside containers. We need to sync up on that come up with a plan, or we run the risk of creating a useless tool. Binary diffs are also bidirectional, FWIW. When you hear complete blobs can you describe what you mean? I suspect that you're thinking of something different than what Alex has actually implemented. Quoting from https://hosted.fedoraproject.org/projects/updatinator/ : --- Each computer running Updatinator tracks a specific image id, and runs a specific version of that image. Whenever it finds a manifest for a later version of that image id (and that manifest is signed by the right gpg key) that manifest and the required blobs for updating to it is automatically downloaded. When the manifest and all required blobs are downloaded Updatinator sends a dbus signal so that the system may let the user apply the update (e.g. automatically, or by showing some ui asking the user if he wants to update). --- My next post will give some concrete #s justifing the use of binary diffs. --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
[ Fixing Alex's address. ] On Mon, 2007-06-25 at 14:58 -0400, C. Scott Ananian wrote: On 6/25/07, Christopher Blizzard [EMAIL PROTECTED] wrote: The broadcast just contains a product/version ID - doesn't have to include the entire update. No more expensive than the presence stuff we have today. You're misunderstanding me. My concern is with waking machines up by broadcasting this information. We don't wake up on presence, but we do want to wake up on (some, urgent) upgrades. That's going to be interesting, yeah. You would need to teach the wireless firmware about it? How about just checking on wakeup? Some kind of wake-on-lan signal? I think that what Alex here can do it either way. You could use the vserver copy on write stuff if you want to use the hammer of a Again, you're misunderstanding me slightly. Vserver has *very odd* semantics for its copy-on-write and for writing inside containers. We need to sync up on that come up with a plan, or we run the risk of creating a useless tool. Can you explain how they are odd? It sure would help everyone. Binary diffs are also bidirectional, FWIW. Yeah, but you always need both sets of information to be able to generate them. So you have to host full file + diff data if you want to host an update. The nice thing about Alex's system is that you only have to host the file data that you're using on your system instead of file + diff data. You end up using less space that way. If you want to downgrade, you have to get the files or use the vserver versions (maybe you could just use the old files handled by the CoW stuff to drive the update system - that might work pretty well!) When you hear complete blobs can you describe what you mean? I suspect that you're thinking of something different than what Alex has actually implemented. Quoting from https://hosted.fedoraproject.org/projects/updatinator/ : --- Each computer running Updatinator tracks a specific image id, and runs a specific version of that image. Whenever it finds a manifest for a later version of that image id (and that manifest is signed by the right gpg key) that manifest and the required blobs for updating to it is automatically downloaded. When the manifest and all required blobs are downloaded Updatinator sends a dbus signal so that the system may let the user apply the update (e.g. automatically, or by showing some ui asking the user if he wants to update). --- My next post will give some concrete #s justifing the use of binary diffs. --scott Keep in mind that those blobs he's talking about are just files. The only place where we would add binary diffs would be for individual files, not entire trees. So what we're downloading today is only the changed files, largely for the sake of expediency and what I describe above for the space savings. Speaking for myself I've never been opposed to use binary diffs. To be sure, all of my original ideas included them. But if I have to choose between having something that works today with full files and saves some space and adding the complexity of binary diffs later, I will use the former. I would love to hear what you have to say about numbers for binary diffs. (I believe that in earlier discussions with people who tried these systems before that any diff system that didn't understand elf was doomed to failure, but I am ready to be shown the money.) I think that both Alex and I are happy to listen to ideas here (esp about the vserver stuff that you mention!) but it's June 25th and we have to be pretty practical about what we can do between now and when we have to ship. The update system needs to be very simple, easy to test and easy to validate + understand. The key to that is using a simple design and simple implementation. Added complexity has a high bar for inclusion here. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Fwd: XO in-field upgrades
[grumble, sent this from the wrong email address.] -- Forwarded message -- Here are some quick numbers justifying the use of binary diffs as an upgrade delivery format. I just grabbed the latest debian security advisory for Etch, which happened to be: http://www.us.debian.org/security/2007/dsa-1320 on clamav. This upgrade appeared not atypical, but feel free to point out if it is widely uncharacteristic for a lightweight package (ie, not mozilla). I downloaded and unpacked the packages corresponding to the 'before' and 'after' versions of all the packages affected. For reference, those versions are 0.90.1-3etch2 and 0.90.1-3etch3. I then deleted the files in /usr/share/doc and /usr/share/man (neither of which 'should' have changed, but apparently they did) and /usr/lib/debug. I wrote a script to generate binary diffs using both xdiff and bsdiff for the files which changed. (I also munged slashes to dashes.) These files were: usr-bin-clamconf usr-bin-freshclamusr-lib-libclamav.so.2.0.1 usr-bin-clamdscan usr-bin-sigtool usr-sbin-clamav-milter usr-bin-clamscan usr-lib-libclamav.a usr-sbin-clamd For all files other than libclamav.a and libclamav.so.2.0.1, the binary diff was less than 164/225 bytes (xdiff/bsdiff). libclamav.a had a 3529/6571 byte diff, and libclamav.so.2.0.1 had a 13398/74461 byte diff. (Note larger diff because all the relocation table entries moved. Maybe a smarter diff is possible.) In comparison, the full libclamav.a blob is 567,140 bytes and libclamav.so.2.0.1 is 432,668 bytes. Details: blob: 17168 usr/bin/clamconf 33120 usr/bin/clamdscan 49600 usr/bin/clamscan 65584 usr/bin/freshclam 77868 usr/bin/sigtool 567140 usr/lib/libclamav.a 432668 usr/lib/libclamav.so.2.0.1 76260 usr/sbin/clamav-milter 49948 usr/sbin/clamd bdiff: 158 usr-bin-clamconf 158 usr-bin-clamdscan 161 usr-bin-clamscan 159 usr-bin-freshclam 162 usr-bin-sigtool 3529 usr-lib-libclamav.a 13398 usr-lib-libclamav.so.2.0.1 164 usr-sbin-clamav-milter 162 usr-sbin-clamd Total bsdiff size: 18,051 Total raw size: 1,369,356 bytes. (75x larger) Further, although my goal of single packet upgrades seems overoptimistic, all but two of the file diffs can fit in a single packet, helping assure atomicity. The total upgrade information fits in 13 packets. To my mind, this makes it clear that the on-wire format should be a binary diff. We could uncompress these diffs on receipt and maintain a blob store as in the current proposal, but it seems much more reasonable to sign, authenticate, and store the *diffs* rather than the *blobs*, so that we don't need to recompute diffs when we want to pass on our upgrade to a neighboring XO. --scott -- ( http://cscott.net/ ) -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
C. Scott Ananian wrote: On 6/25/07, Christopher Blizzard [EMAIL PROTECTED] wrote: That's going to be interesting, yeah. You would need to teach the wireless firmware about it? How about just checking on wakeup? Some kind of wake-on-lan signal? Binding upgrade notifications to a multicast address as I previously proposed fixes this problem without any kind of firmware hacking. Can you explain how they are odd? It sure would help everyone. Caveat: I'm not an expert here. I haven't read the code, just the documentation. So we can all follow along, start here: http://linux-vserver.org/Paper#Unification http://linux-vserver.org/Frequently_Asked_Questions#What_is_vhashify.3F Basically, copy-on-write works by running a tool ('vhashify') which looks for identical files in the different containers and hard links them together, then marks them immutable. The copy-on-write mechanism works by intercepting writes to immutable files and cloning the file before making it writable by the container. It is worth noting we are not using vhashify or any of the other util scripts. The rainbow daemon sets up the chroot for each activity itself. We are a bit non-standard in that we are doing process-level containerization, instead of a more guest-OS system like many vserver users (most?). --Noah signature.asc Description: OpenPGP digital signature ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Mon, 2007-06-25 at 15:35 -0400, C. Scott Ananian wrote: On 6/25/07, Christopher Blizzard [EMAIL PROTECTED] wrote: That's going to be interesting, yeah. You would need to teach the wireless firmware about it? How about just checking on wakeup? Some kind of wake-on-lan signal? Binding upgrade notifications to a multicast address as I previously proposed fixes this problem without any kind of firmware hacking. Ahh, sorry, I thought you meant _really_ asleep - like not waking up on network events. Although does our independent firmware know enough to wake us up on multicast traffic? I thought that it only worked on the lower level protocols and that a packet had to be specifically destined for our MAC address to get a wake up event. I'll show my ignorance of multicast here: does it include specific MAC addresses or is it a wide broadcast at that layer? I always assumed that it was the latter. Can you explain how they are odd? It sure would help everyone. Caveat: I'm not an expert here. I haven't read the code, just the documentation. So we can all follow along, start here: http://linux-vserver.org/Paper#Unification http://linux-vserver.org/Frequently_Asked_Questions#What_is_vhashify.3F Basically, copy-on-write works by running a tool ('vhashify') which looks for identical files in the different containers and hard links them together, then marks them immutable. The copy-on-write mechanism works by intercepting writes to immutable files and cloning the file before making it writable by the container. Quoting from their FAQ: (when running vhashify:) The guest needs to be running because vhashify tries to figure out what files not to hashify by calling the package manager of the guest via vserver enter. In order for the OS cache to benefit from the hardlinking, you'll have to restart the vservers. Holy crap, this sounds like a steaming _pile_ of complexity. Are we seriously going to try to deploy on this? Since vserver is doing file hashification anyway, it seems like it would be a much better idea to use this rather than reinvent the wheel. Some other issues: a) we may need to be aware of which files are hardlinked where in order to do proper updates. b) not clear how/if we can make updates to an entire tree atomically c) how/can we restart the vservers? how important is this? I think we need to bring in a vserver expert (ie, not me) to get these details right at the start. Am happy to get more advice on this, for sure. I suspect that all of the vserver people we can call on are on this list. Our current thinking basically is that we can do an update as part of an update/shutdown procedure. So you can apply the updates on the way down, get a new env on restart. That would handle the vserver restarts and also how to get a new kernel issue that no one else has mentioned. I'm not sure if hashing for updates and hashing for vserver are the kinds of things we want to share or not. I would love to hear more about how vserver does its hashing and see if we can share. I still feel that keeping the update system as simple and uncomplex as possible is a very good way to go - it lets us advance in parallel and come up with something that works well. It also sounds like it's pretty easy to do something like: o Start root container o Start guest container o Apply update o Start activities in that guest container And the hardlink/CoW stuff will then give us an updated container. Still doesn't help with updates on the base system nor kernel bits, but it's a start. Yeah, but you always need both sets of information to be able to generate them. So you have to host full file + diff data if you want to host an update. My proposal would be that XOs pass around binary diffs *only* among themselves, and that if someone needs an older version or to leapfrog versions, they ask the school server. This allows XOs to cheaply host updates in the common case. You could do that with Alex's system as well. But in Alex's case the XO doesn't have to carry both the system it's using + diff. Because the system you're using is the update. The nice thing about Alex's system is that you only have to host the file data that you're using on your system instead of file + diff data. You end up using less space that way. If you look at the numbers I just posted, file+diff is 1.3% larger than just files. If you want to downgrade, you have to get the files or use the vserver versions (maybe you could just use the old files handled by the CoW stuff to drive the update system - that might work pretty well!) Now we're talking! ;-) Keep in mind that those blobs he's talking about are just files. The only place where we would add binary diffs would be for individual files, not entire trees. So what we're downloading today is only the changed files, largely for the sake of expediency and what I describe
Re: XO in-field upgrades
You're misunderstanding me. My concern is with waking machines up by broadcasting this information. We don't wake up on presence, but we do want to wake up on (some, urgent) upgrades. That's going to be interesting, yeah. You would need to teach the wireless firmware about it? How about just checking on wakeup? Some kind of wake-on-lan signal? In general, we won't be waking up the host on broadcast/multicast frames. We can make an exception for anycast addresses (that the firmware listens for), however I can't see a scenario where we wake up on broadcast. M. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Mon, 2007-06-25 at 15:56 -0400, Noah Kantrowitz wrote: Design docs? We're still at the proof-of-concept phase really ;-). But yes, each chroot needs to be generated on the fly when a new activity starts (unless we do some funky magic with unionfs, which is probably not a great idea). The load of a few directory hardlinks should be minimal. Are we expecting to do online updating or will it be more of a windows-style shut it all down then patch? I think that for phase one we can do it during a shutdown/restart cycle. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Fwd: XO in-field upgrades
On 6/25/07, Christopher Blizzard [EMAIL PROTECTED] wrote: On Mon, 2007-06-25 at 15:38 -0400, C. Scott Ananian wrote: To my mind, this makes it clear that the on-wire format should be a binary diff. We could uncompress these diffs on receipt and maintain a blob store as in the current proposal, but it seems much more reasonable to sign, authenticate, and store the *diffs* rather than the *blobs*, so that we don't need to recompute diffs when we want to pass on our upgrade to a neighboring XO. It's not clear what that data means, exactly. Do those files like this: 158 usr-bin-clamconf 158 usr-bin-clamdscan 161 usr-bin-clamscan 159 usr-bin-freshclam 162 usr-bin-sigtool Actually include binary files changes or are they just permissions changes? If so you're 75x larger charge seems a little over the top since they wouldn't be transferred at all in Alex's case outside of the manifest. :) No, they are actually 4-byte changes to each binary, buried in the linker-generated section, presumably caused by an offset change in the library. I stared at this and the patches for a while. You're correct that most of the size of this diff is spent coding the filename, but there is a real diff there. There's a 60x change if you just look at the libraries. So this savings is definitely real. That being said, this is pretty compelling stuff. I would love to see this done against various versions of libgklayout.so, which will probably be our largest offender. If I were bored, I might indulge you. ;-) It also might be interesting to layer a diff system on top of what Alex has proposed so you could do it either way - diffs if ya got 'em, raw if ya don't. That sounds suspiciously like the complexity you're afraid of. Let's just do diffs and call it done. --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
C. Scott Ananian wrote: Has this already been discussed in more depth off-list? To some degree. I should have a concrete spec ready for discussion later today. -- Ivan Krstić [EMAIL PROTECTED] | GPG: 0x147C722D ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On 6/24/07, Ivan Krstić [EMAIL PROTECTED] wrote: I should have a concrete spec ready for discussion later today. I will wait with bated breath. =) Some concrete concerns -- I've got some answers to these, but I'll try to just present the questions at this point: a) Robustness -- what can break our upgrade mechanism? Do we have a fallback? What dependencies does the upgrade mechanism have? b) Current vserver code requires restarting the containers when files inside the container are modified by the root context. There is also a relinking process necessary. Have we thought through these interactions? c) How do XOs know there is an upgrade available? Does this query/notification work both with and without a school server? d) We can't afford to have all the XOs contact Cambridge directly to get updates. Likewise, if the XOs get all their updates from the school server, the machines closest to the mesh portal will be passing the same bits around redundantly. Can we use mesh topography to improve distribution? e) How small/fast can we make an upgrade, to patch (say) a specific 0-day in one of our applications? If we can fit a patch into a single network packet, our distribution success probability should increase dramatically. f) Does our mechanism scale gracefully for both small (0-day security) and large (FC7-to-FC8) upgrades? g) Related to the above three: I've been told that the XOs can't possibly afford the space to store upgrades in order to be able to provide them to neighbors. I'm not sure I agree with this, but this concern should be addressed. h) What happens if we need to patch the kernel to fix a security problem? What if we need to patch firmware? i) What's the development process like? Can we easily create and test upgrades? j) Is the base system upgrade mechanism related in any way to the activity upgrade mechanism? If so, activity upgrades have a whole 'nuther set of concerns. If not, how can we avoid reinventing the same wheels (esp. with regard to distributing an activity upgrade efficiently to a classroom full of machines). Since I can't resist a few stabs at answers: a) I'd like to see two independent upgrade mechanisms available, one of them as simple as practical and hard-coded into firmware. d) If upgrades are broadcast to the mesh from the portal, the flood fill will ensure the bits efficiently reach most of the machines. I hope that listing these issues is helpful. --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel