Re: System update spec proposal
On Tue, 2007-06-26 at 14:21 -0400, Christopher Blizzard wrote: First about approach: you should have given this feedback earlier rather than later since Alex has been off working on an implementation and if you're not giving feedback early then you're wasting Alex's time. Also I would have appreciated it if you had given direct feedback to Alex instead of just dropping your own proposal from space. It's a crappy thing to do. I owe Ivan an apology here. I should have handled this in private email instead of on a public mailing list. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: SW Dev meeting minutes, 6/26/07
On Tue, 2007-06-26 at 22:14 -0400, Kim Quirk wrote: * Ivan discussed the activation feature - it fell behind as we are trying to sort out the updates; but he believes he can still get a minimum activation feature into the release over the next few days. He will touch base with J5 and Blizzard on changes that are coming or might be done in userland init so that won't affect activation. He is also waiting on hardware to test crypto and keys. Hopefully he and Mitch will make progress on this next week when Mitch is at 1CC. I thought that activation was located entirely in firmware and there wouldn't be any changes to the OS to support activation? --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: System update spec proposal
On Wed, 2007-06-27 at 12:26 -0400, Mike C. Fletcher wrote: That said, I would be more comfortable if the fallback included a way for the laptop to check a well-known machine every X period (e.g. in Ivan's proposal) and if there's no locally discovered source, use a publicly available source as the HTTP source by default I think that you're talking about the mail from Scott, not Ivan. And I think that we'll do something like that, yeah. That's one of the easiest parts of the update system. (Also one of the worst mistakes if you get it wrong ala the Hour of Terror: http://www.justdave.net/dave/2005/05/01/one-hour-of-terror/ but that's beside the point.) --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: System update spec proposal
On Wed, 2007-06-27 at 12:39 -0400, Mike C. Fletcher wrote: Could we get a summary of what the problem is: The main objection to vserver from all the kernel hackers (and those of us that have to support them!) is that it's a huge patch that touches core kernel bits and it has no plans to make it upstream. Yes, it's used in a lot of interesting places successfully, but that doesn't mean it's a supportable-over-the-long-term solution. Scale has nothing to do with long term supportability. And these laptops have to be supported for at least 5 years. This isn't a new discussion; it's been going on for months and months. Just quietly, that's all. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: System update spec proposal
On Wed, 2007-06-27 at 17:31 +0200, Alexander Larsson wrote: I have a general question on how this vserver/overlay/whatever system is supposed to handle system files that are not part of the system image, but still exist in the root file system. For instance, take /var/log/messages or /dev/log? Where are they stored? Are they mixed in with the other system files? If so, then rolling back to an older version will give you e.g. your old log files back. Also, that could be complicating the usage of rsync. If you use --delete then it would delete these files (as they are not on the server). Just a note about these particular files. I don't think that on the final version that we're going to be running a kernel or syslog daemon. We're running them right now because they are useful for debugging but I don't want those out there in the field taking up memory and writing to the flash when they don't have to be. I suspect that for most users they will have very little use. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: mesh network vs broadcast/multicast question
On Wed, 2007-06-27 at 11:22 -0400, Michail Bletsas wrote: The mesh TTL field will eventually be tunable on a per packet basis. Yeah, and we really want that. What's left to make that happen? (Frankly, I thought it was done already.) --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: A different proposal for XO upgrade.
Just some comments on this thread. It seems odd to try to optimize the bandwith on the actual local lan as we have a decent amount of bandwith to work with. The only use case that I can come up with to support that is during unboxing and/or a mass re-install. And I don't think that we're ready to try and support that with this first iteration. It's a distribution method and probably something that we can add after the fact. (I know that some Red Hat guys did something like this for a customer where an entire bank's set of terminals could be completely re-imaged after a power failure in 20 seconds using a mutlicast-rsync setup. I should see more about that.) As long as the formats that we pick support something like this we should be pretty safe for now. The case where we do worry about bandwidth is the client - main updates server setup. That's where the bandwidth is likely very slow and expensive and looking at using a diff-style system is worth the investment given our install base. I think Alex is looking into this now. I like the system that scott proposed for how often we should look at updates and the idea of a lead to say hey, I'm getting this update so don't look. (It sounds strangely familiar to an idea that I shared with scott over lunch a month or so ago so of course I like it!) We also might consider setting up other clients to start collecting the update before it's completely finished downloading to start spreading around the bandwidth over a longer period of time and making the entire process more fault tolerant and bandwidth efficient on the local lan. i.e. someone else could pick up the rest of the update if the lead machine vanishes. Also, two machines might be able to get the update faster than one, just due to latency over a lot of the links we're talking about using. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: System update spec proposal
A few notes follow here. First about approach: you should have given this feedback earlier rather than later since Alex has been off working on an implementation and if you're not giving feedback early then you're wasting Alex's time. Also I would have appreciated it if you had given direct feedback to Alex instead of just dropping your own proposal from space. It's a crappy thing to do. So notes on the proposal: 1. There's a lot in here about vserver + updates and all of that is fine. But we've been pretty careful in our proposals to point out that how you get to the bits to the box is different than how they are applied. I don't see anything in here around vserver that couldn't use alex's system instead of rsync. So there's no added value there. 2. rsync is a huge hammer in this case. In fact, I think it's too much of a hammer. We've used it in the past ourselves for these image update systems over the last few years (see also: stateless linux) and it always made things pretty hard. Because you have to use lots random exceptions during its execution and once it starts you can't really control what it does. It's good for moving live image to live image, but I wouldn't really want to use it for an image update system - especially one that will be as distributed as this. Simply put I see rsync as more of a tool for sysadmins than for a task like this. I think that we need something that's actually designed to solve the problems at hand rather than seeing the hammer we have on the shelf and thinking that it's obviously the right solution. 3. It doesn't really solve the scaling + bandwith problems in the same way as Alex's tool does. Still requires a server and doesn't let you propagate changes out to servers as easily as his code does. Basically aside from the vserver bits, which no one has seen, I don't see a particular advantage to using rsync. In fact, I see serious downsides since it misses some of the key critical advantages of using our own tool not the least of which is that we can make our tool do what we want and with rsync you're talking about changing the protocols. Anyway, I'll let Alex respond with more technical points if he chooses to. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: System update spec proposal
On Tue, 2007-06-26 at 18:50 -0400, C. Scott Ananian wrote: On 6/26/07, Christopher Blizzard [EMAIL PROTECTED] wrote: A note about the history of using rsync. We used rsync as the basis for a lot of the Stateless Linux work that we did a few years ago. That approach (although using LVM snapshots instead of CoW snapshots) basically did exactly what you've proposed here. And we used to kill servers all the time with only a handful of clients. Other people report that it's easy to take out other servers using rsync. It's pretty fragile and it doesn't scale well to entire filesystem updates. That's just based on our experience of building systems like what you're suggesting here and how we got to where we are today. I can try to get some benchmark numbers to validate this one way or the other. My understanding is that rsync is a memory hog because it builds the complete list of filenames to sync before doing anything. 'Killing servers' would be their running out of memory. Rsync 3.0 claims to fix this problem, which may also be mitigated by the relatively small scale of our use: my laptop's debian/unstable build has 1,345,731 files. Rsync documents using 100 bytes per file, so that's 100M of core required. Not hard to see that 10 clients or so would tax a machine with 1G main memory. In contrast, XO build 465 has 23,327 files: ~2M of memory. 100 kids simultaneously updating equals 2G of memory, which is within our specs for the school server. Almost two orders of magnitude fewer files for the XO vs. a 'standard' distribution ought to fix the scaling problem, even without moving to rsync 3.0. --scott I think that in our case it wasn't just memory it was also seeking all over the disk. We could probably solve that easily by stuffing the entire image into memory (and it will fit, easily) but your comment serves to prove another point: that firing up a program that has to do a lot of computation every time a client connects is something that's deeply wrong. And that's just for the central server. Also, 2G of memory on the school server is nice - as long as you don't expect to do anything else. Or as long as you don't want to do what I mention above and shove everything into memory to avoid the thrashing problem. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Mon, 2007-06-25 at 12:25 -0400, C. Scott Ananian wrote: mDNS *is* multicast. But the blobs won't be exposed over mDNS, that is far to much data for a protocol like that. Really? Do we know that? What's a typical 0-day patch look like? Have we tried to see how few bits it could be squashed into? The broadcast just contains a product/version ID - doesn't have to include the entire update. No more expensive than the presence stuff we have today. Binary diffs seem much less useful. They enforce a specific base version that you have to have around, and they enforce the direction of upgrade. This is *exactly* why we need to have the big picture view. My understanding is that we *are* expecting all laptops to have identical bases, that the upgrade propagation rate and upgrade (in)frequency will be sufficient that all laptops will be running the same version of the software (save for a few stragglers who go on a long trip for a few weeks), and that the vserver copy-on-write mechanism will be used to perform rollback (so that we only need to worry about the forward direction). I think that what Alex here can do it either way. You could use the vserver copy on write stuff if you want to use the hammer of a filesystem or you could use the image code he has to move it back whether or not the copy on write stuff is there. Or even after you have committed via vserver. I strongly suggest that we make sure that we keep the ability to go both directions. It gives us a lot more flexibility down the road, and for the countries and our users as well. I'm not saying that my assumptions are correct. But I feel that deciding file formats before we've come to a big picture consensus may be premature. If you can cheaply generate at runtime (on the client) a minimal diff between any two versions you can avoid storing unnecessary information, Not sure exactly what you're getting at here: that we transfer complete blobs over the network but store them on the XO as binary diffs? My working assumption is that network and storage costs on the XO should be minimized as much as possible. Transferring complete blobs fails on these grounds. When you hear complete blobs can you describe what you mean? I suspect that you're thinking of something different than what Alex has actually implemented. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
[ Fixing Alex's address. ] On Mon, 2007-06-25 at 14:58 -0400, C. Scott Ananian wrote: On 6/25/07, Christopher Blizzard [EMAIL PROTECTED] wrote: The broadcast just contains a product/version ID - doesn't have to include the entire update. No more expensive than the presence stuff we have today. You're misunderstanding me. My concern is with waking machines up by broadcasting this information. We don't wake up on presence, but we do want to wake up on (some, urgent) upgrades. That's going to be interesting, yeah. You would need to teach the wireless firmware about it? How about just checking on wakeup? Some kind of wake-on-lan signal? I think that what Alex here can do it either way. You could use the vserver copy on write stuff if you want to use the hammer of a Again, you're misunderstanding me slightly. Vserver has *very odd* semantics for its copy-on-write and for writing inside containers. We need to sync up on that come up with a plan, or we run the risk of creating a useless tool. Can you explain how they are odd? It sure would help everyone. Binary diffs are also bidirectional, FWIW. Yeah, but you always need both sets of information to be able to generate them. So you have to host full file + diff data if you want to host an update. The nice thing about Alex's system is that you only have to host the file data that you're using on your system instead of file + diff data. You end up using less space that way. If you want to downgrade, you have to get the files or use the vserver versions (maybe you could just use the old files handled by the CoW stuff to drive the update system - that might work pretty well!) When you hear complete blobs can you describe what you mean? I suspect that you're thinking of something different than what Alex has actually implemented. Quoting from https://hosted.fedoraproject.org/projects/updatinator/ : --- Each computer running Updatinator tracks a specific image id, and runs a specific version of that image. Whenever it finds a manifest for a later version of that image id (and that manifest is signed by the right gpg key) that manifest and the required blobs for updating to it is automatically downloaded. When the manifest and all required blobs are downloaded Updatinator sends a dbus signal so that the system may let the user apply the update (e.g. automatically, or by showing some ui asking the user if he wants to update). --- My next post will give some concrete #s justifing the use of binary diffs. --scott Keep in mind that those blobs he's talking about are just files. The only place where we would add binary diffs would be for individual files, not entire trees. So what we're downloading today is only the changed files, largely for the sake of expediency and what I describe above for the space savings. Speaking for myself I've never been opposed to use binary diffs. To be sure, all of my original ideas included them. But if I have to choose between having something that works today with full files and saves some space and adding the complexity of binary diffs later, I will use the former. I would love to hear what you have to say about numbers for binary diffs. (I believe that in earlier discussions with people who tried these systems before that any diff system that didn't understand elf was doomed to failure, but I am ready to be shown the money.) I think that both Alex and I are happy to listen to ideas here (esp about the vserver stuff that you mention!) but it's June 25th and we have to be pretty practical about what we can do between now and when we have to ship. The update system needs to be very simple, easy to test and easy to validate + understand. The key to that is using a simple design and simple implementation. Added complexity has a high bar for inclusion here. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Mon, 2007-06-25 at 15:45 +0100, David Woodhouse wrote: On Tue, 2007-06-19 at 14:39 +0200, Alexander Larsson wrote: Does OLPC use selinux or xattrs? Because if so we have to extend the manifest format. Not yet, but it's likely to in the near future when we ditch the short-term hacks and manage to implement the proper long-term security plan. So it's worth designing for it from the start. Yeah, I'm pretty sure that want to have support for at least describing xattrs (especially if we want to use this for regular filesystems as well - lots of people are using xattrs for all kinds of things.) --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Mon, 2007-06-25 at 15:35 -0400, C. Scott Ananian wrote:
On 6/25/07, Christopher Blizzard [EMAIL PROTECTED] wrote:
That's going to be interesting, yeah. You would need to teach the
wireless firmware about it? How about just checking on wakeup? Some
kind of wake-on-lan signal?
Binding upgrade notifications to a multicast address as I previously
proposed fixes this problem without any kind of firmware hacking.
Ahh, sorry, I thought you meant _really_ asleep - like not waking up on
network events. Although does our independent firmware know enough to
wake us up on multicast traffic? I thought that it only worked on the
lower level protocols and that a packet had to be specifically destined
for our MAC address to get a wake up event. I'll show my ignorance of
multicast here: does it include specific MAC addresses or is it a wide
broadcast at that layer? I always assumed that it was the latter.
Can you explain how they are odd? It sure would help everyone.
Caveat: I'm not an expert here. I haven't read the code, just the
documentation. So we can all follow along, start here:
http://linux-vserver.org/Paper#Unification
http://linux-vserver.org/Frequently_Asked_Questions#What_is_vhashify.3F
Basically, copy-on-write works by running a tool ('vhashify') which
looks for identical files in the different containers and hard links
them together, then marks them immutable. The copy-on-write mechanism
works by intercepting writes to immutable files and cloning the file
before making it writable by the container.
Quoting from their FAQ:
(when running vhashify:) The guest needs to be running because
vhashify tries to figure out what files not to hashify by calling the
package manager of the guest via vserver enter.
In order for the OS cache to benefit from the hardlinking, you'll have
to restart the vservers.
Holy crap, this sounds like a steaming _pile_ of complexity. Are we
seriously going to try to deploy on this?
Since vserver is doing file hashification anyway, it seems like it
would be a much better idea to use this rather than reinvent the
wheel. Some other issues:
a) we may need to be aware of which files are hardlinked where in
order to do proper updates.
b) not clear how/if we can make updates to an entire tree atomically
c) how/can we restart the vservers? how important is this?
I think we need to bring in a vserver expert (ie, not me) to get these
details right at the start.
Am happy to get more advice on this, for sure. I suspect that all of
the vserver people we can call on are on this list.
Our current thinking basically is that we can do an update as part of an
update/shutdown procedure. So you can apply the updates on the way
down, get a new env on restart. That would handle the vserver restarts
and also how to get a new kernel issue that no one else has mentioned.
I'm not sure if hashing for updates and hashing for vserver are the
kinds of things we want to share or not. I would love to hear more
about how vserver does its hashing and see if we can share. I still
feel that keeping the update system as simple and uncomplex as possible
is a very good way to go - it lets us advance in parallel and come up
with something that works well.
It also sounds like it's pretty easy to do something like:
o Start root container
o Start guest container
o Apply update
o Start activities in that guest container
And the hardlink/CoW stuff will then give us an updated container.
Still doesn't help with updates on the base system nor kernel bits, but
it's a start.
Yeah, but you always need both sets of information to be able to
generate them. So you have to host full file + diff data if you want to
host an update.
My proposal would be that XOs pass around binary diffs *only* among
themselves, and that if someone needs an older version or to leapfrog
versions, they ask the school server. This allows XOs to cheaply host
updates in the common case.
You could do that with Alex's system as well. But in Alex's case the XO
doesn't have to carry both the system it's using + diff. Because the
system you're using is the update.
The nice thing about Alex's system is that you only
have to host the file data that you're using on your system instead of
file + diff data. You end up using less space that way.
If you look at the numbers I just posted, file+diff is 1.3% larger
than just files.
If you want to
downgrade, you have to get the files or use the vserver versions (maybe
you could just use the old files handled by the CoW stuff to drive the
update system - that might work pretty well!)
Now we're talking! ;-)
Keep in mind that those blobs he's talking about are just files. The
only place where we would add binary diffs would be for individual
files, not entire trees. So what we're downloading today is only the
changed files, largely for the sake of expediency and what I describe
Re: XO in-field upgrades
On Mon, 2007-06-25 at 15:56 -0400, Noah Kantrowitz wrote: Design docs? We're still at the proof-of-concept phase really ;-). But yes, each chroot needs to be generated on the fly when a new activity starts (unless we do some funky magic with unionfs, which is probably not a great idea). The load of a few directory hardlinks should be minimal. Are we expecting to do online updating or will it be more of a windows-style shut it all down then patch? I think that for phase one we can do it during a shutdown/restart cycle. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
