Re: Synchronizing xs-0.3 and xo-??? --- backups
On Mon, Apr 28, 2008 at 7:23 PM, Martin Langhoff [EMAIL PROTECTED] wrote: On Tue, Apr 29, 2008 at 5:14 AM, Eben Eliason [EMAIL PROTECTED] wrote: On Mon, Apr 28, 2008 at 12:58 PM, Michael Stone [EMAIL PROTECTED] wrote: On Mon, Apr 28, 2008 at 10:33:13AM -0400, Eben Eliason wrote: Well, I guess I need to know a bit more about the technical details that will be in play in this circumstance. Since we don't have unique usernames or passwords, the only identifier for the individual and her data is her key, right? How does one obtain her key if, for instance, her previous machine was bricked or stolen? And even if she has it, must we really require her to type it in? I suspect we may, unfortunately. Laptops are uniquely identified by their serial-number/uuid pairs. The ssh keys are incidental. Ivan's proposal (correctly, in my opinion) calls for backups to be associated with laptops on the XS with an SN - child OK. relation. I regard the problem of associating a new laptop with an existing child as a problem that can be most conveniently solved with a configuration tool or UI on the XS. Yes - this is the track I am working on. So, backing up one step...this means that in any case where the restore needs to be performed on the same laptop, we can do the retrieval automatically by silently passing the SN from the XO to XS. Correct? It's only in the new-machine case that we need any user intervention, apart from initiation and perhaps confirmation. Well, we need to define a workflow that contemplates trust and real life problems. My fuzzy idea was to - Get the teacher to indicate via a web-based UI on the XS (to be done later) that new laptop UUID/SN should be associated to this (already existing) user+laptop account. - When the laptop registers, accept the locally generated SSH key and somehow alias both accts together or provide access to the old acct I was actually referring to a case where the child wanted to do a restore on their current laptop, rather than a new one, in which case no new association is needed and it seems there wouldn't be any XS-side requirements. Perhaps this is an infrequent use case, in practice? As a final option, we could of course check on the XS for a backup associated with the machine's SN, and secondarily offer a UI for entering the SN association if one doesn't exist. This has the benefit of handling both of the above cases silently as described...as I was thinking along the same lines (check for a related backup regularly), but based on an association made on the XS by the teacher. I don't know that it needs to be regularly. Do you think we should try to automatically discover when a new restoration association is available? It seems just as easy to me to require a simple manual interaction in the XO UI to initiate the check. Perhaps you could do a check initially upon boot/login, so that in the case where a kid boots up a brand new machine for the first time, and the association has been made in advance, it runs automatically... Tricky aspects remain of defining teacher and providing a suitable UI on the XS side, but I will cross that bridge when I get there :-) It seems that teacher basically translates to the individual with access to the XS UI, which seems fine to me. - Eben ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Tue, Apr 29, 2008 at 11:11 PM, Tomeu Vizoso [EMAIL PROTECTED] wrote: That would be _very_ costly on the server side. Let the rsync complete, and do a bulk import. Ok, we just have to take in account that the bulk import will block all activity requiring the DS. This is a once-only operation anyway. cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Wed, Apr 30, 2008 at 3:37 AM, Eben Eliason [EMAIL PROTECTED] wrote: On Mon, Apr 28, 2008 at 7:23 PM, Martin Langhoff I was actually referring to a case where the child wanted to do a restore on their current laptop, rather than a new one, in which case no new association is needed and it seems there wouldn't be any XS-side requirements. Perhaps this is an infrequent use case, in practice? Ah. Of course. The retrieval can happen via - Web UI - authentication TBA - Journal - I don't know that it needs to be regularly. Do you think we should try to automatically discover when a new restoration association is available? It seems just as easy to me to require a simple manual interaction in the XO UI to initiate the check. Perhaps you could do a check initially upon boot/login, so that in the case where a kid boots up a brand new machine for the first time, and the association has been made in advance, it runs automatically... Boot-time sounds good to me :-) cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Mon, Apr 28, 2008 at 10:14 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: On Thu, Apr 24, 2008 at 6:18 AM, Martin Langhoff [EMAIL PROTECTED] wrote: - Initial HTTP check for protocol and XS availability stays. We drop the 'nonce' value. - Q: Any reason for this being REST vs XML-RPC for the registration service? - Preparations are simpler - No need for a master index. Always backup everything (let rsync deal with what needs to be backed up), to a directory that will have the last copy we rsync'd. Make extensive use of rsync smarts. - We must dump _all_ the metadata to a CJSON file - Q: Could we create a single metadata file per document (ahem, Journal entry) instead? - Q: Does it make sense to always maintain the CJSON-formatted metadata anyway? (This would specially make sense if some of the DS corruption reports being discussed are tracked down to Xapian.) Extracting all the metadata from the index is a relatively expensive process, that could affect very negatively the user experience while it lasts. I really recommend to do it in smaller chunks (one hundred entries?). One single metadata file per entry looks best to me, and is already implemented here: http://dev.laptop.org/git?p=users/krstic/ds-backup;a=summary - Initially I thought that transferring the Something missing here? Full restore: We can simplify things here - If the XS is using rsync, the REST API returns a path, the XO rsyncs from there - If the XS is using git internally, there are more options it may force a delay (to get a temp checkout that will be held there for 24hs) or perhaps the client can just execute git archive and pipe the output to itself ;-) - After retrieval of files from the XS, the XO must rebuild its Xapian metadata store Should this happen after each entry has been copied? - Q: How do we trigger this from Sugar on a just-reflashed machine? Which user experience do we want here? Eben? Well, I guess I need to know a bit more about the technical details that will be in play in this circumstance. Since we don't have unique usernames or passwords, the only identifier for the individual and her data is her key, right? How does one obtain her key if, for instance, her previous machine was bricked or stolen? And even if she has it, must we really require her to type it in? I suspect we may, unfortunately. Once these technical considerations are more clear, we can discuss the manner in which one initiates a restore. One clean option is to add a Restore from backup action to the Journal palette, which would launch a modal dialog asking for any required info and displaying progress information. Other questions here include: Should this be modal to the Journal? To the entire OS? Can we allow activities to run (and save, etc) in the background during a restore? I'm happy to create a mockup or two for this once I have a clearer idea of the requirements. - Eben Single-item-restore: I'll punt on this in the short-term (as I have to prepare an XS release too!), but various avenues are open. We can support web-based zipfile download, specially since Tomeu's implemented the handling, and we can support Journal-based browse-and-restore. But want to get the major issues sorted first ;-) Sure. Robson may be interested in lending a hand here. At any point, thinking now about the different possibilities may not do any harm. Thanks, Tomeu ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Mon, Apr 28, 2008 at 10:33:13AM -0400, Eben Eliason wrote: Well, I guess I need to know a bit more about the technical details that will be in play in this circumstance. Since we don't have unique usernames or passwords, the only identifier for the individual and her data is her key, right? How does one obtain her key if, for instance, her previous machine was bricked or stolen? And even if she has it, must we really require her to type it in? I suspect we may, unfortunately. Laptops are uniquely identified by their serial-number/uuid pairs. The ssh keys are incidental. Ivan's proposal (correctly, in my opinion) calls for backups to be associated with laptops on the XS with an SN - child relation. I regard the problem of associating a new laptop with an existing child as a problem that can be most conveniently solved with a configuration tool or UI on the XS. Michael ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Mon, Apr 28, 2008 at 12:58 PM, Michael Stone [EMAIL PROTECTED] wrote: On Mon, Apr 28, 2008 at 10:33:13AM -0400, Eben Eliason wrote: Well, I guess I need to know a bit more about the technical details that will be in play in this circumstance. Since we don't have unique usernames or passwords, the only identifier for the individual and her data is her key, right? How does one obtain her key if, for instance, her previous machine was bricked or stolen? And even if she has it, must we really require her to type it in? I suspect we may, unfortunately. Laptops are uniquely identified by their serial-number/uuid pairs. The ssh keys are incidental. Ivan's proposal (correctly, in my opinion) calls for backups to be associated with laptops on the XS with an SN - child OK. relation. I regard the problem of associating a new laptop with an existing child as a problem that can be most conveniently solved with a configuration tool or UI on the XS. So, backing up one step...this means that in any case where the restore needs to be performed on the same laptop, we can do the retrieval automatically by silently passing the SN from the XO to XS. Correct? It's only in the new-machine case that we need any user intervention, apart from initiation and perhaps confirmation. In the new-machine case, it seems that the association could be done either at the XS or at the XO. Performing it at the XO gives us one common place to put all the UI work for restore, but means that a child (or a teacher helping her) will need to enter an SN by hand. Alternately, if the association is made at the XS in advance (by who?), then the restore can work silently just as it would had this always been the child's machine. As a final option, we could of course check on the XS for a backup associated with the machine's SN, and secondarily offer a UI for entering the SN association if one doesn't exist. This has the benefit of handling both of the above cases silently as described...as long as a backup is associated with the SN, regardless of how the association was made (it could have been configured at the XS by an admin). There's a privacy concern, though: naturally we don't want to immediately offer an entry field into which anyone can type any SN (they could have read it physically from another classmates machine) to gain access to another child's data. Perhaps this is a reason to only allow the association server side... - Eben ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Tue, Apr 29, 2008 at 5:14 AM, Eben Eliason [EMAIL PROTECTED] wrote: On Mon, Apr 28, 2008 at 12:58 PM, Michael Stone [EMAIL PROTECTED] wrote: On Mon, Apr 28, 2008 at 10:33:13AM -0400, Eben Eliason wrote: Well, I guess I need to know a bit more about the technical details that will be in play in this circumstance. Since we don't have unique usernames or passwords, the only identifier for the individual and her data is her key, right? How does one obtain her key if, for instance, her previous machine was bricked or stolen? And even if she has it, must we really require her to type it in? I suspect we may, unfortunately. Laptops are uniquely identified by their serial-number/uuid pairs. The ssh keys are incidental. Ivan's proposal (correctly, in my opinion) calls for backups to be associated with laptops on the XS with an SN - child OK. relation. I regard the problem of associating a new laptop with an existing child as a problem that can be most conveniently solved with a configuration tool or UI on the XS. Yes - this is the track I am working on. So, backing up one step...this means that in any case where the restore needs to be performed on the same laptop, we can do the retrieval automatically by silently passing the SN from the XO to XS. Correct? It's only in the new-machine case that we need any user intervention, apart from initiation and perhaps confirmation. Well, we need to define a workflow that contemplates trust and real life problems. My fuzzy idea was to - Get the teacher to indicate via a web-based UI on the XS (to be done later) that new laptop UUID/SN should be associated to this (already existing) user+laptop account. - When the laptop registers, accept the locally generated SSH key and somehow alias both accts together or provide access to the old acct As a final option, we could of course check on the XS for a backup associated with the machine's SN, and secondarily offer a UI for entering the SN association if one doesn't exist. This has the benefit of handling both of the above cases silently as described...as I was thinking along the same lines (check for a related backup regularly), but based on an association made on the XS by the teacher. Tricky aspects remain of defining teacher and providing a suitable UI on the XS side, but I will cross that bridge when I get there :-) cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Tue, Apr 29, 2008 at 2:14 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: - Q: Does it make sense to always maintain the CJSON-formatted metadata anyway? (This would specially make sense if some of the DS corruption reports being discussed are tracked down to Xapian.) Extracting all the metadata from the index is a relatively expensive process, that could affect very negatively the user experience while it lasts. Is it? Ugh then. (I thought it'd be a cheap op!) I really recommend to do it in smaller chunks (one hundred entries?). One single metadata file per entry looks best to me, and is already implemented here: http://dev.laptop.org/git?p=users/krstic/ds-backup;a=summary Ah, ok. I must have mis-read it, I thought it was exporting to a single large metadata file. - Initially I thought that transferring the Something missing here? Nothing important :-) sorry! Full restore: We can simplify things here - If the XS is using rsync, the REST API returns a path, the XO rsyncs from there - If the XS is using git internally, there are more options it may force a delay (to get a temp checkout that will be held there for 24hs) or perhaps the client can just execute git archive and pipe the output to itself ;-) - After retrieval of files from the XS, the XO must rebuild its Xapian metadata store Should this happen after each entry has been copied? That would be _very_ costly on the server side. Let the rsync complete, and do a bulk import. Single-item-restore: I'll punt on this in the short-term (as I have to prepare an XS release too!), but various avenues are open. We can support web-based zipfile download, specially since Tomeu's implemented the handling, and we can support Journal-based browse-and-restore. But want to get the major issues sorted first ;-) Sure. Robson may be interested in lending a hand here. At any point, thinking now about the different possibilities may not do any harm. Definitely. Right now I am trying to sort out the plumbing that does the real job, and do the web-based stuff later ;-) cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Thu, Apr 24, 2008 at 4:22 PM, Martin Langhoff [EMAIL PROTECTED] wrote: I'll tidy-up and update http://wiki.laptop.org/go/XS_backup_restore - Done. Feedback welcome. It's Friday here, I'll be hacking on this on Monday (NZ time) so you guys have a chance to rip it to shreds before I start... cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
2008/4/22 Eben Eliason [EMAIL PROTECTED]: 2008/4/22 Tomeu Vizoso [EMAIL PROTECTED]: 2008/4/22 Eben Eliason [EMAIL PROTECTED]: 2008/4/22 Tomeu Vizoso [EMAIL PROTECTED]: 2008/4/22 Martin Langhoff [EMAIL PROTECTED]: On Tue, Apr 22, 2008 at 10:18 PM, Ivan Krstić [EMAIL PROTECTED] wrote: My solution is the simplest design I could create that allows both emergency restore (laptop FS has been trashed, get everything back) and individual file restore and sharing via a web interface running on the XS, Indeed, reading bug 4200 the purpose of the protocol becomes clearer. What I don't understand completely is the drive behind retrieving the files via a web interface. I'd think that the main requirement is that the Journal can talk to it in a simple enough fashion -- perhaps it was a shortcut to avoid complex work on the Journal? Yes, having the journal activity browsing its own backup as well as the public sections of the other children could be quite a bit of work. A web app that allowed clicking on a link and having that entry installed in the journal seemed much better in the short-medium term. Well, let's not confuse these two use cases. I think they are quite different. Use case (a) offers a child a way to individually recover entries that they themselves created, but which have since been deleted from their Journal. Use case (b) offers a child a way to browse a repository of work that other children have created, with the ability to download, share, and modify those works. I think (a) deserves to be tied closely to the Journal UI, which serves as a history of what a given child has done, and is likely the place one already will be when discovering one wants something that has been deleted. Case (b) is much more appropriate as a web app, and for that matter likely requires additional UI capabilities for determining what is allowed to be presented in this mannerwe don't want every backed up entry to be publicly shared with the world. Yes, the two use cases are very different and, given enough resources, should be implemented separately. As we have so many important things left to do, it has been suggested that a child could mark one of the entries in its backup as public, so other people can access it. In this way, the backup section of a kid in the server would have the following functionalities: - browse backed up entries, - download one entry, - mark one entry as public, - mark it as accessible by other user of the school server. Does it look like a bad compromise between functionality and cost? It's not particularly terrible, but there are some serious drawbacks: 1. The recovered object, as a download from browse, will be treated as a new entry, with new metadata and activity id. The main purpose of the restore in my mind is to allow one to recover something within the historical context. No, the entry should be recreated exactly as it was in the original place. We are zipping the file plus metadata together. This is already implemented, btw. 2. The child can only recover objects which they also felt comfortable tagging as public. No, the child's entries get backed up as private. Later, the user could browse the backup and restore, mark as public, send to another kid, etc every individual entry. 3. Loss of the clear distinction of the Journal as the place for managing ones files. This is a philosophical issue, but I think it's still valid. Browse is about new discovery, and obtaining new things. The Journal is about history, and managing ones own things. Agreed. Tomeu ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
2008/4/23 Martin Langhoff [EMAIL PROTECTED]: 2008/4/23 Eben Eliason [EMAIL PROTECTED]: 2008/4/22 Martin Langhoff [EMAIL PROTECTED]: On Tue, Apr 22, 2008 at 10:18 PM, Ivan Krstić [EMAIL PROTECTED] wrote: My solution is the simplest design I could create that allows both emergency restore (laptop FS has been trashed, get everything back) and individual file restore and sharing via a web interface running on the XS, Indeed, reading bug 4200 the purpose of the protocol becomes clearer. What I don't understand completely is the drive behind retrieving the files via a web interface. I'd think that the main requirement is that the Journal can talk to it in a simple enough fashion -- perhaps it was a shortcut to avoid complex work on the Journal? I do believe that was the case at the time. I think that the user experience will be much better if we provide support for entry recovery directly within the Journal interface. To complete the picture in my mind of how well/bad rsync will interact with this workload, I am trying to read up a bit on the current datastore implementation, but I haven't found any high-level overview of its internal organization. Is there one? I am not after the API, but after a rough explanation of how things are laid out on-disk. Even if the DS will change, it's important that things work reasonably well with the current DS. We have in ~/.sugar/default/datastore/store all the files that contain the opaque data of every entry, named by an uuid that identifies the entry. In ~/.sugar/default/datastore/store/preview are the values of the metadata property 'preview', png files named by uuid of the entry they refer to. In ~/.sugar/default/datastore/store/index are the files that make part of the xapian index, see the links below for more details. The index not only speeds up lookups, but also store the whole metadata (in a quite inefficient way). http://www.xapian.org/docs/admin_notes.html http://www.xapian.org/docs/quartzdesign.html That's all. I'm more than happy to answer any other question you have about the DS. Thanks, Tomeu ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
Tomeu, Superb - thanks for the info! some questions below... 2008/4/23 Tomeu Vizoso [EMAIL PROTECTED]: We have in ~/.sugar/default/datastore/store all the files that contain the opaque data of every entry, named by an uuid that identifies the entry. So this is the actual data store - Does it also have the metadata? - When a file changes or is renamed, the UUID remains the same? - Do files get moved/renamed under any condition? - Can we only backup this, and ignore the preview and index? IOWs will Journal or Sugar know how to recreate preview and index? cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Tue, Apr 22, 2008 at 12:18 PM, Ivan Krstić [EMAIL PROTECTED] wrote: On Apr 22, 2008, at 12:46 AM, Martin Langhoff wrote: Wad's too simple, I am hoping to use a slightly simplified version of Ivan's. For the restore part, I think I will drastically simplify things from Ivan's approach. My solution is the simplest design I could create that allows both emergency restore (laptop FS has been trashed, get everything back) and individual file restore and sharing via a web interface running on the XS, both in the face of an opaque DS backend (Xapian). These were, at the time I was asked to design this, communicated as requirements for the solution. If you relax the requirements or swap in a less braindead DS backend, indeed you can simplify the whole system. Hi Ivan, do you have around the code for the server-side implementation and would you like to share it with us? Thanks, Tomeu ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
Tomeu, Ivan, with my notes from the last software-status meeting and a few other bits and pieces, my plan is to follow roughly the spec but with a few changes. Making a backup: - Initial HTTP check for protocol and XS availability stays. We drop the 'nonce' value. - Q: Any reason for this being REST vs XML-RPC for the registration service? - Preparations are simpler - No need for a master index. Always backup everything (let rsync deal with what needs to be backed up), to a directory that will have the last copy we rsync'd. Make extensive use of rsync smarts. - We must dump _all_ the metadata to a CJSON file - Q: Could we create a single metadata file per document (ahem, Journal entry) instead? - Q: Does it make sense to always maintain the CJSON-formatted metadata anyway? (This would specially make sense if some of the DS corruption reports being discussed are tracked down to Xapian.) - Initially I thought that transferring the - Control moves from HTTP-based (with the 'nonce' identifier) to an rsync wrapper on the XS side that... - re-checks 'availability/load' criteria - uses a lock to prevent overapping transfers from the same client - sets a flag to indicate that the transfer was successful for XS-side post-processing - XS postprocessing - once a backup is successful, we'll make a hardlinked copy to a timestamped directory. rsync is good about breaking hardlinks, so this allows us to do a cheap storage in a pdumpfs fashion. In any case, this is an XS-side thing - might do it with git too ;-) Full restore: We can simplify things here - If the XS is using rsync, the REST API returns a path, the XO rsyncs from there - If the XS is using git internally, there are more options it may force a delay (to get a temp checkout that will be held there for 24hs) or perhaps the client can just execute git archive and pipe the output to itself ;-) - After retrieval of files from the XS, the XO must rebuild its Xapian metadata store - Q: How do we trigger this from Sugar on a just-reflashed machine? - Q: We'll have to define an interim means of aliasing the acct so that the XS allows restore to a different XO in case of a replaced XO. Single-item-restore: I'll punt on this in the short-term (as I have to prepare an XS release too!), but various avenues are open. We can support web-based zipfile download, specially since Tomeu's implemented the handling, and we can support Journal-based browse-and-restore. But want to get the major issues sorted first ;-) cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Thu, Apr 24, 2008 at 4:18 PM, Martin Langhoff [EMAIL PROTECTED] wrote: with my notes from the last software-status meeting and a few other bits and pieces, my plan is to follow roughly the spec but with a few changes. I'll tidy-up and update http://wiki.laptop.org/go/XS_backup_restore - cheers, martin -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Tue, Apr 22, 2008 at 6:46 AM, Martin Langhoff [EMAIL PROTECTED] wrote: Hi Michael, thanks for looking into this - I had the wikipage in my sight, but not the bugs you listed. Excellent, reading up to fill the gaps. Based on feedback from Peru, Mexico, and Nepal, the restoration from disaster-recovery backups XO/XS coordination feature has been steadily rising in priority. I also notice that Backups is your first line-item on the XS-0.3 roadmap. Yes. I was planning on looking at it next week, but happy to bump it up. My large question is: what changes need to be made to the XO's OS, (currently to candidate-703) in order to make progress in this feature cluster? I was hoping to say here's a Python script that needs to be added to the image and called opportunistically when NM tells us we are connected to the XS (and at a regular interval during that situation) and we have plenty battery or a powercord. Sub-questions: * What backup scheme do we actually intend to deploy? (Current choices appear to be Wad's dumb-rsync method [1] and Ivan's method [2]. Wad's too simple, I am hoping to use a slightly simplified version of Ivan's. For the restore part, I think I will drastically simplify things from Ivan's approach. * Does the chosen scheme permit us to sanely combine old backups with an XO build that makes breaking changes to file layouts? O-o-ouch! I imagine you are thinking about datastore overhaul insurance...? ;-) Yes, when Ivan designed his backup scheme, he had in mind a future rewrite of the DS, using an incompatible data backend. Yes and no. My target layer is purely files, and we use rsync to copy them over either way. *If* the datastore changes, and file layouts change significantly, we'll copy that verbatim to the XS. If (for restore purposes) the XO wants to read data from before the datastore change, it will have to know how to... we'll just provide the old files ;-) ... though the XS could provide translation services to help make that easier. In practice, if the datastore stuff is today in a library on the XO, it will be kept to support legacy data...? The DS is today a dbus service, but a python module that access the data in the old format wouldn't be hard to provide. (Was going to say that any datastore change must support the fact that data will be on old USB sticks, but the Journal saves in a different format to the USB stick.) We have plans to stop using usb sticks and sd cards as extensions of the local DS, and just display the files on it. Do we have a set of requirements for the next releases? Robson is interested in working on restoring and sharing of individual journal entries from the server, what priority have these use cases? Thanks, Tomeu ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Apr 22, 2008, at 12:46 AM, Martin Langhoff wrote: Wad's too simple, I am hoping to use a slightly simplified version of Ivan's. For the restore part, I think I will drastically simplify things from Ivan's approach. My solution is the simplest design I could create that allows both emergency restore (laptop FS has been trashed, get everything back) and individual file restore and sharing via a web interface running on the XS, both in the face of an opaque DS backend (Xapian). These were, at the time I was asked to design this, communicated as requirements for the solution. If you relax the requirements or swap in a less braindead DS backend, indeed you can simplify the whole system. -- Ivan Krstić [EMAIL PROTECTED] | http://radian.org ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
On Tue, Apr 22, 2008 at 10:18 PM, Ivan Krstić [EMAIL PROTECTED] wrote: My solution is the simplest design I could create that allows both emergency restore (laptop FS has been trashed, get everything back) and individual file restore and sharing via a web interface running on the XS, Indeed, reading bug 4200 the purpose of the protocol becomes clearer. What I don't understand completely is the drive behind retrieving the files via a web interface. I'd think that the main requirement is that the Journal can talk to it in a simple enough fashion -- perhaps it was a shortcut to avoid complex work on the Journal? thanks for chiming in! :-) cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
2008/4/22 Martin Langhoff [EMAIL PROTECTED]: On Tue, Apr 22, 2008 at 10:18 PM, Ivan Krstić [EMAIL PROTECTED] wrote: My solution is the simplest design I could create that allows both emergency restore (laptop FS has been trashed, get everything back) and individual file restore and sharing via a web interface running on the XS, Indeed, reading bug 4200 the purpose of the protocol becomes clearer. What I don't understand completely is the drive behind retrieving the files via a web interface. I'd think that the main requirement is that the Journal can talk to it in a simple enough fashion -- perhaps it was a shortcut to avoid complex work on the Journal? I do believe that was the case at the time. I think that the user experience will be much better if we provide support for entry recovery directly within the Journal interface. - Eben ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
2008/4/22 Martin Langhoff [EMAIL PROTECTED]: On Tue, Apr 22, 2008 at 10:18 PM, Ivan Krstić [EMAIL PROTECTED] wrote: My solution is the simplest design I could create that allows both emergency restore (laptop FS has been trashed, get everything back) and individual file restore and sharing via a web interface running on the XS, Indeed, reading bug 4200 the purpose of the protocol becomes clearer. What I don't understand completely is the drive behind retrieving the files via a web interface. I'd think that the main requirement is that the Journal can talk to it in a simple enough fashion -- perhaps it was a shortcut to avoid complex work on the Journal? Yes, having the journal activity browsing its own backup as well as the public sections of the other children could be quite a bit of work. A web app that allowed clicking on a link and having that entry installed in the journal seemed much better in the short-medium term. And here is where Robson's application appears in the game: http://code.google.com/soc/2008/olpc/app.html?csaid=KhwGAARfXjYKOzMDHgpYX28GNx5eQgkFBnlRaUpeRghTVnNWOUo%3D%0A Worth taking in account as this requirement comes from one of the oldest pilots. Thanks, Tomeu ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
2008/4/22 Tomeu Vizoso [EMAIL PROTECTED]: 2008/4/22 Martin Langhoff [EMAIL PROTECTED]: On Tue, Apr 22, 2008 at 10:18 PM, Ivan Krstić [EMAIL PROTECTED] wrote: My solution is the simplest design I could create that allows both emergency restore (laptop FS has been trashed, get everything back) and individual file restore and sharing via a web interface running on the XS, Indeed, reading bug 4200 the purpose of the protocol becomes clearer. What I don't understand completely is the drive behind retrieving the files via a web interface. I'd think that the main requirement is that the Journal can talk to it in a simple enough fashion -- perhaps it was a shortcut to avoid complex work on the Journal? Yes, having the journal activity browsing its own backup as well as the public sections of the other children could be quite a bit of work. A web app that allowed clicking on a link and having that entry installed in the journal seemed much better in the short-medium term. Well, let's not confuse these two use cases. I think they are quite different. Use case (a) offers a child a way to individually recover entries that they themselves created, but which have since been deleted from their Journal. Use case (b) offers a child a way to browse a repository of work that other children have created, with the ability to download, share, and modify those works. I think (a) deserves to be tied closely to the Journal UI, which serves as a history of what a given child has done, and is likely the place one already will be when discovering one wants something that has been deleted. Case (b) is much more appropriate as a web app, and for that matter likely requires additional UI capabilities for determining what is allowed to be presented in this mannerwe don't want every backed up entry to be publicly shared with the world. - Eben ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
2008/4/22 Eben Eliason [EMAIL PROTECTED]: 2008/4/22 Tomeu Vizoso [EMAIL PROTECTED]: 2008/4/22 Martin Langhoff [EMAIL PROTECTED]: On Tue, Apr 22, 2008 at 10:18 PM, Ivan Krstić [EMAIL PROTECTED] wrote: My solution is the simplest design I could create that allows both emergency restore (laptop FS has been trashed, get everything back) and individual file restore and sharing via a web interface running on the XS, Indeed, reading bug 4200 the purpose of the protocol becomes clearer. What I don't understand completely is the drive behind retrieving the files via a web interface. I'd think that the main requirement is that the Journal can talk to it in a simple enough fashion -- perhaps it was a shortcut to avoid complex work on the Journal? Yes, having the journal activity browsing its own backup as well as the public sections of the other children could be quite a bit of work. A web app that allowed clicking on a link and having that entry installed in the journal seemed much better in the short-medium term. Well, let's not confuse these two use cases. I think they are quite different. Use case (a) offers a child a way to individually recover entries that they themselves created, but which have since been deleted from their Journal. Use case (b) offers a child a way to browse a repository of work that other children have created, with the ability to download, share, and modify those works. I think (a) deserves to be tied closely to the Journal UI, which serves as a history of what a given child has done, and is likely the place one already will be when discovering one wants something that has been deleted. Case (b) is much more appropriate as a web app, and for that matter likely requires additional UI capabilities for determining what is allowed to be presented in this mannerwe don't want every backed up entry to be publicly shared with the world. Yes, the two use cases are very different and, given enough resources, should be implemented separately. As we have so many important things left to do, it has been suggested that a child could mark one of the entries in its backup as public, so other people can access it. In this way, the backup section of a kid in the server would have the following functionalities: - browse backed up entries, - download one entry, - mark one entry as public, - mark it as accessible by other user of the school server. Does it look like a bad compromise between functionality and cost? Thanks, Tomeu ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
2008/4/22 Tomeu Vizoso [EMAIL PROTECTED]: 2008/4/22 Eben Eliason [EMAIL PROTECTED]: 2008/4/22 Tomeu Vizoso [EMAIL PROTECTED]: 2008/4/22 Martin Langhoff [EMAIL PROTECTED]: On Tue, Apr 22, 2008 at 10:18 PM, Ivan Krstić [EMAIL PROTECTED] wrote: My solution is the simplest design I could create that allows both emergency restore (laptop FS has been trashed, get everything back) and individual file restore and sharing via a web interface running on the XS, Indeed, reading bug 4200 the purpose of the protocol becomes clearer. What I don't understand completely is the drive behind retrieving the files via a web interface. I'd think that the main requirement is that the Journal can talk to it in a simple enough fashion -- perhaps it was a shortcut to avoid complex work on the Journal? Yes, having the journal activity browsing its own backup as well as the public sections of the other children could be quite a bit of work. A web app that allowed clicking on a link and having that entry installed in the journal seemed much better in the short-medium term. Well, let's not confuse these two use cases. I think they are quite different. Use case (a) offers a child a way to individually recover entries that they themselves created, but which have since been deleted from their Journal. Use case (b) offers a child a way to browse a repository of work that other children have created, with the ability to download, share, and modify those works. I think (a) deserves to be tied closely to the Journal UI, which serves as a history of what a given child has done, and is likely the place one already will be when discovering one wants something that has been deleted. Case (b) is much more appropriate as a web app, and for that matter likely requires additional UI capabilities for determining what is allowed to be presented in this mannerwe don't want every backed up entry to be publicly shared with the world. Yes, the two use cases are very different and, given enough resources, should be implemented separately. As we have so many important things left to do, it has been suggested that a child could mark one of the entries in its backup as public, so other people can access it. In this way, the backup section of a kid in the server would have the following functionalities: - browse backed up entries, - download one entry, - mark one entry as public, - mark it as accessible by other user of the school server. Does it look like a bad compromise between functionality and cost? It's not particularly terrible, but there are some serious drawbacks: 1. The recovered object, as a download from browse, will be treated as a new entry, with new metadata and activity id. The main purpose of the restore in my mind is to allow one to recover something within the historical context. 2. The child can only recover objects which they also felt comfortable tagging as public. 3. Loss of the clear distinction of the Journal as the place for managing ones files. This is a philosophical issue, but I think it's still valid. Browse is about new discovery, and obtaining new things. The Journal is about history, and managing ones own things. - Eben ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
I'm not seeing the use case that I *think* is the primary one: 1) Backup and restore from a disaster recovery perspective. and the secondary that goes with that is: 2) Ability to restore to a new laptop (as in the child lost the laptop or it died and they need to restore their work to a new laptop) You don't need any browsing capability for this. Kim 2008/4/22 Tomeu Vizoso [EMAIL PROTECTED]: 2008/4/22 Eben Eliason [EMAIL PROTECTED]: 2008/4/22 Tomeu Vizoso [EMAIL PROTECTED]: 2008/4/22 Martin Langhoff [EMAIL PROTECTED]: On Tue, Apr 22, 2008 at 10:18 PM, Ivan Krstić [EMAIL PROTECTED] wrote: My solution is the simplest design I could create that allows both emergency restore (laptop FS has been trashed, get everything back) and individual file restore and sharing via a web interface running on the XS, Indeed, reading bug 4200 the purpose of the protocol becomes clearer. What I don't understand completely is the drive behind retrieving the files via a web interface. I'd think that the main requirement is that the Journal can talk to it in a simple enough fashion -- perhaps it was a shortcut to avoid complex work on the Journal? Yes, having the journal activity browsing its own backup as well as the public sections of the other children could be quite a bit of work. A web app that allowed clicking on a link and having that entry installed in the journal seemed much better in the short-medium term. Well, let's not confuse these two use cases. I think they are quite different. Use case (a) offers a child a way to individually recover entries that they themselves created, but which have since been deleted from their Journal. Use case (b) offers a child a way to browse a repository of work that other children have created, with the ability to download, share, and modify those works. I think (a) deserves to be tied closely to the Journal UI, which serves as a history of what a given child has done, and is likely the place one already will be when discovering one wants something that has been deleted. Case (b) is much more appropriate as a web app, and for that matter likely requires additional UI capabilities for determining what is allowed to be presented in this mannerwe don't want every backed up entry to be publicly shared with the world. Yes, the two use cases are very different and, given enough resources, should be implemented separately. As we have so many important things left to do, it has been suggested that a child could mark one of the entries in its backup as public, so other people can access it. In this way, the backup section of a kid in the server would have the following functionalities: - browse backed up entries, - download one entry, - mark one entry as public, - mark it as accessible by other user of the school server. Does it look like a bad compromise between functionality and cost? Thanks, Tomeu ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
2008/4/22 Kim Quirk [EMAIL PROTECTED]: I'm not seeing the use case that I *think* is the primary one: That's still listed as the main one, at least in Ivan's email, and definitely from my perspective. This discussion of how to deal with the secondary one with minimal fuss and with a good fit with the Journal is informative, as I want to support it, but my core priority is in the disaster case. cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
2008/4/23 Eben Eliason [EMAIL PROTECTED]: 2008/4/22 Martin Langhoff [EMAIL PROTECTED]: On Tue, Apr 22, 2008 at 10:18 PM, Ivan Krstić [EMAIL PROTECTED] wrote: My solution is the simplest design I could create that allows both emergency restore (laptop FS has been trashed, get everything back) and individual file restore and sharing via a web interface running on the XS, Indeed, reading bug 4200 the purpose of the protocol becomes clearer. What I don't understand completely is the drive behind retrieving the files via a web interface. I'd think that the main requirement is that the Journal can talk to it in a simple enough fashion -- perhaps it was a shortcut to avoid complex work on the Journal? I do believe that was the case at the time. I think that the user experience will be much better if we provide support for entry recovery directly within the Journal interface. To complete the picture in my mind of how well/bad rsync will interact with this workload, I am trying to read up a bit on the current datastore implementation, but I haven't found any high-level overview of its internal organization. Is there one? I am not after the API, but after a rough explanation of how things are laid out on-disk. Even if the DS will change, it's important that things work reasonably well with the current DS. cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
Hi Michael, thanks for looking into this - I had the wikipage in my sight, but not the bugs you listed. Excellent, reading up to fill the gaps. Based on feedback from Peru, Mexico, and Nepal, the restoration from disaster-recovery backups XO/XS coordination feature has been steadily rising in priority. I also notice that Backups is your first line-item on the XS-0.3 roadmap. Yes. I was planning on looking at it next week, but happy to bump it up. My large question is: what changes need to be made to the XO's OS, (currently to candidate-703) in order to make progress in this feature cluster? I was hoping to say here's a Python script that needs to be added to the image and called opportunistically when NM tells us we are connected to the XS (and at a regular interval during that situation) and we have plenty battery or a powercord. Sub-questions: * What backup scheme do we actually intend to deploy? (Current choices appear to be Wad's dumb-rsync method [1] and Ivan's method [2]. Wad's too simple, I am hoping to use a slightly simplified version of Ivan's. For the restore part, I think I will drastically simplify things from Ivan's approach. * Does the chosen scheme permit us to sanely combine old backups with an XO build that makes breaking changes to file layouts? O-o-ouch! I imagine you are thinking about datastore overhaul insurance...? ;-) Yes and no. My target layer is purely files, and we use rsync to copy them over either way. *If* the datastore changes, and file layouts change significantly, we'll copy that verbatim to the XS. If (for restore purposes) the XO wants to read data from before the datastore change, it will have to know how to... we'll just provide the old files ;-) ... though the XS could provide translation services to help make that easier. In practice, if the datastore stuff is today in a library on the XO, it will be kept to support legacy data...? (Was going to say that any datastore change must support the fact that data will be on old USB sticks, but the Journal saves in a different format to the USB stick.) cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Synchronizing xs-0.3 and xo-??? --- backups
FYI, Awhile back ago, I developed a quick-n-dirty little *rsync* script which can be ran on the XO Laptop at any time and regularly. Here's the latest online version... http://wiki.laptop.org/go/User:Ixo/Script/xo-backup-server I've set it up on my XO. And run it when-ever and where-ever it is connected to the Internet. Especially after large changes or saved important documents. (currently only acts as 'disaster recovery archive', but with slight mods could be updated to save the last N number of backups) A similar idea / concept, was to do the reverse, from the SERVER backup each laptop when available... http://wiki.laptop.org/go/User:Ixo/Script/get-xo Freely CC-GNU-GPL 'ed , hope it helps, -Ixo On Mon, Apr 21, 2008 at 8:38 PM, Michael Stone [EMAIL PROTECTED] wrote: Martin, Based on feedback from Peru, Mexico, and Nepal, the restoration from disaster-recovery backups XO/XS coordination feature has been steadily rising in priority. I also notice that Backups is your first line-item on the XS-0.3 roadmap. My large question is: what changes need to be made to the XO's OS, (currently to candidate-703) in order to make progress in this feature cluster? Relevant tickets #24 ROTcscott (Backup of laptops - short-term solution) #2516 STKtomeu (Automatic backup of laptops to XS) #3334 DSNjg (Exactly what should be backed up?) #4569 DSNjg (Controlling disk usage of backups) #4224 DSNkrstic (Manage SN - identity mapping on XS) #4270 PKGwad (Full restore from school server) #4380 TSTtomeu (Restore individual entry from school server) #4275 DSNtomeu (Keep UI) #4587 ESCtomeu (Mass-export Journal to USB key) #6374 PKGmartin (Package the xo-backup tools for the school server) #4100 STKkrstic (XS should provide human readable index of journal backup) Sub-questions: * What backup scheme do we actually intend to deploy? (Current choices appear to be Wad's dumb-rsync method [1] and Ivan's method [2]. [1]: http://lists.laptop.org/pipermail/server-devel/2008-February/000314.html (outdated) [2]: http://wiki.laptop.org/go/XS_backup_restore * Does the chosen scheme permit us to sanely combine old backups with an XO build that makes breaking changes to file layouts? Regards, Michael Key: ??? - status or author unknown TST - needs to be tested in a build BLD - needs to be put into a build PKG - pkg needs to be built DBG - debugging/diagnosis still needed DSN - design needed STK - stuck; a decision is needed about how to proceed SGN - a signoff is needed ESC - canceled or siginificantly reduced in priority FIN - successfully finished MSG - communication needed ROT - solution has bitrotted ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
