Re: [Sugar-devel] Journal Backup/Restore UI
On Mon, Mar 29, 2010 at 2:22 AM, Bernie Innocenti ber...@codewiz.org wrote: However, it looks like it would take a couple of months of effort to get I think you are overengineering this. - Single-file restore is well solved by Moodle's backup/restore UI + Browse - As you found out, librsync isn't what you want... just rsync _all_ of it, and then inject each entry using copy-to-journal. It doesn't fix the disk usage problem, but it sure solves the restart datastore and the associated issue that you are deleting whatever new was in the Journal. You are not restoring. Stoopid question: can't you 'FUSE'-mount the XS directory with sshfs and then use copy-to-journal? Latency will make that a tad slower, but the local-diskspace issue is gone. m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Journal Backup/Restore UI
On Mon, 2010-03-29 at 10:14 +0200, Martin Langhoff wrote: On Mon, Mar 29, 2010 at 2:22 AM, Bernie Innocenti ber...@codewiz.org wrote: However, it looks like it would take a couple of months of effort to get I think you are overengineering this. - Single-file restore is well solved by Moodle's backup/restore UI + Browse - As you found out, librsync isn't what you want... just rsync _all_ of it, and then inject each entry using copy-to-journal. It doesn't fix the disk usage problem, but it sure solves the restart datastore and the associated issue that you are deleting whatever new was in the Journal. You are not restoring. Ok, how about: - an initial rsync with no destination to list what's available - rsync one file at a time + copy-to-journal. There's still the possibility that one very large file would make the entire process fail. Does the datastore support creating entries with no data and writing the data at a later time? I don't know much about the datastore design, but I got that we use uuids as keys, not hashes of the content. Stoopid question: can't you 'FUSE'-mount the XS directory with sshfs and then use copy-to-journal? Latency will make that a tad slower, but the local-diskspace issue is gone. Very good idea. -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Journal Backup/Restore UI
On Mon, Mar 29, 2010 at 14:26, Bernie Innocenti ber...@codewiz.org wrote: On Mon, 2010-03-29 at 10:14 +0200, Martin Langhoff wrote: On Mon, Mar 29, 2010 at 2:22 AM, Bernie Innocenti ber...@codewiz.org wrote: However, it looks like it would take a couple of months of effort to get I think you are overengineering this. - Single-file restore is well solved by Moodle's backup/restore UI + Browse - As you found out, librsync isn't what you want... just rsync _all_ of it, and then inject each entry using copy-to-journal. It doesn't fix the disk usage problem, but it sure solves the restart datastore and the associated issue that you are deleting whatever new was in the Journal. You are not restoring. Ok, how about: - an initial rsync with no destination to list what's available - rsync one file at a time + copy-to-journal. There's still the possibility that one very large file would make the entire process fail. Does the datastore support creating entries with no data and writing the data at a later time? I don't know much about the datastore design, but I got that we use uuids as keys, not hashes of the content. Yup. We do use hashes but only internally for detecting duplicates. http://wiki.sugarlabs.org/go/Development_Team/Datastore_Rewrite Regards, Tomeu Stoopid question: can't you 'FUSE'-mount the XS directory with sshfs and then use copy-to-journal? Latency will make that a tad slower, but the local-diskspace issue is gone. Very good idea. -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Journal Backup/Restore UI
I'm a bit confused, is the problem reliable full restore with limited local space or the DS having the xapian DB open while the copy happens? Or both? Thanks, Tomeu On Fri, Mar 26, 2010 at 00:51, Bernie Innocenti ber...@codewiz.org wrote: In Paraguay, we started discussing the topic of improving backup and restore procedures. Let's compare notes with a wider audience. We started from an existing backup/restore procedure written by Daniel Drake one year ago. It's meant to be invoked from the Terminal and uses a transactional approach: 1) the technician runs a script to restore the journal from the school server into a temporary directory while Sugar is still running; 2) then, you restart Sugar (by killing the sugar-shell process); 3) on startup, the olpc-session script notices the temporary directory and replaces the datastore with it almost atomically. The problem with this approach was that it only works as long as there's a enough free space in the filesystem, which is rarely the case with laptops that are actually in use. Transactional behavior would be good to have, but I don't see how we could implement it reliably. Even an rsync in-place would have to be done with --delete-before to ensure it works all the time. This opens the question of how we stop the datastore from accessing the journal and mess it up while we're still updating it. We're currently doing it quite a gross way: kill the datastore process before beginning the restore, then restart Sugar: http://trac.paraguayeduca.org/browser/scripts/os-modifications/diario-restaurar We'll thoroughly test this new kludge tomorrow. I'm confident it will work, but it's a shame that users need to call tech support in order to restore a backup. Perhaps it's time to add integrated backup/restore functionality to the datastore itself, with a nice UI in the Journal or in the control panel. The underlying mechanism could be as simple as the one we're testing now, but with proper synchronization with the datastore and no need to restart Sugar. Shall we go on and write a feature page, targeting 0.90? If it's done as a control panel item, it may be sufficiently self-contained to backport it all the way back to 0.84. For the time being, I might add the backup and restore scripts to olpc-utils. Or, better, create a new sugar-utils package for these things that are generally useful on any platform. Suggestions? -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Journal Backup/Restore UI
On Sun, 2010-03-28 at 14:14 +0200, Tomeu Vizoso wrote: I'm a bit confused, is the problem reliable full restore with limited local space or the DS having the xapian DB open while the copy happens? Or both? Both, but somehow we managed to solve them in a quick dirty way: 1) our script kills the datastore process before starting the restore. Then, at the end of the procedure, we restart Sugar. 2) we restore the journal in place, using the --delete-before option of rsync in order to free up space in the filesystem beforehand I've tested this procedure yesterday in a school, and it worked egregiously with two different laptops. The code is here: http://trac.paraguayeduca.org/browser/scripts/os-modifications/diario-restaurar Now we're just missing a nice GUI for end-users: a control panel icon or a button in the journal toolbar. Someone has to file a feature request and discuss it with the Design team. Would anyone like to get started on this while we're still busy fixing the few remaining high-priority bugs in Sugar 0.84? -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Journal Backup/Restore UI
On Sun, Mar 28, 2010 at 2:29 PM, Bernie Innocenti ber...@codewiz.org wrote: On Sun, 2010-03-28 at 14:14 +0200, Tomeu Vizoso wrote: I'm a bit confused, is the problem reliable full restore with limited local space or the DS having the xapian DB open while the copy happens? Or both? Both, but somehow we managed to solve them in a quick dirty way: Doesn't copy-to-journal skip all the ugly? m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Journal Backup/Restore UI
On Sun, 2010-03-28 at 20:08 +0200, Martin Langhoff wrote: On Sun, Mar 28, 2010 at 2:29 PM, Bernie Innocenti ber...@codewiz.org wrote: On Sun, 2010-03-28 at 14:14 +0200, Tomeu Vizoso wrote: I'm a bit confused, is the problem reliable full restore with limited local space or the DS having the xapian DB open while the copy happens? Or both? Both, but somehow we managed to solve them in a quick dirty way: Doesn't copy-to-journal skip all the ugly? Oh, well, indeed. Though we need to figure out how to glue it to the rsync back-end. The easy part is binding rsync to Python: this was already done in duplicity by wrapping librsync with simple C and Python code which we could steal. Then it starts to get hard: the files are probably going to come down from the xs in some random order. librsync provides a sophisticated callback interface, but there seems to be also a whole-file interface. This approach also opens other interesting possibilities such as restoring individual files or merging the backup with the current journal contents. However, it looks like it would take a couple of months of effort to get something like this done. For the time being, we could keep using the plain rsync and maybe just add a stop/resume interface to the datastore? -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Journal Backup/Restore UI
On Sun, 2010-03-28 at 21:22 -0300, Bernie Innocenti wrote: Oh, well, indeed. Though we need to figure out how to glue it to the rsync back-end. The easy part is binding rsync to Python: this was already done in duplicity by wrapping librsync with simple C and Python code which we could steal. Wait, I had overlooked this note: librsync does not speak the (hairy) application-specific protocol used by rsync. They only share an algorithm, not any code. This rules it out for our usecase, unless we also want to rewrite the xs backend. -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Journal Backup/Restore UI
On Thu, Mar 25, 2010 at 11:51 PM, Bernie Innocenti ber...@codewiz.org wrote: Suggestions? Use (perhaps improve?) copy-to-journal cli script. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
