Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-24 Thread Tomeu Vizoso
On Sat, May 24, 2008 at 2:56 AM, Martin Langhoff [EMAIL PROTECTED] wrote: On Fri, May 23, 2008 at 8:19 PM, Tomeu Vizoso [EMAIL PROTECTED] wrote: Ouch, sorry, I should have been more explicit in that this patch introduces a dependency on cjson. I'm going to send next another patch that falls

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-23 Thread Tomeu Vizoso
On Fri, May 23, 2008 at 6:27 AM, Martin Langhoff [EMAIL PROTECTED] wrote: On Fri, May 23, 2008 at 3:25 PM, Martin Langhoff [EMAIL PROTECTED] wrote: I'll diagnose a bit more and post back. I'm baffled. I initially suspected PEBKAC but it keeps happening. With the patch the DS gets renamed,

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-23 Thread Tomeu Vizoso
On Fri, May 23, 2008 at 10:19 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: On Fri, May 23, 2008 at 6:27 AM, Martin Langhoff [EMAIL PROTECTED] wrote: On Fri, May 23, 2008 at 3:25 PM, Martin Langhoff [EMAIL PROTECTED] wrote: I'll diagnose a bit more and post back. I'm baffled. I initially

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-23 Thread Martin Langhoff
On Fri, May 23, 2008 at 8:19 PM, Tomeu Vizoso [EMAIL PROTECTED] wrote: Ouch, sorry, I should have been more explicit in that this patch introduces a dependency on cjson. I'm going to send next another patch that falls back to simplejson (slower) if cjson is not available. Ah, it was pebkac

Re: Datastore backup - request for help

2008-05-22 Thread Michail Bletsas
John Watlington [EMAIL PROTECTED] wrote on 05/21/2008 10:45:31 PM: Michail is choosing a method, which will be used by NM and should be used by other laptop tasks as well. It will almost certainly not be SSID based. My inclination is to just use a DNS name like schoolserver (that will

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-22 Thread Tomeu Vizoso
On Thu, May 22, 2008 at 5:48 AM, Benjamin M. Schwartz [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jameson Chema Quinn wrote: | - at startup, if the file datastore_path/.metadata.exported doesn't | exist, check how many objects need to get their metadata exported |

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-22 Thread Tomeu Vizoso
On Thu, May 22, 2008 at 1:13 AM, Martin Langhoff [EMAIL PROTECTED] wrote: On Thu, May 22, 2008 at 6:14 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: the patch attached maintains a copy of the metadata of each object outside the xapian index. How it works: Fantastic. Except that... erm... arhm...

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-22 Thread Tomeu Vizoso
On Thu, May 22, 2008 at 5:45 AM, Jameson Chema Quinn [EMAIL PROTECTED] wrote: Yay, I am happy about this patch (when there is a patch :) - at every create and update, a json file is created next to the object's file, I definitely think it should be in the same directory as the object file,

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-22 Thread Martin Langhoff
On Thu, May 22, 2008 at 8:55 PM, Tomeu Vizoso [EMAIL PROTECTED] wrote: This process is easier to perform inside the DS process and will happen only on first boot after update. But the backup process itself (rsync) should have a really high nice value, agreed. That goes without saying: my

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-22 Thread Martin Langhoff
On Thu, May 22, 2008 at 8:47 PM, Tomeu Vizoso [EMAIL PROTECTED] wrote: On Thu, May 22, 2008 at 1:13 AM, Martin Langhoff Fantastic. Except that... erm... arhm... you forgot the patch ;-) Ouch. Thanks for the patch. Quick heads up - I've applied it on a 703 laptop and it's completely borked

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-22 Thread Martin Langhoff
On Fri, May 23, 2008 at 3:25 PM, Martin Langhoff [EMAIL PROTECTED] wrote: I'll diagnose a bit more and post back. I'm baffled. I initially suspected PEBKAC but it keeps happening. With the patch the DS gets renamed, and journal never starts. If the datastore code gets to complain somewhere I

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-22 Thread Ivan Krstić
On May 22, 2008, at 8:25 PM, Martin Langhoff wrote: The first sign of oddness is that the DS dir got renamed to /home/olpc/.sugar/default/datastore1211512411.22 I'll diagnose a bit more and post back. This is due to my patch that was pushed out to mitigate the massive DS outage in

Re: [Server-devel] Datastore backup - request for help

2008-05-22 Thread Michail Bletsas
John Watlington [EMAIL PROTECTED] wrote on 05/21/2008 10:45:31 PM: Michail is choosing a method, which will be used by NM and should be used by other laptop tasks as well. It will almost certainly not be SSID based. My inclination is to just use a DNS name like schoolserver (that will

Re: Datastore backup - request for help

2008-05-21 Thread Bert Freudenberg
On 21.05.2008, at 00:34, Martin Langhoff wrote: On Tue, May 20, 2008 at 11:43 PM, Bert Freudenberg [EMAIL PROTECTED] wrote: Ah, Marco rightly pointed out hal-get-property. Much easier that way: hal-get-property --udi $battery --key battery.charge_level.percentage hal-get-property --udi

Re: Datastore backup - request for help

2008-05-21 Thread Martin Langhoff
On Wed, May 21, 2008 at 11:37 PM, Martin Langhoff [EMAIL PROTECTED] wrote: On Wed, May 21, 2008 at 8:40 PM, Bert Freudenberg [EMAIL PROTECTED] wrote: Now, is there a reasonable way to check for whether we are in a school mesh? No idea. No worries, I cannibalised a bit of bash script from

Re: Datastore backup - request for help

2008-05-21 Thread Ivan Krstić
On May 21, 2008, at 4:41 AM, Martin Langhoff wrote: what I am trying to sort out is the following. With the deployments that have APs instead of AAs, how do we tell school APs from any random AP out there? You don't, because you can't depend on controlling that part of infra. If you need to

[PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-21 Thread Tomeu Vizoso
Hi, On Wed, May 21, 2008 at 1:37 PM, Martin Langhoff [EMAIL PROTECTED] wrote: Any idea if someone can lend a hand with the DS issues I mentioned in my opening post? To recap: - Add a dump all metadata to a file mechanism in datastore/xapianindex.py that is fast. It could be one file per

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-21 Thread Martin Langhoff
On Thu, May 22, 2008 at 6:14 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: the patch attached maintains a copy of the metadata of each object outside the xapian index. How it works: Fantastic. Except that... erm... arhm... you forgot the patch ;-) - at every create and update, a json file is

Re: Datastore backup - request for help

2008-05-21 Thread Martin Langhoff
On Thu, May 22, 2008 at 2:45 PM, John Watlington [EMAIL PROTECTED] wrote: Michail is choosing a method, which will be used by NM and should be used by other laptop tasks as well. It will almost certainly not be SSID based. Cool. I'll keep my ears open - Michail, I assume you'll ping me when

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-21 Thread Jameson Chema Quinn
Yay, I am happy about this patch (when there is a patch :) - at every create and update, a json file is created next to the object's file, I definitely think it should be in the same directory as the object file, with a related name. It might even be worth using the macintosh ._name naming

Re: [PATCH] Maintain a metadata copy outside the index (was Re: Datastore backup - request for help)

2008-05-21 Thread Benjamin M. Schwartz
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jameson Chema Quinn wrote: | - at startup, if the file datastore_path/.metadata.exported doesn't | exist, check how many objects need to get their metadata exported | (0.8s for 3000 entries) | That's pretty good. | | - in an idle callback, process

Re: [Server-devel] Datastore backup - request for help

2008-05-21 Thread Martin Langhoff
On Thu, May 22, 2008 at 2:45 PM, John Watlington [EMAIL PROTECTED] wrote: Michail is choosing a method, which will be used by NM and should be used by other laptop tasks as well. It will almost certainly not be SSID based. Cool. I'll keep my ears open - Michail, I assume you'll ping me when

Re: [Server-devel] Datastore backup - request for help

2008-05-21 Thread Martin Langhoff
On Thu, May 22, 2008 at 3:43 PM, John Watlington [EMAIL PROTECTED] wrote: That question was asked at the school server presentation today, by someone from Teaching Matters (IIRC) who knew of you. They commented that 1GB seemed very low for 100 students where Moodle was involved. That's for

Re: Datastore backup - request for help

2008-05-20 Thread Martin Langhoff
On Tue, May 20, 2008 at 5:43 PM, Martin Langhoff [EMAIL PROTECTED] wrote: Now that I have ds_backup that does backup something (thanks Tomeu for the script to grow the ds), I am looking at doing it correctly. Couple more things that I want to do, and I am unsure where to hook into: - On

Re: Datastore backup - request for help

2008-05-20 Thread Tomeu Vizoso
On Tue, May 20, 2008 at 9:47 AM, Martin Langhoff [EMAIL PROTECTED] wrote: On Tue, May 20, 2008 at 5:43 PM, Martin Langhoff [EMAIL PROTECTED] wrote: Now that I have ds_backup that does backup something (thanks Tomeu for the script to grow the ds), I am looking at doing it correctly. Couple

Re: Datastore backup - request for help

2008-05-20 Thread Martin Langhoff
On Tue, May 20, 2008 at 8:16 PM, Tomeu Vizoso [EMAIL PROTECTED] wrote: What about ~/.sugar/default? Ok. I can store the flag straight there. For the locks, it might make sense to make a lock directory, as those have a different semantic (ie: you never want to copy them in a backup, for example).

Re: Datastore backup - request for help

2008-05-20 Thread Marco Pesenti Gritti
On Tue, May 20, 2008 at 10:48 AM, Martin Langhoff [EMAIL PROTECTED] wrote: What about using HAL? Sugar uses it like this: http://dev.laptop.org/git?p=sugar;a=blob;f=src/model/devices/battery.py;h=853d00ec490a0b58ef7f569ac1c9a9ef75228259;hb=HEAD Looks like a ton of work. cat

Re: Datastore backup - request for help

2008-05-20 Thread Martin Langhoff
On Tue, May 20, 2008 at 8:58 PM, Marco Pesenti Gritti [EMAIL PROTECTED] wrote: But it works only on the XO. Fair enough. Is there a portable (and lightweight) way of asking this question from bash? cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask

Re: Datastore backup - request for help

2008-05-20 Thread Bert Freudenberg
On 20.05.2008, at 11:31, Martin Langhoff wrote: On Tue, May 20, 2008 at 8:58 PM, Marco Pesenti Gritti [EMAIL PROTECTED] wrote: But it works only on the XO. Fair enough. Is there a portable (and lightweight) way of asking this question from bash? battery=`hal-find-by-capability

Re: Datastore backup - request for help

2008-05-20 Thread Bert Freudenberg
On 20.05.2008, at 12:58, Bert Freudenberg wrote: On 20.05.2008, at 11:31, Martin Langhoff wrote: On Tue, May 20, 2008 at 8:58 PM, Marco Pesenti Gritti [EMAIL PROTECTED] wrote: But it works only on the XO. Fair enough. Is there a portable (and lightweight) way of asking this question

Re: Datastore backup - request for help

2008-05-20 Thread Martin Langhoff
On Tue, May 20, 2008 at 11:43 PM, Bert Freudenberg [EMAIL PROTECTED] wrote: Ah, Marco rightly pointed out hal-get-property. Much easier that way: hal-get-property --udi $battery --key battery.charge_level.percentage hal-get-property --udi $ac_adapter --key ac_adapter.present Great! Each

Datastore backup - request for help

2008-05-19 Thread Martin Langhoff
Now that I have ds_backup that does backup something (thanks Tomeu for the script to grow the ds), I am looking at doing it correctly. I've published what I have at git://dev.laptop.org/users/martin/ds-backup.git , visible at http://dev.laptop.org/git?p=users/martin/ds-backup.git;a=summary (For