On Thu, May 22, 2008 at 6:14 AM, Tomeu Vizoso <[EMAIL PROTECTED]> wrote: > the patch attached maintains a copy of the metadata of each object > outside the xapian index. How it works:
Fantastic. Except that... erm... arhm... you forgot the patch ;-) > - at every create and update, a json file is created next to the object's > file, > > - it's also deleted along the object, > > - at startup, if the file <datastore_path>/.metadata.exported doesn't > exist, check how many objects need to get their metadata exported > (0.8s for 3000 entries) That's pretty good. > - in an idle callback, process each of those objects one per iteration > (3ms per entry with simplejson, 2ms with cjson). Exporting a few 100 per iteration probably is more efficient ;-) > In my tests this has worked quite well, but I have one concern: can > something bad happen if we have 20k files in the same dir (for a > journal with 10k entries)? Ok, we can split it into a subdir (which will only have 10K files then). If there's a cost to large dirs in jfffs2 then we can use hashed dirs, and that change will be needed for both the main datastore storage _and_ the metadata files. > One side effect of this is that when (if) we agree on a new on-disk > data structure for the DS, it will be easier to convert than if we had > to extract all the metadata from the index. Yes. And as you said earlier, easy recovery if xapian goes to la-la land. cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff _______________________________________________ Devel mailing list [email protected] http://lists.laptop.org/listinfo/devel
