Jamie McCracken wrote: > On Fri, 2008-08-08 at 12:49 +0100, Martyn Russell wrote: >> Laurent Aguerreche wrote: >>> Hello, >> Hi :) >> >>> Le jeudi 07 août 2008 à 16:38 +0100, Martyn Russell a écrit : >>>> Jamie, >>>> >>>> I think we are ready for merge if you are happy with the current state >>>> of the indexer-split branch. >>>> >>>> If you find anything you think needs fixing before the merge. Let us know. >>> I'm experiencing two issues with the indexer-split branch : >>> - if I terminate trackerd with "kill" (to send a SIGTERM signal) sent to >>> it, I see tracker-indexer still running. I can't kill tracker-indexer >>> with "kill", I have to use "kill -9" >>> - tracker kills the performances of my hard-drive while it starts: GUIs >>> of my applications get frozen! >>> >>> Do you intend to fix these issues before the merge (notably the second >>> one...)? >> We have been fixing a few things up today that I noticed last night. >> Issues include: >> >> 1.) Index being too big that indexing is slowed right down: > > this is why trunk used index merging so it would flush to separate > indexes and then merge them into one when finished. > > By experimentation 16MB is a good size for an index hence trunk flushed > in 16mb lumps
Yea, that would fix things. Right now we are trying to evaluate if it is worth doing that or just upgrading to SQLite with FTS. What are your thoughts here Jamie. Would you rather have time spent doing this QDBM work or the SQLite work? I would be inclined to say we should merge and immediately start working on the SQLite work. > doing lots of small writes to a large index causes real IO problems with > ext/3 (ext/4 fixes this) - each word being indexed is a separate seek > and write on the index so you could easily have 100,000+ small updates > being applied. Also when words have hits added the hashtable has to > relocate them to free space which also causes fragmentation - the > optimize qdbm call recovers this by copying the index into a new one > once indexing is finished > > as you can see index merging eliminates this perfomance overhead or the > need to call optimize as the final merged index will have no > fragmentation Yea. Have you had time to look at the rest of the branch to see if it is satisfactory? -- Regards, Martyn _______________________________________________ tracker-list mailing list [email protected] http://mail.gnome.org/mailman/listinfo/tracker-list
