On Mon, 2008-08-11 at 10:55 +0100, Martyn Russell wrote: > Jamie McCracken wrote: > > On Fri, 2008-08-08 at 12:49 +0100, Martyn Russell wrote: > >> Laurent Aguerreche wrote: > >>> Hello, > >> Hi :) > >> > >>> Le jeudi 07 août 2008 à 16:38 +0100, Martyn Russell a écrit : > >>>> Jamie, > >>>> > >>>> I think we are ready for merge if you are happy with the current state > >>>> of the indexer-split branch. > >>>> > >>>> If you find anything you think needs fixing before the merge. Let us > >>>> know. > >>> I'm experiencing two issues with the indexer-split branch : > >>> - if I terminate trackerd with "kill" (to send a SIGTERM signal) sent to > >>> it, I see tracker-indexer still running. I can't kill tracker-indexer > >>> with "kill", I have to use "kill -9" > >>> - tracker kills the performances of my hard-drive while it starts: GUIs > >>> of my applications get frozen! > >>> > >>> Do you intend to fix these issues before the merge (notably the second > >>> one...)? > >> We have been fixing a few things up today that I noticed last night. > >> Issues include: > >> > >> 1.) Index being too big that indexing is slowed right down: > > > > this is why trunk used index merging so it would flush to separate > > indexes and then merge them into one when finished. > > > > By experimentation 16MB is a good size for an index hence trunk flushed > > in 16mb lumps > > Yea, that would fix things. > > Right now we are trying to evaluate if it is worth doing that or just > upgrading to SQLite with FTS. > > What are your thoughts here Jamie. Would you rather have time spent > doing this QDBM work or the SQLite work?
I am doing the sqlite fts fork work atm I guess the merging can wait as there is a lot of more important stuff > > I would be inclined to say we should merge and immediately start working > on the SQLite work. > > > doing lots of small writes to a large index causes real IO problems with > > ext/3 (ext/4 fixes this) - each word being indexed is a separate seek > > and write on the index so you could easily have 100,000+ small updates > > being applied. Also when words have hits added the hashtable has to > > relocate them to free space which also causes fragmentation - the > > optimize qdbm call recovers this by copying the index into a new one > > once indexing is finished > > > > as you can see index merging eliminates this perfomance overhead or the > > need to call optimize as the final merged index will have no > > fragmentation > > Yea. > > Have you had time to look at the rest of the branch to see if it is > satisfactory? > unfortunately it is not Running it I get very poor performance (Im running tracker-applet so dont know if that is affecting/pausing it) Does not handle file moves (see trunk for this as its non trivial) The gio file monitoring is not adequate - inotify needs optimisation EG for inotify we only trigger a change if FILE_WRITE_CLOSED is sent - that way we know the file has finished changing and we can safely index. Currently it indexes whenever the file changes which sucks. FAM would need a pending list (as per trunk) as it only has the file change and we need to wait til changes are stabilised before trigger an index Also we are a differential index so i dont understand why you have removed the differential update code? What does it do when a file updates? Im seeing lots of issues so merge has to wait...still compiling list of problems jamie _______________________________________________ tracker-list mailing list [email protected] http://mail.gnome.org/mailman/listinfo/tracker-list
