On Sat, 21 Nov 2009 17:36:18 -0500, Brett Viren <brett.viren at gmail.com> wrote: > Processed 130871 total files in 38m 7s (57 files/sec.). > Added 102723 new messages to the database (not much, really).
Just be glad that you have so little mail. ;-) > This was ~2GB of mail on a 2.5GHz CPU. That seems pretty reasonable > to me but I'd like to rerun the "notmuch new" under google perftools > to see if there are any obvious bottlenecks that might be cleaned up. To me, here are the obvious things to fix after looking at a profile: 1. We're spending a *lot* of time searching in the Xapian database. But our initial indexing operation should only be *writing* data into the database, so what's this searching about? Well, at each new message, we're looking up the ID from it's In-Reply-To header to find a thread-ID to link to, and then we're looking up all of the IDs from its References header to find thread IDs that need to be merged with ours. So both parent and child lookups. And since those are taking a bunch of time, I think it might make sense to just keep a hashtable mapping message-ID -> thread-ID and do lookups in that, (should have plenty of memory on current machines even with lots of mail). 2. We're hitting the slow Xapian document updates for thread-ID merging. Whenever we find a child that was already in the database with one thread ID that should have ours, we simply want to set its thread ID to ours. But as we've talked about recently, Xapian has a bug (defect 250) that makes it much more expensive than it should be to update a single term. So, we could do a first pass over the messages to find all their thread IDs and get them to settle down before doing any indexing in a separate, second pass. Step (2) should help even if we don't do step (1), but clearly we can do both. It would be great if anyone wants to take a look at either or both of these, otherwise I will when I can. -Carl