On Sat, Nov 21, 2009 at 05:36:18PM -0500, Brett Viren wrote: > On Sat, Nov 21, 2009 at 12:07 PM, Carl Worth <cworth at cworth.org> wrote: > > > Though, frankly, I think we need to fix "notmuch new" to do much better > > than 40 files/sec. > > Just a "me too". > > Processed 130871 total files in 38m 7s (57 files/sec.). > Added 102723 new messages to the database (not much, really). > > This was ~2GB of mail on a 2.5GHz CPU. That seems pretty reasonable > to me but I'd like to rerun the "notmuch new" under google perftools > to see if there are any obvious bottlenecks that might be cleaned up.
FWIW, my 90k+ messages mailbox was imported at a pace of 130 files/sec, and my CPU is "only" 2.2GHz, but I have a SSD. A good share of the bottlenecks is "simply" I/O. Don't forget having a lot of small files sucks I/O wise, as files are most likely spread all over the disk. A good test, if you have enough memory, would be to put your mailbox in a tmpfs, and see how fast that imports. Mike