On Sat, Aug 11, 2012 at 11:50 PM, Jameson Graef Rollins
<jrollins at finestructure.net> wrote:
> On Sat, Aug 11 2012, Ciprian Dorin Craciun <ciprian.craciun at gmail.com>
> wrote:
>> My problem with it is that it doesn't scale... And I don't mean
>> this in a theoretical sense, I mean it in the concrete one: I have
>> about 661k emails... And a single `notmuch sync` takes a few tens of
>> seconds...
>
> Hey, Ciprian. That sounds really slow, which makes me wonder if there
> are other things going on here.
> I have 155k messages, but notmuch new
> takes a fraction of a second for me. This initial indexing certainly
> takes a long time (hours potentially), but additions after that should
> be really fast. What version of notmuch are you using? What version of
> xapian?
Don't think there is anything wrong here... Its just drags with
the file system...
So just to give a complete info:
* hardware: Core i5, 8GiB RAM (7.5GiB of which is the FS cache),
SSD (about 175MiB raw disk access);
* `notmuch --version`: 0.13 (built from sources on latest ArchLinux);
* `notmuch count`: 701820;
* `notmuch new` (after adding 5925 new emails, at touching others):
~~~~
Processed 7017 total files in 3m 19s (35 files/sec.).
Added 6061 new messages to the database. Detected 1116 file renames.
~~~~
* actually the entire thing took almost 5 minutes, but the first
two it didn't display anything just acesing the disk;
* `notmuch new` (another go, but this time I've `time`-d it):
~~~~
No new mail.
real 0m40.546s
user 0m4.523s
sys 0m17.506s
~~~~
* `notmuch new` (yet another go, no change):
~~~~
No new mail.
real 0m39.190s
user 0m4.229s
sys 0m17.697s
~~~~
* just to `du` the maildir (there are also 40k other files in
other maildirs not included in this count):
~~~~
8.7G ......
real 0m22.229s
user 0m1.023s
sys 0m7.890s
~~~~
* on `new` no hooks are run;
* the file system in cause is JFS;
As such I doubt the problem is with notmuch itself, and I guess
it's the file system interaction...
Now I know I have a really obscure corner case, and I'm positively
amazed on how good notmuch handles this situation. I just wandered if
I could have fixed my problem by moving to an embedded DB, thus
skipping all that syscall overhead...
Ciprian.
