[notmuch] Notmuch performance problems on OSX
On Tue, Feb 9, 2010 at 10:09 PM, Olly Betts wrote: > On 2010-02-09, Oliver Charles wrote: >> I just upgraded to xapian-core HEAD and notmuch master tip today, in >> desparation to get away from GMail. Sadly it's still taking at least >> 0.7s to tag a single thread (with one message). I'm really eager to >> solve this, could anyone give me any pointers on how I could go about >> profiling it or finding the cause of this problem? > > The first thing to try is disabling use of F_FULLFSYNC. You'll need to > run this command in the xapian-core source tree to comment out the F_FULLFSYNC > code: > > perl -pi -e 's/^#ifdef F_FULLFSYNC/#if 0/' backends/*/*_io.h > > Then run "make" and "make install". $ time notmuch tag +inbox thread:6e66368b7887184c6d4c63653211b3f2 real0m0.067s user0m0.036s sys 0m0.028s Now this looks a little bit more usable! > Assuming that helps, then (a) you have a workaround, and (b) we'll know for > sure it is F_FULLFSYNC to blame. Looks like this is the case. > I've created a ticket for a change to Xapian which should help here, but > not had a chance to work on it yet: > > http://trac.xapian.org/ticket/426 I will add my info there and follow the ticket if I can. -- Oliver Charles / aCiD2 (Olly, sorry about the double email - in all my excitement I didn't hit reply all :))
[notmuch] Notmuch performance problems on OSX
I just upgraded to xapian-core HEAD and notmuch master tip today, in desparation to get away from GMail. Sadly it's still taking at least 0.7s to tag a single thread (with one message). I'm really eager to solve this, could anyone give me any pointers on how I could go about profiling it or finding the cause of this problem? -- Oliver Charles / aCiD2
Re: [notmuch] Notmuch performance problems on OSX
On Tue, Feb 9, 2010 at 10:09 PM, Olly Betts o...@survex.com wrote: On 2010-02-09, Oliver Charles wrote: I just upgraded to xapian-core HEAD and notmuch master tip today, in desparation to get away from GMail. Sadly it's still taking at least 0.7s to tag a single thread (with one message). I'm really eager to solve this, could anyone give me any pointers on how I could go about profiling it or finding the cause of this problem? The first thing to try is disabling use of F_FULLFSYNC. You'll need to run this command in the xapian-core source tree to comment out the F_FULLFSYNC code: perl -pi -e 's/^#ifdef F_FULLFSYNC/#if 0/' backends/*/*_io.h Then run make and make install. $ time notmuch tag +inbox thread:6e66368b7887184c6d4c63653211b3f2 real0m0.067s user0m0.036s sys 0m0.028s Now this looks a little bit more usable! Assuming that helps, then (a) you have a workaround, and (b) we'll know for sure it is F_FULLFSYNC to blame. Looks like this is the case. I've created a ticket for a change to Xapian which should help here, but not had a chance to work on it yet: http://trac.xapian.org/ticket/426 I will add my info there and follow the ticket if I can. -- Oliver Charles / aCiD2 (Olly, sorry about the double email - in all my excitement I didn't hit reply all :)) ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[notmuch] A functional (but rudimentary) test suite for notmuch
On Thu, Feb 4, 2010 at 11:29 PM, Carl Worth wrote:
> Looking at TAP, one thing I don't like is that it prints the
> success/failure of the test first, before the description of the
> test. That's not so nice in the case of a long-running (perhaps
> infinitely running) test where you might need to interrupt it, but you'd
> still want to know *what* was running for so long.
Ah, the way this works for Test::Harness in Perl is that while a test
suite is running the output is of the form:
t/edit_artist_add_alias.t .. 1/?
And then when it completes it will change to:
t/edit_artist_add_alias.t .. OK
So you can see stuff that's failing to terminate (as I just found
while running my test suites ;))
--
Oliver Charles / aCiD2
[notmuch] strange behavior of indexing of and searching for strings containing '[]'
On Fri, Feb 5, 2010 at 4:44 PM, Jameson Rollins
wrote:
> Does anyone have any idea what's going on here? ?I think I saw mention
> of this issue on IRC somewhere, but I thought I should bring it up
> explicitly here. ?This is definitely some buggy behavior.
Afaik, stuff in between [] is not indexed, but that doesn't quite
explain the other weird results.
--
Oliver Charles / aCiD2
[notmuch] A functional (but rudimentary) test suite for notmuch
Carl, have you considered outputting the test suite in the same format as the test anything protocol? [1] I only mention this because it might be a nice way to easily do some reporting (or perhaps even continuous integration) notmuch, with trivial effort. -- Oliver Charles / aCiD2 [1]: http://en.wikipedia.org/wiki/Test_Anything_Protocol
[notmuch] Notmuch performance problems on OSX
On Thu, Jan 14, 2010 at 11:16 PM, Carl Worth wrote: > Hi Oliver, welcome to notmuch! > > On Thu, 14 Jan 2010 15:30:48 +0000, Oliver Charles googlemail.com> wrote: >> I've installed the latest notmuch from Git at this time of writing, >> along with Xapian from SVN head. However, just tagging a single thread >> with only one message seems to take too long: >> >> $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221 >> >> real ?0m0.812s >> user ?0m0.022s >> sys ? 0m0.037s > > Things work quite a bit faster than that on my machine: > > $ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314 at > mail.gmail.com > > real ? ?0m0.024s > user ? ?0m0.012s > sys ? ? 0m0.004s > > But that could just be system differences. Possibly, though my machine is more than capable of what it's doing - a Q6600 with 4gb RAM >> And tagging all my messages is really horrible: >> >> $ time notmuch tag +foobar tag:inbox >> >> real ?0m5.076s >> user ?0m3.688s >> sys ? 0m0.105s > > For this operation, I can't really compare. How many messages are you > tagging? Here's that operation for me with 525 messages in my inbox: A few thousand (4k, I believe) >> That xapian-svn was built from svn HEAD right now, so I'm assuming it >> contains the #250 fix (http://trac.xapian.org/changeset/13808) > > Which I think means that things could have been even *much* slower > before. ;-) > > The Xapian defect #250 was just one, initial (and obvious) performance > problem. [Though, as I mentioned in a previous thread, if you're using a > Xapian flint database, (look for .notmuch/xapian/iamflint), then you > won't get the benefit of the Xapian fix until you rebuild your notmuch > database from scratch with a current notmuch.] I didn't know about this need to rebuild, but I tried that and didn't have any more success sadly. > Once you've verified that you've got the #250 fix functional, there > could still be lots of performance bugs. And it would be time to start > profiling. > > [...] I'm pressed for time at the moment, but in a few weeks I might have some time to investigate here... -- Oliver Charles / aCiD2
Re: [notmuch] Notmuch performance problems on OSX
On Thu, Jan 14, 2010 at 11:16 PM, Carl Worth cwo...@cworth.org wrote: Hi Oliver, welcome to notmuch! On Thu, 14 Jan 2010 15:30:48 +, Oliver Charles oliver.g.char...@googlemail.com wrote: I've installed the latest notmuch from Git at this time of writing, along with Xapian from SVN head. However, just tagging a single thread with only one message seems to take too long: $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221 real 0m0.812s user 0m0.022s sys 0m0.037s Things work quite a bit faster than that on my machine: $ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d...@mail.gmail.com real 0m0.024s user 0m0.012s sys 0m0.004s But that could just be system differences. Possibly, though my machine is more than capable of what it's doing - a Q6600 with 4gb RAM And tagging all my messages is really horrible: $ time notmuch tag +foobar tag:inbox real 0m5.076s user 0m3.688s sys 0m0.105s For this operation, I can't really compare. How many messages are you tagging? Here's that operation for me with 525 messages in my inbox: A few thousand (4k, I believe) That xapian-svn was built from svn HEAD right now, so I'm assuming it contains the #250 fix (http://trac.xapian.org/changeset/13808) Which I think means that things could have been even *much* slower before. ;-) The Xapian defect #250 was just one, initial (and obvious) performance problem. [Though, as I mentioned in a previous thread, if you're using a Xapian flint database, (look for .notmuch/xapian/iamflint), then you won't get the benefit of the Xapian fix until you rebuild your notmuch database from scratch with a current notmuch.] I didn't know about this need to rebuild, but I tried that and didn't have any more success sadly. Once you've verified that you've got the #250 fix functional, there could still be lots of performance bugs. And it would be time to start profiling. [...] I'm pressed for time at the moment, but in a few weeks I might have some time to investigate here... -- Oliver Charles / aCiD2 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[notmuch] Notmuch performance problems on OSX
Actually, significant performance problems. Ho ho ho. (sorry) I've installed the latest notmuch from Git at this time of writing, along with Xapian from SVN head. However, just tagging a single thread with only one message seems to take too long: $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221 real0m0.812s user0m0.022s sys 0m0.037s And tagging all my messages is really horrible: $ time notmuch tag +foobar tag:inbox real0m5.076s user0m3.688s sys 0m0.105s Here is what my notmuch binary links with: $ otool -L /usr/local/bin/notmuch /usr/local/bin/notmuch: /usr/local/Cellar/gmime/2.4.0/lib/libgmime-2.4.2.dylib (compatibility version 7.0.0, current version 7.0.0) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3) /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0) /usr/local/Cellar/glib/2.20.5/lib/libgobject-2.0.0.dylib (compatibility version 2001.0.0, current version 2001.5.0) /usr/local/Cellar/glib/2.20.5/lib/libglib-2.0.0.dylib (compatibility version 2001.0.0, current version 2001.5.0) /usr/local/Cellar/gettext/0.17/lib/libintl.8.dylib (compatibility version 9.0.0, current version 9.2.0) /usr/local/Cellar/xapian-svn/HEAD/lib/libxapian-1.1.3.dylib (compatibility version 4.0.0, current version 4.0.0) /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0) /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.1.4) That xapian-svn was built from svn HEAD right now, so I'm assuming it contains the #250 fix (http://trac.xapian.org/changeset/13808) Any ideas? -- Oliver Charles / aCiD2
