[notmuch] Notmuch performance problems on OSX
On Thu, Jan 14, 2010 at 11:16 PM, Carl Worth wrote: > Hi Oliver, welcome to notmuch! > > On Thu, 14 Jan 2010 15:30:48 +, Oliver Charles googlemail.com> wrote: >> I've installed the latest notmuch from Git at this time of writing, >> along with Xapian from SVN head. However, just tagging a single thread >> with only one message seems to take too long: >> >> $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221 >> >> real ?0m0.812s >> user ?0m0.022s >> sys ? 0m0.037s > > Things work quite a bit faster than that on my machine: > > $ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314 at > mail.gmail.com > > real ? ?0m0.024s > user ? ?0m0.012s > sys ? ? 0m0.004s > > But that could just be system differences. Possibly, though my machine is more than capable of what it's doing - a Q6600 with 4gb RAM >> And tagging all my messages is really horrible: >> >> $ time notmuch tag +foobar tag:inbox >> >> real ?0m5.076s >> user ?0m3.688s >> sys ? 0m0.105s > > For this operation, I can't really compare. How many messages are you > tagging? Here's that operation for me with 525 messages in my inbox: A few thousand (4k, I believe) >> That xapian-svn was built from svn HEAD right now, so I'm assuming it >> contains the #250 fix (http://trac.xapian.org/changeset/13808) > > Which I think means that things could have been even *much* slower > before. ;-) > > The Xapian defect #250 was just one, initial (and obvious) performance > problem. [Though, as I mentioned in a previous thread, if you're using a > Xapian flint database, (look for .notmuch/xapian/iamflint), then you > won't get the benefit of the Xapian fix until you rebuild your notmuch > database from scratch with a current notmuch.] I didn't know about this need to rebuild, but I tried that and didn't have any more success sadly. > Once you've verified that you've got the #250 fix functional, there > could still be lots of performance bugs. And it would be time to start > profiling. > > [...] I'm pressed for time at the moment, but in a few weeks I might have some time to investigate here... -- Oliver Charles / aCiD2
[notmuch] Introducing notmuchsync
On Mon, 18 Jan 2010 16:41:25 +0100, Michal Sojka wrote: > with which Python version is this meant to work? I get the following error: > $ ./notmuchsync -r > AttributeError: 'str' object has no attribute 'format' DOH, I just used 2.6-style formatting. But as there is little reason for this, I just reverted to old-style formatting. Just pull from git again. I have no clue about the needed version, but it should definitely be <2.6 now. spaetz
[notmuch] Introducing notmuchsync
On Mon, 18 Jan 2010 16:12:28 +0100, "Sebastian Spaeth" wrote: > What does it do? > > - Synchronizes the "S" flag with the "unread" tag (1-way). The > synchronization direction is decided by using either --sync (change > maildir flags according to notmuch) or --revsync (change notmuch tags > according to maildir). By default it always checks the mails from the > previous 30 > days (but can also do --all mails if you have plenty of RAM and time). > - Deletes all mail files that have the "delete" tag > - Quiet/normal/verbose logging Excellent. Sounds like nearly exactly what I thought about doing last weekend, except that I couldn't find a nice Ruby library quickly and then my attention-span was over :) Also, I'm curious as to Carl's opinion to this, but as far as I'm concerned, not everything about notmuch needs to be coded in C. Obviously, things interacting with the database directly need to, but if it could be built upon the notmuch C-based CLI commands, and be fast enough, would that be eligible to make it into the repository? -- - Marten
[notmuch] Introducing notmuchsync
On Monday 18 of January 2010 16:12:28 Sebastian Spaeth wrote:
> Dear list,
>
> I really want to sync my maildir flags/folders with notmuch tags and as
> I haven't seen a script to do that, I've written (the beginnings of?)
> one.
>
> It is probably pre-alpha but I am a fan of release early, release often
> and if someone finds it useful to build upon, it was already worth it. I
> put the code here: http://github.com/spaetz/notmuchsync
Hi,
with which Python version is this meant to work? I get the following error:
$ ./notmuchsync -r
Traceback (most recent call last):
File "./notmuchsync", line 316, in
notmuch.syncTags(frommaildir=True, dryrun=dryrun)
File "./notmuchsync", line 249, in syncTags
searchterm = "{0}..{1}".format(now-2592000,now+2592000)
AttributeError: 'str' object has no attribute 'format'
Michal
[notmuch] Introducing notmuchsync
On Mon, 18 Jan 2010 16:12:28 +0100, "Sebastian Spaeth" wrote: DOH, I should actually have finished the Workflow section before sending it off :-): Workflow 1a) start out with notmuchsync -r in order to initialize the notmuch tag database based on maildir 'S' flags. This is probably what everyone but Carl Worth wants. :-) If the "--all" command works for you, even better, but I doubt it as it would store all your mails in RAM. I'll need to work on some chunking for this. 1b) start out with nnotmuchsync -s if your notmuch tags are authoritative and it will modify your Maildir flags accordingly. After that a periodic: 2) "notmuchsync -s" syncs tags to flags (Currently 'S' only) 3) "notmuchsync -p" purges mail files with "delete" flags
[notmuch] Introducing notmuchsync
Dear list, I really want to sync my maildir flags/folders with notmuch tags and as I haven't seen a script to do that, I've written (the beginnings of?) one. It is probably pre-alpha but I am a fan of release early, release often and if someone finds it useful to build upon, it was already worth it. I put the code here: http://github.com/spaetz/notmuchsync What does it do? - Synchronizes the "S" flag with the "unread" tag (1-way). The synchronization direction is decided by using either --sync (change maildir flags according to notmuch) or --revsync (change notmuch tags according to maildir). By default it always checks the mails from the previous 30 days (but can also do --all mails if you have plenty of RAM and time). - Deletes all mail files that have the "delete" tag - Quiet/normal/verbose logging Workflow 1a) start out with notmuchsync -r in order to initialize the notmuch tag database based on maildir 'S' flags. This is probably what everyone but Carl Worth wants. :-) If the "--all" command works for you, even better, but I doubt it as it would store all your mails in RAM. 1b) start out with not What needs improvment = - It's a python script in one file. The architecture might need some cleanup. Documentation needs work too. - It temporarily slurps in all your mails from the last 30 days into RAM. I am waiting for "notmuchs show blah --output filename --output tags" to improve that :). Generally the parsing of the output of "notmuch show" is a bit hackyish with regexps at the moment. - Support parsing "chunks" of "notmuch show" output or do several runs for -all, using monthly intervals to make --all work. - Support for the "T" (delete), "F" (flag), "D" (draft) tags/flags. Should be pretty easy to add. - Support syncing of the "inbox" tag if the mail is in a dir different than INBOX. No clue yet how to do. Perhaps if a maildir name corresponds to a notmuch tag name, we move the mail there? - It seems to work for me, but is untested in environments with spaces in paths and windows machines. - the --doeverything command that prunes and syncs in one go What else = - It is GPL v2.1+ - Please improve and/or fork this thing. - If you don't like it, just ignore it. Thanks for your understanding. --- Usage: -p --prune Prune deleted mails -s --sync Sync from notmuch tags to maildir flags. By default it will only look for mails from the last 30 days. Use --all to look at earlier mails. Beware, if timestamps are more than 30 days in the future, we won't handle it. -r --revsyncSync tags from maildir to notmuch. See also -s. Options: -d Really verbose debug oputput. -q Log only errors. --dryrun Do not really modify notmuch db or mail files Works together with -p -s -r -- Let me know what you like or don't. If you want github contributor right, I am willing to hand out access quite liberal. And as always: patches welcome. :-) spaetz
Re: [notmuch] Notmuch performance problems on OSX
On Thu, Jan 14, 2010 at 11:16 PM, Carl Worth wrote: > Hi Oliver, welcome to notmuch! > > On Thu, 14 Jan 2010 15:30:48 +, Oliver Charles > wrote: >> I've installed the latest notmuch from Git at this time of writing, >> along with Xapian from SVN head. However, just tagging a single thread >> with only one message seems to take too long: >> >> $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221 >> >> real 0m0.812s >> user 0m0.022s >> sys 0m0.037s > > Things work quite a bit faster than that on my machine: > > $ time notmuch tag +foo > id:5641883d1001140730l22832715ld6bdc95c9938d...@mail.gmail.com > > real 0m0.024s > user 0m0.012s > sys 0m0.004s > > But that could just be system differences. Possibly, though my machine is more than capable of what it's doing - a Q6600 with 4gb RAM >> And tagging all my messages is really horrible: >> >> $ time notmuch tag +foobar tag:inbox >> >> real 0m5.076s >> user 0m3.688s >> sys 0m0.105s > > For this operation, I can't really compare. How many messages are you > tagging? Here's that operation for me with 525 messages in my inbox: A few thousand (4k, I believe) >> That xapian-svn was built from svn HEAD right now, so I'm assuming it >> contains the #250 fix (http://trac.xapian.org/changeset/13808) > > Which I think means that things could have been even *much* slower > before. ;-) > > The Xapian defect #250 was just one, initial (and obvious) performance > problem. [Though, as I mentioned in a previous thread, if you're using a > Xapian flint database, (look for .notmuch/xapian/iamflint), then you > won't get the benefit of the Xapian fix until you rebuild your notmuch > database from scratch with a current notmuch.] I didn't know about this need to rebuild, but I tried that and didn't have any more success sadly. > Once you've verified that you've got the #250 fix functional, there > could still be lots of performance bugs. And it would be time to start > profiling. > > [...] I'm pressed for time at the moment, but in a few weeks I might have some time to investigate here... -- Oliver Charles / aCiD2 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [notmuch] Introducing notmuchsync
On Mon, 18 Jan 2010 16:41:25 +0100, Michal Sojka wrote: > with which Python version is this meant to work? I get the following error: > $ ./notmuchsync -r > AttributeError: 'str' object has no attribute 'format' DOH, I just used 2.6-style formatting. But as there is little reason for this, I just reverted to old-style formatting. Just pull from git again. I have no clue about the needed version, but it should definitely be <2.6 now. spaetz ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [notmuch] Introducing notmuchsync
On Mon, 18 Jan 2010 16:12:28 +0100, "Sebastian Spaeth" wrote: > What does it do? > > - Synchronizes the "S" flag with the "unread" tag (1-way). The > synchronization direction is decided by using either --sync (change > maildir flags according to notmuch) or --revsync (change notmuch tags > according to maildir). By default it always checks the mails from the > previous 30 > days (but can also do --all mails if you have plenty of RAM and time). > - Deletes all mail files that have the "delete" tag > - Quiet/normal/verbose logging Excellent. Sounds like nearly exactly what I thought about doing last weekend, except that I couldn't find a nice Ruby library quickly and then my attention-span was over :) Also, I'm curious as to Carl's opinion to this, but as far as I'm concerned, not everything about notmuch needs to be coded in C. Obviously, things interacting with the database directly need to, but if it could be built upon the notmuch C-based CLI commands, and be fast enough, would that be eligible to make it into the repository? -- - Marten ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [notmuch] Introducing notmuchsync
On Monday 18 of January 2010 16:12:28 Sebastian Spaeth wrote:
> Dear list,
>
> I really want to sync my maildir flags/folders with notmuch tags and as
> I haven't seen a script to do that, I've written (the beginnings of?)
> one.
>
> It is probably pre-alpha but I am a fan of release early, release often
> and if someone finds it useful to build upon, it was already worth it. I
> put the code here: http://github.com/spaetz/notmuchsync
Hi,
with which Python version is this meant to work? I get the following error:
$ ./notmuchsync -r
Traceback (most recent call last):
File "./notmuchsync", line 316, in
notmuch.syncTags(frommaildir=True, dryrun=dryrun)
File "./notmuchsync", line 249, in syncTags
searchterm = "{0}..{1}".format(now-2592000,now+2592000)
AttributeError: 'str' object has no attribute 'format'
Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: [notmuch] Introducing notmuchsync
On Mon, 18 Jan 2010 16:12:28 +0100, "Sebastian Spaeth" wrote: DOH, I should actually have finished the Workflow section before sending it off :-): Workflow 1a) start out with notmuchsync -r in order to initialize the notmuch tag database based on maildir 'S' flags. This is probably what everyone but Carl Worth wants. :-) If the "--all" command works for you, even better, but I doubt it as it would store all your mails in RAM. I'll need to work on some chunking for this. 1b) start out with nnotmuchsync -s if your notmuch tags are authoritative and it will modify your Maildir flags accordingly. After that a periodic: 2) "notmuchsync -s" syncs tags to flags (Currently 'S' only) 3) "notmuchsync -p" purges mail files with "delete" flags ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[notmuch] Introducing notmuchsync
Dear list, I really want to sync my maildir flags/folders with notmuch tags and as I haven't seen a script to do that, I've written (the beginnings of?) one. It is probably pre-alpha but I am a fan of release early, release often and if someone finds it useful to build upon, it was already worth it. I put the code here: http://github.com/spaetz/notmuchsync What does it do? - Synchronizes the "S" flag with the "unread" tag (1-way). The synchronization direction is decided by using either --sync (change maildir flags according to notmuch) or --revsync (change notmuch tags according to maildir). By default it always checks the mails from the previous 30 days (but can also do --all mails if you have plenty of RAM and time). - Deletes all mail files that have the "delete" tag - Quiet/normal/verbose logging Workflow 1a) start out with notmuchsync -r in order to initialize the notmuch tag database based on maildir 'S' flags. This is probably what everyone but Carl Worth wants. :-) If the "--all" command works for you, even better, but I doubt it as it would store all your mails in RAM. 1b) start out with not What needs improvment = - It's a python script in one file. The architecture might need some cleanup. Documentation needs work too. - It temporarily slurps in all your mails from the last 30 days into RAM. I am waiting for "notmuchs show blah --output filename --output tags" to improve that :). Generally the parsing of the output of "notmuch show" is a bit hackyish with regexps at the moment. - Support parsing "chunks" of "notmuch show" output or do several runs for -all, using monthly intervals to make --all work. - Support for the "T" (delete), "F" (flag), "D" (draft) tags/flags. Should be pretty easy to add. - Support syncing of the "inbox" tag if the mail is in a dir different than INBOX. No clue yet how to do. Perhaps if a maildir name corresponds to a notmuch tag name, we move the mail there? - It seems to work for me, but is untested in environments with spaces in paths and windows machines. - the --doeverything command that prunes and syncs in one go What else = - It is GPL v2.1+ - Please improve and/or fork this thing. - If you don't like it, just ignore it. Thanks for your understanding. --- Usage: -p --prune Prune deleted mails -s --sync Sync from notmuch tags to maildir flags. By default it will only look for mails from the last 30 days. Use --all to look at earlier mails. Beware, if timestamps are more than 30 days in the future, we won't handle it. -r --revsyncSync tags from maildir to notmuch. See also -s. Options: -d Really verbose debug oputput. -q Log only errors. --dryrun Do not really modify notmuch db or mail files Works together with -p -s -r -- Let me know what you like or don't. If you want github contributor right, I am willing to hand out access quite liberal. And as always: patches welcome. :-) spaetz ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
