[notmuch] Notmuch performance problems on OSX

2010-01-18 Thread Oliver Charles
On Thu, Jan 14, 2010 at 11:16 PM, Carl Worth  wrote:
> Hi Oliver, welcome to notmuch!
>
> On Thu, 14 Jan 2010 15:30:48 +, Oliver Charles  googlemail.com> wrote:
>> I've installed the latest notmuch from Git at this time of writing,
>> along with Xapian from SVN head. However, just tagging a single thread
>> with only one message seems to take too long:
>>
>> $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221
>>
>> real ?0m0.812s
>> user ?0m0.022s
>> sys ? 0m0.037s
>
> Things work quite a bit faster than that on my machine:
>
> $ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314 at 
> mail.gmail.com
>
> real ? ?0m0.024s
> user ? ?0m0.012s
> sys ? ? 0m0.004s
>
> But that could just be system differences.

Possibly, though my machine is more than capable of what it's doing -
a Q6600 with 4gb RAM

>> And tagging all my messages is really horrible:
>>
>> $ time notmuch tag +foobar tag:inbox
>>
>> real ?0m5.076s
>> user ?0m3.688s
>> sys ? 0m0.105s
>
> For this operation, I can't really compare. How many messages are you
> tagging? Here's that operation for me with 525 messages in my inbox:

A few thousand (4k, I believe)

>> That xapian-svn was built from svn HEAD right now, so I'm assuming it
>> contains the #250 fix (http://trac.xapian.org/changeset/13808)
>
> Which I think means that things could have been even *much* slower
> before. ;-)
>
> The Xapian defect #250 was just one, initial (and obvious) performance
> problem. [Though, as I mentioned in a previous thread, if you're using a
> Xapian flint database, (look for .notmuch/xapian/iamflint), then you
> won't get the benefit of the Xapian fix until you rebuild your notmuch
> database from scratch with a current notmuch.]

I didn't know about this need to rebuild, but I tried that and didn't
have any more success sadly.

> Once you've verified that you've got the #250 fix functional, there
> could still be lots of performance bugs. And it would be time to start
> profiling.
>
> [...]

I'm pressed for time at the moment, but in a few weeks I might have
some time to investigate here...

-- 
Oliver Charles / aCiD2


[notmuch] Introducing notmuchsync

2010-01-18 Thread Sebastian Spaeth
On Mon, 18 Jan 2010 16:41:25 +0100, Michal Sojka  wrote:
> with which Python version is this meant to work? I get the following error:
> $ ./notmuchsync -r

> AttributeError: 'str' object has no attribute 'format'

DOH, I just used 2.6-style formatting. But as there is little reason for
this, I just reverted to old-style formatting. Just pull from git again.

I have no clue about the needed version, but it should definitely be
<2.6 now.

spaetz


[notmuch] Introducing notmuchsync

2010-01-18 Thread Marten Veldthuis
On Mon, 18 Jan 2010 16:12:28 +0100, "Sebastian Spaeth"  wrote:
> What does it do?
> 
>  - Synchronizes the "S" flag with the "unread" tag (1-way). The
>  synchronization direction is decided by using either --sync (change
>  maildir flags according to notmuch) or --revsync (change notmuch tags 
> according to maildir). By default it always checks the mails from the 
> previous 30
>  days (but can also do --all mails if you have plenty of RAM and time).
>  - Deletes all mail files that have the "delete" tag
>  - Quiet/normal/verbose logging 

Excellent. Sounds like nearly exactly what I thought about doing last
weekend, except that I couldn't find a nice Ruby library quickly and
then my attention-span was over :)

Also, I'm curious as to Carl's opinion to this, but as far as I'm
concerned, not everything about notmuch needs to be coded in
C. Obviously, things interacting with the database directly need to, but
if it could be built upon the notmuch C-based CLI commands, and be fast
enough, would that be eligible to make it into the repository?

-- 
- Marten


[notmuch] Introducing notmuchsync

2010-01-18 Thread Michal Sojka
On Monday 18 of January 2010 16:12:28 Sebastian Spaeth wrote:
> Dear list,
> 
> I really want to sync my maildir flags/folders with notmuch tags and as
> I haven't seen a script to do that, I've written (the beginnings of?)
> one.
> 
> It is probably pre-alpha but I am a fan of release early, release often
> and if someone finds it useful to build upon, it was already worth it. I
> put the code here: http://github.com/spaetz/notmuchsync

Hi,

with which Python version is this meant to work? I get the following error:
$ ./notmuchsync -r
Traceback (most recent call last):
  File "./notmuchsync", line 316, in 
notmuch.syncTags(frommaildir=True, dryrun=dryrun)
  File "./notmuchsync", line 249, in syncTags
searchterm = "{0}..{1}".format(now-2592000,now+2592000)
AttributeError: 'str' object has no attribute 'format'

Michal


[notmuch] Introducing notmuchsync

2010-01-18 Thread Sebastian Spaeth
On Mon, 18 Jan 2010 16:12:28 +0100, "Sebastian Spaeth"  wrote:
DOH, I should actually have finished the Workflow section before sending
it off :-):

Workflow

1a) start out with notmuchsync -r in order to initialize the notmuch tag
database based on maildir 'S' flags. This is probably what everyone but
Carl Worth wants. :-) If the "--all" command works for you, even better,
but I doubt it as it would store all your mails in RAM. I'll need to
work on some chunking for this.
1b) start out with nnotmuchsync -s if your notmuch tags are
authoritative and it will modify your Maildir flags accordingly.

After that a periodic:
2) "notmuchsync -s" syncs tags to flags (Currently 'S' only)
3) "notmuchsync -p" purges mail files with "delete" flags


[notmuch] Introducing notmuchsync

2010-01-18 Thread Sebastian Spaeth
Dear list,

I really want to sync my maildir flags/folders with notmuch tags and as
I haven't seen a script to do that, I've written (the beginnings of?)
one.

It is probably pre-alpha but I am a fan of release early, release often
and if someone finds it useful to build upon, it was already worth it. I
put the code here: http://github.com/spaetz/notmuchsync

What does it do?

 - Synchronizes the "S" flag with the "unread" tag (1-way). The
 synchronization direction is decided by using either --sync (change
 maildir flags according to notmuch) or --revsync (change notmuch tags 
according to maildir). By default it always checks the mails from the previous 
30
 days (but can also do --all mails if you have plenty of RAM and time).
 - Deletes all mail files that have the "delete" tag
 - Quiet/normal/verbose logging 

Workflow

1a) start out with notmuchsync -r in order to initialize the notmuch tag
database based on maildir 'S' flags. This is probably what everyone but
Carl Worth wants. :-) If the "--all" command works for you, even better,
but I doubt it as it would store all your mails in RAM.
1b) start out with not

What needs improvment
=
 - It's a python script in one file. The architecture might need some
 cleanup. Documentation needs work too.
 - It temporarily slurps in all your mails from the last 30 days into
 RAM. I am waiting for "notmuchs show blah --output filename --output
 tags" to improve that :). Generally the parsing of the output of
 "notmuch show" is a bit hackyish with regexps at the moment.
 - Support parsing "chunks" of "notmuch show" output or do several
   runs for -all, using monthly intervals to make --all work.
 - Support for the "T" (delete), "F" (flag), "D" (draft) tags/flags.
   Should be pretty easy to add.
 - Support syncing of the "inbox" tag if the mail is in a dir different
 than INBOX. No clue yet how to do. Perhaps if a maildir name
 corresponds to a notmuch tag name, we move the mail there?
 - It seems to work for me, but is untested in environments with spaces
 in paths and windows machines.
 - the --doeverything command that prunes and syncs in one go

What else
=
- It is GPL v2.1+
- Please improve and/or fork this thing.
- If you don't like it, just ignore it. Thanks for your understanding.

---
Usage:
-p --prune  Prune deleted mails
-s --sync   Sync from notmuch tags to maildir flags.
By default it will only look for mails from the last 30 days.
Use --all to look at earlier mails.
Beware, if timestamps are more than 30 days in the
future, 
we won't handle it.
-r --revsyncSync tags from maildir to notmuch. See also -s.
Options:
-d   Really verbose debug oputput.
-q   Log only errors.
--dryrun Do not really modify notmuch db or mail files
 Works together with -p -s -r

--

Let me know what you like or don't. If you want github contributor
right, I am willing to hand out access quite liberal. And as always:
patches welcome. :-)

spaetz


[notmuch] Introducing notmuchsync

2010-01-18 Thread Sebastian Spaeth
Dear list,

I really want to sync my maildir flags/folders with notmuch tags and as
I haven't seen a script to do that, I've written (the beginnings of?)
one.

It is probably pre-alpha but I am a fan of release early, release often
and if someone finds it useful to build upon, it was already worth it. I
put the code here: http://github.com/spaetz/notmuchsync

What does it do?

 - Synchronizes the S flag with the unread tag (1-way). The
 synchronization direction is decided by using either --sync (change
 maildir flags according to notmuch) or --revsync (change notmuch tags 
according to maildir). By default it always checks the mails from the previous 
30
 days (but can also do --all mails if you have plenty of RAM and time).
 - Deletes all mail files that have the delete tag
 - Quiet/normal/verbose logging 

Workflow

1a) start out with notmuchsync -r in order to initialize the notmuch tag
database based on maildir 'S' flags. This is probably what everyone but
Carl Worth wants. :-) If the --all command works for you, even better,
but I doubt it as it would store all your mails in RAM.
1b) start out with not

What needs improvment
=
 - It's a python script in one file. The architecture might need some
 cleanup. Documentation needs work too.
 - It temporarily slurps in all your mails from the last 30 days into
 RAM. I am waiting for notmuchs show blah --output filename --output
 tags to improve that :). Generally the parsing of the output of
 notmuch show is a bit hackyish with regexps at the moment.
 - Support parsing chunks of notmuch show output or do several
   runs for -all, using monthly intervals to make --all work.
 - Support for the T (delete), F (flag), D (draft) tags/flags.
   Should be pretty easy to add.
 - Support syncing of the inbox tag if the mail is in a dir different
 than INBOX. No clue yet how to do. Perhaps if a maildir name
 corresponds to a notmuch tag name, we move the mail there?
 - It seems to work for me, but is untested in environments with spaces
 in paths and windows machines.
 - the --doeverything command that prunes and syncs in one go

What else
=
- It is GPL v2.1+
- Please improve and/or fork this thing.
- If you don't like it, just ignore it. Thanks for your understanding.

---
Usage:
-p --prune  Prune deleted mails
-s --sync   Sync from notmuch tags to maildir flags.
By default it will only look for mails from the last 30 days.
Use --all to look at earlier mails.
Beware, if timestamps are more than 30 days in the
future, 
we won't handle it.
-r --revsyncSync tags from maildir to notmuch. See also -s.
Options:
-d   Really verbose debug oputput.
-q   Log only errors.
--dryrun Do not really modify notmuch db or mail files
 Works together with -p -s -r

--

Let me know what you like or don't. If you want github contributor
right, I am willing to hand out access quite liberal. And as always:
patches welcome. :-)

spaetz
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] Introducing notmuchsync

2010-01-18 Thread Sebastian Spaeth
On Mon, 18 Jan 2010 16:12:28 +0100, Sebastian Spaeth sebast...@sspaeth.de 
wrote:
DOH, I should actually have finished the Workflow section before sending
it off :-):

Workflow

1a) start out with notmuchsync -r in order to initialize the notmuch tag
database based on maildir 'S' flags. This is probably what everyone but
Carl Worth wants. :-) If the --all command works for you, even better,
but I doubt it as it would store all your mails in RAM. I'll need to
work on some chunking for this.
1b) start out with nnotmuchsync -s if your notmuch tags are
authoritative and it will modify your Maildir flags accordingly.

After that a periodic:
2) notmuchsync -s syncs tags to flags (Currently 'S' only)
3) notmuchsync -p purges mail files with delete flags
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] Introducing notmuchsync

2010-01-18 Thread Michal Sojka
On Monday 18 of January 2010 16:12:28 Sebastian Spaeth wrote:
 Dear list,
 
 I really want to sync my maildir flags/folders with notmuch tags and as
 I haven't seen a script to do that, I've written (the beginnings of?)
 one.
 
 It is probably pre-alpha but I am a fan of release early, release often
 and if someone finds it useful to build upon, it was already worth it. I
 put the code here: http://github.com/spaetz/notmuchsync

Hi,

with which Python version is this meant to work? I get the following error:
$ ./notmuchsync -r
Traceback (most recent call last):
  File ./notmuchsync, line 316, in module
notmuch.syncTags(frommaildir=True, dryrun=dryrun)
  File ./notmuchsync, line 249, in syncTags
searchterm = {0}..{1}.format(now-2592000,now+2592000)
AttributeError: 'str' object has no attribute 'format'

Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] Introducing notmuchsync

2010-01-18 Thread Marten Veldthuis
On Mon, 18 Jan 2010 16:12:28 +0100, Sebastian Spaeth sebast...@sspaeth.de 
wrote:
 What does it do?
 
  - Synchronizes the S flag with the unread tag (1-way). The
  synchronization direction is decided by using either --sync (change
  maildir flags according to notmuch) or --revsync (change notmuch tags 
 according to maildir). By default it always checks the mails from the 
 previous 30
  days (but can also do --all mails if you have plenty of RAM and time).
  - Deletes all mail files that have the delete tag
  - Quiet/normal/verbose logging 

Excellent. Sounds like nearly exactly what I thought about doing last
weekend, except that I couldn't find a nice Ruby library quickly and
then my attention-span was over :)

Also, I'm curious as to Carl's opinion to this, but as far as I'm
concerned, not everything about notmuch needs to be coded in
C. Obviously, things interacting with the database directly need to, but
if it could be built upon the notmuch C-based CLI commands, and be fast
enough, would that be eligible to make it into the repository?

-- 
- Marten
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] Notmuch performance problems on OSX

2010-01-18 Thread Oliver Charles
On Thu, Jan 14, 2010 at 11:16 PM, Carl Worth cwo...@cworth.org wrote:
 Hi Oliver, welcome to notmuch!

 On Thu, 14 Jan 2010 15:30:48 +, Oliver Charles 
 oliver.g.char...@googlemail.com wrote:
 I've installed the latest notmuch from Git at this time of writing,
 along with Xapian from SVN head. However, just tagging a single thread
 with only one message seems to take too long:

 $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221

 real  0m0.812s
 user  0m0.022s
 sys   0m0.037s

 Things work quite a bit faster than that on my machine:

 $ time notmuch tag +foo 
 id:5641883d1001140730l22832715ld6bdc95c9938d...@mail.gmail.com

 real    0m0.024s
 user    0m0.012s
 sys     0m0.004s

 But that could just be system differences.

Possibly, though my machine is more than capable of what it's doing -
a Q6600 with 4gb RAM

 And tagging all my messages is really horrible:

 $ time notmuch tag +foobar tag:inbox

 real  0m5.076s
 user  0m3.688s
 sys   0m0.105s

 For this operation, I can't really compare. How many messages are you
 tagging? Here's that operation for me with 525 messages in my inbox:

A few thousand (4k, I believe)

 That xapian-svn was built from svn HEAD right now, so I'm assuming it
 contains the #250 fix (http://trac.xapian.org/changeset/13808)

 Which I think means that things could have been even *much* slower
 before. ;-)

 The Xapian defect #250 was just one, initial (and obvious) performance
 problem. [Though, as I mentioned in a previous thread, if you're using a
 Xapian flint database, (look for .notmuch/xapian/iamflint), then you
 won't get the benefit of the Xapian fix until you rebuild your notmuch
 database from scratch with a current notmuch.]

I didn't know about this need to rebuild, but I tried that and didn't
have any more success sadly.

 Once you've verified that you've got the #250 fix functional, there
 could still be lots of performance bugs. And it would be time to start
 profiling.

 [...]

I'm pressed for time at the moment, but in a few weeks I might have
some time to investigate here...

-- 
Oliver Charles / aCiD2
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch