[notmuch] Quick thoughts on a notmuch daemon

[Marten Veldthuis]

On Mon, 07 Dec 2009 15:02:03 -0400, David Bremner  wrote:
> 
> On Thu, 03 Dec 2009 14:27:05 -0800, Carl Worth  wrote:
> It might be a bit blue sky, but if this daemon could (optionally) talk
> IMAP and translate tags into folders on the fly, this would be very
> useful for people who need imap access to their mail as well as from an
> custom notmuch client.

For me, my IMAP needs are pretty much limited to my iPhone. I'm making
do for now, but to make notmuch viable in the long term, what I'd need
is:

 * notmuch shouldn't choke on mails I had in notmuch's database, and
   then marked read or deleted on my iPhone (which renames them in the
   maildir). This is coming with the moving/deleting patches.
 * notmuch should sync back read/unread state to maildir
 * notmuch should move read stuff out of my inbox. It would be
   acceptable if it moved everything into a designated archive folder
   unless it had the 'inbox' tag assigned, in which case it moved it
   there. Note that we have the moving/deleting patches, then this could
   even be done as a script and some searches.

With this, my inbox would be usable from my iPhone.

-- 
- Marten

[notmuch] [PATCH] notmuch-restore.c: only update tags for messages that differ from dump file.

[da...@tethera.net]

From: David Bremner 

The main feature of this patch is that it compares the list of current
tags on a message with those read by restore. Only if the two lists
differ is the tag list in the message replaced.  In my experiments this leads to
a large performance improvement.

Since I had to rewrite the parsing of tags from the dump file anyway
to keep a list of tags (in case they should be written to the
database), I decided to make it a bit more robust. It sorts the
incoming tags (it is critical for the comparison of the two tag lists
that they are both sorted), and allows arbitrary whitespace (as
determined by "isspace") between tags.

The patch allocates a temporary array to keep track of the current
list of tags using calloc and grows it as neccesary using realloc.
---

This is the second version of the patch, the only difference is in the test 
on line 144 (used to be 142), and the comment before explaining it.

 notmuch-restore.c |   75 
 1 files changed, 63 insertions(+), 12 deletions(-)

diff --git a/notmuch-restore.c b/notmuch-restore.c
index 1b9598d..9d7bd17 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -18,8 +18,17 @@
  * Author: Carl Worth 
  */

+#include 
+#include 
+#include 
 #include "notmuch-client.h"

+#define DEFAULT_TAG_ARRAY_SIZE 2
+/* for qsort */
+static int scmp( const void *sp1, const void *sp2 )
+{
+return( strcmp(*(const char **)sp1, *(const char **)sp2) );
+}
 int
 notmuch_restore_command (unused (void *ctx), int argc, char *argv[])
 {
@@ -31,6 +40,9 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
 ssize_t line_len;
 regex_t regex;
 int rerr;
+char **tag_array=NULL;
+int tag_array_size=DEFAULT_TAG_ARRAY_SIZE;
+

 config = notmuch_config_open (ctx, NULL, NULL);
 if (config == NULL)
@@ -61,11 +73,18 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
  "^([^ ]+) \\(([^)]*)\\)$",
  REG_EXTENDED);

+/* Make an array of pointers to point to individual tokens */
+tag_array=calloc(tag_array_size,sizeof(char*));
+
 while ((line_len = getline (, _size, input)) != -1) {
regmatch_t match[3];
-   char *message_id, *tags, *tag, *next;
+   char *message_id, *tags,  *next;
notmuch_message_t *message;
notmuch_status_t status;
+   int tag_count;
+
+   notmuch_tags_t *tag_list;
+   int i;

chomp_newline (line);

@@ -89,26 +108,55 @@ notmuch_restore_command (unused (void *ctx), int argc, 
char *argv[])
goto NEXT_LINE;
}

-   notmuch_message_freeze (message);
+   next=tags;
+   tag_count=0;
+   while(*next){
+ while(*next && isspace(*next))
+   next++;
+ if (*next) {
+   while (tag_count>= tag_array_size){
+ tag_array_size*=2;
+ tag_array=realloc(tag_array,tag_array_size*sizeof(char *));
+   }
+   tag_array[tag_count]=next;
+   tag_count++;
+ }
+ while (*next && !isspace(*next))
+   next++;
+ if (*next){
+   *next='\0';
+   next++;
+ }
+   }
+
+   qsort(tag_array,tag_count,sizeof(char*),scmp);
+   
+   tag_list = notmuch_message_get_tags (message);
+   i=0;
+   while (notmuch_tags_has_more (tag_list) && i

[notmuch] Quick thoughts on a notmuch daemon

[Carl Worth]

On Mon, 07 Dec 2009 18:51:58 -0500, micah anderson  wrote:
> I've switched mail clients enough over the past few years to know that
> switching itself is a major pain.

Absolutely.

One concept in notmuch (compared to sup) is to (eventually) avoid people
having to go through that pain by their current mail client becoming
"notmuch enabled". For me, I had always liked composing email in emacs,
so I just have to add notmuch support to the existing emacs
message-mode.

Hopefully people working on other email interfaces will do similar
things, (would be great to have Anjal and Thunderbird get some notmuch
support for example).

I definitely didn't like that with sup, to get all the global-search and
tagging features one had to accept the curses UI as well.

>If I switch to an
> 'all-in-one' label-state mail client, like notmuch, I want to be sure
> that in 2 years from now if I happen to decide to change to something
> else (I'm hoping this wont EVER happen because notmuch is very
> promising, but I need to be honest based on past experiences here), then
> I'm going to want to make sure that all of my mail is marked as read,
> deleted mail is deleted and even though I was using labels for
> organizational purposes in notmuch, things are still organized in some
> way appropriate to folders on the filesystem.

I appreciate your caution before making the commitment to a mail
client. I think that's very wise. [*]

And I think that for the case of "I'm giving up on notmuch and want to
switch to something else" you can be quite assured that notmuch will
allow you to take care of everything you need. At a system level, it's
really an almost trivial matter to implement everything you describe
above as small shell scripts based on "notmuch search" commands. And I
can only imagine support for that kind of thing getting better all the
time.

> The way it is now, if I switch to notmuch and then try to switch back,
> my life is a total mess because all of the state is contained within the
> notmuch database, that is frightening for the long-term, but it also
> makes me very worried about simple corruption of that data store. If I
> lose that state, I'm totally screwed. At least in a maildir scenario, if
> I got corruption in one place it might cause me to lose one email, but
> if I get corruption in my notmuch database, I had better have a recent
> backup or I am screwed. 

I highly recommend making very regular backups of the output of "notmuch
dump". It's a quick process to run, and the file it creates is not
large. It's also nicely sorted so that if you are doing some kind of
incremental backup, that should work well.

> I like to think that this concept of label-state, indexable
> storage-backed mail clients is the dawn of a new age, just as maildir
> was to mbox, but I'm still scared because there is no mb2md equivalent
> yet.

I think we're really close to where you could write a notmuch2md script
as a very tiny shell script:

for tag in $(notmuch search --for=tags *); do
notmuch search --for=messages --output=maildir:/some/dir/$tag tag:$tag
done

Being able to do that kind of scriptable output is pretty powerful. And
we're not missing anything fundamental in the system to be able to
support that, (just a little bit of support for the new command-line
options like --for and --output).

-Carl

[*] I know I went through a similarly cautious evaluation when switching
away from CVS. CVS made the transition away so painful that I was
determined to choose a system that satisfied me on two criteria:

1. The system seemed well-designed enough that I could imagine it
   surviving "forever".

2. That when realities exceeded my imagination, the system wouldn't make
   it hard for me to switch away.

So far I'm still quite happy with git on both points.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20091207/0b06434d/attachment-0001.pgp>

[notmuch] [PATCH] notmuch-restore.c: only update tags for messages that differ from dump file.

[Scott Robinson]

Excerpts from david's message of Mon Dec 07 19:14:48 -0800 2009:
> The patch allocates a temporary array to keep track of the current
> list of tags using calloc and grows it as neccesary using realloc.

The codebase has already well established the use of the talloc library. Why
did you break with that?
-- 
Scott Robinson | http://quadhome.com/

Q: Why are my replies five sentences or less?
A: http://five.sentenc.es/

[notmuch] semi-usable notmuch-gtk client

[Chris Wilson]

On Mon, 07 Dec 2009 09:57:12 -0800, Carl Worth  wrote:
> On Mon, 07 Dec 2009 16:42:42 +0530, "Aneesh Kumar K. V"  linux.vnet.ibm.com> wrote:
> > While discussing notmuch with some of the gnome developers they pointed
> > me to http://live.gnome.org/Anjal/ . The interface is suited to the way
> > notmuch displays threads. So i guess it would really cool if we can get
> > anjal to use nomuch as one of the backend.
> > 
> > That should also make sure you get a working mailclient in gtk with
> > minimal changes
> 
> Thanks for mentioning this, Aneesh.
> 
> Hooking up Anjal to notmuch was an idea that Chris Wilson had almost
> immediately after playing with notmuch. I think he even started poking
> around in the Anjal code base to see how much work this would be.
> 
> Chris, did you make any progress worth mentioning in your
> investigations?

Before being distracted again, the impression I got from reading Anjal was
that to integrate with it, you needed to understand evolution-data-server
and camel. Anjal itself seems to be a relatively thin message/folder
viewer and composer, with the details hidden behind camel. So the next
step, I felt, would be to see whether you could write a notmuch data store
for evolution.
-ickle

-- 
Chris Wilson, Intel Open Source Technology Centre

[notmuch] Quick thoughts on a notmuch daemon

[micah anderson]

lose one email, but
if I get corruption in my notmuch database, I had better have a recent
backup or I am screwed. 

The important part is gluing the maildir capabilities in there, and not
getting distracted by the transport (let offlineimap and its ilk handle
this, maybe some eager python hacker might find a way to add this to
offlineimap?!).

I like to think that this concept of label-state, indexable
storage-backed mail clients is the dawn of a new age, just as maildir
was to mbox, but I'm still scared because there is no mb2md equivalent
yet.

micah
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20091207/e4de245d/attachment.pgp>

[notmuch] semi-usable notmuch-gtk client

[Aneesh Kumar K. V]

On Fri, 27 Nov 2009 19:58:38 +0200, Baruch Even  wrote:
> Hi,
> 
> I added a quick hack to show a message thread in notmuch-gtk and this 
> makes it semi-usable. I don't actually parse the reply of notmuch-show 
> but it's already passably usable to read mail.
> 

While discussing notmuch with some of the gnome developers they pointed
me to http://live.gnome.org/Anjal/ . The interface is suited to the way
notmuch displays threads. So i guess it would really cool if we can get
anjal to use nomuch as one of the backend.

That should also make sure you get a working mailclient in gtk with
minimal changes

-aneesh

[notmuch] Quick thoughts on a notmuch daemon

[David Bremner]

On Thu, 03 Dec 2009 14:27:05 -0800, Carl Worth  wrote:
> But with the other commands in place, (such as search and show), then
> this could operate over ssh and all that would be necessary is one
> additional command not in the current command line, (for "give me this
> file"). And then the client could just maintain a local cache of mail
> files as needed.

It might be a bit blue sky, but if this daemon could (optionally) talk
IMAP and translate tags into folders on the fly, this would be very
useful for people who need imap access to their mail as well as from an
custom notmuch client.

Of course it would be nice to be able to just drop a back-end into some 
well tested server that dealt with authentication and the imap protocol.

As long as we're tossing out projects :)

d

[notmuch] semi-usable notmuch-gtk client

[Carl Worth]

On Mon, 07 Dec 2009 16:42:42 +0530, "Aneesh Kumar K. V"  wrote:
> While discussing notmuch with some of the gnome developers they pointed
> me to http://live.gnome.org/Anjal/ . The interface is suited to the way
> notmuch displays threads. So i guess it would really cool if we can get
> anjal to use nomuch as one of the backend.
> 
> That should also make sure you get a working mailclient in gtk with
> minimal changes

Thanks for mentioning this, Aneesh.

Hooking up Anjal to notmuch was an idea that Chris Wilson had almost
immediately after playing with notmuch. I think he even started poking
around in the Anjal code base to see how much work this would be.

Chris, did you make any progress worth mentioning in your
investigations?

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20091207/70bf1196/attachment.pgp>

[notmuch] slowdown with large html heavy threads

[Carl Worth]

On Sat, 05 Dec 2009 15:10:18 -0800, Alexander Botero-Lowry  wrote:
> Carl mentioned that he saw some nasty performance issues with threads
> that had a lot of HTML messages in them.
> 
> If you are experiencing this problem, would you mind trying some
> alternative mm-text-html-renderer setting:
> 
> http://mh-e.sourceforge.net/manual/html/HTML.html
> 
> Also, if you have elinks, I recommend my custom renderer:
> 
> (require 'mm-view)
> (add-to-list 'mm-text-html-renderer-alist 
>   '(elinks mm-inline-render-with-stdin nil "elinks" "-force-html" "-dump"))
> (setq mm-text-html-renderer 'elinks)

Hi Alex,

I'm sorry, but the delays got in my way so much this morning that I
reverted the patch before even attempting the alternate renderer.

Part of my annoyance may have been due to a distinct bug. There was
definitely one instance where I was seeing a big delay when trying to
display a tiny thread (about 13 messages) none of which were HTML. I
reverted the HTML-rendering patch and saw the delay go away, but I also
put the patch back and didn't see the problem come back.

So that problem might have been form something else. But when I looked
at the patch and saw that it was doing this:

-(let ((filename (notmuch-show-get-filename)))
-  (with-temp-buffer
-(insert-file-contents filename nil nil nil t)

for every file, (regardless of it being HTML or not), I was convinced
that there's got to be a better way. Notmuch has already passed the
entire thread through GMime, so we really don't want to do a second pass
of all of the raw message content through an emacs-lisp MIME parser as
well.

Instead, let's find a way to just get the information from GMime to be
cheaply available for emacs to look at (without loading all of the
original files).

-Carl

-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20091207/e79b727c/attachment.pgp>

[notmuch] emacs: Revert of rudimentary HTML support

[Carl Worth]

It was really nice to have the basic conversion of HTML to text, but I'm
unwilling to accept the performance impacts of the current implementation.

I'm seeing emacs stop and pause for seconds when I ask it to display a
thread with even a small number of messages (say, 30 or so). Performance
bugs related to threads of a similar size, (which is of course *tiny*
compared to some of the threads I'll need to deal with), was a source of
a lot of frustration to me when using sup.

So for now I've reverted the patch that added this support, and we'll
just need to find a better way of implementing this.

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20091207/c770638d/attachment.pgp>

[notmuch] notmuch_threads_back and notmuch_messages_back

[Carl Worth]

On Sun, 6 Dec 2009 18:34:37 +0100, Ruben Pollan  
wrote:
> Do I miss something? Or the function is just designed to work on a single node
> not a list of them?

The latter. The documentation for the function is wrong. Sorry about that.

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20091207/b7a24810/attachment.pgp>

[notmuch] emacs: Revert of rudimentary HTML support

[Carl Worth]

It was really nice to have the basic conversion of HTML to text, but I'm
unwilling to accept the performance impacts of the current implementation.

I'm seeing emacs stop and pause for seconds when I ask it to display a
thread with even a small number of messages (say, 30 or so). Performance
bugs related to threads of a similar size, (which is of course *tiny*
compared to some of the threads I'll need to deal with), was a source of
a lot of frustration to me when using sup.

So for now I've reverted the patch that added this support, and we'll
just need to find a better way of implementing this.

-Carl


pgpDikHbGOyCA.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[notmuch] [PATCH 2/2] Save all attachments to a directory

[Keith Amidon]

Prompt for a directory and write all attachments of the current
message to that directory, prompting for a filename for each with a
default value of the filename specified in the attachment.

The behavior of this function differs in two ways from the existing
notmuch-show-save-attachments function:

1) It first prompts for the directory in which to save attachments
   instead of assuming the current default directory.

2) If there is more than one attachment in the message, it assumes all
   should be saved and only prompts for filenames instead of asking
   first whether each attachment should be saved.
---
 notmuch.el |   12 
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/notmuch.el b/notmuch.el
index 8d51709..08bd114 100644
--- a/notmuch.el
+++ b/notmuch.el
@@ -65,6 +65,7 @@
 (define-key map r 'notmuch-show-reply)
 (define-key map | 'notmuch-show-pipe-message)
 (define-key map w 'notmuch-show-save-attachments)
+(define-key map W 'notmuch-show-save-all-attachments)
 (define-key map V 'notmuch-show-view-raw-message)
 (define-key map v 'notmuch-show-view-all-mime-parts)
 (define-key map - 'notmuch-show-remove-tag)
@@ -352,6 +353,17 @@ buffer.
   mm-handle ( (notmuch-count-attachments mm-handle) 1
   (message Done))
 
+(defun notmuch-show-save-all-attachments (directory)
+  Save all attachments of a message to a directory.
+  (interactive G)
+  (let ((dirname (file-name-as-directory directory)))
+(make-directory dirname t)
+(with-current-notmuch-show-message
+ (let ((mm-handle (mm-dissect-buffer))
+   (mm-default-directory dirname))
+   (notmuch-save-attachments mm-handle nil
+  (message Done))
+
 (defun notmuch-reply (query-string)
   (switch-to-buffer (generate-new-buffer notmuch-draft))
   (call-process notmuch-command nil t nil reply query-string)
-- 
1.6.5.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] semi-usable notmuch-gtk client

[Chris Wilson]

On Mon, 07 Dec 2009 09:57:12 -0800, Carl Worth cwo...@cworth.org wrote:
 On Mon, 07 Dec 2009 16:42:42 +0530, Aneesh Kumar K. V 
 aneesh.ku...@linux.vnet.ibm.com wrote:
  While discussing notmuch with some of the gnome developers they pointed
  me to http://live.gnome.org/Anjal/ . The interface is suited to the way
  notmuch displays threads. So i guess it would really cool if we can get
  anjal to use nomuch as one of the backend.
  
  That should also make sure you get a working mailclient in gtk with
  minimal changes
 
 Thanks for mentioning this, Aneesh.
 
 Hooking up Anjal to notmuch was an idea that Chris Wilson had almost
 immediately after playing with notmuch. I think he even started poking
 around in the Anjal code base to see how much work this would be.
 
 Chris, did you make any progress worth mentioning in your
 investigations?

Before being distracted again, the impression I got from reading Anjal was
that to integrate with it, you needed to understand evolution-data-server
and camel. Anjal itself seems to be a relatively thin message/folder
viewer and composer, with the details hidden behind camel. So the next
step, I felt, would be to see whether you could write a notmuch data store
for evolution.
-ickle

-- 
Chris Wilson, Intel Open Source Technology Centre
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Quick thoughts on a notmuch daemon

[Marten Veldthuis]

On Mon, 07 Dec 2009 15:02:03 -0400, David Bremner da...@tethera.net wrote:
 
 On Thu, 03 Dec 2009 14:27:05 -0800, Carl Worth cwo...@cworth.org wrote:
 It might be a bit blue sky, but if this daemon could (optionally) talk
 IMAP and translate tags into folders on the fly, this would be very
 useful for people who need imap access to their mail as well as from an
 custom notmuch client.

For me, my IMAP needs are pretty much limited to my iPhone. I'm making
do for now, but to make notmuch viable in the long term, what I'd need
is:

 * notmuch shouldn't choke on mails I had in notmuch's database, and
   then marked read or deleted on my iPhone (which renames them in the
   maildir). This is coming with the moving/deleting patches.
 * notmuch should sync back read/unread state to maildir
 * notmuch should move read stuff out of my inbox. It would be
   acceptable if it moved everything into a designated archive folder
   unless it had the 'inbox' tag assigned, in which case it moved it
   there. Note that we have the moving/deleting patches, then this could
   even be done as a script and some searches.

With this, my inbox would be usable from my iPhone.

-- 
- Marten
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Quick thoughts on a notmuch daemon

[micah anderson]

Excerpts from Marten Veldthuis's message of Mon Dec 07 17:55:24 -0500 2009:
 On Mon, 07 Dec 2009 15:02:03 -0400, David Bremner da...@tethera.net wrote:
  
  On Thu, 03 Dec 2009 14:27:05 -0800, Carl Worth cwo...@cworth.org wrote:
  It might be a bit blue sky, but if this daemon could (optionally) talk
  IMAP and translate tags into folders on the fly, this would be very
  useful for people who need imap access to their mail as well as from an
  custom notmuch client.
 
 For me, my IMAP needs are pretty much limited to my iPhone. I'm making
 do for now, but to make notmuch viable in the long term, what I'd need
 is:
 
  * notmuch shouldn't choke on mails I had in notmuch's database, and
then marked read or deleted on my iPhone (which renames them in the
maildir). This is coming with the moving/deleting patches.
  * notmuch should sync back read/unread state to maildir
  * notmuch should move read stuff out of my inbox. It would be
acceptable if it moved everything into a designated archive folder
unless it had the 'inbox' tag assigned, in which case it moved it
there. Note that we have the moving/deleting patches, then this could
even be done as a script and some searches.
 
 With this, my inbox would be usable from my iPhone.

I dont have an iPhone, but this would be a *killer* feature for me, and
I suspect for others as well. 

I have two fundamental problems with label-state clients (sup, notmuch)
that I am still trying to come to terms with, this bluesky feature would
change that radically.

The first issue is that I do not have unlimited IMAP storage space on
the server, in fact I have a quota. I know, lots of people have
effectively unlimited space,and others POP/fetchmail things so its not
stored on the server...but lots of us still do have upper limits that we
have to deal with and many of us do use IMAP, for good reasons. With
label-state clients (sup, notmuch) you have to accept the fact that your
mail is going to grow on the IMAP server indefinitely. This is not a
good thing for those of us with quotas, but it is also a bad thing for
some IMAP servers who choke to death on very large IMAP stores (or
consume an awful lot of memory dealing with them). Or you have to setup
kludgy archive operations to periodically deal with the disk space
issue.

The second mind-bending problem is that sometimes I actually do have to
use mutt, sometimes webmail (for various reasons, but one important one
in the early stages of a new email client is that notmuch just doesn't
have support for certain operations such as inline/pgp-mime handling of
emails, which means I need to open those emails in other clients that do
support that). Other people likely are going to need to use things like
iphones or blackberries. Indeed, using other clients is not an
unreasonable thing for people to do. However, switching between notmuch
and these clients is. Because the message state is stored in a DB which
is only useful for notmuch, its a dreadful nightmare to do anything
outside of the notmuch world.

Most importantly, having all of your email state in a notmuch database
format means that it can only be parsed by those tools, and is no longer
in a standard format. I think it is great that 'notmuch restore' can
deal with the sup database format, which may signal the dawn of a new
label-state standardbut still the problem is the glue to the
underlying maildir structure which provides a lot of useful information
contained in reasonably well-defined ways is totally thrown out the
window. 

I've switched mail clients enough over the past few years to know that
switching itself is a major pain. I've settled on Maildir as my storage
method of choice and in a way it is a 'client'. I can serve it via IMAP
if I need, I can read it with different mail clients and maintain state
across those mail clients, there are tools to manipulate and convert
maildirs that have matured over the years. If I switch to an
'all-in-one' label-state mail client, like notmuch, I want to be sure
that in 2 years from now if I happen to decide to change to something
else (I'm hoping this wont EVER happen because notmuch is very
promising, but I need to be honest based on past experiences here), then
I'm going to want to make sure that all of my mail is marked as read,
deleted mail is deleted and even though I was using labels for
organizational purposes in notmuch, things are still organized in some
way appropriate to folders on the filesystem.

The way it is now, if I switch to notmuch and then try to switch back,
my life is a total mess because all of the state is contained within the
notmuch database, that is frightening for the long-term, but it also
makes me very worried about simple corruption of that data store. If I
lose that state, I'm totally screwed. At least in a maildir scenario, if
I got corruption in one place it might cause me to lose one email, but
if I get corruption in my notmuch database, I had better have a recent

[notmuch] [PATCH] notmuch-restore.c: only update tags for messages that differ from dump file.

[david]

From: David Bremner brem...@unb.ca

The main feature of this patch is that it compares the list of current
tags on a message with those read by restore. Only if the two lists
differ is the tag list in the message replaced.  In my experiments this leads to
a large performance improvement.

Since I had to rewrite the parsing of tags from the dump file anyway
to keep a list of tags (in case they should be written to the
database), I decided to make it a bit more robust. It sorts the
incoming tags (it is critical for the comparison of the two tag lists
that they are both sorted), and allows arbitrary whitespace (as
determined by isspace) between tags.

The patch allocates a temporary array to keep track of the current
list of tags using calloc and grows it as neccesary using realloc.
---

This is the second version of the patch, the only difference is in the test 
on line 144 (used to be 142), and the comment before explaining it.

 notmuch-restore.c |   75 
 1 files changed, 63 insertions(+), 12 deletions(-)

diff --git a/notmuch-restore.c b/notmuch-restore.c
index 1b9598d..9d7bd17 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -18,8 +18,17 @@
  * Author: Carl Worth cwo...@cworth.org
  */
 
+#include stdlib.h
+#include string.h
+#include ctype.h
 #include notmuch-client.h
 
+#define DEFAULT_TAG_ARRAY_SIZE 2
+/* for qsort */
+static int scmp( const void *sp1, const void *sp2 )
+{
+return( strcmp(*(const char **)sp1, *(const char **)sp2) );
+}
 int
 notmuch_restore_command (unused (void *ctx), int argc, char *argv[])
 {
@@ -31,6 +40,9 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
 ssize_t line_len;
 regex_t regex;
 int rerr;
+char **tag_array=NULL;
+int tag_array_size=DEFAULT_TAG_ARRAY_SIZE;
+
 
 config = notmuch_config_open (ctx, NULL, NULL);
 if (config == NULL)
@@ -61,11 +73,18 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
  ^([^ ]+) \\(([^)]*)\\)$,
  REG_EXTENDED);
 
+/* Make an array of pointers to point to individual tokens */
+tag_array=calloc(tag_array_size,sizeof(char*));
+
 while ((line_len = getline (line, line_size, input)) != -1) {
regmatch_t match[3];
-   char *message_id, *tags, *tag, *next;
+   char *message_id, *tags,  *next;
notmuch_message_t *message;
notmuch_status_t status;
+   int tag_count;
+
+   notmuch_tags_t *tag_list;
+   int i;
 
chomp_newline (line);
 
@@ -89,26 +108,55 @@ notmuch_restore_command (unused (void *ctx), int argc, 
char *argv[])
goto NEXT_LINE;
}
 
-   notmuch_message_freeze (message);
+   next=tags;
+   tag_count=0;
+   while(*next){
+ while(*next  isspace(*next))
+   next++;
+ if (*next) {
+   while (tag_count= tag_array_size){
+ tag_array_size*=2;
+ tag_array=realloc(tag_array,tag_array_size*sizeof(char *));
+   }
+   tag_array[tag_count]=next;
+   tag_count++;
+ }
+ while (*next  !isspace(*next))
+   next++;
+ if (*next){
+   *next='\0';
+   next++;
+ }
+   }
+
+   qsort(tag_array,tag_count,sizeof(char*),scmp);
+   
+   tag_list = notmuch_message_get_tags (message);
+   i=0;
+   while (notmuch_tags_has_more (tag_list)  itag_count 
+  (strcmp(notmuch_tags_get (tag_list),tag_array[i])==0)){
+ notmuch_tags_advance (tag_list);
+ i++;
+   }
 
-   notmuch_message_remove_all_tags (message);
+   /* the only success condition is for the tag list comparison is to run
+  off the end of both lists at the same time */
+   if (notmuch_tags_has_more (tag_list) || itag_count ){
+ notmuch_message_freeze (message);
+ notmuch_message_remove_all_tags (message);
 
-   next = tags;
-   while (next) {
-   tag = strsep (next,  );
-   if (*tag == '\0')
-   continue;
-   status = notmuch_message_add_tag (message, tag);
+ for (i=0; itag_count; i++) {
+   status = notmuch_message_add_tag (message, tag_array[i]);
if (status) {
fprintf (stderr,
 Error applying tag %s to message %s:\n,
-tag, message_id);
+tag_array[i], message_id);
fprintf (stderr, %s\n,
 notmuch_status_to_string (status));
}
+ }
+ notmuch_message_thaw (message);
}
-
-   notmuch_message_thaw (message);
notmuch_message_destroy (message);
   NEXT_LINE:
free (message_id);
@@ -120,6 +168,9 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
 if (line)
free (line);
 
+if (tag_array)
+  free (tag_array);
+
 notmuch_database_close (notmuch);

Re: [notmuch] [PATCH] notmuch-restore.c: only update tags for messages that differ from dump file.

[Scott Robinson]

Excerpts from david's message of Mon Dec 07 19:14:48 -0800 2009:
 The patch allocates a temporary array to keep track of the current
 list of tags using calloc and grows it as neccesary using realloc.

The codebase has already well established the use of the talloc library. Why
did you break with that?
-- 
Scott Robinson | http://quadhome.com/

Q: Why are my replies five sentences or less?
A: http://five.sentenc.es/
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Quick thoughts on a notmuch daemon

[Carl Worth]

On Mon, 07 Dec 2009 18:51:58 -0500, micah anderson mi...@riseup.net wrote:
 I've switched mail clients enough over the past few years to know that
 switching itself is a major pain.

Absolutely.

One concept in notmuch (compared to sup) is to (eventually) avoid people
having to go through that pain by their current mail client becoming
notmuch enabled. For me, I had always liked composing email in emacs,
so I just have to add notmuch support to the existing emacs
message-mode.

Hopefully people working on other email interfaces will do similar
things, (would be great to have Anjal and Thunderbird get some notmuch
support for example).

I definitely didn't like that with sup, to get all the global-search and
tagging features one had to accept the curses UI as well.

If I switch to an
 'all-in-one' label-state mail client, like notmuch, I want to be sure
 that in 2 years from now if I happen to decide to change to something
 else (I'm hoping this wont EVER happen because notmuch is very
 promising, but I need to be honest based on past experiences here), then
 I'm going to want to make sure that all of my mail is marked as read,
 deleted mail is deleted and even though I was using labels for
 organizational purposes in notmuch, things are still organized in some
 way appropriate to folders on the filesystem.

I appreciate your caution before making the commitment to a mail
client. I think that's very wise. [*]

And I think that for the case of I'm giving up on notmuch and want to
switch to something else you can be quite assured that notmuch will
allow you to take care of everything you need. At a system level, it's
really an almost trivial matter to implement everything you describe
above as small shell scripts based on notmuch search commands. And I
can only imagine support for that kind of thing getting better all the
time.

 The way it is now, if I switch to notmuch and then try to switch back,
 my life is a total mess because all of the state is contained within the
 notmuch database, that is frightening for the long-term, but it also
 makes me very worried about simple corruption of that data store. If I
 lose that state, I'm totally screwed. At least in a maildir scenario, if
 I got corruption in one place it might cause me to lose one email, but
 if I get corruption in my notmuch database, I had better have a recent
 backup or I am screwed. 

I highly recommend making very regular backups of the output of notmuch
dump. It's a quick process to run, and the file it creates is not
large. It's also nicely sorted so that if you are doing some kind of
incremental backup, that should work well.

 I like to think that this concept of label-state, indexable
 storage-backed mail clients is the dawn of a new age, just as maildir
 was to mbox, but I'm still scared because there is no mb2md equivalent
 yet.

I think we're really close to where you could write a notmuch2md script
as a very tiny shell script:

for tag in $(notmuch search --for=tags *); do
notmuch search --for=messages --output=maildir:/some/dir/$tag tag:$tag
done

Being able to do that kind of scriptable output is pretty powerful. And
we're not missing anything fundamental in the system to be able to
support that, (just a little bit of support for the new command-line
options like --for and --output).

-Carl

[*] I know I went through a similarly cautious evaluation when switching
away from CVS. CVS made the transition away so painful that I was
determined to choose a system that satisfied me on two criteria:

1. The system seemed well-designed enough that I could imagine it
   surviving forever.

2. That when realities exceeded my imagination, the system wouldn't make
   it hard for me to switch away.

So far I'm still quite happy with git on both points.


pgpAJYJXcOJTM.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch