[ANNOUNCE] notmuchfs - A virtual maildir file system for notmuch queries

2012-10-01 Thread Tim Stoakes
Vladimir Marek(Vladimir.Marek at Oracle.COM)@011012-11:40:
> > Why?

I know - proceed to the next paragraph for my explanation. Summary -
mutt-kz wasn't done when I did this.

Tim

-- 
Tim Stoakes


Re: [ANNOUNCE] notmuchfs - A virtual maildir file system for notmuch queries

2012-10-01 Thread Tim Stoakes
Vladimir Marek(vladimir.ma...@oracle.com)@011012-11:40:
  Why?

I know - proceed to the next paragraph for my explanation. Summary -
mutt-kz wasn't done when I did this.

Tim

-- 
Tim Stoakes
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[ANNOUNCE] notmuchfs - A virtual maildir file system for notmuch queries

2012-09-29 Thread Tim Stoakes
Hello fellow notmuch-ers,

What is notmuchfs?
--
Notmuchfs implements a virtual file system which creates maildirs from notmuch
mail query results. This is useful for using notmuch with tools which are not
aware of notmuch, only maildirs - such as mutt.

Basically you 'mkdir' your query strings, then point your tool of choice at
that directory, and it turns into a virtual maildir containing the query
results.


How does it work?
-
A notmuchfs virtual file system is mounted referencing a particular existing
(real) directory called the 'backing store'.

Directories created within the backing store, when read, appear to have the
standard maildir format (e.g. cur/, new/, tmp/ sub-directories).

Notmuchfs interprets the names of these backing store directories as notmuch
queries, and fills the cur/ virtual maildir sub-directory with virtual messages
which are the result of executing that query (at the instant in time that the
directory is read).

Each virtual maildir message file, when read, appears to have the exact content
of the message referenced by the notmuch query, augmented with an 'X-Label'
header generated automatically by notmuchfs, containing the notmuch tags of
that message.


Additional 'glue' scripts are provided to augment this, to integrate it more
tightly with MUA-like tools, specifically 'mutt', giving the ability to alter
notmuch tags, create new queries, etc.



Why?

I wanted to use notmuch with mutt. I didn't feel that ploughing though mutt's
codebase to add tight integration was the quickest way to a result, so
notmuchfs was born.

I developed notmuchfs in 2011, but I am only now able to release it. In the
mean-time, the excellent 'mutt-kz' was released, scratching my initial itch.

Notmuchfs solves a broader problem - e.g. a notmuch-aware courier-imap is not
too much of a stretch - so I think there is room for the different approaches.


More information

 https://github.com/tsto/notmuchfs
 https://github.com/tsto/notmuchfs/blob/master/README.md
 https://github.com/tsto/notmuchfs/blob/master/README.MUTT.md
 https://github.com/tsto/notmuchfs/blob/master/ISSUES



Perhaps this will scratch someone else's itch too.

Tim

-- 
Tim Stoakes


[ANNOUNCE] notmuchfs - A virtual maildir file system for notmuch queries

2012-09-29 Thread Tim Stoakes
Hello fellow notmuch-ers,

What is notmuchfs?
--
Notmuchfs implements a virtual file system which creates maildirs from notmuch
mail query results. This is useful for using notmuch with tools which are not
aware of notmuch, only maildirs - such as mutt.

Basically you 'mkdir' your query strings, then point your tool of choice at
that directory, and it turns into a virtual maildir containing the query
results.


How does it work?
-
A notmuchfs virtual file system is mounted referencing a particular existing
(real) directory called the 'backing store'.

Directories created within the backing store, when read, appear to have the
standard maildir format (e.g. cur/, new/, tmp/ sub-directories).

Notmuchfs interprets the names of these backing store directories as notmuch
queries, and fills the cur/ virtual maildir sub-directory with virtual messages
which are the result of executing that query (at the instant in time that the
directory is read).

Each virtual maildir message file, when read, appears to have the exact content
of the message referenced by the notmuch query, augmented with an 'X-Label'
header generated automatically by notmuchfs, containing the notmuch tags of
that message.


Additional 'glue' scripts are provided to augment this, to integrate it more
tightly with MUA-like tools, specifically 'mutt', giving the ability to alter
notmuch tags, create new queries, etc.



Why?

I wanted to use notmuch with mutt. I didn't feel that ploughing though mutt's
codebase to add tight integration was the quickest way to a result, so
notmuchfs was born.

I developed notmuchfs in 2011, but I am only now able to release it. In the
mean-time, the excellent 'mutt-kz' was released, scratching my initial itch.

Notmuchfs solves a broader problem - e.g. a notmuch-aware courier-imap is not
too much of a stretch - so I think there is room for the different approaches.


More information

 https://github.com/tsto/notmuchfs
 https://github.com/tsto/notmuchfs/blob/master/README.md
 https://github.com/tsto/notmuchfs/blob/master/README.MUTT.md
 https://github.com/tsto/notmuchfs/blob/master/ISSUES



Perhaps this will scratch someone else's itch too.

Tim

-- 
Tim Stoakes
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] Add pseudo-compatibility with gmime 2.6

2012-01-17 Thread Tim Stoakes
Thomas Jost(schnouki at schnouki.net)@170112-00:56:
> There are lots of API changes in gmime 2.6 crypto handling. By adding
> preprocessor directives, it is however possible to add gmime 2.6 compatibility
> while preserving compatibility with gmime 2.4 too.
> 
> This is mostly based on id:"8762i8hrb9.fsf at bookbinder.fernseed.info".
> 
> This was tested against both gmime 2.6.4 and 2.4.31. With gmime 2.4.31, the
> crypto tests all work fine (as expected). With gmime 2.6.4, one crypto test
> fails (signature verification with signer key unavailable) but this will be 
> hard
> to fix since the new API does not report the reason why a signature 
> verification
> fails (other than the human-readable error message).

+1 from me.

I literally just coded up a similar patch, and the first email I see
with the newly compiled notmuch is this one. I've now tried Thomas's
(better) patch with gmime-2.6.4 and notmuch
7ddd849015759a329bf8fef8c8b5a93359408962, and can confirm it works fine
for me. This build breakage is otherwise quite painful.

Thanks Thomas.

Tim

-- 
Tim Stoakes


Tag timestamps and synchronization

2011-01-25 Thread Tim Stoakes
dm-list-email-notmuch at scs.stanford.edu(dm-list-email-notmuch at 
scs.stanford.edu)@240111-11:10:
> One of the features I would like to see from notmuch is an easier
> ability to synchronize tags across machines.  At the very least, I
> would need either incremental dump and restore, or some way to
> communicate arbitrary tags to a local imap server that shares
> notmuch's maildir (much as notmuch currently syncs the standard tags),
> so that I synchronize two maildirs with a tool like offlineimap.

David,

I do something like this by using some shell scripts with formail, to
'store' notmuch tags into the X-Label headers of the individual mails.
Offlineimap then syncs these headers. If I need the tags to become
notmuch-ified on the target, I just scan all the mail's X-Label headers.

(Actually it's better than this, since I use maildrop to set notmuch
tags with notmuch-deliver, *and* set X-Label headers to the same thing,
at mail delivery time. Then I use keybindings and shell scripts in mutt
such that whenever I retag a message, it is pushed to both notmuch and
X-Label.)

I'm happy to share this hack glue if it would help.

This is not great for a few reasons - there are a ton of moving parts,
and some double-work. If notmuch could index X-Label headers (a coming
feature I hear) then this would be much cleaner.

This is just one way of doing it, that works for me...

Tim

> As Carl pointed out to me in private email, there has been some
> previous discussion in the following thread:
> 
>   notmuch show id:87hbfnmiux.fsf at yoom.home.cworth.org
> 
> Based on that thread, there seems to be some desire for notmuch to
> keep track of a per-message timestamp when the flags were last
> updated.  This would allow much easier expiration for people who want
> the deleted tag.  It would also allow incremental dump and restore of
> tags, which is exactly what I need to sync tags across servers with
> reasonable amounts of bandwidth.
> 
> Metadata timestamps are one of those things that probably have a lot
> of different applications, so since Carl is considering a new database
> format for the next release anyway, perhaps it also makes sense to add
> a metadata change time for each messages.
> 
> The timestamp would be included in "dump" output, and you could
> request a dump of changes since a particular time.  On restore, you
> might have several options:
> 
>   - overwrite: always set the new tags and timestamp in the database
> to the value in the restore data.
> 
>   - update: always set the tags, but update the to the current time.
> 
>   - conditional T: update only if the message metadata has not been
> updated since time T.
> 
> To sync flags, then you just need to keep track of the last time you
> synced with a particular server--call this time T.  Do a dump since
> time T, upload to server, do a conditional restore for time T on
> server.  Finally do a partial dump from time T on the server and an
> overwrite import on the client.  (This policy makes changes on the
> server always override conflicting ones on the client--perhaps people
> want other policies, like union of the tags, etc.)
> 
> 
> Second, there seems to be some desire in that thread to sync with IMAP
> flags.  This would be particularly great, but the easies way to do it
> is probably *not* to try to implement IMAP, but rather to use an
> existing IMAP server and just modify the maildir so that the IMAP
> server will pick up the flags.
> 
> In the case of dovecot, the arbitrary tag format is very simple.  Each
> maildir has a file called dovecot-keywords mapping numbers 0, 1,
> ... to keywords.  Then mail file names contain lower-case letters for
> the flags they are marked with--0 => a, 1 => b, etc.--allowing up to
> 26 arbitrary tags for each maildir.  One could probably sync to
> dovecot's maildir format relatively easily in a script given
> incremental dump and restore of tags.  Or possibly notmuch could
> natively support dovecot as one of multiple back-end tag storage
> schemes.
> 
> Having a static tag mapping in the .notmuch-config file would be much
> better than hard-coding flag2tag.  However, I'm not sure it's
> sufficient.  The reason is that if you ever completely delete a tag
> (e.g., you have "todo", and "meeting" tags and periodically have no
> messages in either categories in a given mail folder), then an IMAP
> server like dovecot might end up re-allocating the letters
> corresponding to those tags in a different order.  Also, at least for
> dovecot, the flag mappings are per-folder, which you kind of want
> since you are limited to 26 non-standard tags, so global values might
> not work.
> 
> I'm curious to hear people's thoughts/reactions?
> 
> David

-- 
Tim Stoakes


thread sorting ideas

2010-06-23 Thread Tim Stoakes
Dirk Hohndel(hohndel at infradead.org)@210610-09:03:
> Here is a feature that I could really use some times... 
> 
> Instead of sorting the threads based on the date of the first message in
> the thread, sort them based on the newest message in the thread. So if I
> take a quick look at the bottom (I am an "oldest first" kinda person) of
> a search and I see all the threads that have the newest messages.

Indeed, this would be akin to mutt's:
 set sort=threads
 set sort_aux=last-date
I think. Indispensable!

Tim

> My suggestion would be to turn 'o' into a two key command:
> 
> o-o : oldest first, sort by oldest message in thread
> o-O : oldest first, sort by newest message in thread
> o-n : newest first, sort by oldest message in thread
> o-N : newest first, sort by newest message in thread
> o-s : sort by subject?
> o-z : unthreaded, sort by message size?
> 
> you can come up with many more sort ideas...

-- 
Tim Stoakes


Re: thread sorting ideas

2010-06-23 Thread Tim Stoakes
Dirk Hohndel(hohn...@infradead.org)@210610-09:03:
 Here is a feature that I could really use some times... 
 
 Instead of sorting the threads based on the date of the first message in
 the thread, sort them based on the newest message in the thread. So if I
 take a quick look at the bottom (I am an oldest first kinda person) of
 a search and I see all the threads that have the newest messages.

Indeed, this would be akin to mutt's:
 set sort=threads
 set sort_aux=last-date
I think. Indispensable!

Tim

 My suggestion would be to turn 'o' into a two key command:
 
 o-o : oldest first, sort by oldest message in thread
 o-O : oldest first, sort by newest message in thread
 o-n : newest first, sort by oldest message in thread
 o-N : newest first, sort by newest message in thread
 o-s : sort by subject?
 o-z : unthreaded, sort by message size?
 
 you can come up with many more sort ideas...

-- 
Tim Stoakes
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[notmuch] envelope-to

2010-02-25 Thread Tim Stoakes
James Vasile(james at hackervisions.org)@240210-14:59:
> Can notmuch search for envelope-to:?  Or for that matter, can it
> search for arbitrary headers?
> 
> I get a lot of BCC mail.  The envelope-to is a good way for me to know
> which of my various addresses was BCC'd.

I'm very new here, but I think the answer is 'no' - notmuch only seems
to index a few headers. Which is a pity - I use an extensive system of
X-Label headers, and would like to convert them to notmuch tags - I just
can't seem to do it. I hope I'm wrong.

Tim

-- 
Tim Stoakes


[notmuch] [PATCH] notmuch: Respect maildir message flags

2010-02-10 Thread Tim Stoakes
Michiel Buddingh'(michiel at michielbuddingh.net)@061209-20:55:
>
...
> A new patch is attached.  Apologies for the rather verbose Maildir
> handling logic, but I couldn't find a way to minimize the calls to
> is_maildir that was both neat and readable.


Hi notmuch-ers,

My apologies for dredging up an old thread. I don't want to restart the
religious war over whether notmuch should respect Maildir flags -
suffice to say that *I* want that, and the patch posted by Michiel
seemed to be the best way to make that happen.

Since it no longer applies cleanly, I've ported it forward to
79d3f9773c58d6fd7113871362687d8cfc0b1a59, to save someone else the
trouble. It works for me, but that's all the testing I've done.

Tim

-- 
Tim Stoakes



---
 notmuch-new.c |   86 -
 1 files changed, 85 insertions(+), 1 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index f25c71f..3264653 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -39,6 +39,7 @@ typedef struct {
 int total_files;
 int processed_files;
 int added_messages;
+int tag_maildir;
 struct timeval tv_start;

 _filename_list_t *removed_files;
@@ -169,6 +170,60 @@ _entries_resemble_maildir (struct dirent **entries, int 
count)
 return 0;
 }

+/* Tag new mail according to its Maildir attribute flags.
+ *
+ * Test if the mail file's filename contains any of the
+ * standard Maildir attributes, and translate these to
+ * the corresponding standard notmuch tags.
+ *
+ * If the message is not marked as 'seen', or if no
+ * flags are present, tag as 'inbox, unread'.
+ */
+static void
+derive_tags_from_maildir_flags (notmuch_message_t *message,
+   const char * path)
+{
+int seen = FALSE;
+int end_of_flags = FALSE;
+size_t l = strlen(path);
+
+/* Non-experimental message flags start with this */
+char * i = strstr(path, ":2,");
+i = (i) ? i : strstr(path, "!2,"); /* This format is used on VFAT */
+if (i != NULL) {
+   i += 3;
+   for (; i < (path + l) && !end_of_flags; i++) {
+   switch (*i) {
+   case 'F' :
+   notmuch_message_add_tag (message, "flagged");
+   break;
+   case 'R': /* replied */
+   notmuch_message_add_tag (message, "answered");
+   break;
+   case 'D':
+   notmuch_message_add_tag (message, "draft");
+   break;
+   case 'S': /* seen */
+   seen = TRUE;
+   break;
+   case 'T': /* trashed */
+   notmuch_message_add_tag (message, "deleted");
+   break;
+   case 'P': /* passed */
+   notmuch_message_add_tag (message, "forwarded");
+   break;
+   default:
+   end_of_flags = TRUE;
+   break;
+   }
+   }
+}
+
+if (i == NULL || !seen) {
+   tag_inbox_and_unread (message);
+}
+}
+
 /* Examine 'path' recursively as follows:
  *
  *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -222,6 +277,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 notmuch_filenames_t *db_subdirs = NULL;
 struct stat st;
 notmuch_bool_t is_maildir, new_directory;
+int maildir_detected = -1;

 if (stat (path, )) {
fprintf (stderr, "Error reading directory %s: %s\n",
@@ -301,6 +357,28 @@ add_files_recursive (notmuch_database_t *notmuch,
continue;
}

+   /* If this directory is a Maildir folder, we need to
+* ignore any subdirectories marked tmp/, and scan for
+* Maildir attributes on messages contained in the sub-
+* directories 'new' and 'cur'. */
+   if (maildir_detected != 0 &&
+   entry->d_type == DT_DIR &&
+   ((strcmp (entry->d_name, "tmp") == 0) ||
+(strcmp (entry->d_name, "new") == 0) ||
+(strcmp (entry->d_name, "cur") == 0))) {
+
+if (maildir_detected == -1) {
+  maildir_detected = _entries_resemble_maildir(fs_entries, num_fs_entries);
+}
+if (maildir_detected == 1) {
+  if (strcmp (entry->d_name, "tmp") == 0) {
+continue;
+  } else {
+state->tag_maildir = TRUE;
+  }
+}
+  }
+
next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
status = add_files_recursive (notmuch, next, state);
if (status && ret == NOTMUCH_STATUS_SUCCESS)
@@ -412,7 +490,12 @@ add_files_recursive (notmuch_database_t *notmuch,
/* success */
case NOTMUCH_STATUS_SUCCESS:
state->added_messages++;
-   tag_inbox_and_unread (message);
+   if (state->tag_maildir) {
+   der

Re: [notmuch] [PATCH] notmuch: Respect maildir message flags

2010-02-09 Thread Tim Stoakes
Michiel Buddingh'(mich...@michielbuddingh.net)@061209-20:55:

...
 A new patch is attached.  Apologies for the rather verbose Maildir
 handling logic, but I couldn't find a way to minimize the calls to
 is_maildir that was both neat and readable.


Hi notmuch-ers,

My apologies for dredging up an old thread. I don't want to restart the
religious war over whether notmuch should respect Maildir flags -
suffice to say that *I* want that, and the patch posted by Michiel
seemed to be the best way to make that happen.

Since it no longer applies cleanly, I've ported it forward to
79d3f9773c58d6fd7113871362687d8cfc0b1a59, to save someone else the
trouble. It works for me, but that's all the testing I've done.

Tim

-- 
Tim Stoakes



---
 notmuch-new.c |   86 -
 1 files changed, 85 insertions(+), 1 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index f25c71f..3264653 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -39,6 +39,7 @@ typedef struct {
 int total_files;
 int processed_files;
 int added_messages;
+int tag_maildir;
 struct timeval tv_start;
 
 _filename_list_t *removed_files;
@@ -169,6 +170,60 @@ _entries_resemble_maildir (struct dirent **entries, int 
count)
 return 0;
 }
 
+/* Tag new mail according to its Maildir attribute flags.
+ *
+ * Test if the mail file's filename contains any of the
+ * standard Maildir attributes, and translate these to
+ * the corresponding standard notmuch tags.
+ *
+ * If the message is not marked as 'seen', or if no
+ * flags are present, tag as 'inbox, unread'.
+ */
+static void
+derive_tags_from_maildir_flags (notmuch_message_t *message,
+   const char * path)
+{
+int seen = FALSE;
+int end_of_flags = FALSE;
+size_t l = strlen(path);
+
+/* Non-experimental message flags start with this */
+char * i = strstr(path, :2,);
+i = (i) ? i : strstr(path, !2,); /* This format is used on VFAT */
+if (i != NULL) {
+   i += 3;
+   for (; i  (path + l)  !end_of_flags; i++) {
+   switch (*i) {
+   case 'F' :
+   notmuch_message_add_tag (message, flagged);
+   break;
+   case 'R': /* replied */
+   notmuch_message_add_tag (message, answered);
+   break;
+   case 'D':
+   notmuch_message_add_tag (message, draft);
+   break;
+   case 'S': /* seen */
+   seen = TRUE;
+   break;
+   case 'T': /* trashed */
+   notmuch_message_add_tag (message, deleted);
+   break;
+   case 'P': /* passed */
+   notmuch_message_add_tag (message, forwarded);
+   break;
+   default:
+   end_of_flags = TRUE;
+   break;
+   }
+   }
+}
+
+if (i == NULL || !seen) {
+   tag_inbox_and_unread (message);
+}
+}
+
 /* Examine 'path' recursively as follows:
  *
  *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -222,6 +277,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 notmuch_filenames_t *db_subdirs = NULL;
 struct stat st;
 notmuch_bool_t is_maildir, new_directory;
+int maildir_detected = -1;
 
 if (stat (path, st)) {
fprintf (stderr, Error reading directory %s: %s\n,
@@ -301,6 +357,28 @@ add_files_recursive (notmuch_database_t *notmuch,
continue;
}
 
+   /* If this directory is a Maildir folder, we need to
+* ignore any subdirectories marked tmp/, and scan for
+* Maildir attributes on messages contained in the sub-
+* directories 'new' and 'cur'. */
+   if (maildir_detected != 0 
+   entry-d_type == DT_DIR 
+   ((strcmp (entry-d_name, tmp) == 0) ||
+(strcmp (entry-d_name, new) == 0) ||
+(strcmp (entry-d_name, cur) == 0))) {
+
+if (maildir_detected == -1) {
+  maildir_detected = _entries_resemble_maildir(fs_entries, num_fs_entries);
+}
+if (maildir_detected == 1) {
+  if (strcmp (entry-d_name, tmp) == 0) {
+continue;
+  } else {
+state-tag_maildir = TRUE;
+  }
+}
+  }
+
next = talloc_asprintf (notmuch, %s/%s, path, entry-d_name);
status = add_files_recursive (notmuch, next, state);
if (status  ret == NOTMUCH_STATUS_SUCCESS)
@@ -412,7 +490,12 @@ add_files_recursive (notmuch_database_t *notmuch,
/* success */
case NOTMUCH_STATUS_SUCCESS:
state-added_messages++;
-   tag_inbox_and_unread (message);
+   if (state-tag_maildir) {
+   derive_tags_from_maildir_flags (message,
+   entry-d_name);
+   } else {
+   tag_inbox_and_unread (message);
+   }
break;
/* Non-fatal issues (go