Questions about importing mail (mbox)

2011-03-20 Thread Mueen Nawaz

Hi,

I'm trying to experiment with notmuch. 

As I understand it, notmuch does not handle mbox for input. The problem
is that all my mail is currently in mbox format.

So I first tried converting mbox to maildir using mb2md.

It didn't do a good job. When I subsequently tried importing to notmuch,
notmuch complained about lots of non-mail files - I confirmed that
indeed mb2md had botched converting those emails.

So then I tried to convert to mh format using Sylpheed. This seemed to
go well, but then when importing to notmuch, it complained again for
about 20 emails, and a manual check confirmed that some messages did not
get converted properly to mh (they don't show up in Sylpheed).

And then I noticed another discrepancy. mutt shows that I started with
44473 messages in mbox. When I imported into Sylpheed, it showed 44482
messages (no idea where the extra 9 came from). However, notmuch is
reporting that it processed 44482 files, but that it added 35602
messages.

Why only 35602 (it complained for only about 20 messages)? A search
confirmed that some messages that show up in both mutt (in mbox) and
Sylpheed (in mh format) were not indexed.

So I want to know: When you guys switched to notmuch, how did you ensure
you did not miss any emails. I really, really, really don't want to lose
any emails in this process!

Thanks.



RFC: notmuch powered (personal) (end-to-end) e-mail system

2011-03-20 Thread Ciprian Dorin Craciun
Hello all! (Sorry for the long email.)

I'm "struggling" for some time to get rid of the current
"de-facto" email solutions (i.e. GMail, Zimbra), and I've passively
observed for some time the notmuch project and community.

Although I've forwarded all my email to a single account, and I'm
currently mirroring my GMail account locally (by using `mbsync`),
index it by using notmuch, and I collect spam mails for later filter
training, unfortunately I'm unable to "convert" because the current
notmuch-powered solutions have (some of) the following shortcomings (I
don't want to offend anyone, so please take these as observations):
* the most feature full UI is the Emacs one -- thus limited remote
access (I mean from an arbitrary computer with only a web-browser);
(and I'm not a very big fan of Emacs;)
* most are still dependent on external IMAP systems -- this is not
a problem with notmuch itself, but for the integrating clients;
* SPAM -- as above -- is not integrated;
* filtering (tag applying) is not automatic (as in integrated in
notmuch itself or the client), but triggered through external scripts;

As such I'm thinking on implementing a custom end-to-end email
system and I would like to hear your feedback before embarking on such
a task.

I'm targeting the following features:
* (inbound) SMTP integration, thus once an email is received it is
automatically pushed through the system; (I'm primarily targeting
those users that afford to run their own SMTP server; but the solution
could still be adapted for those that only want the other features;)
* automatic spam filtering, and tag applying;
* automatic email triggers based on tags (such as user
notifications, forwarding, etc.)
* remote RPC-like access to the whole system;
* remote Web user interface;

About the overall architecture I'm thinking on adopting the following:
* in general the whole system is decomposed in independent
components (long-lived OS daemons) that each one does a particular job
(see below);
* all the components communicate between each-other through a
message queue system (for example ZeroMQ or RabbitMQ);
* all the communication is JSON based;

The components would be:
* SMTP inbound gateway -- for example I could take qmail or
Postfix and replace the delivery agent with a custom process that
pushes the email into the system; (any other solution suggestions?);
* email store -- as the name suggests it is a simple
key-value-like store that should persist raw email-messages; it should
be as robust as possible, and its contents should be the only thing
needed to reconstruct all the other derived data; (I could use here a
simple process that maintains a maildir, I could go also with a
BerkeleyDB wrapper, or even something more sophisticated;)
* spam filter -- which either classifies the email or trains the
spam filter; (for example I would use bogofilter;)
* email index -- this is where notmuch would come into play; it
would be fed with emails, which it would automatically apply tags and
issue trigger notifications based on tags; it also maintains a set of
filters and tags to automatically apply;
* (maybe) a coordinator that should delegate and monitor requests
to the above components; but if I'm using RabbitMQ and carefully
designing the above components, they could drive each other;
* restful web service that would intermediate access to all the
above components;

For now I have the following uncertainties:
* how should I handle multiple users? I think each user should
have it's own store / notmuch / bogofilter instance (at least in terms
of storage if not even in terms of separate daemon);
* should I keep the emails is a file-system, or a key-value store?
(the file-system is more bug-free, but I'm confident that a BerkeleyDB
instance would be more efficient);
* should I use libnotmuch or for starters just make a notmuch tool wrapper;
* and the most pressing one, transactions: I would like that at no
point does a message get half processed or lost; as such I need
notmuch to behave transactionally -- indexing the message and tagging
it should be atomic and durable; (is there a way with libnotmuch to
control the underlaying BerkeleyDB database?)

Suggestions? Considerations?

Ciprian.


RFC: notmuch powered (personal) (end-to-end) e-mail system

2011-03-20 Thread Austin Clements
Much of the beauty of notmuch is how few assumptions it makes about
your mail system.  It plays well with others.  For example, one deep
insight of notmuch is that it *doesn't* require a custom mail store,
even though a more obvious design might; in fact, it doesn't even
require Maildir.

That said, I think I can see where you're coming from and I also think
you're targeting some of the deficiencies of notmuch, but I also think
you're overengineering the solution.  As a result of notmuch's
simplicity, a fully working mail setup requires a lot of moving parts
besides notmuch and it can take a while for a new user to set all that
up, especially if they're migrating wholesale from some external mail
setup.

On Sun, Mar 20, 2011 at 10:07 AM, Ciprian Dorin Craciun
 wrote:
> ? ?As such I'm thinking on implementing a custom end-to-end email
> system and I would like to hear your feedback before embarking on such
> a task.
>
> ? ?I'm targeting the following features:
> ? ?* (inbound) SMTP integration, thus once an email is received it is
> automatically pushed through the system; (I'm primarily targeting
> those users that afford to run their own SMTP server; but the solution
> could still be adapted for those that only want the other features;)

As others have mentioned, see notmuch-deliver.  I and others have also
suggested inotify support for notmuch before, which would make the
inbound mail mechanism (be it SMTP, IMAP fetching, or whatever)
completely unaware of notmuch, offer some other benefits (for example,
if mail is manipulated outside notmuch via IMAP), and is highly
discoverable for new users (just have notmuch setup ask if they want
notmuch to monitor for new email and then fire up an inotify daemon
the first time notmuch is called).

> ? ?* automatic spam filtering, and tag applying;
> ? ?* automatic email triggers based on tags (such as user
> notifications, forwarding, etc.)

Obviously the above two can be scripted, but I agree that it's
unsatisfying that every user needs to roll their own delivery script.
While tagging and triggering are highly personal, they're not *so*
personal that everyone needs a completely custom solution.  This
should be more approachable.  I'm not sure what the best answer here
is, but I don't think it requires it requires integration with a
monolithic system to do right.

> ? ?* remote RPC-like access to the whole system;

This is another deep insight of notmuch.  It already has an awesome
RPC interface: the CLI.  Perhaps your actual problem is that the only
supported remote transport protocol is SSH.  This comes with a lot of
benefits (authentication, RPC pipelining), but also a lot of baggage
(a full SSH client on the client side).  I've thought about this in
the context of both an HTTP client and an Android client and in both
cases I concluded that a simple HTTPS transport wrapped around the
notmuch CLI would be the way to go.  Just put the CLI arguments in a
POST and send the JSON on stdout back.  This is trivial to prototype
as a Python CGI script, easy to build as a standalone Python server,
and not especially hard to build as a robust C server.

> ? ?* remote Web user interface;

A good web UI would be fantastic.  Based on the rest of your email, I
get the impression this was a requirements driver from much of the
above, especially the integrated tagging/triggering and RPC access.
I've already suggested a simple solution to the RPC problem.  For
tagging/triggering, it's probably worth developing a solution that
allows for machine-editable rules (ideally retaining
user-editableness), which would make it possible to integrate filter
management in to a web UI.  This could be as simple as a standard
delivery script that operates from some simple rule database.


RFC: notmuch powered (personal) (end-to-end) e-mail system

2011-03-20 Thread Ben Gamari
On Sun, 20 Mar 2011 16:07:50 +0200, Ciprian Dorin Craciun  wrote:
> Hello all! (Sorry for the long email.)
> 
> [snip]
> 
> * the most feature full UI is the Emacs one -- thus limited remote
> access (I mean from an arbitrary computer with only a web-browser);
> (and I'm not a very big fan of Emacs;)
> 
There have been a few attempts to put together an HTML front-end to
notmuch[1]. None have made it very far though. It would be nice to see
this space filled.

> * most are still dependent on external IMAP systems -- this is not
> a problem with notmuch itself, but for the integrating clients;
> 
Not entirely sure what you mean by this. You could easily use
e.g. notmuch-deliver as the local delivery agent with a SMTP server and
you'd have no need for IMAP.

> * SPAM -- as above -- is not integrated;
> 
Nor should it be. Mail indexing, viewing, composing, and
filtering are all orthogonal parts of a mail system. It takes all of
ten lines to invoke a spam filter in your filter script.

> * filtering (tag applying) is not automatic (as in integrated in
> notmuch itself or the client), but triggered through external scripts;
> 
Again, there is no reason why this should be incorporated into your
mail indexer.

> As such I'm thinking on implementing a custom end-to-end email
> system and I would like to hear your feedback before embarking on such
> a task.
> 
Notmuch works so well for its audience because it adheres to the UNIX
philosophy of "do one thing and do it well." The goal of an "integrated
end-to-end" mail system might sound nice, but IMHO it's a recipe for a
kludgey, unmaintainable nightmare which is mediocre at performing its
task, on a good day. Perhaps I'm misunderstanding your proposal but it
seems to me like you are taking an easy, already solved problem and
turning it into a difficult one.

> I'm targeting the following features:
> * (inbound) SMTP integration, thus once an email is received it is
> automatically pushed through the system; (I'm primarily targeting
> those users that afford to run their own SMTP server; but the solution
> could still be adapted for those that only want the other features;)
> 
Is there something wrong with Postfix with notmuch-deliver as a LDA?

> * automatic spam filtering, and tag applying;
>
A traditional sorting script with bogofilter/spamassassin?

> * automatic email triggers based on tags (such as user
> notifications, forwarding, etc.)
>
Again, a sorting script?

> * remote RPC-like access to the whole system;
> 
What's wrong with SSH?

> * remote Web user interface;
> 
Nothing fills this need currently. Feel free to write up something but
please don't couple it to some all-inclusive beheamoth of a project.

Personally, I would think more carefully about this project before
proceding. It sounds like you intend on reinventing various portions of
the wheel several times. Nothing you have listed is difficult to do with
a few scripts, notmuch, and an SMTP server. 

> About the overall architecture I'm thinking on adopting the following:
> * in general the whole system is decomposed in independent
> components (long-lived OS daemons) that each one does a particular job
> (see below);
> * all the components communicate between each-other through a
> message queue system (for example ZeroMQ or RabbitMQ);
> * all the communication is JSON based;
> 
> The components would be:
> * SMTP inbound gateway -- for example I could take qmail or
> Postfix and replace the delivery agent with a custom process that
> pushes the email into the system; (any other solution suggestions?);
> * email store -- as the name suggests it is a simple
> key-value-like store that should persist raw email-messages; it should
> be as robust as possible, and its contents should be the only thing
> needed to reconstruct all the other derived data; (I could use here a
> simple process that maintains a maildir, I could go also with a
> BerkeleyDB wrapper, or even something more sophisticated;)
> * spam filter -- which either classifies the email or trains the
> spam filter; (for example I would use bogofilter;)
> * email index -- this is where notmuch would come into play; it
> would be fed with emails, which it would automatically apply tags and
> issue trigger notifications based on tags; it also maintains a set of
> filters and tags to automatically apply;
> * (maybe) a coordinator that should delegate and monitor requests
> to the above components; but if I'm using RabbitMQ and carefully
> designing the above components, they could drive each other;
> * restful web service that would intermediate access to all the
> above components;
> 
> For now I have the following uncertainties:
> * how should I handle multiple users? I think each user should
> have it's own store / notmuch / bogofilter instance (at least in terms
> of storage if not even in terms of separate daemon);
> * should I keep the 

Org-mode support

2011-03-20 Thread micah anderson
On Thu, 17 Mar 2011 00:28:30 +0100, Matthieu Lemerre  wrote:

> > I have one question, how do you add multiple query arguments? For
> > example, in my org file I tried 'notmuch-search:tag:notmuch' - this
> > worked great, but if I did 'notmuch-search:tag:notmuch and subject:Org'
> > the link only works up until the first space.
> 
> This is because org-mode links must be url encoded (this is a org-mode
> limitation, could do nothing about it). So in links that contain spaces,
> such as yours, spaces must be replaced by %20, like this:
> 
> [[notmuch-search:toto%20and%20tata][Notmuch search: toto and tata]]

Thanks, I'm somewhat new to org-mode, so I appreciate that information!

> > I also noticed that David Bremner has done something similar[0] and I
> > wonder if you have looked at his code[1], or if David has looked at
> > yours. Perhaps the start of a collaboration?
> 
> If I remember correctly, I specifically forbid myself to look into this
> code because David had copyright issues with its employer. But I believe
> that both codes are doing essentially the same thing.

It sounds like you are right about the copyright issues. 

I noticed one thing about your org add-on that would be nice to improve:
it doesn't appear as if you support linking to a thread in
notmuch-search-mode. If I have a search buffer, generated by "notmuch
search tag:notmuch" and I find in there your message with the Subject
"Org-mode support" and I put my cursor on it and do M-x org-store-link,
the link that is generated is to "Notmuch search: tag:notmuch" rather
than to the thread id that is under the cursor. A minor thing, but could
be quite useful.

thanks!
micah
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110320/fb9eb2f3/attachment.pgp>


RFC: notmuch powered (personal) (end-to-end) e-mail system

2011-03-20 Thread Brett Viren
On Sun, Mar 20, 2011 at 10:07 AM, Ciprian Dorin Craciun
 wrote:

> ? ?I'm "struggling" for some time to get rid of the current
> "de-facto" email solutions (i.e. GMail, Zimbra), and I've passively
> observed for some time the notmuch project and community.

It sounds like what you want *is* GMail (I don't know Zimbra) but just
that you want it running on your own box instead of on Google's
servers.

> ? ?Suggestions? Considerations?

Based on what you wrote, I think BerkeleyDB will be too limiting.
I suggest for you to look into DBMail[1] for the mail store.


-Brett.

[1] http://www.dbmail.org/


[PATCH 0/1] emacs: Use "message-cited-text" instead of "message-cited-text-face"

2011-03-20 Thread Pieter Praet
On Wed, 16 Mar 2011 08:20:24 -0300, David Bremner  wrote:
> [...] I notice there is no commit message in the body, so I suspect
> the chances of Carl applying it as-is are pretty remote. [...]

David,

Thanks for the heads up!

Guess I should've read the patch submission guidelines first...

Peace

-Pieter


[PATCH 1/1] emacs: Use "message-cited-text" instead of "message-cited-text-face"

2011-03-20 Thread Pieter Praet

(describe-face 'message-cited-text-face)
> message-cited-text-face is an alias for the face `message-cited-text'.
> This face is obsolete since 22.1; use `message-cited-text' instead.

Signed-off-by: Pieter Praet 
---
 emacs/notmuch-wash.el |4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/emacs/notmuch-wash.el b/emacs/notmuch-wash.el
index c4a7a41..0e64eb2 100644
--- a/emacs/notmuch-wash.el
+++ b/emacs/notmuch-wash.el
@@ -125,7 +125,7 @@ is what to put on the button."
 (let* ((cite-start (match-beginning 0))
   (cite-end (match-end 0))
   (cite-lines (count-lines cite-start cite-end)))
-  (overlay-put (make-overlay cite-start cite-end) 'face 
'message-cited-text-face)
+  (overlay-put (make-overlay cite-start cite-end) 'face 
'message-cited-text)
   (when (> cite-lines (+ notmuch-wash-citation-lines-prefix
 notmuch-wash-citation-lines-suffix
 1))
@@ -151,7 +151,7 @@ is what to put on the button."
  (sig-end-marker (make-marker)))
  (set-marker sig-start-marker sig-start)
  (set-marker sig-end-marker (point-max))
- (overlay-put (make-overlay sig-start-marker sig-end-marker) 'face 
'message-cited-text-face)
+ (overlay-put (make-overlay sig-start-marker sig-end-marker) 'face 
'message-cited-text)
  (notmuch-wash-region-to-button
   sig-start-marker sig-end-marker
   "signature" "\n"
-- 
1.7.1


RFC: notmuch powered (personal) (end-to-end) e-mail system

2011-03-20 Thread Ciprian Dorin Craciun
Hello all! (Sorry for the long email.)

I'm struggling for some time to get rid of the current
de-facto email solutions (i.e. GMail, Zimbra), and I've passively
observed for some time the notmuch project and community.

Although I've forwarded all my email to a single account, and I'm
currently mirroring my GMail account locally (by using `mbsync`),
index it by using notmuch, and I collect spam mails for later filter
training, unfortunately I'm unable to convert because the current
notmuch-powered solutions have (some of) the following shortcomings (I
don't want to offend anyone, so please take these as observations):
* the most feature full UI is the Emacs one -- thus limited remote
access (I mean from an arbitrary computer with only a web-browser);
(and I'm not a very big fan of Emacs;)
* most are still dependent on external IMAP systems -- this is not
a problem with notmuch itself, but for the integrating clients;
* SPAM -- as above -- is not integrated;
* filtering (tag applying) is not automatic (as in integrated in
notmuch itself or the client), but triggered through external scripts;

As such I'm thinking on implementing a custom end-to-end email
system and I would like to hear your feedback before embarking on such
a task.

I'm targeting the following features:
* (inbound) SMTP integration, thus once an email is received it is
automatically pushed through the system; (I'm primarily targeting
those users that afford to run their own SMTP server; but the solution
could still be adapted for those that only want the other features;)
* automatic spam filtering, and tag applying;
* automatic email triggers based on tags (such as user
notifications, forwarding, etc.)
* remote RPC-like access to the whole system;
* remote Web user interface;

About the overall architecture I'm thinking on adopting the following:
* in general the whole system is decomposed in independent
components (long-lived OS daemons) that each one does a particular job
(see below);
* all the components communicate between each-other through a
message queue system (for example ZeroMQ or RabbitMQ);
* all the communication is JSON based;

The components would be:
* SMTP inbound gateway -- for example I could take qmail or
Postfix and replace the delivery agent with a custom process that
pushes the email into the system; (any other solution suggestions?);
* email store -- as the name suggests it is a simple
key-value-like store that should persist raw email-messages; it should
be as robust as possible, and its contents should be the only thing
needed to reconstruct all the other derived data; (I could use here a
simple process that maintains a maildir, I could go also with a
BerkeleyDB wrapper, or even something more sophisticated;)
* spam filter -- which either classifies the email or trains the
spam filter; (for example I would use bogofilter;)
* email index -- this is where notmuch would come into play; it
would be fed with emails, which it would automatically apply tags and
issue trigger notifications based on tags; it also maintains a set of
filters and tags to automatically apply;
* (maybe) a coordinator that should delegate and monitor requests
to the above components; but if I'm using RabbitMQ and carefully
designing the above components, they could drive each other;
* restful web service that would intermediate access to all the
above components;

For now I have the following uncertainties:
* how should I handle multiple users? I think each user should
have it's own store / notmuch / bogofilter instance (at least in terms
of storage if not even in terms of separate daemon);
* should I keep the emails is a file-system, or a key-value store?
(the file-system is more bug-free, but I'm confident that a BerkeleyDB
instance would be more efficient);
* should I use libnotmuch or for starters just make a notmuch tool wrapper;
* and the most pressing one, transactions: I would like that at no
point does a message get half processed or lost; as such I need
notmuch to behave transactionally -- indexing the message and tagging
it should be atomic and durable; (is there a way with libnotmuch to
control the underlaying BerkeleyDB database?)

Suggestions? Considerations?

Ciprian.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: RFC: notmuch powered (personal) (end-to-end) e-mail system

2011-03-20 Thread Brett Viren
On Sun, Mar 20, 2011 at 10:07 AM, Ciprian Dorin Craciun
ciprian.crac...@gmail.com wrote:

    I'm struggling for some time to get rid of the current
 de-facto email solutions (i.e. GMail, Zimbra), and I've passively
 observed for some time the notmuch project and community.

It sounds like what you want *is* GMail (I don't know Zimbra) but just
that you want it running on your own box instead of on Google's
servers.

    Suggestions? Considerations?

Based on what you wrote, I think BerkeleyDB will be too limiting.
I suggest for you to look into DBMail[1] for the mail store.


-Brett.

[1] http://www.dbmail.org/
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: RFC: notmuch powered (personal) (end-to-end) e-mail system

2011-03-20 Thread Ben Gamari
On Sun, 20 Mar 2011 16:07:50 +0200, Ciprian Dorin Craciun 
ciprian.crac...@gmail.com wrote:
 Hello all! (Sorry for the long email.)
 
 [snip]
 
 * the most feature full UI is the Emacs one -- thus limited remote
 access (I mean from an arbitrary computer with only a web-browser);
 (and I'm not a very big fan of Emacs;)
 
There have been a few attempts to put together an HTML front-end to
notmuch[1]. None have made it very far though. It would be nice to see
this space filled.

 * most are still dependent on external IMAP systems -- this is not
 a problem with notmuch itself, but for the integrating clients;
 
Not entirely sure what you mean by this. You could easily use
e.g. notmuch-deliver as the local delivery agent with a SMTP server and
you'd have no need for IMAP.

 * SPAM -- as above -- is not integrated;
 
Nor should it be. Mail indexing, viewing, composing, and
filtering are all orthogonal parts of a mail system. It takes all of
ten lines to invoke a spam filter in your filter script.

 * filtering (tag applying) is not automatic (as in integrated in
 notmuch itself or the client), but triggered through external scripts;
 
Again, there is no reason why this should be incorporated into your
mail indexer.

 As such I'm thinking on implementing a custom end-to-end email
 system and I would like to hear your feedback before embarking on such
 a task.
 
Notmuch works so well for its audience because it adheres to the UNIX
philosophy of do one thing and do it well. The goal of an integrated
end-to-end mail system might sound nice, but IMHO it's a recipe for a
kludgey, unmaintainable nightmare which is mediocre at performing its
task, on a good day. Perhaps I'm misunderstanding your proposal but it
seems to me like you are taking an easy, already solved problem and
turning it into a difficult one.

 I'm targeting the following features:
 * (inbound) SMTP integration, thus once an email is received it is
 automatically pushed through the system; (I'm primarily targeting
 those users that afford to run their own SMTP server; but the solution
 could still be adapted for those that only want the other features;)
 
Is there something wrong with Postfix with notmuch-deliver as a LDA?

 * automatic spam filtering, and tag applying;

A traditional sorting script with bogofilter/spamassassin?

 * automatic email triggers based on tags (such as user
 notifications, forwarding, etc.)

Again, a sorting script?

 * remote RPC-like access to the whole system;
 
What's wrong with SSH?

 * remote Web user interface;
 
Nothing fills this need currently. Feel free to write up something but
please don't couple it to some all-inclusive beheamoth of a project.

Personally, I would think more carefully about this project before
proceding. It sounds like you intend on reinventing various portions of
the wheel several times. Nothing you have listed is difficult to do with
a few scripts, notmuch, and an SMTP server. 

 About the overall architecture I'm thinking on adopting the following:
 * in general the whole system is decomposed in independent
 components (long-lived OS daemons) that each one does a particular job
 (see below);
 * all the components communicate between each-other through a
 message queue system (for example ZeroMQ or RabbitMQ);
 * all the communication is JSON based;
 
 The components would be:
 * SMTP inbound gateway -- for example I could take qmail or
 Postfix and replace the delivery agent with a custom process that
 pushes the email into the system; (any other solution suggestions?);
 * email store -- as the name suggests it is a simple
 key-value-like store that should persist raw email-messages; it should
 be as robust as possible, and its contents should be the only thing
 needed to reconstruct all the other derived data; (I could use here a
 simple process that maintains a maildir, I could go also with a
 BerkeleyDB wrapper, or even something more sophisticated;)
 * spam filter -- which either classifies the email or trains the
 spam filter; (for example I would use bogofilter;)
 * email index -- this is where notmuch would come into play; it
 would be fed with emails, which it would automatically apply tags and
 issue trigger notifications based on tags; it also maintains a set of
 filters and tags to automatically apply;
 * (maybe) a coordinator that should delegate and monitor requests
 to the above components; but if I'm using RabbitMQ and carefully
 designing the above components, they could drive each other;
 * restful web service that would intermediate access to all the
 above components;
 
 For now I have the following uncertainties:
 * how should I handle multiple users? I think each user should
 have it's own store / notmuch / bogofilter instance (at least in terms
 of storage if not even in terms of separate daemon);
 * should I keep the emails is a file-system, or a key-value store?
 

Re: RFC: notmuch powered (personal) (end-to-end) e-mail system

2011-03-20 Thread Austin Clements
Much of the beauty of notmuch is how few assumptions it makes about
your mail system.  It plays well with others.  For example, one deep
insight of notmuch is that it *doesn't* require a custom mail store,
even though a more obvious design might; in fact, it doesn't even
require Maildir.

That said, I think I can see where you're coming from and I also think
you're targeting some of the deficiencies of notmuch, but I also think
you're overengineering the solution.  As a result of notmuch's
simplicity, a fully working mail setup requires a lot of moving parts
besides notmuch and it can take a while for a new user to set all that
up, especially if they're migrating wholesale from some external mail
setup.

On Sun, Mar 20, 2011 at 10:07 AM, Ciprian Dorin Craciun
ciprian.crac...@gmail.com wrote:
    As such I'm thinking on implementing a custom end-to-end email
 system and I would like to hear your feedback before embarking on such
 a task.

    I'm targeting the following features:
    * (inbound) SMTP integration, thus once an email is received it is
 automatically pushed through the system; (I'm primarily targeting
 those users that afford to run their own SMTP server; but the solution
 could still be adapted for those that only want the other features;)

As others have mentioned, see notmuch-deliver.  I and others have also
suggested inotify support for notmuch before, which would make the
inbound mail mechanism (be it SMTP, IMAP fetching, or whatever)
completely unaware of notmuch, offer some other benefits (for example,
if mail is manipulated outside notmuch via IMAP), and is highly
discoverable for new users (just have notmuch setup ask if they want
notmuch to monitor for new email and then fire up an inotify daemon
the first time notmuch is called).

    * automatic spam filtering, and tag applying;
    * automatic email triggers based on tags (such as user
 notifications, forwarding, etc.)

Obviously the above two can be scripted, but I agree that it's
unsatisfying that every user needs to roll their own delivery script.
While tagging and triggering are highly personal, they're not *so*
personal that everyone needs a completely custom solution.  This
should be more approachable.  I'm not sure what the best answer here
is, but I don't think it requires it requires integration with a
monolithic system to do right.

    * remote RPC-like access to the whole system;

This is another deep insight of notmuch.  It already has an awesome
RPC interface: the CLI.  Perhaps your actual problem is that the only
supported remote transport protocol is SSH.  This comes with a lot of
benefits (authentication, RPC pipelining), but also a lot of baggage
(a full SSH client on the client side).  I've thought about this in
the context of both an HTTP client and an Android client and in both
cases I concluded that a simple HTTPS transport wrapped around the
notmuch CLI would be the way to go.  Just put the CLI arguments in a
POST and send the JSON on stdout back.  This is trivial to prototype
as a Python CGI script, easy to build as a standalone Python server,
and not especially hard to build as a robust C server.

    * remote Web user interface;

A good web UI would be fantastic.  Based on the rest of your email, I
get the impression this was a requirements driver from much of the
above, especially the integrated tagging/triggering and RPC access.
I've already suggested a simple solution to the RPC problem.  For
tagging/triggering, it's probably worth developing a solution that
allows for machine-editable rules (ideally retaining
user-editableness), which would make it possible to integrate filter
management in to a web UI.  This could be as simple as a standard
delivery script that operates from some simple rule database.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch