Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags

2016-02-03 Thread Jan Pobrislo
On Wed, 03 Feb 2016 08:03:08 -0400
David Bremner  wrote:

> I see, you're talking about this "dovecot-keywords" file I guess
> 
>   http://wiki2.dovecot.org/MailboxFormat/Maildir

Indeed.

> Some questions that spring to mind:
> 
> - This is clearly dovecot specific; I wonder what fraction of
>   our users would benefit. I suppose that's a question about any
> scheme involving maildir-flags a-z; at least those can be
> synchronized ootb by several tools.

Every such maildir extension out there that I know of was invented for
some specific application. Out of the two documented formats there are,
the dovecot-keywords file is:

1) more limited (26 tags maximum)
2) simpler to implement, especially wrt. detecting changes
3) usable out of the box with some tools, as you noted

Of course, one could go invent some another format, but in the end it'd
be application-specific too, and I don't see it succeeding without help
of mail-synchronizer authors.

> - Notmuch new currently only indexes one copy of a message, so two
> files in different maildirs (i.e. a list and inbox) would be pretty
> much a crapshoot which tags get applied. We intend to change this
> behaviour eventually, but no one is working on it currently.
> 
> - even if/when this behaviour changes, there is still the problem of
>   reconciling different tag mappings from several maildirs.

This is nothing new though, current synchronize_tags has the very same
problem. I'm not sure how much of a problem it is in practice, but it
probably should be addressed in some way.

Here possibly the best course of action would be to leave it open-ended
and provide user-definable conflict resolution hook, as I can't really
think of "one size fits all" solution.

> On the other hand, maybe not much change to the notmuch core would be
> needed to at least experiment with this, using e.g. hooks to
> notmuch-insert and notmuch-new.

I was pondering this before and would find it rather neat, but it is
bit more complicated than it might first seem. The hook needs to be
able to add arbitrary files to be watched for changes and deduce from
that which files need their tags re-read.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags

2016-02-03 Thread Jan Pobrislo
On Wed, 03 Feb 2016 11:38:30 -0400
David Bremner  wrote:
> > I was pondering this before and would find it rather neat, but it is
> > bit more complicated than it might first seem. The hook needs to be
> > able to add arbitrary files to be watched for changes and deduce
> > from that which files need their tags re-read.  
> 
> perhaps lastmod: queries can help here. that's more or less what they
> added for.
> 
> d
> 

That only helps with applying tags from notmuch to maildir, not
vice-versa, which is the main thing I was talking about.

But it got me thinking about potential notmuch<->dovecot-keywords
synchronizer running as a separate pass. Although it would have to keep
it's own database of mtimes, tags, etc. is still pretty feasible to
make without changes to notmuch itself.

---
David: Sorry about the duplicate mail. Having to use stupid MUA ATM.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags

2016-02-02 Thread Jan Pobrislo
Hello. I've been thinking about this feature for quite some while.
Having tags stored and synchronized together with mail would certainly
make life easier for a lot of notmuch user. And the upside is that you
can already do that with Dovecot's dsync tool.

But from what I've seen in the patches (as far as I understood them) it
doesn't really work anything like keyword storage in Dovecot - aside
from using maildir flags a-z.

I've been missing comprehensive source of information on various things
maildir-related, so I've written this: https://notmuchmail.org/software/
It includes links to specifications of the two keyword storage formats
for maildir: Dovecot's and Courier's. There's a lot more that can be
added on that page, all the MUAs for starters.

I think that using fixed mapping for flag meaning is a good POC step,
but that won't work without explicit support from the synchronizer to
map specific keywords to always same tags. I don't really know what
offlineimap does to synchronize keywords, if anything. Dsync already
does what it does - which is obviously to use the full dovecot format.

Having the mapping in the maildir rather than database can work well
because you have one unambiguous format that both synchronizer and
notmuch can use without much hassle. And it will scale up to 26
different tags per maildir, after that it will unfortunately stop
working without resorting to dovecot-specific formats, but I think it
still covers 98% of use-cases or so.

On the other hand, you can hack up a script that renames mail
post-dsync to conform to expected static mapping. The change detection
with just one auxiliary file is simple too, as opposed to the
subdirectory format Courier uses.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: Notmuch scripts (again), now with more usenet

2012-11-01 Thread Jan Pobrislo
On Thu, 18 Oct 2012 23:33:22 -0400, Ethan Glasser-Camp 
ethan.glasser.c...@gmail.com wrote:
 c...@webprojekty.cz writes:

  Hello, for quite some time my set of scripts just lied in my repo and
  waited for polish before release. So tonight I finally managed to update
  the docs, remove old stuff, rewrite some unfortunate things etc.
 
  One notable addition is slrn2maildir script which can convert NNTP
  spool, eg. gmane mailing lists or blogs, as fetched by slrnpull to
  maildir format. This way you can follow plethora of mailing lists
  without subscribing, any blog that publishes full atom/rss feed or
  usenet newsgroup.
 
  For details see the readme:
http://webprojekty.cz/ccx/loggerhead/zmuch/view/head:/README
  or check out the code:
bzr branch http://webprojekty.cz/ccx/bzr/zmuch
 
  I hope it's now in the form acceptable for inclusion to contrib.

 Hi! Sorry about the delay, but I'm going through the patch queue now and
 it seems like this branch is just completely gone. I get 502 Bad Gateway
 errors when I follow the first link. Did it move or is there a problem
 with your site?

 Ethan

Hi! I was having some hardware issues and had to migrate the site. It should
be all up again and ready for inclusion.

Added stuff from last time:
* source function  actions for zaw (https://github.com/zsh-users/zaw)
* LICENSE (CC0)


pgp1RSpuhdDIg.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Distributed Notmuch

2012-01-10 Thread Jan Pobrislo
Quoting Ethan Glasser-Camp (2012-01-08 11:23:59)
>Hi guys,
>
> ...
>
>In brainstorming about the One True Mail Setup, my friend suggested to 
>me that Maildir/IMAP are not really the best choices for mail storage. 

In my opinion Maildirs are very good mail storage format, the issue is
just that IMAP can't transfer them in their entirety and simplicity.

>Among other flaws: to synchronize mail via IMAP you have to check the 
>headers of each message, which means a lot of bandwidth;

There are UIDs in IMAP, see:
http://tools.ietf.org/html/rfc3501#section-2.3.1.1
But I do agree IMAP is indeed not a very good protocol.

>compress Maildir, meaning lots of wasted space;

There are several approaches to compressing the filesystem that can be
used with maildirs, but this could easilly become bottleneck for most
setups.

>My friend suggested that instead it might be better to dump mail into
>some kind of database, for example CouchDB, and synchronize it that way. 

Some time ago I pondered putting emails into MongoDB so the client does
not have to deal with parsing MIME, but this does not give you any big
advantage for synchronization. Rather, you'll be running into the
consistency/availability/partition-tolerance issue. You'll have to
choose if you want to support offline write operations and if so, how
will you handle conflicts that will appear. DVCSes are built to make
this as easy as possible, databases usually not. I cannot comment on
CouchDB and it's MVCC, but I still doubt it would be as practical as
true DVCS.

By the way I highly reccomend this blogpost series:
http://blog.mongodb.org/post/475279604/on-distributed-consistency-part-1

> ...
>
>So my question for the wizards on this list is what their idea of the 
>One True Mail Setup would be in a perfect, or slightly better, world, 
>and what needs to be done to get there. I know some people use one 
>notmuch install that they access remotely. For myself, I'm on a pretty 
>limited Internet connection, so low bandwidth/offline access are big for 
>me, and despite Nicolas Sebrecht and Sebastian Spaeth's heroic work on 
>OfflineIMAP, it still uses a lot of bandwidth to sync. And obviously the 
>whole point of this exercise is tag synchronization..

I tend to go offline too with my laptop, so I can see what are you
talking about. For me the Ideal Mail Setup would be:

* access via ssh to limited/pseudoshell account
  - ssh handles autentication far better than sasl-based apps
  - ssh is designed to allow multiple operations in parallel including
large uploads/downloads that can be resumed
* maildir is accessible via sftp (sshfs) and ssh+rsync
* there is notmuch launchable from the restricted shell, every new mail
  is indexed
* there is database of messages, tags and filenames, kept under DVCS.
  With aid of this database full three-way merges may be performed.
* once client is connected, he should have a way to listen for change
  messages that the server will push

This would allow for convenient operation both in online (storage-free)
and offline (replicated) mode.

I think this is actually pretty implementable. I'd use twisted.conch for
ssh server (launchpad.net uses this), which can be easilly tied in with
dovecot's autentication daemon. Change detection can be done via
inotify/lsyncd. The versioning/merging tool can possibly be based off
current nmbug (I haven't examined it yet). But I'm pretty sure I won't
have time for project of such scale in near future.




Re: Distributed Notmuch

2012-01-10 Thread Jan Pobrislo
Quoting Ethan Glasser-Camp (2012-01-08 11:23:59)
Hi guys,

 ...

In brainstorming about the One True Mail Setup, my friend suggested to 
me that Maildir/IMAP are not really the best choices for mail storage. 

In my opinion Maildirs are very good mail storage format, the issue is
just that IMAP can't transfer them in their entirety and simplicity.

Among other flaws: to synchronize mail via IMAP you have to check the 
headers of each message, which means a lot of bandwidth;

There are UIDs in IMAP, see:
http://tools.ietf.org/html/rfc3501#section-2.3.1.1
But I do agree IMAP is indeed not a very good protocol.

compress Maildir, meaning lots of wasted space;

There are several approaches to compressing the filesystem that can be
used with maildirs, but this could easilly become bottleneck for most
setups.

My friend suggested that instead it might be better to dump mail into
some kind of database, for example CouchDB, and synchronize it that way. 

Some time ago I pondered putting emails into MongoDB so the client does
not have to deal with parsing MIME, but this does not give you any big
advantage for synchronization. Rather, you'll be running into the
consistency/availability/partition-tolerance issue. You'll have to
choose if you want to support offline write operations and if so, how
will you handle conflicts that will appear. DVCSes are built to make
this as easy as possible, databases usually not. I cannot comment on
CouchDB and it's MVCC, but I still doubt it would be as practical as
true DVCS.

By the way I highly reccomend this blogpost series:
http://blog.mongodb.org/post/475279604/on-distributed-consistency-part-1

 ...

So my question for the wizards on this list is what their idea of the 
One True Mail Setup would be in a perfect, or slightly better, world, 
and what needs to be done to get there. I know some people use one 
notmuch install that they access remotely. For myself, I'm on a pretty 
limited Internet connection, so low bandwidth/offline access are big for 
me, and despite Nicolas Sebrecht and Sebastian Spaeth's heroic work on 
OfflineIMAP, it still uses a lot of bandwidth to sync. And obviously the 
whole point of this exercise is tag synchronization..

I tend to go offline too with my laptop, so I can see what are you
talking about. For me the Ideal Mail Setup would be:

* access via ssh to limited/pseudoshell account
  - ssh handles autentication far better than sasl-based apps
  - ssh is designed to allow multiple operations in parallel including
large uploads/downloads that can be resumed
* maildir is accessible via sftp (sshfs) and ssh+rsync
* there is notmuch launchable from the restricted shell, every new mail
  is indexed
* there is database of messages, tags and filenames, kept under DVCS.
  With aid of this database full three-way merges may be performed.
* once client is connected, he should have a way to listen for change
  messages that the server will push

This would allow for convenient operation both in online (storage-free)
and offline (replicated) mode.

I think this is actually pretty implementable. I'd use twisted.conch for
ssh server (launchpad.net uses this), which can be easilly tied in with
dovecot's autentication daemon. Change detection can be done via
inotify/lsyncd. The versioning/merging tool can possibly be based off
current nmbug (I haven't examined it yet). But I'm pretty sure I won't
have time for project of such scale in near future.

/braindump
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[ANNOUNCE] mutt with notmuch support

2012-01-04 Thread Jan Pobrislo
On Tue, 3 Jan 2012 13:39:38 +0100
Karel Zak  wrote:

> 
> This is not another curses front-end for notmuch, this is mutt :-)
> 
> I have forked mutt to seriously integrate notmuch to this excellent
> e-mail client. I don't want to use symlinks or any other hacks to
> emulate virtual folders. My wish is mutt linked with libnotmuch.

Good to hear!

First, I'll shamelessly plug in my set of scripts that do the dirty work by
using symlinked maildirs to interact with any maildir capable mail reader.
It's mainly intended to be interface from shell, calling out to mutt or any
mail reader only when necessary, so it's somewhat reverse paradigm to yours.

view @ http://webprojekty.cz/ccx/loggerhead/zmuch/files
bzr branch http://webprojekty.cz/ccx/bzr/zmuch

Sadly I had to take hiatus mid-rewrite of it, but several interesting points
came up when I was talking about it with cworth and others.

We agreed that we wanted to standardize several features currently present in
emacs UI, so we can share configuration among several implementations. Most
important of these were:

saved searches
--

This is the feature I started writing my scripts for. I wanted to refer by a
simple shorthand to complex queries. I know emacs UI has some notion of
remembered queries, but I haven't really bothered trying it out. I assume you
will want to represent these as separate mailboxes, maybe shown using the
sidebar patch, so one has quick overview what's new in which ML.

The way I do this is that command:

$ zmuch search :foss and not :notmuch

will expand to:

$ notmuch search ( to:lists.xmms.se or ( to:cairo-announce at cairographics.org
or to:cairo at cairographics.org ) or ( to:notmuch at notmuchmail.org or
to:notmuch-request at notmuchmail.org ) ) and not (
to:notmuch at notmuchmail.org or to:notmuch-request at notmuchmail.org )

Which would be bit too much to type by hand, even for so few lists.
This would be using .notmuch-config with something like this:

[zmuch_searches]
xmms2=to:lists.xmms.se
cairo=to:cairo-announce at cairographics.org or to:cairo at cairographics.org
foss=:xmms2 or :cairo or :notmuch

Subset of this issue is the question: How to display overview of such saved
queries in a sensible manner? Mutt probably can't go beyond unread/total
number of messages for each query and that's actually what I have currently
implemented. Given the config:

[interesting]
query=is:unread and not ( is:spam or is:mute )

[zmuch_show]
selected=twisted;notmuch;gentoo-cs;cajovna;system;inbox

it would show number of interesting/total messages for each of the listed
saved query. Much nicer generalisation was then discussed on IRC, where you
could use regular saved search instead of separate [interesting] entry,
allowing you to have several such queries, eg. for displaying
unread/flagged/muted/total or anything your heart desires.

way to call notmuch
---

This might be feature you'll be probably missing due to using libnotmuch
directly instead of calling out executable as emacs UI does, but I find it
really useful to be able to layer functionality unix-style. Not only you can
run remote notmuch via ssh, but also my saved search implementation does this
by expanding arguments passed to the notmuch executable and as such is usable
from any UI that has configurable executable to call instead of "notmuch".

colors
--

Again dunno how much this applies to mutt, as it has it's own coloring
paradigm, but I find it neat to be able to do something like this globally:

[tag_colors]
flagged=%F{red}
mute=%B%F{black}
spam=%F{magenta}
unread=%F{cyan}
replied=%F{yellow}
sent=%F{green}
attachment=%B

This is syntax that zsh uses for it's print/prompt formatting and is obviously
ANSI VT specific (%B is for bold/standout). GUI lovers will probably want #RGB
scheme too.


HTH bring up some good ideas and discussion.


Re: [ANNOUNCE] mutt with notmuch support

2012-01-03 Thread Jan Pobrislo
On Tue, 3 Jan 2012 13:39:38 +0100
Karel Zak k...@redhat.com wrote:

 
 This is not another curses front-end for notmuch, this is mutt :-)
 
 I have forked mutt to seriously integrate notmuch to this excellent
 e-mail client. I don't want to use symlinks or any other hacks to
 emulate virtual folders. My wish is mutt linked with libnotmuch.

Good to hear!

First, I'll shamelessly plug in my set of scripts that do the dirty work by
using symlinked maildirs to interact with any maildir capable mail reader.
It's mainly intended to be interface from shell, calling out to mutt or any
mail reader only when necessary, so it's somewhat reverse paradigm to yours.

view @ http://webprojekty.cz/ccx/loggerhead/zmuch/files
bzr branch http://webprojekty.cz/ccx/bzr/zmuch

Sadly I had to take hiatus mid-rewrite of it, but several interesting points
came up when I was talking about it with cworth and others.

We agreed that we wanted to standardize several features currently present in
emacs UI, so we can share configuration among several implementations. Most
important of these were:

saved searches
--

This is the feature I started writing my scripts for. I wanted to refer by a
simple shorthand to complex queries. I know emacs UI has some notion of
remembered queries, but I haven't really bothered trying it out. I assume you
will want to represent these as separate mailboxes, maybe shown using the
sidebar patch, so one has quick overview what's new in which ML.

The way I do this is that command:

$ zmuch search :foss and not :notmuch

will expand to:

$ notmuch search ( to:lists.xmms.se or ( to:cairo-annou...@cairographics.org
or to:ca...@cairographics.org ) or ( to:notmuch@notmuchmail.org or
to:notmuch-requ...@notmuchmail.org ) ) and not (
to:notmuch@notmuchmail.org or to:notmuch-requ...@notmuchmail.org )

Which would be bit too much to type by hand, even for so few lists.
This would be using .notmuch-config with something like this:

[zmuch_searches]
xmms2=to:lists.xmms.se
cairo=to:cairo-annou...@cairographics.org or to:ca...@cairographics.org
foss=:xmms2 or :cairo or :notmuch

Subset of this issue is the question: How to display overview of such saved
queries in a sensible manner? Mutt probably can't go beyond unread/total
number of messages for each query and that's actually what I have currently
implemented. Given the config:

[interesting]
query=is:unread and not ( is:spam or is:mute )

[zmuch_show]
selected=twisted;notmuch;gentoo-cs;cajovna;system;inbox

it would show number of interesting/total messages for each of the listed
saved query. Much nicer generalisation was then discussed on IRC, where you
could use regular saved search instead of separate [interesting] entry,
allowing you to have several such queries, eg. for displaying
unread/flagged/muted/total or anything your heart desires.

way to call notmuch
---

This might be feature you'll be probably missing due to using libnotmuch
directly instead of calling out executable as emacs UI does, but I find it
really useful to be able to layer functionality unix-style. Not only you can
run remote notmuch via ssh, but also my saved search implementation does this
by expanding arguments passed to the notmuch executable and as such is usable
from any UI that has configurable executable to call instead of notmuch.

colors
--

Again dunno how much this applies to mutt, as it has it's own coloring
paradigm, but I find it neat to be able to do something like this globally:

[tag_colors]
flagged=%F{red}
mute=%B%F{black}
spam=%F{magenta}
unread=%F{cyan}
replied=%F{yellow}
sent=%F{green}
attachment=%B

This is syntax that zsh uses for it's print/prompt formatting and is obviously
ANSI VT specific (%B is for bold/standout). GUI lovers will probably want #RGB
scheme too.


HTH bring up some good ideas and discussion.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch