Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags

[David Bremner]

Jan Pobrislo  writes:
>
> I think that using fixed mapping for flag meaning is a good POC step,
> but that won't work without explicit support from the synchronizer to
> map specific keywords to always same tags. I don't really know what
> offlineimap does to synchronize keywords, if anything. Dsync already
> does what it does - which is obviously to use the full dovecot format.
>

I see, you're talking about this "dovecot-keywords" file I guess

  http://wiki2.dovecot.org/MailboxFormat/Maildir

Some questions that spring to mind:

- This is clearly dovecot specific; I wonder what fraction of
  our users would benefit. I suppose that's a question about any scheme
  involving maildir-flags a-z; at least those can be synchronized ootb
  by several tools.

- Notmuch new currently only indexes one copy of a message, so two files
  in different maildirs (i.e. a list and inbox) would be pretty much a
  crapshoot which tags get applied. We intend to change this behaviour
  eventually, but no one is working on it currently.

- even if/when this behaviour changes, there is still the problem of
  reconciling different tag mappings from several maildirs.

On the other hand, maybe not much change to the notmuch core would be
needed to at least experiment with this, using e.g. hooks to
notmuch-insert and notmuch-new.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags

[David Bremner]

Jan Pobrislo  writes:

>
> Every such maildir extension out there that I know of was invented for
> some specific application. Out of the two documented formats there are,
> the dovecot-keywords file is:
>
> 1) more limited (26 tags maximum)
> 2) simpler to implement, especially wrt. detecting changes
> 3) usable out of the box with some tools, as you noted

Actually, I just mean the extra maildir flags. I guess only the dovecot
sync tool syncs dovecot-keywords. And I don't think "notmuch+dsync
users" is a very big group, fwiw.

>> On the other hand, maybe not much change to the notmuch core would be
>> needed to at least experiment with this, using e.g. hooks to
>> notmuch-insert and notmuch-new.
>
> I was pondering this before and would find it rather neat, but it is
> bit more complicated than it might first seem. The hook needs to be
> able to add arbitrary files to be watched for changes and deduce from
> that which files need their tags re-read.

perhaps lastmod: queries can help here. that's more or less what they
added for.

d

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags

[David Bremner]

David Bremner  writes:

> Jan Pobrislo  writes:
>
>>
>> Every such maildir extension out there that I know of was invented for
>> some specific application. Out of the two documented formats there are,
>> the dovecot-keywords file is:
>>
>> 1) more limited (26 tags maximum)
>> 2) simpler to implement, especially wrt. detecting changes
>> 3) usable out of the box with some tools, as you noted
>
> Actually, I just mean the extra maildir flags. I guess only the dovecot
> sync tool syncs dovecot-keywords. And I don't think "notmuch+dsync
> users" is a very big group, fwiw.

Straight rsync would also transfer those files, of course.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags

[Jan Pobrislo]

On Wed, 03 Feb 2016 08:03:08 -0400
David Bremner  wrote:

> I see, you're talking about this "dovecot-keywords" file I guess
> 
>   http://wiki2.dovecot.org/MailboxFormat/Maildir

Indeed.

> Some questions that spring to mind:
> 
> - This is clearly dovecot specific; I wonder what fraction of
>   our users would benefit. I suppose that's a question about any
> scheme involving maildir-flags a-z; at least those can be
> synchronized ootb by several tools.

Every such maildir extension out there that I know of was invented for
some specific application. Out of the two documented formats there are,
the dovecot-keywords file is:

1) more limited (26 tags maximum)
2) simpler to implement, especially wrt. detecting changes
3) usable out of the box with some tools, as you noted

Of course, one could go invent some another format, but in the end it'd
be application-specific too, and I don't see it succeeding without help
of mail-synchronizer authors.

> - Notmuch new currently only indexes one copy of a message, so two
> files in different maildirs (i.e. a list and inbox) would be pretty
> much a crapshoot which tags get applied. We intend to change this
> behaviour eventually, but no one is working on it currently.
> 
> - even if/when this behaviour changes, there is still the problem of
>   reconciling different tag mappings from several maildirs.

This is nothing new though, current synchronize_tags has the very same
problem. I'm not sure how much of a problem it is in practice, but it
probably should be addressed in some way.

Here possibly the best course of action would be to leave it open-ended
and provide user-definable conflict resolution hook, as I can't really
think of "one size fits all" solution.

> On the other hand, maybe not much change to the notmuch core would be
> needed to at least experiment with this, using e.g. hooks to
> notmuch-insert and notmuch-new.

I was pondering this before and would find it rather neat, but it is
bit more complicated than it might first seem. The hook needs to be
able to add arbitrary files to be watched for changes and deduce from
that which files need their tags re-read.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags

[David Bremner]

Jan Pobrislo  writes:

> On Wed, 03 Feb 2016 11:38:30 -0400
> David Bremner  wrote:
>> > I was pondering this before and would find it rather neat, but it is
>> > bit more complicated than it might first seem. The hook needs to be
>> > able to add arbitrary files to be watched for changes and deduce
>> > from that which files need their tags re-read.  
>> 
>> perhaps lastmod: queries can help here. that's more or less what they
>> added for.
>> 
>> d
>> 
>
> That only helps with applying tags from notmuch to maildir, not
> vice-versa, which is the main thing I was talking about.

For changes to dovecot-keywords files, you're on your own. But changes
to the actual maildir flags are file renames, which should trigger an
increase in lastmod counter; if not I'd consider that a bug.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags

[Jan Pobrislo]

On Wed, 03 Feb 2016 11:38:30 -0400
David Bremner  wrote:
> > I was pondering this before and would find it rather neat, but it is
> > bit more complicated than it might first seem. The hook needs to be
> > able to add arbitrary files to be watched for changes and deduce
> > from that which files need their tags re-read.  
> 
> perhaps lastmod: queries can help here. that's more or less what they
> added for.
> 
> d
> 

That only helps with applying tags from notmuch to maildir, not
vice-versa, which is the main thing I was talking about.

But it got me thinking about potential notmuch<->dovecot-keywords
synchronizer running as a separate pass. Although it would have to keep
it's own database of mtimes, tags, etc. is still pretty feasible to
make without changes to notmuch itself.

---
David: Sorry about the duplicate mail. Having to use stupid MUA ATM.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags

[Jan Pobrislo]

Hello. I've been thinking about this feature for quite some while.
Having tags stored and synchronized together with mail would certainly
make life easier for a lot of notmuch user. And the upside is that you
can already do that with Dovecot's dsync tool.

But from what I've seen in the patches (as far as I understood them) it
doesn't really work anything like keyword storage in Dovecot - aside
from using maildir flags a-z.

I've been missing comprehensive source of information on various things
maildir-related, so I've written this: https://notmuchmail.org/software/
It includes links to specifications of the two keyword storage formats
for maildir: Dovecot's and Courier's. There's a lot more that can be
added on that page, all the MUAs for starters.

I think that using fixed mapping for flag meaning is a good POC step,
but that won't work without explicit support from the synchronizer to
map specific keywords to always same tags. I don't really know what
offlineimap does to synchronize keywords, if anything. Dsync already
does what it does - which is obviously to use the full dovecot format.

Having the mapping in the maildir rather than database can work well
because you have one unambiguous format that both synchronizer and
notmuch can use without much hassle. And it will scale up to 26
different tags per maildir, after that it will unfortunately stop
working without resorting to dovecot-specific formats, but I think it
still covers 98% of use-cases or so.

On the other hand, you can hack up a script that renames mail
post-dsync to conform to expected static mapping. The change detection
with just one auxiliary file is simple too, as opposed to the
subdirectory format Courier uses.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH/RFC 0/3] Maildir custom flags and notmuch tags

[David Bremner]

Igor Almeida  writes:

> Building on Bremner's patch at
> <1411805835-3563-1-git-send-email-da...@tethera.net>, this updates
> 'notmuch new' to tag messages based on lower-case letters in the file
> name.
> I'm testing this with an IMAP server, offlineimap and notmuch,
> and seems to work well enough for my unidirectional use case.
> Eventually I'd like to implement the reverse path, i.e. tagging in
> notmuch and seeing IMAP keywords show up in the server, but that's for
> later.
>
> Comments are very welcome.
>

Sorry about the long silence. I _have_ been thinking about and working
on this. This message summarizes my current thinking/plan.

I'm not very enthusiastic about the about the several places this can
get out of sync

1) config info in .notmuch-config can be out of sync with the "real
config" in the database.  This part is only until the next run of
notmuch new, so it's not so bad.

2) When the interpretation of maildir flags changes, there doesn't seem
to be a sensible way of retagging messages.

Some others have objected to supporting a nonstandard extension used (I
think) only by dovecot. For me this is not so bad, as long as the code
required specifically for this is not too large / intrusive.

So here are the pieces I think we need. 

a) some code in notmuch-new to sync flags a-z to a fixed set of tags
  [1]. This soundspretty much the functionality you have now, although I
  haven't really reviewed that part of the series.

b) some code to update metadata directly from the CLI (notmuch
   config/notmuch tag), probably using something like [2] as a back
   end. This metadata maps the fixed tags like $maildir_flag_k to and
   from some friendly format.

c) The use of xapian field processors to translate e.g. a query
   "tag:my-sensible-thing" into "tag:$maildir_flag_k". Optionally do the
   reverse when returning lists of tags. FieldProcessors are only
   available in Xapian 1.3.x, but apparently 1.3 series is nearing a
   stable release, so it's a sane time to optionally depend on it.
   I guess as a short term hack, this could be done by shell wrappers
   (not something we'd ship, but it might make it usable for you).  

d) code to dump and restore the metadata as part of the normal dump
   files. This is at least in progress [2].

In terms of how these changes relate to other goals

b,d) have strong connects with allowing database level config. Among
 other reasons this will help make bindings users less second class
 citizens
  

  c) has overlap with allowing date fields that are not ranges (among
 other things). At least all the machinery to configure xapian 1.2
 versus 1.3, and the understanding how to use field processors, is
 reusable.

I'd understand perfectly if you prefer to rebase your simpler solution
on top of master, but if you want to keep working on this, then I guess
(a), (b), and (c) are all things you would work on.

Probably (b) needs the library api from [2], which now that I start
typing this epic long message, I could split off from the dump/restore
stuff. In particular managing the config metadata only needs about half
of the LOC; the iterator is only needed for dump, I think.

David

[1]: eventually it might make sense to split the tags that are somehow
automatic from those that represent user choice. We already have this
issue with tag:encrypted and friends.

[2]: id:1452394301-29499-1-git-send-email-da...@tethera.net
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH/RFC 0/3] Maildir custom flags and notmuch tags

[Igor Almeida]

Building on Bremner's patch at
<1411805835-3563-1-git-send-email-da...@tethera.net>, this updates
'notmuch new' to tag messages based on lower-case letters in the file
name.
I'm testing this with an IMAP server, offlineimap and notmuch,
and seems to work well enough for my unidirectional use case.
Eventually I'd like to implement the reverse path, i.e. tagging in
notmuch and seeing IMAP keywords show up in the server, but that's for
later.

Comments are very welcome.

Igor Almeida (3):
  David Bremner's patch for custom maildir flags
  notmuch new: tag messages based on maildir custom flags
  notmuch new: sync maildir custom flag user configuration

 lib/database.cc  | 61 +++
 lib/message.cc   | 88 
 lib/notmuch.h| 18 
 notmuch-client.h |  4 +++
 notmuch-config.c | 24 +++-
 notmuch-new.c| 49 +++
 6 files changed, 243 insertions(+), 1 deletion(-)

-- 
2.5.3

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch