[notmuch] [PATCH 0/4] Make tags applied by 'notmuch new' configurable.

[Carl Worth]

On Thu, 03 Dec 2009 10:48:00 +0100, Marten Veldthuis  
wrote:
> I can see one clear case where that would become a problem: mistaggings.
> If I set up something so that in 99% of the cases, it'd tag new mail
> correctly with say "inbox unread mytag", I could quickly pick the
> mistagged ones out and remove the offending tag. 
> 
> If you do it like above however, the autotag script would come merrily
> along and retag them.

Right. A fully automatic, rule-based tag does not allow for manual
deletion of the exact same tag, (nor even manual addition).

I've given this some thought and there are things we could do here. For
example, the automatic tag could be stored internally under a different
prefix than the manual tag of the same name. That would be enough to
support the manual addition of a tag, such that a user-level search such
as:

tag:foo

would map internally to something like:

(auto-tag:foo OR manual-tag:foo)

Then manual tag removal could similarly be stored as yet anther internal
prefix that would work in a negative sense. So the search for "tag:foo"
would now map to:

((auto-tag:foo OR manual-tag:foo) AND NOT manual-tag-removal:foo) [*]

I'm not sure yet if this idea is simply clever or insanely stupid.

It does seem like we could do this and hide all of the complexity
entirely from the user. But hidden complexity can raise its head in
nasty ways. Such as "I wrote this rule in my configuration file, why
isn't it working?" (in the face of a hidden manual-tag-removal).

Another idea would be to just keep the rule-based tags strictly as
configured, explicitly prevent the user from manually adding/removing
tags of the same name. And then the user could play games like the above
in saved searches to be able to simulate manual touchups of rule-based
tags, (without the automatic vs. manual state being hidden).

Yet another idea is to keep all rule-based tagging out of the
configuration, and let the user come up with whatever scheme they most
prefer. This could still allow for the configuration to hold saved
searches, (which could stand in for rule-based tags in a lot of cases
and avoid a lot of the complexity discussed here).

So those are some of my current thoughts on the issue. And that's why I
haven't put any auto tagging into the configuration file yet.

-Carl

[*] Or should that be:

(manual-tag:foo OR (auto-tag:foo AND NOT manual-tag-removal:foo))

It probably wouldn't matter in practice as presumably the addition of
the manual-tag would remove any manual-tag-removal and vice-versa.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 

[notmuch] [PATCH 0/4] Make tags applied by 'notmuch new' configurable.

[Marten Veldthuis]

Excerpts from Carl Worth's message of Wed Dec 02 22:36:13 +0100 2009:
> As a side-note, I would recommend against making your auto-tagging
> scripts process only new messages. You can get a much more reliable
> setup by having your auto-tagging scripts apply to the global
> database. And this is not inefficient at all if you simply ensure that
> you only match messages which need the tag added.

I can see one clear case where that would become a problem: mistaggings.
If I set up something so that in 99% of the cases, it'd tag new mail
correctly with say "inbox unread mytag", I could quickly pick the
mistagged ones out and remove the offending tag. 

If you do it like above however, the autotag script would come merrily
along and retag them.
-- 
- Marten

[notmuch] [PATCH 0/4] Make tags applied by 'notmuch new' configurable.

[Carl Worth]

On Tue, 24 Nov 2009 23:10:26 +0100, Jan Janak  wrote:
> I would like to propose that we make the list of tags applied by 'notmuch new'
> configurable. Right now notmuch applies two tags to all new messages added to
> the database, 'inbox' and 'unread'. The two tags are added by the C code in
> notmuch-new.c and they cannot be changed without editing the source file and
> recompiling notmuch.

Hi Jan,

Sorry to leave this thread sitting without review for so long. My travel
schedule, combined with a big US holiday last week combined to create a
fair amount of backlog.

> This change was motivated by my desire to remove both tags from newly added
> messages. My rules for adding these two tags are more complex and I do it in
> a script run after 'notmuch new'. Instead of 'inbox' and 'unread', I configure
> 'notmuch new' to add a new tag called 'new' (and only that one). This tag
> marks newly added messages that haven't been properly tagged yet by my 
> auto-tagging scripts. The last script I run after 'notmuch new' removes that
> tag. My auto-tagging scripts process only messages with the tag 'new'.

As a side-note, I would recommend against making your auto-tagging
scripts process only new messages. You can get a much more reliable
setup by having your auto-tagging scripts apply to the global
database. And this is not inefficient at all if you simply ensure that
you only match messages which need the tag added.

For example, in your current setup you presumably have something like:

notmuch tag +foo  and tag:new

If you change that to:

notmuch tag +foo  and not tag:foo

then you have the advantage of being able to change the search terms you
are using for the "foo" tag and know that this tag will be applied
globally, (and not only to messages incorporated since the rule
change). Also, you should find that this performs just fine.

You could also have a second specification that would support removing
the "foo" tag if the terms ever changed to become more restrictive:

notmuch tag -foo tag:foo and not ( On a side note; It may seem logical to add/omit the tag 'unread' directly in 
> 'notmuch new' based on the Maildir flags extracted from the filename of the
> message. I suggest that we don't do that in 'notmuch new'. A better way would
> be writing a small script or command that can be run *after* 'notmuch new'.
> We could then distribute it with notmuch (maybe with other small tagging
> scripts for common situations). 

I do appreciate the sentiment here, (provide good support for
scriptability and let the user use that for extended functionality). But
the question is, how far do we take that idea? For example, we could
simply hard-code a single "new" tag and say that anyone that wants
"inbox" and "unread" on new messages should do:

notmuch tag +inbox +unread -new tag:new

Would that make notmuch a more elegant system, or would it just be
harder to use than just allowing the inbox and unread tags in the
configuration file?

> I think Maildir flags should be processed after 'notmuch new' is because if
> there are multiple copies of a message with different flags, we may need to
> see all flags from all filenames to set corresponding tags properly and we may
> also need to take the directory part into consideration (i.e. the new mail is
> in 'new', not 'cur').

I think we're going to have to solve all of these problems anyway. We've
already got outstanding patches for dealing with message renames. And
the message rename case is identical to the case of duplicate messages
with different tags, (differing only in whether both filenames are still
present in the mail store).

> There is one catch for users who already have a configuration file and start
> using the patches. They will need to add the new section and the tags option
> manually if they want to preserve current behavior of applying 'inbox' and
> 'unread' automatically by 'notmuch new'.

Why not support backwards compatibility by simply treating an empty
configuration value as "inbox;unread"? We could even make the
notmuch-config code notice a case like this and write out the new
configuration file with the new option 

[notmuch] [PATCH 0/4] Make tags applied by 'notmuch new' configurable.

[Jan Janak]

Hello,

On 24-11 23:10, Jan Janak wrote:
> I would like to propose that we make the list of tags applied by 'notmuch new'
> configurable. Right now notmuch applies two tags to all new messages added to
> the database, 'inbox' and 'unread'. The two tags are added by the C code in
> notmuch-new.c and they cannot be changed without editing the source file and
> recompiling notmuch.
> 
> The four patches that follow this email allow for configuring the tags to be
> added by 'notmuch new' either in the configuration file or on the command
> line.
> 
> This change was motivated by my desire to remove both tags from newly added
> messages. My rules for adding these two tags are more complex and I do it in
> a script run after 'notmuch new'. Instead of 'inbox' and 'unread', I configure
> 'notmuch new' to add a new tag called 'new' (and only that one). This tag
> marks newly added messages that haven't been properly tagged yet by my 
> auto-tagging scripts. The last script I run after 'notmuch new' removes that
> tag. My auto-tagging scripts process only messages with the tag 'new'.
> 
> On a side note; It may seem logical to add/omit the tag 'unread' directly in 
> 'notmuch new' based on the Maildir flags extracted from the filename of the
> message. I suggest that we don't do that in 'notmuch new'. A better way would
> be writing a small script or command that can be run *after* 'notmuch new'.
> We could then distribute it with notmuch (maybe with other small tagging
> scripts for common situations). 
> 
> I think Maildir flags should be processed after 'notmuch new' is because if
> there are multiple copies of a message with different flags, we may need to
> see all flags from all filenames to set corresponding tags properly and we may
> also need to take the directory part into consideration (i.e. the new mail is
> in 'new', not 'cur').
> 
> The list of tags to be applied by notmuch can be configured in the
> configuration file. There is a new section [new] which contains configuration
> options for 'notmuch new'. There is only one option called 'tags'. The option
> contains a semicolon separated list of tags:
> 
>   [new]
>   tags=inbox;unread  # Emulate the original behavior
> 
> One of the patches updates 'notmuch setup' to create the section and add
> the tags option with tags 'inbox' and 'unread', but only if a new
> configuration file is being created. If the configuration file already exists
> then it just copies the contents from the old configuration file to the new
> one.
> 
> We do not ask the user for the list of tags in the interactive part, that 
> would
> have been too much. Users can edit the configuration file manually if they 
> want
> to change the list of tags. If they create a new configuration file then they
> probably want to accept the default anyway.
> 
> There is one catch for users who already have a configuration file and start
> using the patches. They will need to add the new section and the tags option
> manually if they want to preserve current behavior of applying 'inbox' and
> 'unread' automatically by 'notmuch new'.
> 
> The last patch in the set adds a new command line option to 'notmuch new'.
> The name of the option is --tag and it can be used to override any tags
> configured in the configuration file. For example:
> 
>   notmuch new --tag=outbox --tag=read
> 
> adds the tags 'outbox' and 'read' and ignores any tags from the configuration
> file.
> 
> Comments and opinions are welcome!

I updated this patch series, adding features and bug fixes suggested by others
on the list.

The command line option --tag now *adds* tags, so the tags configured in the
configuration file are still applied...unless you use the new option
--no-config-tag. With this option only the tags specified on the command line
will be applied by 'notmuch-new'.

  -- Jan

[notmuch] [PATCH 0/4] Make tags applied by 'notmuch new' configurable.

[Jan Janak]

I would like to propose that we make the list of tags applied by 'notmuch new'
configurable. Right now notmuch applies two tags to all new messages added to
the database, 'inbox' and 'unread'. The two tags are added by the C code in
notmuch-new.c and they cannot be changed without editing the source file and
recompiling notmuch.

The four patches that follow this email allow for configuring the tags to be
added by 'notmuch new' either in the configuration file or on the command
line.

This change was motivated by my desire to remove both tags from newly added
messages. My rules for adding these two tags are more complex and I do it in
a script run after 'notmuch new'. Instead of 'inbox' and 'unread', I configure
'notmuch new' to add a new tag called 'new' (and only that one). This tag
marks newly added messages that haven't been properly tagged yet by my 
auto-tagging scripts. The last script I run after 'notmuch new' removes that
tag. My auto-tagging scripts process only messages with the tag 'new'.

On a side note; It may seem logical to add/omit the tag 'unread' directly in 
'notmuch new' based on the Maildir flags extracted from the filename of the
message. I suggest that we don't do that in 'notmuch new'. A better way would
be writing a small script or command that can be run *after* 'notmuch new'.
We could then distribute it with notmuch (maybe with other small tagging
scripts for common situations). 

I think Maildir flags should be processed after 'notmuch new' is because if
there are multiple copies of a message with different flags, we may need to
see all flags from all filenames to set corresponding tags properly and we may
also need to take the directory part into consideration (i.e. the new mail is
in 'new', not 'cur').

The list of tags to be applied by notmuch can be configured in the
configuration file. There is a new section [new] which contains configuration
options for 'notmuch new'. There is only one option called 'tags'. The option
contains a semicolon separated list of tags:

  [new]
  tags=inbox;unread  # Emulate the original behavior

One of the patches updates 'notmuch setup' to create the section and add
the tags option with tags 'inbox' and 'unread', but only if a new
configuration file is being created. If the configuration file already exists
then it just copies the contents from the old configuration file to the new
one.

We do not ask the user for the list of tags in the interactive part, that would
have been too much. Users can edit the configuration file manually if they want
to change the list of tags. If they create a new configuration file then they
probably want to accept the default anyway.

There is one catch for users who already have a configuration file and start
using the patches. They will need to add the new section and the tags option
manually if they want to preserve current behavior of applying 'inbox' and
'unread' automatically by 'notmuch new'.

The last patch in the set adds a new command line option to 'notmuch new'.
The name of the option is --tag and it can be used to override any tags
configured in the configuration file. For example:

  notmuch new --tag=outbox --tag=read

adds the tags 'outbox' and 'read' and ignores any tags from the configuration
file.

Comments and opinions are welcome!

   -- Jan

[notmuch] [PATCH 0/4] Make tags applied by 'notmuch new' configurable.

[Bart Trojanowski]

* Jan Janak  [091124 17:11]:
> I would like to propose that we make the list of tags applied by 'notmuch new'
> configurable. Right now notmuch applies two tags to all new messages added to
> the database, 'inbox' and 'unread'. The two tags are added by the C code in
> notmuch-new.c and they cannot be changed without editing the source file and
> recompiling notmuch.

I like it.

-Bart

-- 
WebSig: http://www.jukie.net/~bart/sig/

[notmuch] [PATCH 0/4] Make tags applied by 'notmuch new' configurable.

[Bdale Garbee]

On Tue, 2009-11-24 at 23:10 +0100, Jan Janak wrote:
> Instead of 'inbox' and 'unread', I configure
> 'notmuch new' to add a new tag called 'new' (and only that one). This tag
> marks newly added messages that haven't been properly tagged yet by my 
> auto-tagging scripts. 

Oh, brilliant!  I like it.

Bdale

[notmuch] [PATCH 0/4] Make tags applied by 'notmuch new' configurable.

[Brett Viren]

On Tue, Nov 24, 2009 at 5:10 PM, Jan Janak  wrote:
> I would like to propose that we make the list of tags applied by 'notmuch new'
> configurable. Right now notmuch applies two tags to all new messages added to
> the database, 'inbox' and 'unread'. The two tags are added by the C code in
> notmuch-new.c and they cannot be changed without editing the source file and
> recompiling notmuch.

Personally I would like this.  I can't find myself leaving GNUS and so
am using notmuch as "just" a search engine.  Now I manually remove
"inbox" and "unread" after each "new" since they just clutter up an
inbox I never read.

Cheers,
-Brett.