date:20141005

[PATCH v2 4/4] cli: Add configurable address deduplication for --output=addresses

2014-10-05 Thread Michal Sojka

The code here is an extended version of a patch from Jani Nikula.
---
 completion/notmuch-completion.bash |   6 ++-
 completion/notmuch-completion.zsh  |   3 +-
 doc/man1/notmuch-search.rst|  32 
 notmuch-search.c   | 101 ++---
 test/T090-search-output.sh |   6 +--
 test/T095-search-unique.sh |  63 +++
 6 files changed, 200 insertions(+), 11 deletions(-)
 create mode 100755 test/T095-search-unique.sh

diff --git a/completion/notmuch-completion.bash 
b/completion/notmuch-completion.bash
index c37ddf5..8bc7874 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -305,12 +305,16 @@ _notmuch_search()
COMPREPLY=( $( compgen -W "true false flag all" -- "${cur}" ) )
return
;;
+   --unique)
+   COMPREPLY=( $( compgen -W "none addr addrfold name" -- "${cur}" ) )
+   return
+   ;;
 esac

 ! $split &&
 case "${cur}" in
-*)
-   local options="--format= --output= --sort= --offset= --limit= 
--exclude= --duplicate="
+   local options="--format= --output= --sort= --offset= --limit= 
--exclude= --duplicate= --unique="
compopt -o nospace
COMPREPLY=( $(compgen -W "$options" -- ${cur}) )
;;
diff --git a/completion/notmuch-completion.zsh 
b/completion/notmuch-completion.zsh
index bff8fd5..cf4968c 100644
--- a/completion/notmuch-completion.zsh
+++ b/completion/notmuch-completion.zsh
@@ -53,7 +53,8 @@ _notmuch_search()
 '--max-threads=[display only the first x threads from the search 
results]:number of threads to show: ' \
 '--first=[omit the first x threads from the search results]:number of 
threads to omit: ' \
 '--sort=[sort results]:sorting:((newest-first\:"reverse chronological 
order" oldest-first\:"chronological order"))' \
-'--output=[select what to output]:output:((summary threads messages files 
tags sender recipients addresses))'
+'--output=[select what to output]:output:((summary threads messages files 
tags sender recipients addresses))' \
+'--unique=[address deduplication]:unique:((none\:"no deduplication" 
addr\:"deduplicate by address" addrfold\:"deduplicate by case-insensitive 
address" name\:"deduplicate by name"))'
 }

 _notmuch()
diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
index 3447820..9a9d9c3 100644
--- a/doc/man1/notmuch-search.rst
+++ b/doc/man1/notmuch-search.rst
@@ -85,6 +85,9 @@ Supported options for **search** include
 (--format=text0), as a JSON array (--format=json), or as
 an S-Expression list (--format=sexp).

+Handling of duplicate addresses and/or names can be
+controlled with the --unique option.
+
Note: Searching for **sender** should be much faster than
searching for **recipients** or **addresses**, because
sender addresses are cached directly in the database
@@ -151,6 +154,35 @@ Supported options for **search** include
 prefix. The prefix matches messages based on filenames. This
 option filters filenames of the matching messages.

+``--unique=``\ (**none**\ \|\ **addr**\ \|\ **addrfold**\ \|\ **name**)
+
+Can be used with ``--output=addresses``, ``--output=sender``
+or ``--output=recipients`` to control the address
+deduplication algorithm.
+
+   **none** means that no deduplication is performed. The same
+   address can appear multiple times in the output.
+
+   **addr** means that case-sensitive deduplication is performed
+   on the address part. For example, given the addresses "John
+   Doe " and "Dr. John Doe ",
+   only one will be printed.
+
+   **addrfold** is the same as **addr** but with case folding
+   applied. For example, given the addresses "John Doe
+   " and "John Doe ", only
+   one will be printed. This is the default.
+
+   **name** means that case-sensitive deduplication is performed
+   on the name part. For example, given the addresses "John Doe
+   " and "John Doe ", only one
+   will be printed.
+
+   This option can be given multiple times to output unique
+   combinations of names and addresses. For example,
+   ``--unique=name --unique=addr`` will print unique
+   case-sensitive combinations of name and address.
+
 EXIT STATUS
 ===

diff --git a/notmuch-search.c b/notmuch-search.c
index 0614f10..94d400e 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -33,6 +33,15 @@ typedef enum {
 OUTPUT_ADDRESSES   = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
 } output_t;

+typedef enum {
+UNIQUE_NONE  = 1 << 0,
+UNIQUE_ADDR  = 1 << 1,
+UNIQUE_NAME  = 1 << 2,
+UNIQUE_ADDR_CASEFOLD  = 1 << 3,
+
+UNIQUE_BOTH = UNIQUE_NAME | UNIQUE_ADDR,
+} unique_t;
+
 typedef struct {
 sprinter_t *format;

[PATCH v2 3/4] cli: Add support for parsing multiple keyword arguments

2014-10-05 Thread Michal Sojka

From: Jani Nikula 

This allows having multiple --foo=bar --foo=baz options on the command
line, with the corresponding values OR'd together.

[Test added by Michal Sojka]
---
 command-line-arguments.c  | 6 +-
 command-line-arguments.h  | 1 +
 test/T410-argument-parsing.sh | 3 ++-
 test/arg-test.c   | 9 +
 4 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/command-line-arguments.c b/command-line-arguments.c
index 844d6c3..c6f7269 100644
--- a/command-line-arguments.c
+++ b/command-line-arguments.c
@@ -23,7 +23,10 @@ _process_keyword_arg (const notmuch_opt_desc_t *arg_desc, 
char next, const char
 while (keywords->name) {
if (strcmp (arg_str, keywords->name) == 0) {
if (arg_desc->output_var) {
-   *((int *)arg_desc->output_var) = keywords->value;
+   if (arg_desc->opt_type == NOTMUCH_OPT_KEYWORD_FLAGS)
+   *((int *)arg_desc->output_var) |= keywords->value;
+   else
+   *((int *)arg_desc->output_var) = keywords->value;
}
return TRUE;
}
@@ -152,6 +155,7 @@ parse_option (const char *arg,

switch (try->opt_type) {
case NOTMUCH_OPT_KEYWORD:
+   case NOTMUCH_OPT_KEYWORD_FLAGS:
return _process_keyword_arg (try, next, value);
case NOTMUCH_OPT_BOOLEAN:
return _process_boolean_arg (try, next, value);
diff --git a/command-line-arguments.h b/command-line-arguments.h
index de1734a..085a492 100644
--- a/command-line-arguments.h
+++ b/command-line-arguments.h
@@ -8,6 +8,7 @@ enum notmuch_opt_type {
 NOTMUCH_OPT_BOOLEAN,   /* --verbose  */
 NOTMUCH_OPT_INT,   /* --frob=8   */
 NOTMUCH_OPT_KEYWORD,   /* --format=raw|json|text */
+NOTMUCH_OPT_KEYWORD_FLAGS,  /* the above with values OR'd together */
 NOTMUCH_OPT_STRING,/* --file=/tmp/gnarf.txt  */
 NOTMUCH_OPT_POSITION   /* notmuch dump pos_arg   */
 };
diff --git a/test/T410-argument-parsing.sh b/test/T410-argument-parsing.sh
index 94e9087..2e5d7ae 100755
--- a/test/T410-argument-parsing.sh
+++ b/test/T410-argument-parsing.sh
@@ -3,9 +3,10 @@ test_description="argument parsing"
 . ./test-lib.sh

 test_begin_subtest "sanity check"
-$TEST_DIRECTORY/arg-test  pos1  --keyword=one --string=foo pos2 --int=7 > 
OUTPUT
+$TEST_DIRECTORY/arg-test  pos1  --keyword=one --string=foo pos2 --int=7 
--flag=one --flag=three > OUTPUT
 cat < EXPECTED
 keyword 1
+flags 5
 int 7
 string foo
 positional arg 1 pos1
diff --git a/test/arg-test.c b/test/arg-test.c
index 6c49eac..736686d 100644
--- a/test/arg-test.c
+++ b/test/arg-test.c
@@ -7,6 +7,7 @@ int main(int argc, char **argv){
 int opt_index=1;

 int kw_val=0;
+int fl_val=0;
 int int_val=0;
 char *pos_arg1=NULL;
 char *pos_arg2=NULL;
@@ -17,6 +18,11 @@ int main(int argc, char **argv){
  (notmuch_keyword_t []){ { "one", 1 },
  { "two", 2 },
  { 0, 0 } } },
+   { NOTMUCH_OPT_KEYWORD_FLAGS, _val, "flag", 'f',
+ (notmuch_keyword_t []){ { "one",   1 << 0},
+ { "two",   1 << 1 },
+ { "three", 1 << 2 },
+ { 0, 0 } } },
{ NOTMUCH_OPT_INT, _val, "int", 'i', 0},
{ NOTMUCH_OPT_STRING, _val, "string", 's', 0},
{ NOTMUCH_OPT_POSITION, _arg1, 0,0, 0},
@@ -31,6 +37,9 @@ int main(int argc, char **argv){
 if (kw_val)
printf("keyword %d\n", kw_val);

+if (fl_val)
+   printf("flags %d\n", fl_val);
+
 if (int_val)
printf("int %d\n", int_val);

-- 
2.1.1

[PATCH v2 2/4] cli: Extend the search command for --output=addresses and similar

2014-10-05 Thread Michal Sojka

The new outputs allow printing senders, recipients or both of matching
messages.

This code based on a patch from Jani Nikula.
---
 completion/notmuch-completion.bash |   2 +-
 completion/notmuch-completion.zsh  |   3 +-
 doc/man1/notmuch-search.rst|  22 +++-
 notmuch-search.c   | 100 ++---
 test/T090-search-output.sh |  64 
 5 files changed, 182 insertions(+), 9 deletions(-)

diff --git a/completion/notmuch-completion.bash 
b/completion/notmuch-completion.bash
index 0571dc9..c37ddf5 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -294,7 +294,7 @@ _notmuch_search()
return
;;
--output)
-   COMPREPLY=( $( compgen -W "summary threads messages files tags" -- 
"${cur}" ) )
+   COMPREPLY=( $( compgen -W "summary threads messages files tags 
sender recipients addresses" -- "${cur}" ) )
return
;;
--sort)
diff --git a/completion/notmuch-completion.zsh 
b/completion/notmuch-completion.zsh
index 67a9aba..bff8fd5 100644
--- a/completion/notmuch-completion.zsh
+++ b/completion/notmuch-completion.zsh
@@ -52,7 +52,8 @@ _notmuch_search()
   _arguments -s : \
 '--max-threads=[display only the first x threads from the search 
results]:number of threads to show: ' \
 '--first=[omit the first x threads from the search results]:number of 
threads to omit: ' \
-'--sort=[sort results]:sorting:((newest-first\:"reverse chronological 
order" oldest-first\:"chronological order"))'
+'--sort=[sort results]:sorting:((newest-first\:"reverse chronological 
order" oldest-first\:"chronological order"))' \
+'--output=[select what to output]:output:((summary threads messages files 
tags sender recipients addresses))'
 }

 _notmuch()
diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
index 90160f2..3447820 100644
--- a/doc/man1/notmuch-search.rst
+++ b/doc/man1/notmuch-search.rst
@@ -35,7 +35,7 @@ Supported options for **search** include
 intended for programs that invoke **notmuch(1)** internally. If
 omitted, the latest supported version will be used.

-``--output=(summary|threads|messages|files|tags)``
+
``--output=(summary|threads|messages|files|tags|sender|recipients|addresses)``

 **summary**
 Output a summary of each thread with any message matching
@@ -78,6 +78,26 @@ Supported options for **search** include
 by null characters (--format=text0), as a JSON array
 (--format=json), or as an S-Expression list (--format=sexp).

+   **sender**
+Output all addresses from the *From* header that appear on
+any message matching the search terms, either one per line
+(--format=text), separated by null characters
+(--format=text0), as a JSON array (--format=json), or as
+an S-Expression list (--format=sexp).
+
+   Note: Searching for **sender** should be much faster than
+   searching for **recipients** or **addresses**, because
+   sender addresses are cached directly in the database
+   whereas other addresses need to be fetched from message
+   files.
+
+   **recipients**
+Like **sender** but for addresses from *To*, *Cc* and
+   *Bcc* headers.
+
+   **addresses**
+   Like **sender** and **recipients** together.
+
 ``--sort=``\ (**newest-first**\ \|\ **oldest-first**)
 This option can be used to present results in either
 chronological order (**oldest-first**) or reverse chronological
diff --git a/notmuch-search.c b/notmuch-search.c
index 5ac2a26..0614f10 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -23,11 +23,14 @@
 #include "string-util.h"

 typedef enum {
-OUTPUT_SUMMARY,
-OUTPUT_THREADS,
-OUTPUT_MESSAGES,
-OUTPUT_FILES,
-OUTPUT_TAGS
+OUTPUT_SUMMARY = 1 << 0,
+OUTPUT_THREADS = 1 << 1,
+OUTPUT_MESSAGES= 1 << 2,
+OUTPUT_FILES   = 1 << 3,
+OUTPUT_TAGS= 1 << 4,
+OUTPUT_SENDER  = 1 << 5,
+OUTPUT_RECIPIENTS  = 1 << 6,
+OUTPUT_ADDRESSES   = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
 } output_t;

 typedef struct {
@@ -220,6 +223,67 @@ do_search_threads (search_options_t *o)
 return 0;
 }

+static void
+print_address_list (const search_options_t *o, InternetAddressList *list)
+{
+InternetAddress *address;
+int i;
+
+for (i = 0; i < internet_address_list_length (list); i++) {
+   address = internet_address_list_get_address (list, i);
+   if (INTERNET_ADDRESS_IS_GROUP (address)) {
+   InternetAddressGroup *group;
+   InternetAddressList *group_list;
+
+   group = INTERNET_ADDRESS_GROUP (address);
+   group_list = internet_address_group_get_members (group);
+   if (group_list == NULL)
+   continue;
+
+

[PATCH v2 1/4] cli: Refactor option passing in the search command

2014-10-05 Thread Michal Sojka

Many functions that implement the search command need to access command
line options. Instead of passing each option in a separate variable, put
them in a structure and pass only this structure.

This will become handy in the following patches.
---
 notmuch-search.c | 122 ---
 1 file changed, 62 insertions(+), 60 deletions(-)

diff --git a/notmuch-search.c b/notmuch-search.c
index bc9be45..5ac2a26 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -30,6 +30,16 @@ typedef enum {
 OUTPUT_TAGS
 } output_t;

+typedef struct {
+sprinter_t *format;
+notmuch_query_t *query;
+notmuch_sort_t sort;
+output_t output;
+int offset;
+int limit;
+int dupe;
+} search_options_t;
+
 /* Return two stable query strings that identify exactly the matched
  * and unmatched messages currently in thread.  If there are no
  * matched or unmatched messages, the returned buffers will be
@@ -70,46 +80,42 @@ get_thread_query (notmuch_thread_t *thread,
 }

 static int
-do_search_threads (sprinter_t *format,
-  notmuch_query_t *query,
-  notmuch_sort_t sort,
-  output_t output,
-  int offset,
-  int limit)
+do_search_threads (search_options_t *o)
 {
 notmuch_thread_t *thread;
 notmuch_threads_t *threads;
 notmuch_tags_t *tags;
+sprinter_t *format = o->format;
 time_t date;
 int i;

-if (offset < 0) {
-   offset += notmuch_query_count_threads (query);
-   if (offset < 0)
-   offset = 0;
+if (o->offset < 0) {
+   o->offset += notmuch_query_count_threads (o->query);
+   if (o->offset < 0)
+   o->offset = 0;
 }

-threads = notmuch_query_search_threads (query);
+threads = notmuch_query_search_threads (o->query);
 if (threads == NULL)
return 1;

 format->begin_list (format);

 for (i = 0;
-notmuch_threads_valid (threads) && (limit < 0 || i < offset + limit);
+notmuch_threads_valid (threads) && (o->limit < 0 || i < o->offset + 
o->limit);
 notmuch_threads_move_to_next (threads), i++)
 {
thread = notmuch_threads_get (threads);

-   if (i < offset) {
+   if (i < o->offset) {
notmuch_thread_destroy (thread);
continue;
}

-   if (output == OUTPUT_THREADS) {
+   if (o->output == OUTPUT_THREADS) {
format->set_prefix (format, "thread");
format->string (format,
-   notmuch_thread_get_thread_id (thread));
+  notmuch_thread_get_thread_id (thread));
format->separator (format);
} else { /* output == OUTPUT_SUMMARY */
void *ctx_quote = talloc_new (thread);
@@ -123,7 +129,7 @@ do_search_threads (sprinter_t *format,

format->begin_map (format);

-   if (sort == NOTMUCH_SORT_OLDEST_FIRST)
+   if (o->sort == NOTMUCH_SORT_OLDEST_FIRST)
date = notmuch_thread_get_oldest_date (thread);
else
date = notmuch_thread_get_newest_date (thread);
@@ -215,40 +221,36 @@ do_search_threads (sprinter_t *format,
 }

 static int
-do_search_messages (sprinter_t *format,
-   notmuch_query_t *query,
-   output_t output,
-   int offset,
-   int limit,
-   int dupe)
+do_search_messages (search_options_t *o)
 {
 notmuch_message_t *message;
 notmuch_messages_t *messages;
 notmuch_filenames_t *filenames;
+sprinter_t *format = o->format;
 int i;

-if (offset < 0) {
-   offset += notmuch_query_count_messages (query);
-   if (offset < 0)
-   offset = 0;
+if (o->offset < 0) {
+   o->offset += notmuch_query_count_messages (o->query);
+   if (o->offset < 0)
+   o->offset = 0;
 }

-messages = notmuch_query_search_messages (query);
+messages = notmuch_query_search_messages (o->query);
 if (messages == NULL)
return 1;

 format->begin_list (format);

 for (i = 0;
-notmuch_messages_valid (messages) && (limit < 0 || i < offset + limit);
+notmuch_messages_valid (messages) && (o->limit < 0 || i < o->offset + 
o->limit);
 notmuch_messages_move_to_next (messages), i++)
 {
-   if (i < offset)
+   if (i < o->offset)
continue;

message = notmuch_messages_get (messages);

-   if (output == OUTPUT_FILES) {
+   if (o->output == OUTPUT_FILES) {
int j;
filenames = notmuch_message_get_filenames (message);

@@ -256,7 +258,7 @@ do_search_messages (sprinter_t *format,
 notmuch_filenames_valid (filenames);
 notmuch_filenames_move_to_next (filenames), j++)
{
-   if (dupe < 0 || dupe == j) {
+   if (o->dupe < 0 || o->dupe == j) {
format->string (format,

[PATCH v2 0/4] notmuch search --output=addresses

2014-10-05 Thread Michal Sojka

Hi,

this is a second version of my adaptation of Jani's patch series
adding --output=addresses and similar arguments to notmuch search.

Based on the feedback from others, this version uses Jani's original
"keyword flags" implementation with --flag=a --flab=b syntax. Also the
tests for --output and --unique flags are not mixed together, but are
included in the patches that introduce the new features.

I left the default value of the --unique option the same as before,
because I'm convinced that this is what makes most sense. But of
course, we can discuss about that.

-Michal


Jani Nikula (1):
  cli: Add support for parsing multiple keyword arguments

Michal Sojka (3):
  cli: Refactor option passing in the search command
  cli: Extend the search command for --output=addresses and similar
  cli: Add configurable address deduplication for --output=addresses

 command-line-arguments.c   |   6 +-
 command-line-arguments.h   |   1 +
 completion/notmuch-completion.bash |   8 +-
 completion/notmuch-completion.zsh  |   4 +-
 doc/man1/notmuch-search.rst|  54 ++-
 notmuch-search.c   | 311 +
 test/T090-search-output.sh |  64 
 test/T095-search-unique.sh |  63 
 test/T410-argument-parsing.sh  |   3 +-
 test/arg-test.c|   9 ++
 10 files changed, 451 insertions(+), 72 deletions(-)
 create mode 100755 test/T095-search-unique.sh

-- 
2.1.1

[PATCH 08/11] lib: Implement upgrade to ghost messages feature

2014-10-05 Thread Austin Clements

Quoth David Bremner on Oct 05 at 10:56 am:
> Austin Clements  writes:
> > +if (new_features & NOTMUCH_FEATURE_GHOSTS) {
> > +   t_end = db->metadata_keys_end ("thread_id_");
> > +   for (t = db->metadata_keys_begin ("thread_id_"); t != t_end; ++t)
> > +   ++total;
> > +}
> 
> It would be nice to have the comment below, or something like it, for
> the loop above.

  /* The ghost message upgrade converts all thread_id_*
   * metadata values into ghost message documents. */
sound good?

> > +/* Perform metadata upgrades. */
> > +
> > +/* Prior to NOTMUCH_FEATURE_GHOSTS, thread IDs for missing
> > + * messages were stored as database metadata. Change these to
> > + * ghost messages.
> > + */

[PATCH 06/11] lib: Internal support for querying and creating ghost messages

2014-10-05 Thread Austin Clements

Quoth David Bremner on Oct 05 at 10:30 am:
> Austin Clements  writes:
> 
> > +   message->flags &= ~(1 << NOTMUCH_MESSAGE_FLAG_GHOST);
> 
> What do you think about using bit set / clear / read macros?  I don't
> insist, but I wonder if it would make this part more readable.

I'm used to reading this stuff, so either way is fine with me.  Do we
have bit set / clear / read macros?

> > +   else if (*i == "Tghost")
> > +   message->flags |= (1 << NOTMUCH_MESSAGE_FLAG_GHOST);
> > +   else
> 
> It makes me faintly unhappy to have the prefix hardcoded here.
> Not sure if there is a sensible solution.

I agree, but I also don't want to construct the test string every time
or deconstruct the term string every time.  I could move the "T"
prefix string to a #define and use that both here and in
BOOLEAN_PREFIX_INTERNAL, but that solution may be worse than the
problem.  What do you think?

[PATCH 02/11] lib: Refactor _notmuch_database_link_message

2014-10-05 Thread Austin Clements

Quoth David Bremner on Oct 05 at  9:45 am:
> Austin Clements  writes:
> > +void *local = talloc_new (NULL);
> 
> What's the advantage of using a local talloc context here? Is this just
> an optimization?

There are a few allocations that wind up going in to this local
context because of the call to _consume_metadata_thread_id, so it's
more convenient to free this one context on return from
_notmuch_database_link_message than to worry about tracking these
various allocations.

[PATCH] Add default configuration values to the man page

2014-10-05 Thread Michal Sojka

On Wed, Oct 01 2014, Sergei Shilovsky wrote:
> ---
>  doc/man1/notmuch-config.rst | 18 ++
>  1 file changed, 18 insertions(+)
>
> diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
> index 3c9a568..9de5534 100644
> --- a/doc/man1/notmuch-config.rst
> +++ b/doc/man1/notmuch-config.rst
> @@ -49,19 +49,31 @@ The available configuration items are described below.
>  within a sub-directory of the path configured here named
>  ``.notmuch``.
>
> +Default: ``$MAILDIR``, otherwise ``$HOME/mail``.

To be consistent with the rest, this should be "``$MAILDIR`` variable if
set"

> +
>  **user.name**
>  Your full name.
>
> +Default: ``$NAME`` variable if set, otherwise read from
> +``/etc/passwd``.
> +
>  **user.primary\_email**
>  Your primary email address.
>
> +Default: ``$EMAIL`` variable if set, otherwise constructed from the
> +username and hostname of the current machine.
> +
>  **user.other\_email**
>  A list of other email addresses at which you receive email.
>
> +Default: not set.
> +
>  **new.tags**
>  A list of tags that will be added to all messages incorporated
>  by **notmuch new**.
>
> +Default: ``unread;inbox``.
> +
>  **new.ignore**
>  A list of file and directory names, without path, that will not
>  be searched for messages by **notmuch new**. All the files and
> @@ -69,11 +81,15 @@ The available configuration items are described below.
>  ignored, regardless of the location in the mail store directory
>  hierarchy.
>
> +Default: no tags.

This should be "empty list".

> +
>  **search.exclude\_tags**
>  A list of tags that will be excluded from search results by
>  default. Using an excluded tag in a query will override that
>  exclusion.
>
> +Default: ``deleted;spam``.
> +
>  **maildir.synchronize\_flags**
>  If true, then the following maildir flags (in message filenames)
>  will be synchronized with the corresponding notmuch tags:
> @@ -104,6 +120,8 @@ The available configuration items are described below.
>  are properly synchronized to the maildir flags, as the commands
>  expect the database and maildir to be in sync.
>
> +Default: ``true``.
> +
>  ENVIRONMENT
>  ===

Cheers,
-Michal

[PATCH] NEWS: mention the change in default build flags

2014-10-05 Thread David Bremner

Tomi Ollila  writes:

> On Sun, Oct 05 2014, David Bremner  wrote:
>
>> It blows things up by a factor of six or so, so it's worth giving
>> people a heads up. It won't effect e.g. Debian, that already builds
>> with -g and then strips.
>
> Maybe SomeOne(?) could make a patch that strip(1)s at make install time ?
>

Then I guess the NEWS item would be unneeded?

d

[PATCH 00/11] Add ghost messages and fix thread linking

2014-10-05 Thread David Bremner

Austin Clements  writes:

> This series modifies our database representation of messages that have
> been referenced by other messages, but for which we don't have the
> message itself.  Currently, we store this information as Xapian
> metadata, but this has several downsides for performance and
> complexity and results in hard-to-fix thread linking bugs.  This patch
> series implements "ghost messages", which replace this Xapian metadata
> with Xapian documents that look and act very much like regular message
> documents, but simply have no content.  This simplifies and speeds up
> our thread linking algorithm and fixes the currently broken thread
> linking test.

Other than some style / documentation quibbles already noted, 
the series looks good to me.

d

[PATCH 08/11] lib: Implement upgrade to ghost messages feature

2014-10-05 Thread David Bremner

Austin Clements  writes:
> +if (new_features & NOTMUCH_FEATURE_GHOSTS) {
> + t_end = db->metadata_keys_end ("thread_id_");
> + for (t = db->metadata_keys_begin ("thread_id_"); t != t_end; ++t)
> + ++total;
> +}

It would be nice to have the comment below, or something like it, for
the loop above.

> +/* Perform metadata upgrades. */
> +
> +/* Prior to NOTMUCH_FEATURE_GHOSTS, thread IDs for missing
> + * messages were stored as database metadata. Change these to
> + * ghost messages.
> + */

[PATCH 06/11] lib: Internal support for querying and creating ghost messages

2014-10-05 Thread David Bremner

Austin Clements  writes:

> + message->flags &= ~(1 << NOTMUCH_MESSAGE_FLAG_GHOST);

What do you think about using bit set / clear / read macros?  I don't
insist, but I wonder if it would make this part more readable.

> + else if (*i == "Tghost")
> + message->flags |= (1 << NOTMUCH_MESSAGE_FLAG_GHOST);
> + else

It makes me faintly unhappy to have the prefix hardcoded here.
Not sure if there is a sensible solution.

[PATCH] NEWS: mention the change in default build flags

2014-10-05 Thread Tomi Ollila

On Sun, Oct 05 2014, David Bremner  wrote:

> It blows things up by a factor of six or so, so it's worth giving
> people a heads up. It won't effect e.g. Debian, that already builds
> with -g and then strips.

Maybe SomeOne(?) could make a patch that strip(1)s at make install time ?

Tomi


> ---
>  NEWS | 7 +++
>  1 file changed, 7 insertions(+)
>
> diff --git a/NEWS b/NEWS
> index fa57e5d..eb769fd 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -42,6 +42,13 @@ Python Bindings
>  
>  Add support for `notmuch_query_add_tag_exclude`
>  
> +Build System
> +
> +
> +The notmuch binaries and libraries are now build with debugging symbols
> +by default.  Users concerned with disk space should change the
> +defaults when configuring or use the strip(1) command.
> +
>  Notmuch 0.18.1 (2014-06-25)
>  ===
>  
> -- 
> 2.1.0
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 02/11] lib: Refactor _notmuch_database_link_message

2014-10-05 Thread David Bremner

Austin Clements  writes:
> +void *local = talloc_new (NULL);

What's the advantage of using a local talloc context here? Is this just
an optimization?

d

[PATCH] NEWS: Document "nmbug: Translate to Python"

2014-10-05 Thread W. Trevor King

This is mostly culled from the commit message for 7f2cb3be (nmbug:
Translate to Python, 2014-10-03).  I realized while writing it that
the 7f2cb3be commit message has:

  * 'nmbug log' now execs 'git log', as there's no need to keep the
Python process around once we've launched Git there.

But we dropped that exec in favor of the subprocess approach between
v3 and v4, I just forgot to update the commit message [1].

[1]: id:e630b6763e9d0771718afee41ea15b29bb4a1de8.1409935538.git.wking at 
tremily.us
 http://article.gmane.org/gmane.mail.notmuch.general/19007
---
 NEWS | 48 
 1 file changed, 48 insertions(+)

diff --git a/NEWS b/NEWS
index fa57e5d..c111dd0 100644
--- a/NEWS
+++ b/NEWS
@@ -22,6 +22,54 @@ Library changes
 Add return status to notmuch_database_close and
 notmuch_database_destroy

+nmbug
+-
+
+The Perl script has been translated to Python; you'll need Python 2.7
+or anything from the 3.x line.  This gives us better control over
+subprocesses without resorting to third-party modules.  Most of the
+user-facing interface is the same, but there are a few changes, where
+reproducing the original interface was too difficult or we saw a
+change to make the underlying Git interface accessible:
+
+* 'nmbug help' has been split between the general 'nmbug --help' and
+  the command-specific 'nmbug COMMAND --help'.
+
+* Commands are no longer split into "most common", "other useful", and
+  "less common" sets.
+
+* 'nmbug commit' now only uses a single argument for the optional
+  commit-message text.
+
+* The default repository for 'nmbug push' and 'nmbug fetch' is now the
+  current branch's upstream (branch..remote) instead of
+  'origin'.  When we have to, we extract this remote by hand, but
+  where possible we just call the Git command without a repository
+  argument, and leave it to Git to figure out the default.
+
+* 'nmbug push' accepts multiple refspecs if you want to explicitly
+  specify what to push.  Otherwise, the refspec(s) pushed depend on
+  push.default.  The Perl version hardcoded 'master' as the pushed
+  refspec.
+
+* 'nmbug pull' defaults to the current branch's upstream
+  (branch..remote and branch..merge) instead of hardcoding
+  'origin' and 'master'.  It also supports multiple refspecs if for
+  some crazy reason you need an octopus merge (but mostly to avoid
+  breaking consistency with 'git pull').
+
+* 'nmbug status' now catches stderr, and doesn't print errors like:
+
+No upstream configured for branch 'master'
+
+  The Perl implementation had just learned to avoid crashing on that
+  case, but wasn't yet catching the dying subprocess's stderr.
+
+* 'nmbug archive' now accepts positional arguments for the tree-ish
+  and additional 'git archive' options.  For example, you can run:
+
+$ nmbug archive HEAD -- --format tar.gz
+
 nmbug-status
 

-- 
2.1.0.60.g85f0837

[PATCH] NEWS: mention the change in default build flags

2014-10-05 Thread David Bremner

It blows things up by a factor of six or so, so it's worth giving
people a heads up. It won't effect e.g. Debian, that already builds
with -g and then strips.
---
 NEWS | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/NEWS b/NEWS
index fa57e5d..eb769fd 100644
--- a/NEWS
+++ b/NEWS
@@ -42,6 +42,13 @@ Python Bindings

 Add support for `notmuch_query_add_tag_exclude`

+Build System
+
+
+The notmuch binaries and libraries are now build with debugging symbols
+by default.  Users concerned with disk space should change the
+defaults when configuring or use the strip(1) command.
+
 Notmuch 0.18.1 (2014-06-25)
 ===

-- 
2.1.0

[PATCH] test: Port atomicity test to Python

2014-10-05 Thread David Bremner

Austin Clements  writes:

> Previously, this was implemented using a horrible GDB script (because
> there is no such thing as a non-horrible GDB script).  This GDB script
> often broke with newer versions of GDB for mysterious reasons.  Port
> the test script to GDB's Python API, which makes the code much cleaner
> and, hopefully, more stable.

pushed, 

d

[PATCH 1/2] test: check for debug symbols in notmuch

2014-10-05 Thread David Bremner

Tomi Ollila  writes:

> On Fri, Oct 03 2014, David Bremner  wrote:
>
>> In the future, tests may rely on debug symbols being present in
>> notmuch, so we plan to switch the default flags.
>>
>> The main purpose of this test is to help explain the perhaps
>> mysterious failures of other tests which rely on symbols being
>> present.
>> ---
>
> series LGTM.
>

Series pushed,

d

[PATCH] hex-escape: remove unused variable default_buf_size

2014-10-05 Thread David Bremner

Jani Nikula  writes:

> Found by clang:
>

pushed

d

[PATCH v6 1/2] nmbug: Translate to Python

2014-10-05 Thread David Bremner

"W. Trevor King"  writes:

> This allows us to capture stdout and stderr separately, and do other
> explicit subprocess manipulation without resorting to external
> packages.  It should be compatible with Python 2.7 and later
> (including the 3.x series).
>

Pushed patch 1/2 to master. This probably merits a few lines in NEWS.

d

Re: [PATCH] NEWS: mention the change in default build flags

2014-10-05 Thread Tomi Ollila

On Sun, Oct 05 2014, David Bremner da...@tethera.net wrote:

 It blows things up by a factor of six or so, so it's worth giving
 people a heads up. It won't effect e.g. Debian, that already builds
 with -g and then strips.

Maybe SomeOne(™) could make a patch that strip(1)s at make install time ?

Tomi


 ---
  NEWS | 7 +++
  1 file changed, 7 insertions(+)

 diff --git a/NEWS b/NEWS
 index fa57e5d..eb769fd 100644
 --- a/NEWS
 +++ b/NEWS
 @@ -42,6 +42,13 @@ Python Bindings
  
  Add support for `notmuch_query_add_tag_exclude`
  
 +Build System
 +
 +
 +The notmuch binaries and libraries are now build with debugging symbols
 +by default.  Users concerned with disk space should change the
 +defaults when configuring or use the strip(1) command.
 +
  Notmuch 0.18.1 (2014-06-25)
  ===
  
 -- 
 2.1.0

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 02/11] lib: Refactor _notmuch_database_link_message

2014-10-05 Thread David Bremner

Austin Clements acleme...@csail.mit.edu writes:
 +void *local = talloc_new (NULL);

What's the advantage of using a local talloc context here? Is this just
an optimization?

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 06/11] lib: Internal support for querying and creating ghost messages

2014-10-05 Thread David Bremner

Austin Clements acleme...@csail.mit.edu writes:

 + message-flags = ~(1  NOTMUCH_MESSAGE_FLAG_GHOST);

What do you think about using bit set / clear / read macros?  I don't
insist, but I wonder if it would make this part more readable.

 + else if (*i == Tghost)
 + message-flags |= (1  NOTMUCH_MESSAGE_FLAG_GHOST);
 + else

It makes me faintly unhappy to have the prefix hardcoded here.
Not sure if there is a sensible solution.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 08/11] lib: Implement upgrade to ghost messages feature

2014-10-05 Thread David Bremner

Austin Clements acleme...@csail.mit.edu writes:
 +if (new_features  NOTMUCH_FEATURE_GHOSTS) {
 + t_end = db-metadata_keys_end (thread_id_);
 + for (t = db-metadata_keys_begin (thread_id_); t != t_end; ++t)
 + ++total;
 +}

It would be nice to have the comment below, or something like it, for
the loop above.

 +/* Perform metadata upgrades. */
 +
 +/* Prior to NOTMUCH_FEATURE_GHOSTS, thread IDs for missing
 + * messages were stored as database metadata. Change these to
 + * ghost messages.
 + */
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 00/11] Add ghost messages and fix thread linking

2014-10-05 Thread David Bremner

Austin Clements acleme...@csail.mit.edu writes:

 This series modifies our database representation of messages that have
 been referenced by other messages, but for which we don't have the
 message itself.  Currently, we store this information as Xapian
 metadata, but this has several downsides for performance and
 complexity and results in hard-to-fix thread linking bugs.  This patch
 series implements ghost messages, which replace this Xapian metadata
 with Xapian documents that look and act very much like regular message
 documents, but simply have no content.  This simplifies and speeds up
 our thread linking algorithm and fixes the currently broken thread
 linking test.

Other than some style / documentation quibbles already noted, 
the series looks good to me.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] NEWS: mention the change in default build flags

2014-10-05 Thread David Bremner

Tomi Ollila tomi.oll...@iki.fi writes:

 On Sun, Oct 05 2014, David Bremner da...@tethera.net wrote:

 It blows things up by a factor of six or so, so it's worth giving
 people a heads up. It won't effect e.g. Debian, that already builds
 with -g and then strips.

 Maybe SomeOne(™) could make a patch that strip(1)s at make install time ?


Then I guess the NEWS item would be unneeded?

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Add default configuration values to the man page

2014-10-05 Thread Michal Sojka

On Wed, Oct 01 2014, Sergei Shilovsky wrote:
 ---
  doc/man1/notmuch-config.rst | 18 ++
  1 file changed, 18 insertions(+)

 diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
 index 3c9a568..9de5534 100644
 --- a/doc/man1/notmuch-config.rst
 +++ b/doc/man1/notmuch-config.rst
 @@ -49,19 +49,31 @@ The available configuration items are described below.
  within a sub-directory of the path configured here named
  ``.notmuch``.

 +Default: ``$MAILDIR``, otherwise ``$HOME/mail``.

To be consistent with the rest, this should be ``$MAILDIR`` variable if
set

 +
  **user.name**
  Your full name.

 +Default: ``$NAME`` variable if set, otherwise read from
 +``/etc/passwd``.
 +
  **user.primary\_email**
  Your primary email address.

 +Default: ``$EMAIL`` variable if set, otherwise constructed from the
 +username and hostname of the current machine.
 +
  **user.other\_email**
  A list of other email addresses at which you receive email.

 +Default: not set.
 +
  **new.tags**
  A list of tags that will be added to all messages incorporated
  by **notmuch new**.

 +Default: ``unread;inbox``.
 +
  **new.ignore**
  A list of file and directory names, without path, that will not
  be searched for messages by **notmuch new**. All the files and
 @@ -69,11 +81,15 @@ The available configuration items are described below.
  ignored, regardless of the location in the mail store directory
  hierarchy.

 +Default: no tags.

This should be empty list.

 +
  **search.exclude\_tags**
  A list of tags that will be excluded from search results by
  default. Using an excluded tag in a query will override that
  exclusion.

 +Default: ``deleted;spam``.
 +
  **maildir.synchronize\_flags**
  If true, then the following maildir flags (in message filenames)
  will be synchronized with the corresponding notmuch tags:
 @@ -104,6 +120,8 @@ The available configuration items are described below.
  are properly synchronized to the maildir flags, as the commands
  expect the database and maildir to be in sync.

 +Default: ``true``.
 +
  ENVIRONMENT
  ===

Cheers,
-Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] NEWS: Document nmbug: Translate to Python

2014-10-05 Thread W. Trevor King

This is mostly culled from the commit message for 7f2cb3be (nmbug:
Translate to Python, 2014-10-03).  I realized while writing it that
the 7f2cb3be commit message has:

  * 'nmbug log' now execs 'git log', as there's no need to keep the
Python process around once we've launched Git there.

But we dropped that exec in favor of the subprocess approach between
v3 and v4, I just forgot to update the commit message [1].

[1]: id:e630b6763e9d0771718afee41ea15b29bb4a1de8.1409935538.git.wk...@tremily.us
 http://article.gmane.org/gmane.mail.notmuch.general/19007
---
 NEWS | 48 
 1 file changed, 48 insertions(+)

diff --git a/NEWS b/NEWS
index fa57e5d..c111dd0 100644
--- a/NEWS
+++ b/NEWS
@@ -22,6 +22,54 @@ Library changes
 Add return status to notmuch_database_close and
 notmuch_database_destroy
 
+nmbug
+-
+
+The Perl script has been translated to Python; you'll need Python 2.7
+or anything from the 3.x line.  This gives us better control over
+subprocesses without resorting to third-party modules.  Most of the
+user-facing interface is the same, but there are a few changes, where
+reproducing the original interface was too difficult or we saw a
+change to make the underlying Git interface accessible:
+
+* 'nmbug help' has been split between the general 'nmbug --help' and
+  the command-specific 'nmbug COMMAND --help'.
+
+* Commands are no longer split into most common, other useful, and
+  less common sets.
+
+* 'nmbug commit' now only uses a single argument for the optional
+  commit-message text.
+
+* The default repository for 'nmbug push' and 'nmbug fetch' is now the
+  current branch's upstream (branch.name.remote) instead of
+  'origin'.  When we have to, we extract this remote by hand, but
+  where possible we just call the Git command without a repository
+  argument, and leave it to Git to figure out the default.
+
+* 'nmbug push' accepts multiple refspecs if you want to explicitly
+  specify what to push.  Otherwise, the refspec(s) pushed depend on
+  push.default.  The Perl version hardcoded 'master' as the pushed
+  refspec.
+
+* 'nmbug pull' defaults to the current branch's upstream
+  (branch.name.remote and branch.name.merge) instead of hardcoding
+  'origin' and 'master'.  It also supports multiple refspecs if for
+  some crazy reason you need an octopus merge (but mostly to avoid
+  breaking consistency with 'git pull').
+
+* 'nmbug status' now catches stderr, and doesn't print errors like:
+
+No upstream configured for branch 'master'
+
+  The Perl implementation had just learned to avoid crashing on that
+  case, but wasn't yet catching the dying subprocess's stderr.
+
+* 'nmbug archive' now accepts positional arguments for the tree-ish
+  and additional 'git archive' options.  For example, you can run:
+
+$ nmbug archive HEAD -- --format tar.gz
+
 nmbug-status
 
 
-- 
2.1.0.60.g85f0837

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2 0/4] notmuch search --output=addresses

2014-10-05 Thread Michal Sojka

Hi,

this is a second version of my adaptation of Jani's patch series
adding --output=addresses and similar arguments to notmuch search.

Based on the feedback from others, this version uses Jani's original
keyword flags implementation with --flag=a --flab=b syntax. Also the
tests for --output and --unique flags are not mixed together, but are
included in the patches that introduce the new features.

I left the default value of the --unique option the same as before,
because I'm convinced that this is what makes most sense. But of
course, we can discuss about that.

-Michal


Jani Nikula (1):
  cli: Add support for parsing multiple keyword arguments

Michal Sojka (3):
  cli: Refactor option passing in the search command
  cli: Extend the search command for --output=addresses and similar
  cli: Add configurable address deduplication for --output=addresses

 command-line-arguments.c   |   6 +-
 command-line-arguments.h   |   1 +
 completion/notmuch-completion.bash |   8 +-
 completion/notmuch-completion.zsh  |   4 +-
 doc/man1/notmuch-search.rst|  54 ++-
 notmuch-search.c   | 311 +
 test/T090-search-output.sh |  64 
 test/T095-search-unique.sh |  63 
 test/T410-argument-parsing.sh  |   3 +-
 test/arg-test.c|   9 ++
 10 files changed, 451 insertions(+), 72 deletions(-)
 create mode 100755 test/T095-search-unique.sh

-- 
2.1.1

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2 4/4] cli: Add configurable address deduplication for --output=addresses

2014-10-05 Thread Michal Sojka

The code here is an extended version of a patch from Jani Nikula.
---
 completion/notmuch-completion.bash |   6 ++-
 completion/notmuch-completion.zsh  |   3 +-
 doc/man1/notmuch-search.rst|  32 
 notmuch-search.c   | 101 ++---
 test/T090-search-output.sh |   6 +--
 test/T095-search-unique.sh |  63 +++
 6 files changed, 200 insertions(+), 11 deletions(-)
 create mode 100755 test/T095-search-unique.sh

diff --git a/completion/notmuch-completion.bash 
b/completion/notmuch-completion.bash
index c37ddf5..8bc7874 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -305,12 +305,16 @@ _notmuch_search()
COMPREPLY=( $( compgen -W true false flag all -- ${cur} ) )
return
;;
+   --unique)
+   COMPREPLY=( $( compgen -W none addr addrfold name -- ${cur} ) )
+   return
+   ;;
 esac
 
 ! $split 
 case ${cur} in
-*)
-   local options=--format= --output= --sort= --offset= --limit= 
--exclude= --duplicate=
+   local options=--format= --output= --sort= --offset= --limit= 
--exclude= --duplicate= --unique=
compopt -o nospace
COMPREPLY=( $(compgen -W $options -- ${cur}) )
;;
diff --git a/completion/notmuch-completion.zsh 
b/completion/notmuch-completion.zsh
index bff8fd5..cf4968c 100644
--- a/completion/notmuch-completion.zsh
+++ b/completion/notmuch-completion.zsh
@@ -53,7 +53,8 @@ _notmuch_search()
 '--max-threads=[display only the first x threads from the search 
results]:number of threads to show: ' \
 '--first=[omit the first x threads from the search results]:number of 
threads to omit: ' \
 '--sort=[sort results]:sorting:((newest-first\:reverse chronological 
order oldest-first\:chronological order))' \
-'--output=[select what to output]:output:((summary threads messages files 
tags sender recipients addresses))'
+'--output=[select what to output]:output:((summary threads messages files 
tags sender recipients addresses))' \
+'--unique=[address deduplication]:unique:((none\:no deduplication 
addr\:deduplicate by address addrfold\:deduplicate by case-insensitive 
address name\:deduplicate by name))'
 }
 
 _notmuch()
diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
index 3447820..9a9d9c3 100644
--- a/doc/man1/notmuch-search.rst
+++ b/doc/man1/notmuch-search.rst
@@ -85,6 +85,9 @@ Supported options for **search** include
 (--format=text0), as a JSON array (--format=json), or as
 an S-Expression list (--format=sexp).
 
+Handling of duplicate addresses and/or names can be
+controlled with the --unique option.
+
Note: Searching for **sender** should be much faster than
searching for **recipients** or **addresses**, because
sender addresses are cached directly in the database
@@ -151,6 +154,35 @@ Supported options for **search** include
 prefix. The prefix matches messages based on filenames. This
 option filters filenames of the matching messages.
 
+``--unique=``\ (**none**\ \|\ **addr**\ \|\ **addrfold**\ \|\ **name**)
+
+Can be used with ``--output=addresses``, ``--output=sender``
+or ``--output=recipients`` to control the address
+deduplication algorithm.
+
+   **none** means that no deduplication is performed. The same
+   address can appear multiple times in the output.
+
+   **addr** means that case-sensitive deduplication is performed
+   on the address part. For example, given the addresses John
+   Doe j...@example.com and Dr. John Doe j...@example.com,
+   only one will be printed.
+
+   **addrfold** is the same as **addr** but with case folding
+   applied. For example, given the addresses John Doe
+   j...@example.com and John Doe j...@example.com, only
+   one will be printed. This is the default.
+
+   **name** means that case-sensitive deduplication is performed
+   on the name part. For example, given the addresses John Doe
+   j...@example.com and John Doe j...@doe.name, only one
+   will be printed.
+
+   This option can be given multiple times to output unique
+   combinations of names and addresses. For example,
+   ``--unique=name --unique=addr`` will print unique
+   case-sensitive combinations of name and address.
+
 EXIT STATUS
 ===
 
diff --git a/notmuch-search.c b/notmuch-search.c
index 0614f10..94d400e 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -33,6 +33,15 @@ typedef enum {
 OUTPUT_ADDRESSES   = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
 } output_t;
 
+typedef enum {
+UNIQUE_NONE  = 1  0,
+UNIQUE_ADDR  = 1  1,
+UNIQUE_NAME  = 1  2,
+UNIQUE_ADDR_CASEFOLD  = 1  3,
+
+UNIQUE_BOTH = UNIQUE_NAME | UNIQUE_ADDR,
+} unique_t;
+

[PATCH v2 3/4] cli: Add support for parsing multiple keyword arguments

2014-10-05 Thread Michal Sojka

From: Jani Nikula j...@nikula.org

This allows having multiple --foo=bar --foo=baz options on the command
line, with the corresponding values OR'd together.

[Test added by Michal Sojka]
---
 command-line-arguments.c  | 6 +-
 command-line-arguments.h  | 1 +
 test/T410-argument-parsing.sh | 3 ++-
 test/arg-test.c   | 9 +
 4 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/command-line-arguments.c b/command-line-arguments.c
index 844d6c3..c6f7269 100644
--- a/command-line-arguments.c
+++ b/command-line-arguments.c
@@ -23,7 +23,10 @@ _process_keyword_arg (const notmuch_opt_desc_t *arg_desc, 
char next, const char
 while (keywords-name) {
if (strcmp (arg_str, keywords-name) == 0) {
if (arg_desc-output_var) {
-   *((int *)arg_desc-output_var) = keywords-value;
+   if (arg_desc-opt_type == NOTMUCH_OPT_KEYWORD_FLAGS)
+   *((int *)arg_desc-output_var) |= keywords-value;
+   else
+   *((int *)arg_desc-output_var) = keywords-value;
}
return TRUE;
}
@@ -152,6 +155,7 @@ parse_option (const char *arg,
 
switch (try-opt_type) {
case NOTMUCH_OPT_KEYWORD:
+   case NOTMUCH_OPT_KEYWORD_FLAGS:
return _process_keyword_arg (try, next, value);
case NOTMUCH_OPT_BOOLEAN:
return _process_boolean_arg (try, next, value);
diff --git a/command-line-arguments.h b/command-line-arguments.h
index de1734a..085a492 100644
--- a/command-line-arguments.h
+++ b/command-line-arguments.h
@@ -8,6 +8,7 @@ enum notmuch_opt_type {
 NOTMUCH_OPT_BOOLEAN,   /* --verbose  */
 NOTMUCH_OPT_INT,   /* --frob=8   */
 NOTMUCH_OPT_KEYWORD,   /* --format=raw|json|text */
+NOTMUCH_OPT_KEYWORD_FLAGS,  /* the above with values OR'd together */
 NOTMUCH_OPT_STRING,/* --file=/tmp/gnarf.txt  */
 NOTMUCH_OPT_POSITION   /* notmuch dump pos_arg   */
 };
diff --git a/test/T410-argument-parsing.sh b/test/T410-argument-parsing.sh
index 94e9087..2e5d7ae 100755
--- a/test/T410-argument-parsing.sh
+++ b/test/T410-argument-parsing.sh
@@ -3,9 +3,10 @@ test_description=argument parsing
 . ./test-lib.sh
 
 test_begin_subtest sanity check
-$TEST_DIRECTORY/arg-test  pos1  --keyword=one --string=foo pos2 --int=7  
OUTPUT
+$TEST_DIRECTORY/arg-test  pos1  --keyword=one --string=foo pos2 --int=7 
--flag=one --flag=three  OUTPUT
 cat EOF  EXPECTED
 keyword 1
+flags 5
 int 7
 string foo
 positional arg 1 pos1
diff --git a/test/arg-test.c b/test/arg-test.c
index 6c49eac..736686d 100644
--- a/test/arg-test.c
+++ b/test/arg-test.c
@@ -7,6 +7,7 @@ int main(int argc, char **argv){
 int opt_index=1;
 
 int kw_val=0;
+int fl_val=0;
 int int_val=0;
 char *pos_arg1=NULL;
 char *pos_arg2=NULL;
@@ -17,6 +18,11 @@ int main(int argc, char **argv){
  (notmuch_keyword_t []){ { one, 1 },
  { two, 2 },
  { 0, 0 } } },
+   { NOTMUCH_OPT_KEYWORD_FLAGS, fl_val, flag, 'f',
+ (notmuch_keyword_t []){ { one,   1  0},
+ { two,   1  1 },
+ { three, 1  2 },
+ { 0, 0 } } },
{ NOTMUCH_OPT_INT, int_val, int, 'i', 0},
{ NOTMUCH_OPT_STRING, string_val, string, 's', 0},
{ NOTMUCH_OPT_POSITION, pos_arg1, 0,0, 0},
@@ -31,6 +37,9 @@ int main(int argc, char **argv){
 if (kw_val)
printf(keyword %d\n, kw_val);
 
+if (fl_val)
+   printf(flags %d\n, fl_val);
+
 if (int_val)
printf(int %d\n, int_val);
 
-- 
2.1.1

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2 1/4] cli: Refactor option passing in the search command

2014-10-05 Thread Michal Sojka

Many functions that implement the search command need to access command
line options. Instead of passing each option in a separate variable, put
them in a structure and pass only this structure.

This will become handy in the following patches.
---
 notmuch-search.c | 122 ---
 1 file changed, 62 insertions(+), 60 deletions(-)

diff --git a/notmuch-search.c b/notmuch-search.c
index bc9be45..5ac2a26 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -30,6 +30,16 @@ typedef enum {
 OUTPUT_TAGS
 } output_t;
 
+typedef struct {
+sprinter_t *format;
+notmuch_query_t *query;
+notmuch_sort_t sort;
+output_t output;
+int offset;
+int limit;
+int dupe;
+} search_options_t;
+
 /* Return two stable query strings that identify exactly the matched
  * and unmatched messages currently in thread.  If there are no
  * matched or unmatched messages, the returned buffers will be
@@ -70,46 +80,42 @@ get_thread_query (notmuch_thread_t *thread,
 }
 
 static int
-do_search_threads (sprinter_t *format,
-  notmuch_query_t *query,
-  notmuch_sort_t sort,
-  output_t output,
-  int offset,
-  int limit)
+do_search_threads (search_options_t *o)
 {
 notmuch_thread_t *thread;
 notmuch_threads_t *threads;
 notmuch_tags_t *tags;
+sprinter_t *format = o-format;
 time_t date;
 int i;
 
-if (offset  0) {
-   offset += notmuch_query_count_threads (query);
-   if (offset  0)
-   offset = 0;
+if (o-offset  0) {
+   o-offset += notmuch_query_count_threads (o-query);
+   if (o-offset  0)
+   o-offset = 0;
 }
 
-threads = notmuch_query_search_threads (query);
+threads = notmuch_query_search_threads (o-query);
 if (threads == NULL)
return 1;
 
 format-begin_list (format);
 
 for (i = 0;
-notmuch_threads_valid (threads)  (limit  0 || i  offset + limit);
+notmuch_threads_valid (threads)  (o-limit  0 || i  o-offset + 
o-limit);
 notmuch_threads_move_to_next (threads), i++)
 {
thread = notmuch_threads_get (threads);
 
-   if (i  offset) {
+   if (i  o-offset) {
notmuch_thread_destroy (thread);
continue;
}
 
-   if (output == OUTPUT_THREADS) {
+   if (o-output == OUTPUT_THREADS) {
format-set_prefix (format, thread);
format-string (format,
-   notmuch_thread_get_thread_id (thread));
+  notmuch_thread_get_thread_id (thread));
format-separator (format);
} else { /* output == OUTPUT_SUMMARY */
void *ctx_quote = talloc_new (thread);
@@ -123,7 +129,7 @@ do_search_threads (sprinter_t *format,
 
format-begin_map (format);
 
-   if (sort == NOTMUCH_SORT_OLDEST_FIRST)
+   if (o-sort == NOTMUCH_SORT_OLDEST_FIRST)
date = notmuch_thread_get_oldest_date (thread);
else
date = notmuch_thread_get_newest_date (thread);
@@ -215,40 +221,36 @@ do_search_threads (sprinter_t *format,
 }
 
 static int
-do_search_messages (sprinter_t *format,
-   notmuch_query_t *query,
-   output_t output,
-   int offset,
-   int limit,
-   int dupe)
+do_search_messages (search_options_t *o)
 {
 notmuch_message_t *message;
 notmuch_messages_t *messages;
 notmuch_filenames_t *filenames;
+sprinter_t *format = o-format;
 int i;
 
-if (offset  0) {
-   offset += notmuch_query_count_messages (query);
-   if (offset  0)
-   offset = 0;
+if (o-offset  0) {
+   o-offset += notmuch_query_count_messages (o-query);
+   if (o-offset  0)
+   o-offset = 0;
 }
 
-messages = notmuch_query_search_messages (query);
+messages = notmuch_query_search_messages (o-query);
 if (messages == NULL)
return 1;
 
 format-begin_list (format);
 
 for (i = 0;
-notmuch_messages_valid (messages)  (limit  0 || i  offset + limit);
+notmuch_messages_valid (messages)  (o-limit  0 || i  o-offset + 
o-limit);
 notmuch_messages_move_to_next (messages), i++)
 {
-   if (i  offset)
+   if (i  o-offset)
continue;
 
message = notmuch_messages_get (messages);
 
-   if (output == OUTPUT_FILES) {
+   if (o-output == OUTPUT_FILES) {
int j;
filenames = notmuch_message_get_filenames (message);
 
@@ -256,7 +258,7 @@ do_search_messages (sprinter_t *format,
 notmuch_filenames_valid (filenames);
 notmuch_filenames_move_to_next (filenames), j++)
{
-   if (dupe  0 || dupe == j) {
+   if (o-dupe  0 || o-dupe == j) {
format-string (format, notmuch_filenames_get (filenames));

[PATCH v2 2/4] cli: Extend the search command for --output=addresses and similar

2014-10-05 Thread Michal Sojka

The new outputs allow printing senders, recipients or both of matching
messages.

This code based on a patch from Jani Nikula.
---
 completion/notmuch-completion.bash |   2 +-
 completion/notmuch-completion.zsh  |   3 +-
 doc/man1/notmuch-search.rst|  22 +++-
 notmuch-search.c   | 100 ++---
 test/T090-search-output.sh |  64 
 5 files changed, 182 insertions(+), 9 deletions(-)

diff --git a/completion/notmuch-completion.bash 
b/completion/notmuch-completion.bash
index 0571dc9..c37ddf5 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -294,7 +294,7 @@ _notmuch_search()
return
;;
--output)
-   COMPREPLY=( $( compgen -W summary threads messages files tags -- 
${cur} ) )
+   COMPREPLY=( $( compgen -W summary threads messages files tags 
sender recipients addresses -- ${cur} ) )
return
;;
--sort)
diff --git a/completion/notmuch-completion.zsh 
b/completion/notmuch-completion.zsh
index 67a9aba..bff8fd5 100644
--- a/completion/notmuch-completion.zsh
+++ b/completion/notmuch-completion.zsh
@@ -52,7 +52,8 @@ _notmuch_search()
   _arguments -s : \
 '--max-threads=[display only the first x threads from the search 
results]:number of threads to show: ' \
 '--first=[omit the first x threads from the search results]:number of 
threads to omit: ' \
-'--sort=[sort results]:sorting:((newest-first\:reverse chronological 
order oldest-first\:chronological order))'
+'--sort=[sort results]:sorting:((newest-first\:reverse chronological 
order oldest-first\:chronological order))' \
+'--output=[select what to output]:output:((summary threads messages files 
tags sender recipients addresses))'
 }
 
 _notmuch()
diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
index 90160f2..3447820 100644
--- a/doc/man1/notmuch-search.rst
+++ b/doc/man1/notmuch-search.rst
@@ -35,7 +35,7 @@ Supported options for **search** include
 intended for programs that invoke **notmuch(1)** internally. If
 omitted, the latest supported version will be used.
 
-``--output=(summary|threads|messages|files|tags)``
+
``--output=(summary|threads|messages|files|tags|sender|recipients|addresses)``
 
 **summary**
 Output a summary of each thread with any message matching
@@ -78,6 +78,26 @@ Supported options for **search** include
 by null characters (--format=text0), as a JSON array
 (--format=json), or as an S-Expression list (--format=sexp).
 
+   **sender**
+Output all addresses from the *From* header that appear on
+any message matching the search terms, either one per line
+(--format=text), separated by null characters
+(--format=text0), as a JSON array (--format=json), or as
+an S-Expression list (--format=sexp).
+
+   Note: Searching for **sender** should be much faster than
+   searching for **recipients** or **addresses**, because
+   sender addresses are cached directly in the database
+   whereas other addresses need to be fetched from message
+   files.
+
+   **recipients**
+Like **sender** but for addresses from *To*, *Cc* and
+   *Bcc* headers.
+
+   **addresses**
+   Like **sender** and **recipients** together.
+
 ``--sort=``\ (**newest-first**\ \|\ **oldest-first**)
 This option can be used to present results in either
 chronological order (**oldest-first**) or reverse chronological
diff --git a/notmuch-search.c b/notmuch-search.c
index 5ac2a26..0614f10 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -23,11 +23,14 @@
 #include string-util.h
 
 typedef enum {
-OUTPUT_SUMMARY,
-OUTPUT_THREADS,
-OUTPUT_MESSAGES,
-OUTPUT_FILES,
-OUTPUT_TAGS
+OUTPUT_SUMMARY = 1  0,
+OUTPUT_THREADS = 1  1,
+OUTPUT_MESSAGES= 1  2,
+OUTPUT_FILES   = 1  3,
+OUTPUT_TAGS= 1  4,
+OUTPUT_SENDER  = 1  5,
+OUTPUT_RECIPIENTS  = 1  6,
+OUTPUT_ADDRESSES   = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
 } output_t;
 
 typedef struct {
@@ -220,6 +223,67 @@ do_search_threads (search_options_t *o)
 return 0;
 }
 
+static void
+print_address_list (const search_options_t *o, InternetAddressList *list)
+{
+InternetAddress *address;
+int i;
+
+for (i = 0; i  internet_address_list_length (list); i++) {
+   address = internet_address_list_get_address (list, i);
+   if (INTERNET_ADDRESS_IS_GROUP (address)) {
+   InternetAddressGroup *group;
+   InternetAddressList *group_list;
+
+   group = INTERNET_ADDRESS_GROUP (address);
+   group_list = internet_address_group_get_members (group);
+   if (group_list == NULL)
+   continue;
+
+   print_address_list (o,

Re: [PATCH 02/11] lib: Refactor _notmuch_database_link_message

2014-10-05 Thread Austin Clements

Quoth David Bremner on Oct 05 at  9:45 am:
 Austin Clements acleme...@csail.mit.edu writes:
  +void *local = talloc_new (NULL);
 
 What's the advantage of using a local talloc context here? Is this just
 an optimization?

There are a few allocations that wind up going in to this local
context because of the call to _consume_metadata_thread_id, so it's
more convenient to free this one context on return from
_notmuch_database_link_message than to worry about tracking these
various allocations.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 06/11] lib: Internal support for querying and creating ghost messages

2014-10-05 Thread Austin Clements

Quoth David Bremner on Oct 05 at 10:30 am:
 Austin Clements acleme...@csail.mit.edu writes:
 
  +   message-flags = ~(1  NOTMUCH_MESSAGE_FLAG_GHOST);
 
 What do you think about using bit set / clear / read macros?  I don't
 insist, but I wonder if it would make this part more readable.

I'm used to reading this stuff, so either way is fine with me.  Do we
have bit set / clear / read macros?

  +   else if (*i == Tghost)
  +   message-flags |= (1  NOTMUCH_MESSAGE_FLAG_GHOST);
  +   else
 
 It makes me faintly unhappy to have the prefix hardcoded here.
 Not sure if there is a sensible solution.

I agree, but I also don't want to construct the test string every time
or deconstruct the term string every time.  I could move the T
prefix string to a #define and use that both here and in
BOOLEAN_PREFIX_INTERNAL, but that solution may be worse than the
problem.  What do you think?
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 08/11] lib: Implement upgrade to ghost messages feature

2014-10-05 Thread Austin Clements

Quoth David Bremner on Oct 05 at 10:56 am:
 Austin Clements acleme...@csail.mit.edu writes:
  +if (new_features  NOTMUCH_FEATURE_GHOSTS) {
  +   t_end = db-metadata_keys_end (thread_id_);
  +   for (t = db-metadata_keys_begin (thread_id_); t != t_end; ++t)
  +   ++total;
  +}
 
 It would be nice to have the comment below, or something like it, for
 the loop above.

  /* The ghost message upgrade converts all thread_id_*
   * metadata values into ghost message documents. */
sound good?

  +/* Perform metadata upgrades. */
  +
  +/* Prior to NOTMUCH_FEATURE_GHOSTS, thread IDs for missing
  + * messages were stored as database metadata. Change these to
  + * ghost messages.
  + */
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

37 matches

Mail list logo