v4 of Hex Dump/Restore patches

[Jani Nikula]

Hi David, some minor bikeshedding in patch 4 (can be fixed later too),
and did not look at patches 7, 8, or 9, but otherwise the series LGTM.

BR,
Jani.


On Sat, 08 Dec 2012, david at tethera.net wrote:
> This obsoletes the series at
>  
>  id:1354843607-17980-1-git-send-email-david at tethera.net
>
> One new patch:
>
> [Patch v4 08/10] test/dump-restore: add test for warning/error
>
> Other changes since the last series are as follows:
>
> commit 16ca0f8126fafd49266ffa02534f2e9b73c03027
> Author: David Bremner 
> Date:   Fri Dec 7 20:19:21 2012 -0400
>
> fixup for test/dump-restore
>
> diff --git a/test/dump-restore b/test/dump-restore
> index 2532cbb..b267792 100755
> --- a/test/dump-restore
> +++ b/test/dump-restore
> @@ -212,7 +212,7 @@ test_expect_equal_file EXPECTED OUTPUT
>  
>  test_begin_subtest 'roundtripping random message-ids and tags'
>  
> -${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG}
> +${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
>   --num-messages=100
>  
>   notmuch dump --format=batch-tag| \
>
> commit 9d864bd724bd2d2b26b5d52e3e2f28b4fff2046f
> Author: David Bremner 
> Date:   Fri Dec 7 20:32:50 2012 -0400
>
> fixup for tag-util.c: put * in right palce
>
> diff --git a/tag-util.h b/tag-util.h
> index df05d72..6674674 100644
> --- a/tag-util.h
> +++ b/tag-util.h
> @@ -62,8 +62,8 @@ parse_tag_line (void *ctx, char *line,
>   * ctx is passed to talloc
>   */
>  
> -tag_op_list_t
> -*tag_op_list_create (void *ctx);
> +tag_op_list_t *
> +tag_op_list_create (void *ctx);
>  
>  /*
>   * Add a tag operation (delete iff remove == TRUE) to a list.
>
> commit d2cead797981e8992849e1d96ad95177b42290e2
> Author: David Bremner 
> Date:   Fri Dec 7 20:57:52 2012 -0400
>
> fixup for id:87txrxxyzp.fsf at nikula.org
>
> diff --git a/tag-util.c b/tag-util.c
> index 3d54e9e..a927363 100644
> --- a/tag-util.c
> +++ b/tag-util.c
> @@ -24,9 +24,15 @@ parse_tag_line (void *ctx, char *line,
>  {
>  char *tok = line;
>  size_t tok_len = 0;
> -char *line_for_error = talloc_strdup (ctx, line);
> +char *line_for_error;
>  int ret = 0;
>  
> +line_for_error = talloc_strdup (ctx, line);
> +if (line_for_error == NULL) {
> + fprintf (stderr, "Error: out of memory\n");
> + return -1;
> +}
> +
>  chomp_newline (line);
>  
>  /* remove leading space */
> @@ -58,7 +64,7 @@ parse_tag_line (void *ctx, char *line,
>  
>   /* If tag is terminated by NUL, there's no query string. */
>   if (*(tok + tok_len) == '\0') {
> - fprintf (stderr, "no query string: %s\n", line_for_error);
> + fprintf (stderr, "Warning: no query string: %s\n", line_for_error);
>   ret = 1;
>   goto DONE;
>   }
> @@ -71,19 +77,21 @@ parse_tag_line (void *ctx, char *line,
>  
>   /* Maybe refuse empty tags. */
>   if (! (flags & TAG_FLAG_BE_GENEROUS) && *tag == '\0') {
> - fprintf (stderr, "Error: empty tag: %s\n", line_for_error);
> + fprintf (stderr, "Warning: empty tag: %s\n", line_for_error);
> + ret = 1;
>   goto DONE;
>   }
>  
>   /* Decode tag. */
>   if (hex_decode_inplace (tag) != HEX_SUCCESS) {
> - fprintf (stderr, "Hex decoding of tag %s failed\n",
> + fprintf (stderr, "Warning: Hex decoding of tag %s failed\n",
>tag);
>   ret = 1;
>   goto DONE;
>   }
>  
>   if (tag_op_list_append (ctx, tag_ops, tag, remove)) {
> + /* diagnostics already printed */
>   ret = -1;
>   goto DONE;
>   }
> @@ -98,7 +106,7 @@ parse_tag_line (void *ctx, char *line,
>  
>  /* tok now points to the query string */
>  if (hex_decode_inplace (tok) != HEX_SUCCESS) {
> - fprintf (stderr, "Hex decoding of query %s failed\n",
> + fprintf (stderr, "Warning: Hex decoding of query %s failed\n",
>tok);
>   ret = 1;
>   goto DONE;
>
> commit 162c20ab6d9b9eef56cb2b966a7144a61557eade
> Author: David Bremner 
> Date:   Sat Dec 8 07:29:56 2012 -0400
>
> fixup for id:87624chmfw.fsf at qmul.ac.uk
> 
> update comment
>
> diff --git a/util/string-util.h b/util/string-util.h
> index 696da40..ac7676c 100644
> --- a/util/string-util.h
> +++ b/util/string-util.h
> @@ -3,7 +3,10 @@
>  
>  #include 
>  
> -/* like strtok(3), but without state, and doesn't modify s. usage pattern:
> +/* like strtok(3), but without state, and doesn't modify s.  Return
> + * value is indicated by pointer and length, not null terminator.
> + *
> + * Usage pattern:
>   *
>   * const char *tok = input;
>   * const char *delim = " \t";
>
> commit 590adeec072f235861154b930bfb10bedbe37e8a
> Author: David Bremner 
> Date:   Sat Dec 8 07:42:44 2012 -0400
>
> fixup for id:8738zghl6d.fsf at qmul.ac.uk
>
> diff --git a/tag-util.c b/tag-util.c
> index a927363..c97d240 100644
> --- a/tag-util.c
> +++ b/tag-util.c
> @@ -98,8 +98,6 @@ parse_tag_line 

[Patch v4 04/10] tag-util.[ch]: New files for common tagging routines

[Jani Nikula]

Hi David, some error message bikeshedding below.

BR,
Jani.


On Sat, 08 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> These are meant to be shared between notmuch-tag and notmuch-restore.
>
> The bulk of the routines implement a "tag operation list" abstract
> data type act as a structured representation of a set of tag
> operations (typically coming from a single tag command or line of
> input).
> ---
>  Makefile.local |1 +
>  tag-util.c |  292 
> 
>  tag-util.h |  122 +++
>  3 files changed, 415 insertions(+)
>  create mode 100644 tag-util.c
>  create mode 100644 tag-util.h
>
> diff --git a/Makefile.local b/Makefile.local
> index 0db1713..c274f07 100644
> --- a/Makefile.local
> +++ b/Makefile.local
> @@ -275,6 +275,7 @@ notmuch_client_srcs = \
>   query-string.c  \
>   mime-node.c \
>   crypto.c\
> + tag-util.c
>  
>  notmuch_client_modules = $(notmuch_client_srcs:.c=.o)
>  
> diff --git a/tag-util.c b/tag-util.c
> new file mode 100644
> index 000..80ebdc2
> --- /dev/null
> +++ b/tag-util.c
> @@ -0,0 +1,292 @@
> +#include 
> +#include "string-util.h"
> +#include "tag-util.h"
> +#include "hex-escape.h"
> +
> +#define TAG_OP_LIST_INITIAL_SIZE 10
> +
> +struct _tag_operation_t {
> +const char *tag;
> +notmuch_bool_t remove;
> +};
> +
> +struct _tag_op_list_t {
> +tag_operation_t *ops;
> +size_t count;
> +size_t size;
> +};
> +
> +int
> +parse_tag_line (void *ctx, char *line,
> + tag_op_flag_t flags,
> + char **query_string,
> + tag_op_list_t *tag_ops)
> +{
> +char *tok = line;
> +size_t tok_len = 0;
> +char *line_for_error;
> +int ret = 0;
> +
> +chomp_newline (line);
> +
> +line_for_error = talloc_strdup (ctx, line);
> +if (line_for_error == NULL) {
> + fprintf (stderr, "Error: out of memory\n");
> + return -1;
> +}
> +
> +/* remove leading space */
> +while (*tok == ' ' || *tok == '\t')
> + tok++;
> +
> +/* Skip empty and comment lines. */
> +if (*tok == '\0' || *tok == '#') {
> + ret = 2;
> + goto DONE;
> +}
> +
> +tag_op_list_reset (tag_ops);
> +
> +/* Parse tags. */
> +while ((tok = strtok_len (tok + tok_len, " ", _len)) != NULL) {
> + notmuch_bool_t remove;
> + char *tag;
> +
> + /* Optional explicit end of tags marker. */
> + if (tok_len == 2 && strncmp (tok, "--", tok_len) == 0) {
> + tok = strtok_len (tok + tok_len, " ", _len);
> + if (tok == NULL)
> + fprintf (stderr, "Warning: no query string: %s\n", 
> line_for_error);

I think you should move this warning back... (see below)

> + break;
> + }
> +
> + /* Implicit end of tags. */
> + if (*tok != '-' && *tok != '+')
> + break;
> +
> + /* If tag is terminated by NUL, there's no query string. */
> + if (*(tok + tok_len) == '\0') {
> + fprintf (stderr, "Warning: no query string: %s\n", line_for_error);
> + ret = 1;
> + goto DONE;
> + }
> +
> + /* Terminate, and start next token after terminator. */
> + *(tok + tok_len++) = '\0';
> +
> + remove = (*tok == '-');
> + tag = tok + 1;
> +
> + /* Maybe refuse empty tags. */
> + if (! (flags & TAG_FLAG_BE_GENEROUS) && *tag == '\0') {
> + fprintf (stderr, "Warning: empty tag: %s\n", line_for_error);
> + ret = 1;
> + goto DONE;
> + }
> +
> + /* Decode tag. */
> + if (hex_decode_inplace (tag) != HEX_SUCCESS) {
> + fprintf (stderr, "Warning: Hex decoding of tag %s failed\n",
> +  tag);
> + ret = 1;
> + goto DONE;
> + }
> +
> + if (tag_op_list_append (ctx, tag_ops, tag, remove)) {
> + /* diagnostics already printed */
> + ret = -1;
> + goto DONE;
> + }
> +}
> +
> +if (tok == NULL) {

...here where it was. Now you'll only get the warning for lines like:

+foo +bar --

but not for:

+foo +bar

which are only caught here. (Alternatively, you could have both print
the error message, and set ret and goto DONE also in the first case. But
I don't know if that gains us anything.)

> + ret = 1;
> + goto DONE;
> +}
> +
> +/* tok now points to the query string */
> +if (hex_decode_inplace (tok) != HEX_SUCCESS) {
> + fprintf (stderr, "Warning: Hex decoding of query %s failed\n",
> +  tok);
> + ret = 1;
> + goto DONE;
> +}
> +
> +*query_string = tok;
> +
> +  DONE:
> +if ((ret % 2) != 0)
> + fprintf (stderr, "%s invalid input line %s\n",
> +  ret == 1 ? "Warning: Ignoring" : "Error: Failing at",
> +  line_for_error);

Can't resist urge to nitpick... I think the (ret % 2) is a bit
magical. I'd prefer the explicit (ret != 0 && ret != 2). Another one is
that some of the earlier 

[PATCH] emacs: query: make sync queries use sexp

[Austin Clements]

sexp-at-point is a pretty roundabout way to parse this that I think
will basically parse the S-expression three times (in end-of-sexp, in
beginning-of-sexp, and then for real in read-from-whole-string).  How
about

  (prog1
  (read (current-buffer))
(skip-chars-forward " \t\n\f")
(unless (eobp)
  (error "Trailing garbage after notmuch output")))

?  Technically, just the (read (current-buffer)) part would be
equivalent to the current (json-read) call (which will ignore any
trailing garbage).

There's also another synchronous JSON query in notmuch-mua.el that
should be fixed.

I wouldn't worry too much about the error handling.  I don't think
this can deal worse with errors than the current JSON code does.  My
schema versioning series has a patch [1] that should help improve
error handling.  It's for JSON right now, but will be trivial to port
to S-expressions.

[1] id:1354416002-3557-9-git-send-email-amdragon at mit.edu

Quoth Mark Walters on Dec 08 at  2:11 pm:
> This changes the queries used by notmuch-show from json to sexp (patch
> based on a comment by Tomi on irc as to the trivial change needed).
> 
> The async query parsed used by search is not as easy to convert.
> ---
> 
> It's probably worth making this change: sexps are significantly faster
> but I doubt anyone would notice in show (since the query is small and
> the wash processing etc relatively large).
> 
> At the moment this doesn't do any error fixing. The json version did
> not either but the sexp parser and the json parser might behave
> differently on malformed input.
> 
> Best wishes
> 
> Mark
> 
> 
>  emacs/notmuch-query.el |7 ++-
>  1 files changed, 2 insertions(+), 5 deletions(-)
> 
> diff --git a/emacs/notmuch-query.el b/emacs/notmuch-query.el
> index d66baea..0ee6cca 100644
> --- a/emacs/notmuch-query.el
> +++ b/emacs/notmuch-query.el
> @@ -29,10 +29,7 @@ A thread is a forest or list of trees. A tree is a two 
> element
>  list where the first element is a message, and the second element
>  is a possibly empty forest of replies.
>  "
> -  (let  ((args '("show" "--format=json"))
> -  (json-object-type 'plist)
> -  (json-array-type 'list)
> -  (json-false 'nil))
> +  (let  ((args '("show" "--format=sexp")))
>  (if notmuch-show-process-crypto
>   (setq args (append args '("--decrypt"
>  (setq args (append args search-terms))
> @@ -40,7 +37,7 @@ is a possibly empty forest of replies.
>(progn
>   (apply 'call-process (append (list notmuch-command nil (list t nil) 
> nil) args))
>   (goto-char (point-min))
> - (json-read)
> + (sexp-at-point)
>  
>  ;;
>  ;; Mapping functions across collections of messages.

v4 of Hex Dump/Restore patches

[Mark Walters]

Aside from the minor nit with one comment this series looks good to
me. (I haven't thoroughly checked patch 8)

Best wishes

Mark

On Sat, 08 Dec 2012, david at tethera.net wrote:
> This obsoletes the series at
>  
>  id:1354843607-17980-1-git-send-email-david at tethera.net
>
> One new patch:
>
> [Patch v4 08/10] test/dump-restore: add test for warning/error
>
> Other changes since the last series are as follows:
>
> commit 16ca0f8126fafd49266ffa02534f2e9b73c03027
> Author: David Bremner 
> Date:   Fri Dec 7 20:19:21 2012 -0400
>
> fixup for test/dump-restore
>
> diff --git a/test/dump-restore b/test/dump-restore
> index 2532cbb..b267792 100755
> --- a/test/dump-restore
> +++ b/test/dump-restore
> @@ -212,7 +212,7 @@ test_expect_equal_file EXPECTED OUTPUT
>  
>  test_begin_subtest 'roundtripping random message-ids and tags'
>  
> -${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG}
> +${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
>   --num-messages=100
>  
>   notmuch dump --format=batch-tag| \
>
> commit 9d864bd724bd2d2b26b5d52e3e2f28b4fff2046f
> Author: David Bremner 
> Date:   Fri Dec 7 20:32:50 2012 -0400
>
> fixup for tag-util.c: put * in right palce
>
> diff --git a/tag-util.h b/tag-util.h
> index df05d72..6674674 100644
> --- a/tag-util.h
> +++ b/tag-util.h
> @@ -62,8 +62,8 @@ parse_tag_line (void *ctx, char *line,
>   * ctx is passed to talloc
>   */
>  
> -tag_op_list_t
> -*tag_op_list_create (void *ctx);
> +tag_op_list_t *
> +tag_op_list_create (void *ctx);
>  
>  /*
>   * Add a tag operation (delete iff remove == TRUE) to a list.
>
> commit d2cead797981e8992849e1d96ad95177b42290e2
> Author: David Bremner 
> Date:   Fri Dec 7 20:57:52 2012 -0400
>
> fixup for id:87txrxxyzp.fsf at nikula.org
>
> diff --git a/tag-util.c b/tag-util.c
> index 3d54e9e..a927363 100644
> --- a/tag-util.c
> +++ b/tag-util.c
> @@ -24,9 +24,15 @@ parse_tag_line (void *ctx, char *line,
>  {
>  char *tok = line;
>  size_t tok_len = 0;
> -char *line_for_error = talloc_strdup (ctx, line);
> +char *line_for_error;
>  int ret = 0;
>  
> +line_for_error = talloc_strdup (ctx, line);
> +if (line_for_error == NULL) {
> + fprintf (stderr, "Error: out of memory\n");
> + return -1;
> +}
> +
>  chomp_newline (line);
>  
>  /* remove leading space */
> @@ -58,7 +64,7 @@ parse_tag_line (void *ctx, char *line,
>  
>   /* If tag is terminated by NUL, there's no query string. */
>   if (*(tok + tok_len) == '\0') {
> - fprintf (stderr, "no query string: %s\n", line_for_error);
> + fprintf (stderr, "Warning: no query string: %s\n", line_for_error);
>   ret = 1;
>   goto DONE;
>   }
> @@ -71,19 +77,21 @@ parse_tag_line (void *ctx, char *line,
>  
>   /* Maybe refuse empty tags. */
>   if (! (flags & TAG_FLAG_BE_GENEROUS) && *tag == '\0') {
> - fprintf (stderr, "Error: empty tag: %s\n", line_for_error);
> + fprintf (stderr, "Warning: empty tag: %s\n", line_for_error);
> + ret = 1;
>   goto DONE;
>   }
>  
>   /* Decode tag. */
>   if (hex_decode_inplace (tag) != HEX_SUCCESS) {
> - fprintf (stderr, "Hex decoding of tag %s failed\n",
> + fprintf (stderr, "Warning: Hex decoding of tag %s failed\n",
>tag);
>   ret = 1;
>   goto DONE;
>   }
>  
>   if (tag_op_list_append (ctx, tag_ops, tag, remove)) {
> + /* diagnostics already printed */
>   ret = -1;
>   goto DONE;
>   }
> @@ -98,7 +106,7 @@ parse_tag_line (void *ctx, char *line,
>  
>  /* tok now points to the query string */
>  if (hex_decode_inplace (tok) != HEX_SUCCESS) {
> - fprintf (stderr, "Hex decoding of query %s failed\n",
> + fprintf (stderr, "Warning: Hex decoding of query %s failed\n",
>tok);
>   ret = 1;
>   goto DONE;
>
> commit 162c20ab6d9b9eef56cb2b966a7144a61557eade
> Author: David Bremner 
> Date:   Sat Dec 8 07:29:56 2012 -0400
>
> fixup for id:87624chmfw.fsf at qmul.ac.uk
> 
> update comment
>
> diff --git a/util/string-util.h b/util/string-util.h
> index 696da40..ac7676c 100644
> --- a/util/string-util.h
> +++ b/util/string-util.h
> @@ -3,7 +3,10 @@
>  
>  #include 
>  
> -/* like strtok(3), but without state, and doesn't modify s. usage pattern:
> +/* like strtok(3), but without state, and doesn't modify s.  Return
> + * value is indicated by pointer and length, not null terminator.
> + *
> + * Usage pattern:
>   *
>   * const char *tok = input;
>   * const char *delim = " \t";
>
> commit 590adeec072f235861154b930bfb10bedbe37e8a
> Author: David Bremner 
> Date:   Sat Dec 8 07:42:44 2012 -0400
>
> fixup for id:8738zghl6d.fsf at qmul.ac.uk
>
> diff --git a/tag-util.c b/tag-util.c
> index a927363..c97d240 100644
> --- a/tag-util.c
> +++ b/tag-util.c
> @@ -98,8 +98,6 @@ parse_tag_line (void *ctx, char *line,
>   

[Patch v5 11/11] tag-util: optimization of tag application

[da...@tethera.net]

From: David Bremner 

The idea is not to bother with restore operations if they don't change
the set of tags. This is actually a relatively common case.

In order to avoid fancy datastructures, this method is quadratic in
the number of tags; at least on my mail database this doesn't seem to
be a big problem.
---
 tag-util.c |   68 
 1 file changed, 68 insertions(+)

diff --git a/tag-util.c b/tag-util.c
index 164fdf0..e7233ab 100644
--- a/tag-util.c
+++ b/tag-util.c
@@ -151,6 +151,71 @@ message_error (notmuch_message_t *message,
 fprintf (stderr, "Status: %s\n", notmuch_status_to_string (status));
 }

+static int
+makes_changes (notmuch_message_t *message,
+  tag_op_list_t *list,
+  tag_op_flag_t flags)
+{
+
+size_t i;
+
+notmuch_tags_t *tags;
+notmuch_bool_t changes = FALSE;
+
+/* First, do we delete an existing tag? */
+changes = FALSE;
+for (tags = notmuch_message_get_tags (message);
+! changes && notmuch_tags_valid (tags);
+notmuch_tags_move_to_next (tags)) {
+   const char *cur_tag = notmuch_tags_get (tags);
+   int last_op =  (flags & TAG_FLAG_REMOVE_ALL) ? -1 : 0;
+
+   /* scan backwards to get last operation */
+   i = list->count;
+   while (i > 0) {
+   i--;
+   if (strcmp (cur_tag, list->ops[i].tag) == 0) {
+   last_op = list->ops[i].remove ? -1 : 1;
+   break;
+   }
+   }
+
+   changes = (last_op == -1);
+}
+notmuch_tags_destroy (tags);
+
+if (changes)
+   return TRUE;
+
+/* Now check for adding new tags */
+for (i = 0; i < list->count; i++) {
+   notmuch_bool_t exists = FALSE;
+
+   if (list->ops[i].remove)
+   continue;
+
+   for (tags = notmuch_message_get_tags (message);
+notmuch_tags_valid (tags);
+notmuch_tags_move_to_next (tags)) {
+   const char *cur_tag = notmuch_tags_get (tags);
+   if (strcmp (cur_tag, list->ops[i].tag) == 0) {
+   exists = TRUE;
+   break;
+   }
+   }
+   notmuch_tags_destroy (tags);
+
+   /* the following test is conservative,
+* in the sense it ignores cases like +foo ... -foo
+* but this is OK from a correctness point of view
+*/
+   if (! exists)
+   return TRUE;
+}
+return FALSE;
+
+}
+
 notmuch_status_t
 tag_op_list_apply (notmuch_message_t *message,
   tag_op_list_t *list,
@@ -160,6 +225,9 @@ tag_op_list_apply (notmuch_message_t *message,
 notmuch_status_t status = 0;
 tag_operation_t *tag_ops = list->ops;

+if (! (flags & TAG_FLAG_PRE_OPTIMIZED) && ! makes_changes (message, list, 
flags))
+   return NOTMUCH_STATUS_SUCCESS;
+
 status = notmuch_message_freeze (message);
 if (status) {
message_error (message, status, "freezing message");
-- 
1.7.10.4

[Patch v5 10/11] notmuch-{dump, restore}.1: document new format options

[da...@tethera.net]

From: David Bremner 

More or less arbitrarily, notmuch-dump.1 gets the more detailed
description of the format.
---
 man/man1/notmuch-dump.1|   59 
 man/man1/notmuch-restore.1 |   59 +++-
 2 files changed, 112 insertions(+), 6 deletions(-)

diff --git a/man/man1/notmuch-dump.1 b/man/man1/notmuch-dump.1
index 230deec..770b00f 100644
--- a/man/man1/notmuch-dump.1
+++ b/man/man1/notmuch-dump.1
@@ -5,6 +5,7 @@ notmuch-dump \- creates a plain-text dump of the tags of each 
message
 .SH SYNOPSIS

 .B "notmuch dump"
+.RB  [ "\-\-format=(sup|batch-tag)"  "] [--]"
 .RI "[ --output=<" filename "> ] [--]"
 .RI "[ <" search-term ">...]"

@@ -19,6 +20,64 @@ recreated from the messages themselves.  The output of 
notmuch dump is
 therefore the only critical thing to backup (and much more friendly to
 incremental backup than the native database files.)

+.TP 4
+.B \-\-format=(sup|batch-tag)
+
+Notmuch restore supports two plain text dump formats, both with one message-id
+per line, followed by a list of tags.
+
+.RS 4
+.TP 4
+.B sup
+
+The
+.B sup
+dump file format is specifically chosen to be
+compatible with the format of files produced by sup-dump.
+So if you've previously been using sup for mail, then the
+.B "notmuch restore"
+command provides you a way to import all of your tags (or labels as
+sup calls them).
+Each line has the following form
+
+.RS 4
+.RI < message-id >
+.B (
+.RI < tag "> ..."
+.B )
+
+with zero or more tags are separated by spaces. Note that (malformed)
+message-ids may contain arbitrary non-null characters. Note also
+that tags with spaces will not be correctly restored with this format.
+
+.RE
+
+.RE
+.RS 4
+.TP 4
+.B batch-tag
+
+The
+.B batch-tag
+dump format is intended to more robust against malformed message-ids
+and tags containing whitespace or non-\fBascii\fR(7) characters.
+Each line has the form
+
+.RS 4
+.RI "+<" "encoded-tag" "> " "" "+<" "encoded-tag" "> ... -- " "" " id:<" 
encoded-message-id >
+
+where encoded means that every byte not matching the regex
+.B [A-Za-z0-9@=.,_+-]
+is replace by
+.B %nn
+where nn is the two digit hex encoding.
+The astute reader will notice this is a special case of the batch input
+format for \fBnotmuch-tag\fR(1); note that the single message-id query is
+mandatory for \fBnotmuch-restore\fR(1).
+
+.RE
+
+
 With no search terms, a dump of all messages in the database will be
 generated.  A "--" argument instructs notmuch that the
 remaining arguments are search terms.
diff --git a/man/man1/notmuch-restore.1 b/man/man1/notmuch-restore.1
index 2fa8733..6bba628 100644
--- a/man/man1/notmuch-restore.1
+++ b/man/man1/notmuch-restore.1
@@ -6,6 +6,7 @@ notmuch-restore \- restores the tags from the given file (see 
notmuch dump)

 .B "notmuch restore"
 .RB [ "--accumulate" ]
+.RB [ "--format=(auto|batch-tag|sup)" ]
 .RI "[ --input=<" filename "> ]"

 .SH DESCRIPTION
@@ -15,19 +16,51 @@ Restores the tags from the given file (see

 The input is read from the given filename, if any, or from stdin.

-Note: The dump file format is specifically chosen to be
+
+Supported options for
+.B restore
+include
+.RS 4
+.TP 4
+.B \-\-accumulate
+
+The union of the existing and new tags is applied, instead of
+replacing each message's tags as they are read in from the dump file.
+
+.RE
+.RS 4
+.TP 4
+.B \-\-format=(sup|batch-tag|auto)
+
+Notmuch restore supports two plain text dump formats, with each line
+specifying a message-id and a set of tags.
+For details of the actual formats, see \fBnotmuch-dump\fR(1).
+
+.RS 4
+.TP 4
+.B sup
+
+The
+.B sup
+dump file format is specifically chosen to be
 compatible with the format of files produced by sup-dump.
 So if you've previously been using sup for mail, then the
 .B "notmuch restore"
 command provides you a way to import all of your tags (or labels as
 sup calls them).

-The --accumulate switch causes the union of the existing and new tags to be
-applied, instead of replacing each message's tags as they are read in from the
-dump file.
+.RE
+.RS 4
+.TP 4
+.B batch-tag

-See \fBnotmuch-search-terms\fR(7)
-for details of the supported syntax for .
+The
+.B batch-tag
+dump format is intended to more robust against malformed message-ids
+and tags containing whitespace or non-\fBascii\fR(7) characters.  This
+format hex-escapes all characters those outside of a small character
+set, intended to be suitable for e.g. pathnames in most UNIX-like
+systems.

 .B "notmuch restore"
 updates the maildir flags according to tag changes if the
@@ -36,6 +69,20 @@ configuration option is enabled. See \fBnotmuch-config\fR(1) 
for
 details.

 .RE
+
+.RS 4
+.TP 4
+.B auto
+
+This option (the default) tries to guess the format from the
+input. For correctly formed input in either supported format, this
+heuristic, based the fact that batch-tag format contains no parentheses,
+should be accurate.
+
+.RE
+
+.RE
+
 .SH SEE ALSO

 

[Patch v5 09/11] test/dump-restore: add test for warning/error messages

[da...@tethera.net]

From: David Bremner 

We want to test both that error/warning messages are generated when
they should be, and not generated when they should not be. This varies
between restore and batch tagging.
---
 test/dump-restore |   35 +++
 1 file changed, 35 insertions(+)

diff --git a/test/dump-restore b/test/dump-restore
index 7d91cca..6a989b6 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -181,6 +181,41 @@ notmuch restore < EXPECTED.$test_count
 notmuch dump --format=sup > OUTPUT.$test_count
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count

+test_begin_subtest 'restore: checking error messages'
+notmuch restore 

[Patch v5 08/11] test: second set of dump/restore --format=batch-tag tests

[da...@tethera.net]

From: David Bremner 

These one need the completed functionality in notmuch-restore. Fairly
exotic tags are tested, but no weird message id's.

We test each possible input to autodetection, both explicit (with
--format=auto) and implicit (without --format).
---
 test/dump-restore |   83 +
 1 file changed, 83 insertions(+)

diff --git a/test/dump-restore b/test/dump-restore
index c700aac..7d91cca 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -98,6 +98,89 @@ notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- 
id://' | \
 sort > OUTPUT.$test_count
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count

+test_begin_subtest "format=batch-tag, # round-trip"
+notmuch dump --format=sup | sort > EXPECTED.$test_count
+notmuch dump --format=batch-tag | notmuch restore --format=batch-tag
+notmuch dump --format=sup | sort > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest "format=batch-tag, # blank lines and comments"
+notmuch dump --format=batch-tag| sort > EXPECTED.$test_count
+notmuch restore < OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest "format=batch-tag, # reverse-round-trip empty tag"
+cat  
OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+tag1='comic_swear=$&^%$^%\\//-+$^%$'
+enc1=$($TEST_DIRECTORY/hex-xcode --direction=encode "$tag1")
+
+tag2=$(printf 'this\n tag\t has\n spaces')
+enc2=$($TEST_DIRECTORY/hex-xcode --direction=encode "$tag2")
+
+enc3='%c3%91%c3%a5%c3%b0%c3%a3%c3%a5%c3%a9-%c3%8f%c3%8a'
+tag3=$($TEST_DIRECTORY/hex-xcode --direction=decode $enc3)
+
+notmuch dump --format=batch-tag > BACKUP
+
+notmuch tag +"$tag1" +"$tag2" +"$tag3" -inbox -unread "*"
+
+test_begin_subtest 'format=batch-tag, round trip with strange tags'
+notmuch dump --format=batch-tag > EXPECTED.$test_count
+notmuch dump --format=batch-tag | notmuch restore --format=batch-tag
+notmuch dump --format=batch-tag > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'format=batch-tag, checking encoded output'
+notmuch dump --format=batch-tag -- from:cworth |\
+awk "{ print \"+$enc1 +$enc2 +$enc3 -- \" \$5 }" > EXPECTED.$test_count
+notmuch dump --format=batch-tag -- from:cworth  > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'restoring sane tags'
+notmuch restore --format=batch-tag < BACKUP
+notmuch dump --format=batch-tag > OUTPUT.$test_count
+test_expect_equal_file BACKUP OUTPUT.$test_count
+
+test_begin_subtest 'format=batch-tag, restore=auto'
+notmuch dump --format=batch-tag > EXPECTED.$test_count
+notmuch tag -inbox -unread "*"
+notmuch restore --format=auto < EXPECTED.$test_count
+notmuch dump --format=batch-tag > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'format=sup, restore=auto'
+notmuch dump --format=sup > EXPECTED.$test_count
+notmuch tag -inbox -unread "*"
+notmuch restore --format=auto < EXPECTED.$test_count
+notmuch dump --format=sup > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'format=batch-tag, restore=default'
+notmuch dump --format=batch-tag > EXPECTED.$test_count
+notmuch tag -inbox -unread "*"
+notmuch restore < EXPECTED.$test_count
+notmuch dump --format=batch-tag > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'format=sup, restore=default'
+notmuch dump --format=sup > EXPECTED.$test_count
+notmuch tag -inbox -unread "*"
+notmuch restore < EXPECTED.$test_count
+notmuch dump --format=sup > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
 test_begin_subtest 'roundtripping random message-ids and tags'

 ${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
-- 
1.7.10.4

[Patch v5 07/11] test: update dump-restore roundtripping test for batch-tag format

[da...@tethera.net]

From: David Bremner 

Now we can actually round trip these crazy tags and and message ids.
hex-xcode is no longer needed as it's built in.
---
 test/dump-restore |   15 +++
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/test/dump-restore b/test/dump-restore
index b4c807f..c700aac 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -99,23 +99,22 @@ notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- 
id://' | \
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count

 test_begin_subtest 'roundtripping random message-ids and tags'
-test_subtest_known_broken
+
 ${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
-   --num-messages=10
+   --num-messages=100

- notmuch dump| \
-${TEST_DIRECTORY}/hex-xcode --direction=encode| \
+ notmuch dump --format=batch-tag| \
 sort > EXPECTED.$test_count

  notmuch tag +this_tag_is_very_unlikely_to_be_random '*'

- ${TEST_DIRECTORY}/hex-xcode --direction=decode < EXPECTED.$test_count | \
-notmuch restore 2>/dev/null
+ notmuch restore --format=batch-tag < EXPECTED.$test_count

- notmuch dump| \
-${TEST_DIRECTORY}/hex-xcode --direction=encode| \
+ notmuch dump --format=batch-tag| \
 sort > OUTPUT.$test_count

 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count

 test_done
+
+# Note the database is "poisoned" for sup format at this point.
-- 
1.7.10.4

[Patch v5 06/11] notmuch-restore: normalize case of error messages.

[da...@tethera.net]

From: David Bremner 

In English, (unlike German) one does not capitalize the first word
after a colon.
---
 notmuch-restore.c |4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/notmuch-restore.c b/notmuch-restore.c
index 44bf88d..dba882b 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -39,7 +39,7 @@ tag_message (unused (void *ctx),

 status = notmuch_database_find_message (notmuch, message_id, );
 if (status || message == NULL) {
-   fprintf (stderr, "Warning: Cannot apply tags to %smessage: %s\n",
+   fprintf (stderr, "Warning: cannot apply tags to %smessage: %s\n",
 message ? "" : "missing ", message_id);
if (status)
fprintf (stderr, "%s\n", notmuch_status_to_string (status));
@@ -214,7 +214,7 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])

if (ret == 0) {
if (strncmp ("id:", query_string, 3) != 0) {
-   fprintf (stderr, "Unsupported query: %s\n", query_string);
+   fprintf (stderr, "Warning: unsupported query: %s\n", 
query_string);
continue;
}
/* delete id: from front of string; tag_message
-- 
1.7.10.4

[Patch v5 05/11] notmuch-restore: add support for input format 'batch-tag'

[da...@tethera.net]

From: David Bremner 

This can be enabled with the new --format=batch-tag command line
option to "notmuch restore". The input must consist of lines of the
format:

+|- [...] [--] id:

Each line is interpreted similarly to "notmuch tag" command line
arguments. The delimiter is one or more spaces ' '. Any characters in
 and  MAY be hex encoded with %NN where NN is the
hexadecimal value of the character. Any ' ' and '%' characters in
 and  MUST be hex encoded (using %20 and %25,
respectively). Any characters that are not part of  or
 MUST NOT be hex encoded.

Leading and trailing space ' ' is ignored. Empty lines and lines
beginning with '#' are ignored.

Commit message mainly stolen from Jani's batch tagging commit, to
follow.
---
 notmuch-restore.c |  220 +
 1 file changed, 138 insertions(+), 82 deletions(-)

diff --git a/notmuch-restore.c b/notmuch-restore.c
index f03dcac..44bf88d 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -19,18 +19,22 @@
  */

 #include "notmuch-client.h"
+#include "dump-restore-private.h"
+#include "tag-util.h"
+#include "string-util.h"
+
+static volatile sig_atomic_t interrupted;
+static regex_t regex;

 static int
-tag_message (notmuch_database_t *notmuch, const char *message_id,
-char *file_tags, notmuch_bool_t remove_all,
-notmuch_bool_t synchronize_flags)
+tag_message (unused (void *ctx),
+notmuch_database_t *notmuch,
+const char *message_id,
+tag_op_list_t *tag_ops,
+tag_op_flag_t flags)
 {
 notmuch_status_t status;
-notmuch_tags_t *db_tags;
-char *db_tags_str;
 notmuch_message_t *message = NULL;
-const char *tag;
-char *next;
 int ret = 0;

 status = notmuch_database_find_message (notmuch, message_id, );
@@ -44,55 +48,67 @@ tag_message (notmuch_database_t *notmuch, const char 
*message_id,

 /* In order to detect missing messages, this check/optimization is
  * intentionally done *after* first finding the message. */
-if (! remove_all && (file_tags == NULL || *file_tags == '\0'))
-   goto DONE;
-
-db_tags_str = NULL;
-for (db_tags = notmuch_message_get_tags (message);
-notmuch_tags_valid (db_tags);
-notmuch_tags_move_to_next (db_tags)) {
-   tag = notmuch_tags_get (db_tags);
-
-   if (db_tags_str)
-   db_tags_str = talloc_asprintf_append (db_tags_str, " %s", tag);
-   else
-   db_tags_str = talloc_strdup (message, tag);
-}
+if ((flags & TAG_FLAG_REMOVE_ALL) || tag_op_list_size (tag_ops))
+   tag_op_list_apply (message, tag_ops, flags);

-if (((file_tags == NULL || *file_tags == '\0') &&
-(db_tags_str == NULL || *db_tags_str == '\0')) ||
-   (file_tags && db_tags_str && strcmp (file_tags, db_tags_str) == 0))
-   goto DONE;
+notmuch_message_destroy (message);

-notmuch_message_freeze (message);
+return ret;
+}

-if (remove_all)
-   notmuch_message_remove_all_tags (message);
+/* Sup dump output is one line per message. We match a sequence of
+ * non-space characters for the message-id, then one or more
+ * spaces, then a list of space-separated tags as a sequence of
+ * characters within literal '(' and ')'. */

-next = file_tags;
-while (next) {
-   tag = strsep (, " ");
-   if (*tag == '\0')
-   continue;
-   status = notmuch_message_add_tag (message, tag);
-   if (status) {
-   fprintf (stderr, "Error applying tag %s to message %s:\n",
-tag, message_id);
-   fprintf (stderr, "%s\n", notmuch_status_to_string (status));
-   ret = 1;
-   }
+static int
+parse_sup_line (void *ctx, char *line,
+   char **query_str, tag_op_list_t *tag_ops)
+{
+
+regmatch_t match[3];
+char *file_tags;
+int rerr;
+
+tag_op_list_reset (tag_ops);
+
+chomp_newline (line);
+
+/* Silently ignore blank lines */
+if (line[0] == '\0') {
+   return 1;
 }

-notmuch_message_thaw (message);
+rerr = xregexec (, line, 3, match, 0);
+if (rerr == REG_NOMATCH) {
+   fprintf (stderr, "Warning: Ignoring invalid sup format line: %s\n",
+line);
+   return 1;
+}

-if (synchronize_flags)
-   notmuch_message_tags_to_maildir_flags (message);
+*query_str = talloc_strndup (ctx, line + match[1].rm_so,
+match[1].rm_eo - match[1].rm_so);
+file_tags = talloc_strndup (ctx, line + match[2].rm_so,
+   match[2].rm_eo - match[2].rm_so);

-  DONE:
-if (message)
-   notmuch_message_destroy (message);
+char *tok = file_tags;
+size_t tok_len = 0;
+
+tag_op_list_reset (tag_ops);
+
+while ((tok = strtok_len (tok + tok_len, " ", _len)) != NULL) {
+
+   if (*(tok + tok_len) != '\0') {
+   *(tok + tok_len) = '\0';
+   tok_len++;
+   }
+
+   if 

[Patch v5 04/11] tag-util.[ch]: New files for common tagging routines

[da...@tethera.net]

From: David Bremner 

These are meant to be shared between notmuch-tag and notmuch-restore.

The bulk of the routines implement a "tag operation list" abstract
data type act as a structured representation of a set of tag
operations (typically coming from a single tag command or line of
input).
---
 Makefile.local |1 +
 tag-util.c |  305 
 tag-util.h |  135 +
 3 files changed, 441 insertions(+)
 create mode 100644 tag-util.c
 create mode 100644 tag-util.h

diff --git a/Makefile.local b/Makefile.local
index 0db1713..c274f07 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -275,6 +275,7 @@ notmuch_client_srcs =   \
query-string.c  \
mime-node.c \
crypto.c\
+   tag-util.c

 notmuch_client_modules = $(notmuch_client_srcs:.c=.o)

diff --git a/tag-util.c b/tag-util.c
new file mode 100644
index 000..164fdf0
--- /dev/null
+++ b/tag-util.c
@@ -0,0 +1,305 @@
+#include 
+#include "string-util.h"
+#include "tag-util.h"
+#include "hex-escape.h"
+
+#define TAG_OP_LIST_INITIAL_SIZE 10
+
+struct _tag_operation_t {
+const char *tag;
+notmuch_bool_t remove;
+};
+
+struct _tag_op_list_t {
+tag_operation_t *ops;
+size_t count;
+size_t size;
+};
+
+static int
+line_error (tag_parse_status_t status,
+   const char *line,
+   const char *format, ...)
+{
+va_list va_args;
+
+va_start (va_args, format);
+
+fprintf (stderr, status < 0 ? "Error: " : "Warning: ");
+vfprintf (stderr, format, va_args);
+fprintf (stderr, " [%s]\n", line);
+return status;
+}
+
+int
+parse_tag_line (void *ctx, char *line,
+   tag_op_flag_t flags,
+   char **query_string,
+   tag_op_list_t *tag_ops)
+{
+char *tok = line;
+size_t tok_len = 0;
+char *line_for_error;
+int ret = 0;
+
+chomp_newline (line);
+
+line_for_error = talloc_strdup (ctx, line);
+if (line_for_error == NULL) {
+   fprintf (stderr, "Error: out of memory\n");
+   return -1;
+}
+
+/* remove leading space */
+while (*tok == ' ' || *tok == '\t')
+   tok++;
+
+/* Skip empty and comment lines. */
+if (*tok == '\0' || *tok == '#') {
+   ret = 2;
+   goto DONE;
+}
+
+tag_op_list_reset (tag_ops);
+
+/* Parse tags. */
+while ((tok = strtok_len (tok + tok_len, " ", _len)) != NULL) {
+   notmuch_bool_t remove;
+   char *tag;
+
+   /* Optional explicit end of tags marker. */
+   if (tok_len == 2 && strncmp (tok, "--", tok_len) == 0) {
+   tok = strtok_len (tok + tok_len, " ", _len);
+   if (tok == NULL) {
+   ret = line_error (TAG_PARSE_INVALID, line_for_error,
+ "no query string after --");
+   goto DONE;
+   }
+   break;
+   }
+
+   /* Implicit end of tags. */
+   if (*tok != '-' && *tok != '+')
+   break;
+
+   /* If tag is terminated by NUL, there's no query string. */
+   if (*(tok + tok_len) == '\0') {
+   ret = line_error (TAG_PARSE_INVALID, line_for_error,
+ "no query string");
+   goto DONE;
+   }
+
+   /* Terminate, and start next token after terminator. */
+   *(tok + tok_len++) = '\0';
+
+   remove = (*tok == '-');
+   tag = tok + 1;
+
+   /* Maybe refuse empty tags. */
+   if (! (flags & TAG_FLAG_BE_GENEROUS) && *tag == '\0') {
+   ret = line_error (TAG_PARSE_INVALID, line_for_error,
+ "empty tag");
+   goto DONE;
+   }
+
+   /* Decode tag. */
+   if (hex_decode_inplace (tag) != HEX_SUCCESS) {
+   ret = line_error (TAG_PARSE_INVALID, line_for_error,
+ "hex decoding of tag %s failed", tag);
+   goto DONE;
+   }
+
+   if (tag_op_list_append (ctx, tag_ops, tag, remove)) {
+   ret = line_error (TAG_PARSE_OUT_OF_MEMORY, line_for_error,
+ "aborting");
+   goto DONE;
+   }
+}
+
+if (tok == NULL) {
+   /* use a different error message for testing */
+   ret = line_error (TAG_PARSE_INVALID, line_for_error,
+ "missing query string");
+   goto DONE;
+}
+
+/* tok now points to the query string */
+if (hex_decode_inplace (tok) != HEX_SUCCESS) {
+   ret = line_error (TAG_PARSE_INVALID, line_for_error,
+ "hex decoding of query %s failed", tok);
+   goto DONE;
+}
+
+*query_string = tok;
+
+  DONE:
+talloc_free (line_for_error);
+return ret;
+}
+
+static inline void
+message_error (notmuch_message_t *message,
+  notmuch_status_t status,
+  const char *format, ...)
+{
+va_list va_args;
+
+va_start (va_args, format);
+
+vfprintf (stderr, format, 

[Patch v5 03/11] util: add string-util.[ch]

[da...@tethera.net]

From: David Bremner 

This is to give a home to strtok_len. It's a bit silly to add a header
for one routine, but it needs to be shared between several compilation
units (or at least that's the most natural design).
---
 util/Makefile.local |3 ++-
 util/string-util.c  |   34 ++
 util/string-util.h  |   22 ++
 3 files changed, 58 insertions(+), 1 deletion(-)
 create mode 100644 util/string-util.c
 create mode 100644 util/string-util.h

diff --git a/util/Makefile.local b/util/Makefile.local
index 3ca623e..a11e35b 100644
--- a/util/Makefile.local
+++ b/util/Makefile.local
@@ -3,7 +3,8 @@
 dir := util
 extra_cflags += -I$(srcdir)/$(dir)

-libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c
+libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c \
+ $(dir)/string-util.c

 libutil_modules := $(libutil_c_srcs:.c=.o)

diff --git a/util/string-util.c b/util/string-util.c
new file mode 100644
index 000..44f8cd3
--- /dev/null
+++ b/util/string-util.c
@@ -0,0 +1,34 @@
+/* string-util.c -  Extra or enhanced routines for null terminated strings.
+ *
+ * Copyright (c) 2012 Jani Nikula
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Jani Nikula 
+ */
+
+
+#include "string-util.h"
+
+char *
+strtok_len (char *s, const char *delim, size_t *len)
+{
+/* skip initial delims */
+s += strspn (s, delim);
+
+/* length of token */
+*len = strcspn (s, delim);
+
+return *len ? s : NULL;
+}
diff --git a/util/string-util.h b/util/string-util.h
new file mode 100644
index 000..ac7676c
--- /dev/null
+++ b/util/string-util.h
@@ -0,0 +1,22 @@
+#ifndef _STRING_UTIL_H
+#define _STRING_UTIL_H
+
+#include 
+
+/* like strtok(3), but without state, and doesn't modify s.  Return
+ * value is indicated by pointer and length, not null terminator.
+ *
+ * Usage pattern:
+ *
+ * const char *tok = input;
+ * const char *delim = " \t";
+ * size_t tok_len = 0;
+ *
+ * while ((tok = strtok_len (tok + tok_len, delim, _len)) != NULL) {
+ * // do stuff with string tok of length tok_len
+ * }
+ */
+
+char *strtok_len (char *s, const char *delim, size_t *len);
+
+#endif
-- 
1.7.10.4

[Patch v5 02/11] test: add sanity check for dump --format=batch-tag.

[da...@tethera.net]

From: David Bremner 

It's important this does not rely on restore, since it hasn't been
written yet.
---
 test/dump-restore |   13 +
 1 file changed, 13 insertions(+)

diff --git a/test/dump-restore b/test/dump-restore
index bf31266..b4c807f 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -85,6 +85,19 @@ test_begin_subtest "dump --output=outfile -- from:cworth"
 notmuch dump --output=dump-outfile-dash-inbox.actual -- from:cworth
 test_expect_equal_file dump-cworth.expected dump-outfile-dash-inbox.actual

+test_begin_subtest "Check for a safe set of message-ids"
+notmuch search --output=messages from:cworth | sed s/^id:// > EXPECTED
+notmuch search --output=messages from:cworth | sed s/^id:// |\
+   $TEST_DIRECTORY/hex-xcode --direction=encode > OUTPUT
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest "format=batch-tag, dump sanity check."
+notmuch dump --format=sup from:cworth | cut -f1 -d' ' | \
+sort > EXPECTED.$test_count
+notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
+sort > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
 test_begin_subtest 'roundtripping random message-ids and tags'
 test_subtest_known_broken
 ${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
-- 
1.7.10.4

[Patch v5 01/11] notmuch-dump: add --format=(batch-tag|sup)

[da...@tethera.net]

From: David Bremner 

sup is the old format, and remains the default, at least until
restore is converted to parse this format.

Each line of the batch-tag format is modelled on the syntax of notmuch tag:
- "notmuch tag" is omitted from the front of the line
- The dump format only uses query strings of a single message-id.
- Each space seperated tag/message-id is 'hex-encoded' to remove
  trouble-making characters.
- It is permitted (and will be useful) for there to be no tags before
  the query.

In particular this format won't have the same problem with e.g. spaces
in message-ids or tags; they will be round-trip-able.
---
 dump-restore-private.h |   13 +
 notmuch-dump.c |   48 ++--
 2 files changed, 55 insertions(+), 6 deletions(-)
 create mode 100644 dump-restore-private.h

diff --git a/dump-restore-private.h b/dump-restore-private.h
new file mode 100644
index 000..896a004
--- /dev/null
+++ b/dump-restore-private.h
@@ -0,0 +1,13 @@
+#ifndef DUMP_RESTORE_PRIVATE_H
+#define DUMP_RESTORE_PRIVATE_H
+
+#include "hex-escape.h"
+#include "command-line-arguments.h"
+
+typedef enum dump_formats {
+DUMP_FORMAT_AUTO,
+DUMP_FORMAT_BATCH_TAG,
+DUMP_FORMAT_SUP
+} dump_format_t;
+
+#endif
diff --git a/notmuch-dump.c b/notmuch-dump.c
index 88f598a..d2dad40 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -19,6 +19,7 @@
  */

 #include "notmuch-client.h"
+#include "dump-restore-private.h"

 int
 notmuch_dump_command (unused (void *ctx), int argc, char *argv[])
@@ -43,7 +44,13 @@ notmuch_dump_command (unused (void *ctx), int argc, char 
*argv[])
 char *output_file_name = NULL;
 int opt_index;

+int output_format = DUMP_FORMAT_SUP;
+
 notmuch_opt_desc_t options[] = {
+   { NOTMUCH_OPT_KEYWORD, _format, "format", 'f',
+ (notmuch_keyword_t []){ { "sup", DUMP_FORMAT_SUP },
+ { "batch-tag", DUMP_FORMAT_BATCH_TAG },
+ { 0, 0 } } },
{ NOTMUCH_OPT_STRING, _file_name, "output", 'o', 0  },
{ 0, 0, 0, 0, 0 }
 };
@@ -83,27 +90,56 @@ notmuch_dump_command (unused (void *ctx), int argc, char 
*argv[])
  */
 notmuch_query_set_sort (query, NOTMUCH_SORT_UNSORTED);

+char *buffer = NULL;
+size_t buffer_size = 0;
+
 for (messages = notmuch_query_search_messages (query);
 notmuch_messages_valid (messages);
 notmuch_messages_move_to_next (messages)) {
int first = 1;
+   const char *message_id;
+
message = notmuch_messages_get (messages);
+   message_id = notmuch_message_get_message_id (message);

-   fprintf (output,
-"%s (", notmuch_message_get_message_id (message));
+   if (output_format == DUMP_FORMAT_SUP) {
+   fprintf (output, "%s (", message_id);
+   }

for (tags = notmuch_message_get_tags (message);
 notmuch_tags_valid (tags);
 notmuch_tags_move_to_next (tags)) {
-   if (! first)
-   fprintf (output, " ");
+   const char *tag_str = notmuch_tags_get (tags);

-   fprintf (output, "%s", notmuch_tags_get (tags));
+   if (! first)
+   fputs (" ", output);

first = 0;
+
+   if (output_format == DUMP_FORMAT_SUP) {
+   fputs (tag_str, output);
+   } else {
+   if (hex_encode (notmuch, tag_str,
+   , _size) != HEX_SUCCESS) {
+   fprintf (stderr, "Error: failed to hex-encode tag %s\n",
+tag_str);
+   return 1;
+   }
+   fprintf (output, "+%s", buffer);
+   }
}

-   fprintf (output, ")\n");
+   if (output_format == DUMP_FORMAT_SUP) {
+   fputs (")\n", output);
+   } else {
+   if (hex_encode (notmuch, message_id,
+   , _size) != HEX_SUCCESS) {
+   fprintf (stderr, "Error: failed to hex-encode msg-id %s\n",
+message_id);
+   return 1;
+   }
+   fprintf (output, " -- id:%s\n", buffer);
+   }

notmuch_message_destroy (message);
 }
-- 
1.7.10.4

v5 of Batch-tag dump/restore patches

[da...@tethera.net]

Yet another version. Luckily we all use threaded mailreaders, right?

This obsoletes 

 id:1354979276-20099-1-git-send-email-david at tethera.net

One trivial new patch

[Patch v5 06/11] notmuch-restore: normalize case of error messages.

And a fairly extensive reworking of the error reporting in
parse_tag_line. This version introduces an enum for return values of
parse_tag_line; this could in the future also be used for
parse_sup_line.


commit 55f38bbd034bf8bfbe126cd598cf6085f5f30bf1
Author: David Bremner 
Date:   Sat Dec 8 17:47:21 2012 -0400

tag-util.h: add enum for tag parse status

diff --git a/tag-util.h b/tag-util.h
index 581207a..e828992 100644
--- a/tag-util.h
+++ b/tag-util.h
@@ -30,6 +30,23 @@ typedef enum {

 } tag_op_flag_t;

+/* These should obey the convention that fatal errors are negative,
+ * skipped lines are positive.
+ */
+typedef enum {
+TAG_PARSE_OUT_OF_MEMORY = -1,
+
+/* Line parsed successfuly. */
+TAG_PARSE_SUCCESS = 0,
+
+/* Line has a syntax error */
+TAG_PARSE_INVALID = 1,
+
+/* Line was blank or a comment */
+TAG_PARSE_SKIPPED = 2
+
+} tag_parse_status_t;
+
 /* Parse a string of the following format:
  *
  * +|- [...] [--] 
@@ -45,16 +62,12 @@ typedef enum {
  * Leading and trailing space ' ' is ignored. Empty lines and lines
  * beginning with '#' are ignored.
  *
- * Returns:0   OK,
- * 1   skipped (invalid) line
- * 2   skipped (valid) line
- * -1  fatal(ish) error.
  *
  * Output Parameters:
  * ops contains a list of tag operations
  * query_str the search terms.
  */
-int
+tag_parse_status_t
 parse_tag_line (void *ctx, char *line,
tag_op_flag_t flags,
char **query_str, tag_op_list_t *ops);

commit da6f5cb79c526b8229fb2dbda0ecdce568c2a47c
Author: David Bremner 
Date:   Sat Dec 8 17:48:02 2012 -0400

tag-util.h: uncrustify comments

diff --git a/tag-util.h b/tag-util.h
index e828992..99b0fa0 100644
--- a/tag-util.h
+++ b/tag-util.h
@@ -20,7 +20,7 @@ typedef enum {

 /* Don't try to avoid database operations. Useful when we
  * know that message passed needs these operations.
-  */
+ */
 TAG_FLAG_PRE_OPTIMIZED = (1 << 2),

 /* Accept strange tags that might be user error;
@@ -118,9 +118,9 @@ void
 tag_op_list_reset (tag_op_list_t *list);


- /*
-  *   return the i'th tag in the list
-  */
+/*
+ *   return the i'th tag in the list
+ */

 const char *
 tag_op_list_tag (const tag_op_list_t *list, size_t i);

commit e8b27272340c4ba869f1d39e3264f78368d51d86
Author: David Bremner 
Date:   Sat Dec 8 17:53:53 2012 -0400

fixup for error message test

diff --git a/test/dump-restore b/test/dump-restore
index b267792..8a86782 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -183,7 +183,13 @@ test_expect_equal_file EXPECTED.$test_count 
OUTPUT.$test_count

 test_begin_subtest 'restore: checking error messages'
 notmuch restore 

gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 6:04 PM, Jason A. Donenfeld  wrote:
>> BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to 
>> get that far.
>
> Really? Are you in the UK? I'll have to do this dynamically using the
> XLIST imap extension.

Okay, I've updated it to dynamically find the All Mail folder.

gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 5:20 PM, Patrick Totzke  
wrote:
> I have two new errors:
>
> -
> ./gmail-notmuch.py -u patricktotzke at gmail.com -p mypwd ~/mail/gmail/
>
> I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
> as this is where my all-mail maildir is, and i also tried an absolute path, 
> all the same result.
>
> If I point it to the root of my notmuch directory I get:

Pointing it at the root directory is what you want. In fact, I'm
removing the argument from the script and having it just get that from
the config. So the correct invocation will be:

./gmail-notmuch.py -u blabla -p password

>   File "./gmail-notmuch.py", line 147, in create_progressbar
> return ProgressBar(maxval=total, widgets=[text + ": ", SimpleProgress(), 
> Bar(), Percentage(), " ", ETA(), " ", FileTransferSpeed(unit="emails")])
> TypeError: __init__() got an unexpected keyword argument 'unit'

You're probably running with an old version of python-statusbar.

>
> BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get 
> that far.

Really? Are you in the UK? I'll have to do this dynamically using the
XLIST imap extension.

[Patch v4 08/10] test/dump-restore: add test for warning/error messages

[Mark Walters]

On Sat, 08 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> We want to test both that error/warning messages are generated when
> they should be, and not generated when they should not be. This varies
> between restore and batch tagging.
> ---
>  test/dump-restore |   29 +
>  1 file changed, 29 insertions(+)
>
> diff --git a/test/dump-restore b/test/dump-restore
> index 7d91cca..b267792 100755
> --- a/test/dump-restore
> +++ b/test/dump-restore
> @@ -181,6 +181,35 @@ notmuch restore < EXPECTED.$test_count
>  notmuch dump --format=sup > OUTPUT.$test_count
>  test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
>  
> +test_begin_subtest 'restore: checking error messages'
> +notmuch restore 
 ++a +b
> ++c +d --
> +# this is a harmless comment, do not yell about it.
> +
> +# the previous line was blank; also no yelling please
> ++%zz -- id:whatever
> ++e +f id:%yy
> +# the next line should report an an empty tag error for batch tagging, but 
> restore
> ++ +e -- id:20091117232137.GA7669 at griffis1.net

I don't follow the comment above: do you mean "but not for restore", or
it should report an error but restore anyway or?

Best wishes

Mark

> +# highlight the sketchy id parsing; this should be last
> ++g -- id:foo and bar
> +EOF


> +
> +cat < EXPECTED
> +Warning: no query string: +a +b
> +Warning: Ignoring invalid input line +a +b
> +Warning: no query string: +c +d --
> +Warning: Ignoring invalid input line +c +d --
> +Warning: Hex decoding of tag %zz failed
> +Warning: Ignoring invalid input line +%zz -- id:whatever
> +Warning: Hex decoding of query id:%yy failed
> +Warning: Ignoring invalid input line +e +f id:%yy
> +Warning: Cannot apply tags to missing message: foo and bar
> +EOF
> +
> +test_expect_equal_file EXPECTED OUTPUT
> +
>  test_begin_subtest 'roundtripping random message-ids and tags'
>  
>  ${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-08 17:04:48)
> On Sat, Dec 8, 2012 at 5:20 PM, Patrick Totzke  
> wrote:
> > I have two new errors:
> >
> > -
> > ./gmail-notmuch.py -u patricktotzke at gmail.com -p mypwd ~/mail/gmail/
> >
> > I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
> > as this is where my all-mail maildir is, and i also tried an absolute path, 
> > all the same result.
> >
> > If I point it to the root of my notmuch directory I get:
> 
> Pointing it at the root directory is what you want. In fact, I'm
> removing the argument from the script and having it just get that from
> the config. So the correct invocation will be:
> 
> ./gmail-notmuch.py -u blabla -p password

Okay..
> 
> >   File "./gmail-notmuch.py", line 147, in create_progressbar
> > return ProgressBar(maxval=total, widgets=[text + ": ", 
> > SimpleProgress(), Bar(), Percentage(), " ", ETA(), " ", 
> > FileTransferSpeed(unit="emails")])
> > TypeError: __init__() got an unexpected keyword argument 'unit'
> 
> You're probably running with an old version of python-statusbar.

I just installed it from ubuntu 12.04's repository. the version was 2.2-2.
Installed from pip now, and it seems ok. you'll want to make more specific 
dependencies at some
point..

> >
> > BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to 
> > get that far.
> 
> Really? Are you in the UK? I'll have to do this dynamically using the
> XLIST imap extension.

Yea, but its a *really* old account I first got from google.de.
Reading this dynamically would def. help.


next try..
-

Traceback (most recent call last):
  File "./gmail-notmuch.py", line 246, in 
main()
  File "./gmail-notmuch.py", line 74, in main
new_messages = retag_old_messages(database, messages, destination)
  File "./gmail-notmuch.py", line 176, in retag_old_messages
progressbar.start()
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 
311, in start
self.update(0)
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 
283, in update
self.fd.write(self._format_line() + '\r')
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 
243, in _format_line
widgets = ''.join(self._format_widgets())
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 
223, in _format_widgets
widget = format_updatable(widget, self)
  File "/usr/local/lib/python2.7/dist-packages/progressbar/widgets.py", line 
38, in format_updatable
if hasattr(updatable, 'update'): return updatable.update(pbar)
  File "/usr/local/lib/python2.7/dist-packages/progressbar/widgets.py", line 
184, in update
return '%3d%%' % pbar.percentage()
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 
208, in percentage
return self.currval * 100.0 / self.maxval
ZeroDivisionError: float division by zero
--



Another feature request:
Could you make the progressbar stuff optinal? in the end one will want to run 
this via cronjob
and this shouldn't print anything to stdout.
Also, I can pretty much recommend the 'logging' module.

cheers,
/p
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: signature
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20121208/5aef8270/attachment.pgp>

gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-07 13:49:46)
> Not sure what is causing this. My best guess is that your password was
> incorrect and that I'm not checking the login return value.

Yes, you're right, it was an incorrect passwd.

> One thing you also might want to watch out for is
> exceptions due to a locked notmuch index.
> 
> 
> How do I do this?

See e.g. here: https://github.com/pazz/alot/blob/master/alot/db/manager.py#L163
I'm sure "afew" (https://github.com/teythoon/afew) does a similar thing.


I have two new errors:

-
./gmail-notmuch.py -u patricktotzke at gmail.com -p mypwd ~/mail/gmail/
Logging in...
Selecting all mail...
Error opening database at /home/pazz/mail/gmail/.notmuch: No such file or 
directory
Traceback (most recent call last):
  File "./gmail-notmuch.py", line 225, in 
main()
  File "./gmail-notmuch.py", line 56, in main
database = notmuch.Database(destination, False, 
notmuch.Database.MODE.READ_WRITE)
  File "/home/pazz/.local/lib/python2.7/site-packages/notmuch/database.py", 
line 154, in __init__
self.open(path, mode)
  File "/home/pazz/.local/lib/python2.7/site-packages/notmuch/database.py", 
line 214, in open
raise NotmuchError(status)
notmuch.errors.FileError
-

I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
as this is where my all-mail maildir is, and i also tried an absolute path, all 
the same result.

If I point it to the root of my notmuch directory I get:
-
[~/projects/gmail-notmuch] ./gmail-notmuch.py -u patricktotzke at gmail.com -p 
mypwd /home/pazz/mail
Logging in...
Selecting all mail...
Traceback (most recent call last):
  File "./gmail-notmuch.py", line 225, in 
main()
  File "./gmail-notmuch.py", line 58, in main
messages = discover_messages(imap, total)
  File "./gmail-notmuch.py", line 104, in discover_messages
new_readline.progressbar = create_progressbar("Receiving message list", 
total).start()
  File "./gmail-notmuch.py", line 147, in create_progressbar
return ProgressBar(maxval=total, widgets=[text + ": ", SimpleProgress(), 
Bar(), Percentage(), " ", ETA(), " ", FileTransferSpeed(unit="emails")])
TypeError: __init__() got an unexpected keyword argument 'unit'

BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get 
that far.


HTH,
/p
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: signature
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20121208/9817067c/attachment-0001.pgp>

[PATCH V2 1/1] NEWS: under-the-hood Emacs interface fixes

[David Bremner]

Tomi Ollila  writes:

> Added the following Emacs Interface NEWS entries:
>
> Catch errors bodypart insertions may throw,
> Improved text/calendar content handling and
> Disabled coding conversions when reading in
> `with-current-notmuch-show-message`.
>

Some suggestions about wording:

diff --git a/NEWS b/NEWS
index 6970178..2e1c054 100644
--- a/NEWS
+++ b/NEWS
@@ -28,7 +28,7 @@ Removal of the deprecated `notmuch-folders` variable
   has now been removed. Any remaining users should migrate to
   `notmuch-saved-searches`.

-Handle errors bodypart insertions may throw
+Handle errors from bodypart insertions

   If displaying the text of a message in show mode causes an error (in
   the `notmuch-show-insert-part-*` functions), notmuch no longer cuts
@@ -38,9 +38,9 @@ Handle errors bodypart insertions may throw
 Improved text/calendar content handling

   Carriage returns in embedded text/calendar content caused insertion
-  of calendar content fail. Now CRs are removed before calling icalendar
+  of the calendar content fail. Now CRs are removed before calling icalendar
   to extract icalendar data. In case icalendar extraction fails an error
-  is thrown for bodypart insertion function to react upon it.
+  is thrown for the bodypart insertion function to deal with.

 Disabled coding conversions when reading in `with-current-notmuch-show-message`

[PATCH] test: Adding non-maildir tags does not move message from new to cur

[da...@tethera.net]

From: Michal Sojka 

Some MUA's like mutt show the difference between "new" emails living in maildir
directory new/, and "old" emails living in maildir directory cur/. However
notmuch tag unconditionally moves selected messages from new/ to cur/, even if
no maildir synchronized tag is changed.

While maildir specification forbids messages with tags living in new/, there is
no need to move messages to cur/ when no maildir synchronized tag is changed.
Thus notmuch can remain transparent with respect to other MUA's.

[ Edited commit log to better describe the intended changes, and tag the
  test as broken until the actual changes are implemented -- Louis Rilling ]

Signed-off-by: Louis Rilling 

[ Converted to use test_subtest_known_broken, David Bremner ]
---

Do we agree that the behaviour of moving messages to ./cur on tagging
is broken? If so, maybe it's worth tidying up and applying this.  The
use of cd and ls strikes me as slightly suspect, but I welcome other
opinions.

 test/maildir-sync |9 +
 1 file changed, 9 insertions(+)

diff --git a/test/maildir-sync b/test/maildir-sync
index 0fc742a..6165782 100755
--- a/test/maildir-sync
+++ b/test/maildir-sync
@@ -83,6 +83,15 @@ test_expect_equal "$output" "No new mail."
 # creating new directories in the mail store, then it should be
 # creating all necessary database state for those directories.

+test_begin_subtest "Adding non-maildir tags does not move message from new to 
cur"
+test_subtest_known_broken
+add_message [subject]='"Message to stay in new"' \
+[date]='"Sat, 01 Jan 2000 12:00:00 -"' \
+[filename]='message-to-stay-in-new' [dir]=new
+notmuch tag +donotmove subject:"Message to stay in new"
+output=$(cd "$MAIL_DIR"; ls */message-to-stay-in-new*)
+test_expect_equal "$output" "new/message-to-stay-in-new"
+
 test_begin_subtest "Removing 'S' flag from existing filename adds 'unread' tag"
 add_message [subject]='"Removing S flag"' [filename]='removing-s-flag:2,S' 
[dir]=cur
 output=$(notmuch search subject:"Removing S flag" | notmuch_search_sanitize)
-- 
1.7.10.4

[RFC PATCH] emacs/show: add command to view a patch using diff mode

[da...@tethera.net]

From: David Bremner 

This enables fast navigation between hunks and better highlighting.

The use of notmuch-get-bodypart-internal could be improved; hard
coding to 1 seems to work for the output of git-format-patch. It also
not so nice to call (insert (notmuch-get-bodypart-internal ...)) since the
last thing n-g-b-i does is call buffer-string.
---

The main thing that worries me is the use of
notmuch-get-bodypart-internal.  Do we have a canonical way to get the
message body?  It would also be nice to have "q" actually kill the
buffer, but I didn't find a clean way to do that.

 emacs/notmuch-show.el |   16 
 1 file changed, 16 insertions(+)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 20f8997..442e6ce 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1694,6 +1694,22 @@ to show, nil otherwise."
 (set-buffer-modified-p nil)
 (view-buffer buf 'kill-buffer-if-not-modified)))

+(defun notmuch-show-view-as-patch ()
+  "View the the current message as a patch."
+  (interactive)
+  (let* ((id (notmuch-show-get-message-id))
+(subject (concat "Subject: " (notmuch-show-get-subject) "\n"))
+(diff-default-read-only t)
+(buf (get-buffer-create (concat "*notmuch-patch-" id "*"
+(switch-to-buffer buf)
+(let ((inhibit-read-only t))
+  (erase-buffer)
+  (insert subject)
+  (insert (notmuch-get-bodypart-internal id 1 nil)))
+(set-buffer-modified-p nil)
+(diff-mode)
+(goto-char (point-min
+
 (defun notmuch-show-pipe-message (entire-thread command)
   "Pipe the contents of the current message (or thread) to the given command.

-- 
1.7.10.4

[Patch v4 08/10] test/dump-restore: add test for warning/error messages

[David Bremner]

Mark Walters  writes:
>> ++e +f id:%yy
>> +# the next line should report an an empty tag error for batch tagging, but 
>> restore
>> ++ +e -- id:20091117232137.GA7669 at griffis1.net
>
> I don't follow the comment above: do you mean "but not for restore", or
> it should report an error but restore anyway or?

the former, it should read "but not for restore"

d

Tags with spaces

[Jason A. Donenfeld]

Very helpful! Thanks folks for the explanations.

Another thing to note is that the dump format doesn't add any quotes
to the tags, so this is something of an issue I suppose.

The Gmail Importer Script: Complete

[Jason A. Donenfeld]

Hi everyone,

My script to import messages from gmail into notmuch, including tag
information, is now polished and feature complete.


** Watch a video of it here:
** https://www.youtube.com/watch?v=e-8EHIAr7wA


Read the source here:
http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Download the plaintext here:
http://git.zx2c4.com/gmail-notmuch/plain/gmail-notmuch.py

View the requirements here:
http://git.zx2c4.com/gmail-notmuch/tree/requirements.txt

Enjoy,
Jason

gmail importer script

[Jason A. Donenfeld]

> I actually prefer this approach, but I think it would be more useful to
> leave the syncing of the emails to a different program, and then just
> managing the labels / tags with your tool (which is notmuch territory). So
> the workflow would be:
> So, implementing the mail sync in this script would, as I see it, kind of
> reinventing the wheel.

You're misunderstanding me. Let me make it very clear what the script
does. Actually why don't you just read the source code? Please? Well,
anyway, here:

- It looks in gmail and asks it which messages gmail has
- It downloads those messages
- It applies gmail's labels to those messages as tags

The end.

Along the way it has some smart things to reduce redundant downloads.

For more information, consult the source code.

[PATCH] emacs: query: make sync queries use sexp

[Mark Walters]

This changes the queries used by notmuch-show from json to sexp (patch
based on a comment by Tomi on irc as to the trivial change needed).

The async query parsed used by search is not as easy to convert.
---

It's probably worth making this change: sexps are significantly faster
but I doubt anyone would notice in show (since the query is small and
the wash processing etc relatively large).

At the moment this doesn't do any error fixing. The json version did
not either but the sexp parser and the json parser might behave
differently on malformed input.

Best wishes

Mark


 emacs/notmuch-query.el |7 ++-
 1 files changed, 2 insertions(+), 5 deletions(-)

diff --git a/emacs/notmuch-query.el b/emacs/notmuch-query.el
index d66baea..0ee6cca 100644
--- a/emacs/notmuch-query.el
+++ b/emacs/notmuch-query.el
@@ -29,10 +29,7 @@ A thread is a forest or list of trees. A tree is a two 
element
 list where the first element is a message, and the second element
 is a possibly empty forest of replies.
 "
-  (let  ((args '("show" "--format=json"))
-(json-object-type 'plist)
-(json-array-type 'list)
-(json-false 'nil))
+  (let  ((args '("show" "--format=sexp")))
 (if notmuch-show-process-crypto
(setq args (append args '("--decrypt"
 (setq args (append args search-terms))
@@ -40,7 +37,7 @@ is a possibly empty forest of replies.
   (progn
(apply 'call-process (append (list notmuch-command nil (list t nil) 
nil) args))
(goto-char (point-min))
-   (json-read)
+   (sexp-at-point)

 ;;
 ;; Mapping functions across collections of messages.
-- 
1.7.9.1

gmail importer script

[Rainer M Krug]

On 7 December 2012 16:32, Jason A. Donenfeld  wrote:

>
>
> On Fri, Dec 7, 2012 at 2:57 PM, Rainer M Krug  wrote:
>>
>>  >
>> >
>> > 2) I am downloading with the patched notmuch (to add x-keywords
>> containing the labels) only
>> > the "All Mail" folder - does your script use this information (if
>> present) to tag the emails?
>> >
>> >
>> > What is the patched notmuch? What does that do?
>>
>> Sorry - should have been "patched offlineimap" (
>> https://github.com/aroig/offlineimap
>>
>> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970and
>> http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 )
>>
>> It is synching the gmail labels into the x-keywords header as a comma
>> separated list. These could
>> then be used by notmuch to be added to the labels. It also does sync
>> changes from the X-Keywords
>> to gmail (see
>> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970for
>>  details)
>
>
> I see.
>
> I use the All Mail folder and get the tags using the X-GM-LABELS imap
> extension. I don't muck with the mail headers in the process, fortunately.
>

I actually prefer this approach, but I think it would be more useful to
leave the syncing of the emails to a different program, and then just
managing the labels / tags with your tool (which is notmuch territory). So
the workflow would be:

Initial or download (only remote changes):

1) Download email ("All Mail" folder) with whatever you are using
(offlinemail, ...)
2) call your script to index the downloaded emails (All Mail" folder) with
notmuch and get the labels and set / remove them in notmuch accordingly.

Upload (only local changes):

1) upload emails using whatever (offlineimap, ...)
2) upload the changed tags and create the needed new labels (and possibly
delete empty ons?)

The tricky part will be the synching when changes on both sides were
observed.
The mail sync is implemented in e.g. offlineimap and works reilably. But I
don't know how one could handle tag / label changes on both sides.

So, implementing the mail sync in this script would, as I see it, kind of
reinventing the wheel. One can leave it in, but I would like to have an
option to leave the synching of the emails to a separate program.

Cheers,

Rainer



-- 
NEW GERMAN FAX NUMBER!!!

Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology,
UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa

Cell:   +27 - (0)83 9479 042
Fax:+27 - (0)86 516 2782
Fax:+49 - (0)321 2125 2244
email:  Rainer at krugs.de

Skype:  RMkrug
Google: R.M.Krug at gmail.com
-- next part --
An HTML attachment was scrubbed...
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20121208/ee41fe95/attachment.html>

[PATCH 2/2] contrib: pick: thread tagging (including archiving) implemented

[Mark Walters]

Previously pick had no actions based on the entire thread: this adds
some. Note in this version '*' is bound to `tag thread' which is not
consistent with search or show. However it still might be the most
natural thing (as it is similar to running * in the show pane).
---
 contrib/notmuch-pick/notmuch-pick.el |   30 ++
 1 files changed, 30 insertions(+), 0 deletions(-)

diff --git a/contrib/notmuch-pick/notmuch-pick.el 
b/contrib/notmuch-pick/notmuch-pick.el
index 77e15bb..23442b9 100644
--- a/contrib/notmuch-pick/notmuch-pick.el
+++ b/contrib/notmuch-pick/notmuch-pick.el
@@ -173,6 +173,7 @@
 (define-key map "q" 'notmuch-pick-quit)
 (define-key map "x" 'notmuch-pick-quit)
 (define-key map "?" 'notmuch-help)
+(define-key map "A" 'notmuch-pick-archive-thread)
 (define-key map "a" 'notmuch-pick-archive-message-then-next)
 (define-key map "=" 'notmuch-pick-refresh-view)
 (define-key map "s" 'notmuch-pick-to-search)
@@ -188,6 +189,7 @@
 (define-key map "|" 'notmuch-pick-pipe-message)
 (define-key map "-" 'notmuch-pick-remove-tag)
 (define-key map "+" 'notmuch-pick-add-tag)
+(define-key map "*" 'notmuch-pick-tag-thread)
 (define-key map " " 'notmuch-pick-scroll-or-next)
 (define-key map "b" 'notmuch-pick-scroll-message-window-back)
 map))
@@ -579,6 +581,34 @@ than only the current message."
   "Return a search string for all message ids of messages in the
 current thread."
   (mapconcat 'identity (notmuch-pick-get-messages-ids-thread) " or "))
+
+(defun notmuch-pick-tag-thread ( tag-changes)
+  "Tag all messages in the current thread"
+  (interactive)
+  (setq tag-changes (funcall 'notmuch-tag 
(notmuch-pick-get-messages-ids-thread-search) tag-changes))
+  (notmuch-pick-thread-mapc
+   (lambda () (notmuch-pick-tag-update-display tag-changes
+
+(defun notmuch-pick-archive-thread ( unarchive)
+  "Archive each message in thread.
+
+Archive each message currently shown by applying the tag changes
+in `notmuch-archive-tags' to each (remove the \"inbox\" tag by
+default). If a prefix argument is given, the messages will be
+\"unarchived\", i.e. the tag changes in `notmuch-archive-tags'
+will be reversed.
+
+Note: This command is safe from any race condition of new messages
+being delivered to the same thread. It does not archive the
+entire thread, but only the messages shown in the current
+buffer."
+  (interactive "P")
+  (when notmuch-archive-tags
+(notmuch-pick-tag-thread
+ (notmuch-tag-change-list notmuch-archive-tags unarchive
+
+;; Functions below here display the pick buffer itself.
+
 (defun notmuch-pick-clean-address (address)
   "Try to clean a single email ADDRESS for display. Return
 AUTHOR_NAME if present, otherwise return AUTHOR_EMAIL. Return
-- 
1.7.9.1

[PATCH 1/2] contrib: pick: add thread based utility functions

[Mark Walters]

Previously notmuch-pick had no thread based functionality. This adds a
macro to iterate through all messages in a thread. To simplify this it
adds a text-property marker to the first message of each thread.
---
 contrib/notmuch-pick/notmuch-pick.el |   28 
 1 files changed, 28 insertions(+), 0 deletions(-)

diff --git a/contrib/notmuch-pick/notmuch-pick.el 
b/contrib/notmuch-pick/notmuch-pick.el
index b474231..77e15bb 100644
--- a/contrib/notmuch-pick/notmuch-pick.el
+++ b/contrib/notmuch-pick/notmuch-pick.el
@@ -552,6 +552,33 @@ than only the current message."
  (message (format "Command '%s' exited abnormally with code %d"
   shell-command exit-code)))

+(defun notmuch-pick-thread-top ()
+  (interactive)
+  (while (not (notmuch-pick-get-prop :first))
+(forward-line -1)))
+
+(defmacro notmuch-pick-thread-mapc (function)
+  "Iterate through all messages in the current thread
+ and call FUNCTION for side effects."
+  (save-excursion
+(notmuch-pick-thread-top)
+(loop do (progn
+  (funcall function)
+  (forward-line))
+ while (and (notmuch-pick-get-message-properties)
+(not (notmuch-pick-get-prop :first))
+
+(defun notmuch-pick-get-messages-ids-thread ()
+  "Return all id: queries of messages in the current thread."
+  (let ((message-ids))
+(notmuch-pick-thread-mapc
+ (lambda () (push (notmuch-pick-get-message-id) message-ids)))
+message-ids))
+
+(defun notmuch-pick-get-messages-ids-thread-search ()
+  "Return a search string for all message ids of messages in the
+current thread."
+  (mapconcat 'identity (notmuch-pick-get-messages-ids-thread) " or "))
 (defun notmuch-pick-clean-address (address)
   "Try to clean a single email ADDRESS for display. Return
 AUTHOR_NAME if present, otherwise return AUTHOR_EMAIL. Return
@@ -638,6 +665,7 @@ unchanged ADDRESS if parsing fails."
(push "?" tree-status)))

   (push (concat (if replies "?" "?") "?") tree-status)
+  (plist-put msg :first (and first (eq 0 depth)))
   (notmuch-pick-insert-msg (plist-put msg :tree-status tree-status))
   (pop tree-status)
   (pop tree-status)
-- 
1.7.9.1

[PATCH 0/2] Add some thread based actions to pick

[Mark Walters]

This addes the a macro to notmuch pick to apply a command to all
messages in the current thread. In the second patch we add the ability
to tag/archive whole threads.

This is independent of
id:1354970494-18050-1-git-send-email-markwalters1009 at gmail.com and
id:1354970674-18136-1-git-send-email-markwalters1009 at gmail.com but may
have overlapping context (in the keybindings) so they should be applied first.

I can easily supply a version applying straight to master if preferred.

Best wishes

Mark



Mark Walters (2):
  contrib: pick: add thread based utility functions
  contrib: pick: thread tagging (including archiving) implemented

 contrib/notmuch-pick/notmuch-pick.el |   58 ++
 1 files changed, 58 insertions(+), 0 deletions(-)

-- 
1.7.9.1

[PATCH] contrib: pick: slightly tweak running search and pick from pick buffer

[Mark Walters]

Previously running search or pick from the pick buffer did not close
the message pane (if open). This meant that then new search ends up in
a very small window. Fix this so that the message pane is
shut. However, make it so that the pane is shut after the search
string is entered in case the user is basing the search on something
in the current message.
---

This is essentially an independent patch but the context clashes with
id:1354970494-18050-1-git-send-email-markwalters1009 at gmail.com so that
should be applied first for testing.

If there is any problem with the first patch I can trivially rebase
this one.

Best wishes

Mark


 contrib/notmuch-pick/notmuch-pick.el |   23 +--
 1 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/contrib/notmuch-pick/notmuch-pick.el 
b/contrib/notmuch-pick/notmuch-pick.el
index 36587a6..b474231 100644
--- a/contrib/notmuch-pick/notmuch-pick.el
+++ b/contrib/notmuch-pick/notmuch-pick.el
@@ -175,8 +175,8 @@
 (define-key map "?" 'notmuch-help)
 (define-key map "a" 'notmuch-pick-archive-message-then-next)
 (define-key map "=" 'notmuch-pick-refresh-view)
-(define-key map "s" 'notmuch-search)
-(define-key map "z" 'notmuch-pick)
+(define-key map "s" 'notmuch-pick-to-search)
+(define-key map "z" 'notmuch-pick-to-pick)
 (define-key map "m" 'notmuch-pick-new-mail)
 (define-key map "f" 'notmuch-pick-forward-message)
 (define-key map "r" 'notmuch-pick-reply-sender)
@@ -289,6 +289,25 @@ Does NOT change the database."
   (interactive)
   (notmuch-pick-tag "-"))

+;; The next two functions close the message window before searching or
+;; picking but they do so after the user has entered the query (in
+;; case the user was basing the query on something in the message
+;; window).
+
+(defun notmuch-pick-to-search ()
+  "Run \"notmuch search\" with the given `query' and display results."
+  (interactive)
+  (let ((query (notmuch-read-query "Notmuch search: ")))
+(notmuch-pick-close-message-window)
+(notmuch-search query)))
+
+(defun notmuch-pick-to-pick ()
+  "Run a query and display results in experimental notmuch-pick mode"
+  (interactive)
+  (let ((query (notmuch-read-query "Notmuch pick: ")))
+(notmuch-pick-close-message-window)
+(notmuch-pick query)))
+
 ;; This function should be in notmuch-hello.el but we are trying to
 ;; minimise impact on the rest of the codebase.
 (defun notmuch-pick-from-hello ( search)
-- 
1.7.9.1

[PATCH] contrib: pick: archive message updated

[Mark Walters]

Update pick's archive message to respect notmuch-archive-tags. Also
split archive message into an archiving part and a separate
"then-next" part, to move more inline with show. Update the keybinding
so default behaviour is unchanged.
---

Notmuch pick had fallen behind show so update.

Best wishes 

Mark


 contrib/notmuch-pick/notmuch-pick.el |   21 +
 1 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/contrib/notmuch-pick/notmuch-pick.el 
b/contrib/notmuch-pick/notmuch-pick.el
index 755cbbc..36587a6 100644
--- a/contrib/notmuch-pick/notmuch-pick.el
+++ b/contrib/notmuch-pick/notmuch-pick.el
@@ -173,7 +173,7 @@
 (define-key map "q" 'notmuch-pick-quit)
 (define-key map "x" 'notmuch-pick-quit)
 (define-key map "?" 'notmuch-help)
-(define-key map "a" 'notmuch-pick-archive-message)
+(define-key map "a" 'notmuch-pick-archive-message-then-next)
 (define-key map "=" 'notmuch-pick-refresh-view)
 (define-key map "s" 'notmuch-search)
 (define-key map "z" 'notmuch-pick)
@@ -393,10 +393,23 @@ Does NOT change the database."
   (kill-buffer notmuch-pick-message-buffer))
 t))

-(defun notmuch-pick-archive-message ()
+(defun notmuch-pick-archive-message ( unarchive)
+  "Archive the current message.
+
+Archive the current message by applying the tag changes in
+`notmuch-archive-tags' to it (remove the \"inbox\" tag by
+default). If a prefix argument is given, the message will be
+\"unarchived\", i.e. the tag changes in `notmuch-archive-tags'
+will be reversed."
+  (interactive "P")
+  (when notmuch-archive-tags
+(apply 'notmuch-pick-tag
+  (notmuch-tag-change-list notmuch-archive-tags unarchive
+
+(defun notmuch-pick-archive-message-then-next ( unarchive)
   "Archive the current message and move to next matching message."
-  (interactive)
-  (notmuch-pick-tag "-inbox")
+  (interactive "P")
+  (notmuch-pick-archive-message unarchive)
   (notmuch-pick-next-matching-message))

 (defun notmuch-pick-next-message ()
-- 
1.7.9.1

[Patch v3b 5/9] notmuch-restore: add support for input format 'batch-tag'

[Mark Walters]

Hi

Basically LGTM: just one comment below.

On Fri, 07 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> This can be enabled with the new --format=batch-tag command line
> option to "notmuch restore". The input must consist of lines of the
> format:
>
> +|- [...] [--] id:
>
> Each line is interpreted similarly to "notmuch tag" command line
> arguments. The delimiter is one or more spaces ' '. Any characters in
>  and  MAY be hex encoded with %NN where NN is the
> hexadecimal value of the character. Any ' ' and '%' characters in
>  and  MUST be hex encoded (using %20 and %25,
> respectively). Any characters that are not part of  or
>  MUST NOT be hex encoded.
>
> Leading and trailing space ' ' is ignored. Empty lines and lines
> beginning with '#' are ignored.
>
> Commit message mainly stolen from Jani's batch tagging commit, to
> follow.
> ---
>  notmuch-restore.c |  206 
> ++---
>  1 file changed, 131 insertions(+), 75 deletions(-)
>
> diff --git a/notmuch-restore.c b/notmuch-restore.c
> index f03dcac..ceec2d3 100644
> --- a/notmuch-restore.c
> +++ b/notmuch-restore.c
> @@ -19,18 +19,22 @@
>   */
>  
>  #include "notmuch-client.h"
> +#include "dump-restore-private.h"
> +#include "tag-util.h"
> +#include "string-util.h"
> +
> +static volatile sig_atomic_t interrupted;
> +static regex_t regex;
>  
>  static int
> -tag_message (notmuch_database_t *notmuch, const char *message_id,
> -  char *file_tags, notmuch_bool_t remove_all,
> -  notmuch_bool_t synchronize_flags)
> +tag_message (unused (void *ctx),
> +  notmuch_database_t *notmuch,
> +  const char *message_id,
> +  tag_op_list_t *tag_ops,
> +  tag_op_flag_t flags)
>  {
>  notmuch_status_t status;
> -notmuch_tags_t *db_tags;
> -char *db_tags_str;
>  notmuch_message_t *message = NULL;
> -const char *tag;
> -char *next;
>  int ret = 0;
>  
>  status = notmuch_database_find_message (notmuch, message_id, );
> @@ -44,55 +48,62 @@ tag_message (notmuch_database_t *notmuch, const char 
> *message_id,
>  
>  /* In order to detect missing messages, this check/optimization is
>   * intentionally done *after* first finding the message. */
> -if (! remove_all && (file_tags == NULL || *file_tags == '\0'))
> - goto DONE;
> -
> -db_tags_str = NULL;
> -for (db_tags = notmuch_message_get_tags (message);
> -  notmuch_tags_valid (db_tags);
> -  notmuch_tags_move_to_next (db_tags)) {
> - tag = notmuch_tags_get (db_tags);
> -
> - if (db_tags_str)
> - db_tags_str = talloc_asprintf_append (db_tags_str, " %s", tag);
> - else
> - db_tags_str = talloc_strdup (message, tag);
> -}
> +if ((flags & TAG_FLAG_REMOVE_ALL) || tag_op_list_size (tag_ops))
> + tag_op_list_apply (message, tag_ops, flags);
>  
> -if (((file_tags == NULL || *file_tags == '\0') &&
> -  (db_tags_str == NULL || *db_tags_str == '\0')) ||
> - (file_tags && db_tags_str && strcmp (file_tags, db_tags_str) == 0))
> - goto DONE;
> +notmuch_message_destroy (message);
>  
> -notmuch_message_freeze (message);
> +return ret;
> +}
>  
> -if (remove_all)
> - notmuch_message_remove_all_tags (message);
> +static int
> +parse_sup_line (void *ctx, char *line,
> + char **query_str, tag_op_list_t *tag_ops)
> +{
>  
> -next = file_tags;
> -while (next) {
> - tag = strsep (, " ");
> - if (*tag == '\0')
> - continue;
> - status = notmuch_message_add_tag (message, tag);
> - if (status) {
> - fprintf (stderr, "Error applying tag %s to message %s:\n",
> -  tag, message_id);
> - fprintf (stderr, "%s\n", notmuch_status_to_string (status));
> - ret = 1;
> - }
> +regmatch_t match[3];
> +char *file_tags;
> +int rerr;
> +
> +tag_op_list_reset (tag_ops);
> +
> +chomp_newline (line);
> +
> +/* Silently ignore blank lines */
> +if (line[0] == '\0') {
> + return 1;
> +}
> +
> +rerr = xregexec (, line, 3, match, 0);
> +if (rerr == REG_NOMATCH) {
> + fprintf (stderr, "Warning: Ignoring invalid sup format line: %s\n",
> +  line);
> + return 1;
>  }
>  
> -notmuch_message_thaw (message);
> +*query_str = talloc_strndup (ctx, line + match[1].rm_so,
> +  match[1].rm_eo - match[1].rm_so);
> +file_tags = talloc_strndup (ctx, line + match[2].rm_so,
> + match[2].rm_eo - match[2].rm_so);
>  
> -if (synchronize_flags)
> - notmuch_message_tags_to_maildir_flags (message);
> +char *tok = file_tags;
> +size_t tok_len = 0;
>  
> -  DONE:
> -if (message)
> - notmuch_message_destroy (message);
> +tag_op_list_reset (tag_ops);
> +
> +while ((tok = strtok_len (tok + tok_len, " ", _len)) != NULL) {
> +
> + if (*(tok + tok_len) != '\0') {
> + *(tok + tok_len) 

[Patch v3b 9/9] tag-util: optimization of tag application

[Mark Walters]

On Fri, 07 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> The idea is not to bother with restore operations if they don't change
> the set of tags. This is actually a relatively common case.
>
> In order to avoid fancy datastructures, this method is quadratic in
> the number of tags; at least on my mail database this doesn't seem to
> be a big problem.
> ---
>  tag-util.c |   66 
> 
>  1 file changed, 66 insertions(+)
>
> diff --git a/tag-util.c b/tag-util.c
> index 932ee7f..3d54e9e 100644
> --- a/tag-util.c
> +++ b/tag-util.c
> @@ -124,6 +124,69 @@ message_error (notmuch_message_t *message,
>  fprintf (stderr, "Status: %s\n", notmuch_status_to_string (status));
>  }
>  
> +static int
> +makes_changes (notmuch_message_t *message,
> +tag_op_list_t *list,
> +tag_op_flag_t flags)
> +{
> +
> +size_t i;
> +
> +notmuch_tags_t *tags;
> +notmuch_bool_t changes = FALSE;
> +
> +/* First, do we delete an existing tag? */
> +changes = FALSE;
> +for (tags = notmuch_message_get_tags (message);
> +  ! changes && notmuch_tags_valid (tags);
> +  notmuch_tags_move_to_next (tags)) {
> + const char *cur_tag = notmuch_tags_get (tags);
> + int last_op =  (flags & TAG_FLAG_REMOVE_ALL) ? -1 : 0;
> +
> + /* slight contortions to count down with an unsigned index */
> + for (i = list->count; i-- > 0; /*nothing*/) {
> + if (strcmp (cur_tag, list->ops[i].tag) == 0) {
> + last_op = list->ops[i].remove ? -1 : 1;
> + break;
> + }
> + }

I agree that this is a little ugly but ok I think. Is it worth adding a
comment as to why you are counting backwards? eg " we count backwards to
check whether the last change for the tag foo is removal"

But basically LGTM

Mark


> +
> + changes = (last_op == -1);
> +}
> +notmuch_tags_destroy (tags);
> +
> +if (changes)
> + return TRUE;
> +
> +/* Now check for adding new tags */
> +for (i = 0; i < list->count; i++) {
> + notmuch_bool_t exists = FALSE;
> +
> + if (list->ops[i].remove)
> + continue;
> +
> + for (tags = notmuch_message_get_tags (message);
> +  notmuch_tags_valid (tags);
> +  notmuch_tags_move_to_next (tags)) {
> + const char *cur_tag = notmuch_tags_get (tags);
> + if (strcmp (cur_tag, list->ops[i].tag) == 0) {
> + exists = TRUE;
> + break;
> + }
> + }
> + notmuch_tags_destroy (tags);
> +
> + /* the following test is conservative,
> +  * in the sense it ignores cases like +foo ... -foo
> +  * but this is OK from a correctness point of view
> +  */
> + if (! exists)
> + return TRUE;
> +}
> +return FALSE;
> +
> +}
> +
>  notmuch_status_t
>  tag_op_list_apply (notmuch_message_t *message,
>  tag_op_list_t *list,
> @@ -133,6 +196,9 @@ tag_op_list_apply (notmuch_message_t *message,
>  notmuch_status_t status = 0;
>  tag_operation_t *tag_ops = list->ops;
>  
> +if (! (flags & TAG_FLAG_PRE_OPTIMIZED) && ! makes_changes (message, 
> list, flags))
> + return NOTMUCH_STATUS_SUCCESS;
> +
>  status = notmuch_message_freeze (message);
>  if (status) {
>   message_error (message, status, "freezing message");
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Tags with spaces

[David Bremner]

Austin Clements  writes:

> Tags with spaces are supported, but your shell will strip away the
> quotes you've used in both examples before notmuch ever sees them.
> You'll need both shell quoting and Xapian quoting in this case.  Try
>   notmuch search -- '-tag:"This has spaces"'
> This should be easier from a front-end, since it will take care of any
> necessary shell quoting.

Note that there are also patches in the works that add a hex encoded
format for notmuch-tag on standard input; so far we have tried to do
that for regular command line arguments, but it's a possibility.

d

[Patch v4 10/10] tag-util: optimization of tag application

[da...@tethera.net]

From: David Bremner 

The idea is not to bother with restore operations if they don't change
the set of tags. This is actually a relatively common case.

In order to avoid fancy datastructures, this method is quadratic in
the number of tags; at least on my mail database this doesn't seem to
be a big problem.
---
 tag-util.c |   68 
 1 file changed, 68 insertions(+)

diff --git a/tag-util.c b/tag-util.c
index 80ebdc2..b68ea50 100644
--- a/tag-util.c
+++ b/tag-util.c
@@ -138,6 +138,71 @@ message_error (notmuch_message_t *message,
 fprintf (stderr, "Status: %s\n", notmuch_status_to_string (status));
 }

+static int
+makes_changes (notmuch_message_t *message,
+  tag_op_list_t *list,
+  tag_op_flag_t flags)
+{
+
+size_t i;
+
+notmuch_tags_t *tags;
+notmuch_bool_t changes = FALSE;
+
+/* First, do we delete an existing tag? */
+changes = FALSE;
+for (tags = notmuch_message_get_tags (message);
+! changes && notmuch_tags_valid (tags);
+notmuch_tags_move_to_next (tags)) {
+   const char *cur_tag = notmuch_tags_get (tags);
+   int last_op =  (flags & TAG_FLAG_REMOVE_ALL) ? -1 : 0;
+
+   /* scan backwards to get last operation */
+   i = list->count;
+   while (i > 0) {
+   i--;
+   if (strcmp (cur_tag, list->ops[i].tag) == 0) {
+   last_op = list->ops[i].remove ? -1 : 1;
+   break;
+   }
+   }
+
+   changes = (last_op == -1);
+}
+notmuch_tags_destroy (tags);
+
+if (changes)
+   return TRUE;
+
+/* Now check for adding new tags */
+for (i = 0; i < list->count; i++) {
+   notmuch_bool_t exists = FALSE;
+
+   if (list->ops[i].remove)
+   continue;
+
+   for (tags = notmuch_message_get_tags (message);
+notmuch_tags_valid (tags);
+notmuch_tags_move_to_next (tags)) {
+   const char *cur_tag = notmuch_tags_get (tags);
+   if (strcmp (cur_tag, list->ops[i].tag) == 0) {
+   exists = TRUE;
+   break;
+   }
+   }
+   notmuch_tags_destroy (tags);
+
+   /* the following test is conservative,
+* in the sense it ignores cases like +foo ... -foo
+* but this is OK from a correctness point of view
+*/
+   if (! exists)
+   return TRUE;
+}
+return FALSE;
+
+}
+
 notmuch_status_t
 tag_op_list_apply (notmuch_message_t *message,
   tag_op_list_t *list,
@@ -147,6 +212,9 @@ tag_op_list_apply (notmuch_message_t *message,
 notmuch_status_t status = 0;
 tag_operation_t *tag_ops = list->ops;

+if (! (flags & TAG_FLAG_PRE_OPTIMIZED) && ! makes_changes (message, list, 
flags))
+   return NOTMUCH_STATUS_SUCCESS;
+
 status = notmuch_message_freeze (message);
 if (status) {
message_error (message, status, "freezing message");
-- 
1.7.10.4

[Patch v4 09/10] notmuch-{dump, restore}.1: document new format options

[da...@tethera.net]

From: David Bremner 

More or less arbitrarily, notmuch-dump.1 gets the more detailed
description of the format.
---
 man/man1/notmuch-dump.1|   59 
 man/man1/notmuch-restore.1 |   59 +++-
 2 files changed, 112 insertions(+), 6 deletions(-)

diff --git a/man/man1/notmuch-dump.1 b/man/man1/notmuch-dump.1
index 230deec..770b00f 100644
--- a/man/man1/notmuch-dump.1
+++ b/man/man1/notmuch-dump.1
@@ -5,6 +5,7 @@ notmuch-dump \- creates a plain-text dump of the tags of each 
message
 .SH SYNOPSIS

 .B "notmuch dump"
+.RB  [ "\-\-format=(sup|batch-tag)"  "] [--]"
 .RI "[ --output=<" filename "> ] [--]"
 .RI "[ <" search-term ">...]"

@@ -19,6 +20,64 @@ recreated from the messages themselves.  The output of 
notmuch dump is
 therefore the only critical thing to backup (and much more friendly to
 incremental backup than the native database files.)

+.TP 4
+.B \-\-format=(sup|batch-tag)
+
+Notmuch restore supports two plain text dump formats, both with one message-id
+per line, followed by a list of tags.
+
+.RS 4
+.TP 4
+.B sup
+
+The
+.B sup
+dump file format is specifically chosen to be
+compatible with the format of files produced by sup-dump.
+So if you've previously been using sup for mail, then the
+.B "notmuch restore"
+command provides you a way to import all of your tags (or labels as
+sup calls them).
+Each line has the following form
+
+.RS 4
+.RI < message-id >
+.B (
+.RI < tag "> ..."
+.B )
+
+with zero or more tags are separated by spaces. Note that (malformed)
+message-ids may contain arbitrary non-null characters. Note also
+that tags with spaces will not be correctly restored with this format.
+
+.RE
+
+.RE
+.RS 4
+.TP 4
+.B batch-tag
+
+The
+.B batch-tag
+dump format is intended to more robust against malformed message-ids
+and tags containing whitespace or non-\fBascii\fR(7) characters.
+Each line has the form
+
+.RS 4
+.RI "+<" "encoded-tag" "> " "" "+<" "encoded-tag" "> ... -- " "" " id:<" 
encoded-message-id >
+
+where encoded means that every byte not matching the regex
+.B [A-Za-z0-9@=.,_+-]
+is replace by
+.B %nn
+where nn is the two digit hex encoding.
+The astute reader will notice this is a special case of the batch input
+format for \fBnotmuch-tag\fR(1); note that the single message-id query is
+mandatory for \fBnotmuch-restore\fR(1).
+
+.RE
+
+
 With no search terms, a dump of all messages in the database will be
 generated.  A "--" argument instructs notmuch that the
 remaining arguments are search terms.
diff --git a/man/man1/notmuch-restore.1 b/man/man1/notmuch-restore.1
index 2fa8733..6bba628 100644
--- a/man/man1/notmuch-restore.1
+++ b/man/man1/notmuch-restore.1
@@ -6,6 +6,7 @@ notmuch-restore \- restores the tags from the given file (see 
notmuch dump)

 .B "notmuch restore"
 .RB [ "--accumulate" ]
+.RB [ "--format=(auto|batch-tag|sup)" ]
 .RI "[ --input=<" filename "> ]"

 .SH DESCRIPTION
@@ -15,19 +16,51 @@ Restores the tags from the given file (see

 The input is read from the given filename, if any, or from stdin.

-Note: The dump file format is specifically chosen to be
+
+Supported options for
+.B restore
+include
+.RS 4
+.TP 4
+.B \-\-accumulate
+
+The union of the existing and new tags is applied, instead of
+replacing each message's tags as they are read in from the dump file.
+
+.RE
+.RS 4
+.TP 4
+.B \-\-format=(sup|batch-tag|auto)
+
+Notmuch restore supports two plain text dump formats, with each line
+specifying a message-id and a set of tags.
+For details of the actual formats, see \fBnotmuch-dump\fR(1).
+
+.RS 4
+.TP 4
+.B sup
+
+The
+.B sup
+dump file format is specifically chosen to be
 compatible with the format of files produced by sup-dump.
 So if you've previously been using sup for mail, then the
 .B "notmuch restore"
 command provides you a way to import all of your tags (or labels as
 sup calls them).

-The --accumulate switch causes the union of the existing and new tags to be
-applied, instead of replacing each message's tags as they are read in from the
-dump file.
+.RE
+.RS 4
+.TP 4
+.B batch-tag

-See \fBnotmuch-search-terms\fR(7)
-for details of the supported syntax for .
+The
+.B batch-tag
+dump format is intended to more robust against malformed message-ids
+and tags containing whitespace or non-\fBascii\fR(7) characters.  This
+format hex-escapes all characters those outside of a small character
+set, intended to be suitable for e.g. pathnames in most UNIX-like
+systems.

 .B "notmuch restore"
 updates the maildir flags according to tag changes if the
@@ -36,6 +69,20 @@ configuration option is enabled. See \fBnotmuch-config\fR(1) 
for
 details.

 .RE
+
+.RS 4
+.TP 4
+.B auto
+
+This option (the default) tries to guess the format from the
+input. For correctly formed input in either supported format, this
+heuristic, based the fact that batch-tag format contains no parentheses,
+should be accurate.
+
+.RE
+
+.RE
+
 .SH SEE ALSO

 

[Patch v4 08/10] test/dump-restore: add test for warning/error messages

[da...@tethera.net]

From: David Bremner 

We want to test both that error/warning messages are generated when
they should be, and not generated when they should not be. This varies
between restore and batch tagging.
---
 test/dump-restore |   29 +
 1 file changed, 29 insertions(+)

diff --git a/test/dump-restore b/test/dump-restore
index 7d91cca..b267792 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -181,6 +181,35 @@ notmuch restore < EXPECTED.$test_count
 notmuch dump --format=sup > OUTPUT.$test_count
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count

+test_begin_subtest 'restore: checking error messages'
+notmuch restore 

[Patch v4 07/10] test: second set of dump/restore --format=batch-tag tests

[da...@tethera.net]

From: David Bremner 

These one need the completed functionality in notmuch-restore. Fairly
exotic tags are tested, but no weird message id's.

We test each possible input to autodetection, both explicit (with
--format=auto) and implicit (without --format).
---
 test/dump-restore |   85 -
 1 file changed, 84 insertions(+), 1 deletion(-)

diff --git a/test/dump-restore b/test/dump-restore
index ce81e6f..7d91cca 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -98,9 +98,92 @@ notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- 
id://' | \
 sort > OUTPUT.$test_count
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count

+test_begin_subtest "format=batch-tag, # round-trip"
+notmuch dump --format=sup | sort > EXPECTED.$test_count
+notmuch dump --format=batch-tag | notmuch restore --format=batch-tag
+notmuch dump --format=sup | sort > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest "format=batch-tag, # blank lines and comments"
+notmuch dump --format=batch-tag| sort > EXPECTED.$test_count
+notmuch restore < OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest "format=batch-tag, # reverse-round-trip empty tag"
+cat  
OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+tag1='comic_swear=$&^%$^%\\//-+$^%$'
+enc1=$($TEST_DIRECTORY/hex-xcode --direction=encode "$tag1")
+
+tag2=$(printf 'this\n tag\t has\n spaces')
+enc2=$($TEST_DIRECTORY/hex-xcode --direction=encode "$tag2")
+
+enc3='%c3%91%c3%a5%c3%b0%c3%a3%c3%a5%c3%a9-%c3%8f%c3%8a'
+tag3=$($TEST_DIRECTORY/hex-xcode --direction=decode $enc3)
+
+notmuch dump --format=batch-tag > BACKUP
+
+notmuch tag +"$tag1" +"$tag2" +"$tag3" -inbox -unread "*"
+
+test_begin_subtest 'format=batch-tag, round trip with strange tags'
+notmuch dump --format=batch-tag > EXPECTED.$test_count
+notmuch dump --format=batch-tag | notmuch restore --format=batch-tag
+notmuch dump --format=batch-tag > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'format=batch-tag, checking encoded output'
+notmuch dump --format=batch-tag -- from:cworth |\
+awk "{ print \"+$enc1 +$enc2 +$enc3 -- \" \$5 }" > EXPECTED.$test_count
+notmuch dump --format=batch-tag -- from:cworth  > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'restoring sane tags'
+notmuch restore --format=batch-tag < BACKUP
+notmuch dump --format=batch-tag > OUTPUT.$test_count
+test_expect_equal_file BACKUP OUTPUT.$test_count
+
+test_begin_subtest 'format=batch-tag, restore=auto'
+notmuch dump --format=batch-tag > EXPECTED.$test_count
+notmuch tag -inbox -unread "*"
+notmuch restore --format=auto < EXPECTED.$test_count
+notmuch dump --format=batch-tag > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'format=sup, restore=auto'
+notmuch dump --format=sup > EXPECTED.$test_count
+notmuch tag -inbox -unread "*"
+notmuch restore --format=auto < EXPECTED.$test_count
+notmuch dump --format=sup > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'format=batch-tag, restore=default'
+notmuch dump --format=batch-tag > EXPECTED.$test_count
+notmuch tag -inbox -unread "*"
+notmuch restore < EXPECTED.$test_count
+notmuch dump --format=batch-tag > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'format=sup, restore=default'
+notmuch dump --format=sup > EXPECTED.$test_count
+notmuch tag -inbox -unread "*"
+notmuch restore < EXPECTED.$test_count
+notmuch dump --format=sup > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
 test_begin_subtest 'roundtripping random message-ids and tags'

-${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG}
+${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
--num-messages=100

  notmuch dump --format=batch-tag| \
-- 
1.7.10.4

[Patch v4 06/10] test: update dump-restore roundtripping test for batch-tag format

[da...@tethera.net]

From: David Bremner 

Now we can actually round trip these crazy tags and and message ids.
hex-xcode is no longer needed as it's built in.
---
 test/dump-restore |   17 -
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/test/dump-restore b/test/dump-restore
index b4c807f..ce81e6f 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -99,23 +99,22 @@ notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- 
id://' | \
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count

 test_begin_subtest 'roundtripping random message-ids and tags'
-test_subtest_known_broken
-${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
-   --num-messages=10

- notmuch dump| \
-${TEST_DIRECTORY}/hex-xcode --direction=encode| \
+${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG}
+   --num-messages=100
+
+ notmuch dump --format=batch-tag| \
 sort > EXPECTED.$test_count

  notmuch tag +this_tag_is_very_unlikely_to_be_random '*'

- ${TEST_DIRECTORY}/hex-xcode --direction=decode < EXPECTED.$test_count | \
-notmuch restore 2>/dev/null
+ notmuch restore --format=batch-tag < EXPECTED.$test_count

- notmuch dump| \
-${TEST_DIRECTORY}/hex-xcode --direction=encode| \
+ notmuch dump --format=batch-tag| \
 sort > OUTPUT.$test_count

 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count

 test_done
+
+# Note the database is "poisoned" for sup format at this point.
-- 
1.7.10.4

[Patch v4 05/10] notmuch-restore: add support for input format 'batch-tag'

[da...@tethera.net]

From: David Bremner 

This can be enabled with the new --format=batch-tag command line
option to "notmuch restore". The input must consist of lines of the
format:

+|- [...] [--] id:

Each line is interpreted similarly to "notmuch tag" command line
arguments. The delimiter is one or more spaces ' '. Any characters in
 and  MAY be hex encoded with %NN where NN is the
hexadecimal value of the character. Any ' ' and '%' characters in
 and  MUST be hex encoded (using %20 and %25,
respectively). Any characters that are not part of  or
 MUST NOT be hex encoded.

Leading and trailing space ' ' is ignored. Empty lines and lines
beginning with '#' are ignored.

Commit message mainly stolen from Jani's batch tagging commit, to
follow.
---
 notmuch-restore.c |  220 +
 1 file changed, 138 insertions(+), 82 deletions(-)

diff --git a/notmuch-restore.c b/notmuch-restore.c
index f03dcac..44bf88d 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -19,18 +19,22 @@
  */

 #include "notmuch-client.h"
+#include "dump-restore-private.h"
+#include "tag-util.h"
+#include "string-util.h"
+
+static volatile sig_atomic_t interrupted;
+static regex_t regex;

 static int
-tag_message (notmuch_database_t *notmuch, const char *message_id,
-char *file_tags, notmuch_bool_t remove_all,
-notmuch_bool_t synchronize_flags)
+tag_message (unused (void *ctx),
+notmuch_database_t *notmuch,
+const char *message_id,
+tag_op_list_t *tag_ops,
+tag_op_flag_t flags)
 {
 notmuch_status_t status;
-notmuch_tags_t *db_tags;
-char *db_tags_str;
 notmuch_message_t *message = NULL;
-const char *tag;
-char *next;
 int ret = 0;

 status = notmuch_database_find_message (notmuch, message_id, );
@@ -44,55 +48,67 @@ tag_message (notmuch_database_t *notmuch, const char 
*message_id,

 /* In order to detect missing messages, this check/optimization is
  * intentionally done *after* first finding the message. */
-if (! remove_all && (file_tags == NULL || *file_tags == '\0'))
-   goto DONE;
-
-db_tags_str = NULL;
-for (db_tags = notmuch_message_get_tags (message);
-notmuch_tags_valid (db_tags);
-notmuch_tags_move_to_next (db_tags)) {
-   tag = notmuch_tags_get (db_tags);
-
-   if (db_tags_str)
-   db_tags_str = talloc_asprintf_append (db_tags_str, " %s", tag);
-   else
-   db_tags_str = talloc_strdup (message, tag);
-}
+if ((flags & TAG_FLAG_REMOVE_ALL) || tag_op_list_size (tag_ops))
+   tag_op_list_apply (message, tag_ops, flags);

-if (((file_tags == NULL || *file_tags == '\0') &&
-(db_tags_str == NULL || *db_tags_str == '\0')) ||
-   (file_tags && db_tags_str && strcmp (file_tags, db_tags_str) == 0))
-   goto DONE;
+notmuch_message_destroy (message);

-notmuch_message_freeze (message);
+return ret;
+}

-if (remove_all)
-   notmuch_message_remove_all_tags (message);
+/* Sup dump output is one line per message. We match a sequence of
+ * non-space characters for the message-id, then one or more
+ * spaces, then a list of space-separated tags as a sequence of
+ * characters within literal '(' and ')'. */

-next = file_tags;
-while (next) {
-   tag = strsep (, " ");
-   if (*tag == '\0')
-   continue;
-   status = notmuch_message_add_tag (message, tag);
-   if (status) {
-   fprintf (stderr, "Error applying tag %s to message %s:\n",
-tag, message_id);
-   fprintf (stderr, "%s\n", notmuch_status_to_string (status));
-   ret = 1;
-   }
+static int
+parse_sup_line (void *ctx, char *line,
+   char **query_str, tag_op_list_t *tag_ops)
+{
+
+regmatch_t match[3];
+char *file_tags;
+int rerr;
+
+tag_op_list_reset (tag_ops);
+
+chomp_newline (line);
+
+/* Silently ignore blank lines */
+if (line[0] == '\0') {
+   return 1;
 }

-notmuch_message_thaw (message);
+rerr = xregexec (, line, 3, match, 0);
+if (rerr == REG_NOMATCH) {
+   fprintf (stderr, "Warning: Ignoring invalid sup format line: %s\n",
+line);
+   return 1;
+}

-if (synchronize_flags)
-   notmuch_message_tags_to_maildir_flags (message);
+*query_str = talloc_strndup (ctx, line + match[1].rm_so,
+match[1].rm_eo - match[1].rm_so);
+file_tags = talloc_strndup (ctx, line + match[2].rm_so,
+   match[2].rm_eo - match[2].rm_so);

-  DONE:
-if (message)
-   notmuch_message_destroy (message);
+char *tok = file_tags;
+size_t tok_len = 0;
+
+tag_op_list_reset (tag_ops);
+
+while ((tok = strtok_len (tok + tok_len, " ", _len)) != NULL) {
+
+   if (*(tok + tok_len) != '\0') {
+   *(tok + tok_len) = '\0';
+   tok_len++;
+   }
+
+   if 

[Patch v4 04/10] tag-util.[ch]: New files for common tagging routines

[da...@tethera.net]

From: David Bremner 

These are meant to be shared between notmuch-tag and notmuch-restore.

The bulk of the routines implement a "tag operation list" abstract
data type act as a structured representation of a set of tag
operations (typically coming from a single tag command or line of
input).
---
 Makefile.local |1 +
 tag-util.c |  292 
 tag-util.h |  122 +++
 3 files changed, 415 insertions(+)
 create mode 100644 tag-util.c
 create mode 100644 tag-util.h

diff --git a/Makefile.local b/Makefile.local
index 0db1713..c274f07 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -275,6 +275,7 @@ notmuch_client_srcs =   \
query-string.c  \
mime-node.c \
crypto.c\
+   tag-util.c

 notmuch_client_modules = $(notmuch_client_srcs:.c=.o)

diff --git a/tag-util.c b/tag-util.c
new file mode 100644
index 000..80ebdc2
--- /dev/null
+++ b/tag-util.c
@@ -0,0 +1,292 @@
+#include 
+#include "string-util.h"
+#include "tag-util.h"
+#include "hex-escape.h"
+
+#define TAG_OP_LIST_INITIAL_SIZE 10
+
+struct _tag_operation_t {
+const char *tag;
+notmuch_bool_t remove;
+};
+
+struct _tag_op_list_t {
+tag_operation_t *ops;
+size_t count;
+size_t size;
+};
+
+int
+parse_tag_line (void *ctx, char *line,
+   tag_op_flag_t flags,
+   char **query_string,
+   tag_op_list_t *tag_ops)
+{
+char *tok = line;
+size_t tok_len = 0;
+char *line_for_error;
+int ret = 0;
+
+chomp_newline (line);
+
+line_for_error = talloc_strdup (ctx, line);
+if (line_for_error == NULL) {
+   fprintf (stderr, "Error: out of memory\n");
+   return -1;
+}
+
+/* remove leading space */
+while (*tok == ' ' || *tok == '\t')
+   tok++;
+
+/* Skip empty and comment lines. */
+if (*tok == '\0' || *tok == '#') {
+   ret = 2;
+   goto DONE;
+}
+
+tag_op_list_reset (tag_ops);
+
+/* Parse tags. */
+while ((tok = strtok_len (tok + tok_len, " ", _len)) != NULL) {
+   notmuch_bool_t remove;
+   char *tag;
+
+   /* Optional explicit end of tags marker. */
+   if (tok_len == 2 && strncmp (tok, "--", tok_len) == 0) {
+   tok = strtok_len (tok + tok_len, " ", _len);
+   if (tok == NULL)
+   fprintf (stderr, "Warning: no query string: %s\n", 
line_for_error);
+   break;
+   }
+
+   /* Implicit end of tags. */
+   if (*tok != '-' && *tok != '+')
+   break;
+
+   /* If tag is terminated by NUL, there's no query string. */
+   if (*(tok + tok_len) == '\0') {
+   fprintf (stderr, "Warning: no query string: %s\n", line_for_error);
+   ret = 1;
+   goto DONE;
+   }
+
+   /* Terminate, and start next token after terminator. */
+   *(tok + tok_len++) = '\0';
+
+   remove = (*tok == '-');
+   tag = tok + 1;
+
+   /* Maybe refuse empty tags. */
+   if (! (flags & TAG_FLAG_BE_GENEROUS) && *tag == '\0') {
+   fprintf (stderr, "Warning: empty tag: %s\n", line_for_error);
+   ret = 1;
+   goto DONE;
+   }
+
+   /* Decode tag. */
+   if (hex_decode_inplace (tag) != HEX_SUCCESS) {
+   fprintf (stderr, "Warning: Hex decoding of tag %s failed\n",
+tag);
+   ret = 1;
+   goto DONE;
+   }
+
+   if (tag_op_list_append (ctx, tag_ops, tag, remove)) {
+   /* diagnostics already printed */
+   ret = -1;
+   goto DONE;
+   }
+}
+
+if (tok == NULL) {
+   ret = 1;
+   goto DONE;
+}
+
+/* tok now points to the query string */
+if (hex_decode_inplace (tok) != HEX_SUCCESS) {
+   fprintf (stderr, "Warning: Hex decoding of query %s failed\n",
+tok);
+   ret = 1;
+   goto DONE;
+}
+
+*query_string = tok;
+
+  DONE:
+if ((ret % 2) != 0)
+   fprintf (stderr, "%s invalid input line %s\n",
+ret == 1 ? "Warning: Ignoring" : "Error: Failing at",
+line_for_error);
+
+talloc_free (line_for_error);
+return ret;
+}
+
+static inline void
+message_error (notmuch_message_t *message,
+  notmuch_status_t status,
+  const char *format, ...)
+{
+va_list va_args;
+
+va_start (va_args, format);
+
+vfprintf (stderr, format, va_args);
+fprintf (stderr, "Message-ID: %s\n", notmuch_message_get_message_id 
(message));
+fprintf (stderr, "Status: %s\n", notmuch_status_to_string (status));
+}
+
+notmuch_status_t
+tag_op_list_apply (notmuch_message_t *message,
+  tag_op_list_t *list,
+  tag_op_flag_t flags)
+{
+size_t i;
+notmuch_status_t status = 0;
+tag_operation_t *tag_ops = list->ops;
+
+status = notmuch_message_freeze (message);
+if (status) {
+   message_error 

[Patch v4 03/10] util: add string-util.[ch]

[da...@tethera.net]

From: David Bremner 

This is to give a home to strtok_len. It's a bit silly to add a header
for one routine, but it needs to be shared between several compilation
units (or at least that's the most natural design).
---
 util/Makefile.local |3 ++-
 util/string-util.c  |   34 ++
 util/string-util.h  |   22 ++
 3 files changed, 58 insertions(+), 1 deletion(-)
 create mode 100644 util/string-util.c
 create mode 100644 util/string-util.h

diff --git a/util/Makefile.local b/util/Makefile.local
index 3ca623e..a11e35b 100644
--- a/util/Makefile.local
+++ b/util/Makefile.local
@@ -3,7 +3,8 @@
 dir := util
 extra_cflags += -I$(srcdir)/$(dir)

-libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c
+libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c \
+ $(dir)/string-util.c

 libutil_modules := $(libutil_c_srcs:.c=.o)

diff --git a/util/string-util.c b/util/string-util.c
new file mode 100644
index 000..44f8cd3
--- /dev/null
+++ b/util/string-util.c
@@ -0,0 +1,34 @@
+/* string-util.c -  Extra or enhanced routines for null terminated strings.
+ *
+ * Copyright (c) 2012 Jani Nikula
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Jani Nikula 
+ */
+
+
+#include "string-util.h"
+
+char *
+strtok_len (char *s, const char *delim, size_t *len)
+{
+/* skip initial delims */
+s += strspn (s, delim);
+
+/* length of token */
+*len = strcspn (s, delim);
+
+return *len ? s : NULL;
+}
diff --git a/util/string-util.h b/util/string-util.h
new file mode 100644
index 000..ac7676c
--- /dev/null
+++ b/util/string-util.h
@@ -0,0 +1,22 @@
+#ifndef _STRING_UTIL_H
+#define _STRING_UTIL_H
+
+#include 
+
+/* like strtok(3), but without state, and doesn't modify s.  Return
+ * value is indicated by pointer and length, not null terminator.
+ *
+ * Usage pattern:
+ *
+ * const char *tok = input;
+ * const char *delim = " \t";
+ * size_t tok_len = 0;
+ *
+ * while ((tok = strtok_len (tok + tok_len, delim, _len)) != NULL) {
+ * // do stuff with string tok of length tok_len
+ * }
+ */
+
+char *strtok_len (char *s, const char *delim, size_t *len);
+
+#endif
-- 
1.7.10.4

[Patch v4 02/10] test: add sanity check for dump --format=batch-tag.

[da...@tethera.net]

From: David Bremner 

It's important this does not rely on restore, since it hasn't been
written yet.
---
 test/dump-restore |   13 +
 1 file changed, 13 insertions(+)

diff --git a/test/dump-restore b/test/dump-restore
index bf31266..b4c807f 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -85,6 +85,19 @@ test_begin_subtest "dump --output=outfile -- from:cworth"
 notmuch dump --output=dump-outfile-dash-inbox.actual -- from:cworth
 test_expect_equal_file dump-cworth.expected dump-outfile-dash-inbox.actual

+test_begin_subtest "Check for a safe set of message-ids"
+notmuch search --output=messages from:cworth | sed s/^id:// > EXPECTED
+notmuch search --output=messages from:cworth | sed s/^id:// |\
+   $TEST_DIRECTORY/hex-xcode --direction=encode > OUTPUT
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest "format=batch-tag, dump sanity check."
+notmuch dump --format=sup from:cworth | cut -f1 -d' ' | \
+sort > EXPECTED.$test_count
+notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
+sort > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
 test_begin_subtest 'roundtripping random message-ids and tags'
 test_subtest_known_broken
 ${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
-- 
1.7.10.4

[Patch v4 01/10] notmuch-dump: add --format=(batch-tag|sup)

[da...@tethera.net]

From: David Bremner 

sup is the old format, and remains the default, at least until
restore is converted to parse this format.

Each line of the batch-tag format is modelled on the syntax of notmuch tag:
- "notmuch tag" is omitted from the front of the line
- The dump format only uses query strings of a single message-id.
- Each space seperated tag/message-id is 'hex-encoded' to remove
  trouble-making characters.
- It is permitted (and will be useful) for there to be no tags before
  the query.

In particular this format won't have the same problem with e.g. spaces
in message-ids or tags; they will be round-trip-able.
---
 dump-restore-private.h |   13 +
 notmuch-dump.c |   48 ++--
 2 files changed, 55 insertions(+), 6 deletions(-)
 create mode 100644 dump-restore-private.h

diff --git a/dump-restore-private.h b/dump-restore-private.h
new file mode 100644
index 000..896a004
--- /dev/null
+++ b/dump-restore-private.h
@@ -0,0 +1,13 @@
+#ifndef DUMP_RESTORE_PRIVATE_H
+#define DUMP_RESTORE_PRIVATE_H
+
+#include "hex-escape.h"
+#include "command-line-arguments.h"
+
+typedef enum dump_formats {
+DUMP_FORMAT_AUTO,
+DUMP_FORMAT_BATCH_TAG,
+DUMP_FORMAT_SUP
+} dump_format_t;
+
+#endif
diff --git a/notmuch-dump.c b/notmuch-dump.c
index 88f598a..d2dad40 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -19,6 +19,7 @@
  */

 #include "notmuch-client.h"
+#include "dump-restore-private.h"

 int
 notmuch_dump_command (unused (void *ctx), int argc, char *argv[])
@@ -43,7 +44,13 @@ notmuch_dump_command (unused (void *ctx), int argc, char 
*argv[])
 char *output_file_name = NULL;
 int opt_index;

+int output_format = DUMP_FORMAT_SUP;
+
 notmuch_opt_desc_t options[] = {
+   { NOTMUCH_OPT_KEYWORD, _format, "format", 'f',
+ (notmuch_keyword_t []){ { "sup", DUMP_FORMAT_SUP },
+ { "batch-tag", DUMP_FORMAT_BATCH_TAG },
+ { 0, 0 } } },
{ NOTMUCH_OPT_STRING, _file_name, "output", 'o', 0  },
{ 0, 0, 0, 0, 0 }
 };
@@ -83,27 +90,56 @@ notmuch_dump_command (unused (void *ctx), int argc, char 
*argv[])
  */
 notmuch_query_set_sort (query, NOTMUCH_SORT_UNSORTED);

+char *buffer = NULL;
+size_t buffer_size = 0;
+
 for (messages = notmuch_query_search_messages (query);
 notmuch_messages_valid (messages);
 notmuch_messages_move_to_next (messages)) {
int first = 1;
+   const char *message_id;
+
message = notmuch_messages_get (messages);
+   message_id = notmuch_message_get_message_id (message);

-   fprintf (output,
-"%s (", notmuch_message_get_message_id (message));
+   if (output_format == DUMP_FORMAT_SUP) {
+   fprintf (output, "%s (", message_id);
+   }

for (tags = notmuch_message_get_tags (message);
 notmuch_tags_valid (tags);
 notmuch_tags_move_to_next (tags)) {
-   if (! first)
-   fprintf (output, " ");
+   const char *tag_str = notmuch_tags_get (tags);

-   fprintf (output, "%s", notmuch_tags_get (tags));
+   if (! first)
+   fputs (" ", output);

first = 0;
+
+   if (output_format == DUMP_FORMAT_SUP) {
+   fputs (tag_str, output);
+   } else {
+   if (hex_encode (notmuch, tag_str,
+   , _size) != HEX_SUCCESS) {
+   fprintf (stderr, "Error: failed to hex-encode tag %s\n",
+tag_str);
+   return 1;
+   }
+   fprintf (output, "+%s", buffer);
+   }
}

-   fprintf (output, ")\n");
+   if (output_format == DUMP_FORMAT_SUP) {
+   fputs (")\n", output);
+   } else {
+   if (hex_encode (notmuch, message_id,
+   , _size) != HEX_SUCCESS) {
+   fprintf (stderr, "Error: failed to hex-encode msg-id %s\n",
+message_id);
+   return 1;
+   }
+   fprintf (output, " -- id:%s\n", buffer);
+   }

notmuch_message_destroy (message);
 }
-- 
1.7.10.4

v4 of Hex Dump/Restore patches

[da...@tethera.net]

This obsoletes the series at

 id:1354843607-17980-1-git-send-email-david at tethera.net

One new patch:

[Patch v4 08/10] test/dump-restore: add test for warning/error

Other changes since the last series are as follows:

commit 16ca0f8126fafd49266ffa02534f2e9b73c03027
Author: David Bremner 
Date:   Fri Dec 7 20:19:21 2012 -0400

fixup for test/dump-restore

diff --git a/test/dump-restore b/test/dump-restore
index 2532cbb..b267792 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -212,7 +212,7 @@ test_expect_equal_file EXPECTED OUTPUT

 test_begin_subtest 'roundtripping random message-ids and tags'

-${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG}
+${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
--num-messages=100

  notmuch dump --format=batch-tag| \

commit 9d864bd724bd2d2b26b5d52e3e2f28b4fff2046f
Author: David Bremner 
Date:   Fri Dec 7 20:32:50 2012 -0400

fixup for tag-util.c: put * in right palce

diff --git a/tag-util.h b/tag-util.h
index df05d72..6674674 100644
--- a/tag-util.h
+++ b/tag-util.h
@@ -62,8 +62,8 @@ parse_tag_line (void *ctx, char *line,
  * ctx is passed to talloc
  */

-tag_op_list_t
-*tag_op_list_create (void *ctx);
+tag_op_list_t *
+tag_op_list_create (void *ctx);

 /*
  * Add a tag operation (delete iff remove == TRUE) to a list.

commit d2cead797981e8992849e1d96ad95177b42290e2
Author: David Bremner 
Date:   Fri Dec 7 20:57:52 2012 -0400

fixup for id:87txrxxyzp.fsf at nikula.org

diff --git a/tag-util.c b/tag-util.c
index 3d54e9e..a927363 100644
--- a/tag-util.c
+++ b/tag-util.c
@@ -24,9 +24,15 @@ parse_tag_line (void *ctx, char *line,
 {
 char *tok = line;
 size_t tok_len = 0;
-char *line_for_error = talloc_strdup (ctx, line);
+char *line_for_error;
 int ret = 0;

+line_for_error = talloc_strdup (ctx, line);
+if (line_for_error == NULL) {
+   fprintf (stderr, "Error: out of memory\n");
+   return -1;
+}
+
 chomp_newline (line);

 /* remove leading space */
@@ -58,7 +64,7 @@ parse_tag_line (void *ctx, char *line,

/* If tag is terminated by NUL, there's no query string. */
if (*(tok + tok_len) == '\0') {
-   fprintf (stderr, "no query string: %s\n", line_for_error);
+   fprintf (stderr, "Warning: no query string: %s\n", line_for_error);
ret = 1;
goto DONE;
}
@@ -71,19 +77,21 @@ parse_tag_line (void *ctx, char *line,

/* Maybe refuse empty tags. */
if (! (flags & TAG_FLAG_BE_GENEROUS) && *tag == '\0') {
-   fprintf (stderr, "Error: empty tag: %s\n", line_for_error);
+   fprintf (stderr, "Warning: empty tag: %s\n", line_for_error);
+   ret = 1;
goto DONE;
}

/* Decode tag. */
if (hex_decode_inplace (tag) != HEX_SUCCESS) {
-   fprintf (stderr, "Hex decoding of tag %s failed\n",
+   fprintf (stderr, "Warning: Hex decoding of tag %s failed\n",
 tag);
ret = 1;
goto DONE;
}

if (tag_op_list_append (ctx, tag_ops, tag, remove)) {
+   /* diagnostics already printed */
ret = -1;
goto DONE;
}
@@ -98,7 +106,7 @@ parse_tag_line (void *ctx, char *line,

 /* tok now points to the query string */
 if (hex_decode_inplace (tok) != HEX_SUCCESS) {
-   fprintf (stderr, "Hex decoding of query %s failed\n",
+   fprintf (stderr, "Warning: Hex decoding of query %s failed\n",
 tok);
ret = 1;
goto DONE;

commit 162c20ab6d9b9eef56cb2b966a7144a61557eade
Author: David Bremner 
Date:   Sat Dec 8 07:29:56 2012 -0400

fixup for id:87624chmfw.fsf at qmul.ac.uk

update comment

diff --git a/util/string-util.h b/util/string-util.h
index 696da40..ac7676c 100644
--- a/util/string-util.h
+++ b/util/string-util.h
@@ -3,7 +3,10 @@

 #include 

-/* like strtok(3), but without state, and doesn't modify s. usage pattern:
+/* like strtok(3), but without state, and doesn't modify s.  Return
+ * value is indicated by pointer and length, not null terminator.
+ *
+ * Usage pattern:
  *
  * const char *tok = input;
  * const char *delim = " \t";

commit 590adeec072f235861154b930bfb10bedbe37e8a
Author: David Bremner 
Date:   Sat Dec 8 07:42:44 2012 -0400

fixup for id:8738zghl6d.fsf at qmul.ac.uk

diff --git a/tag-util.c b/tag-util.c
index a927363..c97d240 100644
--- a/tag-util.c
+++ b/tag-util.c
@@ -98,8 +98,6 @@ parse_tag_line (void *ctx, char *line,
 }

 if (tok == NULL) {
-   fprintf (stderr, "Warning: Ignoring invalid input line: %s\n",
-line_for_error);
ret = 1;
goto DONE;
 }
@@ -113,7 +111,13 @@ parse_tag_line (void *ctx, char *line,
 }

 *query_string = tok;
+
   DONE:
+if (ret != 0)
+   fprintf (stderr, "% invalid input line %s\n",
+ret == 1 ? "Warning: Ignoring" : "Error: Failing at",
+  

[Patch v3b 4/9] tag-util.[ch]: New files for common tagging routines

[Mark Walters]

On Fri, 07 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> These are meant to be shared between notmuch-tag and notmuch-restore.
>
> The bulk of the routines implement a "tag operation list" abstract
> data type act as a structured representation of a set of tag
> operations (typically coming from a single tag command or line of
> input).
> ---
>  Makefile.local |1 +
>  tag-util.c |  278 
> 
>  tag-util.h |  119 
>  3 files changed, 398 insertions(+)
>  create mode 100644 tag-util.c
>  create mode 100644 tag-util.h
>
> diff --git a/Makefile.local b/Makefile.local
> index 2b91946..854867d 100644
> --- a/Makefile.local
> +++ b/Makefile.local
> @@ -274,6 +274,7 @@ notmuch_client_srcs = \
>   query-string.c  \
>   mime-node.c \
>   crypto.c\
> + tag-util.c
>  
>  notmuch_client_modules = $(notmuch_client_srcs:.c=.o)
>  
> diff --git a/tag-util.c b/tag-util.c
> new file mode 100644
> index 000..932ee7f
> --- /dev/null
> +++ b/tag-util.c
> @@ -0,0 +1,278 @@
> +#include 
> +#include "string-util.h"
> +#include "tag-util.h"
> +#include "hex-escape.h"
> +
> +#define TAG_OP_LIST_INITIAL_SIZE 10
> +
> +struct _tag_operation_t {
> +const char *tag;
> +notmuch_bool_t remove;
> +};
> +
> +struct _tag_op_list_t {
> +tag_operation_t *ops;
> +size_t count;
> +size_t size;
> +};
> +
> +int
> +parse_tag_line (void *ctx, char *line,
> + tag_op_flag_t flags,
> + char **query_string,
> + tag_op_list_t *tag_ops)
> +{
> +char *tok = line;
> +size_t tok_len = 0;
> +char *line_for_error = talloc_strdup (ctx, line);
> +int ret = 0;
> +
> +chomp_newline (line);
> +
> +/* remove leading space */
> +while (*tok == ' ' || *tok == '\t')
> + tok++;
> +
> +/* Skip empty and comment lines. */
> +if (*tok == '\0' || *tok == '#') {
> + ret = 1;
> + goto DONE;
> +}
> +
> +tag_op_list_reset (tag_ops);
> +
> +/* Parse tags. */
> +while ((tok = strtok_len (tok + tok_len, " ", _len)) != NULL) {
> + notmuch_bool_t remove;
> + char *tag;
> +
> + /* Optional explicit end of tags marker. */
> + if (tok_len == 2 && strncmp (tok, "--", tok_len) == 0) {
> + tok = strtok_len (tok + tok_len, " ", _len);
> + break;
> + }
> +
> + /* Implicit end of tags. */
> + if (*tok != '-' && *tok != '+')
> + break;
> +
> + /* If tag is terminated by NUL, there's no query string. */
> + if (*(tok + tok_len) == '\0') {
> + fprintf (stderr, "no query string: %s\n", line_for_error);
> + ret = 1;
> + goto DONE;
> + }
> +
> + /* Terminate, and start next token after terminator. */
> + *(tok + tok_len++) = '\0';
> +
> + remove = (*tok == '-');
> + tag = tok + 1;
> +
> + /* Maybe refuse empty tags. */
> + if (! (flags & TAG_FLAG_BE_GENEROUS) && *tag == '\0') {
> + fprintf (stderr, "Error: empty tag: %s\n", line_for_error);
> + goto DONE;
> + }
> +
> + /* Decode tag. */
> + if (hex_decode_inplace (tag) != HEX_SUCCESS) {
> + fprintf (stderr, "Hex decoding of tag %s failed\n",
> +  tag);
> + ret = 1;
> + goto DONE;
> + }
> +
> + if (tag_op_list_append (ctx, tag_ops, tag, remove)) {
> + ret = -1;
> + goto DONE;
> + }
> +}
> +
> +if (tok == NULL) {
> + fprintf (stderr, "Warning: Ignoring invalid input line: %s\n",
> +  line_for_error);
> + ret = 1;
> + goto DONE;
> +}
> +
> +/* tok now points to the query string */
> +if (hex_decode_inplace (tok) != HEX_SUCCESS) {
> + fprintf (stderr, "Hex decoding of query %s failed\n",
> +  tok);


For these hex_decode errors would it be worth printing the full line as
well as the bit that fails hex_decode? Perhaps put something under DONE:
to print the whole line if ret is not success?

Otherwise LGTM

Mark


> + ret = 1;
> + goto DONE;
> +}
> +
> +*query_string = tok;
> +  DONE:
> +talloc_free (line_for_error);
> +return ret;
> +}
> +
> +static inline void
> +message_error (notmuch_message_t *message,
> +notmuch_status_t status,
> +const char *format, ...)
> +{
> +va_list va_args;
> +
> +va_start (va_args, format);
> +
> +vfprintf (stderr, format, va_args);
> +fprintf (stderr, "Message-ID: %s\n", notmuch_message_get_message_id 
> (message));
> +fprintf (stderr, "Status: %s\n", notmuch_status_to_string (status));
> +}
> +
> +notmuch_status_t
> +tag_op_list_apply (notmuch_message_t *message,
> +tag_op_list_t *list,
> +tag_op_flag_t flags)
> +{
> +size_t i;
> +notmuch_status_t status = 0;
> +tag_operation_t *tag_ops = list->ops;
> +
> +status = notmuch_message_freeze (message);
> 

[Patch v3b 3/9] util: add string-util.[ch]

[Mark Walters]

On Fri, 07 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> This is to give a home to strtok_len. It's a bit silly to add a header
> for one routine, but it needs to be shared between several compilation
> units (or at least that's the most natural design).
> ---
>  util/Makefile.local |3 ++-
>  util/string-util.c  |   34 ++
>  util/string-util.h  |   19 +++
>  3 files changed, 55 insertions(+), 1 deletion(-)
>  create mode 100644 util/string-util.c
>  create mode 100644 util/string-util.h
>
> diff --git a/util/Makefile.local b/util/Makefile.local
> index 3ca623e..a11e35b 100644
> --- a/util/Makefile.local
> +++ b/util/Makefile.local
> @@ -3,7 +3,8 @@
>  dir := util
>  extra_cflags += -I$(srcdir)/$(dir)
>  
> -libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c
> +libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c \
> +   $(dir)/string-util.c
>  
>  libutil_modules := $(libutil_c_srcs:.c=.o)
>  
> diff --git a/util/string-util.c b/util/string-util.c
> new file mode 100644
> index 000..44f8cd3
> --- /dev/null
> +++ b/util/string-util.c
> @@ -0,0 +1,34 @@
> +/* string-util.c -  Extra or enhanced routines for null terminated strings.
> + *
> + * Copyright (c) 2012 Jani Nikula
> + *
> + * This program is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, either version 3 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see http://www.gnu.org/licenses/ .
> + *
> + * Author: Jani Nikula 
> + */
> +
> +
> +#include "string-util.h"
> +
> +char *
> +strtok_len (char *s, const char *delim, size_t *len)
> +{
> +/* skip initial delims */
> +s += strspn (s, delim);
> +
> +/* length of token */
> +*len = strcspn (s, delim);
> +
> +return *len ? s : NULL;
> +}
> diff --git a/util/string-util.h b/util/string-util.h
> new file mode 100644
> index 000..696da40
> --- /dev/null
> +++ b/util/string-util.h
> @@ -0,0 +1,19 @@
> +#ifndef _STRING_UTIL_H
> +#define _STRING_UTIL_H
> +
> +#include 
> +
> +/* like strtok(3), but without state, and doesn't modify s. usage pattern:
> + *
> + * const char *tok = input;
> + * const char *delim = " \t";
> + * size_t tok_len = 0;
> + *
> + * while ((tok = strtok_len (tok + tok_len, delim, _len)) != NULL) {
> + * // do stuff with string tok of length tok_len
> + * }

Hi

Some possibly silly comments on the comment. strtok(3) seems to return a
null terminated string and this does not. Might be worth including in
the comment? Secondly every time I read this I read "..modify s. usage
pattern" as one ongoing sentence with s. an abbreviation. Perhaps a
capital u? and maybe str or string rather than s?

Best wishes

Mark

> + */
> +
> +char *strtok_len (char *s, const char *delim, size_t *len);
> +
> +#endif
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] test: Fix UTF-8 JSON tests in Python 3

[David Bremner]

Austin Clements  writes:

> This patch sets the PYTHONIOENCODING
> environment variable to utf-8 when invoking json.tool to override
> Python's default encoding choice.

Pushed. One day I'd like to experience, "yes, python really got that
aspect of encoding right" ;).

d

[PATCH 0/3] test fixes, portability

[David Bremner]

Jani Nikula  writes:

> I had a glance at a 'make test' report on OS X [1], and tried to fix a
> few things from there. While at it, noticed that count test is horribly
> broken.

pushed,

d

[PATCH v5 0/5] New output format sexp (Lisp S-Expressions)

[David Bremner]

Peter Feigl  writes:

> This patch series adds a new output format "sexp" to notmuch-reply,
> notmuch-show and notmuch-search. These are useful for the Android mobile
> client, Emacs and perhaps other Lisp programs as well.

pushed, looking forward to an android client ;)

d

[PATCH 00/10] CLI output versioning

[Mark Walters]

Hi

Overall this series looks good. As we discussed on irc I think i would
prefer global NOTMUCH_SCHEMA_MIN as I am a little worried about these
proliferating (eg if someone decides text output also needs versioning
etc) In addition, if we do find the distinction useful it would be easy
to add at a later date.

One tiny comment on the manpage updates: now that you mention two return
values explicitly should the other possibilities be mentioned too or are
they so obvious it is not needed?

Would it be worth having some emacs test for the error handling? (eg set
notmuch-command to something giving some stderr and an error) Inherently
these code paths won't be tested much so I think tests could be
particularly useful.

Best wishes

Mark






On Mon, 03 Dec 2012, Austin Clements  wrote:
> (Sorry; I forgot to include a cover letter.)
>
> This series is intended to help with our long-standing output format
> versioning issue.  While the JSON format is amenable to extension,
> there's still a high barrier to extensions because of the need to
> support them going forward, and an even higher barrier to modifications
> that break backwards compatibility.  Versioning will make the format
> more dynamic, enabling us to easily improve and iterate on it.  It will
> also address the slew of confusing bugs that people encounter when they
> use a mismatched CLI and front-end.
>
> On IRC we've talking about adding version information to the output
> format itself.  This series takes a different and, I think, better
> approach: callers request a specific output format version on the
> command line.  This allows notmuch to remain backwards compatible with
> older format versions when it's easy or necessary.  This also doesn't
> require shoehorning a version number into the output, which would be
> awkward for both the CLI and the consumer.
>
> I called the argument --use-schema, but I'm open to other suggestions.
> --use-schema is technically accurate, but perhaps not as self-describing
> as something like --schema-version or --format-version (to parallel
> --format).
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] TODO: date range queries - check

[David Bremner]

Jameson Graef Rollins  writes:

> On Fri, Dec 07 2012, Jani Nikula  wrote:
>> Fine by me. I was just trying to clean up the file a bit, that's
>> all. The only downside I can think of is potential new users stumbling
>> on this and thinking we still don't have date queries. *shrug*.
>
> That's why it should probably just be modified, instead of removed.
>

Patch? Concrete wording suggestion?

d

[PATCH 07/10] emacs: Use --use-schema for search

[Mark Walters]

On Sun, 02 Dec 2012, Austin Clements  wrote:
> We detect the special exit statuses and use these to produce specific
> diagnostic messages.
> ---
>  emacs/notmuch-lib.el |   32 
>  emacs/notmuch.el |   17 +
>  2 files changed, 45 insertions(+), 4 deletions(-)
>
> diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
> index 9402456..49b0da6 100644
> --- a/emacs/notmuch-lib.el
> +++ b/emacs/notmuch-lib.el
> @@ -325,6 +325,38 @@ string), a property list of face attributes, or a list 
> of these."
>   (put-text-property pos next 'face (cons face cur))
>   (setq pos next)
>  
> +(defun notmuch-pop-up-error (msg)
> +  "Pop up an error buffer displaying MSG."
> +
> +  (let ((buf (get-buffer-create "*Notmuch errors*")))
> +(with-current-buffer buf
> +  (view-mode)
> +  (let ((inhibit-read-only t))
> + (erase-buffer)
> + (insert msg)
> + (unless (bolp)
> +   (insert "\n"))
> + (goto-char (point-min
> +(pop-to-buffer buf)))

I am not sure about the erase-buffer in the above: do we definitely want
to remove all previous error information? For version mismatch possibly
we do but in patch 9 it is done for all show command-line error returns.
Incidentally why does show always pop-up an error but search only for
version-mismatches?

Otherwise this looks good to me (but I am not that familiar with lisp
error handling)

Best wishes

Mark






> +(defun notmuch-version-mismatch-error (exit-status)
> +  "Signal a schema version mismatch error.
> +
> +EXIT-STATUS must be the exit status of the notmuch CLI command,
> +and must have the value 20 or 21.  This function will pop up an
> +error buffer with a descriptive message and signal an error."
> +  (cond ((= exit-status 20)
> +  (notmuch-pop-up-error "Error: Version mismatch.
> +Emacs requested an older output format than supported by the notmuch CLI.
> +You may need to restart Emacs or upgrade your notmuch Emacs package."))
> + ((= exit-status 21)
> +  (notmuch-pop-up-error "Error: Version mismatch.
> +Emacs requested a newer output format than supported by the notmuch CLI.
> +You may need to restart Emacs or upgrade your notmuch package."))
> + (t
> +  (error "Bad exit status %d" exit-status)))
> +  (error "notmuch CLI version mismatch"))
> +
>  ;; Compatibility functions for versions of emacs before emacs 23.
>  ;;
>  ;; Both functions here were copied from emacs 23 with the following 
> copyright:
> diff --git a/emacs/notmuch.el b/emacs/notmuch.el
> index f9454d8..e1f28ca 100644
> --- a/emacs/notmuch.el
> +++ b/emacs/notmuch.el
> @@ -644,6 +644,7 @@ of the result."
>   (exit-status (process-exit-status proc))
>   (never-found-target-thread nil))
>  (when (memq status '(exit signal))
> +  (catch 'return
>   (kill-buffer (process-get proc 'parse-buf))
>   (if (buffer-live-p buffer)
>   (with-current-buffer buffer
> @@ -655,8 +656,16 @@ of the result."
> (insert "Incomplete search results (search process was 
> killed).\n"))
> (when (eq status 'exit)
>   (insert "End of search results.")
> - (unless (= exit-status 0)
> -   (insert (format " (process returned %d)" exit-status)))
> + (cond ((or (= exit-status 20) (= exit-status 21))
> +(kill-buffer)
> +(condition-case err
> +(notmuch-version-mismatch-error exit-status)
> +  ;; Strange things happen when you signal
> +  ;; an error from a sentinel.
> +  (error (throw 'return nil
> +   ((/= exit-status 0)
> +(insert (format " (process returned %d)"
> +exit-status
>   (insert "\n")
>   (if (and atbob
>(not (string= notmuch-search-target-thread 
> "found")))
> @@ -664,7 +673,7 @@ of the result."
> (when (and never-found-target-thread
>  notmuch-search-target-line)
> (goto-char (point-min))
> -   (forward-line (1- notmuch-search-target-line
> +   (forward-line (1- notmuch-search-target-line)
>  
>  (defcustom notmuch-search-line-faces '(("unread" :weight bold)
>  ("flagged" :foreground "blue"))
> @@ -938,7 +947,7 @@ Other optional parameters are used as follows:
>   (let ((proc (start-process
>"notmuch-search" buffer
>notmuch-command "search"
> -  "--format=json"
> +  "--format=json" "--use-schema=1"
>(if oldest-first
>"--sort=oldest-first"
>  "--sort=newest-first")
> -- 
> 1.7.10.4
>
> 

[PATCH 06/10] emacs: Fix bug in resynchronizing after a JSON parse error

[Mark Walters]

This patch looks good to me. Should it go in ahead of the rest of the
series as it essentially a bugfix?

Best wishes

Mark

On Sun, 02 Dec 2012, Austin Clements  wrote:
> Previously, if the input stream consisted only of an error message,
> notmuch-json-begin-compound would signal a (wrong-type-argument
> number-or-marker-p nil) error when reaching the end of the error
> message.  This happened because notmuch-json-scan-to-value would think
> that it reached a value and put the parser into the 'value state.
> Even after notmuch-json-begin-compound signaled the syntax error, the
> parser would remain in this state and when the resynchronization logic
> reached the end of the buffer, the parser would fail because the
> 'value state indicates that characters are available.
>
> This fixes this problem by restoring the parser's previous state if it
> encounters a syntax error.
> ---
>  emacs/notmuch-lib.el |   22 +-
>  1 file changed, 13 insertions(+), 9 deletions(-)
>
> diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
> index 1d0ec17..9402456 100644
> --- a/emacs/notmuch-lib.el
> +++ b/emacs/notmuch-lib.el
> @@ -474,15 +474,19 @@ Entering JSON objects is currently unimplemented."
>(with-current-buffer (notmuch-json-buffer jp)
>  ;; Disallow terminators
>  (setf (notmuch-json-allow-term jp) nil)
> -(or (notmuch-json-scan-to-value jp)
> - (if (/= (char-after) ?\[)
> - (signal 'json-readtable-error (list "expected '['"))
> -   (forward-char)
> -   (push ?\] (notmuch-json-term-stack jp))
> -   ;; Expect a value or terminator next
> -   (setf (notmuch-json-next jp) 'expect-value
> - (notmuch-json-allow-term jp) t)
> -   t
> +;; Save "next" so we can restore it if there's a syntax error
> +(let ((saved-next (notmuch-json-next jp)))
> +  (or (notmuch-json-scan-to-value jp)
> +   (if (/= (char-after) ?\[)
> +   (progn
> + (setf (notmuch-json-next jp) saved-next)
> + (signal 'json-readtable-error (list "expected '['")))
> + (forward-char)
> + (push ?\] (notmuch-json-term-stack jp))
> + ;; Expect a value or terminator next
> + (setf (notmuch-json-next jp) 'expect-value
> +   (notmuch-json-allow-term jp) t)
> + t)
>  
>  (defun notmuch-json-read (jp)
>"Parse the value at point in JP's buffer.
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Tags with spaces

[Tomi Ollila]

On Sat, Dec 08 2012, "Jason A. Donenfeld"  wrote:

> Do we support them? Are we going to?
>
> This doesn't work:
> notmuch search -tag:"This has spaces"
>
> Nor does:
> notmuch search "-tag:This has spaces"

This might work:

notmuch search "-tag:'This has spaces'"

Tags with spaces

[Jason A. Donenfeld]

Do we support them? Are we going to?

This doesn't work:
notmuch search -tag:"This has spaces"

Nor does:
notmuch search "-tag:This has spaces"

Tags with spaces

[Austin Clements]

Quoth Tomi Ollila on Dec 08 at  8:21 am:
> On Sat, Dec 08 2012, "Jason A. Donenfeld"  wrote:
> 
> > Do we support them? Are we going to?
> >
> > This doesn't work:
> > notmuch search -tag:"This has spaces"
> >
> > Nor does:
> > notmuch search "-tag:This has spaces"
> 
> This might work:
> 
> notmuch search "-tag:'This has spaces'"

Xapian's query parser doesn't recognize single quotes, unfortunately.

When in doubt,

$ NOTMUCH_DEBUG_QUERY=1 notmuch search '-tag:"This tag has spaces"'
Final query is:
Xapian::Query((Tmail AND 0 * KThis tag has spaces))

$ NOTMUCH_DEBUG_QUERY=1 notmuch search "-tag:'This tag has spaces'"
Final query is:
Xapian::Query((Tmail AND ((Ztag:(pos=1) AND Zhas:(pos=2) AND Zspace:(pos=3)) 
AND_NOT K'This)))

Tags with spaces

[Austin Clements]

Quoth Jason A. Donenfeld on Dec 08 at  4:44 am:
> Do we support them? Are we going to?
> 
> This doesn't work:
> notmuch search -tag:"This has spaces"
> 
> Nor does:
> notmuch search "-tag:This has spaces"

Tags with spaces are supported, but your shell will strip away the
quotes you've used in both examples before notmuch ever sees them.
You'll need both shell quoting and Xapian quoting in this case.  Try
  notmuch search -- '-tag:"This has spaces"'
This should be easier from a front-end, since it will take care of any
necessary shell quoting.

[Patch v3b 9/9] tag-util: optimization of tag application

[Jani Nikula]

On Fri, 07 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> The idea is not to bother with restore operations if they don't change
> the set of tags. This is actually a relatively common case.
>
> In order to avoid fancy datastructures, this method is quadratic in
> the number of tags; at least on my mail database this doesn't seem to
> be a big problem.
> ---
>  tag-util.c |   66 
> 
>  1 file changed, 66 insertions(+)
>
> diff --git a/tag-util.c b/tag-util.c
> index 932ee7f..3d54e9e 100644
> --- a/tag-util.c
> +++ b/tag-util.c
> @@ -124,6 +124,69 @@ message_error (notmuch_message_t *message,
>  fprintf (stderr, "Status: %s\n", notmuch_status_to_string (status));
>  }
>  
> +static int
> +makes_changes (notmuch_message_t *message,
> +tag_op_list_t *list,
> +tag_op_flag_t flags)
> +{
> +
> +size_t i;
> +
> +notmuch_tags_t *tags;
> +notmuch_bool_t changes = FALSE;
> +
> +/* First, do we delete an existing tag? */
> +changes = FALSE;
> +for (tags = notmuch_message_get_tags (message);
> +  ! changes && notmuch_tags_valid (tags);
> +  notmuch_tags_move_to_next (tags)) {
> + const char *cur_tag = notmuch_tags_get (tags);
> + int last_op =  (flags & TAG_FLAG_REMOVE_ALL) ? -1 : 0;
> +
> + /* slight contortions to count down with an unsigned index */
> + for (i = list->count; i-- > 0; /*nothing*/) {
> + if (strcmp (cur_tag, list->ops[i].tag) == 0) {
> + last_op = list->ops[i].remove ? -1 : 1;
> + break;
> + }
> + }

After some eyeballing it looks correct, but not not exactly pretty. If
you insist on unsigned, you could also have a regular (i = 0; i <
list->count; i++) and use j = list->count - i - 1; within the block. But
that's just style bikeshedding after convincing you to use a count down
loop in the first place...

Otherwise, the patch LGTM.


> +
> + changes = (last_op == -1);
> +}
> +notmuch_tags_destroy (tags);
> +
> +if (changes)
> + return TRUE;
> +
> +/* Now check for adding new tags */
> +for (i = 0; i < list->count; i++) {
> + notmuch_bool_t exists = FALSE;
> +
> + if (list->ops[i].remove)
> + continue;
> +
> + for (tags = notmuch_message_get_tags (message);
> +  notmuch_tags_valid (tags);
> +  notmuch_tags_move_to_next (tags)) {
> + const char *cur_tag = notmuch_tags_get (tags);
> + if (strcmp (cur_tag, list->ops[i].tag) == 0) {
> + exists = TRUE;
> + break;
> + }
> + }
> + notmuch_tags_destroy (tags);
> +
> + /* the following test is conservative,
> +  * in the sense it ignores cases like +foo ... -foo
> +  * but this is OK from a correctness point of view
> +  */
> + if (! exists)
> + return TRUE;
> +}
> +return FALSE;
> +
> +}
> +
>  notmuch_status_t
>  tag_op_list_apply (notmuch_message_t *message,
>  tag_op_list_t *list,
> @@ -133,6 +196,9 @@ tag_op_list_apply (notmuch_message_t *message,
>  notmuch_status_t status = 0;
>  tag_operation_t *tag_ops = list->ops;
>  
> +if (! (flags & TAG_FLAG_PRE_OPTIMIZED) && ! makes_changes (message, 
> list, flags))
> + return NOTMUCH_STATUS_SUCCESS;
> +
>  status = notmuch_message_freeze (message);
>  if (status) {
>   message_error (message, status, "freezing message");
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[Patch v3b 6/9] test: update dump-restore roundtripping test for batch-tag format

[Jani Nikula]

On Fri, 07 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> Now we can actually round trip these crazy tags and and message ids.
> hex-xcode is no longer needed as it's built in.
> ---
>  test/dump-restore |   17 -
>  1 file changed, 8 insertions(+), 9 deletions(-)
>
> diff --git a/test/dump-restore b/test/dump-restore
> index b4c807f..ce81e6f 100755
> --- a/test/dump-restore
> +++ b/test/dump-restore
> @@ -99,23 +99,22 @@ notmuch dump --format=batch-tag from:cworth | sed 
> 's/^.*-- id://' | \
>  test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
>  
>  test_begin_subtest 'roundtripping random message-ids and tags'
> -test_subtest_known_broken
> -${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
> - --num-messages=10
>  
> - notmuch dump| \
> -  ${TEST_DIRECTORY}/hex-xcode --direction=encode| \
> +${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG}
> + --num-messages=100

What happened to the \ at the end of the line? Hmm, does it matter?

Otherwise, LGTM.

> +
> + notmuch dump --format=batch-tag| \
>sort > EXPECTED.$test_count
>  
>   notmuch tag +this_tag_is_very_unlikely_to_be_random '*'
>  
> - ${TEST_DIRECTORY}/hex-xcode --direction=decode < EXPECTED.$test_count | 
> \
> -  notmuch restore 2>/dev/null
> + notmuch restore --format=batch-tag < EXPECTED.$test_count
>  
> - notmuch dump| \
> -  ${TEST_DIRECTORY}/hex-xcode --direction=encode| \
> + notmuch dump --format=batch-tag| \
>sort > OUTPUT.$test_count
>  
>  test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
>  
>  test_done
> +
> +# Note the database is "poisoned" for sup format at this point.
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH V2 1/1] NEWS: under-the-hood Emacs interface fixes

[Tomi Ollila]

Added the following Emacs Interface NEWS entries:

Catch errors bodypart insertions may throw,
Improved text/calendar content handling and
Disabled coding conversions when reading in
`with-current-notmuch-show-message`.

Thanks to Austin for improved content parts.
---

This replaces id:1354488524-12013-1-git-send-email-tomi.ollila at iki.fi

 NEWS | 21 +
 1 file changed, 21 insertions(+)

diff --git a/NEWS b/NEWS
index 34e7a28..6970178 100644
--- a/NEWS
+++ b/NEWS
@@ -28,6 +28,27 @@ Removal of the deprecated `notmuch-folders` variable
   has now been removed. Any remaining users should migrate to
   `notmuch-saved-searches`.

+Handle errors bodypart insertions may throw
+
+  If displaying the text of a message in show mode causes an error (in
+  the `notmuch-show-insert-part-*` functions), notmuch no longer cuts
+  off thread display at the offending message.  The error is now
+  simply displayed in place of the message.
+
+Improved text/calendar content handling
+
+  Carriage returns in embedded text/calendar content caused insertion
+  of calendar content fail. Now CRs are removed before calling icalendar
+  to extract icalendar data. In case icalendar extraction fails an error
+  is thrown for bodypart insertion function to react upon it.
+
+Disabled coding conversions when reading in `with-current-notmuch-show-message`
+
+  Depending on the user's locale, saving attachments containing 8-bit
+  data may have performed an unintentional encoding conversion,
+  corrupting the saved attachment.  This has been fixed by making
+  `with-current-notmuch-show-message` disable coding conversion.
+
 Library changes
 ---

--
1.8.0

[Patch v3b 5/9] notmuch-restore: add support for input format 'batch-tag'

[Jani Nikula]

LGTM.

On Fri, 07 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> This can be enabled with the new --format=batch-tag command line
> option to "notmuch restore". The input must consist of lines of the
> format:
>
> +|- [...] [--] id:
>
> Each line is interpreted similarly to "notmuch tag" command line
> arguments. The delimiter is one or more spaces ' '. Any characters in
>  and  MAY be hex encoded with %NN where NN is the
> hexadecimal value of the character. Any ' ' and '%' characters in
>  and  MUST be hex encoded (using %20 and %25,
> respectively). Any characters that are not part of  or
>  MUST NOT be hex encoded.
>
> Leading and trailing space ' ' is ignored. Empty lines and lines
> beginning with '#' are ignored.
>
> Commit message mainly stolen from Jani's batch tagging commit, to
> follow.
> ---
>  notmuch-restore.c |  206 
> ++---
>  1 file changed, 131 insertions(+), 75 deletions(-)
>
> diff --git a/notmuch-restore.c b/notmuch-restore.c
> index f03dcac..ceec2d3 100644
> --- a/notmuch-restore.c
> +++ b/notmuch-restore.c
> @@ -19,18 +19,22 @@
>   */
>  
>  #include "notmuch-client.h"
> +#include "dump-restore-private.h"
> +#include "tag-util.h"
> +#include "string-util.h"
> +
> +static volatile sig_atomic_t interrupted;
> +static regex_t regex;
>  
>  static int
> -tag_message (notmuch_database_t *notmuch, const char *message_id,
> -  char *file_tags, notmuch_bool_t remove_all,
> -  notmuch_bool_t synchronize_flags)
> +tag_message (unused (void *ctx),
> +  notmuch_database_t *notmuch,
> +  const char *message_id,
> +  tag_op_list_t *tag_ops,
> +  tag_op_flag_t flags)
>  {
>  notmuch_status_t status;
> -notmuch_tags_t *db_tags;
> -char *db_tags_str;
>  notmuch_message_t *message = NULL;
> -const char *tag;
> -char *next;
>  int ret = 0;
>  
>  status = notmuch_database_find_message (notmuch, message_id, );
> @@ -44,55 +48,62 @@ tag_message (notmuch_database_t *notmuch, const char 
> *message_id,
>  
>  /* In order to detect missing messages, this check/optimization is
>   * intentionally done *after* first finding the message. */
> -if (! remove_all && (file_tags == NULL || *file_tags == '\0'))
> - goto DONE;
> -
> -db_tags_str = NULL;
> -for (db_tags = notmuch_message_get_tags (message);
> -  notmuch_tags_valid (db_tags);
> -  notmuch_tags_move_to_next (db_tags)) {
> - tag = notmuch_tags_get (db_tags);
> -
> - if (db_tags_str)
> - db_tags_str = talloc_asprintf_append (db_tags_str, " %s", tag);
> - else
> - db_tags_str = talloc_strdup (message, tag);
> -}
> +if ((flags & TAG_FLAG_REMOVE_ALL) || tag_op_list_size (tag_ops))
> + tag_op_list_apply (message, tag_ops, flags);
>  
> -if (((file_tags == NULL || *file_tags == '\0') &&
> -  (db_tags_str == NULL || *db_tags_str == '\0')) ||
> - (file_tags && db_tags_str && strcmp (file_tags, db_tags_str) == 0))
> - goto DONE;
> +notmuch_message_destroy (message);
>  
> -notmuch_message_freeze (message);
> +return ret;
> +}
>  
> -if (remove_all)
> - notmuch_message_remove_all_tags (message);
> +static int
> +parse_sup_line (void *ctx, char *line,
> + char **query_str, tag_op_list_t *tag_ops)
> +{
>  
> -next = file_tags;
> -while (next) {
> - tag = strsep (, " ");
> - if (*tag == '\0')
> - continue;
> - status = notmuch_message_add_tag (message, tag);
> - if (status) {
> - fprintf (stderr, "Error applying tag %s to message %s:\n",
> -  tag, message_id);
> - fprintf (stderr, "%s\n", notmuch_status_to_string (status));
> - ret = 1;
> - }
> +regmatch_t match[3];
> +char *file_tags;
> +int rerr;
> +
> +tag_op_list_reset (tag_ops);
> +
> +chomp_newline (line);
> +
> +/* Silently ignore blank lines */
> +if (line[0] == '\0') {
> + return 1;
> +}
> +
> +rerr = xregexec (, line, 3, match, 0);
> +if (rerr == REG_NOMATCH) {
> + fprintf (stderr, "Warning: Ignoring invalid sup format line: %s\n",
> +  line);
> + return 1;
>  }
>  
> -notmuch_message_thaw (message);
> +*query_str = talloc_strndup (ctx, line + match[1].rm_so,
> +  match[1].rm_eo - match[1].rm_so);
> +file_tags = talloc_strndup (ctx, line + match[2].rm_so,
> + match[2].rm_eo - match[2].rm_so);
>  
> -if (synchronize_flags)
> - notmuch_message_tags_to_maildir_flags (message);
> +char *tok = file_tags;
> +size_t tok_len = 0;
>  
> -  DONE:
> -if (message)
> - notmuch_message_destroy (message);
> +tag_op_list_reset (tag_ops);
> +
> +while ((tok = strtok_len (tok + tok_len, " ", _len)) != NULL) {
> +
> + if (*(tok + tok_len) != '\0') {
> + *(tok + tok_len) = '\0';
> + tok_len++;
> +

[Patch v3b 4/9] tag-util.[ch]: New files for common tagging routines

[Jani Nikula]

A few small comments, otherwise LGTM.

On Fri, 07 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> These are meant to be shared between notmuch-tag and notmuch-restore.
>
> The bulk of the routines implement a "tag operation list" abstract
> data type act as a structured representation of a set of tag
> operations (typically coming from a single tag command or line of
> input).
> ---
>  Makefile.local |1 +
>  tag-util.c |  278 
> 
>  tag-util.h |  119 
>  3 files changed, 398 insertions(+)
>  create mode 100644 tag-util.c
>  create mode 100644 tag-util.h
>
> diff --git a/Makefile.local b/Makefile.local
> index 2b91946..854867d 100644
> --- a/Makefile.local
> +++ b/Makefile.local
> @@ -274,6 +274,7 @@ notmuch_client_srcs = \
>   query-string.c  \
>   mime-node.c \
>   crypto.c\
> + tag-util.c
>  
>  notmuch_client_modules = $(notmuch_client_srcs:.c=.o)
>  
> diff --git a/tag-util.c b/tag-util.c
> new file mode 100644
> index 000..932ee7f
> --- /dev/null
> +++ b/tag-util.c
> @@ -0,0 +1,278 @@
> +#include 
> +#include "string-util.h"
> +#include "tag-util.h"
> +#include "hex-escape.h"
> +
> +#define TAG_OP_LIST_INITIAL_SIZE 10
> +
> +struct _tag_operation_t {
> +const char *tag;
> +notmuch_bool_t remove;
> +};
> +
> +struct _tag_op_list_t {
> +tag_operation_t *ops;
> +size_t count;
> +size_t size;
> +};
> +
> +int
> +parse_tag_line (void *ctx, char *line,
> + tag_op_flag_t flags,
> + char **query_string,
> + tag_op_list_t *tag_ops)
> +{
> +char *tok = line;
> +size_t tok_len = 0;
> +char *line_for_error = talloc_strdup (ctx, line);

Check the return value?

> +int ret = 0;
> +
> +chomp_newline (line);
> +
> +/* remove leading space */
> +while (*tok == ' ' || *tok == '\t')
> + tok++;
> +
> +/* Skip empty and comment lines. */
> +if (*tok == '\0' || *tok == '#') {
> + ret = 1;
> + goto DONE;
> +}
> +
> +tag_op_list_reset (tag_ops);
> +
> +/* Parse tags. */
> +while ((tok = strtok_len (tok + tok_len, " ", _len)) != NULL) {
> + notmuch_bool_t remove;
> + char *tag;
> +
> + /* Optional explicit end of tags marker. */
> + if (tok_len == 2 && strncmp (tok, "--", tok_len) == 0) {
> + tok = strtok_len (tok + tok_len, " ", _len);
> + break;
> + }
> +
> + /* Implicit end of tags. */
> + if (*tok != '-' && *tok != '+')
> + break;
> +
> + /* If tag is terminated by NUL, there's no query string. */
> + if (*(tok + tok_len) == '\0') {
> + fprintf (stderr, "no query string: %s\n", line_for_error);
> + ret = 1;

Is this intentionally 1 rather than -1? Should the message have a
"Warning:" prefix if 1, or "Error:" prefix if -1?

> + goto DONE;
> + }
> +
> + /* Terminate, and start next token after terminator. */
> + *(tok + tok_len++) = '\0';
> +
> + remove = (*tok == '-');
> + tag = tok + 1;
> +
> + /* Maybe refuse empty tags. */
> + if (! (flags & TAG_FLAG_BE_GENEROUS) && *tag == '\0') {
> + fprintf (stderr, "Error: empty tag: %s\n", line_for_error);

ret = -1; ?

> + goto DONE;
> + }
> +
> + /* Decode tag. */
> + if (hex_decode_inplace (tag) != HEX_SUCCESS) {
> + fprintf (stderr, "Hex decoding of tag %s failed\n",
> +  tag);
> + ret = 1;

or -1? Error vs. Warning prefix?

> + goto DONE;
> + }
> +
> + if (tag_op_list_append (ctx, tag_ops, tag, remove)) {
> + ret = -1;
> + goto DONE;
> + }
> +}
> +
> +if (tok == NULL) {
> + fprintf (stderr, "Warning: Ignoring invalid input line: %s\n",
> +  line_for_error);
> + ret = 1;
> + goto DONE;
> +}
> +
> +/* tok now points to the query string */
> +if (hex_decode_inplace (tok) != HEX_SUCCESS) {
> + fprintf (stderr, "Hex decoding of query %s failed\n",
> +  tok);
> + ret = 1;

or -1? Error vs. Warning prefix?

> + goto DONE;
> +}
> +
> +*query_string = tok;
> +  DONE:
> +talloc_free (line_for_error);
> +return ret;
> +}
> +
> +static inline void
> +message_error (notmuch_message_t *message,
> +notmuch_status_t status,
> +const char *format, ...)
> +{
> +va_list va_args;
> +
> +va_start (va_args, format);
> +
> +vfprintf (stderr, format, va_args);
> +fprintf (stderr, "Message-ID: %s\n", notmuch_message_get_message_id 
> (message));
> +fprintf (stderr, "Status: %s\n", notmuch_status_to_string (status));
> +}
> +
> +notmuch_status_t
> +tag_op_list_apply (notmuch_message_t *message,
> +tag_op_list_t *list,
> +tag_op_flag_t flags)
> +{
> +size_t i;
> +notmuch_status_t status = 0;
> +tag_operation_t *tag_ops = list->ops;
> +
> 

[Patch v3b 3/9] util: add string-util.[ch]

[Jani Nikula]

LGTM.

On Fri, 07 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> This is to give a home to strtok_len. It's a bit silly to add a header
> for one routine, but it needs to be shared between several compilation
> units (or at least that's the most natural design).
> ---
>  util/Makefile.local |3 ++-
>  util/string-util.c  |   34 ++
>  util/string-util.h  |   19 +++
>  3 files changed, 55 insertions(+), 1 deletion(-)
>  create mode 100644 util/string-util.c
>  create mode 100644 util/string-util.h
>
> diff --git a/util/Makefile.local b/util/Makefile.local
> index 3ca623e..a11e35b 100644
> --- a/util/Makefile.local
> +++ b/util/Makefile.local
> @@ -3,7 +3,8 @@
>  dir := util
>  extra_cflags += -I$(srcdir)/$(dir)
>  
> -libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c
> +libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c \
> +   $(dir)/string-util.c
>  
>  libutil_modules := $(libutil_c_srcs:.c=.o)
>  
> diff --git a/util/string-util.c b/util/string-util.c
> new file mode 100644
> index 000..44f8cd3
> --- /dev/null
> +++ b/util/string-util.c
> @@ -0,0 +1,34 @@
> +/* string-util.c -  Extra or enhanced routines for null terminated strings.
> + *
> + * Copyright (c) 2012 Jani Nikula
> + *
> + * This program is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, either version 3 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see http://www.gnu.org/licenses/ .
> + *
> + * Author: Jani Nikula 
> + */
> +
> +
> +#include "string-util.h"
> +
> +char *
> +strtok_len (char *s, const char *delim, size_t *len)
> +{
> +/* skip initial delims */
> +s += strspn (s, delim);
> +
> +/* length of token */
> +*len = strcspn (s, delim);
> +
> +return *len ? s : NULL;
> +}
> diff --git a/util/string-util.h b/util/string-util.h
> new file mode 100644
> index 000..696da40
> --- /dev/null
> +++ b/util/string-util.h
> @@ -0,0 +1,19 @@
> +#ifndef _STRING_UTIL_H
> +#define _STRING_UTIL_H
> +
> +#include 
> +
> +/* like strtok(3), but without state, and doesn't modify s. usage pattern:
> + *
> + * const char *tok = input;
> + * const char *delim = " \t";
> + * size_t tok_len = 0;
> + *
> + * while ((tok = strtok_len (tok + tok_len, delim, _len)) != NULL) {
> + * // do stuff with string tok of length tok_len
> + * }
> + */
> +
> +char *strtok_len (char *s, const char *delim, size_t *len);
> +
> +#endif
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[Patch v3b 2/9] test: add sanity check for dump --format=batch-tag.

[Jani Nikula]

LGTM.

On Fri, 07 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> It's important this does not rely on restore, since it hasn't been
> written yet.
> ---
>  test/dump-restore |   13 +
>  1 file changed, 13 insertions(+)
>
> diff --git a/test/dump-restore b/test/dump-restore
> index bf31266..b4c807f 100755
> --- a/test/dump-restore
> +++ b/test/dump-restore
> @@ -85,6 +85,19 @@ test_begin_subtest "dump --output=outfile -- from:cworth"
>  notmuch dump --output=dump-outfile-dash-inbox.actual -- from:cworth
>  test_expect_equal_file dump-cworth.expected dump-outfile-dash-inbox.actual
>  
> +test_begin_subtest "Check for a safe set of message-ids"
> +notmuch search --output=messages from:cworth | sed s/^id:// > EXPECTED
> +notmuch search --output=messages from:cworth | sed s/^id:// |\
> + $TEST_DIRECTORY/hex-xcode --direction=encode > OUTPUT
> +test_expect_equal_file OUTPUT EXPECTED
> +
> +test_begin_subtest "format=batch-tag, dump sanity check."
> +notmuch dump --format=sup from:cworth | cut -f1 -d' ' | \
> +sort > EXPECTED.$test_count
> +notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
> +sort > OUTPUT.$test_count
> +test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
> +
>  test_begin_subtest 'roundtripping random message-ids and tags'
>  test_subtest_known_broken
>  ${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[Patch v3b 1/9] notmuch-dump: add --format=(batch-tag|sup)

[Jani Nikula]

LGTM.

On Fri, 07 Dec 2012, david at tethera.net wrote:
> From: David Bremner 
>
> sup is the old format, and remains the default, at least until
> restore is converted to parse this format.
>
> Each line of the batch-tag format is modelled on the syntax of notmuch tag:
> - "notmuch tag" is omitted from the front of the line
> - The dump format only uses query strings of a single message-id.
> - Each space seperated tag/message-id is 'hex-encoded' to remove
>   trouble-making characters.
> - It is permitted (and will be useful) for there to be no tags before
>   the query.
>
> In particular this format won't have the same problem with e.g. spaces
> in message-ids or tags; they will be round-trip-able.
> ---
>  dump-restore-private.h |   13 +
>  notmuch-dump.c |   48 
> ++--
>  2 files changed, 55 insertions(+), 6 deletions(-)
>  create mode 100644 dump-restore-private.h
>
> diff --git a/dump-restore-private.h b/dump-restore-private.h
> new file mode 100644
> index 000..896a004
> --- /dev/null
> +++ b/dump-restore-private.h
> @@ -0,0 +1,13 @@
> +#ifndef DUMP_RESTORE_PRIVATE_H
> +#define DUMP_RESTORE_PRIVATE_H
> +
> +#include "hex-escape.h"
> +#include "command-line-arguments.h"
> +
> +typedef enum dump_formats {
> +DUMP_FORMAT_AUTO,
> +DUMP_FORMAT_BATCH_TAG,
> +DUMP_FORMAT_SUP
> +} dump_format_t;
> +
> +#endif
> diff --git a/notmuch-dump.c b/notmuch-dump.c
> index 88f598a..d2dad40 100644
> --- a/notmuch-dump.c
> +++ b/notmuch-dump.c
> @@ -19,6 +19,7 @@
>   */
>  
>  #include "notmuch-client.h"
> +#include "dump-restore-private.h"
>  
>  int
>  notmuch_dump_command (unused (void *ctx), int argc, char *argv[])
> @@ -43,7 +44,13 @@ notmuch_dump_command (unused (void *ctx), int argc, char 
> *argv[])
>  char *output_file_name = NULL;
>  int opt_index;
>  
> +int output_format = DUMP_FORMAT_SUP;
> +
>  notmuch_opt_desc_t options[] = {
> + { NOTMUCH_OPT_KEYWORD, _format, "format", 'f',
> +   (notmuch_keyword_t []){ { "sup", DUMP_FORMAT_SUP },
> +   { "batch-tag", DUMP_FORMAT_BATCH_TAG },
> +   { 0, 0 } } },
>   { NOTMUCH_OPT_STRING, _file_name, "output", 'o', 0  },
>   { 0, 0, 0, 0, 0 }
>  };
> @@ -83,27 +90,56 @@ notmuch_dump_command (unused (void *ctx), int argc, char 
> *argv[])
>   */
>  notmuch_query_set_sort (query, NOTMUCH_SORT_UNSORTED);
>  
> +char *buffer = NULL;
> +size_t buffer_size = 0;
> +
>  for (messages = notmuch_query_search_messages (query);
>notmuch_messages_valid (messages);
>notmuch_messages_move_to_next (messages)) {
>   int first = 1;
> + const char *message_id;
> +
>   message = notmuch_messages_get (messages);
> + message_id = notmuch_message_get_message_id (message);
>  
> - fprintf (output,
> -  "%s (", notmuch_message_get_message_id (message));
> + if (output_format == DUMP_FORMAT_SUP) {
> + fprintf (output, "%s (", message_id);
> + }
>  
>   for (tags = notmuch_message_get_tags (message);
>notmuch_tags_valid (tags);
>notmuch_tags_move_to_next (tags)) {
> - if (! first)
> - fprintf (output, " ");
> + const char *tag_str = notmuch_tags_get (tags);
>  
> - fprintf (output, "%s", notmuch_tags_get (tags));
> + if (! first)
> + fputs (" ", output);
>  
>   first = 0;
> +
> + if (output_format == DUMP_FORMAT_SUP) {
> + fputs (tag_str, output);
> + } else {
> + if (hex_encode (notmuch, tag_str,
> + , _size) != HEX_SUCCESS) {
> + fprintf (stderr, "Error: failed to hex-encode tag %s\n",
> +  tag_str);
> + return 1;
> + }
> + fprintf (output, "+%s", buffer);
> + }
>   }
>  
> - fprintf (output, ")\n");
> + if (output_format == DUMP_FORMAT_SUP) {
> + fputs (")\n", output);
> + } else {
> + if (hex_encode (notmuch, message_id,
> + , _size) != HEX_SUCCESS) {
> + fprintf (stderr, "Error: failed to hex-encode msg-id %s\n",
> +  message_id);
> + return 1;
> + }
> + fprintf (output, " -- id:%s\n", buffer);
> + }
>  
>   notmuch_message_destroy (message);
>  }
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 06/10] emacs: Fix bug in resynchronizing after a JSON parse error

[Mark Walters]

This patch looks good to me. Should it go in ahead of the rest of the
series as it essentially a bugfix?

Best wishes

Mark

On Sun, 02 Dec 2012, Austin Clements amdra...@mit.edu wrote:
 Previously, if the input stream consisted only of an error message,
 notmuch-json-begin-compound would signal a (wrong-type-argument
 number-or-marker-p nil) error when reaching the end of the error
 message.  This happened because notmuch-json-scan-to-value would think
 that it reached a value and put the parser into the 'value state.
 Even after notmuch-json-begin-compound signaled the syntax error, the
 parser would remain in this state and when the resynchronization logic
 reached the end of the buffer, the parser would fail because the
 'value state indicates that characters are available.

 This fixes this problem by restoring the parser's previous state if it
 encounters a syntax error.
 ---
  emacs/notmuch-lib.el |   22 +-
  1 file changed, 13 insertions(+), 9 deletions(-)

 diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
 index 1d0ec17..9402456 100644
 --- a/emacs/notmuch-lib.el
 +++ b/emacs/notmuch-lib.el
 @@ -474,15 +474,19 @@ Entering JSON objects is currently unimplemented.
(with-current-buffer (notmuch-json-buffer jp)
  ;; Disallow terminators
  (setf (notmuch-json-allow-term jp) nil)
 -(or (notmuch-json-scan-to-value jp)
 - (if (/= (char-after) ?\[)
 - (signal 'json-readtable-error (list expected '['))
 -   (forward-char)
 -   (push ?\] (notmuch-json-term-stack jp))
 -   ;; Expect a value or terminator next
 -   (setf (notmuch-json-next jp) 'expect-value
 - (notmuch-json-allow-term jp) t)
 -   t
 +;; Save next so we can restore it if there's a syntax error
 +(let ((saved-next (notmuch-json-next jp)))
 +  (or (notmuch-json-scan-to-value jp)
 +   (if (/= (char-after) ?\[)
 +   (progn
 + (setf (notmuch-json-next jp) saved-next)
 + (signal 'json-readtable-error (list expected '[')))
 + (forward-char)
 + (push ?\] (notmuch-json-term-stack jp))
 + ;; Expect a value or terminator next
 + (setf (notmuch-json-next jp) 'expect-value
 +   (notmuch-json-allow-term jp) t)
 + t)
  
  (defun notmuch-json-read (jp)
Parse the value at point in JP's buffer.
 -- 
 1.7.10.4

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 07/10] emacs: Use --use-schema for search

[Mark Walters]

On Sun, 02 Dec 2012, Austin Clements amdra...@mit.edu wrote:
 We detect the special exit statuses and use these to produce specific
 diagnostic messages.
 ---
  emacs/notmuch-lib.el |   32 
  emacs/notmuch.el |   17 +
  2 files changed, 45 insertions(+), 4 deletions(-)

 diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
 index 9402456..49b0da6 100644
 --- a/emacs/notmuch-lib.el
 +++ b/emacs/notmuch-lib.el
 @@ -325,6 +325,38 @@ string), a property list of face attributes, or a list 
 of these.
   (put-text-property pos next 'face (cons face cur))
   (setq pos next)
  
 +(defun notmuch-pop-up-error (msg)
 +  Pop up an error buffer displaying MSG.
 +
 +  (let ((buf (get-buffer-create *Notmuch errors*)))
 +(with-current-buffer buf
 +  (view-mode)
 +  (let ((inhibit-read-only t))
 + (erase-buffer)
 + (insert msg)
 + (unless (bolp)
 +   (insert \n))
 + (goto-char (point-min
 +(pop-to-buffer buf)))

I am not sure about the erase-buffer in the above: do we definitely want
to remove all previous error information? For version mismatch possibly
we do but in patch 9 it is done for all show command-line error returns.
Incidentally why does show always pop-up an error but search only for
version-mismatches?

Otherwise this looks good to me (but I am not that familiar with lisp
error handling)

Best wishes

Mark






 +(defun notmuch-version-mismatch-error (exit-status)
 +  Signal a schema version mismatch error.
 +
 +EXIT-STATUS must be the exit status of the notmuch CLI command,
 +and must have the value 20 or 21.  This function will pop up an
 +error buffer with a descriptive message and signal an error.
 +  (cond ((= exit-status 20)
 +  (notmuch-pop-up-error Error: Version mismatch.
 +Emacs requested an older output format than supported by the notmuch CLI.
 +You may need to restart Emacs or upgrade your notmuch Emacs package.))
 + ((= exit-status 21)
 +  (notmuch-pop-up-error Error: Version mismatch.
 +Emacs requested a newer output format than supported by the notmuch CLI.
 +You may need to restart Emacs or upgrade your notmuch package.))
 + (t
 +  (error Bad exit status %d exit-status)))
 +  (error notmuch CLI version mismatch))
 +
  ;; Compatibility functions for versions of emacs before emacs 23.
  ;;
  ;; Both functions here were copied from emacs 23 with the following 
 copyright:
 diff --git a/emacs/notmuch.el b/emacs/notmuch.el
 index f9454d8..e1f28ca 100644
 --- a/emacs/notmuch.el
 +++ b/emacs/notmuch.el
 @@ -644,6 +644,7 @@ of the result.
   (exit-status (process-exit-status proc))
   (never-found-target-thread nil))
  (when (memq status '(exit signal))
 +  (catch 'return
   (kill-buffer (process-get proc 'parse-buf))
   (if (buffer-live-p buffer)
   (with-current-buffer buffer
 @@ -655,8 +656,16 @@ of the result.
 (insert Incomplete search results (search process was 
 killed).\n))
 (when (eq status 'exit)
   (insert End of search results.)
 - (unless (= exit-status 0)
 -   (insert (format  (process returned %d) exit-status)))
 + (cond ((or (= exit-status 20) (= exit-status 21))
 +(kill-buffer)
 +(condition-case err
 +(notmuch-version-mismatch-error exit-status)
 +  ;; Strange things happen when you signal
 +  ;; an error from a sentinel.
 +  (error (throw 'return nil
 +   ((/= exit-status 0)
 +(insert (format  (process returned %d)
 +exit-status
   (insert \n)
   (if (and atbob
(not (string= notmuch-search-target-thread 
 found)))
 @@ -664,7 +673,7 @@ of the result.
 (when (and never-found-target-thread
  notmuch-search-target-line)
 (goto-char (point-min))
 -   (forward-line (1- notmuch-search-target-line
 +   (forward-line (1- notmuch-search-target-line)
  
  (defcustom notmuch-search-line-faces '((unread :weight bold)
  (flagged :foreground blue))
 @@ -938,7 +947,7 @@ Other optional parameters are used as follows:
   (let ((proc (start-process
notmuch-search buffer
notmuch-command search
 -  --format=json
 +  --format=json --use-schema=1
(if oldest-first
--sort=oldest-first
  --sort=newest-first)
 -- 
 1.7.10.4

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 00/10] CLI output versioning

[Mark Walters]

Hi

Overall this series looks good. As we discussed on irc I think i would
prefer global NOTMUCH_SCHEMA_MIN as I am a little worried about these
proliferating (eg if someone decides text output also needs versioning
etc) In addition, if we do find the distinction useful it would be easy
to add at a later date.

One tiny comment on the manpage updates: now that you mention two return
values explicitly should the other possibilities be mentioned too or are
they so obvious it is not needed?

Would it be worth having some emacs test for the error handling? (eg set
notmuch-command to something giving some stderr and an error) Inherently
these code paths won't be tested much so I think tests could be
particularly useful.

Best wishes

Mark






On Mon, 03 Dec 2012, Austin Clements amdra...@mit.edu wrote:
 (Sorry; I forgot to include a cover letter.)

 This series is intended to help with our long-standing output format
 versioning issue.  While the JSON format is amenable to extension,
 there's still a high barrier to extensions because of the need to
 support them going forward, and an even higher barrier to modifications
 that break backwards compatibility.  Versioning will make the format
 more dynamic, enabling us to easily improve and iterate on it.  It will
 also address the slew of confusing bugs that people encounter when they
 use a mismatched CLI and front-end.

 On IRC we've talking about adding version information to the output
 format itself.  This series takes a different and, I think, better
 approach: callers request a specific output format version on the
 command line.  This allows notmuch to remain backwards compatible with
 older format versions when it's easy or necessary.  This also doesn't
 require shoehorning a version number into the output, which would be
 awkward for both the CLI and the consumer.

 I called the argument --use-schema, but I'm open to other suggestions.
 --use-schema is technically accurate, but perhaps not as self-describing
 as something like --schema-version or --format-version (to parallel
 --format).
 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [Patch v3b 3/9] util: add string-util.[ch]

[Mark Walters]

On Fri, 07 Dec 2012, da...@tethera.net wrote:
 From: David Bremner brem...@debian.org

 This is to give a home to strtok_len. It's a bit silly to add a header
 for one routine, but it needs to be shared between several compilation
 units (or at least that's the most natural design).
 ---
  util/Makefile.local |3 ++-
  util/string-util.c  |   34 ++
  util/string-util.h  |   19 +++
  3 files changed, 55 insertions(+), 1 deletion(-)
  create mode 100644 util/string-util.c
  create mode 100644 util/string-util.h

 diff --git a/util/Makefile.local b/util/Makefile.local
 index 3ca623e..a11e35b 100644
 --- a/util/Makefile.local
 +++ b/util/Makefile.local
 @@ -3,7 +3,8 @@
  dir := util
  extra_cflags += -I$(srcdir)/$(dir)
  
 -libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c
 +libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c \
 +   $(dir)/string-util.c
  
  libutil_modules := $(libutil_c_srcs:.c=.o)
  
 diff --git a/util/string-util.c b/util/string-util.c
 new file mode 100644
 index 000..44f8cd3
 --- /dev/null
 +++ b/util/string-util.c
 @@ -0,0 +1,34 @@
 +/* string-util.c -  Extra or enhanced routines for null terminated strings.
 + *
 + * Copyright (c) 2012 Jani Nikula
 + *
 + * This program is free software: you can redistribute it and/or modify
 + * it under the terms of the GNU General Public License as published by
 + * the Free Software Foundation, either version 3 of the License, or
 + * (at your option) any later version.
 + *
 + * This program is distributed in the hope that it will be useful,
 + * but WITHOUT ANY WARRANTY; without even the implied warranty of
 + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 + * GNU General Public License for more details.
 + *
 + * You should have received a copy of the GNU General Public License
 + * along with this program.  If not, see http://www.gnu.org/licenses/ .
 + *
 + * Author: Jani Nikula j...@nikula.org
 + */
 +
 +
 +#include string-util.h
 +
 +char *
 +strtok_len (char *s, const char *delim, size_t *len)
 +{
 +/* skip initial delims */
 +s += strspn (s, delim);
 +
 +/* length of token */
 +*len = strcspn (s, delim);
 +
 +return *len ? s : NULL;
 +}
 diff --git a/util/string-util.h b/util/string-util.h
 new file mode 100644
 index 000..696da40
 --- /dev/null
 +++ b/util/string-util.h
 @@ -0,0 +1,19 @@
 +#ifndef _STRING_UTIL_H
 +#define _STRING_UTIL_H
 +
 +#include string.h
 +
 +/* like strtok(3), but without state, and doesn't modify s. usage pattern:
 + *
 + * const char *tok = input;
 + * const char *delim =  \t;
 + * size_t tok_len = 0;
 + *
 + * while ((tok = strtok_len (tok + tok_len, delim, tok_len)) != NULL) {
 + * // do stuff with string tok of length tok_len
 + * }

Hi

Some possibly silly comments on the comment. strtok(3) seems to return a
null terminated string and this does not. Might be worth including in
the comment? Secondly every time I read this I read ..modify s. usage
pattern as one ongoing sentence with s. an abbreviation. Perhaps a
capital u? and maybe str or string rather than s?

Best wishes

Mark

 + */
 +
 +char *strtok_len (char *s, const char *delim, size_t *len);
 +
 +#endif
 -- 
 1.7.10.4

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [Patch v3b 4/9] tag-util.[ch]: New files for common tagging routines

[Mark Walters]

On Fri, 07 Dec 2012, da...@tethera.net wrote:
 From: David Bremner brem...@debian.org

 These are meant to be shared between notmuch-tag and notmuch-restore.

 The bulk of the routines implement a tag operation list abstract
 data type act as a structured representation of a set of tag
 operations (typically coming from a single tag command or line of
 input).
 ---
  Makefile.local |1 +
  tag-util.c |  278 
 
  tag-util.h |  119 
  3 files changed, 398 insertions(+)
  create mode 100644 tag-util.c
  create mode 100644 tag-util.h

 diff --git a/Makefile.local b/Makefile.local
 index 2b91946..854867d 100644
 --- a/Makefile.local
 +++ b/Makefile.local
 @@ -274,6 +274,7 @@ notmuch_client_srcs = \
   query-string.c  \
   mime-node.c \
   crypto.c\
 + tag-util.c
  
  notmuch_client_modules = $(notmuch_client_srcs:.c=.o)
  
 diff --git a/tag-util.c b/tag-util.c
 new file mode 100644
 index 000..932ee7f
 --- /dev/null
 +++ b/tag-util.c
 @@ -0,0 +1,278 @@
 +#include assert.h
 +#include string-util.h
 +#include tag-util.h
 +#include hex-escape.h
 +
 +#define TAG_OP_LIST_INITIAL_SIZE 10
 +
 +struct _tag_operation_t {
 +const char *tag;
 +notmuch_bool_t remove;
 +};
 +
 +struct _tag_op_list_t {
 +tag_operation_t *ops;
 +size_t count;
 +size_t size;
 +};
 +
 +int
 +parse_tag_line (void *ctx, char *line,
 + tag_op_flag_t flags,
 + char **query_string,
 + tag_op_list_t *tag_ops)
 +{
 +char *tok = line;
 +size_t tok_len = 0;
 +char *line_for_error = talloc_strdup (ctx, line);
 +int ret = 0;
 +
 +chomp_newline (line);
 +
 +/* remove leading space */
 +while (*tok == ' ' || *tok == '\t')
 + tok++;
 +
 +/* Skip empty and comment lines. */
 +if (*tok == '\0' || *tok == '#') {
 + ret = 1;
 + goto DONE;
 +}
 +
 +tag_op_list_reset (tag_ops);
 +
 +/* Parse tags. */
 +while ((tok = strtok_len (tok + tok_len,  , tok_len)) != NULL) {
 + notmuch_bool_t remove;
 + char *tag;
 +
 + /* Optional explicit end of tags marker. */
 + if (tok_len == 2  strncmp (tok, --, tok_len) == 0) {
 + tok = strtok_len (tok + tok_len,  , tok_len);
 + break;
 + }
 +
 + /* Implicit end of tags. */
 + if (*tok != '-'  *tok != '+')
 + break;
 +
 + /* If tag is terminated by NUL, there's no query string. */
 + if (*(tok + tok_len) == '\0') {
 + fprintf (stderr, no query string: %s\n, line_for_error);
 + ret = 1;
 + goto DONE;
 + }
 +
 + /* Terminate, and start next token after terminator. */
 + *(tok + tok_len++) = '\0';
 +
 + remove = (*tok == '-');
 + tag = tok + 1;
 +
 + /* Maybe refuse empty tags. */
 + if (! (flags  TAG_FLAG_BE_GENEROUS)  *tag == '\0') {
 + fprintf (stderr, Error: empty tag: %s\n, line_for_error);
 + goto DONE;
 + }
 +
 + /* Decode tag. */
 + if (hex_decode_inplace (tag) != HEX_SUCCESS) {
 + fprintf (stderr, Hex decoding of tag %s failed\n,
 +  tag);
 + ret = 1;
 + goto DONE;
 + }
 +
 + if (tag_op_list_append (ctx, tag_ops, tag, remove)) {
 + ret = -1;
 + goto DONE;
 + }
 +}
 +
 +if (tok == NULL) {
 + fprintf (stderr, Warning: Ignoring invalid input line: %s\n,
 +  line_for_error);
 + ret = 1;
 + goto DONE;
 +}
 +
 +/* tok now points to the query string */
 +if (hex_decode_inplace (tok) != HEX_SUCCESS) {
 + fprintf (stderr, Hex decoding of query %s failed\n,
 +  tok);


For these hex_decode errors would it be worth printing the full line as
well as the bit that fails hex_decode? Perhaps put something under DONE:
to print the whole line if ret is not success?

Otherwise LGTM

Mark


 + ret = 1;
 + goto DONE;
 +}
 +
 +*query_string = tok;
 +  DONE:
 +talloc_free (line_for_error);
 +return ret;
 +}
 +
 +static inline void
 +message_error (notmuch_message_t *message,
 +notmuch_status_t status,
 +const char *format, ...)
 +{
 +va_list va_args;
 +
 +va_start (va_args, format);
 +
 +vfprintf (stderr, format, va_args);
 +fprintf (stderr, Message-ID: %s\n, notmuch_message_get_message_id 
 (message));
 +fprintf (stderr, Status: %s\n, notmuch_status_to_string (status));
 +}
 +
 +notmuch_status_t
 +tag_op_list_apply (notmuch_message_t *message,
 +tag_op_list_t *list,
 +tag_op_flag_t flags)
 +{
 +size_t i;
 +notmuch_status_t status = 0;
 +tag_operation_t *tag_ops = list-ops;
 +
 +status = notmuch_message_freeze (message);
 +if (status) {
 + message_error (message, status, freezing message);
 + return status;
 +}
 +
 +if (flags  TAG_FLAG_REMOVE_ALL) {
 + status = 

Re: [Patch v3b 9/9] tag-util: optimization of tag application

[Mark Walters]

On Fri, 07 Dec 2012, da...@tethera.net wrote:
 From: David Bremner brem...@debian.org

 The idea is not to bother with restore operations if they don't change
 the set of tags. This is actually a relatively common case.

 In order to avoid fancy datastructures, this method is quadratic in
 the number of tags; at least on my mail database this doesn't seem to
 be a big problem.
 ---
  tag-util.c |   66 
 
  1 file changed, 66 insertions(+)

 diff --git a/tag-util.c b/tag-util.c
 index 932ee7f..3d54e9e 100644
 --- a/tag-util.c
 +++ b/tag-util.c
 @@ -124,6 +124,69 @@ message_error (notmuch_message_t *message,
  fprintf (stderr, Status: %s\n, notmuch_status_to_string (status));
  }
  
 +static int
 +makes_changes (notmuch_message_t *message,
 +tag_op_list_t *list,
 +tag_op_flag_t flags)
 +{
 +
 +size_t i;
 +
 +notmuch_tags_t *tags;
 +notmuch_bool_t changes = FALSE;
 +
 +/* First, do we delete an existing tag? */
 +changes = FALSE;
 +for (tags = notmuch_message_get_tags (message);
 +  ! changes  notmuch_tags_valid (tags);
 +  notmuch_tags_move_to_next (tags)) {
 + const char *cur_tag = notmuch_tags_get (tags);
 + int last_op =  (flags  TAG_FLAG_REMOVE_ALL) ? -1 : 0;
 +
 + /* slight contortions to count down with an unsigned index */
 + for (i = list-count; i--  0; /*nothing*/) {
 + if (strcmp (cur_tag, list-ops[i].tag) == 0) {
 + last_op = list-ops[i].remove ? -1 : 1;
 + break;
 + }
 + }

I agree that this is a little ugly but ok I think. Is it worth adding a
comment as to why you are counting backwards? eg  we count backwards to
check whether the last change for the tag foo is removal

But basically LGTM

Mark


 +
 + changes = (last_op == -1);
 +}
 +notmuch_tags_destroy (tags);
 +
 +if (changes)
 + return TRUE;
 +
 +/* Now check for adding new tags */
 +for (i = 0; i  list-count; i++) {
 + notmuch_bool_t exists = FALSE;
 +
 + if (list-ops[i].remove)
 + continue;
 +
 + for (tags = notmuch_message_get_tags (message);
 +  notmuch_tags_valid (tags);
 +  notmuch_tags_move_to_next (tags)) {
 + const char *cur_tag = notmuch_tags_get (tags);
 + if (strcmp (cur_tag, list-ops[i].tag) == 0) {
 + exists = TRUE;
 + break;
 + }
 + }
 + notmuch_tags_destroy (tags);
 +
 + /* the following test is conservative,
 +  * in the sense it ignores cases like +foo ... -foo
 +  * but this is OK from a correctness point of view
 +  */
 + if (! exists)
 + return TRUE;
 +}
 +return FALSE;
 +
 +}
 +
  notmuch_status_t
  tag_op_list_apply (notmuch_message_t *message,
  tag_op_list_t *list,
 @@ -133,6 +196,9 @@ tag_op_list_apply (notmuch_message_t *message,
  notmuch_status_t status = 0;
  tag_operation_t *tag_ops = list-ops;
  
 +if (! (flags  TAG_FLAG_PRE_OPTIMIZED)  ! makes_changes (message, 
 list, flags))
 + return NOTMUCH_STATUS_SUCCESS;
 +
  status = notmuch_message_freeze (message);
  if (status) {
   message_error (message, status, freezing message);
 -- 
 1.7.10.4

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [Patch v3b 5/9] notmuch-restore: add support for input format 'batch-tag'

[Mark Walters]

Hi

Basically LGTM: just one comment below.

On Fri, 07 Dec 2012, da...@tethera.net wrote:
 From: David Bremner brem...@debian.org

 This can be enabled with the new --format=batch-tag command line
 option to notmuch restore. The input must consist of lines of the
 format:

 +tag|-tag [...] [--] id:msg-id

 Each line is interpreted similarly to notmuch tag command line
 arguments. The delimiter is one or more spaces ' '. Any characters in
 tag and search-terms MAY be hex encoded with %NN where NN is the
 hexadecimal value of the character. Any ' ' and '%' characters in
 tag and msg-id MUST be hex encoded (using %20 and %25,
 respectively). Any characters that are not part of tag or
 search-terms MUST NOT be hex encoded.

 Leading and trailing space ' ' is ignored. Empty lines and lines
 beginning with '#' are ignored.

 Commit message mainly stolen from Jani's batch tagging commit, to
 follow.
 ---
  notmuch-restore.c |  206 
 ++---
  1 file changed, 131 insertions(+), 75 deletions(-)

 diff --git a/notmuch-restore.c b/notmuch-restore.c
 index f03dcac..ceec2d3 100644
 --- a/notmuch-restore.c
 +++ b/notmuch-restore.c
 @@ -19,18 +19,22 @@
   */
  
  #include notmuch-client.h
 +#include dump-restore-private.h
 +#include tag-util.h
 +#include string-util.h
 +
 +static volatile sig_atomic_t interrupted;
 +static regex_t regex;
  
  static int
 -tag_message (notmuch_database_t *notmuch, const char *message_id,
 -  char *file_tags, notmuch_bool_t remove_all,
 -  notmuch_bool_t synchronize_flags)
 +tag_message (unused (void *ctx),
 +  notmuch_database_t *notmuch,
 +  const char *message_id,
 +  tag_op_list_t *tag_ops,
 +  tag_op_flag_t flags)
  {
  notmuch_status_t status;
 -notmuch_tags_t *db_tags;
 -char *db_tags_str;
  notmuch_message_t *message = NULL;
 -const char *tag;
 -char *next;
  int ret = 0;
  
  status = notmuch_database_find_message (notmuch, message_id, message);
 @@ -44,55 +48,62 @@ tag_message (notmuch_database_t *notmuch, const char 
 *message_id,
  
  /* In order to detect missing messages, this check/optimization is
   * intentionally done *after* first finding the message. */
 -if (! remove_all  (file_tags == NULL || *file_tags == '\0'))
 - goto DONE;
 -
 -db_tags_str = NULL;
 -for (db_tags = notmuch_message_get_tags (message);
 -  notmuch_tags_valid (db_tags);
 -  notmuch_tags_move_to_next (db_tags)) {
 - tag = notmuch_tags_get (db_tags);
 -
 - if (db_tags_str)
 - db_tags_str = talloc_asprintf_append (db_tags_str,  %s, tag);
 - else
 - db_tags_str = talloc_strdup (message, tag);
 -}
 +if ((flags  TAG_FLAG_REMOVE_ALL) || tag_op_list_size (tag_ops))
 + tag_op_list_apply (message, tag_ops, flags);
  
 -if (((file_tags == NULL || *file_tags == '\0') 
 -  (db_tags_str == NULL || *db_tags_str == '\0')) ||
 - (file_tags  db_tags_str  strcmp (file_tags, db_tags_str) == 0))
 - goto DONE;
 +notmuch_message_destroy (message);
  
 -notmuch_message_freeze (message);
 +return ret;
 +}
  
 -if (remove_all)
 - notmuch_message_remove_all_tags (message);
 +static int
 +parse_sup_line (void *ctx, char *line,
 + char **query_str, tag_op_list_t *tag_ops)
 +{
  
 -next = file_tags;
 -while (next) {
 - tag = strsep (next,  );
 - if (*tag == '\0')
 - continue;
 - status = notmuch_message_add_tag (message, tag);
 - if (status) {
 - fprintf (stderr, Error applying tag %s to message %s:\n,
 -  tag, message_id);
 - fprintf (stderr, %s\n, notmuch_status_to_string (status));
 - ret = 1;
 - }
 +regmatch_t match[3];
 +char *file_tags;
 +int rerr;
 +
 +tag_op_list_reset (tag_ops);
 +
 +chomp_newline (line);
 +
 +/* Silently ignore blank lines */
 +if (line[0] == '\0') {
 + return 1;
 +}
 +
 +rerr = xregexec (regex, line, 3, match, 0);
 +if (rerr == REG_NOMATCH) {
 + fprintf (stderr, Warning: Ignoring invalid sup format line: %s\n,
 +  line);
 + return 1;
  }
  
 -notmuch_message_thaw (message);
 +*query_str = talloc_strndup (ctx, line + match[1].rm_so,
 +  match[1].rm_eo - match[1].rm_so);
 +file_tags = talloc_strndup (ctx, line + match[2].rm_so,
 + match[2].rm_eo - match[2].rm_so);
  
 -if (synchronize_flags)
 - notmuch_message_tags_to_maildir_flags (message);
 +char *tok = file_tags;
 +size_t tok_len = 0;
  
 -  DONE:
 -if (message)
 - notmuch_message_destroy (message);
 +tag_op_list_reset (tag_ops);
 +
 +while ((tok = strtok_len (tok + tok_len,  , tok_len)) != NULL) {
 +
 + if (*(tok + tok_len) != '\0') {
 + *(tok + tok_len) = '\0';
 + tok_len++;
 + }
 +
 + if (tag_op_list_append (ctx, tag_ops, tok, 

[PATCH] contrib: pick: archive message updated

[Mark Walters]

Update pick's archive message to respect notmuch-archive-tags. Also
split archive message into an archiving part and a separate
then-next part, to move more inline with show. Update the keybinding
so default behaviour is unchanged.
---

Notmuch pick had fallen behind show so update.

Best wishes 

Mark


 contrib/notmuch-pick/notmuch-pick.el |   21 +
 1 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/contrib/notmuch-pick/notmuch-pick.el 
b/contrib/notmuch-pick/notmuch-pick.el
index 755cbbc..36587a6 100644
--- a/contrib/notmuch-pick/notmuch-pick.el
+++ b/contrib/notmuch-pick/notmuch-pick.el
@@ -173,7 +173,7 @@
 (define-key map q 'notmuch-pick-quit)
 (define-key map x 'notmuch-pick-quit)
 (define-key map ? 'notmuch-help)
-(define-key map a 'notmuch-pick-archive-message)
+(define-key map a 'notmuch-pick-archive-message-then-next)
 (define-key map = 'notmuch-pick-refresh-view)
 (define-key map s 'notmuch-search)
 (define-key map z 'notmuch-pick)
@@ -393,10 +393,23 @@ Does NOT change the database.
   (kill-buffer notmuch-pick-message-buffer))
 t))
 
-(defun notmuch-pick-archive-message ()
+(defun notmuch-pick-archive-message (optional unarchive)
+  Archive the current message.
+
+Archive the current message by applying the tag changes in
+`notmuch-archive-tags' to it (remove the \inbox\ tag by
+default). If a prefix argument is given, the message will be
+\unarchived\, i.e. the tag changes in `notmuch-archive-tags'
+will be reversed.
+  (interactive P)
+  (when notmuch-archive-tags
+(apply 'notmuch-pick-tag
+  (notmuch-tag-change-list notmuch-archive-tags unarchive
+
+(defun notmuch-pick-archive-message-then-next (optional unarchive)
   Archive the current message and move to next matching message.
-  (interactive)
-  (notmuch-pick-tag -inbox)
+  (interactive P)
+  (notmuch-pick-archive-message unarchive)
   (notmuch-pick-next-matching-message))
 
 (defun notmuch-pick-next-message ()
-- 
1.7.9.1

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] contrib: pick: slightly tweak running search and pick from pick buffer

[Mark Walters]

Previously running search or pick from the pick buffer did not close
the message pane (if open). This meant that then new search ends up in
a very small window. Fix this so that the message pane is
shut. However, make it so that the pane is shut after the search
string is entered in case the user is basing the search on something
in the current message.
---

This is essentially an independent patch but the context clashes with
id:1354970494-18050-1-git-send-email-markwalters1...@gmail.com so that
should be applied first for testing.

If there is any problem with the first patch I can trivially rebase
this one.

Best wishes

Mark


 contrib/notmuch-pick/notmuch-pick.el |   23 +--
 1 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/contrib/notmuch-pick/notmuch-pick.el 
b/contrib/notmuch-pick/notmuch-pick.el
index 36587a6..b474231 100644
--- a/contrib/notmuch-pick/notmuch-pick.el
+++ b/contrib/notmuch-pick/notmuch-pick.el
@@ -175,8 +175,8 @@
 (define-key map ? 'notmuch-help)
 (define-key map a 'notmuch-pick-archive-message-then-next)
 (define-key map = 'notmuch-pick-refresh-view)
-(define-key map s 'notmuch-search)
-(define-key map z 'notmuch-pick)
+(define-key map s 'notmuch-pick-to-search)
+(define-key map z 'notmuch-pick-to-pick)
 (define-key map m 'notmuch-pick-new-mail)
 (define-key map f 'notmuch-pick-forward-message)
 (define-key map r 'notmuch-pick-reply-sender)
@@ -289,6 +289,25 @@ Does NOT change the database.
   (interactive)
   (notmuch-pick-tag -))
 
+;; The next two functions close the message window before searching or
+;; picking but they do so after the user has entered the query (in
+;; case the user was basing the query on something in the message
+;; window).
+
+(defun notmuch-pick-to-search ()
+  Run \notmuch search\ with the given `query' and display results.
+  (interactive)
+  (let ((query (notmuch-read-query Notmuch search: )))
+(notmuch-pick-close-message-window)
+(notmuch-search query)))
+
+(defun notmuch-pick-to-pick ()
+  Run a query and display results in experimental notmuch-pick mode
+  (interactive)
+  (let ((query (notmuch-read-query Notmuch pick: )))
+(notmuch-pick-close-message-window)
+(notmuch-pick query)))
+
 ;; This function should be in notmuch-hello.el but we are trying to
 ;; minimise impact on the rest of the codebase.
 (defun notmuch-pick-from-hello (optional search)
-- 
1.7.9.1

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 0/2] Add some thread based actions to pick

[Mark Walters]

This addes the a macro to notmuch pick to apply a command to all
messages in the current thread. In the second patch we add the ability
to tag/archive whole threads.

This is independent of
id:1354970494-18050-1-git-send-email-markwalters1...@gmail.com and
id:1354970674-18136-1-git-send-email-markwalters1...@gmail.com but may
have overlapping context (in the keybindings) so they should be applied first.

I can easily supply a version applying straight to master if preferred.

Best wishes

Mark



Mark Walters (2):
  contrib: pick: add thread based utility functions
  contrib: pick: thread tagging (including archiving) implemented

 contrib/notmuch-pick/notmuch-pick.el |   58 ++
 1 files changed, 58 insertions(+), 0 deletions(-)

-- 
1.7.9.1

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/2] contrib: pick: add thread based utility functions

[Mark Walters]

Previously notmuch-pick had no thread based functionality. This adds a
macro to iterate through all messages in a thread. To simplify this it
adds a text-property marker to the first message of each thread.
---
 contrib/notmuch-pick/notmuch-pick.el |   28 
 1 files changed, 28 insertions(+), 0 deletions(-)

diff --git a/contrib/notmuch-pick/notmuch-pick.el 
b/contrib/notmuch-pick/notmuch-pick.el
index b474231..77e15bb 100644
--- a/contrib/notmuch-pick/notmuch-pick.el
+++ b/contrib/notmuch-pick/notmuch-pick.el
@@ -552,6 +552,33 @@ than only the current message.
  (message (format Command '%s' exited abnormally with code %d
   shell-command exit-code)))
 
+(defun notmuch-pick-thread-top ()
+  (interactive)
+  (while (not (notmuch-pick-get-prop :first))
+(forward-line -1)))
+
+(defmacro notmuch-pick-thread-mapc (function)
+  Iterate through all messages in the current thread
+ and call FUNCTION for side effects.
+  (save-excursion
+(notmuch-pick-thread-top)
+(loop do (progn
+  (funcall function)
+  (forward-line))
+ while (and (notmuch-pick-get-message-properties)
+(not (notmuch-pick-get-prop :first))
+
+(defun notmuch-pick-get-messages-ids-thread ()
+  Return all id: queries of messages in the current thread.
+  (let ((message-ids))
+(notmuch-pick-thread-mapc
+ (lambda () (push (notmuch-pick-get-message-id) message-ids)))
+message-ids))
+
+(defun notmuch-pick-get-messages-ids-thread-search ()
+  Return a search string for all message ids of messages in the
+current thread.
+  (mapconcat 'identity (notmuch-pick-get-messages-ids-thread)  or ))
 (defun notmuch-pick-clean-address (address)
   Try to clean a single email ADDRESS for display. Return
 AUTHOR_NAME if present, otherwise return AUTHOR_EMAIL. Return
@@ -638,6 +665,7 @@ unchanged ADDRESS if parsing fails.
(push ├ tree-status)))
 
   (push (concat (if replies ┬ ─) ►) tree-status)
+  (plist-put msg :first (and first (eq 0 depth)))
   (notmuch-pick-insert-msg (plist-put msg :tree-status tree-status))
   (pop tree-status)
   (pop tree-status)
-- 
1.7.9.1

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 2/2] contrib: pick: thread tagging (including archiving) implemented

[Mark Walters]

Previously pick had no actions based on the entire thread: this adds
some. Note in this version '*' is bound to `tag thread' which is not
consistent with search or show. However it still might be the most
natural thing (as it is similar to running * in the show pane).
---
 contrib/notmuch-pick/notmuch-pick.el |   30 ++
 1 files changed, 30 insertions(+), 0 deletions(-)

diff --git a/contrib/notmuch-pick/notmuch-pick.el 
b/contrib/notmuch-pick/notmuch-pick.el
index 77e15bb..23442b9 100644
--- a/contrib/notmuch-pick/notmuch-pick.el
+++ b/contrib/notmuch-pick/notmuch-pick.el
@@ -173,6 +173,7 @@
 (define-key map q 'notmuch-pick-quit)
 (define-key map x 'notmuch-pick-quit)
 (define-key map ? 'notmuch-help)
+(define-key map A 'notmuch-pick-archive-thread)
 (define-key map a 'notmuch-pick-archive-message-then-next)
 (define-key map = 'notmuch-pick-refresh-view)
 (define-key map s 'notmuch-pick-to-search)
@@ -188,6 +189,7 @@
 (define-key map | 'notmuch-pick-pipe-message)
 (define-key map - 'notmuch-pick-remove-tag)
 (define-key map + 'notmuch-pick-add-tag)
+(define-key map * 'notmuch-pick-tag-thread)
 (define-key map   'notmuch-pick-scroll-or-next)
 (define-key map b 'notmuch-pick-scroll-message-window-back)
 map))
@@ -579,6 +581,34 @@ than only the current message.
   Return a search string for all message ids of messages in the
 current thread.
   (mapconcat 'identity (notmuch-pick-get-messages-ids-thread)  or ))
+
+(defun notmuch-pick-tag-thread (optional tag-changes)
+  Tag all messages in the current thread
+  (interactive)
+  (setq tag-changes (funcall 'notmuch-tag 
(notmuch-pick-get-messages-ids-thread-search) tag-changes))
+  (notmuch-pick-thread-mapc
+   (lambda () (notmuch-pick-tag-update-display tag-changes
+
+(defun notmuch-pick-archive-thread (optional unarchive)
+  Archive each message in thread.
+
+Archive each message currently shown by applying the tag changes
+in `notmuch-archive-tags' to each (remove the \inbox\ tag by
+default). If a prefix argument is given, the messages will be
+\unarchived\, i.e. the tag changes in `notmuch-archive-tags'
+will be reversed.
+
+Note: This command is safe from any race condition of new messages
+being delivered to the same thread. It does not archive the
+entire thread, but only the messages shown in the current
+buffer.
+  (interactive P)
+  (when notmuch-archive-tags
+(notmuch-pick-tag-thread
+ (notmuch-tag-change-list notmuch-archive-tags unarchive
+
+;; Functions below here display the pick buffer itself.
+
 (defun notmuch-pick-clean-address (address)
   Try to clean a single email ADDRESS for display. Return
 AUTHOR_NAME if present, otherwise return AUTHOR_EMAIL. Return
-- 
1.7.9.1

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Jason A. Donenfeld]

 I actually prefer this approach, but I think it would be more useful to
 leave the syncing of the emails to a different program, and then just
 managing the labels / tags with your tool (which is notmuch territory). So
 the workflow would be:
 So, implementing the mail sync in this script would, as I see it, kind of
 reinventing the wheel.

You're misunderstanding me. Let me make it very clear what the script
does. Actually why don't you just read the source code? Please? Well,
anyway, here:

- It looks in gmail and asks it which messages gmail has
- It downloads those messages
- It applies gmail's labels to those messages as tags

The end.

Along the way it has some smart things to reduce redundant downloads.

For more information, consult the source code.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

The Gmail Importer Script: Complete

[Jason A. Donenfeld]

Hi everyone,

My script to import messages from gmail into notmuch, including tag
information, is now polished and feature complete.


** Watch a video of it here:
** https://www.youtube.com/watch?v=e-8EHIAr7wA


Read the source here:
http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Download the plaintext here:
http://git.zx2c4.com/gmail-notmuch/plain/gmail-notmuch.py

View the requirements here:
http://git.zx2c4.com/gmail-notmuch/tree/requirements.txt

Enjoy,
Jason
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Tags with spaces

[Jason A. Donenfeld]

Very helpful! Thanks folks for the explanations.

Another thing to note is that the dump format doesn't add any quotes
to the tags, so this is something of an issue I suppose.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] TODO: date range queries - check

[David Bremner]

Jameson Graef Rollins jroll...@finestructure.net writes:

 On Fri, Dec 07 2012, Jani Nikula j...@nikula.org wrote:
 Fine by me. I was just trying to clean up the file a bit, that's
 all. The only downside I can think of is potential new users stumbling
 on this and thinking we still don't have date queries. *shrug*.

 That's why it should probably just be modified, instead of removed.


Patch? Concrete wording suggestion?

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v5 0/5] New output format sexp (Lisp S-Expressions)

[David Bremner]

Peter Feigl cra...@gmx.net writes:

 This patch series adds a new output format sexp to notmuch-reply,
 notmuch-show and notmuch-search. These are useful for the Android mobile
 client, Emacs and perhaps other Lisp programs as well.

pushed, looking forward to an android client ;)

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 0/3] test fixes, portability

[David Bremner]

Jani Nikula j...@nikula.org writes:

 I had a glance at a 'make test' report on OS X [1], and tried to fix a
 few things from there. While at it, noticed that count test is horribly
 broken.

pushed,

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] test: Fix UTF-8 JSON tests in Python 3

[David Bremner]

Austin Clements amdra...@mit.edu writes:

 This patch sets the PYTHONIOENCODING
 environment variable to utf-8 when invoking json.tool to override
 Python's default encoding choice.

Pushed. One day I'd like to experience, yes, python really got that
aspect of encoding right ;).

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] emacs: query: make sync queries use sexp

[Mark Walters]

This changes the queries used by notmuch-show from json to sexp (patch
based on a comment by Tomi on irc as to the trivial change needed).

The async query parsed used by search is not as easy to convert.
---

It's probably worth making this change: sexps are significantly faster
but I doubt anyone would notice in show (since the query is small and
the wash processing etc relatively large).

At the moment this doesn't do any error fixing. The json version did
not either but the sexp parser and the json parser might behave
differently on malformed input.

Best wishes

Mark


 emacs/notmuch-query.el |7 ++-
 1 files changed, 2 insertions(+), 5 deletions(-)

diff --git a/emacs/notmuch-query.el b/emacs/notmuch-query.el
index d66baea..0ee6cca 100644
--- a/emacs/notmuch-query.el
+++ b/emacs/notmuch-query.el
@@ -29,10 +29,7 @@ A thread is a forest or list of trees. A tree is a two 
element
 list where the first element is a message, and the second element
 is a possibly empty forest of replies.
 
-  (let  ((args '(show --format=json))
-(json-object-type 'plist)
-(json-array-type 'list)
-(json-false 'nil))
+  (let  ((args '(show --format=sexp)))
 (if notmuch-show-process-crypto
(setq args (append args '(--decrypt
 (setq args (append args search-terms))
@@ -40,7 +37,7 @@ is a possibly empty forest of replies.
   (progn
(apply 'call-process (append (list notmuch-command nil (list t nil) 
nil) args))
(goto-char (point-min))
-   (json-read)
+   (sexp-at-point)
 
 ;;
 ;; Mapping functions across collections of messages.
-- 
1.7.9.1

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

v4 of Hex Dump/Restore patches

[david]

This obsoletes the series at
 
 id:1354843607-17980-1-git-send-email-da...@tethera.net

One new patch:

[Patch v4 08/10] test/dump-restore: add test for warning/error

Other changes since the last series are as follows:

commit 16ca0f8126fafd49266ffa02534f2e9b73c03027
Author: David Bremner brem...@debian.org
Date:   Fri Dec 7 20:19:21 2012 -0400

fixup for test/dump-restore

diff --git a/test/dump-restore b/test/dump-restore
index 2532cbb..b267792 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -212,7 +212,7 @@ test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest 'roundtripping random message-ids and tags'
 
-${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG}
+${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
--num-messages=100
 
  notmuch dump --format=batch-tag| \

commit 9d864bd724bd2d2b26b5d52e3e2f28b4fff2046f
Author: David Bremner brem...@debian.org
Date:   Fri Dec 7 20:32:50 2012 -0400

fixup for tag-util.c: put * in right palce

diff --git a/tag-util.h b/tag-util.h
index df05d72..6674674 100644
--- a/tag-util.h
+++ b/tag-util.h
@@ -62,8 +62,8 @@ parse_tag_line (void *ctx, char *line,
  * ctx is passed to talloc
  */
 
-tag_op_list_t
-*tag_op_list_create (void *ctx);
+tag_op_list_t *
+tag_op_list_create (void *ctx);
 
 /*
  * Add a tag operation (delete iff remove == TRUE) to a list.

commit d2cead797981e8992849e1d96ad95177b42290e2
Author: David Bremner brem...@debian.org
Date:   Fri Dec 7 20:57:52 2012 -0400

fixup for id:87txrxxyzp@nikula.org

diff --git a/tag-util.c b/tag-util.c
index 3d54e9e..a927363 100644
--- a/tag-util.c
+++ b/tag-util.c
@@ -24,9 +24,15 @@ parse_tag_line (void *ctx, char *line,
 {
 char *tok = line;
 size_t tok_len = 0;
-char *line_for_error = talloc_strdup (ctx, line);
+char *line_for_error;
 int ret = 0;
 
+line_for_error = talloc_strdup (ctx, line);
+if (line_for_error == NULL) {
+   fprintf (stderr, Error: out of memory\n);
+   return -1;
+}
+
 chomp_newline (line);
 
 /* remove leading space */
@@ -58,7 +64,7 @@ parse_tag_line (void *ctx, char *line,
 
/* If tag is terminated by NUL, there's no query string. */
if (*(tok + tok_len) == '\0') {
-   fprintf (stderr, no query string: %s\n, line_for_error);
+   fprintf (stderr, Warning: no query string: %s\n, line_for_error);
ret = 1;
goto DONE;
}
@@ -71,19 +77,21 @@ parse_tag_line (void *ctx, char *line,
 
/* Maybe refuse empty tags. */
if (! (flags  TAG_FLAG_BE_GENEROUS)  *tag == '\0') {
-   fprintf (stderr, Error: empty tag: %s\n, line_for_error);
+   fprintf (stderr, Warning: empty tag: %s\n, line_for_error);
+   ret = 1;
goto DONE;
}
 
/* Decode tag. */
if (hex_decode_inplace (tag) != HEX_SUCCESS) {
-   fprintf (stderr, Hex decoding of tag %s failed\n,
+   fprintf (stderr, Warning: Hex decoding of tag %s failed\n,
 tag);
ret = 1;
goto DONE;
}
 
if (tag_op_list_append (ctx, tag_ops, tag, remove)) {
+   /* diagnostics already printed */
ret = -1;
goto DONE;
}
@@ -98,7 +106,7 @@ parse_tag_line (void *ctx, char *line,
 
 /* tok now points to the query string */
 if (hex_decode_inplace (tok) != HEX_SUCCESS) {
-   fprintf (stderr, Hex decoding of query %s failed\n,
+   fprintf (stderr, Warning: Hex decoding of query %s failed\n,
 tok);
ret = 1;
goto DONE;

commit 162c20ab6d9b9eef56cb2b966a7144a61557eade
Author: David Bremner brem...@debian.org
Date:   Sat Dec 8 07:29:56 2012 -0400

fixup for id:87624chmfw@qmul.ac.uk

update comment

diff --git a/util/string-util.h b/util/string-util.h
index 696da40..ac7676c 100644
--- a/util/string-util.h
+++ b/util/string-util.h
@@ -3,7 +3,10 @@
 
 #include string.h
 
-/* like strtok(3), but without state, and doesn't modify s. usage pattern:
+/* like strtok(3), but without state, and doesn't modify s.  Return
+ * value is indicated by pointer and length, not null terminator.
+ *
+ * Usage pattern:
  *
  * const char *tok = input;
  * const char *delim =  \t;

commit 590adeec072f235861154b930bfb10bedbe37e8a
Author: David Bremner brem...@debian.org
Date:   Sat Dec 8 07:42:44 2012 -0400

fixup for id:8738zghl6d@qmul.ac.uk

diff --git a/tag-util.c b/tag-util.c
index a927363..c97d240 100644
--- a/tag-util.c
+++ b/tag-util.c
@@ -98,8 +98,6 @@ parse_tag_line (void *ctx, char *line,
 }
 
 if (tok == NULL) {
-   fprintf (stderr, Warning: Ignoring invalid input line: %s\n,
-line_for_error);
ret = 1;
goto DONE;
 }
@@ -113,7 +111,13 @@ parse_tag_line (void *ctx, char *line,
 }
 
 *query_string = tok;
+
   DONE:
+if (ret != 0)
+   fprintf (stderr, % invalid input line 

[Patch v4 01/10] notmuch-dump: add --format=(batch-tag|sup)

[david]

From: David Bremner brem...@debian.org

sup is the old format, and remains the default, at least until
restore is converted to parse this format.

Each line of the batch-tag format is modelled on the syntax of notmuch tag:
- notmuch tag is omitted from the front of the line
- The dump format only uses query strings of a single message-id.
- Each space seperated tag/message-id is 'hex-encoded' to remove
  trouble-making characters.
- It is permitted (and will be useful) for there to be no tags before
  the query.

In particular this format won't have the same problem with e.g. spaces
in message-ids or tags; they will be round-trip-able.
---
 dump-restore-private.h |   13 +
 notmuch-dump.c |   48 ++--
 2 files changed, 55 insertions(+), 6 deletions(-)
 create mode 100644 dump-restore-private.h

diff --git a/dump-restore-private.h b/dump-restore-private.h
new file mode 100644
index 000..896a004
--- /dev/null
+++ b/dump-restore-private.h
@@ -0,0 +1,13 @@
+#ifndef DUMP_RESTORE_PRIVATE_H
+#define DUMP_RESTORE_PRIVATE_H
+
+#include hex-escape.h
+#include command-line-arguments.h
+
+typedef enum dump_formats {
+DUMP_FORMAT_AUTO,
+DUMP_FORMAT_BATCH_TAG,
+DUMP_FORMAT_SUP
+} dump_format_t;
+
+#endif
diff --git a/notmuch-dump.c b/notmuch-dump.c
index 88f598a..d2dad40 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -19,6 +19,7 @@
  */
 
 #include notmuch-client.h
+#include dump-restore-private.h
 
 int
 notmuch_dump_command (unused (void *ctx), int argc, char *argv[])
@@ -43,7 +44,13 @@ notmuch_dump_command (unused (void *ctx), int argc, char 
*argv[])
 char *output_file_name = NULL;
 int opt_index;
 
+int output_format = DUMP_FORMAT_SUP;
+
 notmuch_opt_desc_t options[] = {
+   { NOTMUCH_OPT_KEYWORD, output_format, format, 'f',
+ (notmuch_keyword_t []){ { sup, DUMP_FORMAT_SUP },
+ { batch-tag, DUMP_FORMAT_BATCH_TAG },
+ { 0, 0 } } },
{ NOTMUCH_OPT_STRING, output_file_name, output, 'o', 0  },
{ 0, 0, 0, 0, 0 }
 };
@@ -83,27 +90,56 @@ notmuch_dump_command (unused (void *ctx), int argc, char 
*argv[])
  */
 notmuch_query_set_sort (query, NOTMUCH_SORT_UNSORTED);
 
+char *buffer = NULL;
+size_t buffer_size = 0;
+
 for (messages = notmuch_query_search_messages (query);
 notmuch_messages_valid (messages);
 notmuch_messages_move_to_next (messages)) {
int first = 1;
+   const char *message_id;
+
message = notmuch_messages_get (messages);
+   message_id = notmuch_message_get_message_id (message);
 
-   fprintf (output,
-%s (, notmuch_message_get_message_id (message));
+   if (output_format == DUMP_FORMAT_SUP) {
+   fprintf (output, %s (, message_id);
+   }
 
for (tags = notmuch_message_get_tags (message);
 notmuch_tags_valid (tags);
 notmuch_tags_move_to_next (tags)) {
-   if (! first)
-   fprintf (output,  );
+   const char *tag_str = notmuch_tags_get (tags);
 
-   fprintf (output, %s, notmuch_tags_get (tags));
+   if (! first)
+   fputs ( , output);
 
first = 0;
+
+   if (output_format == DUMP_FORMAT_SUP) {
+   fputs (tag_str, output);
+   } else {
+   if (hex_encode (notmuch, tag_str,
+   buffer, buffer_size) != HEX_SUCCESS) {
+   fprintf (stderr, Error: failed to hex-encode tag %s\n,
+tag_str);
+   return 1;
+   }
+   fprintf (output, +%s, buffer);
+   }
}
 
-   fprintf (output, )\n);
+   if (output_format == DUMP_FORMAT_SUP) {
+   fputs ()\n, output);
+   } else {
+   if (hex_encode (notmuch, message_id,
+   buffer, buffer_size) != HEX_SUCCESS) {
+   fprintf (stderr, Error: failed to hex-encode msg-id %s\n,
+message_id);
+   return 1;
+   }
+   fprintf (output,  -- id:%s\n, buffer);
+   }
 
notmuch_message_destroy (message);
 }
-- 
1.7.10.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[Patch v4 02/10] test: add sanity check for dump --format=batch-tag.

[david]

From: David Bremner brem...@debian.org

It's important this does not rely on restore, since it hasn't been
written yet.
---
 test/dump-restore |   13 +
 1 file changed, 13 insertions(+)

diff --git a/test/dump-restore b/test/dump-restore
index bf31266..b4c807f 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -85,6 +85,19 @@ test_begin_subtest dump --output=outfile -- from:cworth
 notmuch dump --output=dump-outfile-dash-inbox.actual -- from:cworth
 test_expect_equal_file dump-cworth.expected dump-outfile-dash-inbox.actual
 
+test_begin_subtest Check for a safe set of message-ids
+notmuch search --output=messages from:cworth | sed s/^id://  EXPECTED
+notmuch search --output=messages from:cworth | sed s/^id:// |\
+   $TEST_DIRECTORY/hex-xcode --direction=encode  OUTPUT
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest format=batch-tag, dump sanity check.
+notmuch dump --format=sup from:cworth | cut -f1 -d' ' | \
+sort  EXPECTED.$test_count
+notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
+sort  OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
 test_begin_subtest 'roundtripping random message-ids and tags'
 test_subtest_known_broken
 ${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
-- 
1.7.10.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[Patch v4 03/10] util: add string-util.[ch]

[david]

From: David Bremner brem...@debian.org

This is to give a home to strtok_len. It's a bit silly to add a header
for one routine, but it needs to be shared between several compilation
units (or at least that's the most natural design).
---
 util/Makefile.local |3 ++-
 util/string-util.c  |   34 ++
 util/string-util.h  |   22 ++
 3 files changed, 58 insertions(+), 1 deletion(-)
 create mode 100644 util/string-util.c
 create mode 100644 util/string-util.h

diff --git a/util/Makefile.local b/util/Makefile.local
index 3ca623e..a11e35b 100644
--- a/util/Makefile.local
+++ b/util/Makefile.local
@@ -3,7 +3,8 @@
 dir := util
 extra_cflags += -I$(srcdir)/$(dir)
 
-libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c
+libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c \
+ $(dir)/string-util.c
 
 libutil_modules := $(libutil_c_srcs:.c=.o)
 
diff --git a/util/string-util.c b/util/string-util.c
new file mode 100644
index 000..44f8cd3
--- /dev/null
+++ b/util/string-util.c
@@ -0,0 +1,34 @@
+/* string-util.c -  Extra or enhanced routines for null terminated strings.
+ *
+ * Copyright (c) 2012 Jani Nikula
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Jani Nikula j...@nikula.org
+ */
+
+
+#include string-util.h
+
+char *
+strtok_len (char *s, const char *delim, size_t *len)
+{
+/* skip initial delims */
+s += strspn (s, delim);
+
+/* length of token */
+*len = strcspn (s, delim);
+
+return *len ? s : NULL;
+}
diff --git a/util/string-util.h b/util/string-util.h
new file mode 100644
index 000..ac7676c
--- /dev/null
+++ b/util/string-util.h
@@ -0,0 +1,22 @@
+#ifndef _STRING_UTIL_H
+#define _STRING_UTIL_H
+
+#include string.h
+
+/* like strtok(3), but without state, and doesn't modify s.  Return
+ * value is indicated by pointer and length, not null terminator.
+ *
+ * Usage pattern:
+ *
+ * const char *tok = input;
+ * const char *delim =  \t;
+ * size_t tok_len = 0;
+ *
+ * while ((tok = strtok_len (tok + tok_len, delim, tok_len)) != NULL) {
+ * // do stuff with string tok of length tok_len
+ * }
+ */
+
+char *strtok_len (char *s, const char *delim, size_t *len);
+
+#endif
-- 
1.7.10.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[Patch v4 06/10] test: update dump-restore roundtripping test for batch-tag format

[david]

From: David Bremner brem...@debian.org

Now we can actually round trip these crazy tags and and message ids.
hex-xcode is no longer needed as it's built in.
---
 test/dump-restore |   17 -
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/test/dump-restore b/test/dump-restore
index b4c807f..ce81e6f 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -99,23 +99,22 @@ notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- 
id://' | \
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
 
 test_begin_subtest 'roundtripping random message-ids and tags'
-test_subtest_known_broken
-${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
-   --num-messages=10
 
- notmuch dump| \
-${TEST_DIRECTORY}/hex-xcode --direction=encode| \
+${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG}
+   --num-messages=100
+
+ notmuch dump --format=batch-tag| \
 sort  EXPECTED.$test_count
 
  notmuch tag +this_tag_is_very_unlikely_to_be_random '*'
 
- ${TEST_DIRECTORY}/hex-xcode --direction=decode  EXPECTED.$test_count | \
-notmuch restore 2/dev/null
+ notmuch restore --format=batch-tag  EXPECTED.$test_count
 
- notmuch dump| \
-${TEST_DIRECTORY}/hex-xcode --direction=encode| \
+ notmuch dump --format=batch-tag| \
 sort  OUTPUT.$test_count
 
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
 
 test_done
+
+# Note the database is poisoned for sup format at this point.
-- 
1.7.10.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[Patch v4 07/10] test: second set of dump/restore --format=batch-tag tests

[david]

From: David Bremner brem...@debian.org

These one need the completed functionality in notmuch-restore. Fairly
exotic tags are tested, but no weird message id's.

We test each possible input to autodetection, both explicit (with
--format=auto) and implicit (without --format).
---
 test/dump-restore |   85 -
 1 file changed, 84 insertions(+), 1 deletion(-)

diff --git a/test/dump-restore b/test/dump-restore
index ce81e6f..7d91cca 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -98,9 +98,92 @@ notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- 
id://' | \
 sort  OUTPUT.$test_count
 test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
 
+test_begin_subtest format=batch-tag, # round-trip
+notmuch dump --format=sup | sort  EXPECTED.$test_count
+notmuch dump --format=batch-tag | notmuch restore --format=batch-tag
+notmuch dump --format=sup | sort  OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest format=batch-tag, # blank lines and comments
+notmuch dump --format=batch-tag| sort  EXPECTED.$test_count
+notmuch restore EOF
+# this line is a comment; the next has only white space
+
+
+# the previous line is empty
+EOF
+notmuch dump --format=batch-tag | sort  OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest format=batch-tag, # reverse-round-trip empty tag
+cat EOF EXPECTED.$test_count
++ -- id:20091117232137.ga7...@griffis1.net
+EOF
+notmuch restore --format=batch-tag  EXPECTED.$test_count
+notmuch dump --format=batch-tag id:20091117232137.ga7...@griffis1.net  
OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+tag1='comic_swear=$^%$^%\\//-+$^%$'
+enc1=$($TEST_DIRECTORY/hex-xcode --direction=encode $tag1)
+
+tag2=$(printf 'this\n tag\t has\n spaces')
+enc2=$($TEST_DIRECTORY/hex-xcode --direction=encode $tag2)
+
+enc3='%c3%91%c3%a5%c3%b0%c3%a3%c3%a5%c3%a9-%c3%8f%c3%8a'
+tag3=$($TEST_DIRECTORY/hex-xcode --direction=decode $enc3)
+
+notmuch dump --format=batch-tag  BACKUP
+
+notmuch tag +$tag1 +$tag2 +$tag3 -inbox -unread *
+
+test_begin_subtest 'format=batch-tag, round trip with strange tags'
+notmuch dump --format=batch-tag  EXPECTED.$test_count
+notmuch dump --format=batch-tag | notmuch restore --format=batch-tag
+notmuch dump --format=batch-tag  OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'format=batch-tag, checking encoded output'
+notmuch dump --format=batch-tag -- from:cworth |\
+awk { print \+$enc1 +$enc2 +$enc3 -- \ \$5 }  EXPECTED.$test_count
+notmuch dump --format=batch-tag -- from:cworth   OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'restoring sane tags'
+notmuch restore --format=batch-tag  BACKUP
+notmuch dump --format=batch-tag  OUTPUT.$test_count
+test_expect_equal_file BACKUP OUTPUT.$test_count
+
+test_begin_subtest 'format=batch-tag, restore=auto'
+notmuch dump --format=batch-tag  EXPECTED.$test_count
+notmuch tag -inbox -unread *
+notmuch restore --format=auto  EXPECTED.$test_count
+notmuch dump --format=batch-tag  OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'format=sup, restore=auto'
+notmuch dump --format=sup  EXPECTED.$test_count
+notmuch tag -inbox -unread *
+notmuch restore --format=auto  EXPECTED.$test_count
+notmuch dump --format=sup  OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'format=batch-tag, restore=default'
+notmuch dump --format=batch-tag  EXPECTED.$test_count
+notmuch tag -inbox -unread *
+notmuch restore  EXPECTED.$test_count
+notmuch dump --format=batch-tag  OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest 'format=sup, restore=default'
+notmuch dump --format=sup  EXPECTED.$test_count
+notmuch tag -inbox -unread *
+notmuch restore  EXPECTED.$test_count
+notmuch dump --format=sup  OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
 test_begin_subtest 'roundtripping random message-ids and tags'
 
-${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG}
+${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \
--num-messages=100
 
  notmuch dump --format=batch-tag| \
-- 
1.7.10.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[Patch v4 04/10] tag-util.[ch]: New files for common tagging routines

[david]

From: David Bremner brem...@debian.org

These are meant to be shared between notmuch-tag and notmuch-restore.

The bulk of the routines implement a tag operation list abstract
data type act as a structured representation of a set of tag
operations (typically coming from a single tag command or line of
input).
---
 Makefile.local |1 +
 tag-util.c |  292 
 tag-util.h |  122 +++
 3 files changed, 415 insertions(+)
 create mode 100644 tag-util.c
 create mode 100644 tag-util.h

diff --git a/Makefile.local b/Makefile.local
index 0db1713..c274f07 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -275,6 +275,7 @@ notmuch_client_srcs =   \
query-string.c  \
mime-node.c \
crypto.c\
+   tag-util.c
 
 notmuch_client_modules = $(notmuch_client_srcs:.c=.o)
 
diff --git a/tag-util.c b/tag-util.c
new file mode 100644
index 000..80ebdc2
--- /dev/null
+++ b/tag-util.c
@@ -0,0 +1,292 @@
+#include assert.h
+#include string-util.h
+#include tag-util.h
+#include hex-escape.h
+
+#define TAG_OP_LIST_INITIAL_SIZE 10
+
+struct _tag_operation_t {
+const char *tag;
+notmuch_bool_t remove;
+};
+
+struct _tag_op_list_t {
+tag_operation_t *ops;
+size_t count;
+size_t size;
+};
+
+int
+parse_tag_line (void *ctx, char *line,
+   tag_op_flag_t flags,
+   char **query_string,
+   tag_op_list_t *tag_ops)
+{
+char *tok = line;
+size_t tok_len = 0;
+char *line_for_error;
+int ret = 0;
+
+chomp_newline (line);
+
+line_for_error = talloc_strdup (ctx, line);
+if (line_for_error == NULL) {
+   fprintf (stderr, Error: out of memory\n);
+   return -1;
+}
+
+/* remove leading space */
+while (*tok == ' ' || *tok == '\t')
+   tok++;
+
+/* Skip empty and comment lines. */
+if (*tok == '\0' || *tok == '#') {
+   ret = 2;
+   goto DONE;
+}
+
+tag_op_list_reset (tag_ops);
+
+/* Parse tags. */
+while ((tok = strtok_len (tok + tok_len,  , tok_len)) != NULL) {
+   notmuch_bool_t remove;
+   char *tag;
+
+   /* Optional explicit end of tags marker. */
+   if (tok_len == 2  strncmp (tok, --, tok_len) == 0) {
+   tok = strtok_len (tok + tok_len,  , tok_len);
+   if (tok == NULL)
+   fprintf (stderr, Warning: no query string: %s\n, 
line_for_error);
+   break;
+   }
+
+   /* Implicit end of tags. */
+   if (*tok != '-'  *tok != '+')
+   break;
+
+   /* If tag is terminated by NUL, there's no query string. */
+   if (*(tok + tok_len) == '\0') {
+   fprintf (stderr, Warning: no query string: %s\n, line_for_error);
+   ret = 1;
+   goto DONE;
+   }
+
+   /* Terminate, and start next token after terminator. */
+   *(tok + tok_len++) = '\0';
+
+   remove = (*tok == '-');
+   tag = tok + 1;
+
+   /* Maybe refuse empty tags. */
+   if (! (flags  TAG_FLAG_BE_GENEROUS)  *tag == '\0') {
+   fprintf (stderr, Warning: empty tag: %s\n, line_for_error);
+   ret = 1;
+   goto DONE;
+   }
+
+   /* Decode tag. */
+   if (hex_decode_inplace (tag) != HEX_SUCCESS) {
+   fprintf (stderr, Warning: Hex decoding of tag %s failed\n,
+tag);
+   ret = 1;
+   goto DONE;
+   }
+
+   if (tag_op_list_append (ctx, tag_ops, tag, remove)) {
+   /* diagnostics already printed */
+   ret = -1;
+   goto DONE;
+   }
+}
+
+if (tok == NULL) {
+   ret = 1;
+   goto DONE;
+}
+
+/* tok now points to the query string */
+if (hex_decode_inplace (tok) != HEX_SUCCESS) {
+   fprintf (stderr, Warning: Hex decoding of query %s failed\n,
+tok);
+   ret = 1;
+   goto DONE;
+}
+
+*query_string = tok;
+
+  DONE:
+if ((ret % 2) != 0)
+   fprintf (stderr, %s invalid input line %s\n,
+ret == 1 ? Warning: Ignoring : Error: Failing at,
+line_for_error);
+
+talloc_free (line_for_error);
+return ret;
+}
+
+static inline void
+message_error (notmuch_message_t *message,
+  notmuch_status_t status,
+  const char *format, ...)
+{
+va_list va_args;
+
+va_start (va_args, format);
+
+vfprintf (stderr, format, va_args);
+fprintf (stderr, Message-ID: %s\n, notmuch_message_get_message_id 
(message));
+fprintf (stderr, Status: %s\n, notmuch_status_to_string (status));
+}
+
+notmuch_status_t
+tag_op_list_apply (notmuch_message_t *message,
+  tag_op_list_t *list,
+  tag_op_flag_t flags)
+{
+size_t i;
+notmuch_status_t status = 0;
+tag_operation_t *tag_ops = list-ops;
+
+status = notmuch_message_freeze (message);
+if (status) {
+   message_error (message, status, freezing 

[Patch v4 10/10] tag-util: optimization of tag application

[david]

From: David Bremner brem...@debian.org

The idea is not to bother with restore operations if they don't change
the set of tags. This is actually a relatively common case.

In order to avoid fancy datastructures, this method is quadratic in
the number of tags; at least on my mail database this doesn't seem to
be a big problem.
---
 tag-util.c |   68 
 1 file changed, 68 insertions(+)

diff --git a/tag-util.c b/tag-util.c
index 80ebdc2..b68ea50 100644
--- a/tag-util.c
+++ b/tag-util.c
@@ -138,6 +138,71 @@ message_error (notmuch_message_t *message,
 fprintf (stderr, Status: %s\n, notmuch_status_to_string (status));
 }
 
+static int
+makes_changes (notmuch_message_t *message,
+  tag_op_list_t *list,
+  tag_op_flag_t flags)
+{
+
+size_t i;
+
+notmuch_tags_t *tags;
+notmuch_bool_t changes = FALSE;
+
+/* First, do we delete an existing tag? */
+changes = FALSE;
+for (tags = notmuch_message_get_tags (message);
+! changes  notmuch_tags_valid (tags);
+notmuch_tags_move_to_next (tags)) {
+   const char *cur_tag = notmuch_tags_get (tags);
+   int last_op =  (flags  TAG_FLAG_REMOVE_ALL) ? -1 : 0;
+
+   /* scan backwards to get last operation */
+   i = list-count;
+   while (i  0) {
+   i--;
+   if (strcmp (cur_tag, list-ops[i].tag) == 0) {
+   last_op = list-ops[i].remove ? -1 : 1;
+   break;
+   }
+   }
+
+   changes = (last_op == -1);
+}
+notmuch_tags_destroy (tags);
+
+if (changes)
+   return TRUE;
+
+/* Now check for adding new tags */
+for (i = 0; i  list-count; i++) {
+   notmuch_bool_t exists = FALSE;
+
+   if (list-ops[i].remove)
+   continue;
+
+   for (tags = notmuch_message_get_tags (message);
+notmuch_tags_valid (tags);
+notmuch_tags_move_to_next (tags)) {
+   const char *cur_tag = notmuch_tags_get (tags);
+   if (strcmp (cur_tag, list-ops[i].tag) == 0) {
+   exists = TRUE;
+   break;
+   }
+   }
+   notmuch_tags_destroy (tags);
+
+   /* the following test is conservative,
+* in the sense it ignores cases like +foo ... -foo
+* but this is OK from a correctness point of view
+*/
+   if (! exists)
+   return TRUE;
+}
+return FALSE;
+
+}
+
 notmuch_status_t
 tag_op_list_apply (notmuch_message_t *message,
   tag_op_list_t *list,
@@ -147,6 +212,9 @@ tag_op_list_apply (notmuch_message_t *message,
 notmuch_status_t status = 0;
 tag_operation_t *tag_ops = list-ops;
 
+if (! (flags  TAG_FLAG_PRE_OPTIMIZED)  ! makes_changes (message, list, 
flags))
+   return NOTMUCH_STATUS_SUCCESS;
+
 status = notmuch_message_freeze (message);
 if (status) {
message_error (message, status, freezing message);
-- 
1.7.10.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Tags with spaces

[David Bremner]

Austin Clements amdra...@mit.edu writes:

 Tags with spaces are supported, but your shell will strip away the
 quotes you've used in both examples before notmuch ever sees them.
 You'll need both shell quoting and Xapian quoting in this case.  Try
   notmuch search -- '-tag:This has spaces'
 This should be easier from a front-end, since it will take care of any
 necessary shell quoting.

Note that there are also patches in the works that add a hex encoded
format for notmuch-tag on standard input; so far we have tried to do
that for regular command line arguments, but it's a possibility.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-07 13:49:46)
 Not sure what is causing this. My best guess is that your password was
 incorrect and that I'm not checking the login return value.

Yes, you're right, it was an incorrect passwd.

 One thing you also might want to watch out for is
 exceptions due to a locked notmuch index.
 
 
 How do I do this?

See e.g. here: https://github.com/pazz/alot/blob/master/alot/db/manager.py#L163
I'm sure afew (https://github.com/teythoon/afew) does a similar thing.


I have two new errors:

-
./gmail-notmuch.py -u patricktot...@gmail.com -p mypwd ~/mail/gmail/
Logging in...
Selecting all mail...
Error opening database at /home/pazz/mail/gmail/.notmuch: No such file or 
directory
Traceback (most recent call last):
  File ./gmail-notmuch.py, line 225, in module
main()
  File ./gmail-notmuch.py, line 56, in main
database = notmuch.Database(destination, False, 
notmuch.Database.MODE.READ_WRITE)
  File /home/pazz/.local/lib/python2.7/site-packages/notmuch/database.py, 
line 154, in __init__
self.open(path, mode)
  File /home/pazz/.local/lib/python2.7/site-packages/notmuch/database.py, 
line 214, in open
raise NotmuchError(status)
notmuch.errors.FileError
-

I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
as this is where my all-mail maildir is, and i also tried an absolute path, all 
the same result.

If I point it to the root of my notmuch directory I get:
-
[~/projects/gmail-notmuch] ./gmail-notmuch.py -u patricktot...@gmail.com -p 
mypwd /home/pazz/mail
Logging in...
Selecting all mail...
Traceback (most recent call last):
  File ./gmail-notmuch.py, line 225, in module
main()
  File ./gmail-notmuch.py, line 58, in main
messages = discover_messages(imap, total)
  File ./gmail-notmuch.py, line 104, in discover_messages
new_readline.progressbar = create_progressbar(Receiving message list, 
total).start()
  File ./gmail-notmuch.py, line 147, in create_progressbar
return ProgressBar(maxval=total, widgets=[text + : , SimpleProgress(), 
Bar(), Percentage(),  , ETA(),  , FileTransferSpeed(unit=emails)])
TypeError: __init__() got an unexpected keyword argument 'unit'

BTW: I had to change the hardcoded [Gmail] to [Google Mail] for it to get 
that far.


HTH,
/p


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch