Do path: searches handle spaces correctly?

2014-09-22 Thread Keith Amidon
I'm trying to update an archiving tool that used the old folder: search
terms to use the newer folder: or path: terms.  From playing around with
it, I think that the path: term is what I want to use.  However, I am
getting some behavior I don't understand unless path: searches don't
properly handle spaces or I don't know how to quote them properly.

If I do:

notmuch search --output=files path:'dir/INBOX/**'

I get:

base/dir/INBOX/cur/1411244319_3.18990.kea-tablet,U=24,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
base/dir/INBOX/cur/1411244319_2.18990.kea-tablet,U=23,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
base/dir/INBOX/INBOX/Sent 
Items/cur/1411244324_2.18990.kea-tablet,U=3,FMD5=e99e6ef6cc1489bb6d8b47a4c49bb989:2,S
base/dir/INBOX/cur/1411244319_1.18990.kea-tablet,U=22,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,RS
base/dir/INBOX/cur/1411244319_0.18990.kea-tablet,U=21,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
base/dir/INBOX/cur/1411244318_3.18990.kea-tablet,U=20,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
base/dir/INBOX/cur/1411244318_2.18990.kea-tablet,U=19,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
base/dir/INBOX/cur/1411244318_1.18990.kea-tablet,U=18,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S
base/dir/INBOX/INBOX/Sent 
Items/cur/1411244324_1.18990.kea-tablet,U=2,FMD5=e99e6ef6cc1489bb6d8b47a4c49bb989:2,S
base/dir/INBOX/cur/1411244318_0.18990.kea-tablet,U=17,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S

Notice that this contains results for dir/INBOX/INBOX/Sent Items,
which has a space in it.  If I do:

notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items'

I don't get any results, but it seems like the two results above should
be shown, right?  Am I doing something wrong here?  If it looks like I'm
doing it correctly, what can I do to help troubleshoot the issue?

 Thanks, Keith
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: Do path: searches handle spaces correctly?

2014-09-22 Thread David Bremner
Keith Amidon ke...@awakenetworks.com writes:


 notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items'

 I don't get any results, but it seems like the two results above should
 be shown, right?  Am I doing something wrong here?  If it looks like I'm
 doing it correctly, what can I do to help troubleshoot the issue?

Note that path:, unlike folder:, does not add the maildir subdirs. Dunno
if that's the only issue you are having, but I guess it's one.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 1/5] cli: Refactor option passing in the search command

2014-09-22 Thread Michal Sojka
Many functions that implement the search command need to access command
line options. Instead of passing each option in a separate variable, put
them in a structure and pass only this structure.

This will become handy in the following patches.
---
 notmuch-search.c | 122 ---
 1 file changed, 62 insertions(+), 60 deletions(-)

diff --git a/notmuch-search.c b/notmuch-search.c
index bc9be45..5ac2a26 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -30,6 +30,16 @@ typedef enum {
 OUTPUT_TAGS
 } output_t;
 
+typedef struct {
+sprinter_t *format;
+notmuch_query_t *query;
+notmuch_sort_t sort;
+output_t output;
+int offset;
+int limit;
+int dupe;
+} search_options_t;
+
 /* Return two stable query strings that identify exactly the matched
  * and unmatched messages currently in thread.  If there are no
  * matched or unmatched messages, the returned buffers will be
@@ -70,46 +80,42 @@ get_thread_query (notmuch_thread_t *thread,
 }
 
 static int
-do_search_threads (sprinter_t *format,
-  notmuch_query_t *query,
-  notmuch_sort_t sort,
-  output_t output,
-  int offset,
-  int limit)
+do_search_threads (search_options_t *o)
 {
 notmuch_thread_t *thread;
 notmuch_threads_t *threads;
 notmuch_tags_t *tags;
+sprinter_t *format = o-format;
 time_t date;
 int i;
 
-if (offset  0) {
-   offset += notmuch_query_count_threads (query);
-   if (offset  0)
-   offset = 0;
+if (o-offset  0) {
+   o-offset += notmuch_query_count_threads (o-query);
+   if (o-offset  0)
+   o-offset = 0;
 }
 
-threads = notmuch_query_search_threads (query);
+threads = notmuch_query_search_threads (o-query);
 if (threads == NULL)
return 1;
 
 format-begin_list (format);
 
 for (i = 0;
-notmuch_threads_valid (threads)  (limit  0 || i  offset + limit);
+notmuch_threads_valid (threads)  (o-limit  0 || i  o-offset + 
o-limit);
 notmuch_threads_move_to_next (threads), i++)
 {
thread = notmuch_threads_get (threads);
 
-   if (i  offset) {
+   if (i  o-offset) {
notmuch_thread_destroy (thread);
continue;
}
 
-   if (output == OUTPUT_THREADS) {
+   if (o-output == OUTPUT_THREADS) {
format-set_prefix (format, thread);
format-string (format,
-   notmuch_thread_get_thread_id (thread));
+  notmuch_thread_get_thread_id (thread));
format-separator (format);
} else { /* output == OUTPUT_SUMMARY */
void *ctx_quote = talloc_new (thread);
@@ -123,7 +129,7 @@ do_search_threads (sprinter_t *format,
 
format-begin_map (format);
 
-   if (sort == NOTMUCH_SORT_OLDEST_FIRST)
+   if (o-sort == NOTMUCH_SORT_OLDEST_FIRST)
date = notmuch_thread_get_oldest_date (thread);
else
date = notmuch_thread_get_newest_date (thread);
@@ -215,40 +221,36 @@ do_search_threads (sprinter_t *format,
 }
 
 static int
-do_search_messages (sprinter_t *format,
-   notmuch_query_t *query,
-   output_t output,
-   int offset,
-   int limit,
-   int dupe)
+do_search_messages (search_options_t *o)
 {
 notmuch_message_t *message;
 notmuch_messages_t *messages;
 notmuch_filenames_t *filenames;
+sprinter_t *format = o-format;
 int i;
 
-if (offset  0) {
-   offset += notmuch_query_count_messages (query);
-   if (offset  0)
-   offset = 0;
+if (o-offset  0) {
+   o-offset += notmuch_query_count_messages (o-query);
+   if (o-offset  0)
+   o-offset = 0;
 }
 
-messages = notmuch_query_search_messages (query);
+messages = notmuch_query_search_messages (o-query);
 if (messages == NULL)
return 1;
 
 format-begin_list (format);
 
 for (i = 0;
-notmuch_messages_valid (messages)  (limit  0 || i  offset + limit);
+notmuch_messages_valid (messages)  (o-limit  0 || i  o-offset + 
o-limit);
 notmuch_messages_move_to_next (messages), i++)
 {
-   if (i  offset)
+   if (i  o-offset)
continue;
 
message = notmuch_messages_get (messages);
 
-   if (output == OUTPUT_FILES) {
+   if (o-output == OUTPUT_FILES) {
int j;
filenames = notmuch_message_get_filenames (message);
 
@@ -256,7 +258,7 @@ do_search_messages (sprinter_t *format,
 notmuch_filenames_valid (filenames);
 notmuch_filenames_move_to_next (filenames), j++)
{
-   if (dupe  0 || dupe == j) {
+   if (o-dupe  0 || o-dupe == j) {
format-string (format, notmuch_filenames_get (filenames));
 

[PATCH 5/5] cli: Add tests for 'search --output=addresses' and similar

2014-09-22 Thread Michal Sojka
---
 test/T090-search-output.sh | 59 +++
 test/T095-search-unique.sh | 63 ++
 2 files changed, 122 insertions(+)
 create mode 100755 test/T095-search-unique.sh

diff --git a/test/T090-search-output.sh b/test/T090-search-output.sh
index 947d572..ebc8c37 100755
--- a/test/T090-search-output.sh
+++ b/test/T090-search-output.sh
@@ -387,6 +387,65 @@ cat EOF EXPECTED
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest --output=sender
+notmuch search --output=sender '*' | sort OUTPUT
+cat EOF EXPECTED
+Adrian Perez de Castro ape...@igalia.com
+Alexander Botero-Lowry alex.boterolo...@gmail.com
+Aron Griffis agrif...@n01se.net
+Carl Worth cwo...@cworth.org
+Chris Wilson ch...@chris-wilson.co.uk
+Fran├žois Boulogne boulogn...@gmail.com
+Ingmar Vanhassel ing...@exherbo.org
+Israel Herraiz i...@herraiz.org
+Jan Janak j...@ryngle.com
+Jjgod Jiang gzjj...@gmail.com
+Keith Packard kei...@keithp.com
+Lars Kellogg-Stedman l...@seas.harvard.edu
+Mikhail Gusarov dotted...@dottedmag.net
+Olivier Berger olivier.ber...@it-sudparis.eu
+Rolland Santimano rollandsantim...@yahoo.com
+Stewart Smith stew...@flamingspork.com
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest --output=recipients
+notmuch search --output=recipients '*' | sort OUTPUT
+cat EOF EXPECTED
+Allan McRae al...@archlinux.org
+Discussion about the Arch User Repository (AUR) aur-gene...@archlinux.org
+Keith Packard kei...@keithp.com
+Mikhail Gusarov dotted...@dottedmag.net
+notmuch@notmuchmail.org
+olivier.ber...@it-sudparis.eu
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest --output=addresses
+notmuch search --output=addresses '*' | sort OUTPUT
+cat EOF EXPECTED
+Adrian Perez de Castro ape...@igalia.com
+Alexander Botero-Lowry alex.boterolo...@gmail.com
+Allan McRae al...@archlinux.org
+Aron Griffis agrif...@n01se.net
+Carl Worth cwo...@cworth.org
+Chris Wilson ch...@chris-wilson.co.uk
+Discussion about the Arch User Repository (AUR) aur-gene...@archlinux.org
+Fran├žois Boulogne boulogn...@gmail.com
+Ingmar Vanhassel ing...@exherbo.org
+Israel Herraiz i...@herraiz.org
+Jan Janak j...@ryngle.com
+Jjgod Jiang gzjj...@gmail.com
+Keith Packard kei...@keithp.com
+Lars Kellogg-Stedman l...@seas.harvard.edu
+Mikhail Gusarov dotted...@dottedmag.net
+Olivier Berger olivier.ber...@it-sudparis.eu
+Rolland Santimano rollandsantim...@yahoo.com
+Stewart Smith stew...@flamingspork.com
+notmuch@notmuchmail.org
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest sanitize output for quoted-printable line-breaks in author 
and subject
 add_message [subject]='two =?ISO-8859-1?Q?line=0A_subject?=
headers'
diff --git a/test/T095-search-unique.sh b/test/T095-search-unique.sh
new file mode 100755
index 000..8fd8fc0
--- /dev/null
+++ b/test/T095-search-unique.sh
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+test_description='address deduplication in notmuch search --output=addresses'
+. ./test-lib.sh
+
+add_message '[to]=Real Name f...@example.com, Real Name b...@example.com'
+add_message '[to]=Nickname f...@example.com' '[cc]=Real Name 
b...@example.com'
+add_message '[to]=Nickname f...@example.com' '[bcc]=Real Name 
b...@example.com'
+
+test_begin_subtest --output=recipients
+notmuch search --output=recipients * OUTPUT
+cat EOF EXPECTED
+Real Name f...@example.com
+Real Name b...@example.com
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest --output=recipients --unique=none
+notmuch search --output=recipients --unique=none * OUTPUT
+cat EOF EXPECTED
+Real Name f...@example.com
+Real Name b...@example.com
+Nickname f...@example.com
+Real Name b...@example.com
+Nickname f...@example.com
+Real Name b...@example.com
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest --output=recipients --unique=addr
+notmuch search --output=recipients --unique=addr * OUTPUT
+cat EOF EXPECTED
+Real Name f...@example.com
+Real Name b...@example.com
+Real Name b...@example.com
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest --output=recipients --unique=addrfold
+notmuch search --output=recipients --unique=addrfold * OUTPUT
+cat EOF EXPECTED
+Real Name f...@example.com
+Real Name b...@example.com
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest --output=recipients --unique=name
+notmuch search --output=recipients --unique=name * OUTPUT
+cat EOF EXPECTED
+Real Name f...@example.com
+Nickname f...@example.com
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest --output=recipients --unique=name,addrfold
+notmuch search --output=recipients --unique=name,addrfold * OUTPUT
+cat EOF EXPECTED
+Real Name f...@example.com
+Real Name b...@example.com
+Nickname f...@example.com
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_done
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 4/5] cli: Add configurable address deduplication for --output=addresses

2014-09-22 Thread Michal Sojka
The code here is an extended version of a path from Jani Nikula.
---
 completion/notmuch-completion.bash |   6 ++-
 completion/notmuch-completion.zsh  |   3 +-
 doc/man1/notmuch-search.rst|  33 
 notmuch-search.c   | 101 ++---
 4 files changed, 135 insertions(+), 8 deletions(-)

diff --git a/completion/notmuch-completion.bash 
b/completion/notmuch-completion.bash
index c37ddf5..8bc7874 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -305,12 +305,16 @@ _notmuch_search()
COMPREPLY=( $( compgen -W true false flag all -- ${cur} ) )
return
;;
+   --unique)
+   COMPREPLY=( $( compgen -W none addr addrfold name -- ${cur} ) )
+   return
+   ;;
 esac
 
 ! $split 
 case ${cur} in
-*)
-   local options=--format= --output= --sort= --offset= --limit= 
--exclude= --duplicate=
+   local options=--format= --output= --sort= --offset= --limit= 
--exclude= --duplicate= --unique=
compopt -o nospace
COMPREPLY=( $(compgen -W $options -- ${cur}) )
;;
diff --git a/completion/notmuch-completion.zsh 
b/completion/notmuch-completion.zsh
index bff8fd5..cf4968c 100644
--- a/completion/notmuch-completion.zsh
+++ b/completion/notmuch-completion.zsh
@@ -53,7 +53,8 @@ _notmuch_search()
 '--max-threads=[display only the first x threads from the search 
results]:number of threads to show: ' \
 '--first=[omit the first x threads from the search results]:number of 
threads to omit: ' \
 '--sort=[sort results]:sorting:((newest-first\:reverse chronological 
order oldest-first\:chronological order))' \
-'--output=[select what to output]:output:((summary threads messages files 
tags sender recipients addresses))'
+'--output=[select what to output]:output:((summary threads messages files 
tags sender recipients addresses))' \
+'--unique=[address deduplication]:unique:((none\:no deduplication 
addr\:deduplicate by address addrfold\:deduplicate by case-insensitive 
address name\:deduplicate by name))'
 }
 
 _notmuch()
diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
index 6094906..a92779a 100644
--- a/doc/man1/notmuch-search.rst
+++ b/doc/man1/notmuch-search.rst
@@ -85,6 +85,9 @@ Supported options for **search** include
 (--format=text0), as a JSON array (--format=json), or as
 an S-Expression list (--format=sexp).
 
+Handling of duplicate addresses and/or names can be
+controlled with the --unique option.
+
Note: Searching for **sender** should much be faster than
searching for **recipients** or **addresses**, because
sender addresses are cached directly in the database
@@ -151,6 +154,36 @@ Supported options for **search** include
 prefix. The prefix matches messages based on filenames. This
 option filters filenames of the matching messages.
 
+``--unique=``\ (**none**\ \|\ **addr**\ \|\ **addrfold**\ \|\ **name**)[,\ 
...]
+
+Can be used with ``--output=addresses``, ``--output=sender``
+or ``--output=recipients`` to control the address
+deduplication algorithm.
+
+   **none** means that no deduplication is performed. The same
+   address can appear multiple times in the output.
+
+   **addr** means that case-sensitive deduplication is performed
+   on the address part. For example, given the addresses John
+   Doe j...@example.com and Dr. John Doe j...@example.com,
+   only one will be printed.
+
+   **addrfold** means that case-insensitive deduplication is
+   performed on the address part. For example, given the
+   addresses John Doe j...@example.com and John Doe
+   j...@example.com, only one will be printed. This is the
+   default.
+
+   **name** means that case-sensitive deduplication is performed
+   on the name part. For example, given the addresses John Doe
+   j...@example.com and John Doe j...@doe.name, only one
+   will be printed.
+
+   It is possible to combine the above flags (except **none**) by
+   separating them with comma. For example,
+   ``--unique=name,addr`` will print unique case-sensitive
+   combinations of name and address.
+
 EXIT STATUS
 ===
 
diff --git a/notmuch-search.c b/notmuch-search.c
index 0614f10..00d6771 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -33,6 +33,15 @@ typedef enum {
 OUTPUT_ADDRESSES   = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
 } output_t;
 
+typedef enum {
+UNIQUE_NONE  = 1  0,
+UNIQUE_ADDR  = 1  1,
+UNIQUE_NAME  = 1  2,
+UNIQUE_ADDR_CASEFOLD  = 1  3,
+
+UNIQUE_BOTH = UNIQUE_NAME | UNIQUE_ADDR,
+} unique_t;
+
 typedef struct {
 sprinter_t *format;
 notmuch_query_t *query;
@@ -41,6 +50,7 @@ typedef struct {
 int offset;
 int 

[PATCH 2/5] cli: Extend the search command for --output=addresses and similar

2014-09-22 Thread Michal Sojka
The new outputs allow printing senders, recipients or both of matching
messages.

This code based on a patch from Jani Nikula.
---
 completion/notmuch-completion.bash |   2 +-
 completion/notmuch-completion.zsh  |   3 +-
 doc/man1/notmuch-search.rst|  22 +++-
 notmuch-search.c   | 100 ++---
 4 files changed, 118 insertions(+), 9 deletions(-)

diff --git a/completion/notmuch-completion.bash 
b/completion/notmuch-completion.bash
index 0571dc9..c37ddf5 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -294,7 +294,7 @@ _notmuch_search()
return
;;
--output)
-   COMPREPLY=( $( compgen -W summary threads messages files tags -- 
${cur} ) )
+   COMPREPLY=( $( compgen -W summary threads messages files tags 
sender recipients addresses -- ${cur} ) )
return
;;
--sort)
diff --git a/completion/notmuch-completion.zsh 
b/completion/notmuch-completion.zsh
index 67a9aba..bff8fd5 100644
--- a/completion/notmuch-completion.zsh
+++ b/completion/notmuch-completion.zsh
@@ -52,7 +52,8 @@ _notmuch_search()
   _arguments -s : \
 '--max-threads=[display only the first x threads from the search 
results]:number of threads to show: ' \
 '--first=[omit the first x threads from the search results]:number of 
threads to omit: ' \
-'--sort=[sort results]:sorting:((newest-first\:reverse chronological 
order oldest-first\:chronological order))'
+'--sort=[sort results]:sorting:((newest-first\:reverse chronological 
order oldest-first\:chronological order))' \
+'--output=[select what to output]:output:((summary threads messages files 
tags sender recipients addresses))'
 }
 
 _notmuch()
diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
index 90160f2..6094906 100644
--- a/doc/man1/notmuch-search.rst
+++ b/doc/man1/notmuch-search.rst
@@ -35,7 +35,7 @@ Supported options for **search** include
 intended for programs that invoke **notmuch(1)** internally. If
 omitted, the latest supported version will be used.
 
-``--output=(summary|threads|messages|files|tags)``
+
``--output=(summary|threads|messages|files|tags|sender|recipients|addresses)``
 
 **summary**
 Output a summary of each thread with any message matching
@@ -78,6 +78,26 @@ Supported options for **search** include
 by null characters (--format=text0), as a JSON array
 (--format=json), or as an S-Expression list (--format=sexp).
 
+   **sender**
+Output all addresses from the *From* header that appear on
+any message matching the search terms, either one per line
+(--format=text), separated by null characters
+(--format=text0), as a JSON array (--format=json), or as
+an S-Expression list (--format=sexp).
+
+   Note: Searching for **sender** should much be faster than
+   searching for **recipients** or **addresses**, because
+   sender addresses are cached directly in the database
+   whereas other addresses need to be fetched from the
+   message file by parsing it.
+
+   **recipients**
+Like **sender** but for addresses from *To*, *Cc* and
+   *Bcc* headers.
+
+   **addresses**
+   Like **sender** and **recipients** together.
+
 ``--sort=``\ (**newest-first**\ \|\ **oldest-first**)
 This option can be used to present results in either
 chronological order (**oldest-first**) or reverse chronological
diff --git a/notmuch-search.c b/notmuch-search.c
index 5ac2a26..0614f10 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -23,11 +23,14 @@
 #include string-util.h
 
 typedef enum {
-OUTPUT_SUMMARY,
-OUTPUT_THREADS,
-OUTPUT_MESSAGES,
-OUTPUT_FILES,
-OUTPUT_TAGS
+OUTPUT_SUMMARY = 1  0,
+OUTPUT_THREADS = 1  1,
+OUTPUT_MESSAGES= 1  2,
+OUTPUT_FILES   = 1  3,
+OUTPUT_TAGS= 1  4,
+OUTPUT_SENDER  = 1  5,
+OUTPUT_RECIPIENTS  = 1  6,
+OUTPUT_ADDRESSES   = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
 } output_t;
 
 typedef struct {
@@ -220,6 +223,67 @@ do_search_threads (search_options_t *o)
 return 0;
 }
 
+static void
+print_address_list (const search_options_t *o, InternetAddressList *list)
+{
+InternetAddress *address;
+int i;
+
+for (i = 0; i  internet_address_list_length (list); i++) {
+   address = internet_address_list_get_address (list, i);
+   if (INTERNET_ADDRESS_IS_GROUP (address)) {
+   InternetAddressGroup *group;
+   InternetAddressList *group_list;
+
+   group = INTERNET_ADDRESS_GROUP (address);
+   group_list = internet_address_group_get_members (group);
+   if (group_list == NULL)
+   continue;
+
+   print_address_list (o, group_list);
+   } else {
+   

[PATCH 0/5] notmuch search --output=addresses

2014-09-22 Thread Michal Sojka
Hello all,

this patch series adds support for --output=addresses and similar
arguments in notmuch search. It is a reworked and extended version of
id:1410021689-15901-1-git-send-email-j...@nikula.org.

Compared to Jani's patches, I do not use keyword-flag command line
parser for the --output option, because this would not be useful for
most but one combination of keywords. Instead, this one combination is
explicitely mentioned as another keyword.

The flag command-line parser is introduced later, with (IMHO) better
syntax: --flags=one,two,... This feature is used to control address
deduplication via the --unique option.

Other extensions are added documentation, tests and shell completions.

The whole test suite passes with these patches.

The functionality presented here will be useful for improving Emacs
address completion (id:1411150602-21892-1-git-send-email-sojk...@fel.cvut.cz).

Another nice use case is sending patched with git:

  git send-email --cc-cmd='nmsenders id:' ...

where nmsenders script contains:

  notmuch search --output=sender $(notmuch search --output=threads $1)



Michal Sojka (5):
  cli: Refactor option passing in the search command
  cli: Extend the search command for --output=addresses and similar
  cli: Add support for parsing command line flag options
  cli: Add configurable address deduplication for --output=addresses
  cli: Add tests for 'search --output=addresses' and similar

 command-line-arguments.c   |  40 +
 command-line-arguments.h   |   1 +
 completion/notmuch-completion.bash |   8 +-
 completion/notmuch-completion.zsh  |   4 +-
 doc/man1/notmuch-search.rst|  55 ++-
 notmuch-search.c   | 311 +
 test/Makefile.local|   2 +-
 test/T090-search-output.sh |  59 +++
 test/T095-search-unique.sh |  63 
 test/T410-argument-parsing.sh  |   3 +-
 test/arg-test.c|   8 +
 11 files changed, 482 insertions(+), 72 deletions(-)
 create mode 100755 test/T095-search-unique.sh

-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 00/11] notmuch insert updates

2014-09-22 Thread Jani Nikula
This series refactors and cleans up insert, improves error handling and
reporting, and adds post-insert hook. I intend to add documentation and
more tests, but the code is ready for review. Also, at least some of the
cleanups and fixes in the beginning of the series could go in without
additional tests or documentation.

BR,
Jani.


Jani Nikula (11):
  lib: actually return failures from
notmuch_message_tags_to_maildir_flags
  cli/insert: rename check_folder_name to is_valid_folder_name
  cli/insert: move add_file_to_database to a better place
  cli/insert: rename file copy function
  cli/insert: clean up sync_dir
  cli/insert: use a single recursive mkdir function
  cli/insert: abstract temporary filename generation
  cli/insert: rehash file writing functions
  cli/insert: add fail path to add_file_to_database
  cli/insert: require succesful message indexing for success status
  cli/insert: add post-insert hook

 lib/message.cc  |2 +-
 notmuch-insert.c|  462 +--
 test/T070-insert.sh |2 +-
 3 files changed, 262 insertions(+), 204 deletions(-)

-- 
1.7.2.5

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 03/11] cli/insert: move add_file_to_database to a better place

2014-09-22 Thread Jani Nikula
Move add_file_to_database around to keep the filesystem related
functions grouped together, improving readability. No functional
changes.
---
 notmuch-insert.c |   92 +++---
 1 files changed, 46 insertions(+), 46 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 770275b..ccb091a 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -294,52 +294,6 @@ copy_stdin (int fdin, int fdout)
 return (!interrupted  !empty);
 }
 
-/* Add the specified message file to the notmuch database, applying tags.
- * The file is renamed to encode notmuch tags as maildir flags. */
-static void
-add_file_to_database (notmuch_database_t *notmuch, const char *path,
- tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
-{
-notmuch_message_t *message;
-notmuch_status_t status;
-
-status = notmuch_database_add_message (notmuch, path, message);
-switch (status) {
-case NOTMUCH_STATUS_SUCCESS:
-case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
-   break;
-default:
-case NOTMUCH_STATUS_FILE_NOT_EMAIL:
-case NOTMUCH_STATUS_READ_ONLY_DATABASE:
-case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
-case NOTMUCH_STATUS_OUT_OF_MEMORY:
-case NOTMUCH_STATUS_FILE_ERROR:
-case NOTMUCH_STATUS_NULL_POINTER:
-case NOTMUCH_STATUS_TAG_TOO_LONG:
-case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
-case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
-case NOTMUCH_STATUS_LAST_STATUS:
-   fprintf (stderr, Error: failed to add `%s' to notmuch database: %s\n,
-path, notmuch_status_to_string (status));
-   return;
-}
-
-if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
-   /* Don't change tags of an existing message. */
-   if (synchronize_flags) {
-   status = notmuch_message_tags_to_maildir_flags (message);
-   if (status != NOTMUCH_STATUS_SUCCESS)
-   fprintf (stderr, Error: failed to sync tags to maildir 
flags\n);
-   }
-} else {
-   tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
-
-   tag_op_list_apply (message, tag_ops, flags);
-}
-
-notmuch_message_destroy (message);
-}
-
 static notmuch_bool_t
 write_message (void *ctx, int fdin, const char *dir, char **newpath)
 {
@@ -389,6 +343,52 @@ write_message (void *ctx, int fdin, const char *dir, char 
**newpath)
 return FALSE;
 }
 
+/* Add the specified message file to the notmuch database, applying tags.
+ * The file is renamed to encode notmuch tags as maildir flags. */
+static void
+add_file_to_database (notmuch_database_t *notmuch, const char *path,
+ tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
+{
+notmuch_message_t *message;
+notmuch_status_t status;
+
+status = notmuch_database_add_message (notmuch, path, message);
+switch (status) {
+case NOTMUCH_STATUS_SUCCESS:
+case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+   break;
+default:
+case NOTMUCH_STATUS_FILE_NOT_EMAIL:
+case NOTMUCH_STATUS_READ_ONLY_DATABASE:
+case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
+case NOTMUCH_STATUS_OUT_OF_MEMORY:
+case NOTMUCH_STATUS_FILE_ERROR:
+case NOTMUCH_STATUS_NULL_POINTER:
+case NOTMUCH_STATUS_TAG_TOO_LONG:
+case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
+case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
+case NOTMUCH_STATUS_LAST_STATUS:
+   fprintf (stderr, Error: failed to add `%s' to notmuch database: %s\n,
+path, notmuch_status_to_string (status));
+   return;
+}
+
+if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+   /* Don't change tags of an existing message. */
+   if (synchronize_flags) {
+   status = notmuch_message_tags_to_maildir_flags (message);
+   if (status != NOTMUCH_STATUS_SUCCESS)
+   fprintf (stderr, Error: failed to sync tags to maildir 
flags\n);
+   }
+} else {
+   tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
+
+   tag_op_list_apply (message, tag_ops, flags);
+}
+
+notmuch_message_destroy (message);
+}
+
 int
 notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
 {
-- 
1.7.2.5

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 01/11] lib: actually return failures from notmuch_message_tags_to_maildir_flags

2014-09-22 Thread Jani Nikula
The function takes great care to preserve the first error status it
encounters, yet fails to return that status to the caller. Fix it.
---
 lib/message.cc |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 68f7e68..7e82548 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1497,7 +1497,7 @@ notmuch_message_tags_to_maildir_flags (notmuch_message_t 
*message)
 talloc_free (to_set);
 talloc_free (to_clear);
 
-return NOTMUCH_STATUS_SUCCESS;
+return status;
 }
 
 notmuch_status_t
-- 
1.7.2.5

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 02/11] cli/insert: rename check_folder_name to is_valid_folder_name

2014-09-22 Thread Jani Nikula
An is something predicate conveys the meaning better. While at it,
improve the function documentation and error message. Besides the
error message change, no functional changes.
---
 notmuch-insert.c |   13 -
 1 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 8dfc8bb..770275b 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -83,10 +83,13 @@ sync_dir (const char *dir)
 return ret;
 }
 
-/* Check the specified folder name does not contain a directory
- * component .. to prevent writes outside of the Maildir hierarchy. */
+/*
+ * Check the specified folder name does not contain a directory
+ * component .. to prevent writes outside of the Maildir
+ * hierarchy. Return TRUE on valid folder name, FALSE otherwise.
+ */
 static notmuch_bool_t
-check_folder_name (const char *folder)
+is_valid_folder_name (const char *folder)
 {
 const char *p = folder;
 
@@ -449,8 +452,8 @@ notmuch_insert_command (notmuch_config_t *config, int argc, 
char *argv[])
 if (folder == NULL) {
maildir = db_path;
 } else {
-   if (! check_folder_name (folder)) {
-   fprintf (stderr, Error: bad folder name: %s\n, folder);
+   if (! is_valid_folder_name (folder)) {
+   fprintf (stderr, Error: invalid folder name: '%s'\n, folder);
return EXIT_FAILURE;
}
maildir = talloc_asprintf (config, %s/%s, db_path, folder);
-- 
1.7.2.5

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 06/11] cli/insert: use a single recursive mkdir function

2014-09-22 Thread Jani Nikula
Combine make_directory() and make_directory_and_parents() into a
single recursive mkdir_recursive() function. Clarify the code and
improve error handling. Improve error messages. Switch to using the
new function in maildir_create_folder(). Constify talloc context.
---
 notmuch-insert.c |  131 +++---
 1 files changed, 55 insertions(+), 76 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 7375c54..cdeeb41 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -104,96 +104,78 @@ is_valid_folder_name (const char *folder)
 }
 }
 
-/* Make the given directory, succeeding if it already exists. */
+/*
+ * Make the given directory and its parents as necessary, using the
+ * given mode. Return TRUE on success, FALSE otherwise. Partial
+ * results are not cleaned up on errors.
+ */
 static notmuch_bool_t
-make_directory (char *path, int mode)
+mkdir_recursive (const void *ctx, const char *path, int mode)
 {
-notmuch_bool_t ret;
-char *slash;
+struct stat st;
+int r;
+char *parent = NULL, *slash;
 
-if (mkdir (path, mode) != 0)
-   return (errno == EEXIST);
+/* First check the common case: directory already exists. */
+r = stat (path, st);
+if (r == 0) {
+if (! S_ISDIR (st.st_mode)) {
+   fprintf (stderr, Error: '%s' is not a directory: %s\n,
+path, strerror (EEXIST));
+   return FALSE;
+   }
 
-/* Sync the parent directory for durability. */
-ret = TRUE;
-slash = strrchr (path, '/');
-if (slash) {
-   *slash = '\0';
-   ret = sync_dir (path);
-   *slash = '/';
+   return TRUE;
+} else if (errno != ENOENT) {
+   fprintf (stderr, Error: stat '%s': %s\n, path, strerror (errno));
+   return FALSE;
 }
-return ret;
-}
-
-/* Make the given directory including its parent directories as necessary.
- * Return TRUE on success, FALSE on error. */
-static notmuch_bool_t
-make_directory_and_parents (char *path, int mode)
-{
-struct stat st;
-char *start;
-char *end;
-notmuch_bool_t ret;
 
-/* First check the common case: directory already exists. */
-if (stat (path, st) == 0)
-   return S_ISDIR (st.st_mode) ? TRUE : FALSE;
-
-for (start = path; *start != '\0'; start = end + 1) {
-   /* start points to the first unprocessed character.
-* Find the next slash from start onwards. */
-   end = strchr (start, '/');
-
-   /* If there are no more slashes then all the parent directories
-* have been made.  Now attempt to make the whole path. */
-   if (end == NULL)
-   return make_directory (path, mode);
-
-   /* Make the path up to the next slash, unless the current
-* directory component is actually empty. */
-   if (end  start) {
-   *end = '\0';
-   ret = make_directory (path, mode);
-   *end = '/';
-   if (! ret)
-   return FALSE;
+/* mkdir parents, if any */
+slash = strrchr (path, '/');
+if (slash  slash != path) {
+   parent = talloc_strndup (ctx, path, slash - path);
+   if (! parent) {
+   fprintf (stderr, Error: %s\n, strerror (ENOMEM));
+   return FALSE;
}
+
+   if (! mkdir_recursive (ctx, parent, mode))
+   return FALSE;
 }
 
-return TRUE;
+if (mkdir (path, mode)) {
+   fprintf (stderr, Error: mkdir '%s': %s\n, path, strerror (errno));
+   return FALSE;
+}
+
+return parent ? sync_dir (parent) : TRUE;
 }
 
-/* Create the given maildir folder, i.e. dir and its subdirectories
- * 'cur', 'new', 'tmp'. */
+/*
+ * Create the given maildir folder, i.e. maildir and its
+ * subdirectories cur/new/tmp. Return TRUE on success, FALSE
+ * otherwise. Partial results are not cleaned up on errors.
+ */
 static notmuch_bool_t
-maildir_create_folder (void *ctx, const char *dir)
+maildir_create_folder (const void *ctx, const char *maildir)
 {
+const char *subdirs[] = { cur, new, tmp };
 const int mode = 0700;
 char *subdir;
-char *tail;
-
-/* Create 'cur' directory, including parent directories. */
-subdir = talloc_asprintf (ctx, %s/cur, dir);
-if (! subdir) {
-   fprintf (stderr, Out of memory.\n);
-   return FALSE;
-}
-if (! make_directory_and_parents (subdir, mode))
-   return FALSE;
-
-tail = subdir + strlen (subdir) - 3;
+unsigned int i;
 
-/* Create 'new' directory. */
-strcpy (tail, new);
-if (! make_directory (subdir, mode))
-   return FALSE;
+for (i = 0; i  ARRAY_SIZE (subdirs); i++) {
+   subdir = talloc_asprintf (ctx, %s/%s, maildir, subdirs[i]);
+   if (! subdir) {
+   fprintf (stderr, Error: %s\n, strerror (ENOMEM));
+   return FALSE;
+   }
 
-/* Create 'tmp' directory. */
-strcpy (tail, tmp);
-if (! make_directory (subdir, mode))
-   return FALSE;
+   if (! mkdir_recursive (ctx, subdir, 

[PATCH 04/11] cli/insert: rename file copy function

2014-09-22 Thread Jani Nikula
The copying has nothing to do with stdin, so call it copy_fd
instead. While at it, improve documentation and reverse the
parameters, as destination is traditionally the first parameter.
---
 notmuch-insert.c |   11 ++-
 1 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index ccb091a..5d47806 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -251,11 +251,12 @@ maildir_open_tmp_file (void *ctx, const char *dir,
 return fd;
 }
 
-/* Copy the contents of standard input (fdin) into fdout.
- * Returns TRUE if a non-empty file was written successfully.
- * Otherwise, return FALSE. */
+/*
+ * Copy fdin to fdout, return TRUE on success, and FALSE on errors and
+ * empty input.
+ */
 static notmuch_bool_t
-copy_stdin (int fdin, int fdout)
+copy_fd (int fdout, int fdin)
 {
 notmuch_bool_t empty = TRUE;
 
@@ -308,7 +309,7 @@ write_message (void *ctx, int fdin, const char *dir, char 
**newpath)
 
 cleanup_path = tmppath;
 
-if (! copy_stdin (fdin, fdout))
+if (! copy_fd (fdout, fdin))
goto FAIL;
 
 if (fsync (fdout) != 0) {
-- 
1.7.2.5

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 05/11] cli/insert: clean up sync_dir

2014-09-22 Thread Jani Nikula
Clarify the code slightly, improve error messages. Apart from the
error message changes, no functional changes.
---
 notmuch-insert.c |   17 +
 1 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 5d47806..7375c54 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -67,20 +67,21 @@ safe_gethostname (char *hostname, size_t len)
 static notmuch_bool_t
 sync_dir (const char *dir)
 {
-notmuch_bool_t ret;
-int fd;
+int fd, r;
 
 fd = open (dir, O_RDONLY);
 if (fd == -1) {
-   fprintf (stderr, Error: open() dir failed: %s\n, strerror (errno));
+   fprintf (stderr, Error: open %s: %s\n, dir, strerror (errno));
return FALSE;
 }
-ret = (fsync (fd) == 0);
-if (! ret) {
-   fprintf (stderr, Error: fsync() dir failed: %s\n, strerror (errno));
-}
+
+r = fsync (fd);
+if (r)
+   fprintf (stderr, Error: fsync %s: %s\n, dir, strerror (errno));
+
 close (fd);
-return ret;
+
+return r == 0;
 }
 
 /*
-- 
1.7.2.5

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 07/11] cli/insert: abstract temporary filename generation

2014-09-22 Thread Jani Nikula
This will clean up the usage. There's the slight functional change of
potentially ending up doing extra gethostname and getpid calls, but
this is neglible.
---
 notmuch-insert.c |   39 +++
 1 files changed, 27 insertions(+), 12 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index cdeeb41..a1d564c 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -179,6 +179,31 @@ maildir_create_folder (const void *ctx, const char 
*maildir)
 return TRUE;
 }
 
+/*
+ * Generate a temporary file basename, no path, do not create an
+ * actual file. Return the basename, or NULL on errors.
+ */
+static char *
+tempfilename (const void *ctx)
+{
+char *filename;
+char hostname[256];
+struct timeval tv;
+pid_t pid;
+
+/* We follow the Dovecot file name generation algorithm. */
+pid = getpid ();
+safe_gethostname (hostname, sizeof (hostname));
+gettimeofday (tv, NULL);
+
+filename = talloc_asprintf (ctx, %ld.M%ldP%d.%s,
+   tv.tv_sec, tv.tv_usec, pid, hostname);
+if (! filename)
+   fprintf (stderr, Error: %s\n, strerror (ENOMEM));
+
+return filename;
+}
+
 /* Open a unique file in the 'tmp' sub-directory of dir.
  * Returns the file descriptor on success, or -1 on failure.
  * On success, file paths for the message in the 'tmp' and 'new'
@@ -188,23 +213,13 @@ static int
 maildir_open_tmp_file (void *ctx, const char *dir,
   char **tmppath, char **newpath, char **newdir)
 {
-pid_t pid;
-char hostname[256];
-struct timeval tv;
 char *filename;
 int fd = -1;
 
-/* We follow the Dovecot file name generation algorithm. */
-pid = getpid ();
-safe_gethostname (hostname, sizeof (hostname));
 do {
-   gettimeofday (tv, NULL);
-   filename = talloc_asprintf (ctx, %ld.M%ldP%d.%s,
-   tv.tv_sec, tv.tv_usec, pid, hostname);
-   if (! filename) {
-   fprintf (stderr, Out of memory\n);
+   filename = tempfilename (ctx);
+   if (! filename)
return -1;
-   }
 
*tmppath = talloc_asprintf (ctx, %s/tmp/%s, dir, filename);
if (! *tmppath) {
-- 
1.7.2.5

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 09/11] cli/insert: add fail path to add_file_to_database

2014-09-22 Thread Jani Nikula
Handle failures gracefully in add_file_to_database, renamed simply
add_file while at it. Add keep option to not remove the message from
database if tagging or tag syncing to maildir flags fails. Expand the
function documentation to cover the changes.
---
 notmuch-insert.c |   89 -
 1 files changed, 54 insertions(+), 35 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 5ef6e66..80f52d4 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -364,50 +364,70 @@ FAIL:
 return NULL;
 }
 
-/* Add the specified message file to the notmuch database, applying tags.
- * The file is renamed to encode notmuch tags as maildir flags. */
-static void
-add_file_to_database (notmuch_database_t *notmuch, const char *path,
- tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
+/*
+ * Add the specified message file to the notmuch database, applying
+ * tags in tag_ops. If synchronize_flags is TRUE, the tags are
+ * synchronized to maildir flags (which may result in message file
+ * rename).
+ *
+ * Return NOTMUCH_STATUS_SUCCESS on success, errors otherwise. If keep
+ * is TRUE, errors in tag changes and flag syncing are ignored and
+ * success status is returned; otherwise such errors cause the message
+ * to be removed from the database. Failure to add the message to the
+ * database results in error status regardless of keep.
+ */
+static notmuch_status_t
+add_file (notmuch_database_t *notmuch, const char *path, tag_op_list_t 
*tag_ops,
+ notmuch_bool_t synchronize_flags, notmuch_bool_t keep)
 {
 notmuch_message_t *message;
 notmuch_status_t status;
 
 status = notmuch_database_add_message (notmuch, path, message);
-switch (status) {
-case NOTMUCH_STATUS_SUCCESS:
-case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
-   break;
-default:
-case NOTMUCH_STATUS_FILE_NOT_EMAIL:
-case NOTMUCH_STATUS_READ_ONLY_DATABASE:
-case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
-case NOTMUCH_STATUS_OUT_OF_MEMORY:
-case NOTMUCH_STATUS_FILE_ERROR:
-case NOTMUCH_STATUS_NULL_POINTER:
-case NOTMUCH_STATUS_TAG_TOO_LONG:
-case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
-case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
-case NOTMUCH_STATUS_LAST_STATUS:
-   fprintf (stderr, Error: failed to add `%s' to notmuch database: %s\n,
-path, notmuch_status_to_string (status));
-   return;
-}
-
-if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
-   /* Don't change tags of an existing message. */
-   if (synchronize_flags) {
-   status = notmuch_message_tags_to_maildir_flags (message);
-   if (status != NOTMUCH_STATUS_SUCCESS)
-   fprintf (stderr, Error: failed to sync tags to maildir 
flags\n);
+if (status == NOTMUCH_STATUS_SUCCESS) {
+   status = tag_op_list_apply (message, tag_ops, 0);
+   if (status) {
+   fprintf (stderr, %s: failed to apply tags to file '%s': %s\n,
+keep ? Warning : Error,
+path, notmuch_status_to_string (status));
+   goto DONE;
}
+} else if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+   status = NOTMUCH_STATUS_SUCCESS;
+} else if (status == NOTMUCH_STATUS_FILE_NOT_EMAIL) {
+   fprintf (stderr, Error: delivery of non-mail file: '%s'\n, path);
+   return status;
 } else {
-   tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
+   fprintf (stderr, Error: failed to add '%s' to notmuch database: %s\n,
+path, notmuch_status_to_string (status));
+   return status;
+}
 
-   tag_op_list_apply (message, tag_ops, flags);
+if (synchronize_flags) {
+   status = notmuch_message_tags_to_maildir_flags (message);
+   if (status != NOTMUCH_STATUS_SUCCESS)
+   fprintf (stderr, %s: failed to sync tags to maildir flags for 
'%s': %s\n,
+keep ? Warning : Error,
+path, notmuch_status_to_string (status));
+
+   /*
+* Note: Unfortunately a failed maildir flag sync might
+* already have renamed the file, in which case the cleanup
+* path will fail.
+*/
 }
 
+DONE:
 notmuch_message_destroy (message);
+
+if (status) {
+   if (keep)
+   status = NOTMUCH_STATUS_SUCCESS;
+   else
+   notmuch_database_remove_message (notmuch, path);
+}
+
+return status;
 }
 
 int
@@ -508,8 +528,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc, 
char *argv[])
 /* Add the message to the index.
  * Even if adding the message to the notmuch database fails,
  * the message is on disk and we consider the delivery completed. */
-add_file_to_database (notmuch, newpath, tag_ops,
-   synchronize_flags);
+add_file (notmuch, newpath, tag_ops, synchronize_flags, TRUE);
 
 notmuch_database_destroy (notmuch);
 

[PATCH 10/11] cli/insert: require succesful message indexing for success status

2014-09-22 Thread Jani Nikula
Add --keep option to keep any remaining stuff in index or file. We
could distinguish between failures to index and failures to apply tags
or maildir sync, but for simplicity just have one.
---
 notmuch-insert.c|   20 +++-
 test/T070-insert.sh |2 +-
 2 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 80f52d4..f27b9cb 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -433,6 +433,7 @@ DONE:
 int
 notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
 {
+notmuch_status_t status;
 notmuch_database_t *notmuch;
 struct sigaction action;
 const char *db_path;
@@ -442,6 +443,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc, 
char *argv[])
 char *query_string = NULL;
 const char *folder = NULL;
 notmuch_bool_t create_folder = FALSE;
+notmuch_bool_t keep = FALSE;
 notmuch_bool_t synchronize_flags;
 const char *maildir;
 char *newpath;
@@ -451,6 +453,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc, 
char *argv[])
 notmuch_opt_desc_t options[] = {
{ NOTMUCH_OPT_STRING, folder, folder, 0, 0 },
{ NOTMUCH_OPT_BOOLEAN, create_folder, create-folder, 0, 0 },
+   { NOTMUCH_OPT_BOOLEAN, keep, keep, 0, 0 },
{ NOTMUCH_OPT_END, 0, 0, 0, 0 }
 };
 
@@ -525,11 +528,18 @@ notmuch_insert_command (notmuch_config_t *config, int 
argc, char *argv[])
return EXIT_FAILURE;
 }
 
-/* Add the message to the index.
- * Even if adding the message to the notmuch database fails,
- * the message is on disk and we consider the delivery completed. */
-add_file (notmuch, newpath, tag_ops, synchronize_flags, TRUE);
+/* Index the message. */
+status = add_file (notmuch, newpath, tag_ops, synchronize_flags, keep);
+if (status) {
+   if (keep) {
+   status = NOTMUCH_STATUS_SUCCESS;
+   } else {
+   /* If maildir flag sync failed, this might fail. */
+   unlink (newpath);
+   }
+}
 
 notmuch_database_destroy (notmuch);
-return EXIT_SUCCESS;
+
+return status ? EXIT_FAILURE : EXIT_SUCCESS;
 }
diff --git a/test/T070-insert.sh b/test/T070-insert.sh
index ea9db07..aacc643 100755
--- a/test/T070-insert.sh
+++ b/test/T070-insert.sh
@@ -23,7 +23,7 @@ test_expect_code 1 Insert zero-length file \
 
 # This test is a proxy for other errors that may occur while trying to
 # add a message to the notmuch database, e.g. database locked.
-test_expect_code 0 Insert non-message \
+test_expect_code 1 Insert non-message \
 echo bad_message | notmuch insert
 
 test_begin_subtest Database empty so far
-- 
1.7.2.5

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] Add configurable changed tag to messages that have been changed on disk

2014-09-22 Thread Gaute Hope

Excerpts from Gaute Hope's message of August 6, 2014 10:29:

Austin Clements amdra...@mit.edu wrote on Fri, 01 Aug 2014 14:55:05 -0400:

I have a prototype implementation of message modification times on my
lastmod-v1 branch at

  https://github.com/aclements/notmuch/tree/lastmod-v1

It builds on my database features series that's currently awaiting
review [1].

The series uses a monotonic revision number, rather than wall-clock
time, for reasons related to Xapian's concurrent control and detailed
in the main commit's commit message.  The implementation isn't quite
useful from the CLI yet because I haven't added any way to query the
database's current revision number.  (I'm still thinking about how I
want to do this, since search/show don't have a good way to deliver
additional information right now.  I might just add the last
modification for each individual message/max of all messages in a
thread, similar to what Thomas Jost's patch did long ago.)

[1] id:1406859003-11561-1-git-send-email-amdra...@mit.edu



this should allow me to do what I wish to accomplish. The message
deletion is still a problem though, I can see two options at the moment:


Hi list,

While exploring the possibility of syncing maildir/X-keywords with tags
I had some thoughts about lastmod and message modification:

As briefly discussed on #notmuch, I noticed that it seems that 'notmuch
new' does not detect that a message source has been changed, unless the
file is also re-named.

This means that for instance if the X-Keywords fields have been updated
in a message (from GMail with offlineimap, synclabels = yes) the lastmod
field will remain unchanged, and a source modification will be
undetectable to a client program using this value.

Would it not make sense that if a message has a more recent mtime than
at index time it is re-indexed?

Also, for the lastmod branch I would wish for a notmuch_message_touch()
method where the lastmod time is updated to the last. As well as a
notmuch_database_reindex_message () - possibly defined/documented
behaviour for notmuch_database_add_message () when the filename is
already added (in which case I would expect notmuch to re-index the
message).

Doing notmuch_database_remove_message followed by _add_message could
risk deleting the entry if this file is the only on-disk-representation.

Cheers, Gaute

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: Do path: searches handle spaces correctly?

2014-09-22 Thread Austin Clements
Quoth Keith Amidon on Sep 22 at  7:42 am:
 On Mon, 2014-09-22 at 11:20 +0200, David Bremner wrote:
  Keith Amidon ke...@awakenetworks.com writes:
  
   notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items'
  
   I don't get any results, but it seems like the two results above should
   be shown, right?  Am I doing something wrong here?  If it looks like I'm
   doing it correctly, what can I do to help troubleshoot the issue?
  
  Note that path:, unlike folder:, does not add the maildir subdirs. Dunno
  if that's the only issue you are having, but I guess it's one.
 
 Darn it!  I made a mistake in my original email.  In the test I was
 doing I actually had:
 
 notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items/**'
 
 which I believe should have picked up all the subdirectory paths.  I
 just retested to make sure that it still didn't work, and it doesn't.
 
 Am I still missing something?   Thanks for the help, Keith

I assume you're doing this from the command line?  Does the following
work?

notmuch search --output=files 'path:dir/INBOX/INBOX/Sent Items/**'

Shell quoting and Xapian quoting interact in often confusing ways.  In
your original command line, the single quotes suppress shell argument
splitting, but never make it to notmuch, so notmuch sees two search
terms path:dir/INBOX/INBOX/Sent and Items/**.  In the command line
I suggested, the single quotes play the same role, but for the entire
query.  Because of the quoting, notmuch *does* see the inner double
quotes, which makes the path a single term (note that Xapian only
accepts double quotes, not single quotes).  In general, it's good to
enclose the entire command line query in single quotes so shell
parsing doesn't get in the way of query parsing.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] Add configurable changed tag to messages that have been changed on disk

2014-09-22 Thread Tomi Ollila
On Mon, Sep 22 2014, Gaute Hope e...@gaute.vetsj.com wrote:

 Excerpts from Gaute Hope's message of August 6, 2014 10:29:
 Austin Clements amdra...@mit.edu wrote on Fri, 01 Aug 2014 14:55:05 -0400:
 I have a prototype implementation of message modification times on my
 lastmod-v1 branch at
 
   https://github.com/aclements/notmuch/tree/lastmod-v1
 
 It builds on my database features series that's currently awaiting
 review [1].
 
 The series uses a monotonic revision number, rather than wall-clock
 time, for reasons related to Xapian's concurrent control and detailed
 in the main commit's commit message.  The implementation isn't quite
 useful from the CLI yet because I haven't added any way to query the
 database's current revision number.  (I'm still thinking about how I
 want to do this, since search/show don't have a good way to deliver
 additional information right now.  I might just add the last
 modification for each individual message/max of all messages in a
 thread, similar to what Thomas Jost's patch did long ago.)
 
 [1] id:1406859003-11561-1-git-send-email-amdra...@mit.edu
  
 this should allow me to do what I wish to accomplish. The message
 deletion is still a problem though, I can see two options at the moment:

 Hi list,

 While exploring the possibility of syncing maildir/X-keywords with tags
 I had some thoughts about lastmod and message modification:

 As briefly discussed on #notmuch, I noticed that it seems that 'notmuch
 new' does not detect that a message source has been changed, unless the
 file is also re-named.

 This means that for instance if the X-Keywords fields have been updated
 in a message (from GMail with offlineimap, synclabels = yes) the lastmod
 field will remain unchanged, and a source modification will be
 undetectable to a client program using this value.

 Would it not make sense that if a message has a more recent mtime than
 at index time it is re-indexed?

That would require notmuch to scan the contents of a directory for changed
mtimes of the files -- now notmuch skips looking for the files unless
directory mtime has changed. Directory mtime changes when files are
added/deleted/renamed (as that is what directory needs to know) -- file
mtime change are stored in file information and therefore change there does
not probagate to parent directory (and, if such happened, to it's parent
and so on...)

That would mean the scanning would be slower than it is now.

Tomi


 Also, for the lastmod branch I would wish for a notmuch_message_touch()
 method where the lastmod time is updated to the last. As well as a
 notmuch_database_reindex_message () - possibly defined/documented
 behaviour for notmuch_database_add_message () when the filename is
 already added (in which case I would expect notmuch to re-index the
 message).

 Doing notmuch_database_remove_message followed by _add_message could
 risk deleting the entry if this file is the only on-disk-representation.

 Cheers, Gaute
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] Add configurable changed tag to messages that have been changed on disk

2014-09-22 Thread Austin Clements
On Mon, 22 Sep 2014, Gaute Hope e...@gaute.vetsj.com wrote:
 Excerpts from Gaute Hope's message of August 6, 2014 10:29:
 Austin Clements amdra...@mit.edu wrote on Fri, 01 Aug 2014 14:55:05 -0400:
 I have a prototype implementation of message modification times on my
 lastmod-v1 branch at
 
   https://github.com/aclements/notmuch/tree/lastmod-v1
 
 It builds on my database features series that's currently awaiting
 review [1].
 
 The series uses a monotonic revision number, rather than wall-clock
 time, for reasons related to Xapian's concurrent control and detailed
 in the main commit's commit message.  The implementation isn't quite
 useful from the CLI yet because I haven't added any way to query the
 database's current revision number.  (I'm still thinking about how I
 want to do this, since search/show don't have a good way to deliver
 additional information right now.  I might just add the last
 modification for each individual message/max of all messages in a
 thread, similar to what Thomas Jost's patch did long ago.)
 
 [1] id:1406859003-11561-1-git-send-email-amdra...@mit.edu
  
 this should allow me to do what I wish to accomplish. The message
 deletion is still a problem though, I can see two options at the moment:

 Hi list,

 While exploring the possibility of syncing maildir/X-keywords with tags
 I had some thoughts about lastmod and message modification:

 As briefly discussed on #notmuch, I noticed that it seems that 'notmuch
 new' does not detect that a message source has been changed, unless the
 file is also re-named.

 This means that for instance if the X-Keywords fields have been updated
 in a message (from GMail with offlineimap, synclabels = yes) the lastmod
 field will remain unchanged, and a source modification will be
 undetectable to a client program using this value.

 Would it not make sense that if a message has a more recent mtime than
 at index time it is re-indexed?

This has the potential to make notmuch new substantially more expensive.
Currently, if there are no changes, it only has to stat each directory
in your maildir (in fact, some restructuring of new would let us
eliminate almost all database access during a no-op notmuch new as
well).  Checking for changes to individual messages would require
stat'ing every single message file as well as accessing the database to
check the paths and mtimes of every message, increasing the number of
stat calls and disk accesses by several orders of magnitude.

It may be that this is fast enough that it's okay, but it would be good
to gather some evidence first.  That includes hot and cold caches, and
maildir over NFS.

With respect to X-Keywords specifically, note that it's a fairly basic
design decision that notmuch never modifies message files.  This gives
us strong robustness guarantees we would be loathe to part with.

It has puzzled me ever since offlineimap added X-Keywords why they
didn't just translate these keywords into folders and create hard links
of message files.  Anything could interact smoothly with that.

 Also, for the lastmod branch I would wish for a notmuch_message_touch()
 method where the lastmod time is updated to the last. As well as a
 notmuch_database_reindex_message () - possibly defined/documented
 behaviour for notmuch_database_add_message () when the filename is
 already added (in which case I would expect notmuch to re-index the
 message).

What's the use case for these?

 Doing notmuch_database_remove_message followed by _add_message could
 risk deleting the entry if this file is the only on-disk-representation.

 Cheers, Gaute
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] lib: Simplify close and codify aborting atomic section

2014-09-22 Thread Austin Clements
In Xapian, closing a database implicitly aborts any outstanding
transaction and commits changes.  For historical reasons,
notmuch_database_close had grown to almost, but not quite duplicate
this behavior.  Before closing the database, it would explicitly (and
unnecessarily) commit it.  However, if there was an outstanding
transaction (ie atomic section), commit would throw a Xapian
exception, which notmuch_database_close would unnecessarily print to
stderr, even though notmuch_database_close would ultimately abort the
transaction anyway when it called close.

This patch simplifies notmuch_database_close to just call
Database::close.  This works for both read-only and read/write
databases, takes care of committing changes, unifies the exception
handling path, and codifies aborting outstanding transactions.  This
is currently the only way to abort an atomic section (and may remain
so, since it would be difficult to roll back things we may have cached
from rolled-back modifications).
---
 lib/database.cc | 23 +++
 lib/notmuch.h   |  5 +
 2 files changed, 12 insertions(+), 16 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index a3a7cd3..1f7ff2a 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -903,28 +903,19 @@ notmuch_database_close (notmuch_database_t *notmuch)
 {
 notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
 
-try {
-   if (notmuch-xapian_db != NULL 
-   notmuch-mode == NOTMUCH_DATABASE_MODE_READ_WRITE)
-   (static_cast Xapian::WritableDatabase * 
(notmuch-xapian_db))-flush ();
-} catch (const Xapian::Error error) {
-   status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
-   if (! notmuch-exception_reported) {
-   fprintf (stderr, Error: A Xapian exception occurred flushing 
database: %s\n,
-error.get_msg().c_str());
-   }
-}
-
 /* Many Xapian objects (and thus notmuch objects) hold references to
  * the database, so merely deleting the database may not suffice to
- * close it.  Thus, we explicitly close it here. */
+ * close it.  Thus, we explicitly close it here.  This will
+ * implicitly abort any outstanding transaction and commit changes. */
 if (notmuch-xapian_db != NULL) {
try {
notmuch-xapian_db-close();
} catch (const Xapian::Error error) {
-   /* don't clobber previous error status */
-   if (status == NOTMUCH_STATUS_SUCCESS)
-   status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+   status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+   if (! notmuch-exception_reported) {
+   fprintf (stderr, Error: A Xapian exception occurred closing 
database: %s\n,
+error.get_msg().c_str());
+   }
}
 }
 
diff --git a/lib/notmuch.h b/lib/notmuch.h
index fe2340b..5c40c67 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -292,6 +292,11 @@ notmuch_database_open (const char *path,
  * notmuch_database_close can be called multiple times.  Later calls
  * have no effect.
  *
+ * If the caller is currently in an atomic section (there was a
+ * notmuch_database_begin_atomic without a matching
+ * notmuch_database_end_atomic), this will abort the atomic section,
+ * discarding any modifications made in the atomic section.
+ *
  * Return value:
  *
  * NOTMUCH_STATUS_SUCCESS: Successfully closed the database.
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] lib: Simplify close and codify aborting atomic section

2014-09-22 Thread W. Trevor King
On Mon, Sep 22, 2014 at 11:43:35AM -0400, Austin Clements wrote:
 This patch simplifies notmuch_database_close to just call
 Database::close.  This works for both read-only and read/write
 databases, takes care of committing changes, unifies the exception
 handling path, and codifies aborting outstanding transactions.

If we're dropping the flush call here, where will it be triggered
instead?  We'll need to flush/commit our changes to the database at
some point before closing.  Do clients now need an explicit
flush/commit command (explicit, client-initiated flushes sound like a
good idea to me).

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy


signature.asc
Description: OpenPGP digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [announce] nmhive v0.1.0, a bookmarklet/server for nmbug tags

2014-09-22 Thread W. Trevor King
On Mon, Sep 22, 2014 at 10:19:35AM -0700, W. Trevor King wrote:
 I like nmbug's distributed tag maintenance, but not everyone has
 notmuch/nmbug installed locally (yet ;).  However, everyone that I
 know does have a browser and a mail client.  They can submit
 messages with their mail client already, but we've been missing a
 way for them to help tag messages.

Ah, and the other piece to this workflow is the existing nmbug-status,
which collects the results of canned searches so folks without a local
notmuch can use the tags [1].  Folks using a local nmhive will
probably want to run their own status-genertion via a post-commit hook
in their nmhive repository.  Then their users will have their
search-results updated after each web-initiated change.  If you also
wanted them to see updates from changes to tethera's nmbug repository,
you'd probably also want a cron job that tried to fetch and merge
tethera's changes with the nmhive changes:

  -o---o---oo  tethera/master
\\
 \o  nmhive/status  (auto-generated merge for nmbug-status)
  \  /
   o---oo   nmhive/master (with web-initiated changes)

You'd want to resolve conflicts somehow, but any resolution strategy
is probably fine, since it's unlikely that we get conflicts very
often.

Cheers,
Trevor

[1]: http://nmbug.tethera.net/status/

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy


signature.asc
Description: OpenPGP digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] lib: Simplify close and codify aborting atomic section

2014-09-22 Thread Austin Clements
Quoth W. Trevor King on Sep 22 at  9:59 am:
 On Mon, Sep 22, 2014 at 11:43:35AM -0400, Austin Clements wrote:
  This patch simplifies notmuch_database_close to just call
  Database::close.  This works for both read-only and read/write
  databases, takes care of committing changes, unifies the exception
  handling path, and codifies aborting outstanding transactions.
 
 If we're dropping the flush call here, where will it be triggered
 instead?  We'll need to flush/commit our changes to the database at
 some point before closing.  Do clients now need an explicit
 flush/commit command (explicit, client-initiated flushes sound like a
 good idea to me).

The call to Database::close implicitly flushes/commits, as mentioned
in the comment in the patch, so there's no need for any new APIs or
client changes.  The call to Database::flush in notmuch_database_close
was entirely redundant with the call to Database::close.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] emacs: jump: fix compile warning on emacs 23

2014-09-22 Thread Austin Clements
LGTM.  I'm a little surprised this is necessary, but whatever.

I think the eval-and-compile has to be top-level; it's certainly not
wrong for it to be top-level.  (I like the comment in the
eval-and-compile implementation: ;; Remember, it's magic.)

On Thu, 04 Sep 2014, Mark Walters markwalters1...@gmail.com wrote:
 notmuch-jump uses window-body-width which is not defined in emacs
 23. To get around this it does

 (unless (fboundp 'window-body-width)
   ;; Compatibility for Emacs pre-24
   (defalias 'window-body-width 'window-width))

 This makes sure window-body-width is defined and all should be
 well. But it seems that the byte compiler does not realise that this
 guarantees that window-body-width will be defined and so, when
 compiling with emacs 23, it gives an error

 In end of data:
 notmuch-jump.el:172:1:Warning: the function `window-body-width' is not known 
 to be defined.

 Domo and I came to following on irc: wrap the (unless (fboundp ...))
 inside eval-and-compile which ensures that both the test and the
 defalias (if needed) happen at both compile and load time.  This fixes
 the warning.
 ---
 I think Domo and I were both not completely sure whether the
 eval-and-compile should be inside or outside the (unless fboundp ..)
 or not (ie should it wrap the unless fboundp or just the defalias) nor
 whether it should be eval-and-compile or eval-when-compile.

 We think this is the right version but it would be good to have confirmation.

 I tested notmuch jump inside emacs 23 and 24 with notmuch-emacs
 compiled with emacs 23 or 24 (ie all four combinations) and it seemed to work.

 Best wishes

 Mark



  emacs/notmuch-jump.el |7 ---
  1 file changed, 4 insertions(+), 3 deletions(-)

 diff --git a/emacs/notmuch-jump.el b/emacs/notmuch-jump.el
 index 5eb0949..2706b6c 100644
 --- a/emacs/notmuch-jump.el
 +++ b/emacs/notmuch-jump.el
 @@ -25,9 +25,10 @@
  (require 'notmuch-lib)
  (require 'notmuch-hello)
  
 -(unless (fboundp 'window-body-width)
 -  ;; Compatibility for Emacs pre-24
 -  (defalias 'window-body-width 'window-width))
 +(eval-and-compile
 +  (unless (fboundp 'window-body-width)
 +;; Compatibility for Emacs pre-24
 +(defalias 'window-body-width 'window-width)))
  
  ;;;###autoload
  (defun notmuch-jump-search ()
 -- 
 1.7.10.4

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] lib: Simplify close and codify aborting atomic section

2014-09-22 Thread W. Trevor King
On Mon, Sep 22, 2014 at 06:50:50PM +, Austin Clements wrote:
 Quoth W. Trevor King on Sep 22 at  9:59 am:
  On Mon, Sep 22, 2014 at 11:43:35AM -0400, Austin Clements wrote:
   This patch simplifies notmuch_database_close to just call
   Database::close.  This works for both read-only and read/write
   databases, takes care of committing changes, unifies the
   exception handling path, and codifies aborting outstanding
   transactions.
  
  If we're dropping the flush call here, where will it be triggered
  instead?  We'll need to flush/commit our changes to the database
  at some point before closing.  Do clients now need an explicit
  flush/commit command (explicit, client-initiated flushes sound
  like a good idea to me).
 
 The call to Database::close implicitly flushes/commits, as mentioned
 in the comment in the patch, so there's no need for any new APIs or
 client changes.  The call to Database::flush in
 notmuch_database_close was entirely redundant with the call to
 Database::close.

Ah, I thought the implicit flush/commit was just in our code.  Since
it's also in the underlying Xapian close, then this patch looks pretty
good to me.  I'd mention Xapian's explicit close in the notmuch.h
message.  Xapain's docs say [1]:

  For a WritableDatabase, if a transaction is active it will be
  aborted, while if no transaction is active commit() will be
  implicitly called.

Cheers,
Trevor

[1]: 
http://xapian.org/docs/apidoc/html/classXapian_1_1Database.html#a59f5f8b137723dcaaabdbdccbc0cf1eb

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy


signature.asc
Description: OpenPGP digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: Do path: searches handle spaces correctly?

2014-09-22 Thread Keith Amidon
On Mon, 2014-09-22 at 15:06 +, Austin Clements wrote:
 I assume you're doing this from the command line?  Does the following
 work?
 
 notmuch search --output=files 'path:dir/INBOX/INBOX/Sent Items/**'
 
 Shell quoting and Xapian quoting interact in often confusing ways.  In
 your original command line, the single quotes suppress shell argument
 splitting, but never make it to notmuch, so notmuch sees two search
 terms path:dir/INBOX/INBOX/Sent and Items/**.  In the command line
 I suggested, the single quotes play the same role, but for the entire
 query.  Because of the quoting, notmuch *does* see the inner double
 quotes, which makes the path a single term (note that Xapian only
 accepts double quotes, not single quotes).  In general, it's good to
 enclose the entire command line query in single quotes so shell
 parsing doesn't get in the way of query parsing.

Bingo.  That was it.  Thanks for clarifying how the quoting interacts. 

What do you think about adding a section to the notmuch-search-terms web
page that talks in more detail about the quoting interactions?  If there
were on obvious section about this in the man page it might have helped
me find the issue myself.  If this seems like a good idea I'll take a
crack at it later this week when I get time to research the Xapian
quoting in a bit more detail.

--- Keith

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Do path: searches handle spaces correctly?

2014-09-22 Thread David Bremner
Keith Amidon  writes:

>
> notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items'
>
> I don't get any results, but it seems like the two results above should
> be shown, right?  Am I doing something wrong here?  If it looks like I'm
> doing it correctly, what can I do to help troubleshoot the issue?

Note that path:, unlike folder:, does not add the maildir subdirs. Dunno
if that's the only issue you are having, but I guess it's one.

d


[PATCH 1/5] cli: Refactor option passing in the search command

2014-09-22 Thread Michal Sojka
Many functions that implement the search command need to access command
line options. Instead of passing each option in a separate variable, put
them in a structure and pass only this structure.

This will become handy in the following patches.
---
 notmuch-search.c | 122 ---
 1 file changed, 62 insertions(+), 60 deletions(-)

diff --git a/notmuch-search.c b/notmuch-search.c
index bc9be45..5ac2a26 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -30,6 +30,16 @@ typedef enum {
 OUTPUT_TAGS
 } output_t;

+typedef struct {
+sprinter_t *format;
+notmuch_query_t *query;
+notmuch_sort_t sort;
+output_t output;
+int offset;
+int limit;
+int dupe;
+} search_options_t;
+
 /* Return two stable query strings that identify exactly the matched
  * and unmatched messages currently in thread.  If there are no
  * matched or unmatched messages, the returned buffers will be
@@ -70,46 +80,42 @@ get_thread_query (notmuch_thread_t *thread,
 }

 static int
-do_search_threads (sprinter_t *format,
-  notmuch_query_t *query,
-  notmuch_sort_t sort,
-  output_t output,
-  int offset,
-  int limit)
+do_search_threads (search_options_t *o)
 {
 notmuch_thread_t *thread;
 notmuch_threads_t *threads;
 notmuch_tags_t *tags;
+sprinter_t *format = o->format;
 time_t date;
 int i;

-if (offset < 0) {
-   offset += notmuch_query_count_threads (query);
-   if (offset < 0)
-   offset = 0;
+if (o->offset < 0) {
+   o->offset += notmuch_query_count_threads (o->query);
+   if (o->offset < 0)
+   o->offset = 0;
 }

-threads = notmuch_query_search_threads (query);
+threads = notmuch_query_search_threads (o->query);
 if (threads == NULL)
return 1;

 format->begin_list (format);

 for (i = 0;
-notmuch_threads_valid (threads) && (limit < 0 || i < offset + limit);
+notmuch_threads_valid (threads) && (o->limit < 0 || i < o->offset + 
o->limit);
 notmuch_threads_move_to_next (threads), i++)
 {
thread = notmuch_threads_get (threads);

-   if (i < offset) {
+   if (i < o->offset) {
notmuch_thread_destroy (thread);
continue;
}

-   if (output == OUTPUT_THREADS) {
+   if (o->output == OUTPUT_THREADS) {
format->set_prefix (format, "thread");
format->string (format,
-   notmuch_thread_get_thread_id (thread));
+  notmuch_thread_get_thread_id (thread));
format->separator (format);
} else { /* output == OUTPUT_SUMMARY */
void *ctx_quote = talloc_new (thread);
@@ -123,7 +129,7 @@ do_search_threads (sprinter_t *format,

format->begin_map (format);

-   if (sort == NOTMUCH_SORT_OLDEST_FIRST)
+   if (o->sort == NOTMUCH_SORT_OLDEST_FIRST)
date = notmuch_thread_get_oldest_date (thread);
else
date = notmuch_thread_get_newest_date (thread);
@@ -215,40 +221,36 @@ do_search_threads (sprinter_t *format,
 }

 static int
-do_search_messages (sprinter_t *format,
-   notmuch_query_t *query,
-   output_t output,
-   int offset,
-   int limit,
-   int dupe)
+do_search_messages (search_options_t *o)
 {
 notmuch_message_t *message;
 notmuch_messages_t *messages;
 notmuch_filenames_t *filenames;
+sprinter_t *format = o->format;
 int i;

-if (offset < 0) {
-   offset += notmuch_query_count_messages (query);
-   if (offset < 0)
-   offset = 0;
+if (o->offset < 0) {
+   o->offset += notmuch_query_count_messages (o->query);
+   if (o->offset < 0)
+   o->offset = 0;
 }

-messages = notmuch_query_search_messages (query);
+messages = notmuch_query_search_messages (o->query);
 if (messages == NULL)
return 1;

 format->begin_list (format);

 for (i = 0;
-notmuch_messages_valid (messages) && (limit < 0 || i < offset + limit);
+notmuch_messages_valid (messages) && (o->limit < 0 || i < o->offset + 
o->limit);
 notmuch_messages_move_to_next (messages), i++)
 {
-   if (i < offset)
+   if (i < o->offset)
continue;

message = notmuch_messages_get (messages);

-   if (output == OUTPUT_FILES) {
+   if (o->output == OUTPUT_FILES) {
int j;
filenames = notmuch_message_get_filenames (message);

@@ -256,7 +258,7 @@ do_search_messages (sprinter_t *format,
 notmuch_filenames_valid (filenames);
 notmuch_filenames_move_to_next (filenames), j++)
{
-   if (dupe < 0 || dupe == j) {
+   if (o->dupe < 0 || o->dupe == j) {
format->string (format, 

[PATCH 5/5] cli: Add tests for 'search --output=addresses' and similar

2014-09-22 Thread Michal Sojka
---
 test/T090-search-output.sh | 59 +++
 test/T095-search-unique.sh | 63 ++
 2 files changed, 122 insertions(+)
 create mode 100755 test/T095-search-unique.sh

diff --git a/test/T090-search-output.sh b/test/T090-search-output.sh
index 947d572..ebc8c37 100755
--- a/test/T090-search-output.sh
+++ b/test/T090-search-output.sh
@@ -387,6 +387,65 @@ cat OUTPUT
+cat OUTPUT
+cat OUTPUT
+cat OUTPUT
+cat OUTPUT
+cat OUTPUT
+cat OUTPUT
+cat OUTPUT
+cat OUTPUT
+cat 

[PATCH 3/5] cli: Add support for parsing command line "flag" options

2014-09-22 Thread Michal Sojka
This allows having multiple flags as a value of a command line
option (e.g. --foo=bar,baz). The values of the given flags are OR'd
together.

This was inspired by a similar patch from Jani Nikula.
---
 command-line-arguments.c  | 40 
 command-line-arguments.h  |  1 +
 test/Makefile.local   |  2 +-
 test/T410-argument-parsing.sh |  3 ++-
 test/arg-test.c   |  8 
 5 files changed, 52 insertions(+), 2 deletions(-)

diff --git a/command-line-arguments.c b/command-line-arguments.c
index 844d6c3..50723e4 100644
--- a/command-line-arguments.c
+++ b/command-line-arguments.c
@@ -3,6 +3,7 @@
 #include 
 #include "error_util.h"
 #include "command-line-arguments.h"
+#include "string-util.h"

 /*
   Search the array of keywords for a given argument, assigning the
@@ -36,6 +37,43 @@ _process_keyword_arg (const notmuch_opt_desc_t *arg_desc, 
char next, const char
 return FALSE;
 }

+/*
+  An arguments composed of comma-separated flags is parsed and the
+  output variable is assigned the ORed flag values. Return FALSE in
+  case of error.
+*/
+static notmuch_bool_t
+_process_flags_arg (const notmuch_opt_desc_t *arg_desc, char next, const char 
*arg_str) {
+
+const notmuch_keyword_t *keyword;
+const char *flag = arg_str;
+size_t flen = 0;
+notmuch_bool_t match;
+
+if (next == '\0' || *arg_str == '\0') {
+   /* No flag given */
+   fprintf (stderr, "Option \"%s\" needs an flags argument.\n", 
arg_desc->name);
+   return FALSE;
+}
+
+while ((flag = strtok_len_c (flag + flen, ",", ))) {
+   for (keyword = arg_desc->keywords, match = FALSE; keyword->name; 
keyword++) {
+   if (strncmp (flag, keyword->name, flen) == 0 &&
+   flen == strlen (keyword->name)) {
+   *((int *)arg_desc->output_var) |= keyword->value;
+   match = TRUE;
+   break;
+   }
+   }
+   if (! match) {
+   fprintf (stderr, "Unknown flag argument \"%.*s\" for option 
\"%s\".\n", (int)flen, flag, arg_desc->name);
+   return FALSE;
+   }
+}
+return TRUE;
+}
+
+
 static notmuch_bool_t
 _process_boolean_arg (const notmuch_opt_desc_t *arg_desc, char next, const 
char *arg_str) {

@@ -153,6 +191,8 @@ parse_option (const char *arg,
switch (try->opt_type) {
case NOTMUCH_OPT_KEYWORD:
return _process_keyword_arg (try, next, value);
+   case NOTMUCH_OPT_FLAGS:
+   return _process_flags_arg (try, next, value);
case NOTMUCH_OPT_BOOLEAN:
return _process_boolean_arg (try, next, value);
case NOTMUCH_OPT_INT:
diff --git a/command-line-arguments.h b/command-line-arguments.h
index de1734a..192ce06 100644
--- a/command-line-arguments.h
+++ b/command-line-arguments.h
@@ -8,6 +8,7 @@ enum notmuch_opt_type {
 NOTMUCH_OPT_BOOLEAN,   /* --verbose  */
 NOTMUCH_OPT_INT,   /* --frob=8   */
 NOTMUCH_OPT_KEYWORD,   /* --format=raw|json|text */
+NOTMUCH_OPT_FLAGS, /* --flags=name,addr,casefold */
 NOTMUCH_OPT_STRING,/* --file=/tmp/gnarf.txt  */
 NOTMUCH_OPT_POSITION   /* notmuch dump pos_arg   */
 };
diff --git a/test/Makefile.local b/test/Makefile.local
index a2d58fc..efa3410 100644
--- a/test/Makefile.local
+++ b/test/Makefile.local
@@ -13,7 +13,7 @@ smtp_dummy_srcs = \
 smtp_dummy_modules = $(smtp_dummy_srcs:.c=.o)

 $(dir)/arg-test: $(dir)/arg-test.o command-line-arguments.o util/libutil.a
-   $(call quiet,CC) $^ -o $@
+   $(call quiet,CC) $^ -o $@ -ltalloc

 $(dir)/hex-xcode: $(dir)/hex-xcode.o command-line-arguments.o util/libutil.a
$(call quiet,CC) $^ $(TALLOC_LDFLAGS) -o $@
diff --git a/test/T410-argument-parsing.sh b/test/T410-argument-parsing.sh
index 94e9087..9c4cc84 100755
--- a/test/T410-argument-parsing.sh
+++ b/test/T410-argument-parsing.sh
@@ -3,9 +3,10 @@ test_description="argument parsing"
 . ./test-lib.sh

 test_begin_subtest "sanity check"
-$TEST_DIRECTORY/arg-test  pos1  --keyword=one --string=foo pos2 --int=7 > 
OUTPUT
+$TEST_DIRECTORY/arg-test  pos1  --keyword=one --flags=one,two --string=foo 
pos2 --int=7 > OUTPUT
 cat < EXPECTED
 keyword 1
+flags 3
 int 7
 string foo
 positional arg 1 pos1
diff --git a/test/arg-test.c b/test/arg-test.c
index 6c49eac..0d75fe7 100644
--- a/test/arg-test.c
+++ b/test/arg-test.c
@@ -7,6 +7,7 @@ int main(int argc, char **argv){
 int opt_index=1;

 int kw_val=0;
+int fl_val=0;
 int int_val=0;
 char *pos_arg1=NULL;
 char *pos_arg2=NULL;
@@ -17,6 +18,10 @@ int main(int argc, char **argv){
  (notmuch_keyword_t []){ { "one", 1 },
  { "two", 2 },
  { 0, 0 } } },
+   { NOTMUCH_OPT_FLAGS, _val, "flags", 'f',
+ (notmuch_keyword_t []){ { "one", 1 << 0},
+ { "two", 1 << 1 },
+

[PATCH 4/5] cli: Add configurable address deduplication for --output=addresses

2014-09-22 Thread Michal Sojka
The code here is an extended version of a path from Jani Nikula.
---
 completion/notmuch-completion.bash |   6 ++-
 completion/notmuch-completion.zsh  |   3 +-
 doc/man1/notmuch-search.rst|  33 
 notmuch-search.c   | 101 ++---
 4 files changed, 135 insertions(+), 8 deletions(-)

diff --git a/completion/notmuch-completion.bash 
b/completion/notmuch-completion.bash
index c37ddf5..8bc7874 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -305,12 +305,16 @@ _notmuch_search()
COMPREPLY=( $( compgen -W "true false flag all" -- "${cur}" ) )
return
;;
+   --unique)
+   COMPREPLY=( $( compgen -W "none addr addrfold name" -- "${cur}" ) )
+   return
+   ;;
 esac

 ! $split &&
 case "${cur}" in
-*)
-   local options="--format= --output= --sort= --offset= --limit= 
--exclude= --duplicate="
+   local options="--format= --output= --sort= --offset= --limit= 
--exclude= --duplicate= --unique="
compopt -o nospace
COMPREPLY=( $(compgen -W "$options" -- ${cur}) )
;;
diff --git a/completion/notmuch-completion.zsh 
b/completion/notmuch-completion.zsh
index bff8fd5..cf4968c 100644
--- a/completion/notmuch-completion.zsh
+++ b/completion/notmuch-completion.zsh
@@ -53,7 +53,8 @@ _notmuch_search()
 '--max-threads=[display only the first x threads from the search 
results]:number of threads to show: ' \
 '--first=[omit the first x threads from the search results]:number of 
threads to omit: ' \
 '--sort=[sort results]:sorting:((newest-first\:"reverse chronological 
order" oldest-first\:"chronological order"))' \
-'--output=[select what to output]:output:((summary threads messages files 
tags sender recipients addresses))'
+'--output=[select what to output]:output:((summary threads messages files 
tags sender recipients addresses))' \
+'--unique=[address deduplication]:unique:((none\:"no deduplication" 
addr\:"deduplicate by address" addrfold\:"deduplicate by case-insensitive 
address" name\:"deduplicate by name"))'
 }

 _notmuch()
diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
index 6094906..a92779a 100644
--- a/doc/man1/notmuch-search.rst
+++ b/doc/man1/notmuch-search.rst
@@ -85,6 +85,9 @@ Supported options for **search** include
 (--format=text0), as a JSON array (--format=json), or as
 an S-Expression list (--format=sexp).

+Handling of duplicate addresses and/or names can be
+controlled with the --unique option.
+
Note: Searching for **sender** should much be faster than
searching for **recipients** or **addresses**, because
sender addresses are cached directly in the database
@@ -151,6 +154,36 @@ Supported options for **search** include
 prefix. The prefix matches messages based on filenames. This
 option filters filenames of the matching messages.

+``--unique=``\ (**none**\ \|\ **addr**\ \|\ **addrfold**\ \|\ **name**)[,\ 
...]
+
+Can be used with ``--output=addresses``, ``--output=sender``
+or ``--output=recipients`` to control the address
+deduplication algorithm.
+
+   **none** means that no deduplication is performed. The same
+   address can appear multiple times in the output.
+
+   **addr** means that case-sensitive deduplication is performed
+   on the address part. For example, given the addresses "John
+   Doe " and "Dr. John Doe ",
+   only one will be printed.
+
+   **addrfold** means that case-insensitive deduplication is
+   performed on the address part. For example, given the
+   addresses "John Doe " and "John Doe
+   ", only one will be printed. This is the
+   default.
+
+   **name** means that case-sensitive deduplication is performed
+   on the name part. For example, given the addresses "John Doe
+   " and "John Doe ", only one
+   will be printed.
+
+   It is possible to combine the above flags (except **none**) by
+   separating them with comma. For example,
+   ``--unique=name,addr`` will print unique case-sensitive
+   combinations of name and address.
+
 EXIT STATUS
 ===

diff --git a/notmuch-search.c b/notmuch-search.c
index 0614f10..00d6771 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -33,6 +33,15 @@ typedef enum {
 OUTPUT_ADDRESSES   = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
 } output_t;

+typedef enum {
+UNIQUE_NONE  = 1 << 0,
+UNIQUE_ADDR  = 1 << 1,
+UNIQUE_NAME  = 1 << 2,
+UNIQUE_ADDR_CASEFOLD  = 1 << 3,
+
+UNIQUE_BOTH = UNIQUE_NAME | UNIQUE_ADDR,
+} unique_t;
+
 typedef struct {
 sprinter_t *format;
 notmuch_query_t *query;
@@ -41,6 +50,7 @@ typedef struct {
 int offset;
 int limit;
 int dupe;
+unique_t unique;
 } 

[PATCH 2/5] cli: Extend the search command for --output=addresses and similar

2014-09-22 Thread Michal Sojka
The new outputs allow printing senders, recipients or both of matching
messages.

This code based on a patch from Jani Nikula.
---
 completion/notmuch-completion.bash |   2 +-
 completion/notmuch-completion.zsh  |   3 +-
 doc/man1/notmuch-search.rst|  22 +++-
 notmuch-search.c   | 100 ++---
 4 files changed, 118 insertions(+), 9 deletions(-)

diff --git a/completion/notmuch-completion.bash 
b/completion/notmuch-completion.bash
index 0571dc9..c37ddf5 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -294,7 +294,7 @@ _notmuch_search()
return
;;
--output)
-   COMPREPLY=( $( compgen -W "summary threads messages files tags" -- 
"${cur}" ) )
+   COMPREPLY=( $( compgen -W "summary threads messages files tags 
sender recipients addresses" -- "${cur}" ) )
return
;;
--sort)
diff --git a/completion/notmuch-completion.zsh 
b/completion/notmuch-completion.zsh
index 67a9aba..bff8fd5 100644
--- a/completion/notmuch-completion.zsh
+++ b/completion/notmuch-completion.zsh
@@ -52,7 +52,8 @@ _notmuch_search()
   _arguments -s : \
 '--max-threads=[display only the first x threads from the search 
results]:number of threads to show: ' \
 '--first=[omit the first x threads from the search results]:number of 
threads to omit: ' \
-'--sort=[sort results]:sorting:((newest-first\:"reverse chronological 
order" oldest-first\:"chronological order"))'
+'--sort=[sort results]:sorting:((newest-first\:"reverse chronological 
order" oldest-first\:"chronological order"))' \
+'--output=[select what to output]:output:((summary threads messages files 
tags sender recipients addresses))'
 }

 _notmuch()
diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
index 90160f2..6094906 100644
--- a/doc/man1/notmuch-search.rst
+++ b/doc/man1/notmuch-search.rst
@@ -35,7 +35,7 @@ Supported options for **search** include
 intended for programs that invoke **notmuch(1)** internally. If
 omitted, the latest supported version will be used.

-``--output=(summary|threads|messages|files|tags)``
+
``--output=(summary|threads|messages|files|tags|sender|recipients|addresses)``

 **summary**
 Output a summary of each thread with any message matching
@@ -78,6 +78,26 @@ Supported options for **search** include
 by null characters (--format=text0), as a JSON array
 (--format=json), or as an S-Expression list (--format=sexp).

+   **sender**
+Output all addresses from the *From* header that appear on
+any message matching the search terms, either one per line
+(--format=text), separated by null characters
+(--format=text0), as a JSON array (--format=json), or as
+an S-Expression list (--format=sexp).
+
+   Note: Searching for **sender** should much be faster than
+   searching for **recipients** or **addresses**, because
+   sender addresses are cached directly in the database
+   whereas other addresses need to be fetched from the
+   message file by parsing it.
+
+   **recipients**
+Like **sender** but for addresses from *To*, *Cc* and
+   *Bcc* headers.
+
+   **addresses**
+   Like **sender** and **recipients** together.
+
 ``--sort=``\ (**newest-first**\ \|\ **oldest-first**)
 This option can be used to present results in either
 chronological order (**oldest-first**) or reverse chronological
diff --git a/notmuch-search.c b/notmuch-search.c
index 5ac2a26..0614f10 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -23,11 +23,14 @@
 #include "string-util.h"

 typedef enum {
-OUTPUT_SUMMARY,
-OUTPUT_THREADS,
-OUTPUT_MESSAGES,
-OUTPUT_FILES,
-OUTPUT_TAGS
+OUTPUT_SUMMARY = 1 << 0,
+OUTPUT_THREADS = 1 << 1,
+OUTPUT_MESSAGES= 1 << 2,
+OUTPUT_FILES   = 1 << 3,
+OUTPUT_TAGS= 1 << 4,
+OUTPUT_SENDER  = 1 << 5,
+OUTPUT_RECIPIENTS  = 1 << 6,
+OUTPUT_ADDRESSES   = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
 } output_t;

 typedef struct {
@@ -220,6 +223,67 @@ do_search_threads (search_options_t *o)
 return 0;
 }

+static void
+print_address_list (const search_options_t *o, InternetAddressList *list)
+{
+InternetAddress *address;
+int i;
+
+for (i = 0; i < internet_address_list_length (list); i++) {
+   address = internet_address_list_get_address (list, i);
+   if (INTERNET_ADDRESS_IS_GROUP (address)) {
+   InternetAddressGroup *group;
+   InternetAddressList *group_list;
+
+   group = INTERNET_ADDRESS_GROUP (address);
+   group_list = internet_address_group_get_members (group);
+   if (group_list == NULL)
+   continue;
+
+   print_address_list (o, group_list);
+   } else 

[PATCH 0/5] notmuch search --output=addresses

2014-09-22 Thread Michal Sojka
Hello all,

this patch series adds support for --output=addresses and similar
arguments in notmuch search. It is a reworked and extended version of
id:1410021689-15901-1-git-send-email-jani at nikula.org.

Compared to Jani's patches, I do not use keyword-flag command line
parser for the --output option, because this would not be useful for
most but one combination of keywords. Instead, this one combination is
explicitely mentioned as another keyword.

The flag command-line parser is introduced later, with (IMHO) better
syntax: --flags=one,two,... This feature is used to control address
deduplication via the --unique option.

Other extensions are added documentation, tests and shell completions.

The whole test suite passes with these patches.

The functionality presented here will be useful for improving Emacs
address completion (id:1411150602-21892-1-git-send-email-sojkam1 at 
fel.cvut.cz).

Another nice use case is sending patched with git:

  git send-email --cc-cmd='nmsenders id:' ...

where nmsenders script contains:

  notmuch search --output=sender $(notmuch search --output=threads $1)



Michal Sojka (5):
  cli: Refactor option passing in the search command
  cli: Extend the search command for --output=addresses and similar
  cli: Add support for parsing command line "flag" options
  cli: Add configurable address deduplication for --output=addresses
  cli: Add tests for 'search --output=addresses' and similar

 command-line-arguments.c   |  40 +
 command-line-arguments.h   |   1 +
 completion/notmuch-completion.bash |   8 +-
 completion/notmuch-completion.zsh  |   4 +-
 doc/man1/notmuch-search.rst|  55 ++-
 notmuch-search.c   | 311 +
 test/Makefile.local|   2 +-
 test/T090-search-output.sh |  59 +++
 test/T095-search-unique.sh |  63 
 test/T410-argument-parsing.sh  |   3 +-
 test/arg-test.c|   8 +
 11 files changed, 482 insertions(+), 72 deletions(-)
 create mode 100755 test/T095-search-unique.sh

-- 
2.1.0



[PATCH 00/11] notmuch insert updates

2014-09-22 Thread Jani Nikula
This series refactors and cleans up insert, improves error handling and
reporting, and adds post-insert hook. I intend to add documentation and
more tests, but the code is ready for review. Also, at least some of the
cleanups and fixes in the beginning of the series could go in without
additional tests or documentation.

BR,
Jani.


Jani Nikula (11):
  lib: actually return failures from
notmuch_message_tags_to_maildir_flags
  cli/insert: rename check_folder_name to is_valid_folder_name
  cli/insert: move add_file_to_database to a better place
  cli/insert: rename file copy function
  cli/insert: clean up sync_dir
  cli/insert: use a single recursive mkdir function
  cli/insert: abstract temporary filename generation
  cli/insert: rehash file writing functions
  cli/insert: add fail path to add_file_to_database
  cli/insert: require succesful message indexing for success status
  cli/insert: add post-insert hook

 lib/message.cc  |2 +-
 notmuch-insert.c|  462 +--
 test/T070-insert.sh |2 +-
 3 files changed, 262 insertions(+), 204 deletions(-)

-- 
1.7.2.5



[PATCH 01/11] lib: actually return failures from notmuch_message_tags_to_maildir_flags

2014-09-22 Thread Jani Nikula
The function takes great care to preserve the first error status it
encounters, yet fails to return that status to the caller. Fix it.
---
 lib/message.cc |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 68f7e68..7e82548 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1497,7 +1497,7 @@ notmuch_message_tags_to_maildir_flags (notmuch_message_t 
*message)
 talloc_free (to_set);
 talloc_free (to_clear);

-return NOTMUCH_STATUS_SUCCESS;
+return status;
 }

 notmuch_status_t
-- 
1.7.2.5



[PATCH 02/11] cli/insert: rename check_folder_name to is_valid_folder_name

2014-09-22 Thread Jani Nikula
An "is something" predicate conveys the meaning better. While at it,
improve the function documentation and error message. Besides the
error message change, no functional changes.
---
 notmuch-insert.c |   13 -
 1 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 8dfc8bb..770275b 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -83,10 +83,13 @@ sync_dir (const char *dir)
 return ret;
 }

-/* Check the specified folder name does not contain a directory
- * component ".." to prevent writes outside of the Maildir hierarchy. */
+/*
+ * Check the specified folder name does not contain a directory
+ * component ".." to prevent writes outside of the Maildir
+ * hierarchy. Return TRUE on valid folder name, FALSE otherwise.
+ */
 static notmuch_bool_t
-check_folder_name (const char *folder)
+is_valid_folder_name (const char *folder)
 {
 const char *p = folder;

@@ -449,8 +452,8 @@ notmuch_insert_command (notmuch_config_t *config, int argc, 
char *argv[])
 if (folder == NULL) {
maildir = db_path;
 } else {
-   if (! check_folder_name (folder)) {
-   fprintf (stderr, "Error: bad folder name: %s\n", folder);
+   if (! is_valid_folder_name (folder)) {
+   fprintf (stderr, "Error: invalid folder name: '%s'\n", folder);
return EXIT_FAILURE;
}
maildir = talloc_asprintf (config, "%s/%s", db_path, folder);
-- 
1.7.2.5



[PATCH 03/11] cli/insert: move add_file_to_database to a better place

2014-09-22 Thread Jani Nikula
Move add_file_to_database around to keep the filesystem related
functions grouped together, improving readability. No functional
changes.
---
 notmuch-insert.c |   92 +++---
 1 files changed, 46 insertions(+), 46 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 770275b..ccb091a 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -294,52 +294,6 @@ copy_stdin (int fdin, int fdout)
 return (!interrupted && !empty);
 }

-/* Add the specified message file to the notmuch database, applying tags.
- * The file is renamed to encode notmuch tags as maildir flags. */
-static void
-add_file_to_database (notmuch_database_t *notmuch, const char *path,
- tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
-{
-notmuch_message_t *message;
-notmuch_status_t status;
-
-status = notmuch_database_add_message (notmuch, path, );
-switch (status) {
-case NOTMUCH_STATUS_SUCCESS:
-case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
-   break;
-default:
-case NOTMUCH_STATUS_FILE_NOT_EMAIL:
-case NOTMUCH_STATUS_READ_ONLY_DATABASE:
-case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
-case NOTMUCH_STATUS_OUT_OF_MEMORY:
-case NOTMUCH_STATUS_FILE_ERROR:
-case NOTMUCH_STATUS_NULL_POINTER:
-case NOTMUCH_STATUS_TAG_TOO_LONG:
-case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
-case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
-case NOTMUCH_STATUS_LAST_STATUS:
-   fprintf (stderr, "Error: failed to add `%s' to notmuch database: %s\n",
-path, notmuch_status_to_string (status));
-   return;
-}
-
-if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
-   /* Don't change tags of an existing message. */
-   if (synchronize_flags) {
-   status = notmuch_message_tags_to_maildir_flags (message);
-   if (status != NOTMUCH_STATUS_SUCCESS)
-   fprintf (stderr, "Error: failed to sync tags to maildir 
flags\n");
-   }
-} else {
-   tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
-
-   tag_op_list_apply (message, tag_ops, flags);
-}
-
-notmuch_message_destroy (message);
-}
-
 static notmuch_bool_t
 write_message (void *ctx, int fdin, const char *dir, char **newpath)
 {
@@ -389,6 +343,52 @@ write_message (void *ctx, int fdin, const char *dir, char 
**newpath)
 return FALSE;
 }

+/* Add the specified message file to the notmuch database, applying tags.
+ * The file is renamed to encode notmuch tags as maildir flags. */
+static void
+add_file_to_database (notmuch_database_t *notmuch, const char *path,
+ tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
+{
+notmuch_message_t *message;
+notmuch_status_t status;
+
+status = notmuch_database_add_message (notmuch, path, );
+switch (status) {
+case NOTMUCH_STATUS_SUCCESS:
+case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+   break;
+default:
+case NOTMUCH_STATUS_FILE_NOT_EMAIL:
+case NOTMUCH_STATUS_READ_ONLY_DATABASE:
+case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
+case NOTMUCH_STATUS_OUT_OF_MEMORY:
+case NOTMUCH_STATUS_FILE_ERROR:
+case NOTMUCH_STATUS_NULL_POINTER:
+case NOTMUCH_STATUS_TAG_TOO_LONG:
+case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
+case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
+case NOTMUCH_STATUS_LAST_STATUS:
+   fprintf (stderr, "Error: failed to add `%s' to notmuch database: %s\n",
+path, notmuch_status_to_string (status));
+   return;
+}
+
+if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+   /* Don't change tags of an existing message. */
+   if (synchronize_flags) {
+   status = notmuch_message_tags_to_maildir_flags (message);
+   if (status != NOTMUCH_STATUS_SUCCESS)
+   fprintf (stderr, "Error: failed to sync tags to maildir 
flags\n");
+   }
+} else {
+   tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
+
+   tag_op_list_apply (message, tag_ops, flags);
+}
+
+notmuch_message_destroy (message);
+}
+
 int
 notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
 {
-- 
1.7.2.5



[PATCH 04/11] cli/insert: rename file copy function

2014-09-22 Thread Jani Nikula
The copying has nothing to do with stdin, so call it copy_fd
instead. While at it, improve documentation and reverse the
parameters, as destination is traditionally the first parameter.
---
 notmuch-insert.c |   11 ++-
 1 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index ccb091a..5d47806 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -251,11 +251,12 @@ maildir_open_tmp_file (void *ctx, const char *dir,
 return fd;
 }

-/* Copy the contents of standard input (fdin) into fdout.
- * Returns TRUE if a non-empty file was written successfully.
- * Otherwise, return FALSE. */
+/*
+ * Copy fdin to fdout, return TRUE on success, and FALSE on errors and
+ * empty input.
+ */
 static notmuch_bool_t
-copy_stdin (int fdin, int fdout)
+copy_fd (int fdout, int fdin)
 {
 notmuch_bool_t empty = TRUE;

@@ -308,7 +309,7 @@ write_message (void *ctx, int fdin, const char *dir, char 
**newpath)

 cleanup_path = tmppath;

-if (! copy_stdin (fdin, fdout))
+if (! copy_fd (fdout, fdin))
goto FAIL;

 if (fsync (fdout) != 0) {
-- 
1.7.2.5



[PATCH 05/11] cli/insert: clean up sync_dir

2014-09-22 Thread Jani Nikula
Clarify the code slightly, improve error messages. Apart from the
error message changes, no functional changes.
---
 notmuch-insert.c |   17 +
 1 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 5d47806..7375c54 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -67,20 +67,21 @@ safe_gethostname (char *hostname, size_t len)
 static notmuch_bool_t
 sync_dir (const char *dir)
 {
-notmuch_bool_t ret;
-int fd;
+int fd, r;

 fd = open (dir, O_RDONLY);
 if (fd == -1) {
-   fprintf (stderr, "Error: open() dir failed: %s\n", strerror (errno));
+   fprintf (stderr, "Error: open %s: %s\n", dir, strerror (errno));
return FALSE;
 }
-ret = (fsync (fd) == 0);
-if (! ret) {
-   fprintf (stderr, "Error: fsync() dir failed: %s\n", strerror (errno));
-}
+
+r = fsync (fd);
+if (r)
+   fprintf (stderr, "Error: fsync %s: %s\n", dir, strerror (errno));
+
 close (fd);
-return ret;
+
+return r == 0;
 }

 /*
-- 
1.7.2.5



[PATCH 06/11] cli/insert: use a single recursive mkdir function

2014-09-22 Thread Jani Nikula
Combine make_directory() and make_directory_and_parents() into a
single recursive mkdir_recursive() function. Clarify the code and
improve error handling. Improve error messages. Switch to using the
new function in maildir_create_folder(). Constify talloc context.
---
 notmuch-insert.c |  131 +++---
 1 files changed, 55 insertions(+), 76 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 7375c54..cdeeb41 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -104,96 +104,78 @@ is_valid_folder_name (const char *folder)
 }
 }

-/* Make the given directory, succeeding if it already exists. */
+/*
+ * Make the given directory and its parents as necessary, using the
+ * given mode. Return TRUE on success, FALSE otherwise. Partial
+ * results are not cleaned up on errors.
+ */
 static notmuch_bool_t
-make_directory (char *path, int mode)
+mkdir_recursive (const void *ctx, const char *path, int mode)
 {
-notmuch_bool_t ret;
-char *slash;
+struct stat st;
+int r;
+char *parent = NULL, *slash;

-if (mkdir (path, mode) != 0)
-   return (errno == EEXIST);
+/* First check the common case: directory already exists. */
+r = stat (path, );
+if (r == 0) {
+if (! S_ISDIR (st.st_mode)) {
+   fprintf (stderr, "Error: '%s' is not a directory: %s\n",
+path, strerror (EEXIST));
+   return FALSE;
+   }

-/* Sync the parent directory for durability. */
-ret = TRUE;
-slash = strrchr (path, '/');
-if (slash) {
-   *slash = '\0';
-   ret = sync_dir (path);
-   *slash = '/';
+   return TRUE;
+} else if (errno != ENOENT) {
+   fprintf (stderr, "Error: stat '%s': %s\n", path, strerror (errno));
+   return FALSE;
 }
-return ret;
-}
-
-/* Make the given directory including its parent directories as necessary.
- * Return TRUE on success, FALSE on error. */
-static notmuch_bool_t
-make_directory_and_parents (char *path, int mode)
-{
-struct stat st;
-char *start;
-char *end;
-notmuch_bool_t ret;

-/* First check the common case: directory already exists. */
-if (stat (path, ) == 0)
-   return S_ISDIR (st.st_mode) ? TRUE : FALSE;
-
-for (start = path; *start != '\0'; start = end + 1) {
-   /* start points to the first unprocessed character.
-* Find the next slash from start onwards. */
-   end = strchr (start, '/');
-
-   /* If there are no more slashes then all the parent directories
-* have been made.  Now attempt to make the whole path. */
-   if (end == NULL)
-   return make_directory (path, mode);
-
-   /* Make the path up to the next slash, unless the current
-* directory component is actually empty. */
-   if (end > start) {
-   *end = '\0';
-   ret = make_directory (path, mode);
-   *end = '/';
-   if (! ret)
-   return FALSE;
+/* mkdir parents, if any */
+slash = strrchr (path, '/');
+if (slash && slash != path) {
+   parent = talloc_strndup (ctx, path, slash - path);
+   if (! parent) {
+   fprintf (stderr, "Error: %s\n", strerror (ENOMEM));
+   return FALSE;
}
+
+   if (! mkdir_recursive (ctx, parent, mode))
+   return FALSE;
 }

-return TRUE;
+if (mkdir (path, mode)) {
+   fprintf (stderr, "Error: mkdir '%s': %s\n", path, strerror (errno));
+   return FALSE;
+}
+
+return parent ? sync_dir (parent) : TRUE;
 }

-/* Create the given maildir folder, i.e. dir and its subdirectories
- * 'cur', 'new', 'tmp'. */
+/*
+ * Create the given maildir folder, i.e. maildir and its
+ * subdirectories cur/new/tmp. Return TRUE on success, FALSE
+ * otherwise. Partial results are not cleaned up on errors.
+ */
 static notmuch_bool_t
-maildir_create_folder (void *ctx, const char *dir)
+maildir_create_folder (const void *ctx, const char *maildir)
 {
+const char *subdirs[] = { "cur", "new", "tmp" };
 const int mode = 0700;
 char *subdir;
-char *tail;
-
-/* Create 'cur' directory, including parent directories. */
-subdir = talloc_asprintf (ctx, "%s/cur", dir);
-if (! subdir) {
-   fprintf (stderr, "Out of memory.\n");
-   return FALSE;
-}
-if (! make_directory_and_parents (subdir, mode))
-   return FALSE;
-
-tail = subdir + strlen (subdir) - 3;
+unsigned int i;

-/* Create 'new' directory. */
-strcpy (tail, "new");
-if (! make_directory (subdir, mode))
-   return FALSE;
+for (i = 0; i < ARRAY_SIZE (subdirs); i++) {
+   subdir = talloc_asprintf (ctx, "%s/%s", maildir, subdirs[i]);
+   if (! subdir) {
+   fprintf (stderr, "Error: %s\n", strerror (ENOMEM));
+   return FALSE;
+   }

-/* Create 'tmp' directory. */
-strcpy (tail, "tmp");
-if (! make_directory (subdir, mode))
-   return FALSE;
+   if (! 

[PATCH 08/11] cli/insert: rehash file writing functions

2014-09-22 Thread Jani Nikula
Make the function calls make more sense as independent building blocks
of the big picture, with clear inputs and outputs. Split up
write_message into two. Improve function documentation. Cleanup and
clarify the error paths.
---
 notmuch-insert.c |  127 --
 1 files changed, 75 insertions(+), 52 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index a1d564c..5ef6e66 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -204,47 +204,37 @@ tempfilename (const void *ctx)
 return filename;
 }

-/* Open a unique file in the 'tmp' sub-directory of dir.
- * Returns the file descriptor on success, or -1 on failure.
- * On success, file paths for the message in the 'tmp' and 'new'
- * directories are returned via tmppath and newpath,
- * and the path of the 'new' directory itself in newdir. */
+/*
+ * Create a unique temporary file in maildir/tmp, return fd and full
+ * path to file in *path_out, or -1 on errors (in which case *path_out
+ * is not touched).
+ */
 static int
-maildir_open_tmp_file (void *ctx, const char *dir,
-  char **tmppath, char **newpath, char **newdir)
+maildir_mktemp (const void *ctx, const char *maildir, char **path_out)
 {
-char *filename;
-int fd = -1;
+char *filename, *path;
+int fd;

 do {
filename = tempfilename (ctx);
if (! filename)
return -1;

-   *tmppath = talloc_asprintf (ctx, "%s/tmp/%s", dir, filename);
-   if (! *tmppath) {
-   fprintf (stderr, "Out of memory\n");
+   path = talloc_asprintf (ctx, "%s/tmp/%s", maildir, filename);
+   if (! path) {
+   fprintf (stderr, "Error: %s\n", strerror (ENOMEM));
return -1;
}

-   fd = open (*tmppath, O_WRONLY | O_CREAT | O_TRUNC | O_EXCL, 0600);
+   fd = open (path, O_WRONLY | O_CREAT | O_TRUNC | O_EXCL, 0600);
 } while (fd == -1 && errno == EEXIST);

 if (fd == -1) {
-   fprintf (stderr, "Error: opening %s: %s\n", *tmppath, strerror (errno));
+   fprintf (stderr, "Error: open '%s': %s\n", path, strerror (errno));
return -1;
 }

-*newdir = talloc_asprintf (ctx, "%s/new", dir);
-*newpath = talloc_asprintf (ctx, "%s/new/%s", dir, filename);
-if (! *newdir || ! *newpath) {
-   fprintf (stderr, "Out of memory\n");
-   close (fd);
-   unlink (*tmppath);
-   return -1;
-}
-
-talloc_free (filename);
+*path_out = path;

 return fd;
 }
@@ -293,53 +283,85 @@ copy_fd (int fdout, int fdin)
 return (!interrupted && !empty);
 }

-static notmuch_bool_t
-write_message (void *ctx, int fdin, const char *dir, char **newpath)
+/*
+ * Write fdin to a new temp file in maildir/tmp, return full path to
+ * the file, or NULL on errors.
+ */
+static char *
+maildir_write_tmp (const void *ctx, int fdin, const char *maildir)
 {
-char *tmppath;
-char *newdir;
-char *cleanup_path;
+char *path;
 int fdout;

-fdout = maildir_open_tmp_file (ctx, dir, , newpath, );
+fdout = maildir_mktemp (ctx, maildir, );
 if (fdout < 0)
-   return FALSE;
-
-cleanup_path = tmppath;
+   return NULL;

 if (! copy_fd (fdout, fdin))
goto FAIL;

-if (fsync (fdout) != 0) {
-   fprintf (stderr, "Error: fsync failed: %s\n", strerror (errno));
+if (fsync (fdout)) {
+   fprintf (stderr, "Error: fsync '%s': %s\n", path, strerror (errno));
goto FAIL;
 }

 close (fdout);
-fdout = -1;
-
-/* Atomically move the new message file from the Maildir 'tmp' directory
- * to the 'new' directory.  We follow the Dovecot recommendation to
- * simply use rename() instead of link() and unlink().
- * See also: http://wiki.dovecot.org/MailboxFormat/Maildir#Mail_delivery
- */
-if (rename (tmppath, *newpath) != 0) {
-   fprintf (stderr, "Error: rename() failed: %s\n", strerror (errno));
+
+return path;
+
+FAIL:
+close (fdout);
+unlink (path);
+
+return NULL;
+}
+
+/*
+ * Write fdin to a new file in maildir/new, using an intermediate temp
+ * file in maildir/tmp, return full path to the new file, or NULL on
+ * errors.
+ */
+static char *
+maildir_write_new (const void *ctx, int fdin, const char *maildir)
+{
+char *cleanpath, *tmppath, *newpath, *newdir;
+
+tmppath = maildir_write_tmp (ctx, fdin, maildir);
+if (! tmppath)
+   return NULL;
+cleanpath = tmppath;
+
+newpath = talloc_strdup (ctx, tmppath);
+if (! newpath) {
+   fprintf (stderr, "Error: %s\n", strerror (ENOMEM));
goto FAIL;
 }

-cleanup_path = *newpath;
+/* sanity checks needed? */
+memcpy (newpath + strlen (maildir) + 1, "new", 3);
+
+if (rename (tmppath, newpath)) {
+   fprintf (stderr, "Error: rename '%s' '%s': %s\n",
+tmppath, newpath, strerror (errno));
+   goto FAIL;
+}
+cleanpath = newpath;
+
+newdir = talloc_asprintf (ctx, "%s/%s", maildir, "new");
+

[PATCH 09/11] cli/insert: add fail path to add_file_to_database

2014-09-22 Thread Jani Nikula
Handle failures gracefully in add_file_to_database, renamed simply
add_file while at it. Add keep option to not remove the message from
database if tagging or tag syncing to maildir flags fails. Expand the
function documentation to cover the changes.
---
 notmuch-insert.c |   89 -
 1 files changed, 54 insertions(+), 35 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 5ef6e66..80f52d4 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -364,50 +364,70 @@ FAIL:
 return NULL;
 }

-/* Add the specified message file to the notmuch database, applying tags.
- * The file is renamed to encode notmuch tags as maildir flags. */
-static void
-add_file_to_database (notmuch_database_t *notmuch, const char *path,
- tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
+/*
+ * Add the specified message file to the notmuch database, applying
+ * tags in tag_ops. If synchronize_flags is TRUE, the tags are
+ * synchronized to maildir flags (which may result in message file
+ * rename).
+ *
+ * Return NOTMUCH_STATUS_SUCCESS on success, errors otherwise. If keep
+ * is TRUE, errors in tag changes and flag syncing are ignored and
+ * success status is returned; otherwise such errors cause the message
+ * to be removed from the database. Failure to add the message to the
+ * database results in error status regardless of keep.
+ */
+static notmuch_status_t
+add_file (notmuch_database_t *notmuch, const char *path, tag_op_list_t 
*tag_ops,
+ notmuch_bool_t synchronize_flags, notmuch_bool_t keep)
 {
 notmuch_message_t *message;
 notmuch_status_t status;

 status = notmuch_database_add_message (notmuch, path, );
-switch (status) {
-case NOTMUCH_STATUS_SUCCESS:
-case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
-   break;
-default:
-case NOTMUCH_STATUS_FILE_NOT_EMAIL:
-case NOTMUCH_STATUS_READ_ONLY_DATABASE:
-case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
-case NOTMUCH_STATUS_OUT_OF_MEMORY:
-case NOTMUCH_STATUS_FILE_ERROR:
-case NOTMUCH_STATUS_NULL_POINTER:
-case NOTMUCH_STATUS_TAG_TOO_LONG:
-case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
-case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
-case NOTMUCH_STATUS_LAST_STATUS:
-   fprintf (stderr, "Error: failed to add `%s' to notmuch database: %s\n",
-path, notmuch_status_to_string (status));
-   return;
-}
-
-if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
-   /* Don't change tags of an existing message. */
-   if (synchronize_flags) {
-   status = notmuch_message_tags_to_maildir_flags (message);
-   if (status != NOTMUCH_STATUS_SUCCESS)
-   fprintf (stderr, "Error: failed to sync tags to maildir 
flags\n");
+if (status == NOTMUCH_STATUS_SUCCESS) {
+   status = tag_op_list_apply (message, tag_ops, 0);
+   if (status) {
+   fprintf (stderr, "%s: failed to apply tags to file '%s': %s\n",
+keep ? "Warning" : "Error",
+path, notmuch_status_to_string (status));
+   goto DONE;
}
+} else if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+   status = NOTMUCH_STATUS_SUCCESS;
+} else if (status == NOTMUCH_STATUS_FILE_NOT_EMAIL) {
+   fprintf (stderr, "Error: delivery of non-mail file: '%s'\n", path);
+   return status;
 } else {
-   tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
+   fprintf (stderr, "Error: failed to add '%s' to notmuch database: %s\n",
+path, notmuch_status_to_string (status));
+   return status;
+}

-   tag_op_list_apply (message, tag_ops, flags);
+if (synchronize_flags) {
+   status = notmuch_message_tags_to_maildir_flags (message);
+   if (status != NOTMUCH_STATUS_SUCCESS)
+   fprintf (stderr, "%s: failed to sync tags to maildir flags for 
'%s': %s\n",
+keep ? "Warning" : "Error",
+path, notmuch_status_to_string (status));
+
+   /*
+* Note: Unfortunately a failed maildir flag sync might
+* already have renamed the file, in which case the cleanup
+* path will fail.
+*/
 }

+DONE:
 notmuch_message_destroy (message);
+
+if (status) {
+   if (keep)
+   status = NOTMUCH_STATUS_SUCCESS;
+   else
+   notmuch_database_remove_message (notmuch, path);
+}
+
+return status;
 }

 int
@@ -508,8 +528,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc, 
char *argv[])
 /* Add the message to the index.
  * Even if adding the message to the notmuch database fails,
  * the message is on disk and we consider the delivery completed. */
-add_file_to_database (notmuch, newpath, tag_ops,
-   synchronize_flags);
+add_file (notmuch, newpath, tag_ops, synchronize_flags, TRUE);

 notmuch_database_destroy (notmuch);

[PATCH 10/11] cli/insert: require succesful message indexing for success status

2014-09-22 Thread Jani Nikula
Add --keep option to keep any remaining stuff in index or file. We
could distinguish between failures to index and failures to apply tags
or maildir sync, but for simplicity just have one.
---
 notmuch-insert.c|   20 +++-
 test/T070-insert.sh |2 +-
 2 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 80f52d4..f27b9cb 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -433,6 +433,7 @@ DONE:
 int
 notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
 {
+notmuch_status_t status;
 notmuch_database_t *notmuch;
 struct sigaction action;
 const char *db_path;
@@ -442,6 +443,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc, 
char *argv[])
 char *query_string = NULL;
 const char *folder = NULL;
 notmuch_bool_t create_folder = FALSE;
+notmuch_bool_t keep = FALSE;
 notmuch_bool_t synchronize_flags;
 const char *maildir;
 char *newpath;
@@ -451,6 +453,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc, 
char *argv[])
 notmuch_opt_desc_t options[] = {
{ NOTMUCH_OPT_STRING, , "folder", 0, 0 },
{ NOTMUCH_OPT_BOOLEAN, _folder, "create-folder", 0, 0 },
+   { NOTMUCH_OPT_BOOLEAN, , "keep", 0, 0 },
{ NOTMUCH_OPT_END, 0, 0, 0, 0 }
 };

@@ -525,11 +528,18 @@ notmuch_insert_command (notmuch_config_t *config, int 
argc, char *argv[])
return EXIT_FAILURE;
 }

-/* Add the message to the index.
- * Even if adding the message to the notmuch database fails,
- * the message is on disk and we consider the delivery completed. */
-add_file (notmuch, newpath, tag_ops, synchronize_flags, TRUE);
+/* Index the message. */
+status = add_file (notmuch, newpath, tag_ops, synchronize_flags, keep);
+if (status) {
+   if (keep) {
+   status = NOTMUCH_STATUS_SUCCESS;
+   } else {
+   /* If maildir flag sync failed, this might fail. */
+   unlink (newpath);
+   }
+}

 notmuch_database_destroy (notmuch);
-return EXIT_SUCCESS;
+
+return status ? EXIT_FAILURE : EXIT_SUCCESS;
 }
diff --git a/test/T070-insert.sh b/test/T070-insert.sh
index ea9db07..aacc643 100755
--- a/test/T070-insert.sh
+++ b/test/T070-insert.sh
@@ -23,7 +23,7 @@ test_expect_code 1 "Insert zero-length file" \

 # This test is a proxy for other errors that may occur while trying to
 # add a message to the notmuch database, e.g. database locked.
-test_expect_code 0 "Insert non-message" \
+test_expect_code 1 "Insert non-message" \
 "echo bad_message | notmuch insert"

 test_begin_subtest "Database empty so far"
-- 
1.7.2.5



[PATCH 11/11] cli/insert: add post-insert hook

2014-09-22 Thread Jani Nikula
The post-new hook might no longer be needed or run very often if
notmuch insert is being used. Therefore a post-insert hook is needed
(arguably pre-insert not so much, so don't add one). Also add the
--no-hooks option to skip hooks.
---
 notmuch-insert.c |7 +++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index f27b9cb..adadd12 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -444,6 +444,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc, 
char *argv[])
 const char *folder = NULL;
 notmuch_bool_t create_folder = FALSE;
 notmuch_bool_t keep = FALSE;
+notmuch_bool_t no_hooks = FALSE;
 notmuch_bool_t synchronize_flags;
 const char *maildir;
 char *newpath;
@@ -454,6 +455,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc, 
char *argv[])
{ NOTMUCH_OPT_STRING, , "folder", 0, 0 },
{ NOTMUCH_OPT_BOOLEAN, _folder, "create-folder", 0, 0 },
{ NOTMUCH_OPT_BOOLEAN, , "keep", 0, 0 },
+   { NOTMUCH_OPT_BOOLEAN,  _hooks, "no-hooks", 'n', 0 },
{ NOTMUCH_OPT_END, 0, 0, 0, 0 }
 };

@@ -541,5 +543,10 @@ notmuch_insert_command (notmuch_config_t *config, int 
argc, char *argv[])

 notmuch_database_destroy (notmuch);

+if (! no_hooks && status == NOTMUCH_STATUS_SUCCESS) {
+   /* Ignore hook failures. */
+   notmuch_run_hook (db_path, "post-insert");
+}
+
 return status ? EXIT_FAILURE : EXIT_SUCCESS;
 }
-- 
1.7.2.5



[PATCH] Add configurable changed tag to messages that have been changed on disk

2014-09-22 Thread Gaute Hope
Excerpts from Gaute Hope's message of August 6, 2014 10:29:
> Austin Clements  wrote on Fri, 01 Aug 2014 14:55:05 
> -0400:
>> I have a prototype implementation of message modification times on my
>> lastmod-v1 branch at
>> 
>>   https://github.com/aclements/notmuch/tree/lastmod-v1
>> 
>> It builds on my database features series that's currently awaiting
>> review [1].
>> 
>> The series uses a monotonic revision number, rather than wall-clock
>> time, for reasons related to Xapian's concurrent control and detailed
>> in the main commit's commit message.  The implementation isn't quite
>> useful from the CLI yet because I haven't added any way to query the
>> database's current revision number.  (I'm still thinking about how I
>> want to do this, since search/show don't have a good way to deliver
>> "additional" information right now.  I might just add the last
>> modification for each individual message/max of all messages in a
>> thread, similar to what Thomas Jost's patch did long ago.)
>> 
>> [1] id:1406859003-11561-1-git-send-email-amdragon at mit.edu

> this should allow me to do what I wish to accomplish. The message
> deletion is still a problem though, I can see two options at the moment:

Hi list,

While exploring the possibility of syncing maildir/X-keywords with tags
I had some thoughts about lastmod and message modification:

As briefly discussed on #notmuch, I noticed that it seems that 'notmuch
new' does not detect that a message source has been changed, unless the
file is also re-named.

This means that for instance if the X-Keywords fields have been updated
in a message (from GMail with offlineimap, synclabels = yes) the lastmod
field will remain unchanged, and a source modification will be
undetectable to a client program using this value.

Would it not make sense that if a message has a more recent mtime than
at index time it is re-indexed?

Also, for the lastmod branch I would wish for a notmuch_message_touch()
method where the lastmod time is updated to the last. As well as a
notmuch_database_reindex_message () - possibly defined/documented
behaviour for notmuch_database_add_message () when the filename is
already added (in which case I would expect notmuch to re-index the
message).

Doing notmuch_database_remove_message followed by _add_message could
risk deleting the entry if this file is the only on-disk-representation.

Cheers, Gaute



Do path: searches handle spaces correctly?

2014-09-22 Thread Keith Amidon
On Mon, 2014-09-22 at 11:20 +0200, David Bremner wrote:
> Keith Amidon  writes:
> >
> > notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items'
> >
> > I don't get any results, but it seems like the two results above should
> > be shown, right?  Am I doing something wrong here?  If it looks like I'm
> > doing it correctly, what can I do to help troubleshoot the issue?
> 
> Note that path:, unlike folder:, does not add the maildir subdirs. Dunno
> if that's the only issue you are having, but I guess it's one.

Darn it!  I made a mistake in my original email.  In the test I was
doing I actually had:

notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items/**'

which I believe should have picked up all the subdirectory paths.  I
just retested to make sure that it still didn't work, and it doesn't.

Am I still missing something?   Thanks for the help, Keith




Do path: searches handle spaces correctly?

2014-09-22 Thread Austin Clements
Quoth Keith Amidon on Sep 22 at  7:42 am:
> On Mon, 2014-09-22 at 11:20 +0200, David Bremner wrote:
> > Keith Amidon  writes:
> > >
> > > notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items'
> > >
> > > I don't get any results, but it seems like the two results above should
> > > be shown, right?  Am I doing something wrong here?  If it looks like I'm
> > > doing it correctly, what can I do to help troubleshoot the issue?
> > 
> > Note that path:, unlike folder:, does not add the maildir subdirs. Dunno
> > if that's the only issue you are having, but I guess it's one.
> 
> Darn it!  I made a mistake in my original email.  In the test I was
> doing I actually had:
> 
> notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items/**'
> 
> which I believe should have picked up all the subdirectory paths.  I
> just retested to make sure that it still didn't work, and it doesn't.
> 
> Am I still missing something?   Thanks for the help, Keith

I assume you're doing this from the command line?  Does the following
work?

notmuch search --output=files 'path:"dir/INBOX/INBOX/Sent Items/**"'

Shell quoting and Xapian quoting interact in often confusing ways.  In
your original command line, the single quotes suppress shell argument
splitting, but never make it to notmuch, so notmuch sees two search
terms "path:dir/INBOX/INBOX/Sent" and "Items/**".  In the command line
I suggested, the single quotes play the same role, but for the entire
query.  Because of the quoting, notmuch *does* see the inner double
quotes, which makes the path a single term (note that Xapian only
accepts double quotes, not single quotes).  In general, it's good to
enclose the entire command line query in single quotes so shell
parsing doesn't get in the way of query parsing.


[PATCH] Add configurable changed tag to messages that have been changed on disk

2014-09-22 Thread Tomi Ollila
On Mon, Sep 22 2014, Gaute Hope  wrote:

> Excerpts from Gaute Hope's message of August 6, 2014 10:29:
>> Austin Clements  wrote on Fri, 01 Aug 2014 14:55:05 
>> -0400:
>>> I have a prototype implementation of message modification times on my
>>> lastmod-v1 branch at
>>> 
>>>   https://github.com/aclements/notmuch/tree/lastmod-v1
>>> 
>>> It builds on my database features series that's currently awaiting
>>> review [1].
>>> 
>>> The series uses a monotonic revision number, rather than wall-clock
>>> time, for reasons related to Xapian's concurrent control and detailed
>>> in the main commit's commit message.  The implementation isn't quite
>>> useful from the CLI yet because I haven't added any way to query the
>>> database's current revision number.  (I'm still thinking about how I
>>> want to do this, since search/show don't have a good way to deliver
>>> "additional" information right now.  I might just add the last
>>> modification for each individual message/max of all messages in a
>>> thread, similar to what Thomas Jost's patch did long ago.)
>>> 
>>> [1] id:1406859003-11561-1-git-send-email-amdragon at mit.edu
>  
>> this should allow me to do what I wish to accomplish. The message
>> deletion is still a problem though, I can see two options at the moment:
>
> Hi list,
>
> While exploring the possibility of syncing maildir/X-keywords with tags
> I had some thoughts about lastmod and message modification:
>
> As briefly discussed on #notmuch, I noticed that it seems that 'notmuch
> new' does not detect that a message source has been changed, unless the
> file is also re-named.
>
> This means that for instance if the X-Keywords fields have been updated
> in a message (from GMail with offlineimap, synclabels = yes) the lastmod
> field will remain unchanged, and a source modification will be
> undetectable to a client program using this value.
>
> Would it not make sense that if a message has a more recent mtime than
> at index time it is re-indexed?

That would require notmuch to scan the contents of a directory for changed
mtimes of the files -- now notmuch skips looking for the files unless
directory mtime has changed. Directory mtime changes when files are
added/deleted/renamed (as that is what directory needs to know) -- file
mtime change are stored in file information and therefore change there does
not probagate to parent directory (and, if such happened, to it's parent
and so on...)

That would mean the scanning would be slower than it is now.

Tomi

>
> Also, for the lastmod branch I would wish for a notmuch_message_touch()
> method where the lastmod time is updated to the last. As well as a
> notmuch_database_reindex_message () - possibly defined/documented
> behaviour for notmuch_database_add_message () when the filename is
> already added (in which case I would expect notmuch to re-index the
> message).
>
> Doing notmuch_database_remove_message followed by _add_message could
> risk deleting the entry if this file is the only on-disk-representation.
>
> Cheers, Gaute


[PATCH] Add configurable changed tag to messages that have been changed on disk

2014-09-22 Thread Austin Clements
On Mon, 22 Sep 2014, Gaute Hope  wrote:
> Excerpts from Gaute Hope's message of August 6, 2014 10:29:
>> Austin Clements  wrote on Fri, 01 Aug 2014 14:55:05 
>> -0400:
>>> I have a prototype implementation of message modification times on my
>>> lastmod-v1 branch at
>>> 
>>>   https://github.com/aclements/notmuch/tree/lastmod-v1
>>> 
>>> It builds on my database features series that's currently awaiting
>>> review [1].
>>> 
>>> The series uses a monotonic revision number, rather than wall-clock
>>> time, for reasons related to Xapian's concurrent control and detailed
>>> in the main commit's commit message.  The implementation isn't quite
>>> useful from the CLI yet because I haven't added any way to query the
>>> database's current revision number.  (I'm still thinking about how I
>>> want to do this, since search/show don't have a good way to deliver
>>> "additional" information right now.  I might just add the last
>>> modification for each individual message/max of all messages in a
>>> thread, similar to what Thomas Jost's patch did long ago.)
>>> 
>>> [1] id:1406859003-11561-1-git-send-email-amdragon at mit.edu
>  
>> this should allow me to do what I wish to accomplish. The message
>> deletion is still a problem though, I can see two options at the moment:
>
> Hi list,
>
> While exploring the possibility of syncing maildir/X-keywords with tags
> I had some thoughts about lastmod and message modification:
>
> As briefly discussed on #notmuch, I noticed that it seems that 'notmuch
> new' does not detect that a message source has been changed, unless the
> file is also re-named.
>
> This means that for instance if the X-Keywords fields have been updated
> in a message (from GMail with offlineimap, synclabels = yes) the lastmod
> field will remain unchanged, and a source modification will be
> undetectable to a client program using this value.
>
> Would it not make sense that if a message has a more recent mtime than
> at index time it is re-indexed?

This has the potential to make notmuch new substantially more expensive.
Currently, if there are no changes, it only has to stat each directory
in your maildir (in fact, some restructuring of new would let us
eliminate almost all database access during a no-op notmuch new as
well).  Checking for changes to individual messages would require
stat'ing every single message file as well as accessing the database to
check the paths and mtimes of every message, increasing the number of
stat calls and disk accesses by several orders of magnitude.

It may be that this is fast enough that it's okay, but it would be good
to gather some evidence first.  That includes hot and cold caches, and
maildir over NFS.

With respect to X-Keywords specifically, note that it's a fairly basic
design decision that notmuch never modifies message files.  This gives
us strong robustness guarantees we would be loathe to part with.

It has puzzled me ever since offlineimap added X-Keywords why they
didn't just translate these keywords into folders and create hard links
of message files.  Anything could interact smoothly with that.

> Also, for the lastmod branch I would wish for a notmuch_message_touch()
> method where the lastmod time is updated to the last. As well as a
> notmuch_database_reindex_message () - possibly defined/documented
> behaviour for notmuch_database_add_message () when the filename is
> already added (in which case I would expect notmuch to re-index the
> message).

What's the use case for these?

> Doing notmuch_database_remove_message followed by _add_message could
> risk deleting the entry if this file is the only on-disk-representation.
>
> Cheers, Gaute


[PATCH] lib: Simplify close and codify aborting atomic section

2014-09-22 Thread Austin Clements
In Xapian, closing a database implicitly aborts any outstanding
transaction and commits changes.  For historical reasons,
notmuch_database_close had grown to almost, but not quite duplicate
this behavior.  Before closing the database, it would explicitly (and
unnecessarily) commit it.  However, if there was an outstanding
transaction (ie atomic section), commit would throw a Xapian
exception, which notmuch_database_close would unnecessarily print to
stderr, even though notmuch_database_close would ultimately abort the
transaction anyway when it called close.

This patch simplifies notmuch_database_close to just call
Database::close.  This works for both read-only and read/write
databases, takes care of committing changes, unifies the exception
handling path, and codifies aborting outstanding transactions.  This
is currently the only way to abort an atomic section (and may remain
so, since it would be difficult to roll back things we may have cached
from rolled-back modifications).
---
 lib/database.cc | 23 +++
 lib/notmuch.h   |  5 +
 2 files changed, 12 insertions(+), 16 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index a3a7cd3..1f7ff2a 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -903,28 +903,19 @@ notmuch_database_close (notmuch_database_t *notmuch)
 {
 notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;

-try {
-   if (notmuch->xapian_db != NULL &&
-   notmuch->mode == NOTMUCH_DATABASE_MODE_READ_WRITE)
-   (static_cast  
(notmuch->xapian_db))->flush ();
-} catch (const Xapian::Error ) {
-   status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
-   if (! notmuch->exception_reported) {
-   fprintf (stderr, "Error: A Xapian exception occurred flushing 
database: %s\n",
-error.get_msg().c_str());
-   }
-}
-
 /* Many Xapian objects (and thus notmuch objects) hold references to
  * the database, so merely deleting the database may not suffice to
- * close it.  Thus, we explicitly close it here. */
+ * close it.  Thus, we explicitly close it here.  This will
+ * implicitly abort any outstanding transaction and commit changes. */
 if (notmuch->xapian_db != NULL) {
try {
notmuch->xapian_db->close();
} catch (const Xapian::Error ) {
-   /* don't clobber previous error status */
-   if (status == NOTMUCH_STATUS_SUCCESS)
-   status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+   status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+   if (! notmuch->exception_reported) {
+   fprintf (stderr, "Error: A Xapian exception occurred closing 
database: %s\n",
+error.get_msg().c_str());
+   }
}
 }

diff --git a/lib/notmuch.h b/lib/notmuch.h
index fe2340b..5c40c67 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -292,6 +292,11 @@ notmuch_database_open (const char *path,
  * notmuch_database_close can be called multiple times.  Later calls
  * have no effect.
  *
+ * If the caller is currently in an atomic section (there was a
+ * notmuch_database_begin_atomic without a matching
+ * notmuch_database_end_atomic), this will abort the atomic section,
+ * discarding any modifications made in the atomic section.
+ *
  * Return value:
  *
  * NOTMUCH_STATUS_SUCCESS: Successfully closed the database.
-- 
2.1.0



[PATCH] lib: Simplify close and codify aborting atomic section

2014-09-22 Thread W. Trevor King
On Mon, Sep 22, 2014 at 11:43:35AM -0400, Austin Clements wrote:
> This patch simplifies notmuch_database_close to just call
> Database::close.  This works for both read-only and read/write
> databases, takes care of committing changes, unifies the exception
> handling path, and codifies aborting outstanding transactions.

If we're dropping the flush call here, where will it be triggered
instead?  We'll need to flush/commit our changes to the database at
some point before closing.  Do clients now need an explicit
flush/commit command (explicit, client-initiated flushes sound like a
good idea to me).

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20140922/e44bf0ac/attachment.pgp>


[announce] nmhive v0.1.0

2014-09-22 Thread W. Trevor King
I like nmbug's distributed tag maintenance, but not everyone has
notmuch/nmbug installed locally (yet ;).  However, everyone that I
know does have a browser and a mail client.  They can submit messages
with their mail client already, but we've been missing a way for them
to help tag messages.  Nmhive is a lightweight server that allows
clients to read and write notmuch tags using a JSON API [1].  It uses
nmbug locally to commit after each write, and the admin can then pull
tag changes made by nmhive and push them into the native nmbug
ecosystem:

  web client ? nmhive ? nmhive's nmbug ? admin's nmbug ? tethera's nmbug

To make it easy for folks to drive nmhive, the repository also
contains a bookmarklet [2] that you can use to interactively manage
tags from a message's Gmane page (e.g., [3,4]).  The setup currently
uses my ported-to-Python nmbug [5] if you want to play with it
locally.  I haven't added user authentication yet, so it's probably
best to just run your own nmhive for now.  If anyone has preferences
for authentication, send a patch :).  Or at least let me know, and
I'll see what I can do ;).  I'd appreciate any other feedback folks
have as well.

Cheers,
Trevor

[1]: https://github.com/wking/nmhive
[2]: https://github.com/wking/nmhive/blob/v0.1.0/nmbug.js
[3]: http://thread.gmane.org/gmane.mail.notmuch.general/19091/focus=19092
[4]: http://article.gmane.org/gmane.mail.notmuch.general/19092
[5]: http://thread.gmane.org/gmane.mail.notmuch.general/19007
 id:e630b6763e9d0771718afee41ea15b29bb4a1de8.1409935538.git.wking at 
tremily.us

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20140922/b07e5322/attachment.pgp>


[announce] nmhive v0.1.0, a bookmarklet/server for nmbug tags

2014-09-22 Thread W. Trevor King
On Mon, Sep 22, 2014 at 10:19:35AM -0700, W. Trevor King wrote:
> I like nmbug's distributed tag maintenance, but not everyone has
> notmuch/nmbug installed locally (yet ;).  However, everyone that I
> know does have a browser and a mail client.  They can submit
> messages with their mail client already, but we've been missing a
> way for them to help tag messages.

Ah, and the other piece to this workflow is the existing nmbug-status,
which collects the results of canned searches so folks without a local
notmuch can use the tags [1].  Folks using a local nmhive will
probably want to run their own status-genertion via a post-commit hook
in their nmhive repository.  Then their users will have their
search-results updated after each web-initiated change.  If you also
wanted them to see updates from changes to tethera's nmbug repository,
you'd probably also want a cron job that tried to fetch and merge
tethera's changes with the nmhive changes:

  -o---o---oo  tethera/master
\\
 \o  nmhive/status  (auto-generated merge for nmbug-status)
  \  /
   o---oo   nmhive/master (with web-initiated changes)

You'd want to resolve conflicts somehow, but any resolution strategy
is probably fine, since it's unlikely that we get conflicts very
often.

Cheers,
Trevor

[1]: http://nmbug.tethera.net/status/

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20140922/de3e518b/attachment.pgp>


[PATCH] lib: Simplify close and codify aborting atomic section

2014-09-22 Thread Austin Clements
Quoth W. Trevor King on Sep 22 at  9:59 am:
> On Mon, Sep 22, 2014 at 11:43:35AM -0400, Austin Clements wrote:
> > This patch simplifies notmuch_database_close to just call
> > Database::close.  This works for both read-only and read/write
> > databases, takes care of committing changes, unifies the exception
> > handling path, and codifies aborting outstanding transactions.
> 
> If we're dropping the flush call here, where will it be triggered
> instead?  We'll need to flush/commit our changes to the database at
> some point before closing.  Do clients now need an explicit
> flush/commit command (explicit, client-initiated flushes sound like a
> good idea to me).

The call to Database::close implicitly flushes/commits, as mentioned
in the comment in the patch, so there's no need for any new APIs or
client changes.  The call to Database::flush in notmuch_database_close
was entirely redundant with the call to Database::close.


[PATCH] emacs: jump: fix compile warning on emacs 23

2014-09-22 Thread Austin Clements
LGTM.  I'm a little surprised this is necessary, but whatever.

I think the eval-and-compile has to be top-level; it's certainly not
wrong for it to be top-level.  (I like the comment in the
eval-and-compile implementation: ";; Remember, it's magic.")

On Thu, 04 Sep 2014, Mark Walters  wrote:
> notmuch-jump uses window-body-width which is not defined in emacs
> 23. To get around this it does
>
> (unless (fboundp 'window-body-width)
>   ;; Compatibility for Emacs pre-24
>   (defalias 'window-body-width 'window-width))
>
> This makes sure window-body-width is defined and all should be
> well. But it seems that the byte compiler does not realise that this
> guarantees that window-body-width will be defined and so, when
> compiling with emacs 23, it gives an error
>
> In end of data:
> notmuch-jump.el:172:1:Warning: the function `window-body-width' is not known 
> to be defined.
>
> Domo and I came to following on irc: wrap the (unless (fboundp ...))
> inside eval-and-compile which ensures that both the test and the
> defalias (if needed) happen at both compile and load time.  This fixes
> the warning.
> ---
> I think Domo and I were both not completely sure whether the
> eval-and-compile should be inside or outside the (unless fboundp ..)
> or not (ie should it wrap the unless fboundp or just the defalias) nor
> whether it should be eval-and-compile or eval-when-compile.
>
> We think this is the right version but it would be good to have confirmation.
>
> I tested notmuch jump inside emacs 23 and 24 with notmuch-emacs
> compiled with emacs 23 or 24 (ie all four combinations) and it seemed to work.
>
> Best wishes
>
> Mark
>
>
>
>  emacs/notmuch-jump.el |7 ---
>  1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/emacs/notmuch-jump.el b/emacs/notmuch-jump.el
> index 5eb0949..2706b6c 100644
> --- a/emacs/notmuch-jump.el
> +++ b/emacs/notmuch-jump.el
> @@ -25,9 +25,10 @@
>  (require 'notmuch-lib)
>  (require 'notmuch-hello)
>  
> -(unless (fboundp 'window-body-width)
> -  ;; Compatibility for Emacs pre-24
> -  (defalias 'window-body-width 'window-width))
> +(eval-and-compile
> +  (unless (fboundp 'window-body-width)
> +;; Compatibility for Emacs pre-24
> +(defalias 'window-body-width 'window-width)))
>  
>  ;;;###autoload
>  (defun notmuch-jump-search ()
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] lib: Simplify close and codify aborting atomic section

2014-09-22 Thread W. Trevor King
On Mon, Sep 22, 2014 at 06:50:50PM +, Austin Clements wrote:
> Quoth W. Trevor King on Sep 22 at  9:59 am:
> > On Mon, Sep 22, 2014 at 11:43:35AM -0400, Austin Clements wrote:
> > > This patch simplifies notmuch_database_close to just call
> > > Database::close.  This works for both read-only and read/write
> > > databases, takes care of committing changes, unifies the
> > > exception handling path, and codifies aborting outstanding
> > > transactions.
> > 
> > If we're dropping the flush call here, where will it be triggered
> > instead?  We'll need to flush/commit our changes to the database
> > at some point before closing.  Do clients now need an explicit
> > flush/commit command (explicit, client-initiated flushes sound
> > like a good idea to me).
> 
> The call to Database::close implicitly flushes/commits, as mentioned
> in the comment in the patch, so there's no need for any new APIs or
> client changes.  The call to Database::flush in
> notmuch_database_close was entirely redundant with the call to
> Database::close.

Ah, I thought the implicit flush/commit was just in our code.  Since
it's also in the underlying Xapian close, then this patch looks pretty
good to me.  I'd mention Xapian's explicit close in the notmuch.h
message.  Xapain's docs say [1]:

  For a WritableDatabase, if a transaction is active it will be
  aborted, while if no transaction is active commit() will be
  implicitly called.

Cheers,
Trevor

[1]: 
http://xapian.org/docs/apidoc/html/classXapian_1_1Database.html#a59f5f8b137723dcaaabdbdccbc0cf1eb

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20140922/4cb115a4/attachment.pgp>


Do path: searches handle spaces correctly?

2014-09-22 Thread Keith Amidon
On Mon, 2014-09-22 at 15:06 +, Austin Clements wrote:
> I assume you're doing this from the command line?  Does the following
> work?
> 
> notmuch search --output=files 'path:"dir/INBOX/INBOX/Sent Items/**"'
> 
> Shell quoting and Xapian quoting interact in often confusing ways.  In
> your original command line, the single quotes suppress shell argument
> splitting, but never make it to notmuch, so notmuch sees two search
> terms "path:dir/INBOX/INBOX/Sent" and "Items/**".  In the command line
> I suggested, the single quotes play the same role, but for the entire
> query.  Because of the quoting, notmuch *does* see the inner double
> quotes, which makes the path a single term (note that Xapian only
> accepts double quotes, not single quotes).  In general, it's good to
> enclose the entire command line query in single quotes so shell
> parsing doesn't get in the way of query parsing.

Bingo.  That was it.  Thanks for clarifying how the quoting interacts. 

What do you think about adding a section to the notmuch-search-terms web
page that talks in more detail about the quoting interactions?  If there
were on obvious section about this in the man page it might have helped
me find the issue myself.  If this seems like a good idea I'll take a
crack at it later this week when I get time to research the Xapian
quoting in a bit more detail.

--- Keith