Re: [PATCH] lib/n_d_index_file: check return value from _n_m_add_filename

2021-04-18 Thread David Bremner
David Bremner  writes:

> Ignoring this return value seems like a bad idea in general, and in
> particular it has been hiding one or more bugs related to handling
> long directory names.
> ---
>
> This is not a fix for the aforementioned bugs, but it at least makes
> clear part of the problem.  The XDDIRENTRYn: terms are not checked
> for length in the same way as XDIRECTORY terms. It isn't clear the
> same hashing strategy will work, as the XDDIRECTORY terms are used to
> create lists of child directories. It may be the best we can do is
> enforce a limit on the length of path elements in trees indexed by
> notmuch.

Applied to master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] test-lib: unset XDG_CONFIG_HOME

2021-04-18 Thread David Bremner
Đoàn Trần Công Danh  writes:

> lib/open.cc:_load_key_file will only open xdg-config files in
> $XDG_CONFIG_HOME if it's defined, $HOME/.config will be considered if
> and only if XDG_CONFIG_HOME not defined.
>
> Let's unset said variable before running the test.

applied to master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: v1 deletion patches

2021-04-18 Thread David Bremner
David Bremner  writes:

> This version has a better commit message for 2/2 and portability fixes
> (thanks to Tomi) for 1/2.
>
> It obsoletes the WIP version [1]
>
> [1]: id:20210414021627.1236560-1-da...@tethera.net

Applied to master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] compat: expose canonicalize_file_name to C++

2021-04-18 Thread David Bremner
Đoàn Trần Công Danh  writes:

>
> Anyway, I see some failure in the testsuite due to:
> - *My* hostname(1) (from coreutils) doesn't understand "-f"

ah, any suggestions for a portable replacement? I guess some perl one
liner might work.

> - All emacs tests depend on dtach(1) but the
>   test_require_external_prereq dtach is missing.  Would you want
>   to see those patches for test_require_external_prereq dtach?
>   It's pretty trivial.

Sure.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] compat: expose canonicalize_file_name to C++

2021-04-17 Thread David Bremner
Đoàn Trần Công Danh  writes:

>
> However, I see that lib/open.cc uses g_key_file_get_value from GLib
> already, we may switch to g_canonicalize_file_name then?
>

Yes that could work. I think the treatment of NULL input might need some
extra care with g_canonicalize_file_name; at least my 5 minute attempt
to do a replacement causes many test failures. Do you want to make a
patch that uses g_canonicalize_file_name? The one other place we use
canonicalize_file_name (not totally coincidentally) also uses glib, so
in principle we could completely eliminate the compat function.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] ruby: fix ruby 3.1 warnings

2021-04-17 Thread David Bremner
Felipe Contreras  writes:

>   init.c:214:5: warning: ‘rb_cData’ is deprecated: by: rb_cObject.  Will be 
> removed in 3.1. [-Wdeprecated-declarations]
>
> Signed-off-by: Felipe Contreras 

Hi Felipe;

Thanks for the patch. I have a couple of questions / comments

- It doesn't apply against current master (5248f55d5f1). Can you rebase
  it?

- What version of ruby is generating those warnings? I don't see it with
  ruby 2.7. I guess that's not necessarily a blocker, as long as the
  patched version still builds and runs with older ruby.

d


signature.asc
Description: PGP signature
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] compat: expose canonicalize_file_name to C++

2021-04-17 Thread David Bremner
Đoàn Trần Công Danh  writes:

> When compat canonicalize_file_name was introduced, it was limited to
> C code only because it was used by C code only during that time.
>
> From 5ec6fd4d, (lib/open: check for split configuration when creating
> database., 2021-02-16), lib/open.cc, which is C++, relies on the
> existent of canonicalize_file_name.
>
> Let's remove the language restriction to support those platforms don't
> have canonicalize_file_name(3).

Thanks for the patch, but it seems that we need to change some other
things to make this work with C++?

I set HAVE_CANONICALIZE_FILE_NAME = 0 in Makefile.config, and with g++
10.2 on Debian I get

CXX  -g -O2 lib/database.o
In file included from /usr/include/c++/10/cstdlib:75,
 from /usr/include/c++/10/stdlib.h:36,
 from ./util/xutil.h:24,
 from lib/notmuch-private.h:34,
 from lib/database-private.h:33,
 from lib/database.cc:21:
/usr/include/stdlib.h:790:14: error: declaration of ‘char* 
canonicalize_file_name(const char*) throw ()’ has a different exception 
specifier
  790 | extern char *canonicalize_file_name (const char *__name)
  |  ^~
In file included from lib/notmuch-private.h:30,
 from lib/database-private.h:33,
 from lib/database.cc:21:
./compat/compat.h:42:1: note: from previous declaration ‘char* 
canonicalize_file_name(const char*)’
   42 | canonicalize_file_name (const char *path);
  | ^~
make: *** [Makefile.local:200: lib/database.o] Error 1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] test: put shim at end of LD_PRELOAD path

2021-04-16 Thread David Bremner
David Bremner  writes:

> Certain tools like the address-sanitizer fail if they are not the
> first LD_PRELOADed library. It does not seem to matter for our shims,
> as long as they are loaded before libnotmuch.

For what it's worth, I've applied this change to master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH v2 0/2] scaffolding for autocrypt support

2021-04-16 Thread David Bremner
Daniel Kahn Gillmor  writes:

> Hi David, all--
>
> On Sun 2021-02-21 15:21:30 +0000, David Edmondson wrote:
>> I started looking at how to add autocrypt support based on
>> https://git.sr.ht/~zge/autocrypt.
>
> Thanks for this work, i'm glad to see the interest in autocrypt!
>
> I tend to think that the autocrypt handling belongs in libnotmuch, and
> not just in the emacs frontend, so i'm a bit concerned about what we'll
> have to prune out of the emacs frontend if we do manage to land the
> features in libnotmuch itself.

FTR, I'm waiting for the two of you to come to some concensus before
proceding with these patches.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] NEWS: user visible config related changes for 0.32

2021-04-16 Thread David Bremner
David Bremner  writes:

> ---
>  NEWS | 41 +
>  1 file changed, 41 insertions(+)
>

applied to master

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 2/2] lib: directly traverse postlists in _n_message_delete

2021-04-16 Thread David Bremner
This is intended to fix the slow behaviour of "notmuch new" (and possibly
"notmuch reindex") when large numbers of files are deleted.

The underlying issue [1] seems to be the Xapian glass backend spending
a large amount of time in db.has_positions when running queries with
large-ish amounts of unflushed changes.

This commit removes two uses of Xapian queries [2], and replaces them with
an approximation of what Xapian would do after optimizing the
queries. This avoids the calls to has_positions (which are in any case
un-needed because we are only using boolean terms here).

[1] Thanks to "andres" on IRC for narrowing down the performance
bottleneck.

[2] Thanks to Olly Betts of Xapian fame for talking me a through a fix
that does not require people to update Xapian.
---
 lib/message.cc | 68 +-
 1 file changed, 39 insertions(+), 29 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 0c2eeab5..42d56acb 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1356,11 +1356,10 @@ notmuch_status_t
 _notmuch_message_delete (notmuch_message_t *message)
 {
 notmuch_status_t status;
-const char *mid, *tid, *query_string;
+const char *mid, *tid;
 notmuch_message_t *ghost;
 notmuch_private_status_t private_status;
 notmuch_database_t *notmuch;
-notmuch_query_t *query;
 unsigned int count = 0;
 bool is_ghost;
 
@@ -1382,16 +1381,33 @@ _notmuch_message_delete (notmuch_message_t *message)
 if (is_ghost)
return NOTMUCH_STATUS_SUCCESS;
 
-query_string = talloc_asprintf (message, "thread:%s", tid);
-query = notmuch_query_create (notmuch, query_string);
-if (query == NULL)
-   return NOTMUCH_STATUS_OUT_OF_MEMORY;
-status = notmuch_query_count_messages (query, );
-if (status) {
-   notmuch_query_destroy (query);
-   return status;
+/* look for a non-ghost message in the same thread */
+try {
+   Xapian::PostingIterator thread_doc, thread_doc_end;
+   Xapian::PostingIterator mail_doc, mail_doc_end;
+
+   _notmuch_database_find_doc_ids (message->notmuch, "thread", tid, 
_doc,
+   _doc_end);
+   _notmuch_database_find_doc_ids (message->notmuch, "type", "mail", 
_doc, _doc_end);
+
+   while (count == 0 &&
+  thread_doc != thread_doc_end &&
+  mail_doc != mail_doc_end) {
+   thread_doc.skip_to (*mail_doc);
+   if (thread_doc != thread_doc_end) {
+   if (*thread_doc == *mail_doc) {
+   count++;
+   } else {
+   mail_doc.skip_to (*thread_doc);
+   if (mail_doc != mail_doc_end && *thread_doc == *mail_doc)
+   count++;
+   }
+   }
+   }
+} catch (Xapian::Error ) {
+   LOG_XAPIAN_EXCEPTION (message, error);
+   return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
 }
-
 if (count > 0) {
/* reintroduce a ghost in its place because there are still
 * other active messages in this thread: */
@@ -1410,27 +1426,21 @@ _notmuch_message_delete (notmuch_message_t *message)
notmuch_message_destroy (ghost);
status = COERCE_STATUS (private_status, "Error converting to ghost 
message");
 } else {
-   /* the thread is empty; drop all ghost messages from it */
-   notmuch_messages_t *messages;
-   status = _notmuch_query_search_documents (query,
- "ghost",
- );
-   if (status == NOTMUCH_STATUS_SUCCESS) {
-   notmuch_status_t last_error = NOTMUCH_STATUS_SUCCESS;
-   while (notmuch_messages_valid (messages)) {
-   message = notmuch_messages_get (messages);
-   status = _notmuch_message_delete (message);
-   if (status) /* we'll report the last failure we see;
-* if there is more than one failure, we
-* forget about previous ones */
-   last_error = status;
-   notmuch_message_destroy (message);
-   notmuch_messages_move_to_next (messages);
+   /* the thread now contains only ghosts: delete them */
+   try {
+   Xapian::PostingIterator doc, doc_end;
+
+   _notmuch_database_find_doc_ids (message->notmuch, "thread", tid, 
, _end);
+
+   for (; doc != doc_end; doc++) {
+   message->notmuch->writable_xapian_db->delete_document (*doc);
}
-   status = last_error;
+   } catch (Xapian::Error ) {
+   LOG_XAPIAN_EXCEPTION (message, error);
+   return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
}
+
 }
-notmuch_query_destroy (query);
 return status;
 }
 
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to 

[PATCH 1/2] perf-tests: add test for removing files.

2021-04-16 Thread David Bremner
Without the proposed fix for deletion performance [1], this test is
probably unreasonably slow compared to others at the same
small/medium/large scale.

[1]: id:20210414021627.1236560-3-da...@tethera.net
---
 performance-test/T00-new.sh | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/performance-test/T00-new.sh b/performance-test/T00-new.sh
index a14dd13f..53de1c27 100755
--- a/performance-test/T00-new.sh
+++ b/performance-test/T00-new.sh
@@ -26,6 +26,16 @@ perl -nle 'rename "$_.renamed", $_' $manifest
 
 time_run "new ($count mv back)" 'notmuch new'
 
+xargs tar cf backup.tar < $manifest
+
+perl -nle 'unlink $_; unlink $_.copy' $manifest
+
+time_run "new ($count rm)" 'notmuch new'
+
+tar xf backup.tar
+
+time_run "new ($count restore)" 'notmuch new'
+
 perl -nle 'link $_, "$_.copy"' $manifest
 
 time_run "new ($count cp)" 'notmuch new'
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


v1 deletion patches

2021-04-16 Thread David Bremner
This version has a better commit message for 2/2 and portability fixes
(thanks to Tomi) for 1/2.

It obsoletes the WIP version [1]

[1]: id:20210414021627.1236560-1-da...@tethera.net
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Deletion speed improvements for notmuch-new

2021-04-13 Thread David Bremner
These are a bit rough around the edges, but I wanted to see if we
could get them into shape for the 0.32 release, since the performance
improvement for me is pretty drastic. I would appreciate testing,
since I think the performance test may be kindof best case in avoiding
long threads.

[PATCH 1/2] WIP: add performance test for removing files.
-- this needs someone (Tomi?) to fix my gnu centric tar usage

[PATCH 2/2] WIP: replace use of queries in n_m_delete with postlist

this leads to (at least) a 30x improvement in the test introduced
above with the medium corpus; I ran out of patience waiting for the
unpatched version.

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 1/2] WIP: add performance test for removing files.

2021-04-13 Thread David Bremner
No doubt this is non-portable use of tar.
---
 performance-test/T00-new.sh | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/performance-test/T00-new.sh b/performance-test/T00-new.sh
index a14dd13f..1eeac6d0 100755
--- a/performance-test/T00-new.sh
+++ b/performance-test/T00-new.sh
@@ -26,6 +26,16 @@ perl -nle 'rename "$_.renamed", $_' $manifest
 
 time_run "new ($count mv back)" 'notmuch new'
 
+tar --create --file backup.tar --files-from=$manifest
+
+perl -nle 'unlink $_; unlink $_.copy' $manifest
+
+time_run "new ($count rm)" 'notmuch new'
+
+tar --extract --file backup.tar
+
+time_run "new ($count restore)" 'notmuch new'
+
 perl -nle 'link $_, "$_.copy"' $manifest
 
 time_run "new ($count cp)" 'notmuch new'
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 2/2] WIP: replace use of queries in n_m_delete with postlist access

2021-04-13 Thread David Bremner
This improves performance because it removes some interleaving of
queries and deletions, which is causing a lot of time to be spenting
checking for position information in the glass backend
---
 lib/message.cc | 68 +-
 1 file changed, 39 insertions(+), 29 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 0c2eeab5..42d56acb 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1356,11 +1356,10 @@ notmuch_status_t
 _notmuch_message_delete (notmuch_message_t *message)
 {
 notmuch_status_t status;
-const char *mid, *tid, *query_string;
+const char *mid, *tid;
 notmuch_message_t *ghost;
 notmuch_private_status_t private_status;
 notmuch_database_t *notmuch;
-notmuch_query_t *query;
 unsigned int count = 0;
 bool is_ghost;
 
@@ -1382,16 +1381,33 @@ _notmuch_message_delete (notmuch_message_t *message)
 if (is_ghost)
return NOTMUCH_STATUS_SUCCESS;
 
-query_string = talloc_asprintf (message, "thread:%s", tid);
-query = notmuch_query_create (notmuch, query_string);
-if (query == NULL)
-   return NOTMUCH_STATUS_OUT_OF_MEMORY;
-status = notmuch_query_count_messages (query, );
-if (status) {
-   notmuch_query_destroy (query);
-   return status;
+/* look for a non-ghost message in the same thread */
+try {
+   Xapian::PostingIterator thread_doc, thread_doc_end;
+   Xapian::PostingIterator mail_doc, mail_doc_end;
+
+   _notmuch_database_find_doc_ids (message->notmuch, "thread", tid, 
_doc,
+   _doc_end);
+   _notmuch_database_find_doc_ids (message->notmuch, "type", "mail", 
_doc, _doc_end);
+
+   while (count == 0 &&
+  thread_doc != thread_doc_end &&
+  mail_doc != mail_doc_end) {
+   thread_doc.skip_to (*mail_doc);
+   if (thread_doc != thread_doc_end) {
+   if (*thread_doc == *mail_doc) {
+   count++;
+   } else {
+   mail_doc.skip_to (*thread_doc);
+   if (mail_doc != mail_doc_end && *thread_doc == *mail_doc)
+   count++;
+   }
+   }
+   }
+} catch (Xapian::Error ) {
+   LOG_XAPIAN_EXCEPTION (message, error);
+   return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
 }
-
 if (count > 0) {
/* reintroduce a ghost in its place because there are still
 * other active messages in this thread: */
@@ -1410,27 +1426,21 @@ _notmuch_message_delete (notmuch_message_t *message)
notmuch_message_destroy (ghost);
status = COERCE_STATUS (private_status, "Error converting to ghost 
message");
 } else {
-   /* the thread is empty; drop all ghost messages from it */
-   notmuch_messages_t *messages;
-   status = _notmuch_query_search_documents (query,
- "ghost",
- );
-   if (status == NOTMUCH_STATUS_SUCCESS) {
-   notmuch_status_t last_error = NOTMUCH_STATUS_SUCCESS;
-   while (notmuch_messages_valid (messages)) {
-   message = notmuch_messages_get (messages);
-   status = _notmuch_message_delete (message);
-   if (status) /* we'll report the last failure we see;
-* if there is more than one failure, we
-* forget about previous ones */
-   last_error = status;
-   notmuch_message_destroy (message);
-   notmuch_messages_move_to_next (messages);
+   /* the thread now contains only ghosts: delete them */
+   try {
+   Xapian::PostingIterator doc, doc_end;
+
+   _notmuch_database_find_doc_ids (message->notmuch, "thread", tid, 
, _end);
+
+   for (; doc != doc_end; doc++) {
+   message->notmuch->writable_xapian_db->delete_document (*doc);
}
-   status = last_error;
+   } catch (Xapian::Error ) {
+   LOG_XAPIAN_EXCEPTION (message, error);
+   return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
}
+
 }
-notmuch_query_destroy (query);
 return status;
 }
 
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: v2 convert remaining CLI to new configuration

2021-04-10 Thread David Bremner
David Bremner  writes:

> This obsoletes [1] and [2].
>
> I've been using it for a month, I think it's probably ready to
> apply to master. Since the last posted version the only difference (other than
> incorporating [2]) is that I changed the call to access to test the
> config file not to check for write access.

I applied this series to master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 06/28] lib/config: add notmuch_config_get_values_string

2021-04-06 Thread David Bremner
This is to support the less common (at least in the notmuch codebase)
case of accessing a ;-delimited list config value with an arbitrary
string key.
---
 lib/config.cc  | 14 ++
 lib/notmuch.h  | 17 -
 test/T590-libconfig.sh | 23 +++
 3 files changed, 49 insertions(+), 5 deletions(-)

diff --git a/lib/config.cc b/lib/config.cc
index 8bd3c35a..24418f98 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -261,13 +261,19 @@ _notmuch_config_load_from_database (notmuch_database_t 
*notmuch)
 notmuch_config_values_t *
 notmuch_config_get_values (notmuch_database_t *notmuch, notmuch_config_key_t 
key)
 {
-notmuch_config_values_t *values = NULL;
-bool ok = false;
-
 const char *key_str = _notmuch_config_key_to_string (key);
 
 if (! key_str)
-   goto DONE;
+   return NULL;
+
+return notmuch_config_get_values_string (notmuch, key_str);
+}
+
+notmuch_config_values_t *
+notmuch_config_get_values_string (notmuch_database_t *notmuch, const char 
*key_str)
+{
+notmuch_config_values_t *values = NULL;
+bool ok = false;
 
 values = talloc (notmuch, notmuch_config_values_t);
 if (unlikely (! values))
diff --git a/lib/notmuch.h b/lib/notmuch.h
index da556e50..dd3d06a1 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -2494,7 +2494,6 @@ notmuch_config_list_move_to_next (notmuch_config_list_t 
*config_list);
 void
 notmuch_config_list_destroy (notmuch_config_list_t *config_list);
 
-
 /**
  * Configuration keys known to libnotmuch
  */
@@ -2565,6 +2564,22 @@ notmuch_config_set (notmuch_database_t *notmuch, 
notmuch_config_key_t key, const
 notmuch_config_values_t *
 notmuch_config_get_values (notmuch_database_t *notmuch, notmuch_config_key_t 
key);
 
+/**
+ * Returns an iterator for a ';'-delimited list of configuration values
+ *
+ * These values reflect all configuration information given at the
+ * time the database was opened.
+ *
+ * @param[in] notmuch database
+ * @param[in] key configuration key
+ *
+ * @since libnotmuch 5.4 (notmuch 0.32)
+ *
+ * @retval NULL in case of error.
+ */
+notmuch_config_values_t *
+notmuch_config_get_values_string (notmuch_database_t *notmuch, const char 
*key);
+
 /**
  * Is the given 'config_values' iterator pointing at a valid element.
  *
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index dc9964cf..edbba6c4 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -255,6 +255,29 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 restore_database
 
+test_begin_subtest "notmuch_config_get_values_string"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR} ${NOTMUCH_CONFIG} %NULL%
+{
+notmuch_config_values_t *values;
+EXPECT0(notmuch_database_set_config (db, "test.list", "x;y;z"));
+for (values = notmuch_config_get_values_string (db, "test.list");
+notmuch_config_values_valid (values);
+notmuch_config_values_move_to_next (values))
+{
+ puts (notmuch_config_values_get (values));
+}
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+x
+y
+z
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+restore_database
+
 test_begin_subtest "notmuch_config_get_values (restart)"
 cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR} ${NOTMUCH_CONFIG} %NULL%
 {
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 01/28] lib/open: fix leaks calling _trial_open

2021-04-06 Thread David Bremner
_trial_open can't know if the PATH_ERROR return value will cause the
error message to be returned from the library, so it's up the caller
to clean up if not.
---
 lib/open.cc | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/lib/open.cc b/lib/open.cc
index 8f572b0b..dc191d64 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -280,6 +280,9 @@ _notmuch_choose_xapian_path (void *ctx, const char 
*database_path,
 if (status != NOTMUCH_STATUS_PATH_ERROR)
goto DONE;
 
+if (*message_ptr)
+   free (*message_ptr);
+
 notmuch_path = talloc_asprintf (ctx, "%s/.notmuch", database_path);
 status = _db_dir_exists (notmuch_path, message_ptr);
 if (status)
@@ -648,6 +651,9 @@ notmuch_database_create_with_config (const char 
*database_path,
goto DONE;
 }
 
+if (message)
+   free (message);
+
 status = _finish_open (notmuch,
   profile,
   NOTMUCH_DATABASE_MODE_READ_WRITE,
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 11/28] CLI: load merged config at top level

2021-04-06 Thread David Bremner
This paves the way for the conversion of the remaining subcommands to
the new configuration framework.
---
 notmuch-client.h |  1 +
 notmuch.c| 29 ++---
 2 files changed, 27 insertions(+), 3 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 70458629..b560e3ed 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -272,6 +272,7 @@ typedef enum {
 NOTMUCH_COMMAND_DATABASE_EARLY = 1 << 2,
 NOTMUCH_COMMAND_DATABASE_WRITE = 1 << 3,
 NOTMUCH_COMMAND_DATABASE_CREATE= 1 << 4,
+NOTMUCH_COMMAND_CONFIG_LOAD= 1 << 5,
 } notmuch_command_mode_t;
 
 notmuch_config_t *
diff --git a/notmuch.c b/notmuch.c
index f2bf9c24..742544c2 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -175,7 +175,7 @@ static command_t commands[] = {
 { "reindex", notmuch_reindex_command, NOTMUCH_COMMAND_DATABASE_EARLY |
   NOTMUCH_COMMAND_DATABASE_WRITE,
   "Re-index all messages matching the search terms." },
-{ "config", notmuch_config_command, NOTMUCH_COMMAND_CONFIG_OPEN,
+{ "config", notmuch_config_command, NOTMUCH_COMMAND_CONFIG_OPEN | 
NOTMUCH_COMMAND_CONFIG_LOAD,
   "Get or set settings in the notmuch configuration file." },
 #if WITH_EMACS
 { "emacs-mua", NULL, 0,
@@ -471,7 +471,7 @@ main (int argc, char *argv[])
 notmuch_config_t *config = NULL;
 notmuch_database_t *notmuch = NULL;
 int opt_index;
-int ret;
+int ret = EXIT_SUCCESS;
 
 notmuch_opt_desc_t options[] = {
{ .opt_string = _file_name, .name = "config" },
@@ -559,7 +559,30 @@ main (int argc, char *argv[])
return EXIT_FAILURE;
}
}
-} else {
+}
+
+if (command->mode & NOTMUCH_COMMAND_CONFIG_LOAD) {
+   char *status_string = NULL;
+   notmuch_status_t status;
+   status = notmuch_database_load_config (NULL,
+  config_file_name,
+  NULL,
+  ,
+  _string);
+   if (status) {
+   if (status_string) {
+   fputs (status_string, stderr);
+   free (status_string);
+   }
+
+   if (status == NOTMUCH_STATUS_NO_CONFIG)
+   fputs ("Try running 'notmuch setup' to create a 
configuration.", stderr);
+   goto DONE;
+   }
+
+}
+
+if (command->mode & NOTMUCH_COMMAND_CONFIG_OPEN) {
config = notmuch_config_open (local, config_file_name, command->mode);
if (! config) {
ret = EXIT_FAILURE;
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 07/28] lib/config: add config_pairs iterators

2021-04-06 Thread David Bremner
The layer of shims here seems a bit wasteful compared to just calling
the corresponding string map functions directly, but it allows control
over the API (calling with notmuch_database_t *) and flexibility for
future changes.
---
 lib/config.cc  | 45 +
 lib/notmuch.h  | 75 ++
 test/T590-libconfig.sh | 54 ++
 3 files changed, 174 insertions(+)

diff --git a/lib/config.cc b/lib/config.cc
index 24418f98..ab45ae7b 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -38,6 +38,10 @@ struct _notmuch_config_values {
 void *children; /* talloc_context */
 };
 
+struct _notmuch_config_pairs {
+notmuch_string_map_iterator_t *iter;
+};
+
 static const char *_notmuch_config_key_to_string (notmuch_config_key_t key);
 
 static int
@@ -339,6 +343,47 @@ notmuch_config_values_destroy (notmuch_config_values_t 
*values)
 talloc_free (values);
 }
 
+notmuch_config_pairs_t *
+notmuch_config_get_pairs (notmuch_database_t *notmuch,
+ const char *prefix)
+{
+notmuch_config_pairs_t *pairs = talloc (notmuch, notmuch_config_pairs_t);
+
+pairs->iter = _notmuch_string_map_iterator_create (notmuch->config, 
prefix, false);
+return pairs;
+}
+
+notmuch_bool_t
+notmuch_config_pairs_valid (notmuch_config_pairs_t *pairs)
+{
+return _notmuch_string_map_iterator_valid (pairs->iter);
+}
+
+void
+notmuch_config_pairs_move_to_next (notmuch_config_pairs_t *pairs)
+{
+_notmuch_string_map_iterator_move_to_next (pairs->iter);
+}
+
+const char *
+notmuch_config_pairs_key (notmuch_config_pairs_t *pairs)
+{
+return _notmuch_string_map_iterator_key (pairs->iter);
+}
+
+const char *
+notmuch_config_pairs_value (notmuch_config_pairs_t *pairs)
+{
+return _notmuch_string_map_iterator_value (pairs->iter);
+}
+
+void
+notmuch_config_pairs_destroy (notmuch_config_pairs_t *pairs)
+{
+_notmuch_string_map_iterator_destroy (pairs->iter);
+talloc_free (pairs);
+}
+
 notmuch_status_t
 _notmuch_config_load_from_file (notmuch_database_t *notmuch,
GKeyFile *file)
diff --git a/lib/notmuch.h b/lib/notmuch.h
index dd3d06a1..2a013be0 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -245,6 +245,7 @@ typedef struct _notmuch_directory notmuch_directory_t;
 typedef struct _notmuch_filenames notmuch_filenames_t;
 typedef struct _notmuch_config_list notmuch_config_list_t;
 typedef struct _notmuch_config_values notmuch_config_values_t;
+typedef struct _notmuch_config_pairs notmuch_config_pairs_t;
 typedef struct _notmuch_indexopts notmuch_indexopts_t;
 #endif /* __DOXYGEN__ */
 
@@ -2639,6 +2640,80 @@ notmuch_config_values_start (notmuch_config_values_t 
*values);
 void
 notmuch_config_values_destroy (notmuch_config_values_t *values);
 
+
+/**
+ * Returns an iterator for a (key, value) configuration pairs
+ *
+ * @param[in] notmuch database
+ * @param[in] prefix prefix for keys. Pass "" for all keys.
+ *
+ * @since libnotmuch 5.4 (notmuch 0.32)
+ *
+ * @retval NULL in case of error.
+ */
+notmuch_config_pairs_t *
+notmuch_config_get_pairs (notmuch_database_t *notmuch,
+ const char *prefix);
+
+/**
+ * Is the given 'config_pairs' iterator pointing at a valid element.
+ *
+ * @param[in] pairs iterator
+ *
+ * @since libnotmuch 5.4 (notmuch 0.32)
+ *
+ * @retval FALSE if passed a NULL pointer, or the iterator is exhausted.
+ *
+ */
+notmuch_bool_t
+notmuch_config_pairs_valid (notmuch_config_pairs_t *pairs);
+
+/**
+ * Move the 'config_pairs' iterator to the next element
+ *
+ * @param[in,out] pairs iterator
+ *
+ * @since libnotmuch 5.4 (notmuch 0.32)
+ *
+ */
+void
+notmuch_config_pairs_move_to_next (notmuch_config_pairs_t *pairs);
+
+/**
+ * Get the current key from the 'config_pairs' iterator
+ *
+ * @param[in] pairs iterator
+ *
+ * @since libnotmuch 5.4 (notmuch 0.32)
+ *
+ * @retval a string with the same lifetime as the iterator
+ */
+const char *
+notmuch_config_pairs_key (notmuch_config_pairs_t *pairs);
+
+/**
+ * Get the current value from the 'config_pairs' iterator
+ *
+ * @param[in] pairs iterator
+ *
+ * @since libnotmuch 5.4 (notmuch 0.32)
+ *
+ * @retval a string with the same lifetime as the iterator
+ */
+const char *
+notmuch_config_pairs_value (notmuch_config_pairs_t *pairs);
+
+/**
+ * Destroy a config_pairs iterator, along with any associated
+ * resources.
+ *
+ * @param[in,out] pairs iterator
+ *
+ * @since libnotmuch 5.4 (notmuch 0.32)
+ */
+void
+notmuch_config_pairs_destroy (notmuch_config_pairs_t *pairs);
+
 /**
  * get a configuration value from an open database as Boolean
  *
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index edbba6c4..49cc382a 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -770,5 +770,59 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 restore_database
 
+test_begin_subtest "notmuch_config_get_pairs: prefix (ndlc)"
+cat c_head2 - c_tail <<'EOF' | test_C ${MAIL_DIR} 

[PATCH 15/28] CLI/config: switch "notmuch config list" to merged config

2021-04-06 Thread David Bremner
Use the database opened at the top level rather than opening another
notmuch_database_t.

Test output changes because keys are now listed in alphabetical order,
and because a missing database is no longer an error.
---
 notmuch-config.c| 69 -
 test/T030-config.sh | 27 --
 2 files changed, 23 insertions(+), 73 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index 50008f78..ae315cb7 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -972,69 +972,22 @@ _notmuch_config_list_built_with ()
 }
 
 static int
-_list_db_config (notmuch_config_t *config)
+notmuch_config_command_list (notmuch_database_t *notmuch)
 {
-notmuch_database_t *notmuch;
-notmuch_config_list_t *list;
-
-if (notmuch_database_open (notmuch_config_get_database_path (config),
-  NOTMUCH_DATABASE_MODE_READ_ONLY, ))
-   return EXIT_FAILURE;
+notmuch_config_pairs_t *list;
 
-/* XXX Handle UUID mismatch? */
-
-
-if (print_status_database ("notmuch config", notmuch,
-  notmuch_database_get_config_list (notmuch, "", 
)))
-   return EXIT_FAILURE;
-
-for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next 
(list)) {
-   printf ("%s=%s\n", notmuch_config_list_key (list), 
notmuch_config_list_value (list));
+_notmuch_config_list_built_with ();
+for (list = notmuch_config_get_pairs (notmuch, "");
+notmuch_config_pairs_valid (list);
+notmuch_config_pairs_move_to_next (list)) {
+   const char *value = notmuch_config_pairs_value (list);
+   if (value)
+   printf ("%s=%s\n", notmuch_config_pairs_key (list), value);
 }
-notmuch_config_list_destroy (list);
-
+notmuch_config_pairs_destroy (list);
 return EXIT_SUCCESS;
 }
 
-static int
-notmuch_config_command_list (notmuch_config_t *config)
-{
-char **groups;
-size_t g, groups_length;
-
-groups = g_key_file_get_groups (config->key_file, _length);
-if (groups == NULL)
-   return 1;
-
-for (g = 0; g < groups_length; g++) {
-   char **keys;
-   size_t k, keys_length;
-
-   keys = g_key_file_get_keys (config->key_file,
-   groups[g], _length, NULL);
-   if (keys == NULL)
-   continue;
-
-   for (k = 0; k < keys_length; k++) {
-   char *value;
-
-   value = g_key_file_get_string (config->key_file,
-  groups[g], keys[k], NULL);
-   if (value != NULL) {
-   printf ("%s.%s=%s\n", groups[g], keys[k], value);
-   free (value);
-   }
-   }
-
-   g_strfreev (keys);
-}
-
-g_strfreev (groups);
-
-_notmuch_config_list_built_with ();
-return _list_db_config (config);
-}
-
 int
 notmuch_config_command (notmuch_config_t *config, notmuch_database_t *notmuch,
int argc, char *argv[])
@@ -1074,7 +1027,7 @@ notmuch_config_command (notmuch_config_t *config, 
notmuch_database_t *notmuch,
}
ret = notmuch_config_command_set (config, argv[1], argc - 2, argv + 2);
 } else if (strcmp (argv[0], "list") == 0) {
-   ret = notmuch_config_command_list (config);
+   ret = notmuch_config_command_list (notmuch);
 } else {
fprintf (stderr, "Unrecognized argument for notmuch config: %s\n",
 argv[0]);
diff --git a/test/T030-config.sh b/test/T030-config.sh
index ba29cbac..ffbd7681 100755
--- a/test/T030-config.sh
+++ b/test/T030-config.sh
@@ -46,25 +46,22 @@ notmuch config set foo.nonexistent
 test_expect_equal "$(notmuch config get foo.nonexistent)" ""
 
 test_begin_subtest "List all items"
-notmuch config list > STDOUT 2> STDERR
-printf "%s\n\n%s\n" "$(< STDOUT)" "$(< STDERR)" | notmuch_config_sanitize 
> OUTPUT
-
+notmuch config list 2>&1 | notmuch_config_sanitize > OUTPUT
 cat < EXPECTED
-database.path=MAIL_DIR
-user.name=Notmuch Test Suite
-user.primary_email=test_su...@notmuchmail.org
-user.other_email=test_suite_ot...@notmuchmail.org;test_su...@otherdomain.org
-new.tags=unread;inbox;
-new.ignore=
-search.exclude_tags=
-maildir.synchronize_flags=true
-foo.string=this is another string value
-foo.list=this;is another;list value;
 built_with.compact=something
 built_with.field_processor=something
 built_with.retry_lock=something
-
-Error: Cannot open database at MAIL_DIR/.notmuch: No such file or directory.
+database.mail_root=MAIL_DIR
+database.path=MAIL_DIR
+foo.list=this;is another;list value;
+foo.string=this is another string value
+maildir.synchronize_flags=true
+new.ignore=
+new.tags=unread;inbox;
+search.exclude_tags=
+user.name=Notmuch Test Suite
+user.other_email=test_suite_ot...@notmuchmail.org;test_su...@otherdomain.org
+user.primary_email=test_su...@notmuchmail.org
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
-- 
2.30.2
___
notmuch mailing list -- 

[PATCH 17/28] CLI/config: use notmuch_database_reopen

2021-04-06 Thread David Bremner
This allows `notmuch config` to use the common configuration
information carried by the notmuch_database_t opened at the top level.
---
 notmuch-config.c | 17 -
 test/T750-user-header.sh |  3 ++-
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index 0c618d51..2278ab2f 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -696,9 +696,8 @@ notmuch_config_command_get (notmuch_database_t *notmuch, 
char *item)
 }
 
 static int
-_set_db_config (notmuch_config_t *config, const char *key, int argc, char 
**argv)
+_set_db_config (notmuch_database_t *notmuch, const char *key, int argc, char 
**argv)
 {
-notmuch_database_t *notmuch;
 const char *val = "";
 
 if (argc > 1) {
@@ -711,12 +710,11 @@ _set_db_config (notmuch_config_t *config, const char 
*key, int argc, char **argv
val = argv[0];
 }
 
-if (notmuch_database_open (notmuch_config_get_database_path (config),
-  NOTMUCH_DATABASE_MODE_READ_WRITE, ))
+if (print_status_database ("notmuch config", notmuch,
+  notmuch_database_reopen (notmuch,
+   
NOTMUCH_DATABASE_MODE_READ_WRITE)))
return EXIT_FAILURE;
 
-/* XXX Handle UUID mismatch? */
-
 if (print_status_database ("notmuch config", notmuch,
   notmuch_database_set_config (notmuch, key, val)))
return EXIT_FAILURE;
@@ -729,7 +727,8 @@ _set_db_config (notmuch_config_t *config, const char *key, 
int argc, char **argv
 }
 
 static int
-notmuch_config_command_set (notmuch_config_t *config, char *item, int argc, 
char *argv[])
+notmuch_config_command_set (notmuch_config_t *config, notmuch_database_t 
*notmuch, char *item,
+   int argc, char *argv[])
 {
 char *group, *key;
 config_key_info_t *key_info;
@@ -744,7 +743,7 @@ notmuch_config_command_set (notmuch_config_t *config, char 
*item, int argc, char
return 1;
 
 if (key_info && key_info->in_db) {
-   return _set_db_config (config, item, argc, argv);
+   return _set_db_config (notmuch, item, argc, argv);
 }
 
 if (_item_split (item, , ))
@@ -842,7 +841,7 @@ notmuch_config_command (notmuch_config_t *config, 
notmuch_database_t *notmuch,
 "one argument.\n");
return EXIT_FAILURE;
}
-   ret = notmuch_config_command_set (config, argv[1], argc - 2, argv + 2);
+   ret = notmuch_config_command_set (config, notmuch, argv[1], argc - 2, 
argv + 2);
 } else if (strcmp (argv[0], "list") == 0) {
ret = notmuch_config_command_list (notmuch);
 } else {
diff --git a/test/T750-user-header.sh b/test/T750-user-header.sh
index b19db571..586788b3 100755
--- a/test/T750-user-header.sh
+++ b/test/T750-user-header.sh
@@ -5,7 +5,8 @@ test_description='indexing user specified headers'
 test_begin_subtest "error adding user header before initializing DB"
 notmuch config set index.header.List List-Id 2>&1 | notmuch_dir_sanitize > 
OUTPUT
 cat < EXPECTED
-Error: Cannot open database at MAIL_DIR/.notmuch: No such file or directory.
+notmuch config: Illegal argument for function
+Cannot reopen closed or nonexistent database
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 19/28] CLI/config: drop obsolete notmuch_config_get_*

2021-04-06 Thread David Bremner
These are no longer used, replaced by notmuch_config_get.
---
 notmuch-client.h |  29 
 notmuch-config.c | 121 ---
 2 files changed, 150 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 7911c20c..a1d8dfe9 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -289,70 +289,41 @@ notmuch_config_save (notmuch_config_t *config);
 bool
 notmuch_config_is_new (notmuch_config_t *config);
 
-const char *
-notmuch_config_get_database_path (notmuch_config_t *config);
-
 void
 notmuch_config_set_database_path (notmuch_config_t *config,
  const char *database_path);
 
-const char *
-notmuch_config_get_user_name (notmuch_config_t *config);
-
 void
 notmuch_config_set_user_name (notmuch_config_t *config,
  const char *user_name);
 
-const char *
-notmuch_config_get_user_primary_email (notmuch_config_t *config);
-
 void
 notmuch_config_set_user_primary_email (notmuch_config_t *config,
   const char *primary_email);
 
-const char **
-notmuch_config_get_user_other_email (notmuch_config_t *config,
-size_t *length);
-
 void
 notmuch_config_set_user_other_email (notmuch_config_t *config,
 const char *other_email[],
 size_t length);
 
-const char **
-notmuch_config_get_new_tags (notmuch_config_t *config,
-size_t *length);
 void
 notmuch_config_set_new_tags (notmuch_config_t *config,
 const char *new_tags[],
 size_t length);
 
-const char **
-notmuch_config_get_new_ignore (notmuch_config_t *config,
-  size_t *length);
-
 void
 notmuch_config_set_new_ignore (notmuch_config_t *config,
   const char *new_ignore[],
   size_t length);
 
-bool
-notmuch_config_get_maildir_synchronize_flags (notmuch_config_t *config);
-
 void
 notmuch_config_set_maildir_synchronize_flags (notmuch_config_t *config,
  bool synchronize_flags);
 
-const char **
-notmuch_config_get_search_exclude_tags (notmuch_config_t *config, size_t 
*length);
-
 void
 notmuch_config_set_search_exclude_tags (notmuch_config_t *config,
const char *list[],
size_t length);
-const char *
-_notmuch_config_get_path (notmuch_config_t *config);
-
 int
 notmuch_run_hook (notmuch_database_t *notmuch, const char *hook);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index 2278ab2f..c13f0ecf 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -327,11 +327,6 @@ notmuch_config_close (notmuch_config_t *config)
 talloc_free (config);
 }
 
-const char *
-_notmuch_config_get_path (notmuch_config_t *config)
-{
-return config->filename;
-}
 /* Save any changes made to the notmuch configuration.
  *
  * Any comments originally in the file will be preserved.
@@ -395,22 +390,6 @@ notmuch_config_is_new (notmuch_config_t *config)
 return config->is_new;
 }
 
-static const char *
-_config_get (notmuch_config_t *config, char **field,
-const char *group, const char *key)
-{
-/* read from config file and cache value, if not cached already */
-if (*field == NULL) {
-   char *value;
-   value = g_key_file_get_string (config->key_file, group, key, NULL);
-   if (value) {
-   *field = talloc_strdup (config, value);
-   free (value);
-   }
-}
-return *field;
-}
-
 static void
 _config_set (notmuch_config_t *config, char **field,
 const char *group, const char *key, const char *value)
@@ -422,38 +401,6 @@ _config_set (notmuch_config_t *config, char **field,
 *field = NULL;
 }
 
-static const char **
-_config_get_list (notmuch_config_t *config,
- const char *section, const char *key,
- const char ***outlist, size_t *list_length, size_t 
*ret_length)
-{
-assert (outlist);
-
-/* read from config file and cache value, if not cached already */
-if (*outlist == NULL) {
-
-   char **inlist = g_key_file_get_string_list (config->key_file,
-   section, key, list_length, 
NULL);
-   if (inlist) {
-   unsigned int i;
-
-   *outlist = talloc_size (config, sizeof (char *) * (*list_length + 
1));
-
-   for (i = 0; i < *list_length; i++)
-   (*outlist)[i] = talloc_strdup (*outlist, inlist[i]);
-
-   (*outlist)[i] = NULL;
-
-   g_strfreev (inlist);
-   }
-}
-
-if (ret_length)
-   *ret_length = *list_length;
-
-return *outlist;
-}
-
 static void
 _config_set_list (notmuch_config_t *config,
  const char *group, const char *key,
@@ -467,24 +414,6 @@ _config_set_list (notmuch_config_t 

[PATCH 16/28] CLI/config: migrate notmuch_config_open to new config

2021-04-06 Thread David Bremner
notmuch_config_open will be preserved in the medium term for use by
the commands that are manipulating the config file directly (config
and setup)
---
 lib/config.cc |   5 +-
 notmuch-client.h  |   2 +-
 notmuch-config.c  | 371 +-
 notmuch.c |   2 +-
 test/T030-config.sh   |   2 +-
 test/T590-libconfig.sh|  12 +-
 .../config-with-comments  |   2 -
 7 files changed, 106 insertions(+), 290 deletions(-)

diff --git a/lib/config.cc b/lib/config.cc
index 36f242f2..26280842 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -598,7 +598,7 @@ _notmuch_config_default (notmuch_database_t *notmuch, 
notmuch_config_key_t key)
 case NOTMUCH_CONFIG_EXCLUDE_TAGS:
return "";
 case NOTMUCH_CONFIG_NEW_TAGS:
-   return "inbox;unread";
+   return "unread;inbox";
 case NOTMUCH_CONFIG_SYNC_MAILDIR_FLAGS:
return "true";
 case NOTMUCH_CONFIG_USER_NAME:
@@ -615,9 +615,10 @@ _notmuch_config_default (notmuch_database_t *notmuch, 
notmuch_config_key_t key)
else
email = _get_email_from_passwd_file (notmuch);
return email;
+case NOTMUCH_CONFIG_NEW_IGNORE:
+   return "";
 case NOTMUCH_CONFIG_HOOK_DIR:
 case NOTMUCH_CONFIG_BACKUP_DIR:
-case NOTMUCH_CONFIG_NEW_IGNORE:
 case NOTMUCH_CONFIG_OTHER_EMAIL:
return NULL;
 default:
diff --git a/notmuch-client.h b/notmuch-client.h
index b560e3ed..7911c20c 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -276,7 +276,7 @@ typedef enum {
 } notmuch_command_mode_t;
 
 notmuch_config_t *
-notmuch_config_open (void *ctx,
+notmuch_config_open (notmuch_database_t *notmuch,
 const char *filename,
 notmuch_command_mode_t config_mode);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index ae315cb7..0c618d51 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -31,87 +31,88 @@ static const char toplevel_config_comment[] =
 "\n"
 " For more information about notmuch, see https://notmuchmail.org;;
 
-static const char database_config_comment[] =
-" Database configuration\n"
-"\n"
-" The only value supported here is 'path' which should be the top-level\n"
-" directory where your mail currently exists and to where mail will be\n"
-" delivered in the future. Files should be individual email messages.\n"
-" Notmuch will store its database within a sub-directory of the path\n"
-" configured here named \".notmuch\".\n";
-
-static const char new_config_comment[] =
-" Configuration for \"notmuch new\"\n"
-"\n"
-" The following options are supported here:\n"
-"\n"
-"\ttagsA list (separated by ';') of the tags that will be\n"
-"\tadded to all messages incorporated by \"notmuch new\".\n"
-"\n"
-"\tignore  A list (separated by ';') of file and directory names\n"
-"\tthat will not be searched for messages by \"notmuch new\".\n"
-"\n"
-"\tNOTE: *Every* file/directory that goes by one of those\n"
-"\tnames will be ignored, independent of its depth/location\n"
-"\tin the mail store.\n";
-
-static const char user_config_comment[] =
-" User configuration\n"
-"\n"
-" Here is where you can let notmuch know how you would like to be\n"
-" addressed. Valid settings are\n"
-"\n"
-"\tnameYour full name.\n"
-"\tprimary_email   Your primary email address.\n"
-"\tother_email A list (separated by ';') of other email addresses\n"
-"\tat which you receive email.\n"
-"\n"
-" Notmuch will use the various email addresses configured here when\n"
-" formatting replies. It will avoid including your own addresses in the\n"
-" recipient list of replies, and will set the From address based on the\n"
-" address to which the original email was addressed.\n";
-
-static const char maildir_config_comment[] =
-" Maildir compatibility configuration\n"
-"\n"
-" The following option is supported here:\n"
-"\n"
-"\tsynchronize_flags  Valid values are true and false.\n"
-"\n"
-"\tIf true, then the following maildir flags (in message filenames)\n"
-"\twill be synchronized with the corresponding notmuch tags:\n"
-"\n"
-"\t\tFlag  Tag\n"
-"\t\t  ---\n"
-"\t\tD draft\n"
-"\t\tF flagged\n"
-"\t\tP passed\n"
-"\t\tR replied\n"
-"\t\tS unread (added when 'S' flag is not present)\n"
-"\n"
-"\tThe \"notmuch new\" command will notice flag changes in filenames\n"
-"\tand update tags, while the \"notmuch tag\" and \"notmuch restore\"\n"
-"\tcommands will notice tag changes and update flags in filenames\n";
-
-static const char search_config_comment[] =
-" Search configuration\n"
-

[PATCH 20/28] CLI/config: drop cached data from notmuch_config_t

2021-04-06 Thread David Bremner
The idea is to preserve notmuch_config_t as a thin wrapper for GKeyFile.
---
 notmuch-config.c | 48 +---
 1 file changed, 9 insertions(+), 39 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index c13f0ecf..24c16833 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -118,19 +118,6 @@ struct _notmuch_config {
 char *filename;
 GKeyFile *key_file;
 bool is_new;
-
-char *database_path;
-char *user_name;
-char *user_primary_email;
-const char **user_other_email;
-size_t user_other_email_length;
-const char **new_tags;
-size_t new_tags_length;
-const char **new_ignore;
-size_t new_ignore_length;
-bool maildir_synchronize_flags;
-const char **search_exclude_tags;
-size_t search_exclude_tags_length;
 };
 
 static int
@@ -142,7 +129,6 @@ notmuch_config_destructor (notmuch_config_t *config)
 return 0;
 }
 
-
 static bool
 get_config_from_file (notmuch_config_t *config, bool create_new)
 {
@@ -275,9 +261,6 @@ notmuch_config_open (notmuch_database_t *notmuch,
 
 talloc_set_destructor (config, notmuch_config_destructor);
 
-/* non-zero defaults */
-config->maildir_synchronize_flags = true;
-
 if (filename) {
config->filename = talloc_strdup (config, filename);
 } else if ((notmuch_config_env = getenv ("NOTMUCH_CONFIG"))) {
@@ -391,48 +374,40 @@ notmuch_config_is_new (notmuch_config_t *config)
 }
 
 static void
-_config_set (notmuch_config_t *config, char **field,
+_config_set (notmuch_config_t *config,
 const char *group, const char *key, const char *value)
 {
 g_key_file_set_string (config->key_file, group, key, value);
-
-/* drop the cached value */
-talloc_free (*field);
-*field = NULL;
 }
 
 static void
 _config_set_list (notmuch_config_t *config,
  const char *group, const char *key,
  const char *list[],
- size_t length, const char ***config_var )
+ size_t length)
 {
 g_key_file_set_string_list (config->key_file, group, key, list, length);
-
-/* drop the cached value */
-talloc_free (*config_var);
-*config_var = NULL;
 }
 
 void
 notmuch_config_set_database_path (notmuch_config_t *config,
  const char *database_path)
 {
-_config_set (config, >database_path, "database", "path", 
database_path);
+_config_set (config, "database", "path", database_path);
 }
 
 void
 notmuch_config_set_user_name (notmuch_config_t *config,
  const char *user_name)
 {
-_config_set (config, >user_name, "user", "name", user_name);
+_config_set (config, "user", "name", user_name);
 }
 
 void
 notmuch_config_set_user_primary_email (notmuch_config_t *config,
   const char *primary_email)
 {
-_config_set (config, >user_primary_email, "user", "primary_email", 
primary_email);
+_config_set (config, "user", "primary_email", primary_email);
 }
 
 void
@@ -440,8 +415,7 @@ notmuch_config_set_user_other_email (notmuch_config_t 
*config,
 const char *list[],
 size_t length)
 {
-_config_set_list (config, "user", "other_email", list, length,
- &(config->user_other_email));
+_config_set_list (config, "user", "other_email", list, length);
 }
 
 void
@@ -449,8 +423,7 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 const char *list[],
 size_t length)
 {
-_config_set_list (config, "new", "tags", list, length,
- &(config->new_tags));
+_config_set_list (config, "new", "tags", list, length);
 }
 
 void
@@ -458,8 +431,7 @@ notmuch_config_set_new_ignore (notmuch_config_t *config,
   const char *list[],
   size_t length)
 {
-_config_set_list (config, "new", "ignore", list, length,
- &(config->new_ignore));
+_config_set_list (config, "new", "ignore", list, length);
 }
 
 void
@@ -467,8 +439,7 @@ notmuch_config_set_search_exclude_tags (notmuch_config_t 
*config,
const char *list[],
size_t length)
 {
-_config_set_list (config, "search", "exclude_tags", list, length,
- &(config->search_exclude_tags));
+_config_set_list (config, "search", "exclude_tags", list, length);
 }
 
 
@@ -745,5 +716,4 @@ notmuch_config_set_maildir_synchronize_flags 
(notmuch_config_t *config,
 {
 g_key_file_set_boolean (config->key_file,
"maildir", "synchronize_flags", synchronize_flags);
-config->maildir_synchronize_flags = synchronize_flags;
 }
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to 

[PATCH 13/28] test/setup: check file output instead of notmuch config list

2021-04-06 Thread David Bremner
This allows verification of comments added to new / updated config
file.
---
 test/T040-setup.sh| 17 +---
 .../config-with-comments  | 88 +++
 2 files changed, 91 insertions(+), 14 deletions(-)
 create mode 100644 test/setup.expected-output/config-with-comments

diff --git a/test/T040-setup.sh b/test/T040-setup.sh
index 5eb281d7..daeca3e4 100755
--- a/test/T040-setup.sh
+++ b/test/T040-setup.sh
@@ -10,7 +10,7 @@ Error: cannot load config file.
 Try running 'notmuch setup' to create a configuration."
 
 test_begin_subtest "Create a new config interactively"
-notmuch --config=new-notmuch-config > /dev/null < log.${test_count} <&1 | notmuch_dir_sanitize > OUTPUT
diff --git a/test/setup.expected-output/config-with-comments 
b/test/setup.expected-output/config-with-comments
new file mode 100644
index ..d9f796f2
--- /dev/null
+++ b/test/setup.expected-output/config-with-comments
@@ -0,0 +1,88 @@
+# .notmuch-config - Configuration file for the notmuch mail system
+#
+# For more information about notmuch, see https://notmuchmail.org
+
+# Database configuration
+#
+# The only value supported here is 'path' which should be the top-level
+# directory where your mail currently exists and to where mail will be
+# delivered in the future. Files should be individual email messages.
+# Notmuch will store its database within a sub-directory of the path
+# configured here named ".notmuch".
+#
+[database]
+path=/path/to/maildir
+
+# User configuration
+#
+# Here is where you can let notmuch know how you would like to be
+# addressed. Valid settings are
+#
+#  nameYour full name.
+#  primary_email   Your primary email address.
+#  other_email A list (separated by ';') of other email addresses
+#  at which you receive email.
+#
+# Notmuch will use the various email addresses configured here when
+# formatting replies. It will avoid including your own addresses in the
+# recipient list of replies, and will set the From address based on the
+# address to which the original email was addressed.
+#
+[user]
+name=Test Suite
+primary_email=test.su...@example.com
+other_email=another.su...@example.com;
+
+# Configuration for "notmuch new"
+#
+# The following options are supported here:
+#
+#  tagsA list (separated by ';') of the tags that will be
+#  added to all messages incorporated by "notmuch new".
+#
+#  ignore  A list (separated by ';') of file and directory names
+#  that will not be searched for messages by "notmuch new".
+#
+#  NOTE: *Every* file/directory that goes by one of those
+#  names will be ignored, independent of its depth/location
+#  in the mail store.
+#
+[new]
+tags=foo;bar;
+ignore=
+
+# Search configuration
+#
+# The following option is supported here:
+#
+#  exclude_tags
+#  A ;-separated list of tags that will be excluded from
+#  search results by default.  Using an excluded tag in a
+#  query will override that exclusion.
+#
+[search]
+exclude_tags=baz;
+
+# Maildir compatibility configuration
+#
+# The following option is supported here:
+#
+#  synchronize_flags  Valid values are true and false.
+#
+#  If true, then the following maildir flags (in message filenames)
+#  will be synchronized with the corresponding notmuch tags:
+#
+#  FlagTag
+#  ---
+#  D   draft
+#  F   flagged
+#  P   passed
+#  R   replied
+#  S   unread (added when 'S' flag is not present)
+#
+#  The "notmuch new" command will notice flag changes in filenames
+#  and update tags, while the "notmuch tag" and "notmuch restore"
+#  commands will notice tag changes and update flags in filenames
+#
+[maildir]
+synchronize_flags=true
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 23/28] CLI+lib: detect missing database in split configurations.

2021-04-06 Thread David Bremner
Eventually we want to do all opening of databases in the top
level (main function). This means that detection of missing databases
needs to move out of subcommands. It also requires updating the
library to use the new NO_DATABASE status code.
---
 lib/open.cc  | 24 +---
 notmuch.c| 47 +---
 test/T040-setup.sh   |  7 ++
 test/T055-path-config.sh | 16 ++
 4 files changed, 54 insertions(+), 40 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index d0493950..387e4ddb 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -209,8 +209,8 @@ _choose_database_path (void *ctx,
 }
 
 if (*database_path == NULL) {
-   *message = strdup ("Error: Cannot open a database for a NULL path.\n");
-   return NOTMUCH_STATUS_NULL_POINTER;
+   *message = strdup ("Error: could not locate database.\n");
+   return NOTMUCH_STATUS_NO_DATABASE;
 }
 
 if (*database_path[0] != '/') {
@@ -750,12 +750,12 @@ _maybe_load_config_from_database (notmuch_database_t 
*notmuch,
 char *message; /* ignored */
 
 if (_db_dir_exists (database_path, ))
-   return NOTMUCH_STATUS_SUCCESS;
+   return NOTMUCH_STATUS_NO_DATABASE;
 
 _set_database_path (notmuch, database_path);
 
 if (_notmuch_choose_xapian_path (notmuch, database_path, 
>xapian_path, ))
-   return NOTMUCH_STATUS_SUCCESS;
+   return NOTMUCH_STATUS_NO_DATABASE;
 
 (void) _finish_open (notmuch, profile, NOTMUCH_DATABASE_MODE_READ_ONLY, 
key_file, );
 
@@ -798,19 +798,27 @@ notmuch_database_load_config (const char *database_path,
 status = _choose_database_path (local, profile, key_file,
_path, , );
 switch (status) {
-/* weirdly NULL_POINTER is what is returned if we fail to find
- * a database */
-case NOTMUCH_STATUS_NULL_POINTER:
+case NOTMUCH_STATUS_NO_DATABASE:
 case NOTMUCH_STATUS_SUCCESS:
+   if (! warning)
+   warning = status;
break;
 default:
goto DONE;
 }
 
+
 if (database_path) {
status = _maybe_load_config_from_database (notmuch, key_file, 
database_path, profile);
-   if (status)
+   switch (status) {
+   case NOTMUCH_STATUS_NO_DATABASE:
+   case NOTMUCH_STATUS_SUCCESS:
+   if (! warning)
+   warning = status;
+   break;
+   default:
goto DONE;
+   }
 }
 
 if (key_file) {
diff --git a/notmuch.c b/notmuch.c
index ff9ad18b..bcb4b768 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -380,34 +380,12 @@ notmuch_command (notmuch_config_t *config,
 notmuch_database_t *notmuch,
 unused(int argc), unused(char **argv))
 {
-char *db_path;
-struct stat st;
-
 /* If the user has never configured notmuch, then run
  * notmuch_setup_command which will give a nice welcome message,
  * and interactively guide the user through the configuration. */
 if (notmuch_config_is_new (config))
return notmuch_setup_command (config, notmuch, 0, NULL);
 
-/* Notmuch is already configured, but is there a database? */
-db_path = talloc_asprintf (config, "%s/%s",
-  notmuch_config_get (notmuch, 
NOTMUCH_CONFIG_DATABASE_PATH),
-  ".notmuch");
-if (stat (db_path, )) {
-   if (errno != ENOENT) {
-   fprintf (stderr, "Error looking for notmuch database at %s: %s\n",
-db_path, strerror (errno));
-   return EXIT_FAILURE;
-   }
-   printf ("Notmuch is configured, but there's not yet a database 
at\n\n\t%s\n\n",
-   db_path);
-   printf ("You probably want to run \"notmuch new\" now to create that 
database.\n\n"
-   "Note that the first run of \"notmuch new\" can take a very 
long time\n"
-   "and that the resulting database will use roughly the same 
amount of\n"
-   "storage space as the email being indexed.\n\n");
-   return EXIT_SUCCESS;
-}
-
 printf ("Notmuch is configured and appears to have a database. 
Excellent!\n\n"
"At this point you can start exploring the functionality of notmuch 
by\n"
"using commands such as:\n\n"
@@ -576,12 +554,27 @@ main (int argc, char *argv[])
fputs ("Try running 'notmuch setup' to create a configuration.", 
stderr);
goto DONE;
}
-
-   if (status && (status != NOTMUCH_STATUS_NO_CONFIG)) {
-   if (status_string) {
-   fputs (status_string, stderr);
-   free (status_string);
+   switch (status) {
+   case NOTMUCH_STATUS_NO_CONFIG:
+   if (! (command->mode & NOTMUCH_COMMAND_CONFIG_CREATE)) {
+   fputs ("Try running 'notmuch setup' to create a 
configuration.", stderr);
+   goto DONE;
+   }
+   break;
+   case NOTMUCH_STATUS_NO_DATABASE:
+   if (! 

[PATCH 14/28] CLI/setup: switch to new configuration framework

2021-04-06 Thread David Bremner
Most of the changes are the elimination of notmuch_config_t accessor
use. We also migrate some of the diagnostics to the top level where we
are opening the files in question.
---
 notmuch-setup.c | 51 +++--
 notmuch.c   | 21 
 2 files changed, 37 insertions(+), 35 deletions(-)

diff --git a/notmuch-setup.c b/notmuch-setup.c
index 67214470..e9b81be8 100644
--- a/notmuch-setup.c
+++ b/notmuch-setup.c
@@ -88,14 +88,17 @@ welcome_message_post_setup (void)
 }
 
 static void
-print_tag_list (const char **tags, size_t tags_len)
+print_tag_list (notmuch_config_values_t *tags)
 {
-unsigned int i;
+bool first = false;
 
-for (i = 0; i < tags_len; i++) {
-   if (i != 0)
+for (;
+notmuch_config_values_valid (tags);
+notmuch_config_values_move_to_next (tags)) {
+   if (! first)
printf (" ");
-   printf ("%s", tags[i]);
+   first = false;
+   printf ("%s", notmuch_config_values_get (tags));
 }
 }
 
@@ -122,19 +125,13 @@ parse_tag_list (void *ctx, char *response)
 
 int
 notmuch_setup_command (notmuch_config_t *config,
-  unused(notmuch_database_t *notmuch),
+  notmuch_database_t *notmuch,
   int argc, char *argv[])
 {
 char *response = NULL;
 size_t response_size = 0;
-const char **old_other_emails;
-size_t old_other_emails_len;
 GPtrArray *other_emails;
-unsigned int i;
-const char **new_tags;
-size_t new_tags_len;
-const char **search_exclude_tags;
-size_t search_exclude_tags_len;
+notmuch_config_values_t *new_tags, *search_exclude_tags, *emails;
 
 #define prompt(format, ...) \
 do {\
@@ -157,26 +154,27 @@ notmuch_setup_command (notmuch_config_t *config,
 if (notmuch_config_is_new (config))
welcome_message_pre_setup ();
 
-prompt ("Your full name [%s]: ", notmuch_config_get_user_name (config));
+prompt ("Your full name [%s]: ", notmuch_config_get (notmuch, 
NOTMUCH_CONFIG_USER_NAME));
 if (strlen (response))
notmuch_config_set_user_name (config, response);
 
 prompt ("Your primary email address [%s]: ",
-   notmuch_config_get_user_primary_email (config));
+   notmuch_config_get (notmuch, NOTMUCH_CONFIG_PRIMARY_EMAIL));
 if (strlen (response))
notmuch_config_set_user_primary_email (config, response);
 
 other_emails = g_ptr_array_new ();
 
-old_other_emails = notmuch_config_get_user_other_email (config,
-   
_other_emails_len);
-for (i = 0; i < old_other_emails_len; i++) {
-   prompt ("Additional email address [%s]: ", old_other_emails[i]);
+for (emails = notmuch_config_get_values (notmuch, 
NOTMUCH_CONFIG_OTHER_EMAIL);
+notmuch_config_values_valid (emails);
+notmuch_config_values_move_to_next (emails)) {
+   const char *email = notmuch_config_values_get (emails);
+
+   prompt ("Additional email address [%s]: ", email);
if (strlen (response))
g_ptr_array_add (other_emails, talloc_strdup (config, response));
else
-   g_ptr_array_add (other_emails, talloc_strdup (config,
- old_other_emails[i]));
+   g_ptr_array_add (other_emails, talloc_strdup (config, email));
 }
 
 do {
@@ -192,7 +190,7 @@ notmuch_setup_command (notmuch_config_t *config,
 g_ptr_array_free (other_emails, true);
 
 prompt ("Top-level directory of your email archive [%s]: ",
-   notmuch_config_get_database_path (config));
+   notmuch_config_get (notmuch, NOTMUCH_CONFIG_DATABASE_PATH));
 if (strlen (response)) {
const char *absolute_path;
 
@@ -200,10 +198,10 @@ notmuch_setup_command (notmuch_config_t *config,
notmuch_config_set_database_path (config, absolute_path);
 }
 
-new_tags = notmuch_config_get_new_tags (config, _tags_len);
+new_tags = notmuch_config_get_values (notmuch, NOTMUCH_CONFIG_NEW_TAGS);
 
 printf ("Tags to apply to all new messages (separated by spaces) [");
-print_tag_list (new_tags, new_tags_len);
+print_tag_list (new_tags);
 prompt ("]: ");
 
 if (strlen (response)) {
@@ -215,11 +213,10 @@ notmuch_setup_command (notmuch_config_t *config,
g_ptr_array_free (tags, true);
 }
 
-
-search_exclude_tags = notmuch_config_get_search_exclude_tags (config, 
_exclude_tags_len);
+search_exclude_tags = notmuch_config_get_values (notmuch, 
NOTMUCH_CONFIG_EXCLUDE_TAGS);
 
 printf ("Tags to exclude when searching messages (separated by spaces) [");
-print_tag_list (search_exclude_tags, search_exclude_tags_len);
+print_tag_list (search_exclude_tags);
 prompt ("]: ");
 
 if (strlen (response)) {
diff --git a/notmuch.c b/notmuch.c
index 

[PATCH 26/28] CLI/config: remove calls to notmuch_config_open from top level

2021-04-06 Thread David Bremner
This will allow simplifying the subcommand interface.

Change the internal API to notmuch_config_open to not tie it to the
implementation of subcommands in notmuch.c.

It also fixes a previously broken test, since notmuch_config_open does
not understand the notion of the empty string as a config file name.
---
 notmuch-client.h |  3 +--
 notmuch-config.c | 28 +---
 notmuch-setup.c  | 11 ++-
 notmuch.c| 40 ++--
 test/T055-path-config.sh |  1 -
 5 files changed, 46 insertions(+), 37 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index a1d8dfe9..a5b3c518 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -267,7 +267,6 @@ json_quote_str (const void *ctx, const char *str);
 /* notmuch-config.c */
 
 typedef enum {
-NOTMUCH_COMMAND_CONFIG_OPEN= 1 << 0,
 NOTMUCH_COMMAND_CONFIG_CREATE  = 1 << 1,
 NOTMUCH_COMMAND_DATABASE_EARLY = 1 << 2,
 NOTMUCH_COMMAND_DATABASE_WRITE = 1 << 3,
@@ -278,7 +277,7 @@ typedef enum {
 notmuch_config_t *
 notmuch_config_open (notmuch_database_t *notmuch,
 const char *filename,
-notmuch_command_mode_t config_mode);
+bool create);
 
 void
 notmuch_config_close (notmuch_config_t *config);
diff --git a/notmuch-config.c b/notmuch-config.c
index c4283576..7aac8e94 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -248,7 +248,7 @@ get_config_from_file (notmuch_config_t *config, bool 
create_new)
 notmuch_config_t *
 notmuch_config_open (notmuch_database_t *notmuch,
 const char *filename,
-notmuch_command_mode_t config_mode)
+bool create)
 {
 char *notmuch_config_env = NULL;
 
@@ -272,13 +272,9 @@ notmuch_config_open (notmuch_database_t *notmuch,
 
 config->key_file = g_key_file_new ();
 
-if (config_mode & NOTMUCH_COMMAND_CONFIG_OPEN) {
-   bool create_new = (config_mode & NOTMUCH_COMMAND_CONFIG_CREATE) != 0;
-
-   if (! get_config_from_file (config, create_new)) {
-   talloc_free (config);
-   return NULL;
-   }
+if (! get_config_from_file (config, create)) {
+   talloc_free (config);
+   return NULL;
 }
 
 if (config->is_new)
@@ -582,13 +578,14 @@ _set_db_config (notmuch_database_t *notmuch, const char 
*key, int argc, char **a
 }
 
 static int
-notmuch_config_command_set (notmuch_config_t *config, notmuch_database_t 
*notmuch,
+notmuch_config_command_set (unused(notmuch_config_t *config), 
notmuch_database_t *notmuch,
int argc, char *argv[])
 {
 char *group, *key;
 config_key_info_t *key_info;
+notmuch_config_t *config;
 bool update_database = false;
-int opt_index;
+int opt_index, ret;
 char *item;
 
 notmuch_opt_desc_t options[] = {
@@ -629,6 +626,11 @@ notmuch_config_command_set (notmuch_config_t *config, 
notmuch_database_t *notmuc
 if (_item_split (item, , ))
return 1;
 
+config = notmuch_config_open (notmuch,
+ notmuch_config_path (notmuch), false);
+if (! config)
+   return 1;
+
 /* With only the name of an item, we clear it from the
  * configuration file.
  *
@@ -649,7 +651,11 @@ notmuch_config_command_set (notmuch_config_t *config, 
notmuch_database_t *notmuc
break;
 }
 
-return notmuch_config_save (config);
+ret = notmuch_config_save (config);
+
+notmuch_config_close (config);
+
+return ret;
 }
 
 static
diff --git a/notmuch-setup.c b/notmuch-setup.c
index e9b81be8..221ce934 100644
--- a/notmuch-setup.c
+++ b/notmuch-setup.c
@@ -124,7 +124,7 @@ parse_tag_list (void *ctx, char *response)
 }
 
 int
-notmuch_setup_command (notmuch_config_t *config,
+notmuch_setup_command (unused(notmuch_config_t *config),
   notmuch_database_t *notmuch,
   int argc, char *argv[])
 {
@@ -132,6 +132,7 @@ notmuch_setup_command (notmuch_config_t *config,
 size_t response_size = 0;
 GPtrArray *other_emails;
 notmuch_config_values_t *new_tags, *search_exclude_tags, *emails;
+notmuch_config_t *config;
 
 #define prompt(format, ...) \
 do {\
@@ -151,6 +152,11 @@ notmuch_setup_command (notmuch_config_t *config,
fprintf (stderr, "Warning: ignoring --uuid=%s\n",
 notmuch_requested_db_uuid);
 
+config = notmuch_config_open (notmuch,
+ notmuch_config_path (notmuch), true);
+if (! config)
+   return EXIT_FAILURE;
+
 if (notmuch_config_is_new (config))
welcome_message_pre_setup ();
 
@@ -232,6 +238,9 @@ notmuch_setup_command (notmuch_config_t *config,
 if (notmuch_config_save (config))
return EXIT_FAILURE;
 
+if (config)
+   notmuch_config_close (config);
+

[PATCH 09/28] lib/config: set default for primary user email

2021-04-06 Thread David Bremner
This is mainly copying code from the CLI into the lib. The CLI copy
will be deleted in a later commit.
---
 lib/config.cc  | 70 +++---
 test/T590-libconfig.sh |  7 ++---
 test/test-lib.sh   |  6 +++-
 3 files changed, 73 insertions(+), 10 deletions(-)

diff --git a/lib/config.cc b/lib/config.cc
index 38416632..36f242f2 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -23,6 +23,7 @@
 #include "database-private.h"
 
 #include 
+#include 
 
 static const std::string CONFIG_PREFIX = "C";
 
@@ -488,6 +489,63 @@ _get_name_from_passwd_file (void *ctx)
 return name;
 }
 
+static char *
+_get_username_from_passwd_file (void *ctx)
+{
+long pw_buf_size;
+char *pw_buf;
+struct passwd passwd, *ignored;
+char *name;
+int e;
+
+pw_buf_size = sysconf (_SC_GETPW_R_SIZE_MAX);
+if (pw_buf_size == -1) pw_buf_size = 64;
+pw_buf = (char *) talloc_zero_size (ctx, pw_buf_size);
+
+while ((e = getpwuid_r (getuid (), , pw_buf,
+   pw_buf_size, )) == ERANGE) {
+   pw_buf_size = pw_buf_size * 2;
+   pw_buf = (char *) talloc_zero_size (ctx, pw_buf_size);
+}
+
+if (e == 0)
+   name = talloc_strdup (ctx, passwd.pw_name);
+else
+   name = talloc_strdup (ctx, "");
+
+talloc_free (pw_buf);
+
+return name;
+}
+
+static const char *
+_get_email_from_passwd_file (void *ctx)
+{
+
+char hostname[256];
+struct hostent *hostent;
+const char *domainname;
+char *email;
+
+char *username = _get_username_from_passwd_file (ctx);
+
+gethostname (hostname, 256);
+hostname[255] = '\0';
+
+hostent = gethostbyname (hostname);
+if (hostent && (domainname = strchr (hostent->h_name, '.')))
+   domainname += 1;
+else
+   domainname = "(none)";
+
+email = talloc_asprintf (ctx, "%s@%s.%s",
+username, hostname, domainname);
+
+talloc_free (username);
+talloc_free (email);
+return email;
+}
+
 static const char *
 _notmuch_config_key_to_string (notmuch_config_key_t key)
 {
@@ -523,7 +581,7 @@ static const char *
 _notmuch_config_default (notmuch_database_t *notmuch, notmuch_config_key_t key)
 {
 char *path;
-const char *name;
+const char *name, *email;
 
 switch (key) {
 case NOTMUCH_CONFIG_DATABASE_PATH:
@@ -549,13 +607,17 @@ _notmuch_config_default (notmuch_database_t *notmuch, 
notmuch_config_key_t key)
name = talloc_strdup (notmuch, name);
else
name = _get_name_from_passwd_file (notmuch);
-
return name;
-   break;
+case NOTMUCH_CONFIG_PRIMARY_EMAIL:
+   email = getenv ("EMAIL");
+   if (email)
+   email = talloc_strdup (notmuch, email);
+   else
+   email = _get_email_from_passwd_file (notmuch);
+   return email;
 case NOTMUCH_CONFIG_HOOK_DIR:
 case NOTMUCH_CONFIG_BACKUP_DIR:
 case NOTMUCH_CONFIG_NEW_IGNORE:
-case NOTMUCH_CONFIG_PRIMARY_EMAIL:
 case NOTMUCH_CONFIG_OTHER_EMAIL:
return NULL;
 default:
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 2c6e726c..292778d5 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -397,7 +397,7 @@ MAIL_DIR/.notmuch/backups
 inbox;unread
 NULL
 true
-NULL
+USERNAME@FQDN
 NULL
 USER_FULL_NAME
 == stderr ==
@@ -626,9 +626,6 @@ EOF
 
 
 test_begin_subtest "notmuch_database_get_config (ndlc)"
-echo NOTMUCH_CONFIG=$NOTMUCH_CONFIG
-echo NOTMUCH_PROFILE=$NOTMUCH_PROFILE
-echo HOME=$HOME
 cat c_head2 - c_tail <<'EOF' | test_C ${MAIL_DIR} %NULL% %NULL%
 {
EXPECT0(notmuch_database_get_config (db, "test.key1", ));
@@ -742,7 +739,7 @@ MAIL_DIR/.notmuch/backups
 inbox;unread
 NULL
 true
-NULL
+USERNAME@FQDN
 NULL
 USER_FULL_NAME
 == stderr ==
diff --git a/test/test-lib.sh b/test/test-lib.sh
index fa2e401e..1f438042 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -107,6 +107,9 @@ unset GREP_OPTIONS
 # For emacsclient
 unset ALTERNATE_EDITOR
 
+# for reproducibility
+unset EMAIL
+
 add_gnupg_home ()
 {
 [ -e "${GNUPGHOME}/gpg.conf" ] && return
@@ -697,8 +700,9 @@ notmuch_built_with_sanitize ()
 notmuch_passwd_sanitize ()
 {
 local user=$(id -un)
+local fqdn=$(hostname -f)
 local full_name=$(getent passwd $user | cut -d: -f 5 | cut -d, -f1)
-sed "s/$full_name/USER_FULL_NAME/"
+sed -e "s/$user/USERNAME/" -e "s/$fqdn/FQDN/" -e 
"s/$full_name/USER_FULL_NAME/"
 }
 
 notmuch_config_sanitize ()
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 05/28] lib: provide notmuch_database_load_config

2021-04-06 Thread David Bremner
This is mainly targetted at notmuch-config.c, or similar applications
that don't necessarily need both a configuration file and a database
to exist.
---
 lib/notmuch.h  |  27 ++
 lib/open.cc|  96 ++
 test/T590-libconfig.sh | 182 -
 3 files changed, 302 insertions(+), 3 deletions(-)

diff --git a/lib/notmuch.h b/lib/notmuch.h
index f9efd79e..da556e50 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -442,6 +442,33 @@ notmuch_database_open_with_config (const char 
*database_path,
   const char *profile,
   notmuch_database_t **database,
   char **error_message);
+
+
+/**
+ * Loads configuration from config file, database, and/or defaults
+ *
+ * For description of arguments, @see notmuch_database_open_with_config
+ *
+ * @retval NOTMUCH_STATUS_SUCCESS: Successfully loaded (some) configuration.
+ *
+ * @retval NOTMUCH_STATUS_OUT_OF_MEMORY: Out of memory.
+ *
+ * @retval NOTMUCH_STATUS_FILE_ERROR: An error occurred trying to open the
+ * database or config file (such as permission denied, or file not found,
+ * etc.)
+ *
+ * @retval NOTMUCH_STATUS_XAPIAN_EXCEPTION: A Xapian exception occurred.
+ *
+ * @since libnotmuch 5.4 (notmuch 0.32)
+ */
+
+notmuch_status_t
+notmuch_database_load_config (const char *database_path,
+ const char *config_path,
+ const char *profile,
+ notmuch_database_t **database,
+ char **error_message);
+
 /**
  * Create a new notmuch database located at 'database_path', using
  * configuration in 'config_path'.
diff --git a/lib/open.cc b/lib/open.cc
index 9bb313da..6c3ef46f 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -738,3 +738,99 @@ notmuch_database_reopen (notmuch_database_t *notmuch,
 notmuch->open = true;
 return NOTMUCH_STATUS_SUCCESS;
 }
+
+notmuch_status_t
+_maybe_load_config_from_database (notmuch_database_t *notmuch,
+ GKeyFile *key_file,
+ const char *database_path,
+ const char *profile)
+{
+char *message; /* ignored */
+
+if (_db_dir_exists (database_path, ))
+   return NOTMUCH_STATUS_SUCCESS;
+
+_set_database_path (notmuch, database_path);
+
+if (_notmuch_choose_xapian_path (notmuch, database_path, 
>xapian_path, ))
+   return NOTMUCH_STATUS_SUCCESS;
+
+(void) _finish_open (notmuch, profile, NOTMUCH_DATABASE_MODE_READ_ONLY, 
key_file, );
+
+return NOTMUCH_STATUS_SUCCESS;
+}
+
+notmuch_status_t
+notmuch_database_load_config (const char *database_path,
+ const char *config_path,
+ const char *profile,
+ notmuch_database_t **database,
+ char **status_string)
+{
+notmuch_status_t status = NOTMUCH_STATUS_SUCCESS, warning = 
NOTMUCH_STATUS_SUCCESS;
+void *local = talloc_new (NULL);
+notmuch_database_t *notmuch = NULL;
+char *message = NULL;
+GKeyFile *key_file = NULL;
+bool split = false;
+
+_init_libs ();
+
+notmuch = _alloc_notmuch ();
+if (! notmuch) {
+   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+   goto DONE;
+}
+
+status = _load_key_file (config_path, profile, _file);
+switch (status) {
+case NOTMUCH_STATUS_SUCCESS:
+case NOTMUCH_STATUS_NO_CONFIG:
+   warning = status;
+   break;
+default:
+   message = strdup ("Error: cannot load config file.\n");
+   goto DONE;
+}
+
+status = _choose_database_path (local, profile, key_file,
+   _path, , );
+switch (status) {
+/* weirdly NULL_POINTER is what is returned if we fail to find
+ * a database */
+case NOTMUCH_STATUS_NULL_POINTER:
+case NOTMUCH_STATUS_SUCCESS:
+   break;
+default:
+   goto DONE;
+}
+
+if (database_path) {
+   status = _maybe_load_config_from_database (notmuch, key_file, 
database_path, profile);
+   if (status)
+   goto DONE;
+}
+
+if (key_file) {
+   status = _notmuch_config_load_from_file (notmuch, key_file);
+   if (status)
+   goto DONE;
+}
+status = _notmuch_config_load_defaults (notmuch);
+if (status)
+   goto DONE;
+
+  DONE:
+talloc_free (local);
+
+if (status_string)
+   *status_string = message;
+
+if (database)
+   *database = notmuch;
+
+if (status)
+   return status;
+else
+   return warning;
+}
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 310668a9..dc9964cf 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -538,7 +538,7 @@ cat c_head - c_tail <<'EOF' | test_C %NULL% '' %NULL%
printf("test.key2 = %s\n", val);
 }
 EOF
-NOTMUCH_CONFIG=${old_NOTMUCH_CONFIG}
+export 

[PATCH 02/28] lib: add missing status strings

2021-04-06 Thread David Bremner
---
 lib/database.cc | 8 
 1 file changed, 8 insertions(+)

diff --git a/lib/database.cc b/lib/database.cc
index 9743c1ca..d2ccdd58 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -293,12 +293,20 @@ notmuch_status_to_string (notmuch_status_t status)
return "Operation requires a database upgrade";
 case NOTMUCH_STATUS_PATH_ERROR:
return "Path supplied is illegal for this function";
+case NOTMUCH_STATUS_IGNORED:
+   return "Argument was ignored";
+case NOTMUCH_STATUS_ILLEGAL_ARGUMENT:
+   return "Illegal argument for function";
 case NOTMUCH_STATUS_MALFORMED_CRYPTO_PROTOCOL:
return "Crypto protocol missing, malformed, or unintelligible";
 case NOTMUCH_STATUS_FAILED_CRYPTO_CONTEXT_CREATION:
return "Crypto engine initialization failure";
 case NOTMUCH_STATUS_UNKNOWN_CRYPTO_PROTOCOL:
return "Unknown crypto protocol";
+case NOTMUCH_STATUS_NO_CONFIG:
+   return "No configuration file found";
+case NOTMUCH_STATUS_DATABASE_EXISTS:
+   return "Database exists, not recreated";
 default:
 case NOTMUCH_STATUS_LAST_STATUS:
return "Unknown error status value";
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 21/28] CLI/config: default to storing all config in external files

2021-04-06 Thread David Bremner
Previously the fact that some configuration options were only stored
in the database (and thus editing the config file had no effect) was a
source of user confusion. This was fixed with the series ending at
d9af0af1646.

On the other hand, the underlying partition of config options into
those stored by default in the database and those stored in the config
file remained. This is also confusing, since now some invocations of
"notmuch config set" modify the config file, and others silently
modify the database instead.

With this commit, it is up to the user to decide which configuration
to modify. A new "--database" option is provided for notmuch config to
enable modifying the configuration information in the database;
otherwise the default is to update an external config file.
---
 doc/man1/notmuch-config.rst |  6 +-
 doc/man1/notmuch.rst|  4 +++-
 notmuch-config.c| 43 ++---
 test/T030-config.sh | 20 +
 test/T600-named-queries.sh  | 17 ++-
 test/T750-user-header.sh|  8 ---
 6 files changed, 66 insertions(+), 32 deletions(-)

diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 223d2de5..32290a38 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -7,7 +7,7 @@ SYNOPSIS
 
 **notmuch** **config** **get** <*section*>.<*item*>
 
-**notmuch** **config** **set** <*section*>.<*item*> [*value* ...]
+**notmuch** **config** **set** [--database] <*section*>.<*item*> [*value* ...]
 
 **notmuch** **config** **list**
 
@@ -30,6 +30,10 @@ configuration file and corresponding database.
 If no values are provided, the specified configuration item will
 be removed from the configuration file.
 
+With the `--database` option, updates configuration metadata
+stored in the database, rather than the default (text)
+configuration file.
+
 **list**
 Every configuration item is printed to stdout, each on a separate
 line of the form::
diff --git a/doc/man1/notmuch.rst b/doc/man1/notmuch.rst
index fecfd08a..48351588 100644
--- a/doc/man1/notmuch.rst
+++ b/doc/man1/notmuch.rst
@@ -48,7 +48,9 @@ Supported global options for ``notmuch`` include
 
 ``--config=FILE``
 Specify the configuration file to use. This overrides any
-configuration file specified by ${NOTMUCH\_CONFIG}.
+configuration file specified by ${NOTMUCH\_CONFIG}. The empty
+string is a permitted and sometimes useful value of *FILE*, which
+tells ``notmuch`` to use only configuration metadata from the database.
 
 ``--uuid=HEX``
 Enforce that the database UUID (a unique identifier which persists
diff --git a/notmuch-config.c b/notmuch-config.c
index 24c16833..c4283576 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -511,16 +511,15 @@ validate_field_name (const char *str)
 
 typedef struct config_key {
 const char *name;
-bool in_db;
 bool prefix;
 bool (*validate)(const char *);
 } config_key_info_t;
 
 static struct config_key
 config_key_table[] = {
-{ "index.decrypt",   true,   false,  NULL },
-{ "index.header.",   true,   true,   validate_field_name },
-{ "query.",  true,   true,   NULL },
+{ "index.decrypt",   false,  NULL },
+{ "index.header.",   true,   validate_field_name },
+{ "query.",  true,   NULL },
 };
 
 static config_key_info_t *
@@ -583,11 +582,36 @@ _set_db_config (notmuch_database_t *notmuch, const char 
*key, int argc, char **a
 }
 
 static int
-notmuch_config_command_set (notmuch_config_t *config, notmuch_database_t 
*notmuch, char *item,
+notmuch_config_command_set (notmuch_config_t *config, notmuch_database_t 
*notmuch,
int argc, char *argv[])
 {
 char *group, *key;
 config_key_info_t *key_info;
+bool update_database = false;
+int opt_index;
+char *item;
+
+notmuch_opt_desc_t options[] = {
+   { .opt_bool = _database, .name = "database" },
+   { }
+};
+
+opt_index = parse_arguments (argc, argv, options, 1);
+if (opt_index < 0)
+   return EXIT_FAILURE;
+
+argc -= opt_index;
+argv += opt_index;
+
+if (argc < 1) {
+   fprintf (stderr, "Error: notmuch config set requires at least "
+"one argument.\n");
+   return EXIT_FAILURE;
+}
+
+item = argv[0];
+argv++;
+argc--;
 
 if (STRNCMP_LITERAL (item, BUILT_WITH_PREFIX) == 0) {
fprintf (stderr, "Error: read only option: %s\n", item);
@@ -598,7 +622,7 @@ notmuch_config_command_set (notmuch_config_t *config, 
notmuch_database_t *notmuc
 if (key_info && key_info->validate && (! key_info->validate (item)))
return 1;
 
-if (key_info && key_info->in_db) {
+if (update_database) {
return _set_db_config (notmuch, item, argc, argv);
 }
 
@@ -692,12 +716,7 @@ notmuch_config_command (notmuch_config_t *config, 
notmuch_database_t *notmuch,
}
ret = notmuch_config_command_get 

[PATCH 08/28] lib/config: set defaults for user full name

2021-04-06 Thread David Bremner
This just copies code from from the CLI into the library. New test
infrastructure is needed because apparently we have never tested this
code path.
---
 lib/config.cc  | 48 +-
 test/T590-libconfig.sh | 13 
 test/test-lib.sh   |  7 ++
 3 files changed, 63 insertions(+), 5 deletions(-)

diff --git a/lib/config.cc b/lib/config.cc
index ab45ae7b..38416632 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -22,6 +22,8 @@
 #include "notmuch-private.h"
 #include "database-private.h"
 
+#include 
+
 static const std::string CONFIG_PREFIX = "C";
 
 struct _notmuch_config_list {
@@ -451,6 +453,41 @@ notmuch_config_get_bool (notmuch_database_t *notmuch, 
notmuch_config_key_t key,
 return NOTMUCH_STATUS_SUCCESS;
 }
 
+static const char *
+_get_name_from_passwd_file (void *ctx)
+{
+long pw_buf_size;
+char *pw_buf;
+struct passwd passwd, *ignored;
+const char *name;
+int e;
+
+pw_buf_size = sysconf (_SC_GETPW_R_SIZE_MAX);
+if (pw_buf_size == -1) pw_buf_size = 64;
+pw_buf = (char *) talloc_size (ctx, pw_buf_size);
+
+while ((e = getpwuid_r (getuid (), , pw_buf,
+   pw_buf_size, )) == ERANGE) {
+   pw_buf_size = pw_buf_size * 2;
+   pw_buf = (char *) talloc_zero_size (ctx, pw_buf_size);
+}
+
+if (e == 0) {
+   char *comma = strchr (passwd.pw_gecos, ',');
+   if (comma)
+   name = talloc_strndup (ctx, passwd.pw_gecos,
+  comma - passwd.pw_gecos);
+   else
+   name = talloc_strdup (ctx, passwd.pw_gecos);
+} else {
+   name = talloc_strdup (ctx, "");
+}
+
+talloc_free (pw_buf);
+
+return name;
+}
+
 static const char *
 _notmuch_config_key_to_string (notmuch_config_key_t key)
 {
@@ -486,6 +523,7 @@ static const char *
 _notmuch_config_default (notmuch_database_t *notmuch, notmuch_config_key_t key)
 {
 char *path;
+const char *name;
 
 switch (key) {
 case NOTMUCH_CONFIG_DATABASE_PATH:
@@ -505,10 +543,18 @@ _notmuch_config_default (notmuch_database_t *notmuch, 
notmuch_config_key_t key)
return "inbox;unread";
 case NOTMUCH_CONFIG_SYNC_MAILDIR_FLAGS:
return "true";
+case NOTMUCH_CONFIG_USER_NAME:
+   name = getenv ("NAME");
+   if (name)
+   name = talloc_strdup (notmuch, name);
+   else
+   name = _get_name_from_passwd_file (notmuch);
+
+   return name;
+   break;
 case NOTMUCH_CONFIG_HOOK_DIR:
 case NOTMUCH_CONFIG_BACKUP_DIR:
 case NOTMUCH_CONFIG_NEW_IGNORE:
-case NOTMUCH_CONFIG_USER_NAME:
 case NOTMUCH_CONFIG_PRIMARY_EMAIL:
 case NOTMUCH_CONFIG_OTHER_EMAIL:
return NULL;
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 49cc382a..2c6e726c 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -384,6 +384,9 @@ cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR} '' %NULL%
 }
 }
 EOF
+
+notmuch_passwd_sanitize < OUTPUT > OUTPUT.clean
+
 cat <<'EOF' >EXPECTED
 == stdout ==
 MAIL_DIR
@@ -396,11 +399,11 @@ NULL
 true
 NULL
 NULL
-NULL
+USER_FULL_NAME
 == stderr ==
 EOF
 unset MAILDIR
-test_expect_equal_file EXPECTED OUTPUT
+test_expect_equal_file EXPECTED OUTPUT.clean
 
 backup_database
 test_begin_subtest "override config from \${NOTMUCH_CONFIG}"
@@ -727,6 +730,8 @@ cat c_head2 - c_tail <<'EOF' | test_C ${MAIL_DIR} 
/nonexistent %NULL%
 }
 }
 EOF
+
+notmuch_passwd_sanitize < OUTPUT > OUTPUT.clean
 cat <<'EOF' >EXPECTED
 == stdout ==
 MAIL_DIR
@@ -739,10 +744,10 @@ NULL
 true
 NULL
 NULL
-NULL
+USER_FULL_NAME
 == stderr ==
 EOF
-test_expect_equal_file EXPECTED OUTPUT
+test_expect_equal_file EXPECTED OUTPUT.clean
 
 backup_database
 test_begin_subtest "override config from \${HOME}/.notmuch-config (ndlc)"
diff --git a/test/test-lib.sh b/test/test-lib.sh
index fc176af8..fa2e401e 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -694,6 +694,13 @@ notmuch_built_with_sanitize ()
 sed 's/^built_with[.]\(.*\)=.*$/built_with.\1=something/'
 }
 
+notmuch_passwd_sanitize ()
+{
+local user=$(id -un)
+local full_name=$(getent passwd $user | cut -d: -f 5 | cut -d, -f1)
+sed "s/$full_name/USER_FULL_NAME/"
+}
+
 notmuch_config_sanitize ()
 {
 notmuch_dir_sanitize | notmuch_built_with_sanitize
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 27/28] CLI: drop notmuch_config_t from subcommand interface.

2021-04-06 Thread David Bremner
At this point it is unused in all subcommands.
---
 notmuch-client.h  | 42 ++
 notmuch-compact.c |  3 +--
 notmuch-config.c  |  7 +++
 notmuch-count.c   |  3 +--
 notmuch-dump.c|  3 +--
 notmuch-insert.c  |  3 +--
 notmuch-new.c |  3 +--
 notmuch-reindex.c |  3 +--
 notmuch-reply.c   |  3 +--
 notmuch-restore.c |  3 +--
 notmuch-search.c  |  6 ++
 notmuch-setup.c   |  3 +--
 notmuch-show.c|  3 +--
 notmuch-tag.c |  3 +--
 notmuch.c | 18 +++---
 15 files changed, 37 insertions(+), 69 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index a5b3c518..8815bb11 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -171,60 +171,46 @@ void
 notmuch_exit_if_unsupported_format (void);
 
 int
-notmuch_count_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-  int argc, char *argv[]);
+notmuch_count_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_dump_command (notmuch_config_t *config, notmuch_database_t *notmuch,
- int argc, char *argv[]);
+notmuch_dump_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_new_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-int argc, char *argv[]);
+notmuch_new_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_insert_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-   int argc, char *argv[]);
+notmuch_insert_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_reindex_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-int argc, char *argv[]);
+notmuch_reindex_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_reply_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-  int argc, char *argv[]);
+notmuch_reply_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_restore_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-int argc, char *argv[]);
+notmuch_restore_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_search_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-   int argc, char *argv[]);
+notmuch_search_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_address_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-int argc, char *argv[]);
+notmuch_address_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_setup_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-  int argc, char *argv[]);
+notmuch_setup_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_show_command (notmuch_config_t *config, notmuch_database_t *notmuch,
- int argc, char *argv[]);
+notmuch_show_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_tag_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-int argc, char *argv[]);
+notmuch_tag_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_config_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-   int argc, char *argv[]);
+notmuch_config_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 int
-notmuch_compact_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-int argc, char *argv[]);
+notmuch_compact_command (notmuch_database_t *notmuch, int argc, char *argv[]);
 
 const char *
 notmuch_time_relative_date (const void *ctx, time_t then);
diff --git a/notmuch-compact.c b/notmuch-compact.c
index cc45afa1..2648434e 100644
--- a/notmuch-compact.c
+++ b/notmuch-compact.c
@@ -27,8 +27,7 @@ status_update_cb (const char *msg, unused (void *closure))
 }
 
 int
-notmuch_compact_command (unused(notmuch_config_t *config), notmuch_database_t 
*notmuch,
-int argc, char *argv[])
+notmuch_compact_command (notmuch_database_t *notmuch, int argc, char *argv[])
 {
 const char *backup_path = NULL;
 notmuch_status_t ret;
diff --git a/notmuch-config.c b/notmuch-config.c
index 7aac8e94..34482b4e 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -578,7 +578,7 @@ _set_db_config (notmuch_database_t *notmuch, const char 
*key, int argc, char **a
 }
 
 static int
-notmuch_config_command_set (unused(notmuch_config_t *config), 
notmuch_database_t *notmuch,
+notmuch_config_command_set (notmuch_database_t *notmuch,
int argc, char *argv[])
 {
 char *group, *key;
@@ -691,8 +691,7 @@ notmuch_config_command_list (notmuch_database_t *notmuch)
 }
 
 int
-notmuch_config_command (notmuch_config_t *config, notmuch_database_t *notmuch,
-

[PATCH 18/28] CLI/notmuch: switch notmuch_command to notmuch_config_get

2021-04-06 Thread David Bremner
The goal at this point is to remove the dependence on
notmuch_config_get_* without breaking any existing functionality. This
is a step towards removing notmuch_config_get_* in a future commit.
---
 notmuch.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/notmuch.c b/notmuch.c
index 4132e561..ff9ad18b 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -391,7 +391,7 @@ notmuch_command (notmuch_config_t *config,
 
 /* Notmuch is already configured, but is there a database? */
 db_path = talloc_asprintf (config, "%s/%s",
-  notmuch_config_get_database_path (config),
+  notmuch_config_get (notmuch, 
NOTMUCH_CONFIG_DATABASE_PATH),
   ".notmuch");
 if (stat (db_path, )) {
if (errno != ENOENT) {
@@ -422,8 +422,8 @@ notmuch_command (notmuch_config_t *config,
"or any other interface described at https://notmuchmail.org\n\n;
"And don't forget to run \"notmuch new\" whenever new mail 
arrives.\n\n"
"Have fun, and may your inbox never have much mail.\n\n",
-   notmuch_config_get_user_name (config),
-   notmuch_config_get_user_primary_email (config));
+   notmuch_config_get (notmuch, NOTMUCH_CONFIG_USER_NAME),
+   notmuch_config_get (notmuch, NOTMUCH_CONFIG_PRIMARY_EMAIL));
 
 return EXIT_SUCCESS;
 }
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 25/28] CLI/config: support set/get with split configuration

2021-04-06 Thread David Bremner
There are two small code changes. The main one is to retrieve the
possibly updated config file name found during the database opening
call. The second change is to allow empty config file names, as
a (currently broken) way of specifying that configuration should only
be taken from the database.
---
 notmuch.c|  5 +++-
 test/T055-path-config.sh | 60 +++-
 2 files changed, 63 insertions(+), 2 deletions(-)

diff --git a/notmuch.c b/notmuch.c
index bcb4b768..61663908 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -454,7 +454,7 @@ main (int argc, char *argv[])
 int ret = EXIT_SUCCESS;
 
 notmuch_opt_desc_t options[] = {
-   { .opt_string = _file_name, .name = "config" },
+   { .opt_string = _file_name, .name = "config", .allow_empty = 
TRUE },
{ .opt_inherit = notmuch_shared_options },
{ }
 };
@@ -581,6 +581,9 @@ main (int argc, char *argv[])
 }
 
 if (command->mode & NOTMUCH_COMMAND_CONFIG_OPEN) {
+   if (! config_file_name)
+   config_file_name = notmuch_config_path (notmuch);
+
config = notmuch_config_open (notmuch, config_file_name, command->mode);
if (! config) {
ret = EXIT_FAILURE;
diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
index 0a34e67f..535c41e9 100755
--- a/test/T055-path-config.sh
+++ b/test/T055-path-config.sh
@@ -215,7 +215,65 @@ EOF
 
test_expect_equal_file EXPECTED OUTPUT
 
-restore_config
+   test_begin_subtest "Set config value ($config)"
+   name=${RANDOM}
+   value=${RANDOM}
+   notmuch config set test${test_count}.${name} ${value}
+   output=$(notmuch config get test${test_count}.${name})
+   notmuch config set test${test_count}.${name}
+   output2=$(notmuch config get test${test_count}.${name})
+   test_expect_equal "${output}+${output2}" "${value}+"
+
+   test_begin_subtest "Set config value in database ($config)"
+   name=${RANDOM}
+   value=${RANDOM}
+   notmuch config set --database test${test_count}.${name} ${value}
+   output=$(notmuch config get test${test_count}.${name})
+   notmuch config set --database test${test_count}.${name}
+   output2=$(notmuch config get test${test_count}.${name})
+   test_expect_equal "${output}+${output2}" "${value}+"
+
+   test_begin_subtest "Config list ($config)"
+   notmuch config list | notmuch_dir_sanitize | sed -e 
"s/^database.backup_dir=.*$/database.backup_dir/"  \
+   -e 
"s/^database.hook_dir=.*$/database.hook_dir/" \
+   -e 
"s/^database.path=.*$/database.path/" > OUTPUT
+   cat < EXPECTED
+built_with.compact=true
+built_with.field_processor=true
+built_with.retry_lock=true
+database.backup_dir
+database.hook_dir
+database.mail_root=MAIL_DIR
+database.path
+maildir.synchronize_flags=true
+new.ignore=
+new.tags=unread;inbox
+search.exclude_tags=
+user.name=Notmuch Test Suite
+user.other_email=test_suite_ot...@notmuchmail.org;test_su...@otherdomain.org
+user.primary_email=test_su...@notmuchmail.org
+EOF
+   test_expect_equal_file EXPECTED OUTPUT
+
+   case $config in
+   XDG*)
+  test_begin_subtest "Set shadowed config value in database ($config)"
+  test_subtest_known_broken
+  name=${RANDOM}
+  value=${RANDOM}
+  key=test${test_count}.${name}
+  notmuch config set --database ${key}  ${value}
+  notmuch config set ${key} shadow${value}
+  output=$(notmuch --config='' config get ${key})
+  notmuch config set --database ${key}
+  output2=$(notmuch --config='' config get ${key})
+  notmuch config set ${key}
+  test_expect_equal "${output}+${output2}" "${value}+"
+  ;;
+   esac
+   restore_config
+   rm -rf home/.local
+   rm -rf home/.config
 done
 
 test_done
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 28/28] CLI: rename notmuch_config_t to notmuch_conffile_t

2021-04-06 Thread David Bremner
The name is a compromise between keeping function names from getting
too long, and avoiding confusion with the newer notmuch_config library
API. Try to make clear that the residual functionality is only about
updating configuration files.
---
 notmuch-client.h | 56 
 notmuch-config.c | 84 
 notmuch-setup.c  | 38 +++---
 3 files changed, 89 insertions(+), 89 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 8815bb11..270553ad 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -156,7 +156,7 @@ chomp_newline (char *str)
  */
 extern int notmuch_format_version;
 
-typedef struct _notmuch_config notmuch_config_t;
+typedef struct _notmuch_conffile notmuch_conffile_t;
 
 /* Commands that support structured output should support the
  * following argument
@@ -260,55 +260,55 @@ typedef enum {
 NOTMUCH_COMMAND_CONFIG_LOAD= 1 << 5,
 } notmuch_command_mode_t;
 
-notmuch_config_t *
-notmuch_config_open (notmuch_database_t *notmuch,
-const char *filename,
-bool create);
+notmuch_conffile_t *
+notmuch_conffile_open (notmuch_database_t *notmuch,
+  const char *filename,
+  bool create);
 
 void
-notmuch_config_close (notmuch_config_t *config);
+notmuch_conffile_close (notmuch_conffile_t *config);
 
 int
-notmuch_config_save (notmuch_config_t *config);
+notmuch_conffile_save (notmuch_conffile_t *config);
 
 bool
-notmuch_config_is_new (notmuch_config_t *config);
+notmuch_conffile_is_new (notmuch_conffile_t *config);
 
 void
-notmuch_config_set_database_path (notmuch_config_t *config,
- const char *database_path);
+notmuch_conffile_set_database_path (notmuch_conffile_t *config,
+   const char *database_path);
 
 void
-notmuch_config_set_user_name (notmuch_config_t *config,
- const char *user_name);
+notmuch_conffile_set_user_name (notmuch_conffile_t *config,
+   const char *user_name);
 
 void
-notmuch_config_set_user_primary_email (notmuch_config_t *config,
-  const char *primary_email);
+notmuch_conffile_set_user_primary_email (notmuch_conffile_t *config,
+const char *primary_email);
 
 void
-notmuch_config_set_user_other_email (notmuch_config_t *config,
-const char *other_email[],
-size_t length);
+notmuch_conffile_set_user_other_email (notmuch_conffile_t *config,
+  const char *other_email[],
+  size_t length);
 
 void
-notmuch_config_set_new_tags (notmuch_config_t *config,
-const char *new_tags[],
-size_t length);
+notmuch_conffile_set_new_tags (notmuch_conffile_t *config,
+  const char *new_tags[],
+  size_t length);
 
 void
-notmuch_config_set_new_ignore (notmuch_config_t *config,
-  const char *new_ignore[],
-  size_t length);
+notmuch_conffile_set_new_ignore (notmuch_conffile_t *config,
+const char *new_ignore[],
+size_t length);
 
 void
-notmuch_config_set_maildir_synchronize_flags (notmuch_config_t *config,
- bool synchronize_flags);
+notmuch_conffile_set_maildir_synchronize_flags (notmuch_conffile_t *config,
+   bool synchronize_flags);
 
 void
-notmuch_config_set_search_exclude_tags (notmuch_config_t *config,
-   const char *list[],
-   size_t length);
+notmuch_conffile_set_search_exclude_tags (notmuch_conffile_t *config,
+ const char *list[],
+ size_t length);
 int
 notmuch_run_hook (notmuch_database_t *notmuch, const char *hook);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index 34482b4e..16e86916 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -114,14 +114,14 @@ struct config_group {
 },
 };
 
-struct _notmuch_config {
+struct _notmuch_conffile {
 char *filename;
 GKeyFile *key_file;
 bool is_new;
 };
 
 static int
-notmuch_config_destructor (notmuch_config_t *config)
+notmuch_conffile_destructor (notmuch_conffile_t *config)
 {
 if (config->key_file)
g_key_file_free (config->key_file);
@@ -130,7 +130,7 @@ notmuch_config_destructor (notmuch_config_t *config)
 }
 
 static bool
-get_config_from_file (notmuch_config_t *config, bool create_new)
+get_config_from_file (notmuch_conffile_t *config, bool create_new)
 {
 #define BUF_SIZE 4096
 char 

[PATCH 22/28] lib: add NOTMUCH_STATUS_NO_DATABASE

2021-04-06 Thread David Bremner
This will allow more precise return values from various open related functions.
---
 bindings/python-cffi/notmuch2/_build.py  | 1 +
 bindings/python-cffi/notmuch2/_errors.py | 3 +++
 lib/database.cc  | 2 ++
 lib/notmuch-private.h| 1 +
 lib/notmuch.h| 4 
 5 files changed, 11 insertions(+)

diff --git a/bindings/python-cffi/notmuch2/_build.py 
b/bindings/python-cffi/notmuch2/_build.py
index f67b4de6..f712b6c5 100644
--- a/bindings/python-cffi/notmuch2/_build.py
+++ b/bindings/python-cffi/notmuch2/_build.py
@@ -51,6 +51,7 @@ ffibuilder.cdef(
 NOTMUCH_STATUS_FAILED_CRYPTO_CONTEXT_CREATION,
 NOTMUCH_STATUS_UNKNOWN_CRYPTO_PROTOCOL,
 NOTMUCH_STATUS_NO_CONFIG,
+NOTMUCH_STATUS_NO_DATABASE,
 NOTMUCH_STATUS_DATABASE_EXISTS,
 NOTMUCH_STATUS_LAST_STATUS
 } notmuch_status_t;
diff --git a/bindings/python-cffi/notmuch2/_errors.py 
b/bindings/python-cffi/notmuch2/_errors.py
index 65064d4e..9301073e 100644
--- a/bindings/python-cffi/notmuch2/_errors.py
+++ b/bindings/python-cffi/notmuch2/_errors.py
@@ -52,6 +52,8 @@ class NotmuchError(Exception):
 IllegalArgumentError,
 capi.lib.NOTMUCH_STATUS_NO_CONFIG:
 NoConfigError,
+capi.lib.NOTMUCH_STATUS_NO_DATABASE:
+NoDatabaseError,
 capi.lib.NOTMUCH_STATUS_DATABASE_EXISTS:
 DatabaseExistsError,
 }
@@ -99,6 +101,7 @@ class UpgradeRequiredError(NotmuchError): pass
 class PathError(NotmuchError): pass
 class IllegalArgumentError(NotmuchError): pass
 class NoConfigError(NotmuchError): pass
+class NoDatabaseError(NotmuchError): pass
 class DatabaseExistsError(NotmuchError): pass
 
 class ObjectDestroyedError(NotmuchError):
diff --git a/lib/database.cc b/lib/database.cc
index d2ccdd58..96458f6f 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -305,6 +305,8 @@ notmuch_status_to_string (notmuch_status_t status)
return "Unknown crypto protocol";
 case NOTMUCH_STATUS_NO_CONFIG:
return "No configuration file found";
+case NOTMUCH_STATUS_NO_DATABASE:
+   return "No database found";
 case NOTMUCH_STATUS_DATABASE_EXISTS:
return "Database exists, not recreated";
 default:
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 387f0c1e..10b1b024 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -142,6 +142,7 @@ typedef enum _notmuch_private_status {
 NOTMUCH_PRIVATE_STATUS_FAILED_CRYPTO_CONTEXT_CREATION  = 
NOTMUCH_STATUS_FAILED_CRYPTO_CONTEXT_CREATION,
 NOTMUCH_PRIVATE_STATUS_UNKNOWN_CRYPTO_PROTOCOL = 
NOTMUCH_STATUS_UNKNOWN_CRYPTO_PROTOCOL,
 NOTMUCH_PRIVATE_STATUS_NO_CONFIG   = 
NOTMUCH_STATUS_NO_CONFIG,
+NOTMUCH_PRIVATE_STATUS_NO_DATABASE = 
NOTMUCH_STATUS_NO_DATABASE,
 NOTMUCH_PRIVATE_STATUS_DATABASE_EXISTS = 
NOTMUCH_STATUS_DATABASE_EXISTS,
 
 /* Then add our own private values. */
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 2a013be0..9ea05fc5 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -212,6 +212,10 @@ typedef enum _notmuch_status {
  * Unable to load a config file
  */
 NOTMUCH_STATUS_NO_CONFIG,
+/**
+ * Unable to load a database
+ */
+NOTMUCH_STATUS_NO_DATABASE,
 /**
  * Database exists, so not (re)-created
  */
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 12/28] CLI/config: use merged config for "config get"

2021-04-06 Thread David Bremner
This commit starts the conversion of notmuch-config.c
functionality (as opposed to just interface) to the new config
framework.

The change to T030-config is because of the move of the
canonicalization database paths from the notmuch_config_t accessor to
the internal function _choose_database_path.
---
 notmuch-config.c| 91 +
 test/T030-config.sh | 31 ---
 2 files changed, 36 insertions(+), 86 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index e1a583e6..50008f78 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -864,89 +864,18 @@ _config_key_info (const char *item)
 return NULL;
 }
 
-static bool
-_stored_in_db (const char *item)
-{
-config_key_info_t *info;
-
-info = _config_key_info (item);
-
-return (info && info->in_db);
-}
-
 static int
-_print_db_config (notmuch_config_t *config, const char *name)
+notmuch_config_command_get (notmuch_database_t *notmuch, char *item)
 {
-notmuch_database_t *notmuch;
-char *val;
-
-if (notmuch_database_open (notmuch_config_get_database_path (config),
-  NOTMUCH_DATABASE_MODE_READ_ONLY, ))
-   return EXIT_FAILURE;
-
-/* XXX Handle UUID mismatch? */
-
-if (print_status_database ("notmuch config", notmuch,
-  notmuch_database_get_config (notmuch, name, 
)))
-   return EXIT_FAILURE;
-
-puts (val);
+notmuch_config_values_t *list;
 
-return EXIT_SUCCESS;
-}
-
-static int
-notmuch_config_command_get (notmuch_config_t *config, char *item)
-{
-if (strcmp (item, "database.path") == 0) {
-   printf ("%s\n", notmuch_config_get_database_path (config));
-} else if (strcmp (item, "user.name") == 0) {
-   printf ("%s\n", notmuch_config_get_user_name (config));
-} else if (strcmp (item, "user.primary_email") == 0) {
-   printf ("%s\n", notmuch_config_get_user_primary_email (config));
-} else if (strcmp (item, "user.other_email") == 0) {
-   const char **other_email;
-   size_t i, length;
-
-   other_email = notmuch_config_get_user_other_email (config, );
-   for (i = 0; i < length; i++)
-   printf ("%s\n", other_email[i]);
-} else if (strcmp (item, "new.tags") == 0) {
-   const char **tags;
-   size_t i, length;
-
-   tags = notmuch_config_get_new_tags (config, );
-   for (i = 0; i < length; i++)
-   printf ("%s\n", tags[i]);
-} else if (STRNCMP_LITERAL (item, BUILT_WITH_PREFIX) == 0) {
-   printf ("%s\n",
-   notmuch_built_with (item + strlen (BUILT_WITH_PREFIX)) ? "true" 
: "false");
-} else if (_stored_in_db (item)) {
-   return _print_db_config (config, item);
-} else {
-   char **value;
-   size_t i, length;
-   char *group, *key;
-
-   if (_item_split (item, , ))
-   return 1;
-
-   value = g_key_file_get_string_list (config->key_file,
-   group, key,
-   , NULL);
-   if (value == NULL) {
-   fprintf (stderr, "Unknown configuration item: %s.%s\n",
-group, key);
-   return 1;
-   }
-
-   for (i = 0; i < length; i++)
-   printf ("%s\n", value[i]);
-
-   g_strfreev (value);
+for (list = notmuch_config_get_values_string (notmuch, item);
+notmuch_config_values_valid (list);
+notmuch_config_values_move_to_next (list)) {
+   const char *val = notmuch_config_values_get (list);
+   puts (val);
 }
-
-return 0;
+return EXIT_SUCCESS;
 }
 
 static int
@@ -1107,7 +1036,7 @@ notmuch_config_command_list (notmuch_config_t *config)
 }
 
 int
-notmuch_config_command (notmuch_config_t *config, unused(notmuch_database_t 
*notmuch),
+notmuch_config_command (notmuch_config_t *config, notmuch_database_t *notmuch,
int argc, char *argv[])
 {
 int ret;
@@ -1136,7 +1065,7 @@ notmuch_config_command (notmuch_config_t *config, 
unused(notmuch_database_t *not
 "one argument.\n");
return EXIT_FAILURE;
}
-   ret = notmuch_config_command_get (config, argv[1]);
+   ret = notmuch_config_command_get (notmuch, argv[1]);
 } else if (strcmp (argv[0], "set") == 0) {
if (argc < 2) {
fprintf (stderr, "Error: notmuch config set requires at least "
diff --git a/test/T030-config.sh b/test/T030-config.sh
index 6610bf8e..ba29cbac 100755
--- a/test/T030-config.sh
+++ b/test/T030-config.sh
@@ -7,9 +7,12 @@ test_begin_subtest "Get string value"
 test_expect_equal "$(notmuch config get user.name)" "Notmuch Test Suite"
 
 test_begin_subtest "Get list value"
-test_expect_equal "$(notmuch config get new.tags)" "\
+cat < EXPECTED
+inbox
 unread
-inbox"
+EOF
+notmuch config get new.tags | sort > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "Set string value"
 notmuch config set foo.string "this is a string 

[PATCH 10/28] lib/open: canonicalize relative path read from config file

2021-04-06 Thread David Bremner
This matches functionality in the the CLI function
notmuch_config_get_database_path, which was previously used in the CLI
code for all calls to open a database.
---
 lib/open.cc | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index 6c3ef46f..d0493950 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -196,11 +196,13 @@ _choose_database_path (void *ctx,
 if (! *database_path && key_file) {
char *path = g_key_file_get_value (key_file, "database", "path", NULL);
if (path) {
-   *database_path = talloc_strdup (ctx, path);
+   if (path[0] == '/')
+   *database_path = talloc_strdup (ctx, path);
+   else
+   *database_path = talloc_asprintf (ctx, "%s/%s", getenv 
("HOME"), path);
g_free (path);
}
 }
-
 if (! *database_path) {
*database_path = _xdg_dir (ctx, "XDG_DATA_HOME", ".local/share", 
profile);
*split = true;
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 04/28] lib/open: pull _load_key_file out of _choose_database_path

2021-04-06 Thread David Bremner
Although this increases code duplication, it also increases
flexibility in handling various combinations of missing config file
and missing database.
---
 lib/open.cc | 36 ++--
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index dc191d64..9bb313da 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -183,27 +183,18 @@ _db_dir_exists (const char *database_path, char **message)
 
 static notmuch_status_t
 _choose_database_path (void *ctx,
-  const char *config_path,
   const char *profile,
-  GKeyFile **key_file,
+  GKeyFile *key_file,
   const char **database_path,
   bool *split,
   char **message)
 {
-notmuch_status_t status;
-
-status = _load_key_file (config_path, profile, key_file);
-if (status) {
-   *message = strdup ("Error: cannot load config file.\n");
-   return status;
-}
-
 if (! *database_path) {
*database_path = getenv ("NOTMUCH_DATABASE");
 }
 
-if (! *database_path && *key_file) {
-   char *path = g_key_file_get_value (*key_file, "database", "path", NULL);
+if (! *database_path && key_file) {
+   char *path = g_key_file_get_value (key_file, "database", "path", NULL);
if (path) {
*database_path = talloc_strdup (ctx, path);
g_free (path);
@@ -500,8 +491,14 @@ notmuch_database_open_with_config (const char 
*database_path,
goto DONE;
 }
 
-if ((status = _choose_database_path (local, config_path, profile,
-_file, _path, ,
+status = _load_key_file (config_path, profile, _file);
+if (status) {
+   message = strdup ("Error: cannot load config file.\n");
+   goto DONE;
+}
+
+if ((status = _choose_database_path (local, profile, key_file,
+_path, ,
 )))
goto DONE;
 
@@ -591,11 +588,14 @@ notmuch_database_create_with_config (const char 
*database_path,
goto DONE;
 }
 
-_init_libs ();
+status = _load_key_file (config_path, profile, _file);
+if (status) {
+   message = strdup ("Error: cannot load config file.\n");
+   goto DONE;
+}
 
-if ((status = _choose_database_path (local, config_path, profile,
-_file, _path, ,
-)))
+if ((status = _choose_database_path (local, profile, key_file,
+_path, , )))
goto DONE;
 
 status = _db_dir_exists (database_path, );
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 24/28] lib: provide notmuch_config_path

2021-04-06 Thread David Bremner
Since the library searches in several locations for a config file, the
caller does not know which of these is chosen in the usual case of
passing NULL as a config file. This changes provides an API for the
caller to retrieve the name of the config file chosen. It will be
tested in a following commit.
---
 lib/config.cc  |  6 ++
 lib/database-private.h |  3 +++
 lib/notmuch.h  | 16 +++-
 lib/open.cc| 28 +---
 4 files changed, 41 insertions(+), 12 deletions(-)

diff --git a/lib/config.cc b/lib/config.cc
index 26280842..7a0da971 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -654,6 +654,12 @@ notmuch_config_get (notmuch_database_t *notmuch, 
notmuch_config_key_t key)
 return _notmuch_string_map_get (notmuch->config, 
_notmuch_config_key_to_string (key));
 }
 
+const char *
+notmuch_config_path (notmuch_database_t *notmuch)
+{
+return notmuch->config_path;
+}
+
 notmuch_status_t
 notmuch_config_set (notmuch_database_t *notmuch, notmuch_config_key_t key, 
const char *val)
 {
diff --git a/lib/database-private.h b/lib/database-private.h
index 2900382d..0d12ec1e 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -192,6 +192,9 @@ struct _notmuch_database {
 /* Path to actual database */
 const char *xapian_path;
 
+/* Path to config loaded, if any */
+const char *config_path;
+
 int atomic_nesting;
 /* true if changes have been made in this atomic section */
 bool atomic_dirty;
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 9ea05fc5..4b053932 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -454,7 +454,12 @@ notmuch_database_open_with_config (const char 
*database_path,
  *
  * For description of arguments, @see notmuch_database_open_with_config
  *
- * @retval NOTMUCH_STATUS_SUCCESS: Successfully loaded (some) configuration.
+ * @retval NOTMUCH_STATUS_SUCCESS: Successfully loaded configuration.
+ *
+ * @retval NOTMUCH_STATUS_NO_CONFIG: No config file was loaded. Not fatal.
+ *
+ * @retval NOTMUCH_STATUS_NO_DATABASE: No config information was
+ * loaded from a database. Not fatal.
  *
  * @retval NOTMUCH_STATUS_OUT_OF_MEMORY: Out of memory.
  *
@@ -2737,6 +2742,15 @@ notmuch_status_t
 notmuch_config_get_bool (notmuch_database_t *notmuch,
 notmuch_config_key_t key,
 notmuch_bool_t *val);
+
+/**
+ * return the path of the config file loaded, if any
+ *
+ * @retval NULL if no config file was loaded
+ */
+const char *
+notmuch_config_path (notmuch_database_t *notmuch);
+
 /**
  * get the current default indexing options for a given database.
  *
diff --git a/lib/open.cc b/lib/open.cc
index 387e4ddb..5d80a884 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -113,12 +113,12 @@ _choose_dir (notmuch_database_t *notmuch,
 }
 
 static notmuch_status_t
-_load_key_file (const char *path,
+_load_key_file (notmuch_database_t *notmuch,
+   const char *path,
const char *profile,
GKeyFile **key_file)
 {
 notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
-void *local = talloc_new (NULL);
 
 if (path && EMPTY_STRING (path))
goto DONE;
@@ -126,11 +126,13 @@ _load_key_file (const char *path,
 if (! path)
path = getenv ("NOTMUCH_CONFIG");
 
-if (! path) {
-   const char *dir = _xdg_dir (local, "XDG_CONFIG_HOME", ".config", 
profile);
+if (path)
+   path = talloc_strdup (notmuch, path);
+else {
+   const char *dir = _xdg_dir (notmuch, "XDG_CONFIG_HOME", ".config", 
profile);
 
if (dir) {
-   path = talloc_asprintf (local, "%s/config", dir);
+   path = talloc_asprintf (notmuch, "%s/config", dir);
if (access (path, R_OK) != 0)
path = NULL;
}
@@ -139,13 +141,13 @@ _load_key_file (const char *path,
 if (! path) {
const char *home = getenv ("HOME");
 
-   path = talloc_asprintf (local, "%s/.notmuch-config", home);
+   path = talloc_asprintf (notmuch, "%s/.notmuch-config", home);
 
if (! profile)
profile = getenv ("NOTMUCH_PROFILE");
 
if (profile)
-   path = talloc_asprintf (local, "%s.%s", path, profile);
+   path = talloc_asprintf (notmuch, "%s.%s", path, profile);
 }
 
 *key_file = g_key_file_new ();
@@ -154,7 +156,9 @@ _load_key_file (const char *path,
 }
 
   DONE:
-talloc_free (local);
+if (path)
+   notmuch->config_path = path;
+
 return status;
 }
 
@@ -232,6 +236,7 @@ _alloc_notmuch ()
 notmuch->exception_reported = false;
 notmuch->status_string = NULL;
 notmuch->writable_xapian_db = NULL;
+notmuch->config_path = NULL;
 notmuch->atomic_nesting = 0;
 notmuch->view = 1;
 return notmuch;
@@ -493,7 +498,7 @@ notmuch_database_open_with_config (const char 
*database_path,
goto DONE;
 }
 
-status = _load_key_file (config_path, profile, _file);
+status = _load_key_file 

[PATCH 03/28] test: convert random-corpus to use n_d_open_with_config

2021-04-06 Thread David Bremner
Remove one more usage of notmuch_config_get_database_path
---
 test/random-corpus.c | 13 ++---
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/test/random-corpus.c b/test/random-corpus.c
index d0354e35..7cde22c5 100644
--- a/test/random-corpus.c
+++ b/test/random-corpus.c
@@ -141,7 +141,6 @@ main (int argc, char **argv)
 void *ctx = talloc_new (NULL);
 
 const char *config_path = NULL;
-notmuch_config_t *config;
 notmuch_database_t *notmuch;
 
 int num_messages = 500;
@@ -179,12 +178,12 @@ main (int argc, char **argv)
exit (1);
 }
 
-config = notmuch_config_open (ctx, config_path, false);
-if (config == NULL)
-   return 1;
-
-if (notmuch_database_open (notmuch_config_get_database_path (config),
-  NOTMUCH_DATABASE_MODE_READ_WRITE, ))
+if (notmuch_database_open_with_config (NULL,
+  NOTMUCH_DATABASE_MODE_READ_WRITE,
+  config_path,
+  NULL,
+  ,
+  NULL))
return 1;
 
 srandom (seed);
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


v2 convert remaining CLI to new configuration

2021-04-06 Thread David Bremner
This obsoletes [1] and [2].

I've been using it for a month, I think it's probably ready to
apply to master. Since the last posted version the only difference (other than
incorporating [2]) is that I changed the call to access to test the
config file not to check for write access.

[1]: id:20210304130139.4046183-1-da...@tethera.net
[2]: id:20210327122028.526710-1-da...@tethera.net

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] lib/open: fix leaks calling _trial_open

2021-03-27 Thread David Bremner
_trial_open can't know if the PATH_ERROR return value will cause the
error message to be returned from the library, so it's up the caller
to clean up if not.
---
 lib/open.cc | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/lib/open.cc b/lib/open.cc
index 8f572b0b..dc191d64 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -280,6 +280,9 @@ _notmuch_choose_xapian_path (void *ctx, const char 
*database_path,
 if (status != NOTMUCH_STATUS_PATH_ERROR)
goto DONE;
 
+if (*message_ptr)
+   free (*message_ptr);
+
 notmuch_path = talloc_asprintf (ctx, "%s/.notmuch", database_path);
 status = _db_dir_exists (notmuch_path, message_ptr);
 if (status)
@@ -648,6 +651,9 @@ notmuch_database_create_with_config (const char 
*database_path,
goto DONE;
 }
 
+if (message)
+   free (message);
+
 status = _finish_open (notmuch,
   profile,
   NOTMUCH_DATABASE_MODE_READ_WRITE,
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Notmuch emacs string-trim

2021-03-26 Thread David Bremner
Sami M'Barek  writes:

> When invoking something like notmuch-hello-search “tag:unread” in
> emacs I get “Wrong type argument: conspiracy, “tag:unread””. I’m
> assuming this relates to the switch between notmuch-hello-trim and
> string-trim.

Hi Sami;

notmuch-hello-search is an internal function expecting to called via a
widget action, with the widget as a first argument.  Did you maybe want
the function notmuch-search?

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: v4 flexible path location

2021-03-24 Thread David Bremner
David Bremner  writes:

> Since v3 [1]
>
> - rebase against the uncrustified version (unfortunately this makes an
>   interdiff too noisy to be helpful).
> - improve (?) the documentation of database.mail_root in notmuch-config(1).
> - tweak commit messages
> - move the call to _init_libs from [10/22] to [9/22]
>
> I'm going to mark this version as ready to apply, so object now, or
> grumble later.
>

As promised, I have now applied the updated version of this series to
master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH 2/2] CLI/new: drop the write lock to run the pre-new hook.

2021-03-24 Thread David Bremner
David Bremner  writes:

> This fixes a bug reported in [1]. In principle it could be possible
> avoid one of these reopens, but it complicates the logic in main with
> respect to creating new databases.

I have applied these two patches to master

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] test: add known broken test for indexing RFC822 group names

2021-03-23 Thread David Bremner
David Bremner  writes:

> Austin Clements diagnosed this indexing problem in [1].
>
> [1]: id:20130711215207.gr2...@mit.edu

BTW, I followed Austin's suggestion in the linked message, and confirmed
that the database has no XTO terms for the test message.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] test: add known broken test for indexing RFC822 group names

2021-03-23 Thread David Bremner
Austin Clements diagnosed this indexing problem in [1].

[1]: id:20130711215207.gr2...@mit.edu
---

Hi Tomi;

Here's a test that demonstrates the bug / missing feature.


 test/T050-new.sh | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/test/T050-new.sh b/test/T050-new.sh
index 2985e24c..109ca4ef 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -339,6 +339,13 @@ test_expect_code 1 "NOTMUCH_NEW --debug 2>&1"
 
 notmuch config set new.tags $OLDCONFIG
 
+test_begin_subtest "RFC822 group names are indexed"
+test_subtest_known_broken
+generate_message [to]="undisclosed-recipients:"
+NOTMUCH_NEW > OUTPUT
+output=$(notmuch search --output=messages to:undisclosed-recipients)
+test_expect_equal "${output}" "${gen_msg_id}"
+
 test_begin_subtest "Long directory names don't cause rescan"
 test_subtest_known_broken
 printf -v name 'z%.0s' {1..234}
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Query emails sent to undisclosed-recipients

2021-03-23 Thread David Bremner
Tomi Ollila  writes:

> On Tue, Mar 23 2021, Firmin Martin wrote:
>
>> Hi,
>>
>> I have emails whose the "To" field is undisclosed recipients. In JSON:
>>
>> ```
>> "To": "undisclosed-recipients: ;"
>> ```
>>
>> I would want to tag such email as spam, but I can't query them
>> using 
>>
>> ```
>>  notmuch show --format=json to:"undisclosed-recipients: ;"
>> ```
>>
>> or any variation (regex etc.).
>>
>> This question has already been addressed in 2013 [1]. Are there any plan
>> to implement this feature or available workaround ?
>
> Tried. many things. did not work. notmuch-search-terms(7) tells
>
>  to:
>
> (so no regex syntax...)
>
> I don't know why that doesn't work. IIRC no plan, but patches welcome >;D

The (light) technical background is that regex syntax in notmuch
requires value slots, and someone (TM) would need to evaluate how much
adding a value slot for to: would cost in terms of database size / speed
of queries.

I think there's a separate question about address groups being ignored,
discussed in the linked thread.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] lib/n_d_index_file: check return value from _n_m_add_filename

2021-03-20 Thread David Bremner
Ignoring this return value seems like a bad idea in general, and in
particular it has been hiding one or more bugs related to handling
long directory names.
---

This is not a fix for the aforementioned bugs, but it at least makes
clear part of the problem.  The XDDIRENTRYn: terms are not checked
for length in the same way as XDIRECTORY terms. It isn't clear the
same hashing strategy will work, as the XDDIRECTORY terms are used to
create lists of child directories. It may be the best we can do is
enforce a limit on the length of path elements in trees indexed by
notmuch.

 lib/add-message.cc | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/add-message.cc b/lib/add-message.cc
index 485debad..0c34d318 100644
--- a/lib/add-message.cc
+++ b/lib/add-message.cc
@@ -529,7 +529,9 @@ notmuch_database_index_file (notmuch_database_t *notmuch,
goto DONE;
}
 
-   _notmuch_message_add_filename (message, filename);
+   ret = _notmuch_message_add_filename (message, filename);
+   if (ret)
+   goto DONE;
 
if (is_new || is_ghost) {
_notmuch_message_add_term (message, "type", "mail");
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: v4 flexible path location

2021-03-20 Thread David Bremner
David Bremner  writes:

> Since v3 [1]
>
> - rebase against the uncrustified version (unfortunately this makes an
>   interdiff too noisy to be helpful).
> - improve (?) the documentation of database.mail_root in notmuch-config(1).
> - tweak commit messages
> - move the call to _init_libs from [10/22] to [9/22]
>
> I'm going to mark this version as ready to apply, so object now, or
> grumble later.

Rebasing this on top of the hook fix, I noticed the following

diff --git a/lib/open.cc b/lib/open.cc
index c9390359..8f572b0b 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -508,7 +508,7 @@ notmuch_database_open_with_config (const char 
*database_path,
 
 _set_database_path (notmuch, database_path);
 
-status = _notmuch_choose_xapian_path (local, database_path,
+status = _notmuch_choose_xapian_path (notmuch, database_path,
  >xapian_path, );
 if (status)
goto DONE;

Like all good memory bugs, this was mysteriously undetected until reopen
was called multiple times in the hook fix.

This is now fixed in my git repo. I'll hold off posting another series
for now.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 2/2] CLI/new: drop the write lock to run the pre-new hook.

2021-03-18 Thread David Bremner
This fixes a bug reported in [1]. In principle it could be possible
avoid one of these reopens, but it complicates the logic in main with
respect to creating new databases.

[1]: id:9c1993df-84bd-4199-a9c8-bada98498...@bubblegen.co.uk
---
 notmuch-new.c  | 10 ++
 test/T400-hooks.sh |  1 -
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index 223d68bb..8214fb23 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -1170,9 +1170,19 @@ notmuch_new_command (unused(notmuch_config_t *config), 
notmuch_database_t *notmu
 }
 
 if (hooks) {
+   /* Drop write lock to run hook */
+   status = notmuch_database_reopen (notmuch, 
NOTMUCH_DATABASE_MODE_READ_ONLY);
+   if (print_status_database ("notmuch new", notmuch, status))
+   return EXIT_FAILURE;
+
ret = notmuch_run_hook (notmuch, "pre-new");
if (ret)
return EXIT_FAILURE;
+
+   /* acquire write lock again */
+   status = notmuch_database_reopen (notmuch, 
NOTMUCH_DATABASE_MODE_READ_WRITE);
+   if (print_status_database ("notmuch new", notmuch, status))
+   return EXIT_FAILURE;
 }
 
 notmuch_exit_if_unmatched_db_uuid (notmuch);
diff --git a/test/T400-hooks.sh b/test/T400-hooks.sh
index a2b67d59..de8e4ba4 100755
--- a/test/T400-hooks.sh
+++ b/test/T400-hooks.sh
@@ -159,7 +159,6 @@ EOF
 test_expect_equal_file write.expected write.output
 
 test_begin_subtest "pre-new with write access [${config}]"
-test_subtest_known_broken
 rm -rf ${HOOK_DIR}
 create_write_hook "pre-new" write.expected write.output $HOOK_DIR
 NOTMUCH_NEW
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 1/2] test: Add tests for write access to database from hooks.

2021-03-18 Thread David Bremner
Recent changes to configuration handling meant the pre-new hook was
run while the database was open read only, limiting what could be done
in the hook. Add some known broken tests for this problem, as well as
a regression test for write access from the post-new hook.
---
 test/T400-hooks.sh | 28 
 test/test-lib.sh   |  1 +
 2 files changed, 29 insertions(+)

diff --git a/test/T400-hooks.sh b/test/T400-hooks.sh
index a3dd4c63..a2b67d59 100755
--- a/test/T400-hooks.sh
+++ b/test/T400-hooks.sh
@@ -2,6 +2,8 @@
 test_description='hooks'
 . $(dirname "$0")/test-lib.sh || exit 1
 
+test_require_external_prereq xapian-delve
+
 create_echo_hook () {
 local TOKEN="${RANDOM}"
 mkdir -p ${HOOK_DIR}
@@ -13,6 +15,19 @@ EOF
 echo "${TOKEN}" > ${2}
 }
 
+create_write_hook () {
+local TOKEN="${RANDOM}"
+mkdir -p ${HOOK_DIR}
+cat <"${HOOK_DIR}/${1}"
+#!/bin/sh
+if xapian-delve ${MAIL_DIR}/.notmuch/xapian | grep -q "writing = false"; then
+   echo "${TOKEN}" > ${3}
+fi
+EOF
+chmod +x "${HOOK_DIR}/${1}"
+echo "${TOKEN}" > ${2}
+}
+
 create_failing_hook () {
 local HOOK_DIR=${2}
 mkdir -p ${HOOK_DIR}
@@ -137,6 +152,19 @@ EOF
 chmod +x "${HOOK_DIR}/pre-new"
 test_expect_code 1 "notmuch new"
 
+test_begin_subtest "post-new with write access [${config}]"
+rm -rf ${HOOK_DIR}
+create_write_hook "post-new" write.expected write.output $HOOK_DIR
+NOTMUCH_NEW
+test_expect_equal_file write.expected write.output
+
+test_begin_subtest "pre-new with write access [${config}]"
+test_subtest_known_broken
+rm -rf ${HOOK_DIR}
+create_write_hook "pre-new" write.expected write.output $HOOK_DIR
+NOTMUCH_NEW
+test_expect_equal_file write.expected write.output
+
 rm -rf ${HOOK_DIR}
 done
 test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 29baa0c1..fc176af8 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -1271,3 +1271,4 @@ test_declare_external_prereq openssl
 test_declare_external_prereq gpgsm
 test_declare_external_prereq ${NOTMUCH_PYTHON}
 test_declare_external_prereq xapian-metadata
+test_declare_external_prereq xapian-delve
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH 04/22] lib: support reopening databases for write access.

2021-03-18 Thread David Bremner
David Bremner  writes:

> In the future Xapian will apparently support this more conveniently
> for the cases other than READ_ONLY => READ_ONLY
>
> Conceptually this function seems to fit better in lib/open.cc;
> database.cc is still large enough that moving the function makes
> sense.

I have applied the first 4 patches of the series to master, as I need
them for a bug fix for pre-new hooks.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] test: add known broken test for long directory bug

2021-03-18 Thread David Bremner
Tomi Ollila  writes:

> On Wed, Mar 17 2021, David Bremner wrote:
>
> could do printf -v name 'z%.0s' {1..234}
>
>> +generate_message [dir]=$name
>> +NOTMUCH_NEW  > OUTPUT
>> +notmuch new  >> OUTPUT
>
> 2 spaces in lines above
>
> apart from those 2 spaces lgtm.
>
> Tomi

Thanks. Pushed with those changes.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Database locked when running hooks

2021-03-18 Thread David Bremner
Matthew Lear  writes:
>
> The problem can be reproduced by creating $DATABASEDIR/.notmuch/hooks/pre-new 
> and having it perform a notmuch CLI command to write to the DB, such as apply 
> a tag.
>
> I described the problem in https://github.com/gauteh/lieer/issues/195 
> 
>
> Assuming the database should be able to be opened for writing in the context 
> of the pre-new (and post-new) hooks, this seems like a bug.

Yes, this is a bug I introduced recently, thanks for bringing that to my
attention. It should be just the pre-new hook which is (incorrectly) run
while the database is locked, the post-new should continue to work as
before.  It should be fairly easy to fix, I just need to initially open
the database read only, then re-open it for read/write after running the
hook.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: bug: chokes on long directory names (was: Re: out of memory on idle machine)

2021-03-17 Thread David Bremner
Gregor Zattler  writes:

> Hi David, Olly, notmuch and xapian developers,
> * David Bremner  [11. Feb. 2021]:
>> David Bremner  writes:
>> As a kind of desperation move, you could try bisecting your mailstore,
>> to see how small of a set of messages you can duplicate the problem
>> with.
>
> this I did, somehow.  I found the culprit: It's a maildir
> with one single mail in it.  The name of the maildir is
> exceptionally long [because generated from a List-Id:
> -Header] and the mail arrived at the very day, my notmuch
> database corrupted.  This maildir alone provokes that every
> next notmuch new will rescan all (?) files.

Hi Gregor;

I am very impressed with your persistence. I suspect it is a bug in
notmuch. I don't know all the details yet, but in the normal case the
directory name is added to the database prefixed with XDIRECTORY. I
noticed this isn't happening in the case of directories 234 or
longer. That is roughly the Xapian term limit of 245 characters in
total. I'm not sure why the discrepency of one character, but the main
point is that notmuch is probably improperly ignoring an error from
Xapian when adding these overlong terms.

Thanks again for the debugging, I suspect would have never found this
bug on my own.

David
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] test: add known broken test for long directory bug

2021-03-17 Thread David Bremner
In [1] Gregor Zattler explained the results of his hard working
tracking down a bug in notmuch with long directories. This test
duplicates the bug.

[1]: id:20210317194728.GB5561@no.workgroup
---
 test/T050-new.sh | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/test/T050-new.sh b/test/T050-new.sh
index 76bda959..f84dc2b0 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -339,6 +339,20 @@ test_expect_code 1 "NOTMUCH_NEW --debug 2>&1"
 
 notmuch config set new.tags $OLDCONFIG
 
+test_begin_subtest "Long directory names don't cause rescan"
+test_subtest_known_broken
+name=$(printf 'z%.0s' {1..234})
+generate_message [dir]=$name
+NOTMUCH_NEW  > OUTPUT
+notmuch new  >> OUTPUT
+rm -r ${MAIL_DIR}/${name}
+notmuch new >> OUTPUT
+cat < EXPECTED
+Added 1 new message to the database.
+No new mail.
+No new mail. Removed 1 message.
+EOF
+test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "Xapian exception: read only files"
 chmod u-w ${MAIL_DIR}/.notmuch/xapian/*.*
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 16/22] CLI/new: support split database and mail location

2021-03-14 Thread David Bremner
This adds new state variable for the mail root, and uses it most
places db_path was used. The notable exception is dumps from
backups. The latter will be dealt with properly in a future commit.
---
 notmuch-new.c| 54 ++--
 test/T055-path-config.sh | 41 ++
 2 files changed, 76 insertions(+), 19 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index 223d68bb..30062512 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -43,6 +43,7 @@ enum verbosity {
 
 typedef struct {
 const char *db_path;
+const char *mail_root;
 
 int output_is_a_tty;
 enum verbosity verbosity;
@@ -307,18 +308,18 @@ _setup_ignore (notmuch_database_t *notmuch, 
add_files_state_t *state)
 }
 
 static char *
-_get_relative_path (const char *db_path, const char *dirpath, const char 
*entry)
+_get_relative_path (const char *mail_root, const char *dirpath, const char 
*entry)
 {
-size_t db_path_len = strlen (db_path);
+size_t mail_root_len = strlen (mail_root);
 
 /* paranoia? */
-if (strncmp (dirpath, db_path, db_path_len) != 0) {
+if (strncmp (dirpath, mail_root, mail_root_len) != 0) {
fprintf (stderr, "Warning: '%s' is not a subdirectory of '%s'\n",
-dirpath, db_path);
+dirpath, mail_root);
return NULL;
 }
 
-dirpath += db_path_len;
+dirpath += mail_root_len;
 while (*dirpath == '/')
dirpath++;
 
@@ -346,7 +347,7 @@ _entry_in_ignore_list (add_files_state_t *state, const char 
*dirpath,
 if (state->ignore_regex_length == 0)
return false;
 
-path = _get_relative_path (state->db_path, dirpath, entry);
+path = _get_relative_path (state->mail_root, dirpath, entry);
 if (! path)
return false;
 
@@ -1050,22 +1051,34 @@ _maybe_upgrade (notmuch_database_t *notmuch, 
add_files_state_t *state)
 if (notmuch_database_needs_upgrade (notmuch)) {
time_t now = time (NULL);
struct tm *gm_time = gmtime ();
+   struct stat st;
+   int err;
notmuch_status_t status;
char *dot_notmuch_path = talloc_asprintf (notmuch, "%s/%s", 
state->db_path, ".notmuch");
 
+   const char *backup_name;
+
+   err = stat (dot_notmuch_path, );
+   if (err) {
+   if (errno == ENOENT) {
+   dot_notmuch_path = NULL;
+   } else {
+   fprintf (stderr, "Failed to stat %s: %s\n", dot_notmuch_path, 
strerror (errno));
+   return EXIT_FAILURE;
+   }
+   }
+
/* since dump files are written atomically, the amount of
 * harm from overwriting one within a second seems
 * relatively small. */
-
-   const char *backup_name =
-   talloc_asprintf (notmuch, "%s/dump-%04d%02d%02dT%02d%02d%02d.gz",
-dot_notmuch_path,
-gm_time->tm_year + 1900,
-gm_time->tm_mon + 1,
-gm_time->tm_mday,
-gm_time->tm_hour,
-gm_time->tm_min,
-gm_time->tm_sec);
+   backup_name = talloc_asprintf (notmuch, 
"%s/dump-%04d%02d%02dT%02d%02d%02d.gz",
+  dot_notmuch_path ? dot_notmuch_path : 
state->db_path,
+  gm_time->tm_year + 1900,
+  gm_time->tm_mon + 1,
+  gm_time->tm_mday,
+  gm_time->tm_hour,
+  gm_time->tm_min,
+  gm_time->tm_sec);
 
if (state->verbosity >= VERBOSITY_NORMAL) {
printf ("Welcome to a new version of notmuch! Your database will 
now be upgraded.\n");
@@ -1108,7 +1121,7 @@ notmuch_new_command (unused(notmuch_config_t *config), 
notmuch_database_t *notmu
 };
 struct timeval tv_start;
 int ret = 0;
-const char *db_path;
+const char *db_path, *mail_root;
 struct sigaction action;
 _filename_node_t *f;
 int opt_index;
@@ -1153,6 +1166,9 @@ notmuch_new_command (unused(notmuch_config_t *config), 
notmuch_database_t *notmu
 db_path = notmuch_config_get (notmuch, NOTMUCH_CONFIG_DATABASE_PATH);
 add_files_state.db_path = db_path;
 
+mail_root = notmuch_config_get (notmuch, NOTMUCH_CONFIG_MAIL_ROOT);
+add_files_state.mail_root = mail_root;
+
 if (! _setup_ignore (notmuch, _files_state))
return EXIT_FAILURE;
 
@@ -1179,7 +1195,7 @@ notmuch_new_command (unused(notmuch_config_t *config), 
notmuch_database_t *notmu
 
 if (notmuch_database_get_revision (notmuch, NULL) == 0) {
int count = 0;
-   count_files (db_path, , _files_state);
+   count_files (mail_root, , _files_state);
if (interrupted)
return EXIT_FAILURE;
 
@@ -1225,7 +1241,7 @@ notmuch_new_command (unused(notmuch_config_t *config), 

[PATCH 18/22] CLI/insert: support split database and mail root

2021-03-14 Thread David Bremner
The new test is in T055-path-config because it uses the helper
function split_config, and because it seems easier to put the
database path related tests in one place.
---
 notmuch-insert.c | 12 +++-
 test/T055-path-config.sh | 10 ++
 2 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index cc5aeefc..865b6b69 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -449,7 +449,7 @@ notmuch_insert_command (unused(notmuch_config_t *config), 
notmuch_database_t *no
 {
 notmuch_status_t status, close_status;
 struct sigaction action;
-const char *db_path;
+const char *mail_root;
 notmuch_config_values_t *new_tags = NULL;
 tag_op_list_t *tag_ops;
 char *query_string = NULL;
@@ -481,13 +481,7 @@ notmuch_insert_command (unused(notmuch_config_t *config), 
notmuch_database_t *no
 
 notmuch_process_shared_options (argv[0]);
 
-
-db_path = notmuch_config_get (notmuch, NOTMUCH_CONFIG_DATABASE_PATH);
-
-if (! db_path)
-   INTERNAL_ERROR ("Unable to retrieve database path");
-else
-   db_path = talloc_strdup (local, db_path);
+mail_root = notmuch_config_get (notmuch, NOTMUCH_CONFIG_MAIL_ROOT);
 
 new_tags = notmuch_config_get_values (notmuch, NOTMUCH_CONFIG_NEW_TAGS);
 
@@ -533,7 +527,7 @@ notmuch_insert_command (unused(notmuch_config_t *config), 
notmuch_database_t *no
return EXIT_FAILURE;
 }
 
-maildir = talloc_asprintf (local, "%s/%s", db_path, folder);
+maildir = talloc_asprintf (local, "%s/%s", mail_root, folder);
 if (! maildir) {
fprintf (stderr, "Out of memory\n");
return EXIT_FAILURE;
diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
index f9a8e200..fb5174ac 100755
--- a/test/T055-path-config.sh
+++ b/test/T055-path-config.sh
@@ -167,6 +167,16 @@ On Tue, 05 Jan 2010 15:43:56 -, Sender 
 wrote:
 > basic reply test
 EOF
 test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "insert+search ($config)"
+generate_message \
+   "[subject]=\"insert-subject\"" \
+   "[date]=\"Sat, 01 Jan 2000 12:00:00 -\"" \
+   "[body]=\"insert-message\""
+mkdir -p "$MAIL_DIR"/{cur,new,tmp}
+notmuch insert < "$gen_msg_filename"
+cur_msg_filename=$(notmuch search --output=files "subject:insert-subject")
+test_expect_equal_file "$cur_msg_filename" "$gen_msg_filename"
+
 restore_config
 done
 
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 12/22] lib/open: use _finish_open in n_d_create_with_config

2021-03-14 Thread David Bremner
This avoids reading the configuration file twice.
---
 lib/open.cc | 19 +++
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index 7673e875..54e71fec 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -442,8 +442,7 @@ notmuch_database_open_with_config (const char 
*database_path,
 if (status)
goto DONE;
 
-if (! (notmuch->xapian_path = talloc_asprintf (local, "%s/%s", 
notmuch_path, "xapian"))) {
-   message = strdup ("Out of memory\n");
+if (! (notmuch->xapian_path = talloc_asprintf (notmuch, "%s/%s", 
notmuch_path, "xapian"))) {
status = NOTMUCH_STATUS_OUT_OF_MEMORY;
goto DONE;
 }
@@ -550,12 +549,16 @@ notmuch_database_create_with_config (const char 
*database_path,
goto DONE;
 }
 
-/* XXX this reads the config file twice, which is a bit wasteful */
-status = notmuch_database_open_with_config (database_path,
-   
NOTMUCH_DATABASE_MODE_READ_WRITE,
-   config_path,
-   profile,
-   , );
+if (! (notmuch->xapian_path = talloc_asprintf (notmuch, "%s/%s", 
notmuch_path, "xapian"))) {
+   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+   goto DONE;
+}
+
+status = _finish_open (notmuch,
+  profile,
+  NOTMUCH_DATABASE_MODE_READ_WRITE,
+  key_file,
+  );
 if (status)
goto DONE;
 
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 10/22] lib/open: reuse directory checks from n_d_c_with_config

2021-03-14 Thread David Bremner
Make checks more uniform between creating new databases and opening
existing ones.
---
 lib/open.cc  | 66 ++--
 test/T030-config.sh  |  2 +-
 test/T560-lib-error.sh   |  4 +--
 test/T750-user-header.sh |  2 +-
 4 files changed, 47 insertions(+), 27 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index f48b97ae..0defa9c0 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -152,6 +152,29 @@ _load_key_file (const char *path,
 return status;
 }
 
+static notmuch_status_t
+_db_dir_exists (const char *database_path, char **message)
+{
+struct stat st;
+int err;
+
+err = stat (database_path, );
+if (err) {
+   IGNORE_RESULT (asprintf (message, "Error: Cannot open database at %s: 
%s.\n",
+database_path, strerror (errno)));
+   return NOTMUCH_STATUS_FILE_ERROR;
+}
+
+if (! S_ISDIR (st.st_mode)) {
+   IGNORE_RESULT (asprintf (message, "Error: Cannot open database at %s: "
+"Not a directory.\n",
+database_path));
+   return NOTMUCH_STATUS_FILE_ERROR;
+}
+
+return NOTMUCH_STATUS_SUCCESS;
+}
+
 static notmuch_status_t
 _choose_database_path (void *ctx,
   const char *config_path,
@@ -251,8 +274,6 @@ notmuch_database_open_with_config (const char 
*database_path,
 notmuch_database_t *notmuch = NULL;
 char *notmuch_path, *incompat_features;
 char *message = NULL;
-struct stat st;
-int err;
 unsigned int version;
 GKeyFile *key_file = NULL;
 
@@ -270,19 +291,19 @@ notmuch_database_open_with_config (const char 
*database_path,
 
 _set_database_path (notmuch, database_path);
 
+status = _db_dir_exists (database_path, );
+if (status)
+   goto DONE;
+
 if (! (notmuch_path = talloc_asprintf (local, "%s/%s", database_path, 
".notmuch"))) {
message = strdup ("Out of memory\n");
status = NOTMUCH_STATUS_OUT_OF_MEMORY;
goto DONE;
 }
 
-err = stat (notmuch_path, );
-if (err) {
-   IGNORE_RESULT (asprintf (, "Error opening database at %s: %s\n",
-notmuch_path, strerror (errno)));
-   status = NOTMUCH_STATUS_FILE_ERROR;
+status = _db_dir_exists (notmuch_path, );
+if (status)
goto DONE;
-}
 
 if (! (notmuch->xapian_path = talloc_asprintf (local, "%s/%s", 
notmuch_path, "xapian"))) {
message = strdup ("Out of memory\n");
@@ -471,9 +492,16 @@ notmuch_database_create_with_config (const char 
*database_path,
 char *notmuch_path = NULL;
 char *message = NULL;
 GKeyFile *key_file = NULL;
-struct stat st;
-int err;
 void *local = talloc_new (NULL);
+int err;
+
+_init_libs ();
+
+notmuch = _alloc_notmuch ();
+if (! notmuch) {
+   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+   goto DONE;
+}
 
 _init_libs ();
 
@@ -481,21 +509,11 @@ notmuch_database_create_with_config (const char 
*database_path,
 _file, _path, )))
goto DONE;
 
-err = stat (database_path, );
-if (err) {
-   IGNORE_RESULT (asprintf (, "Error: Cannot create database at 
%s: %s.\n",
-database_path, strerror (errno)));
-   status = NOTMUCH_STATUS_FILE_ERROR;
+status = _db_dir_exists (database_path, );
+if (status)
goto DONE;
-}
 
-if (! S_ISDIR (st.st_mode)) {
-   IGNORE_RESULT (asprintf (, "Error: Cannot create database at 
%s: "
-"Not a directory.\n",
-database_path));
-   status = NOTMUCH_STATUS_FILE_ERROR;
-   goto DONE;
-}
+_set_database_path (notmuch, database_path);
 
 notmuch_path = talloc_asprintf (local, "%s/%s", database_path, ".notmuch");
 
@@ -503,6 +521,8 @@ notmuch_database_create_with_config (const char 
*database_path,
 if (err) {
if (errno == EEXIST) {
status = NOTMUCH_STATUS_DATABASE_EXISTS;
+   talloc_free (notmuch);
+   notmuch = NULL;
} else {
IGNORE_RESULT (asprintf (, "Error: Cannot create directory 
%s: %s.\n",
 notmuch_path, strerror (errno)));
diff --git a/test/T030-config.sh b/test/T030-config.sh
index 32efa945..6610bf8e 100755
--- a/test/T030-config.sh
+++ b/test/T030-config.sh
@@ -61,7 +61,7 @@ built_with.compact=something
 built_with.field_processor=something
 built_with.retry_lock=something
 
-Error opening database at MAIL_DIR/.notmuch: No such file or directory
+Error: Cannot open database at MAIL_DIR/.notmuch: No such file or directory.
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
diff --git a/test/T560-lib-error.sh b/test/T560-lib-error.sh
index ade376ef..03df69d9 100755
--- a/test/T560-lib-error.sh
+++ b/test/T560-lib-error.sh
@@ -76,7 +76,7 @@ EOF
 cat <<'EOF' >EXPECTED
 == stdout ==
 == stderr ==
-Error opening database 

[PATCH 06/22] lib/open: support NOTMUCH_DATABASE environment variable

2021-03-14 Thread David Bremner
The additional code is trivial, but making sure we get the priority of
various options correct takes a few tests.
---
 lib/open.cc|  4 +++
 test/T590-libconfig.sh | 67 +-
 2 files changed, 70 insertions(+), 1 deletion(-)

diff --git a/lib/open.cc b/lib/open.cc
index d05a19a7..6e46168a 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -168,6 +168,10 @@ _choose_database_path (void *ctx,
return status;
 }
 
+if (! *database_path) {
+   *database_path = getenv ("NOTMUCH_DATABASE");
+}
+
 if (! *database_path && *key_file) {
char *path = g_key_file_get_value (*key_file, "database", "path", NULL);
if (path) {
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index e7e6e08a..4e510e97 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -27,7 +27,7 @@ int main (int argc, char** argv)
   ,
   );
if (stat != NOTMUCH_STATUS_SUCCESS) {
- fprintf (stderr, "error opening database: %d %s\n", stat, msg ? msg : "");
+ fprintf (stderr, "error opening database\n%s\n%s\n", 
notmuch_status_to_string (stat), msg ? msg : "");
  exit (1);
}
 EOF
@@ -505,4 +505,69 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 restore_database
 
+test_begin_subtest "no config, fail to open database"
+old_NOTMUCH_CONFIG=${NOTMUCH_CONFIG}
+unset NOTMUCH_CONFIG
+cat c_head - c_tail <<'EOF' | test_C %NULL% '' %NULL%
+{
+   printf("NOT RUN");
+}
+EOF
+NOTMUCH_CONFIG=${old_NOTMUCH_CONFIG}
+cat <<'EOF' >EXPECTED
+== stdout ==
+== stderr ==
+error opening database
+Erroneous NULL pointer
+Error: Cannot open a database for a NULL path.
+
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "open database from NOTMUCH_DATABASE"
+old_NOTMUCH_CONFIG=${NOTMUCH_CONFIG}
+unset NOTMUCH_CONFIG
+export NOTMUCH_DATABASE=${MAIL_DIR}
+cat c_head - c_tail <<'EOF' | test_C %NULL% '' %NULL%
+{
+   EXPECT0(notmuch_database_get_config (db, "test.key1", ));
+   printf("test.key1 = %s\n", val);
+   EXPECT0(notmuch_database_get_config (db, "test.key2", ));
+   printf("test.key2 = %s\n", val);
+}
+EOF
+NOTMUCH_CONFIG=${old_NOTMUCH_CONFIG}
+unset NOTMUCH_DATABASE
+cat <<'EOF' >EXPECTED
+== stdout ==
+test.key1 = testvalue1
+test.key2 = testvalue2
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "NOTMUCH_DATABASE overrides config"
+old_path=$(notmuch config get database.path)
+notmuch config set database.path /nonexistent
+export NOTMUCH_DATABASE=${MAIL_DIR}
+cat c_head - c_tail <<'EOF' | test_C %NULL% '' %NULL%
+{
+   EXPECT0(notmuch_database_get_config (db, "test.key1", ));
+   printf("test.key1 = %s\n", val);
+   EXPECT0(notmuch_database_get_config (db, "test.key2", ));
+   printf("test.key2 = %s\n", val);
+}
+EOF
+NOTMUCH_CONFIG=${old_NOTMUCH_CONFIG}
+unset NOTMUCH_DATABASE
+cat <<'EOF' >EXPECTED
+== stdout ==
+test.key1 = testvalue1
+test.key2 = testvalue2
+== stderr ==
+EOF
+notmuch config set database.path "${old_path}"
+test_expect_equal_file EXPECTED OUTPUT
+
+
 test_done
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 09/22] lib/open: factor out library initialization

2021-03-14 Thread David Bremner
This is slightly more tidy, but more importantly it allows for re-use
of this code in n_d_create_with_config. That re-use will be crucial
when we no longer call n_d_open_with_config from
n_d_create_with_config.
---
 lib/open.cc | 34 ++
 1 file changed, 22 insertions(+), 12 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index a62d4dee..f48b97ae 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -220,6 +220,24 @@ _set_database_path (notmuch_database_t *notmuch,
 _notmuch_config_cache (notmuch, NOTMUCH_CONFIG_DATABASE_PATH, path);
 }
 
+static void
+_init_libs ()
+{
+
+static int initialized = 0;
+
+/* Initialize the GLib type system and threads */
+#if ! GLIB_CHECK_VERSION (2, 35, 1)
+g_type_init ();
+#endif
+
+/* Initialize gmime */
+if (! initialized) {
+   g_mime_init ();
+   initialized = 1;
+}
+}
+
 notmuch_status_t
 notmuch_database_open_with_config (const char *database_path,
   notmuch_database_mode_t mode,
@@ -237,7 +255,8 @@ notmuch_database_open_with_config (const char 
*database_path,
 int err;
 unsigned int version;
 GKeyFile *key_file = NULL;
-static int initialized = 0;
+
+_init_libs ();
 
 notmuch = _alloc_notmuch ();
 if (! notmuch) {
@@ -271,17 +290,6 @@ notmuch_database_open_with_config (const char 
*database_path,
goto DONE;
 }
 
-/* Initialize the GLib type system and threads */
-#if ! GLIB_CHECK_VERSION (2, 35, 1)
-g_type_init ();
-#endif
-
-/* Initialize gmime */
-if (! initialized) {
-   g_mime_init ();
-   initialized = 1;
-}
-
 try {
std::string last_thread_id;
std::string last_mod;
@@ -467,6 +475,8 @@ notmuch_database_create_with_config (const char 
*database_path,
 int err;
 void *local = talloc_new (NULL);
 
+_init_libs ();
+
 if ((status = _choose_database_path (local, config_path, profile,
 _file, _path, )))
goto DONE;
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 03/22] lib: save path of xapian database in notmuch struct.

2021-03-14 Thread David Bremner
This will allow re-opening in a different mode (read/write
vs. read-only) with current Xapian API. It will also prove useful when
updating the compact functions to support more flexible database
location.
---
 lib/database-private.h |  4 
 lib/open.cc| 19 ++-
 2 files changed, 14 insertions(+), 9 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index d83cf0d0..d936b216 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -189,8 +189,12 @@ operator& (notmuch_field_flag_t a, notmuch_field_flag_t b)
 struct _notmuch_database {
 bool exception_reported;
 
+/* Path to database parent directory and or/mail root */
 char *path;
 
+/* Path to actual database */
+const char *xapian_path;
+
 int atomic_nesting;
 /* true if changes have been made in this atomic section */
 bool atomic_dirty;
diff --git a/lib/open.cc b/lib/open.cc
index 0c965d0d..12d3613f 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -199,7 +199,7 @@ notmuch_database_open_with_config (const char 
*database_path,
 notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
 void *local = talloc_new (NULL);
 notmuch_database_t *notmuch = NULL;
-char *notmuch_path, *xapian_path, *incompat_features;
+char *notmuch_path, *incompat_features;
 char *message = NULL;
 struct stat st;
 int err;
@@ -225,12 +225,6 @@ notmuch_database_open_with_config (const char 
*database_path,
goto DONE;
 }
 
-if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, 
"xapian"))) {
-   message = strdup ("Out of memory\n");
-   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
-   goto DONE;
-}
-
 /* Initialize the GLib type system and threads */
 #if ! GLIB_CHECK_VERSION (2, 35, 1)
 g_type_init ();
@@ -252,16 +246,23 @@ notmuch_database_open_with_config (const char 
*database_path,
 notmuch->writable_xapian_db = NULL;
 notmuch->atomic_nesting = 0;
 notmuch->view = 1;
+
+if (! (notmuch->xapian_path = talloc_asprintf (notmuch, "%s/%s", 
notmuch_path, "xapian"))) {
+   message = strdup ("Out of memory\n");
+   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+   goto DONE;
+}
+
 try {
std::string last_thread_id;
std::string last_mod;
 
if (mode == NOTMUCH_DATABASE_MODE_READ_WRITE) {
-   notmuch->writable_xapian_db = new Xapian::WritableDatabase 
(xapian_path,
+   notmuch->writable_xapian_db = new Xapian::WritableDatabase 
(notmuch->xapian_path,

DB_ACTION);
notmuch->xapian_db = notmuch->writable_xapian_db;
} else {
-   notmuch->xapian_db = new Xapian::Database (xapian_path);
+   notmuch->xapian_db = new Xapian::Database (notmuch->xapian_path);
}
 
/* Check version.  As of database version 3, we represent
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 13/22] lib/open: Use check for existing database by trial opening

2021-03-14 Thread David Bremner
This is a bit heavyweight for now, but it will make more sense when we
check multiple locations for the Xapian database.
---
 lib/open.cc| 61 +-
 test/T360-symbol-hiding.sh |  2 +-
 2 files changed, 48 insertions(+), 15 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index 54e71fec..956d50b9 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -232,6 +232,51 @@ _alloc_notmuch ()
 return notmuch;
 }
 
+static notmuch_status_t
+_trial_open (const char *xapian_path, char **message_ptr)
+{
+try {
+   Xapian::Database db (xapian_path);
+} catch (const Xapian::DatabaseOpeningError ) {
+   IGNORE_RESULT (asprintf (message_ptr,
+"Cannot open Xapian database at %s: %s\n",
+xapian_path,
+error.get_msg ().c_str ()));
+   return NOTMUCH_STATUS_PATH_ERROR;
+} catch (const Xapian::Error ) {
+   IGNORE_RESULT (asprintf (message_ptr,
+"A Xapian exception occurred opening database: 
%s\n",
+error.get_msg ().c_str ()));
+   return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+}
+return NOTMUCH_STATUS_SUCCESS;
+}
+
+static notmuch_status_t
+_choose_xapian_path (void *ctx, const char *database_path, const char 
**xapian_path,
+char **message_ptr)
+{
+notmuch_status_t status;
+const char *trial_path, *notmuch_path;
+
+status = _db_dir_exists (database_path, message_ptr);
+if (status)
+   goto DONE;
+
+notmuch_path = talloc_asprintf (ctx, "%s/.notmuch", database_path);
+status = _db_dir_exists (notmuch_path, message_ptr);
+if (status)
+   goto DONE;
+
+trial_path = talloc_asprintf (ctx, "%s/xapian", notmuch_path);
+status = _trial_open (trial_path, message_ptr);
+
+  DONE:
+if (status == NOTMUCH_STATUS_SUCCESS)
+   *xapian_path = trial_path;
+return status;
+}
+
 static void
 _set_database_path (notmuch_database_t *notmuch,
const char *database_path)
@@ -410,7 +455,6 @@ notmuch_database_open_with_config (const char 
*database_path,
 notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
 void *local = talloc_new (NULL);
 notmuch_database_t *notmuch = NULL;
-char *notmuch_path;
 char *message = NULL;
 GKeyFile *key_file = NULL;
 
@@ -426,27 +470,16 @@ notmuch_database_open_with_config (const char 
*database_path,
 )))
goto DONE;
 
-_set_database_path (notmuch, database_path);
-
 status = _db_dir_exists (database_path, );
 if (status)
goto DONE;
 
-if (! (notmuch_path = talloc_asprintf (local, "%s/%s", database_path, 
".notmuch"))) {
-   message = strdup ("Out of memory\n");
-   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
-   goto DONE;
-}
+_set_database_path (notmuch, database_path);
 
-status = _db_dir_exists (notmuch_path, );
+status = _choose_xapian_path (local, database_path, >xapian_path, 
);
 if (status)
goto DONE;
 
-if (! (notmuch->xapian_path = talloc_asprintf (notmuch, "%s/%s", 
notmuch_path, "xapian"))) {
-   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
-   goto DONE;
-}
-
 status = _finish_open (notmuch, profile, mode, key_file, );
 
   DONE:
diff --git a/test/T360-symbol-hiding.sh b/test/T360-symbol-hiding.sh
index 729b9d72..642457bf 100755
--- a/test/T360-symbol-hiding.sh
+++ b/test/T360-symbol-hiding.sh
@@ -17,7 +17,7 @@ $TEST_DIRECTORY/symbol-test ${PWD}/fakedb ${PWD}/nonexistent 
2>&1 \
| notmuch_dir_sanitize | sed -e "s,\`,\',g" -e "s,No [^[:space:]]* 
database,No XX database,g" > OUTPUT
 
 cat < EXPECTED
-A Xapian exception occurred opening database: Couldn't stat 
'CWD/fakedb/.notmuch/xapian'
+Cannot open Xapian database at CWD/fakedb/.notmuch/xapian: Couldn't stat 
'CWD/fakedb/.notmuch/xapian'
 caught No XX database found at path 'CWD/nonexistent'
 EOF
 test_expect_equal_file EXPECTED OUTPUT
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 02/22] lib: publish API for notmuch_database_reopen

2021-03-14 Thread David Bremner
Include the (currently unused) mode argument which will specify which
mode to re-open the database in. Functionality and docs to be
finalized in a followup commit.
---
 lib/database.cc   | 3 ++-
 lib/message.cc| 3 ++-
 lib/notmuch-private.h | 3 ---
 lib/notmuch.h | 7 +++
 4 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index f27616da..82ac5f20 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -518,7 +518,8 @@ notmuch_database_close (notmuch_database_t *notmuch)
 }
 
 notmuch_status_t
-_notmuch_database_reopen (notmuch_database_t *notmuch)
+notmuch_database_reopen (notmuch_database_t *notmuch,
+unused(notmuch_database_mode_t mode))
 {
 if (_notmuch_database_mode (notmuch) != NOTMUCH_DATABASE_MODE_READ_ONLY)
return NOTMUCH_STATUS_UNSUPPORTED_OPERATION;
diff --git a/lib/message.cc b/lib/message.cc
index dc4b47dc..73a7805f 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -459,7 +459,8 @@ _notmuch_message_ensure_metadata (notmuch_message_t 
*message, void *field)
/* all the way without an exception */
break;
} catch (const Xapian::DatabaseModifiedError ) {
-   notmuch_status_t status = _notmuch_database_reopen 
(message->notmuch);
+   notmuch_status_t status = notmuch_database_reopen (message->notmuch,
+  
NOTMUCH_DATABASE_MODE_READ_ONLY);
if (status != NOTMUCH_STATUS_SUCCESS)
INTERNAL_ERROR ("unhandled error from notmuch_database_reopen: 
%s\n",
notmuch_status_to_string (status));
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 3c52cfc2..ef814378 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -204,9 +204,6 @@ _notmuch_message_id_compressed (void *ctx, const char 
*message_id);
 notmuch_status_t
 _notmuch_database_ensure_writable (notmuch_database_t *notmuch);
 
-notmuch_status_t
-_notmuch_database_reopen (notmuch_database_t *notmuch);
-
 void
 _notmuch_database_log (notmuch_database_t *notmuch,
   const char *format, ...);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 58c568d5..db86ffc0 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -879,6 +879,13 @@ notmuch_database_find_message_by_filename 
(notmuch_database_t *notmuch,
 notmuch_tags_t *
 notmuch_database_get_all_tags (notmuch_database_t *db);
 
+/**
+ * Reopen an open notmuch database.
+ *
+ */
+notmuch_status_t
+notmuch_database_reopen (notmuch_database_t *db, notmuch_database_mode_t mode);
+
 /**
  * Create a new query for 'database'.
  *
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 17/22] lib/open: support XDG_DATA_HOME as a fallback database location.

2021-03-14 Thread David Bremner
This changes some error reporting, either intentionally by reporting
the highest level missing directory, or by side effect from looking in
XDG locations when given null database location.
---
 lib/open.cc  | 13 +++--
 test/T055-path-config.sh | 35 +--
 test/T560-lib-error.sh   |  4 ++--
 test/T590-libconfig.sh   |  4 ++--
 4 files changed, 48 insertions(+), 8 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index 2816f72a..3097121e 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -181,6 +181,7 @@ _choose_database_path (void *ctx,
   const char *profile,
   GKeyFile **key_file,
   const char **database_path,
+  bool *split,
   char **message)
 {
 notmuch_status_t status;
@@ -203,6 +204,11 @@ _choose_database_path (void *ctx,
}
 }
 
+if (! *database_path) {
+   *database_path = _xdg_dir (ctx, "XDG_DATA_HOME", ".local/share", 
profile);
+   *split = true;
+}
+
 if (*database_path == NULL) {
*message = strdup ("Error: Cannot open a database for a NULL path.\n");
return NOTMUCH_STATUS_NULL_POINTER;
@@ -462,6 +468,7 @@ notmuch_database_open_with_config (const char 
*database_path,
 notmuch_database_t *notmuch = NULL;
 char *message = NULL;
 GKeyFile *key_file = NULL;
+bool split = false;
 
 _init_libs ();
 
@@ -471,7 +478,8 @@ notmuch_database_open_with_config (const char 
*database_path,
goto DONE;
 }
 
-if ((status = _choose_database_path (local, config_path, profile, 
_file, _path,
+if ((status = _choose_database_path (local, config_path, profile,
+_file, _path, ,
 )))
goto DONE;
 
@@ -563,7 +571,8 @@ notmuch_database_create_with_config (const char 
*database_path,
 _init_libs ();
 
 if ((status = _choose_database_path (local, config_path, profile,
-_file, _path, )))
+_file, _path, ,
+)))
goto DONE;
 
 status = _db_dir_exists (database_path, );
diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
index 6df17f80..f9a8e200 100755
--- a/test/T055-path-config.sh
+++ b/test/T055-path-config.sh
@@ -37,8 +37,31 @@ symlink_config () {
 unset DATABASE_PATH
 }
 
-for config in traditional split symlink; do
-# start each set of tests with a known set of messages
+xdg_config () {
+local dir
+local profile=${1:-default}
+
+if [[ $profile != default ]]; then
+   export NOTMUCH_PROFILE=$profile
+fi
+
+backup_config
+DATABASE_PATH="${HOME}/.local/share/notmuch/${profile}"
+rm -rf $DATABASE_PATH
+mkdir -p $DATABASE_PATH
+
+config_dir="${HOME}/.config/notmuch/${profile}"
+mkdir -p ${config_dir}
+CONFIG_PATH=$config_dir/config
+mv ${NOTMUCH_CONFIG} $CONFIG_PATH
+unset NOTMUCH_CONFIG
+
+notmuch --config=${CONFIG_PATH} config set database.mail_root 
${TMP_DIRECTORY}/mail
+notmuch --config=${CONFIG_PATH} config set database.path
+}
+
+for config in traditional split XDG XDG+profile symlink; do
+#start each set of tests with an known set of messages
 add_email_corpus
 
 case $config in
@@ -49,6 +72,14 @@ for config in traditional split symlink; do
split_config
mv mail/.notmuch/xapian $DATABASE_PATH
;;
+   XDG)
+   xdg_config
+   mv mail/.notmuch/xapian $DATABASE_PATH
+   ;;
+   XDG+profile)
+   xdg_config ${RANDOM}
+   mv mail/.notmuch/xapian $DATABASE_PATH
+   ;;
symlink)
symlink_config
;;
diff --git a/test/T560-lib-error.sh b/test/T560-lib-error.sh
index 03df69d9..89447e9a 100755
--- a/test/T560-lib-error.sh
+++ b/test/T560-lib-error.sh
@@ -22,7 +22,7 @@ EOF
 cat <<'EOF' >EXPECTED
 == stdout ==
 == stderr ==
-Error: Cannot open a database for a NULL path.
+Error: Cannot open database at CWD/home/.local/share/notmuch/default: No such 
file or directory.
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
@@ -93,7 +93,7 @@ EOF
 cat <<'EOF' >EXPECTED
 == stdout ==
 == stderr ==
-Error: Cannot open a database for a NULL path.
+Error: Cannot open database at CWD/home/.local/share/notmuch/default: No such 
file or directory.
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 5cf70987..c21c139b 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -519,8 +519,8 @@ cat <<'EOF' >EXPECTED
 == stdout ==
 == stderr ==
 error opening database
-Erroneous NULL pointer
-Error: Cannot open a database for a NULL path.
+Something went wrong trying to read or write a file
+Error: Cannot open database at CWD/home/.local/share/notmuch/default: No such 
file or directory.
 
 EOF
 

[PATCH 05/22] CLI/show: complete conversion to new configuration framework.

2021-03-14 Thread David Bremner
In order to open the database in main() for this command, we may need
to re-open it in the (possibly less common) case where crypto options
require write access.
---
 notmuch-show.c   | 28 
 notmuch.c|  2 +-
 test/T035-read-config.sh |  2 --
 3 files changed, 9 insertions(+), 23 deletions(-)

diff --git a/notmuch-show.c b/notmuch-show.c
index b6b53991..5fd1ce25 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -1225,10 +1225,9 @@ static const notmuch_show_format_t *formatters[] = {
 };
 
 int
-notmuch_show_command (notmuch_config_t *config, unused(notmuch_database_t 
*notmuch),
+notmuch_show_command (unused (notmuch_config_t *config), notmuch_database_t 
*notmuch,
  int argc, char *argv[])
 {
-notmuch_database_t *notmuch;
 notmuch_query_t *query;
 char *query_string;
 int opt_index, ret;
@@ -1245,7 +1244,7 @@ notmuch_show_command (notmuch_config_t *config, 
unused(notmuch_database_t *notmu
 bool entire_thread_set = false;
 bool single_message;
 bool unthreaded = FALSE;
-char *status_string = NULL;
+notmuch_status_t status;
 
 notmuch_opt_desc_t options[] = {
{ .opt_keyword = , .name = "format", .keywords =
@@ -1336,26 +1335,15 @@ notmuch_show_command (notmuch_config_t *config, 
unused(notmuch_database_t *notmu
 "Warning: --include-html only implemented for format=text, 
format=json and format=sexp\n");
 }
 
-notmuch_database_mode_t mode = NOTMUCH_DATABASE_MODE_READ_ONLY;
-
-if (params.crypto.decrypt == NOTMUCH_DECRYPT_TRUE)
-   mode = NOTMUCH_DATABASE_MODE_READ_WRITE;
-if (notmuch_database_open_with_config (NULL,
-  mode,
-  _notmuch_config_get_path (config),
-  NULL,
-  ,
-  _string)) {
-   if (status_string) {
-   fputs (status_string, stderr);
-   free (status_string);
+if (params.crypto.decrypt == NOTMUCH_DECRYPT_TRUE) {
+   status = notmuch_database_reopen (notmuch, 
NOTMUCH_DATABASE_MODE_READ_WRITE);
+   if (status) {
+   fprintf (stderr, "Error reopening database for READ_WRITE: %s\n",
+ notmuch_status_to_string (status));
+   return EXIT_FAILURE;
}
-
-   return EXIT_FAILURE;
 }
 
-config = NULL;
-
 notmuch_exit_if_unmatched_db_uuid (notmuch);
 
 query_string = query_string_from_args (notmuch, argc - opt_index, argv + 
opt_index);
diff --git a/notmuch.c b/notmuch.c
index 5e56209b..f2bf9c24 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -156,7 +156,7 @@ static command_t commands[] = {
   "Search for messages matching the given search terms." },
 { "address", notmuch_address_command, NOTMUCH_COMMAND_DATABASE_EARLY,
   "Get addresses from messages matching the given search terms." },
-{ "show", notmuch_show_command, NOTMUCH_COMMAND_CONFIG_OPEN,
+{ "show", notmuch_show_command, NOTMUCH_COMMAND_DATABASE_EARLY,
   "Show all messages matching the search terms." },
 { "count", notmuch_count_command, NOTMUCH_COMMAND_DATABASE_EARLY,
   "Count messages matching the search terms." },
diff --git a/test/T035-read-config.sh b/test/T035-read-config.sh
index 205d1736..ac0f420b 100755
--- a/test/T035-read-config.sh
+++ b/test/T035-read-config.sh
@@ -378,7 +378,6 @@ restore_database
 test_expect_equal "$output" "OK"
 
 test_begin_subtest "show with alternate config (xdg)"
-test_subtest_known_broken
 backup_database
 notmuch tag -- +foobar17 '*'
 xdg_config
@@ -389,7 +388,6 @@ restore_config
 test_expect_equal "$output" "OK"
 
 test_begin_subtest "show with alternate config (xdg+profile)"
-test_subtest_known_broken
 backup_database
 notmuch tag -- +foobar17 '*'
 xdg_config foobar17
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 07/22] lib/open: allocate notmuch_t struct early

2021-03-14 Thread David Bremner
This gives more flexibility in restructuring the database opening
code.
---
 lib/open.cc | 57 +
 1 file changed, 36 insertions(+), 21 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index 6e46168a..de5f3467 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -192,6 +192,23 @@ _choose_database_path (void *ctx,
 return NOTMUCH_STATUS_SUCCESS;
 }
 
+notmuch_database_t *
+_alloc_notmuch ()
+{
+notmuch_database_t *notmuch;
+
+notmuch = talloc_zero (NULL, notmuch_database_t);
+if (! notmuch)
+   return NULL;
+
+notmuch->exception_reported = false;
+notmuch->status_string = NULL;
+notmuch->writable_xapian_db = NULL;
+notmuch->atomic_nesting = 0;
+notmuch->view = 1;
+return notmuch;
+}
+
 notmuch_status_t
 notmuch_database_open_with_config (const char *database_path,
   notmuch_database_mode_t mode,
@@ -211,10 +228,19 @@ notmuch_database_open_with_config (const char 
*database_path,
 GKeyFile *key_file = NULL;
 static int initialized = 0;
 
-if ((status = _choose_database_path (local, config_path, profile, 
_file, _path,
-)))
+notmuch = _alloc_notmuch ();
+if (! notmuch) {
+   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+   goto DONE;
+}
+
+if ((status = _choose_database_path (local, config_path, profile,
+_file, _path, )))
goto DONE;
 
+notmuch->path = talloc_strdup (notmuch, database_path);
+strip_trailing (notmuch->path, '/');
+
 if (! (notmuch_path = talloc_asprintf (local, "%s/%s", database_path, 
".notmuch"))) {
message = strdup ("Out of memory\n");
status = NOTMUCH_STATUS_OUT_OF_MEMORY;
@@ -229,6 +255,12 @@ notmuch_database_open_with_config (const char 
*database_path,
goto DONE;
 }
 
+if (! (notmuch->xapian_path = talloc_asprintf (local, "%s/%s", 
notmuch_path, "xapian"))) {
+   message = strdup ("Out of memory\n");
+   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+   goto DONE;
+}
+
 /* Initialize the GLib type system and threads */
 #if ! GLIB_CHECK_VERSION (2, 35, 1)
 g_type_init ();
@@ -240,23 +272,6 @@ notmuch_database_open_with_config (const char 
*database_path,
initialized = 1;
 }
 
-notmuch = talloc_zero (NULL, notmuch_database_t);
-notmuch->exception_reported = false;
-notmuch->status_string = NULL;
-notmuch->path = talloc_strdup (notmuch, database_path);
-
-strip_trailing (notmuch->path, '/');
-
-notmuch->writable_xapian_db = NULL;
-notmuch->atomic_nesting = 0;
-notmuch->view = 1;
-
-if (! (notmuch->xapian_path = talloc_asprintf (notmuch, "%s/%s", 
notmuch_path, "xapian"))) {
-   message = strdup ("Out of memory\n");
-   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
-   goto DONE;
-}
-
 try {
std::string last_thread_id;
std::string last_mod;
@@ -442,8 +457,8 @@ notmuch_database_create_with_config (const char 
*database_path,
 int err;
 void *local = talloc_new (NULL);
 
-if ((status = _choose_database_path (local, config_path, profile, 
_file, _path,
-)))
+if ((status = _choose_database_path (local, config_path, profile,
+_file, _path, )))
goto DONE;
 
 err = stat (database_path, );
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 21/22] lib/config: add configuration variable for backup directory

2021-03-14 Thread David Bremner
Like the hook directory, we primarily need a way to communicate this
directory between various components, but we may as well let the user
configure it.

Most of the diff is generalizing choose_hook_dir to work for both
backup and hook directories.
---
 lib/config.cc  |  3 +++
 lib/notmuch.h  |  1 +
 lib/open.cc| 47 --
 test/T590-libconfig.sh |  1 +
 4 files changed, 37 insertions(+), 15 deletions(-)

diff --git a/lib/config.cc b/lib/config.cc
index f2060e28..8bd3c35a 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -410,6 +410,8 @@ _notmuch_config_key_to_string (notmuch_config_key_t key)
return "database.mail_root";
 case NOTMUCH_CONFIG_HOOK_DIR:
return "database.hook_dir";
+case NOTMUCH_CONFIG_BACKUP_DIR:
+   return "database.backup_dir";
 case NOTMUCH_CONFIG_EXCLUDE_TAGS:
return "search.exclude_tags";
 case NOTMUCH_CONFIG_NEW_TAGS:
@@ -453,6 +455,7 @@ _notmuch_config_default (notmuch_database_t *notmuch, 
notmuch_config_key_t key)
 case NOTMUCH_CONFIG_SYNC_MAILDIR_FLAGS:
return "true";
 case NOTMUCH_CONFIG_HOOK_DIR:
+case NOTMUCH_CONFIG_BACKUP_DIR:
 case NOTMUCH_CONFIG_NEW_IGNORE:
 case NOTMUCH_CONFIG_USER_NAME:
 case NOTMUCH_CONFIG_PRIMARY_EMAIL:
diff --git a/lib/notmuch.h b/lib/notmuch.h
index c9b01719..f9efd79e 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -2476,6 +2476,7 @@ typedef enum _notmuch_config_key {
 NOTMUCH_CONFIG_DATABASE_PATH = NOTMUCH_CONFIG_FIRST,
 NOTMUCH_CONFIG_MAIL_ROOT,
 NOTMUCH_CONFIG_HOOK_DIR,
+NOTMUCH_CONFIG_BACKUP_DIR,
 NOTMUCH_CONFIG_EXCLUDE_TAGS,
 NOTMUCH_CONFIG_NEW_TAGS,
 NOTMUCH_CONFIG_NEW_IGNORE,
diff --git a/lib/open.cc b/lib/open.cc
index aad7fa77..c9390359 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -71,39 +71,43 @@ _xdg_dir (void *ctx,
 }
 
 static notmuch_status_t
-_choose_hook_dir (notmuch_database_t *notmuch,
- const char *profile,
- char **message)
+_choose_dir (notmuch_database_t *notmuch,
+const char *profile,
+notmuch_config_key_t key,
+const char *xdg_var,
+const char *xdg_subdir,
+const char *subdir,
+char **message = NULL)
 {
-const char *config;
-const char *hook_dir;
+const char *parent;
+const char *dir;
 struct stat st;
 int err;
 
-hook_dir = notmuch_config_get (notmuch, NOTMUCH_CONFIG_HOOK_DIR);
+dir = notmuch_config_get (notmuch, key);
 
-if (hook_dir)
+if (dir)
return NOTMUCH_STATUS_SUCCESS;
 
-config = _xdg_dir (notmuch, "XDG_CONFIG_HOME", ".config", profile);
-if (! config)
+parent = _xdg_dir (notmuch, xdg_var, xdg_subdir, profile);
+if (! parent)
return NOTMUCH_STATUS_PATH_ERROR;
 
-hook_dir = talloc_asprintf (notmuch, "%s/hooks", config);
+dir = talloc_asprintf (notmuch, "%s/%s", parent, subdir);
 
-err = stat (hook_dir, );
+err = stat (dir, );
 if (err) {
if (errno == ENOENT) {
char *notmuch_path = dirname (talloc_strdup (notmuch, 
notmuch->xapian_path));
-   hook_dir = talloc_asprintf (notmuch, "%s/hooks", notmuch_path);
+   dir = talloc_asprintf (notmuch, "%s/%s", notmuch_path, subdir);
} else {
IGNORE_RESULT (asprintf (message, "Error: Cannot stat %s: %s.\n",
-hook_dir, strerror (errno)));
+dir, strerror (errno)));
return NOTMUCH_STATUS_FILE_ERROR;
}
 }
 
-_notmuch_config_cache (notmuch, NOTMUCH_CONFIG_HOOK_DIR, hook_dir);
+_notmuch_config_cache (notmuch, key, dir);
 
 return NOTMUCH_STATUS_SUCCESS;
 }
@@ -428,10 +432,23 @@ _finish_open (notmuch_database_t *notmuch,
if (status)
goto DONE;
 
-   status = _choose_hook_dir (notmuch, profile, );
+   status = _choose_dir (notmuch, profile,
+ NOTMUCH_CONFIG_HOOK_DIR,
+ "XDG_CONFIG_HOME",
+ ".config",
+ "hooks",
+ );
if (status)
goto DONE;
 
+   status = _choose_dir (notmuch, profile,
+ NOTMUCH_CONFIG_BACKUP_DIR,
+ "XDG_DATA_HOME",
+ ".local/share",
+ "backups",
+ );
+   if (status)
+   goto DONE;
status = _notmuch_config_load_defaults (notmuch);
if (status)
goto DONE;
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index c21c139b..310668a9 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -366,6 +366,7 @@ cat <<'EOF' >EXPECTED
 MAIL_DIR
 MAIL_DIR
 MAIL_DIR/.notmuch/hooks
+MAIL_DIR/.notmuch/backups
 
 inbox;unread
 NULL
-- 
2.30.1

[PATCH 04/22] lib: support reopening databases for write access.

2021-03-14 Thread David Bremner
In the future Xapian will apparently support this more conveniently
for the cases other than READ_ONLY => READ_ONLY

Conceptually this function seems to fit better in lib/open.cc;
database.cc is still large enough that moving the function makes
sense.
---
 lib/database.cc |  23 
 lib/notmuch.h   |   6 +++
 lib/open.cc |  46 
 test/T595-reopen.sh | 125 
 4 files changed, 177 insertions(+), 23 deletions(-)
 create mode 100755 test/T595-reopen.sh

diff --git a/lib/database.cc b/lib/database.cc
index 82ac5f20..fc13b53b 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -517,29 +517,6 @@ notmuch_database_close (notmuch_database_t *notmuch)
 return status;
 }
 
-notmuch_status_t
-notmuch_database_reopen (notmuch_database_t *notmuch,
-unused(notmuch_database_mode_t mode))
-{
-if (_notmuch_database_mode (notmuch) != NOTMUCH_DATABASE_MODE_READ_ONLY)
-   return NOTMUCH_STATUS_UNSUPPORTED_OPERATION;
-
-try {
-   notmuch->xapian_db->reopen ();
-} catch (const Xapian::Error ) {
-   if (! notmuch->exception_reported) {
-   _notmuch_database_log (notmuch, "Error: A Xapian exception 
reopening database: %s\n",
-  error.get_msg ().c_str ());
-   notmuch->exception_reported = true;
-   }
-   return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
-}
-
-notmuch->view++;
-
-return NOTMUCH_STATUS_SUCCESS;
-}
-
 static int
 unlink_cb (const char *path,
   unused (const struct stat *sb),
diff --git a/lib/notmuch.h b/lib/notmuch.h
index db86ffc0..b5688087 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -882,6 +882,12 @@ notmuch_database_get_all_tags (notmuch_database_t *db);
 /**
  * Reopen an open notmuch database.
  *
+ * @param [in] db  open notmuch database
+ * @param [in] modemode (read only or read-write) for reopened database.
+ *
+ * @retval #NOTMUCH_STATUS_SUCCESS
+ * @retval #NOTMUCH_STATUS_ILLEGAL_ARGUMENTThe passed database was not 
open.
+ * @retval #NOTMUCH_STATUS_XAPIAN_EXCEPTIONA Xapian exception occured
  */
 notmuch_status_t
 notmuch_database_reopen (notmuch_database_t *db, notmuch_database_mode_t mode);
diff --git a/lib/open.cc b/lib/open.cc
index 12d3613f..d05a19a7 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -511,3 +511,49 @@ notmuch_database_create_with_config (const char 
*database_path,
talloc_free (notmuch);
 return status;
 }
+
+notmuch_status_t
+notmuch_database_reopen (notmuch_database_t *notmuch,
+notmuch_database_mode_t new_mode)
+{
+notmuch_database_mode_t cur_mode = _notmuch_database_mode (notmuch);
+
+if (notmuch->xapian_db == NULL) {
+   _notmuch_database_log (notmuch, "Cannot reopen closed or nonexistent 
database\n");
+   return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
+}
+
+try {
+   if (cur_mode == new_mode &&
+   new_mode == NOTMUCH_DATABASE_MODE_READ_ONLY) {
+   notmuch->xapian_db->reopen ();
+   } else {
+   notmuch->xapian_db->close ();
+
+   delete notmuch->xapian_db;
+   notmuch->xapian_db = NULL;
+   /* no need to free the same object twice */
+   notmuch->writable_xapian_db = NULL;
+
+   if (new_mode == NOTMUCH_DATABASE_MODE_READ_WRITE) {
+   notmuch->writable_xapian_db = new Xapian::WritableDatabase 
(notmuch->xapian_path,
+   
DB_ACTION);
+   notmuch->xapian_db = notmuch->writable_xapian_db;
+   } else {
+   notmuch->xapian_db = new Xapian::Database (notmuch->xapian_path,
+  DB_ACTION);
+   }
+   }
+} catch (const Xapian::Error ) {
+   if (! notmuch->exception_reported) {
+   _notmuch_database_log (notmuch, "Error: A Xapian exception 
reopening database: %s\n",
+  error.get_msg ().c_str ());
+   notmuch->exception_reported = true;
+   }
+   return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+}
+
+notmuch->view++;
+notmuch->open = true;
+return NOTMUCH_STATUS_SUCCESS;
+}
diff --git a/test/T595-reopen.sh b/test/T595-reopen.sh
new file mode 100755
index ..7375e2ac
--- /dev/null
+++ b/test/T595-reopen.sh
@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+test_description="library reopen API"
+
+. $(dirname "$0")/test-lib.sh || exit 1
+
+add_email_corpus
+
+cat < c_head
+#include 
+#include 
+#include 
+
+int main (int argc, char** argv)
+{
+   notmuch_database_t *db;
+   char *val;
+   notmuch_status_t stat;
+   notmuch_database_mode_t mode = NOTMUCH_DATABASE_MODE_READ_ONLY;
+
+   char *msg = NULL;
+
+   for (int i = 1; i < argc; i++)
+  if (strcmp (argv[i], "%NULL%") == 0) argv[i] = NULL;
+
+   if (argv[2] && (argv[2][0] == 'w' || argv[2][0] == 'W'))
+ mode = NOTMUCH_DATABASE_MODE_READ_WRITE;
+
+ 

[PATCH 11/22] lib/open: factor out the second half of n_d_open_with_config

2021-03-14 Thread David Bremner
The idea is to allow reuse in n_d_create_with_config. This is
primarily code movement, with some changes in error messages to reduce
the number of input parameters.
---
 lib/open.cc | 115 ++--
 1 file changed, 67 insertions(+), 48 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index 0defa9c0..7673e875 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -261,55 +261,18 @@ _init_libs ()
 }
 }
 
-notmuch_status_t
-notmuch_database_open_with_config (const char *database_path,
-  notmuch_database_mode_t mode,
-  const char *config_path,
-  const char *profile,
-  notmuch_database_t **database,
-  char **status_string)
+static notmuch_status_t
+_finish_open (notmuch_database_t *notmuch,
+ const char *profile,
+ notmuch_database_mode_t mode,
+ GKeyFile *key_file,
+ char **message_ptr)
 {
 notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
-void *local = talloc_new (NULL);
-notmuch_database_t *notmuch = NULL;
-char *notmuch_path, *incompat_features;
+char *incompat_features;
 char *message = NULL;
 unsigned int version;
-GKeyFile *key_file = NULL;
-
-_init_libs ();
-
-notmuch = _alloc_notmuch ();
-if (! notmuch) {
-   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
-   goto DONE;
-}
-
-if ((status = _choose_database_path (local, config_path, profile,
-_file, _path, )))
-   goto DONE;
-
-_set_database_path (notmuch, database_path);
-
-status = _db_dir_exists (database_path, );
-if (status)
-   goto DONE;
-
-if (! (notmuch_path = talloc_asprintf (local, "%s/%s", database_path, 
".notmuch"))) {
-   message = strdup ("Out of memory\n");
-   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
-   goto DONE;
-}
-
-status = _db_dir_exists (notmuch_path, );
-if (status)
-   goto DONE;
-
-if (! (notmuch->xapian_path = talloc_asprintf (local, "%s/%s", 
notmuch_path, "xapian"))) {
-   message = strdup ("Out of memory\n");
-   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
-   goto DONE;
-}
+const char *database_path = notmuch_database_get_path (notmuch);
 
 try {
std::string last_thread_id;
@@ -332,7 +295,7 @@ notmuch_database_open_with_config (const char 
*database_path,
 "Error: Notmuch database at %s\n"
 "   has a newer database format 
version (%u) than supported by this\n"
 "   version of notmuch (%u).\n",
-notmuch_path, version, 
NOTMUCH_DATABASE_VERSION));
+database_path, version, 
NOTMUCH_DATABASE_VERSION));
notmuch_database_destroy (notmuch);
notmuch = NULL;
status = NOTMUCH_STATUS_FILE_ERROR;
@@ -342,7 +305,7 @@ notmuch_database_open_with_config (const char 
*database_path,
/* Check features. */
incompat_features = NULL;
notmuch->features = _notmuch_database_parse_features (
-   local, notmuch->xapian_db->get_metadata ("features").c_str (),
+   notmuch, notmuch->xapian_db->get_metadata ("features").c_str (),
version, mode == NOTMUCH_DATABASE_MODE_READ_WRITE ? 'w' : 'r',
_features);
if (incompat_features) {
@@ -350,7 +313,7 @@ notmuch_database_open_with_config (const char 
*database_path,
 "Error: Notmuch database at %s\n"
 "   requires features (%s)\n"
 "   not supported by this version of 
notmuch.\n",
-notmuch_path, incompat_features));
+database_path, incompat_features));
notmuch_database_destroy (notmuch);
notmuch = NULL;
status = NOTMUCH_STATUS_FILE_ERROR;
@@ -430,6 +393,62 @@ notmuch_database_open_with_config (const char 
*database_path,
notmuch = NULL;
status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
 }
+  DONE:
+if (message_ptr)
+   *message_ptr = message;
+return status;
+}
+
+notmuch_status_t
+notmuch_database_open_with_config (const char *database_path,
+  notmuch_database_mode_t mode,
+  const char *config_path,
+  const char *profile,
+  notmuch_database_t **database,
+  char **status_string)
+{
+notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
+void *local = talloc_new (NULL);
+notmuch_database_t *notmuch = NULL;
+char *notmuch_path;
+char *message = NULL;
+GKeyFile *key_file 

[PATCH 22/22] CLI/new: use configuration variable for backup directory

2021-03-14 Thread David Bremner
The stat is essentially replaced by the mkdir for error detection
purposes.  This changes the default location for backups to make
things tidier, even in non-split configurations. Hopefully there is
not too many user scripts relying on the previous location.

Because the default location may not exist, replace the use of stat
for error detection with a call to mkdir.
---
 notmuch-new.c| 18 ++
 test/T055-path-config.sh | 15 +++
 test/T530-upgrade.sh |  2 +-
 3 files changed, 22 insertions(+), 13 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index 30062512..bae28109 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -1051,28 +1051,22 @@ _maybe_upgrade (notmuch_database_t *notmuch, 
add_files_state_t *state)
 if (notmuch_database_needs_upgrade (notmuch)) {
time_t now = time (NULL);
struct tm *gm_time = gmtime ();
-   struct stat st;
int err;
notmuch_status_t status;
-   char *dot_notmuch_path = talloc_asprintf (notmuch, "%s/%s", 
state->db_path, ".notmuch");
-
+   const char *backup_dir = notmuch_config_get (notmuch, 
NOTMUCH_CONFIG_BACKUP_DIR);
const char *backup_name;
 
-   err = stat (dot_notmuch_path, );
-   if (err) {
-   if (errno == ENOENT) {
-   dot_notmuch_path = NULL;
-   } else {
-   fprintf (stderr, "Failed to stat %s: %s\n", dot_notmuch_path, 
strerror (errno));
-   return EXIT_FAILURE;
-   }
+   err = mkdir (backup_dir, 0755);
+   if (err && errno != EEXIST) {
+   fprintf (stderr, "Failed to create %s: %s\n", backup_dir, strerror 
(errno));
+   return EXIT_FAILURE;
}
 
/* since dump files are written atomically, the amount of
 * harm from overwriting one within a second seems
 * relatively small. */
backup_name = talloc_asprintf (notmuch, 
"%s/dump-%04d%02d%02dT%02d%02d%02d.gz",
-  dot_notmuch_path ? dot_notmuch_path : 
state->db_path,
+  backup_dir,
   gm_time->tm_year + 1900,
   gm_time->tm_mon + 1,
   gm_time->tm_mday,
diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
index e4812c82..d8828342 100755
--- a/test/T055-path-config.sh
+++ b/test/T055-path-config.sh
@@ -2,6 +2,8 @@
 test_description='Configuration of mail-root and database path'
 . $(dirname "$0")/test-lib.sh || exit 1
 
+test_require_external_prereq xapian-metdata
+
 backup_config () {
 local test_name=$(basename $0 .sh)
 cp ${NOTMUCH_CONFIG} notmuch-config-backup.${test_name}
@@ -13,6 +15,7 @@ restore_config () {
 unset CONFIG_PATH
 unset DATABASE_PATH
 unset NOTMUCH_PROFILE
+unset XAPIAN_PATH
 cp notmuch-config-backup.${test_name} ${NOTMUCH_CONFIG}
 }
 
@@ -25,6 +28,7 @@ split_config () {
 notmuch config set database.path $dir
 notmuch config set database.mail_root $MAIL_DIR
 DATABASE_PATH=$dir
+XAPIAN_PATH="$dir/xapian"
 }
 
 symlink_config () {
@@ -34,6 +38,7 @@ symlink_config () {
 ln -s $MAIL_DIR $dir
 notmuch config set database.path $dir
 notmuch config set database.mail_root $MAIL_DIR
+XAPIAN_PATH="$MAIL_DIR/.notmuch/xapian"
 unset DATABASE_PATH
 }
 
@@ -56,6 +61,7 @@ xdg_config () {
 mv ${NOTMUCH_CONFIG} $CONFIG_PATH
 unset NOTMUCH_CONFIG
 
+XAPIAN_PATH="${DATABASE_PATH}/xapian"
 notmuch --config=${CONFIG_PATH} config set database.mail_root 
${TMP_DIRECTORY}/mail
 notmuch --config=${CONFIG_PATH} config set database.path
 }
@@ -67,6 +73,7 @@ for config in traditional split XDG XDG+profile symlink; do
 case $config in
traditional)
backup_config
+   XAPIAN_PATH="$MAIL_DIR/.notmuch/xapian"
;;
split)
split_config
@@ -184,6 +191,14 @@ EOF
 notmuch search --output=messages '*' | sort > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "upgrade backup ($config)"
+features=$(xapian-metadata get $XAPIAN_PATH features | grep -v "^relative 
directory paths")
+xapian-metadata set $XAPIAN_PATH features "$features"
+output=$(notmuch new | grep Welcome)
+test_expect_equal \
+   "$output" \
+   "Welcome to a new version of notmuch! Your database will now be 
upgraded."
+
 restore_config
 done
 
diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
index c599dacf..cce29f45 100755
--- a/test/T530-upgrade.sh
+++ b/test/T530-upgrade.sh
@@ -5,7 +5,7 @@ test_description='database upgrades'
 test_require_external_prereq xapian-metadata
 
 XAPIAN_PATH=$MAIL_DIR/.notmuch/xapian
-BACKUP_PATH=$MAIL_DIR/.notmuch
+BACKUP_PATH=$MAIL_DIR/.notmuch/backups
 
 delete_feature () {
 local key=$1
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe 

[PATCH 14/22] lib: support splitting mail from database location.

2021-03-14 Thread David Bremner
Introduce a new configuration value for the mail root, and use it to
locate mail messages in preference to the database.path (which
previously implied the mail messages were also in this location.

Initially only a subset of the CLI is tested in a split
configuration. Further changes will be needed for the remainder of the
CLI to work in split configurations.
---
 doc/man1/notmuch-config.rst | 15 +--
 lib/config.cc   | 11 +++--
 lib/database.cc |  2 +-
 lib/message-file.c  |  2 +-
 lib/message.cc  |  2 +-
 lib/notmuch.h   |  1 +
 lib/open.cc |  5 +++
 test/T055-path-config.sh| 90 +
 test/T590-libconfig.sh  |  1 +
 9 files changed, 120 insertions(+), 9 deletions(-)
 create mode 100755 test/T055-path-config.sh

diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index bc597957..223d2de5 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -43,12 +43,21 @@ configuration file and corresponding database.
 The available configuration items are described below.
 
 **database.path**
+Notmuch will store its database here, (in
+sub-directory named ``.notmuch`` if **database.mail\_root**
+is unset).
+
+Default: ``$MAILDIR`` variable if set, otherwise ``$HOME/mail``.
+
+**database.mail_root**
 The top-level directory where your mail currently exists and to
 where mail will be delivered in the future. Files should be
-individual email messages. Notmuch will store its database within
-a sub-directory of the path configured here named ``.notmuch``.
+individual email messages.
 
-Default: ``$MAILDIR`` variable if set, otherwise ``$HOME/mail``.
+History: this configuration value was introduced in notmuch 0.32.
+
+Default: For compatibility with older configurations, the value of
+database.path is used if **database.mail\_root** is unset.
 
 **database.hook_dir**
 
diff --git a/lib/config.cc b/lib/config.cc
index 483a02ef..f2060e28 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -406,6 +406,8 @@ _notmuch_config_key_to_string (notmuch_config_key_t key)
 switch (key) {
 case NOTMUCH_CONFIG_DATABASE_PATH:
return "database.path";
+case NOTMUCH_CONFIG_MAIL_ROOT:
+   return "database.mail_root";
 case NOTMUCH_CONFIG_HOOK_DIR:
return "database.hook_dir";
 case NOTMUCH_CONFIG_EXCLUDE_TAGS:
@@ -428,7 +430,7 @@ _notmuch_config_key_to_string (notmuch_config_key_t key)
 }
 
 static const char *
-_notmuch_config_default (void *ctx, notmuch_config_key_t key)
+_notmuch_config_default (notmuch_database_t *notmuch, notmuch_config_key_t key)
 {
 char *path;
 
@@ -436,11 +438,14 @@ _notmuch_config_default (void *ctx, notmuch_config_key_t 
key)
 case NOTMUCH_CONFIG_DATABASE_PATH:
path = getenv ("MAILDIR");
if (path)
-   path = talloc_strdup (ctx, path);
+   path = talloc_strdup (notmuch, path);
else
-   path = talloc_asprintf (ctx, "%s/mail",
+   path = talloc_asprintf (notmuch, "%s/mail",
getenv ("HOME"));
return path;
+case NOTMUCH_CONFIG_MAIL_ROOT:
+   /* by default, mail root is the same as database path */
+   return notmuch_database_get_path (notmuch);
 case NOTMUCH_CONFIG_EXCLUDE_TAGS:
return "";
 case NOTMUCH_CONFIG_NEW_TAGS:
diff --git a/lib/database.cc b/lib/database.cc
index 580b2d58..248d1dc2 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1350,7 +1350,7 @@ _notmuch_database_relative_path (notmuch_database_t 
*notmuch,
 const char *db_path, *relative;
 unsigned int db_path_len;
 
-db_path = notmuch_database_get_path (notmuch);
+db_path = notmuch_config_get (notmuch, NOTMUCH_CONFIG_MAIL_ROOT);
 db_path_len = strlen (db_path);
 
 relative = path;
diff --git a/lib/message-file.c b/lib/message-file.c
index 15b0bfad..9e9b387f 100644
--- a/lib/message-file.c
+++ b/lib/message-file.c
@@ -64,7 +64,7 @@ _notmuch_message_file_open_ctx (notmuch_database_t *notmuch,
 if (unlikely (message == NULL))
return NULL;
 
-const char *prefix = notmuch_database_get_path (notmuch);
+const char *prefix = notmuch_config_get (notmuch, 
NOTMUCH_CONFIG_MAIL_ROOT);
 
 if (prefix == NULL)
goto FAIL;
diff --git a/lib/message.cc b/lib/message.cc
index 73a7805f..0c2eeab5 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1102,7 +1102,7 @@ _notmuch_message_ensure_filename_list (notmuch_message_t 
*message)
 
*colon = '\0';
 
-   db_path = notmuch_database_get_path (message->notmuch);
+   db_path = notmuch_config_get (message->notmuch, 
NOTMUCH_CONFIG_MAIL_ROOT);
 
directory = _notmuch_database_get_directory_path (local,
  message->notmuch,
diff --git a/lib/notmuch.h b/lib/notmuch.h
index b5688087..c9b01719 100644
--- a/lib/notmuch.h

[PATCH 20/22] lib/open: fix hook directory calculation in split configuration

2021-03-14 Thread David Bremner
Choose sibling directory of xapian database, as .notmuch may not
exist.

libgen.h is already used in debugger.c, so it is not a new dependency
/ potential portability problem.
---
 lib/open.cc| 6 --
 test/T400-hooks.sh | 9 -
 2 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index 8321a3be..aad7fa77 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -1,4 +1,6 @@
 #include 
+#include 
+
 #include "database-private.h"
 #include "parse-time-vrp.h"
 
@@ -92,8 +94,8 @@ _choose_hook_dir (notmuch_database_t *notmuch,
 err = stat (hook_dir, );
 if (err) {
if (errno == ENOENT) {
-   const char *database_path = notmuch_database_get_path (notmuch);
-   hook_dir = talloc_asprintf (notmuch, "%s/.notmuch/hooks", 
database_path);
+   char *notmuch_path = dirname (talloc_strdup (notmuch, 
notmuch->xapian_path));
+   hook_dir = talloc_asprintf (notmuch, "%s/hooks", notmuch_path);
} else {
IGNORE_RESULT (asprintf (message, "Error: Cannot stat %s: %s.\n",
 hook_dir, strerror (errno)));
diff --git a/test/T400-hooks.sh b/test/T400-hooks.sh
index a3dd4c63..8267d057 100755
--- a/test/T400-hooks.sh
+++ b/test/T400-hooks.sh
@@ -28,9 +28,10 @@ add_message
 # create maildir structure for notmuch-insert
 mkdir -p "$MAIL_DIR"/{cur,new,tmp}
 
-for config in traditional profile explicit XDG; do
+for config in traditional profile explicit XDG split; do
 unset NOTMUCH_PROFILE
 notmuch config set database.hook_dir
+notmuch config set database.path ${MAIL_DIR}
 case $config in
traditional)
HOOK_DIR=${MAIL_DIR}/.notmuch/hooks
@@ -50,6 +51,12 @@ for config in traditional profile explicit XDG; do
XDG)
HOOK_DIR=${HOME}/.config/notmuch/default/hooks
;;
+   split)
+   dir="$TMP_DIRECTORY/database.$test_count"
+   notmuch config set database.path $dir
+   notmuch config set database.mail_root $MAIL_DIR
+   HOOK_DIR=${dir}/hooks
+   ;;
 esac
 
 test_begin_subtest "pre-new is run [${config}]"
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 08/22] lib: remove "path" from notmuch struct

2021-03-14 Thread David Bremner
This removes duplication between the struct element and the
configuration string_map entry. Create a simple wrapper for setting
the database path that makes sure the trailing / is stripped.
---
 lib/config.cc  |  3 +++
 lib/database-private.h |  3 ---
 lib/database.cc|  2 +-
 lib/open.cc| 14 --
 4 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/lib/config.cc b/lib/config.cc
index 6993e67f..483a02ef 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -496,5 +496,8 @@ notmuch_config_set (notmuch_database_t *notmuch, 
notmuch_config_key_t key, const
 void
 _notmuch_config_cache (notmuch_database_t *notmuch, notmuch_config_key_t key, 
const char *val)
 {
+if (notmuch->config == NULL)
+   notmuch->config = _notmuch_string_map_create (notmuch);
+
 _notmuch_string_map_set (notmuch->config, _notmuch_config_key_to_string 
(key), val);
 }
diff --git a/lib/database-private.h b/lib/database-private.h
index d936b216..2900382d 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -189,9 +189,6 @@ operator& (notmuch_field_flag_t a, notmuch_field_flag_t b)
 struct _notmuch_database {
 bool exception_reported;
 
-/* Path to database parent directory and or/mail root */
-char *path;
-
 /* Path to actual database */
 const char *xapian_path;
 
diff --git a/lib/database.cc b/lib/database.cc
index fc13b53b..580b2d58 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -752,7 +752,7 @@ notmuch_database_destroy (notmuch_database_t *notmuch)
 const char *
 notmuch_database_get_path (notmuch_database_t *notmuch)
 {
-return notmuch->path;
+return notmuch_config_get (notmuch, NOTMUCH_CONFIG_DATABASE_PATH);
 }
 
 unsigned int
diff --git a/lib/open.cc b/lib/open.cc
index de5f3467..a62d4dee 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -209,6 +209,17 @@ _alloc_notmuch ()
 return notmuch;
 }
 
+static void
+_set_database_path (notmuch_database_t *notmuch,
+   const char *database_path)
+{
+char *path = talloc_strdup (notmuch, database_path);
+
+strip_trailing (path, '/');
+
+_notmuch_config_cache (notmuch, NOTMUCH_CONFIG_DATABASE_PATH, path);
+}
+
 notmuch_status_t
 notmuch_database_open_with_config (const char *database_path,
   notmuch_database_mode_t mode,
@@ -238,8 +249,7 @@ notmuch_database_open_with_config (const char 
*database_path,
 _file, _path, )))
goto DONE;
 
-notmuch->path = talloc_strdup (notmuch, database_path);
-strip_trailing (notmuch->path, '/');
+_set_database_path (notmuch, database_path);
 
 if (! (notmuch_path = talloc_asprintf (local, "%s/%s", database_path, 
".notmuch"))) {
message = strdup ("Out of memory\n");
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 19/22] lib/compact: enable split config

2021-03-14 Thread David Bremner
This promotes _choose_xapian_path from static to extern linkage in
order to share between open.cc and database.cc.
---
 lib/database.cc  | 13 -
 lib/notmuch-private.h|  5 +
 lib/open.cc  |  9 +
 test/T055-path-config.sh |  7 +++
 4 files changed, 21 insertions(+), 13 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 248d1dc2..9743c1ca 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -610,11 +610,12 @@ notmuch_database_compact_db (notmuch_database_t *notmuch,
 void *closure)
 {
 void *local;
-char *notmuch_path, *xapian_path, *compact_xapian_path;
+const char *xapian_path, *compact_xapian_path;
 const char *path;
 notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
 struct stat statbuf;
 bool keep_backup;
+char *message;
 
 ret = _notmuch_database_ensure_writable (notmuch);
 if (ret)
@@ -628,15 +629,9 @@ notmuch_database_compact_db (notmuch_database_t *notmuch,
 if (! local)
return NOTMUCH_STATUS_OUT_OF_MEMORY;
 
-if (! (notmuch_path = talloc_asprintf (local, "%s/%s", path, ".notmuch"))) 
{
-   ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
-   goto DONE;
-}
-
-if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, 
"xapian"))) {
-   ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
+ret = _notmuch_choose_xapian_path (local, path, _path, );
+if (ret)
goto DONE;
-}
 
 if (! (compact_xapian_path = talloc_asprintf (local, "%s.compact", 
xapian_path))) {
ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index ef814378..387f0c1e 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -724,6 +724,11 @@ _notmuch_config_load_defaults (notmuch_database_t *db);
 void
 _notmuch_config_cache (notmuch_database_t *db, notmuch_config_key_t key, const 
char *val);
 
+/* open.cc */
+notmuch_status_t
+_notmuch_choose_xapian_path (void *ctx, const char *database_path, const char 
**xapian_path,
+char **message);
+
 NOTMUCH_END_DECLS
 
 #ifdef __cplusplus
diff --git a/lib/open.cc b/lib/open.cc
index 3097121e..8321a3be 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -258,9 +258,9 @@ _trial_open (const char *xapian_path, char **message_ptr)
 return NOTMUCH_STATUS_SUCCESS;
 }
 
-static notmuch_status_t
-_choose_xapian_path (void *ctx, const char *database_path, const char 
**xapian_path,
-char **message_ptr)
+notmuch_status_t
+_notmuch_choose_xapian_path (void *ctx, const char *database_path,
+const char **xapian_path, char **message_ptr)
 {
 notmuch_status_t status;
 const char *trial_path, *notmuch_path;
@@ -489,7 +489,8 @@ notmuch_database_open_with_config (const char 
*database_path,
 
 _set_database_path (notmuch, database_path);
 
-status = _choose_xapian_path (local, database_path, >xapian_path, 
);
+status = _notmuch_choose_xapian_path (local, database_path,
+ >xapian_path, );
 if (status)
goto DONE;
 
diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
index fb5174ac..e4812c82 100755
--- a/test/T055-path-config.sh
+++ b/test/T055-path-config.sh
@@ -177,6 +177,13 @@ EOF
 cur_msg_filename=$(notmuch search --output=files "subject:insert-subject")
 test_expect_equal_file "$cur_msg_filename" "$gen_msg_filename"
 
+
+test_begin_subtest "compact+search ($config)"
+notmuch search --output=messages '*' | sort > EXPECTED
+notmuch compact
+notmuch search --output=messages '*' | sort > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 restore_config
 done
 
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 15/22] lib/open: check for split configuration when creating database.

2021-03-14 Thread David Bremner
The main functionality will be tested when notmuch-new is converted to
support split configuration. Here only the somewhat odd case of split
mail root which is actually symlinked to the database path is tested.
---
 lib/open.cc  | 53 ++--
 test/T055-path-config.sh | 15 ++--
 2 files changed, 53 insertions(+), 15 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index fe42f7f0..2816f72a 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -545,11 +545,12 @@ notmuch_database_create_with_config (const char 
*database_path,
 {
 notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
 notmuch_database_t *notmuch = NULL;
-char *notmuch_path = NULL;
+const char *notmuch_path = NULL;
 char *message = NULL;
 GKeyFile *key_file = NULL;
 void *local = talloc_new (NULL);
 int err;
+bool split = false;
 
 _init_libs ();
 
@@ -571,20 +572,38 @@ notmuch_database_create_with_config (const char 
*database_path,
 
 _set_database_path (notmuch, database_path);
 
-notmuch_path = talloc_asprintf (local, "%s/%s", database_path, ".notmuch");
+if (key_file && ! split) {
+   char *mail_root = canonicalize_file_name (
+   g_key_file_get_value (key_file, "database", "mail_root", NULL));
+   char *db_path = canonicalize_file_name (database_path);
 
-err = mkdir (notmuch_path, 0755);
-if (err) {
-   if (errno == EEXIST) {
-   status = NOTMUCH_STATUS_DATABASE_EXISTS;
-   talloc_free (notmuch);
-   notmuch = NULL;
-   } else {
-   IGNORE_RESULT (asprintf (, "Error: Cannot create directory 
%s: %s.\n",
-notmuch_path, strerror (errno)));
-   status = NOTMUCH_STATUS_FILE_ERROR;
+   split = (mail_root && (0 != strcmp (mail_root, db_path)));
+
+   free (mail_root);
+   free (db_path);
+}
+
+if (split) {
+   notmuch_path = database_path;
+} else {
+   if (! (notmuch_path = talloc_asprintf (local, "%s/%s", database_path, 
".notmuch"))) {
+   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+   goto DONE;
+   }
+
+   err = mkdir (notmuch_path, 0755);
+   if (err) {
+   if (errno == EEXIST) {
+   status = NOTMUCH_STATUS_DATABASE_EXISTS;
+   talloc_free (notmuch);
+   notmuch = NULL;
+   } else {
+   IGNORE_RESULT (asprintf (, "Error: Cannot create 
directory %s: %s.\n",
+notmuch_path, strerror (errno)));
+   status = NOTMUCH_STATUS_FILE_ERROR;
+   }
+   goto DONE;
}
-   goto DONE;
 }
 
 if (! (notmuch->xapian_path = talloc_asprintf (notmuch, "%s/%s", 
notmuch_path, "xapian"))) {
@@ -592,6 +611,14 @@ notmuch_database_create_with_config (const char 
*database_path,
goto DONE;
 }
 
+status = _trial_open (notmuch->xapian_path, );
+if (status == NOTMUCH_STATUS_SUCCESS) {
+   notmuch_database_destroy (notmuch);
+   notmuch = NULL;
+   status = NOTMUCH_STATUS_DATABASE_EXISTS;
+   goto DONE;
+}
+
 status = _finish_open (notmuch,
   profile,
   NOTMUCH_DATABASE_MODE_READ_WRITE,
diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
index c6920ca9..19a8a4ab 100755
--- a/test/T055-path-config.sh
+++ b/test/T055-path-config.sh
@@ -27,9 +27,17 @@ split_config () {
 DATABASE_PATH=$dir
 }
 
+symlink_config () {
+local dir
+backup_config
+dir="$TMP_DIRECTORY/link.$test_count"
+ln -s $MAIL_DIR $dir
+notmuch config set database.path $dir
+notmuch config set database.mail_root $MAIL_DIR
+unset DATABASE_PATH
+}
 
-
-for config in traditional split; do
+for config in traditional split symlink; do
 # start each set of tests with a known set of messages
 add_email_corpus
 
@@ -41,6 +49,9 @@ for config in traditional split; do
split_config
mv mail/.notmuch/xapian $DATABASE_PATH
;;
+   symlink)
+   symlink_config
+   ;;
 esac
 
 test_begin_subtest "count ($config)"
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 01/22] test/setup: add check for config exists, but no database.

2021-03-14 Thread David Bremner
This code path is not currently tested, and will need updating if the
location of the xapian database changes.
---
 test/T040-setup.sh | 16 
 1 file changed, 16 insertions(+)

diff --git a/test/T040-setup.sh b/test/T040-setup.sh
index 1223ebf7..5eb281d7 100755
--- a/test/T040-setup.sh
+++ b/test/T040-setup.sh
@@ -34,4 +34,20 @@ built_with.compact=something
 built_with.field_processor=something
 built_with.retry_lock=something"
 
+test_begin_subtest "notmuch with a config but without a database suggests 
notmuch new"
+notmuch 2>&1 | notmuch_dir_sanitize > OUTPUT
+cat < EXPECTED
+Notmuch is configured, but there's not yet a database at
+
+   MAIL_DIR/.notmuch
+
+You probably want to run "notmuch new" now to create that database.
+
+Note that the first run of "notmuch new" can take a very long time
+and that the resulting database will use roughly the same amount of
+storage space as the email being indexed.
+
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


v4 flexible path location

2021-03-14 Thread David Bremner
Since v3 [1]

- rebase against the uncrustified version (unfortunately this makes an
  interdiff too noisy to be helpful).
- improve (?) the documentation of database.mail_root in notmuch-config(1).
- tweak commit messages
- move the call to _init_libs from [10/22] to [9/22]

I'm going to mark this version as ready to apply, so object now, or
grumble later.

[1]: id:20210307172734.1091264-1-da...@tethera.net

[PATCH 09/22] lib/open: factor out library initialization
[PATCH 10/22] lib/open: reuse directory checks from n_d_c_with_config

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Clean up glib / GKeyFile memory use

2021-03-14 Thread David Bremner
David Bremner  writes:

> It turns out I was too optimistic / lazy about how the g_key_file API
> manages memory.  None of the resulting memory leaks are large (unless
> you somehow keep War and Peace in your config file), but it is more
> tidy to clean them up, and makes it easier to spot more significant
> leaks in the already noisy output from valgrind.
>
> This series applies on top of master. I'll have to rebase the other
> config series on top (and possibly clean up a few more glib related
> memory leaks).

I have a applied a rebased version (on top of the uncrustify patches) to
master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: uncrustify all the things

2021-03-13 Thread David Bremner
David Bremner  writes:
> I'll probabably apply these relatively quickly to minimimize the
> amount of pain for people (like me) rebasing patch series against
> master.

As promised, applied to master. Further cleanup welcome, preferably in a
form that survives re-running uncrustify.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] test: put shim at end of LD_PRELOAD path

2021-03-13 Thread David Bremner
Certain tools like the address-sanitizer fail if they are not the
first LD_PRELOADed library. It does not seem to matter for our shims,
as long as they are loaded before libnotmuch.
---

Using the address sanitizer with notmuch is still a manual process,
but this allows one to manually LD_PRELOAD libasan to make some more
tests pass.

 test/test-lib.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index 29baa0c1..89e218df 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -1117,7 +1117,7 @@ notmuch_with_shim () {
 base_name="$1"
 shift
 shim_file="${base_name}.so"
-LD_PRELOAD=./${shim_file}${LD_PRELOAD:+:$LD_PRELOAD} notmuch-shared "$@"
+LD_PRELOAD=${LD_PRELOAD:+:$LD_PRELOAD}:./${shim_file} notmuch-shared "$@"
 }
 
 # Creates a script that counts how much time it is executed and calls
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 1/4] lib: run uncrustify

2021-03-12 Thread David Bremner
From: uncrustify 

This is the result of running

 $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc

in the lib directory
---
 lib/config.cc   | 72 +
 lib/database.cc | 24 --
 lib/directory.cc|  4 ++-
 lib/features.cc |  2 +-
 lib/index.cc| 28 ++--
 lib/indexopts.c |  1 +
 lib/message-file.c  |  5 +--
 lib/message-property.cc |  2 ++
 lib/message.cc  | 21 +++-
 lib/notmuch-private.h   |  8 ++---
 lib/notmuch.h   |  7 ++--
 lib/open.cc | 14 
 lib/parse-time-vrp.cc   |  4 +--
 lib/prefix.cc   |  4 +--
 lib/string-map.c|  6 ++--
 lib/thread-fp.cc|  6 ++--
 lib/thread.cc   |  3 +-
 17 files changed, 130 insertions(+), 81 deletions(-)

diff --git a/lib/config.cc b/lib/config.cc
index 948751bc..e6b660a9 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -38,7 +38,7 @@ struct _notmuch_config_values {
 void *children; /* talloc_context */
 };
 
-static const char * _notmuch_config_key_to_string (notmuch_config_key_t key);
+static const char *_notmuch_config_key_to_string (notmuch_config_key_t key);
 
 static int
 _notmuch_config_list_destroy (notmuch_config_list_t *list)
@@ -104,7 +104,7 @@ notmuch_database_get_config (notmuch_database_t *notmuch,
 const char *key,
 char **value)
 {
-const char* stored_val;
+const char *stored_val;
 notmuch_status_t status;
 
 if (! notmuch->config) {
@@ -119,7 +119,7 @@ notmuch_database_get_config (notmuch_database_t *notmuch,
 if (! stored_val) {
/* XXX in principle this API should be fixed so empty string
 * is distinguished from not found */
-   *value = strdup("");
+   *value = strdup ("");
 } else {
*value = strdup (stored_val);
 }
@@ -152,7 +152,8 @@ notmuch_database_get_config_list (notmuch_database_t 
*notmuch,
talloc_set_destructor (list, _notmuch_config_list_destroy);
 
 } catch (const Xapian::Error ) {
-   _notmuch_database_log (notmuch, "A Xapian exception occurred getting 
metadata iterator: %s.\n",
+   _notmuch_database_log (notmuch,
+  "A Xapian exception occurred getting metadata 
iterator: %s.\n",
   error.get_msg ().c_str ());
notmuch->exception_reported = true;
status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
@@ -167,7 +168,7 @@ notmuch_database_get_config_list (notmuch_database_t 
*notmuch,
if (status != NOTMUCH_STATUS_XAPIAN_EXCEPTION)
_notmuch_config_list_destroy (list);
}
-}  else {
+} else {
talloc_set_destructor (list, _notmuch_config_list_destroy);
 }
 
@@ -183,7 +184,9 @@ notmuch_config_list_valid (notmuch_config_list_t *metadata)
 return true;
 }
 
-static inline char * _key_from_iterator (notmuch_config_list_t *list) {
+static inline char *
+_key_from_iterator (notmuch_config_list_t *list)
+{
 return talloc_strdup (list, (*list->iterator).c_str () + 
CONFIG_PREFIX.length ());
 }
 
@@ -239,7 +242,7 @@ _notmuch_config_load_from_database (notmuch_database_t 
*notmuch)
 if (notmuch->config == NULL)
notmuch->config = _notmuch_string_map_create (notmuch);
 
-if (unlikely(notmuch->config == NULL))
+if (unlikely (notmuch->config == NULL))
return NOTMUCH_STATUS_OUT_OF_MEMORY;
 
 status = notmuch_database_get_config_list (notmuch, "", );
@@ -267,7 +270,7 @@ notmuch_config_get_values (notmuch_database_t *notmuch, 
notmuch_config_key_t key
goto DONE;
 
 values = talloc (notmuch, notmuch_config_values_t);
-if (unlikely(! values))
+if (unlikely (! values))
goto DONE;
 
 values->children = talloc_new (values);
@@ -279,17 +282,18 @@ notmuch_config_get_values (notmuch_database_t *notmuch, 
notmuch_config_key_t key
 values->iterator = strsplit_len (values->string, ';', &(values->tok_len));
 ok = true;
 
- DONE:
-if (!ok) {
+  DONE:
+if (! ok) {
if (values)
-   talloc_free(values);
+   talloc_free (values);
return NULL;
 }
 return values;
 }
 
 notmuch_bool_t
-notmuch_config_values_valid (notmuch_config_values_t *values) {
+notmuch_config_values_valid (notmuch_config_values_t *values)
+{
 if (! values)
return false;
 
@@ -297,12 +301,14 @@ notmuch_config_values_valid (notmuch_config_values_t 
*values) {
 }
 
 const char *
-notmuch_config_values_get (notmuch_config_values_t *values) {
+notmuch_config_values_get (notmuch_config_values_t *values)
+{
 return talloc_strndup (values, values->iterator, values->tok_len);
 }
 
 void
-notmuch_config_values_start (notmuch_config_values_t *values) {
+notmuch_config_values_start (notmuch_config_values_t *values)
+{
 if (values == NULL)
return;
 if (values->children) {
@@ -315,13 +321,15 @@ 

[PATCH 3/4] test: run uncrustify

2021-03-12 Thread David Bremner
From: uncrustify 

This is the result of running:

$ uncrustify --replace --config ../devel/uncrustify.cfg *.cc *.c *.h

in the test directory.
---
 test/ghost-report.cc| 1 +
 test/make-db-version.cc | 1 +
 test/random-corpus.c| 1 +
 3 files changed, 3 insertions(+)

diff --git a/test/ghost-report.cc b/test/ghost-report.cc
index fad9a71d..9d9e7a7f 100644
--- a/test/ghost-report.cc
+++ b/test/ghost-report.cc
@@ -12,5 +12,6 @@ main (int argc, char **argv)
 }
 
 Xapian::Database db (argv[1]);
+
 std::cout << db.get_termfreq ("Tghost") << std::endl;
 }
diff --git a/test/make-db-version.cc b/test/make-db-version.cc
index 78feaf72..238584e2 100644
--- a/test/make-db-version.cc
+++ b/test/make-db-version.cc
@@ -17,6 +17,7 @@ main (int argc, char **argv)
 }
 
 std::string nmpath (argv[1]);
+
 nmpath += "/.notmuch";
 if (mkdir (nmpath.c_str (), 0777) < 0) {
perror (("failed to create " + nmpath).c_str ());
diff --git a/test/random-corpus.c b/test/random-corpus.c
index ff413252..d0354e35 100644
--- a/test/random-corpus.c
+++ b/test/random-corpus.c
@@ -190,6 +190,7 @@ main (int argc, char **argv)
 srandom (seed);
 
 int count;
+
 for (count = 0; count < num_messages; count++) {
int j;
/* explicitly allow zero tags */
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 2/4] util: run uncrustify

2021-03-12 Thread David Bremner
From: uncrustify 

This is the result of running

 $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h

in the util directory
---
 util/crypto.c  | 18 --
 util/crypto.h  |  6 --
 util/gmime-extra.c |  1 +
 util/repair.c  | 14 --
 util/string-util.c |  3 ++-
 util/zlib-extra.c  |  3 ++-
 util/zlib-extra.h  |  2 +-
 7 files changed, 30 insertions(+), 17 deletions(-)

diff --git a/util/crypto.c b/util/crypto.c
index c09f467b..156a6550 100644
--- a/util/crypto.c
+++ b/util/crypto.c
@@ -48,7 +48,8 @@ _notmuch_crypto_decrypt (bool *attempted,
notmuch_message_properties_t *list = NULL;
 
for (list = notmuch_message_get_properties (message, "session-key", 
TRUE);
-notmuch_message_properties_valid (list); 
notmuch_message_properties_move_to_next (list)) {
+notmuch_message_properties_valid (list); 
notmuch_message_properties_move_to_next (
+list)) {
if (err && *err) {
g_error_free (*err);
*err = NULL;
@@ -61,12 +62,13 @@ _notmuch_crypto_decrypt (bool *attempted,
  
notmuch_message_properties_value (list),
  decrypt_result, err);
} else if (GMIME_IS_APPLICATION_PKCS7_MIME (part)) {
-   GMimeApplicationPkcs7Mime *pkcs7 = GMIME_APPLICATION_PKCS7_MIME 
(part);
+   GMimeApplicationPkcs7Mime *pkcs7 = GMIME_APPLICATION_PKCS7_MIME 
(part);
GMimeSecureMimeType type = 
g_mime_application_pkcs7_mime_get_smime_type (pkcs7);
if (type == GMIME_SECURE_MIME_TYPE_ENVELOPED_DATA) {
ret = g_mime_application_pkcs7_mime_decrypt (pkcs7,
 
GMIME_DECRYPT_NONE,
-
notmuch_message_properties_value (list),
+
notmuch_message_properties_value (
+list),
 
decrypt_result, err);
}
}
@@ -90,6 +92,7 @@ _notmuch_crypto_decrypt (bool *attempted,
 if (attempted)
*attempted = true;
 GMimeDecryptFlags flags = GMIME_DECRYPT_NONE;
+
 if (decrypt == NOTMUCH_DECRYPT_TRUE && decrypt_result)
flags |= GMIME_DECRYPT_EXPORT_SESSION_KEY;
 if (GMIME_IS_MULTIPART_ENCRYPTED (part)) {
@@ -128,7 +131,8 @@ _notmuch_message_crypto_new (void *ctx)
 }
 
 notmuch_status_t
-_notmuch_message_crypto_potential_sig_list (_notmuch_message_crypto_t 
*msg_crypto, GMimeSignatureList *sigs)
+_notmuch_message_crypto_potential_sig_list (_notmuch_message_crypto_t 
*msg_crypto,
+   GMimeSignatureList *sigs)
 {
 if (! msg_crypto)
return NOTMUCH_STATUS_NULL_POINTER;
@@ -157,7 +161,8 @@ _notmuch_message_crypto_potential_sig_list 
(_notmuch_message_crypto_t *msg_crypt
 
 
 bool
-_notmuch_message_crypto_potential_payload (_notmuch_message_crypto_t 
*msg_crypto, GMimeObject *part, GMimeObject *parent, int childnum)
+_notmuch_message_crypto_potential_payload (_notmuch_message_crypto_t 
*msg_crypto, GMimeObject *part,
+  GMimeObject *parent, int childnum)
 {
 const char *protected_headers = NULL;
 const char *forwarded = NULL;
@@ -175,7 +180,8 @@ _notmuch_message_crypto_potential_payload 
(_notmuch_message_crypto_t *msg_crypto
  * encryption protocol should be "control information" metadata,
  * not payload.  So we skip it. (see
  * https://tools.ietf.org/html/rfc1847#page-8) */
-if (parent && GMIME_IS_MULTIPART_ENCRYPTED (parent) && childnum == 
GMIME_MULTIPART_ENCRYPTED_VERSION) {
+if (parent && GMIME_IS_MULTIPART_ENCRYPTED (parent) && childnum ==
+   GMIME_MULTIPART_ENCRYPTED_VERSION) {
const char *enc_type = g_mime_object_get_content_type_parameter 
(parent, "protocol");
GMimeContentType *ct = g_mime_object_get_content_type (part);
if (ct && enc_type) {
diff --git a/util/crypto.h b/util/crypto.h
index 4fa5599c..3c5d384b 100644
--- a/util/crypto.h
+++ b/util/crypto.h
@@ -80,7 +80,8 @@ _notmuch_message_crypto_new (void *ctx);
  * consider a particular signature as relevant for the message.
  */
 notmuch_status_t
-_notmuch_message_crypto_potential_sig_list (_notmuch_message_crypto_t 
*msg_crypto, GMimeSignatureList *sigs);
+_notmuch_message_crypto_potential_sig_list (_notmuch_message_crypto_t 
*msg_crypto,
+   GMimeSignatureList *sigs);
 
 /* call successful_decryption during a depth-first-search on a message
  * to indicate that a part was successfully decrypted.
@@ -95,7 +96,8 @@ _notmuch_message_crypto_successful_decryption 
(_notmuch_message_crypto_t *msg_cr
  * this message.
 

uncrustify all the things

2021-03-12 Thread David Bremner
This replaces and expands id:20210307171658.1084026-1-da...@tethera.net

There has been some style drift in the intervening ~ 2 years since we
last did these. The addition of a line length limit also triggers some
changes.

compat and parse-date-time are unchanged since the last uncrustify run.

I did ignore a couple of suggestions from uncrustify. One is aligning

foo->bar->
   flub->gub

as

foo->bar->
flub->gub


The other is line breaks in the table of status-codes in
lib/notmuch-private.h. I'm not sure the best approach to the latter.

I'll probabably apply these relatively quickly to minimimize the
amount of pain for people (like me) rebasing patch series against
master.


___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 4/4] cli: run uncrustify

2021-03-12 Thread David Bremner
From: uncrustify 

This is the result of running

 $ uncrustify --replace --config devel/uncrustify.cfg *.c *.h

in the top level source directory

Line breaks were then adjusted manually to keep argc and argv
together.
---
 command-line-arguments.c | 18 ++-
 gmime-filter-reply.h | 16 +-
 mime-node.c  | 24 +---
 notmuch-client.h | 47 ++--
 notmuch-compact.c|  3 ++-
 notmuch-config.c |  7 --
 notmuch-count.c  | 13 ++-
 notmuch-dump.c   |  5 +++--
 notmuch-insert.c |  3 ++-
 notmuch-new.c| 21 +++---
 notmuch-reindex.c|  3 ++-
 notmuch-reply.c  |  7 --
 notmuch-restore.c| 10 ++---
 notmuch-search.c |  9 +---
 notmuch-show.c   | 37 +--
 notmuch-tag.c|  3 ++-
 notmuch-time.c   |  4 ++--
 notmuch.c| 33 ++--
 sprinter-json.c  |  2 +-
 sprinter-sexp.c  |  2 +-
 sprinter-text.c  |  2 +-
 21 files changed, 177 insertions(+), 92 deletions(-)

diff --git a/command-line-arguments.c b/command-line-arguments.c
index 169b12a3..5dea8281 100644
--- a/command-line-arguments.c
+++ b/command-line-arguments.c
@@ -47,17 +47,23 @@ _process_keyword_arg (const notmuch_opt_desc_t *arg_desc, 
char next,
continue;
 
*arg_desc->opt_keyword = keywords->value;
-   fprintf (stderr, "Warning: No known keyword option given for 
\"%s\", choosing value \"%s\"."
-"  Please specify the argument explicitly!\n", 
arg_desc->name, arg_desc->keyword_no_arg_value);
+   fprintf (stderr,
+"Warning: No known keyword option given for \"%s\", 
choosing value \"%s\"."
+"  Please specify the argument explicitly!\n", 
arg_desc->name,
+arg_desc->keyword_no_arg_value);
 
return OPT_GIVEBACK;
}
-   fprintf (stderr, "No matching keyword for option \"%s\" and default 
value \"%s\" is invalid.\n", arg_str, arg_desc->name);
+   fprintf (stderr,
+"No matching keyword for option \"%s\" and default value 
\"%s\" is invalid.\n",
+arg_str,
+arg_desc->name);
return OPT_FAILED;
 }
 
 if (next != '\0')
-   fprintf (stderr, "Unknown keyword argument \"%s\" for option 
\"%s\".\n", arg_str, arg_desc->name);
+   fprintf (stderr, "Unknown keyword argument \"%s\" for option 
\"%s\".\n", arg_str,
+arg_desc->name);
 else
fprintf (stderr, "Option \"%s\" needs a keyword argument.\n", 
arg_desc->name);
 return OPT_FAILED;
@@ -74,7 +80,8 @@ _process_boolean_arg (const notmuch_opt_desc_t *arg_desc, 
char next,
 } else if (strcmp (arg_str, "false") == 0) {
value = false;
 } else {
-   fprintf (stderr, "Unknown argument \"%s\" for (boolean) option 
\"%s\".\n", arg_str, arg_desc->name);
+   fprintf (stderr, "Unknown argument \"%s\" for (boolean) option 
\"%s\".\n", arg_str,
+arg_desc->name);
return OPT_FAILED;
 }
 
@@ -202,6 +209,7 @@ parse_option (int argc, char **argv, const 
notmuch_opt_desc_t *options, int opt_
 const notmuch_opt_desc_t *try;
 
 const char *next_arg = NULL;
+
 if (opt_index < argc - 1  && strncmp (argv[opt_index + 1], "--", 2) != 0)
next_arg = argv[opt_index + 1];
 
diff --git a/gmime-filter-reply.h b/gmime-filter-reply.h
index 5a1e606e..7cdefcd1 100644
--- a/gmime-filter-reply.h
+++ b/gmime-filter-reply.h
@@ -24,11 +24,17 @@
 G_BEGIN_DECLS
 
 #define GMIME_TYPE_FILTER_REPLY(g_mime_filter_reply_get_type ())
-#define GMIME_FILTER_REPLY(obj)(G_TYPE_CHECK_INSTANCE_CAST ((obj), 
GMIME_TYPE_FILTER_REPLY, GMimeFilterReply))
-#define GMIME_FILTER_REPLY_CLASS(klass)(G_TYPE_CHECK_CLASS_CAST ((klass), 
GMIME_TYPE_FILTER_REPLY, GMimeFilterReplyClass))
-#define GMIME_IS_FILTER_REPLY(obj) (G_TYPE_CHECK_INSTANCE_TYPE ((obj), 
GMIME_TYPE_FILTER_REPLY))
-#define GMIME_IS_FILTER_REPLY_CLASS(klass) (G_TYPE_CHECK_CLASS_TYPE ((klass), 
GMIME_TYPE_FILTER_REPLY))
-#define GMIME_FILTER_REPLY_GET_CLASS(obj)  (G_TYPE_INSTANCE_GET_CLASS ((obj), 
GMIME_TYPE_FILTER_REPLY, GMimeFilterReplyClass))
+#define GMIME_FILTER_REPLY(obj)(G_TYPE_CHECK_INSTANCE_CAST ((obj), 
\
+   
GMIME_TYPE_FILTER_REPLY, \
+   
GMimeFilterReply))
+#define GMIME_FILTER_REPLY_CLASS(klass)(G_TYPE_CHECK_CLASS_CAST ((klass), 
GMIME_TYPE_FILTER_REPLY, \
+
GMimeFilterReplyClass))
+#define GMIME_IS_FILTER_REPLY(obj) (G_TYPE_CHECK_INSTANCE_TYPE ((obj), 
\
+ 

Re: [PATCH] test: clean up some extra whitespace.

2021-03-12 Thread David Bremner
David Bremner  writes:

> The extra space is mainly just untidy.

Applied to master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] test: T020-compact.sh: fix work directory location

2021-03-12 Thread David Bremner
Tomi Ollila  writes:

> Fix use of $TEST_DIRECTORY ($NOTMUCH_BUILDDIR/test/) with use of
> $TMP_DIRECTORY ($NOTMUCH_BUILDDIR/test/tmp.T020-compact/ in case
> of T020-compact.sh) as root directory where to write test files
> and directories.

Thanks for catching this. I had previously noticed the odd location, but
not got as far as fixing it.

applied to master.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] devel/uncrustify: add line length limits.

2021-03-12 Thread David Bremner
David Bremner  writes:

> A generous limit of 102 is chosen to moderate the amount of resulting
> reformatting.

applied to master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] devel/uncrustify: add line length limits.

2021-03-07 Thread David Bremner
A generous limit of 102 is chosen to moderate the amount of resulting
reformatting.
---
 devel/uncrustify.cfg | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/devel/uncrustify.cfg b/devel/uncrustify.cfg
index c36c33d6..d203d4e1 100644
--- a/devel/uncrustify.cfg
+++ b/devel/uncrustify.cfg
@@ -119,3 +119,9 @@ cmt_star_cont   = true
 # indent_brace = 0
 
 indent_class = true
+
+# line width / line splitting
+code_width102
+ls_for_split_full True
+ls_func_split_fullTrue
+ls_code_width True
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Notmuch and backups

2021-03-07 Thread David Bremner
Matt Armstrong  writes:
>
> David, your recent changes that allow configurable separation of the
> mail store and .notmuch files look like good ideas to me.

Thanks!

> Separately, I've recently set up backups of my system and noticed that
> the Xapian files are an order of magnitude larger than the notmuch dumps
> (3.6 GiB vs. 15 MiB).  Is there any turn key code in notmuch itself to
> keep a rolling set of "recent" notmuch dumps?
>

There is nothing turnkey. One thing that might be of interest is the
ability to do incremental backups via lastmod: queries. This would
require seperately tracking the output of "notmuch count --lastmod
'*'. It's not clear the extra complexity is worth for daily backups.

Personally I backup tags indirectly as part of a tag synchronization
script called "gitmuch" that I write many years ago. Others report doing
the same thing with nmbug, which is shipped with notmuch source.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 07/22] lib/open: allocate notmuch_t struct early

2021-03-07 Thread David Bremner
This gives more flexibility in restructuring the database opening
code.
---
 lib/open.cc | 49 -
 1 file changed, 32 insertions(+), 17 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index e1732d35..21a6b0f4 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -192,6 +192,23 @@ _choose_database_path (void *ctx,
 return NOTMUCH_STATUS_SUCCESS;
 }
 
+notmuch_database_t *
+_alloc_notmuch ()
+{
+notmuch_database_t *notmuch;
+
+notmuch = talloc_zero (NULL, notmuch_database_t);
+if (! notmuch)
+   return NULL;
+
+notmuch->exception_reported = false;
+notmuch->status_string = NULL;
+notmuch->writable_xapian_db = NULL;
+notmuch->atomic_nesting = 0;
+notmuch->view = 1;
+return notmuch;
+}
+
 notmuch_status_t
 notmuch_database_open_with_config (const char *database_path,
   notmuch_database_mode_t mode,
@@ -211,10 +228,19 @@ notmuch_database_open_with_config (const char 
*database_path,
 GKeyFile *key_file = NULL;
 static int initialized = 0;
 
+notmuch = _alloc_notmuch ();
+if (! notmuch) {
+   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+   goto DONE;
+}
+
 if ((status = _choose_database_path (local, config_path, profile,
 _file, _path, )))
goto DONE;
 
+notmuch->path = talloc_strdup (notmuch, database_path);
+strip_trailing (notmuch->path, '/');
+
 if (! (notmuch_path = talloc_asprintf (local, "%s/%s", database_path, 
".notmuch"))) {
message = strdup ("Out of memory\n");
status = NOTMUCH_STATUS_OUT_OF_MEMORY;
@@ -229,6 +255,12 @@ notmuch_database_open_with_config (const char 
*database_path,
goto DONE;
 }
 
+if (! (notmuch->xapian_path = talloc_asprintf (local, "%s/%s", 
notmuch_path, "xapian"))) {
+   message = strdup ("Out of memory\n");
+   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+   goto DONE;
+}
+
 /* Initialize the GLib type system and threads */
 #if ! GLIB_CHECK_VERSION (2, 35, 1)
 g_type_init ();
@@ -240,23 +272,6 @@ notmuch_database_open_with_config (const char 
*database_path,
initialized = 1;
 }
 
-notmuch = talloc_zero (NULL, notmuch_database_t);
-notmuch->exception_reported = false;
-notmuch->status_string = NULL;
-notmuch->path = talloc_strdup (notmuch, database_path);
-
-strip_trailing (notmuch->path, '/');
-
-notmuch->writable_xapian_db = NULL;
-notmuch->atomic_nesting = 0;
-notmuch->view = 1;
-
-if (! (notmuch->xapian_path = talloc_asprintf (notmuch, "%s/%s", 
notmuch_path, "xapian"))) {
-   message = strdup ("Out of memory\n");
-   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
-   goto DONE;
-}
-
 try {
std::string last_thread_id;
std::string last_mod;
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 03/22] lib: save path of xapian database in notmuch struct.

2021-03-07 Thread David Bremner
This will allow re-opening in a different mode (read/write
vs. read-only) with current Xapian API.
---
 lib/database-private.h |  4 
 lib/open.cc| 19 ++-
 2 files changed, 14 insertions(+), 9 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index d83cf0d0..d936b216 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -189,8 +189,12 @@ operator& (notmuch_field_flag_t a, notmuch_field_flag_t b)
 struct _notmuch_database {
 bool exception_reported;
 
+/* Path to database parent directory and or/mail root */
 char *path;
 
+/* Path to actual database */
+const char *xapian_path;
+
 int atomic_nesting;
 /* true if changes have been made in this atomic section */
 bool atomic_dirty;
diff --git a/lib/open.cc b/lib/open.cc
index 5336a93f..c9cb78e6 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -199,7 +199,7 @@ notmuch_database_open_with_config (const char 
*database_path,
 notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
 void *local = talloc_new (NULL);
 notmuch_database_t *notmuch = NULL;
-char *notmuch_path, *xapian_path, *incompat_features;
+char *notmuch_path, *incompat_features;
 char *message = NULL;
 struct stat st;
 int err;
@@ -225,12 +225,6 @@ notmuch_database_open_with_config (const char 
*database_path,
goto DONE;
 }
 
-if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, 
"xapian"))) {
-   message = strdup ("Out of memory\n");
-   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
-   goto DONE;
-}
-
 /* Initialize the GLib type system and threads */
 #if ! GLIB_CHECK_VERSION (2, 35, 1)
 g_type_init ();
@@ -252,16 +246,23 @@ notmuch_database_open_with_config (const char 
*database_path,
 notmuch->writable_xapian_db = NULL;
 notmuch->atomic_nesting = 0;
 notmuch->view = 1;
+
+if (! (notmuch->xapian_path = talloc_asprintf (notmuch, "%s/%s", 
notmuch_path, "xapian"))) {
+   message = strdup ("Out of memory\n");
+   status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+   goto DONE;
+}
+
 try {
std::string last_thread_id;
std::string last_mod;
 
if (mode == NOTMUCH_DATABASE_MODE_READ_WRITE) {
-   notmuch->writable_xapian_db = new Xapian::WritableDatabase 
(xapian_path,
+   notmuch->writable_xapian_db = new Xapian::WritableDatabase 
(notmuch->xapian_path,

DB_ACTION);
notmuch->xapian_db = notmuch->writable_xapian_db;
} else {
-   notmuch->xapian_db = new Xapian::Database (xapian_path);
+   notmuch->xapian_db = new Xapian::Database (notmuch->xapian_path);
}
 
/* Check version.  As of database version 3, we represent
-- 
2.30.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


  1   2   3   4   5   6   7   8   9   10   >