Re: v1: convert remaining CLI to new configuration

David Bremner writes: > This completes the planned overhaul of configuration handling. It > obsoletes [1], and needs to be applied on top of [2]. I will post a > rebased version of [2] fairly soon. > > [1]: id:20210220164448.3956011-1-da...@tethera.net > [2]:

[PATCH 16/22] CLI/new: support split database and mail location

This adds new state variable for the mail root, and uses it most places db_path was used. The notable exception is dumps from backups. --- notmuch-new.c| 54 ++-- test/T055-path-config.sh | 41 ++ 2 files changed, 76

[PATCH 04/22] lib: support reopening databases for write access.

In the future xapian will apparently support this more conveniently for the cases other than READ_ONLY => READ_ONLY --- lib/database.cc | 23 lib/notmuch.h | 6 +++ lib/open.cc | 45 test/T595-reopen.sh | 125

[PATCH 17/22] lib/open: support XDG_DATA_HOME as a fallback database location.

This changes some error reporting, either intentionally by reporting the highest level missing directory, or by side effect from looking in XDG locations when given null database location. --- lib/open.cc | 18 +++--- test/T055-path-config.sh | 35

[PATCH 14/22] support splitting mail from database location.

Introduce a new configuration value for the mail root, and use it in preference to the database.path (which previously implied the mail was also in this location. Further changes to the CLI will be needed to work in this split configuration. --- doc/man1/notmuch-config.rst | 14 --

[PATCH 19/22] lib/compact: enable split config

This promotes _choose_xapian_path from static to extern linkage in order to share between open.cc and database.cc. --- lib/database.cc | 13 - lib/notmuch-private.h| 4 lib/open.cc | 6 +++--- test/T055-path-config.sh | 7 +++ 4 files changed, 18

[PATCH 08/22] lib: remove "path" from notmuch struct

This removes duplication between the struct element and the configuration string_map entry. --- lib/config.cc | 3 +++ lib/database-private.h | 3 --- lib/database.cc| 2 +- lib/open.cc| 13 +++-- 4 files changed, 15 insertions(+), 6 deletions(-) diff

[PATCH 20/22] lib/open: fix hook directory calculation in split configuration

Choose sibling directory of xapian database, as .notmuch may not exist. --- lib/open.cc| 6 -- test/T400-hooks.sh | 9 - 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/lib/open.cc b/lib/open.cc index 1e2a62cd..4324deae 100644 --- a/lib/open.cc +++

[PATCH 06/22] lib/open: support NOTMUCH_DATABASE environment variable

The additional code is trivial, but making sure we get the priority of various options correct takes a few tests. --- lib/open.cc| 4 +++ test/T590-libconfig.sh | 67 +- 2 files changed, 70 insertions(+), 1 deletion(-) diff --git a/lib/open.cc

[PATCH 13/22] lib/open: Use check for existing database by trial opening

This is a bit heavyweight for now, but it will make more sense when we check multiple locations for the xapian database. --- lib/open.cc| 67 +++--- test/T360-symbol-hiding.sh | 2 +- 2 files changed, 49 insertions(+), 20 deletions(-) diff --git

[PATCH 03/22] lib: save path of xapian database in notmuch struct.

This will allow re-opening in a different mode (read/write vs. read-only) with current Xapian API. --- lib/database-private.h | 4 lib/open.cc| 19 ++- 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/lib/database-private.h b/lib/database-private.h

[PATCH 01/22] test/setup: add check for config exists, but no database.

This code path is not currently tested, and will need updating if the location of the xapian database changes. --- test/T040-setup.sh | 16 1 file changed, 16 insertions(+) diff --git a/test/T040-setup.sh b/test/T040-setup.sh index 1223ebf7..5eb281d7 100755 ---

[PATCH 15/22] lib/open: check for split configuration when creating database.

This will be tested when notmuch-new is converted to support split configuration. --- lib/open.cc | 53 ++-- test/T055-path-config.sh | 15 ++-- 2 files changed, 53 insertions(+), 15 deletions(-) diff --git a/lib/open.cc b/lib/open.cc

[PATCH 18/22] CLI/insert: support split database and mail root

The new test is in T055-path-config because it uses the helper function split_config, and because it seems easier to follow the database path related tests in one place. --- notmuch-insert.c | 12 +++- test/T055-path-config.sh | 10 ++ 2 files changed, 13 insertions(+), 9

[PATCH 05/22] CLI/show: complete conversion to new configuration framework.

In order to open the database in main() for this command, we may need to re-open it in the (possibly less common) case where crypto options require it. --- notmuch-show.c | 26 +++--- notmuch.c| 2 +- test/T035-read-config.sh | 2 -- 3 files

[PATCH 09/22] lib/open: factor out library initialization

This is slightly more tidy, but more importantly it allows for re-use of this code in n_d_create_with_config. --- lib/open.cc | 29 ++--- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/lib/open.cc b/lib/open.cc index d599aeac..7f25bd36 100644 ---

[PATCH 11/22] lib/open: factor out the second half of n_d_open_with_config

The idea is to allow reuse in n_d_create_with_config. This is primarily code movement, with some changes in error messages to reduce the number of input parameters. --- lib/open.cc | 115 +++- 1 file changed, 68 insertions(+), 47 deletions(-) diff

[PATCH 21/22] lib/config: add configuration variable for backup directory

Like the hook directory, we primarily need a way to communicate this directory between various components, but we may as well let the user configure it. --- lib/config.cc | 3 +++ lib/notmuch.h | 1 + lib/open.cc| 47 --

[PATCH 10/22] lib/open: reuse directory checks from n_d_c_with_config

Make checks more uniform between creating new databases and opening existing ones. --- lib/open.cc | 73 +--- test/T030-config.sh | 2 +- test/T560-lib-error.sh | 4 +-- test/T750-user-header.sh | 2 +- 4 files changed, 50 insertions(+),

[PATCH 12/22] lib/open: use _finish_open in n_d_create_with_config

This avoids reading the configuration file twice. --- lib/open.cc | 19 +++ 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/lib/open.cc b/lib/open.cc index d052f95d..ebf70c15 100644 --- a/lib/open.cc +++ b/lib/open.cc @@ -429,8 +429,7 @@

[PATCH 22/22] CLI/new: use configuration variable for backup directory

The stat is dropped to avoid a race condition between stat and mkdir. This changes the default location for backups to make things tidier. Hopefully there is not user scripts relying on this location. --- notmuch-new.c| 18 ++ test/T055-path-config.sh | 15

[PATCH 02/22] lib: publish API for notmuch_database_reopen

Include the (currently unused) mode argument which will specify which mode to re-open the database in. Functionality and docs to be finalized in a followup commit. --- lib/database.cc | 3 ++- lib/message.cc| 3 ++- lib/notmuch-private.h | 3 --- lib/notmuch.h | 7 +++ 4

[PATCH 07/22] lib/open: allocate notmuch_t struct early

This gives more flexibility in restructuring the database opening code. --- lib/open.cc | 46 +- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/lib/open.cc b/lib/open.cc index f08b5de5..c7fb6f81 100644 --- a/lib/open.cc +++

v2 support flexible database location

This obsoletes [1]. It is mainly a rebase against master, but also adds one test (see below). I have been running this code in production for a few weeks now, although without using any of the fancy new features. [1]: id:20210208004109.1192719-1-da...@tethera.net diff --git a/test/T040-setup.sh

[PATCH 26/27] CLI: drop notmuch_config_t from subcommand interface.

At this point it is unused in all subcommands. --- notmuch-client.h | 28 ++-- notmuch-compact.c | 2 +- notmuch-config.c | 6 +++--- notmuch-count.c | 2 +- notmuch-dump.c| 2 +- notmuch-insert.c | 2 +- notmuch-new.c | 2 +- notmuch-reindex.c | 2 +-

[PATCH 02/27] test: convert random-corpus to use n_d_open_with_config

Remove one more usage of notmuch_config_get_database_path --- test/random-corpus.c | 13 ++--- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/test/random-corpus.c b/test/random-corpus.c index ff413252..67e106e0 100644 --- a/test/random-corpus.c +++ b/test/random-corpus.c @@

[PATCH 07/27] lib/config: set defaults for user full name

This just copies code from from the CLI into the library. New test infrastructure is needed because apparently we have never tested this code path. --- lib/config.cc | 48 +- test/T590-libconfig.sh | 13 test/test-lib.sh | 7

[PATCH 13/27] CLI/setup: switch to new configuration framework

Most of the changes are the elimination of notmuch_config_t accessor use. We also migrate some of the diagnostics to the top level where we are opening the files in question. --- notmuch-setup.c | 55 +++-- notmuch.c | 21 --- 2

[PATCH 10/27] CLI: load merged config at top level

This paves the way for the conversion of the remaining subcommands to the new configuration framework. --- notmuch-client.h | 1 + notmuch.c| 29 ++--- 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/notmuch-client.h b/notmuch-client.h index

[PATCH 24/27] CLI/config: support set/get with split configuration

The code change is tiny, and transient, but the tests will serve as regression tests the config file opening is moved into subcommands. --- notmuch.c| 5 +++- test/T055-path-config.sh | 60 +++- 2 files changed, 63 insertions(+), 2 deletions(-)

[PATCH 14/27] CLI/config: switch "notmuch config list" to merged config

Use the database opened at the top level rather than opening another notmuch_database_t. Test output changes because keys are now listed in alphabetical order, and because a missing database is no longer an error. --- notmuch-config.c| 69 -

[PATCH 21/27] lib: add NOTMUCH_STATUS_NO_DATABASE

This will allow more precise return values from various open related functions. --- bindings/python-cffi/notmuch2/_build.py | 1 + bindings/python-cffi/notmuch2/_errors.py | 3 +++ lib/database.cc | 2 ++ lib/notmuch-private.h| 1 + lib/notmuch.h

[PATCH 19/27] CLI/config: drop cached data from notmuch_config_t

The idea is to preserve notmuch_config_t as a thin wrapper for GKeyFile. --- notmuch-config.c | 48 +--- 1 file changed, 9 insertions(+), 39 deletions(-) diff --git a/notmuch-config.c b/notmuch-config.c index 774c8936..93a1018e 100644 ---

[PATCH 03/27] lib/open: pull _load_key_file out of _choose_database_path

Although this increases code duplication, it also increases flexibility in handling various combinations of missing config file and missing database. --- lib/open.cc | 36 +++- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/lib/open.cc

[PATCH 18/27] CLI/config: drop obsolete notmuch_config_get_*

These are no longer used, replaced by notmuch_config_get. --- notmuch-client.h | 29 notmuch-config.c | 119 --- 2 files changed, 148 deletions(-) diff --git a/notmuch-client.h b/notmuch-client.h index dfdfc876..677f2f39 100644 ---

[PATCH 11/27] CLI/config: use merged config for "config get"

This commit starts the conversion of notmuch-config.c functionality (as opposed to just interface) to the new config framework. The change to T030-config is because of the move of the canonicalization database paths from the notmuch_config_t accessor to the internal function

[PATCH 12/27] test/setup: check file output instead of notmuch config list

This allows verification of comments added to new / updated config file. --- test/T040-setup.sh| 17 +--- .../config-with-comments | 88 +++ 2 files changed, 91 insertions(+), 14 deletions(-) create mode 100644

[PATCH 01/27] lib: add missing status strings

--- lib/database.cc | 8 1 file changed, 8 insertions(+) diff --git a/lib/database.cc b/lib/database.cc index ffd42ac4..87ad1385 100644 --- a/lib/database.cc +++ b/lib/database.cc @@ -292,12 +292,20 @@ notmuch_status_to_string (notmuch_status_t status) return "Operation requires

[PATCH 17/27] CLI/notmuch: switch notmuch_command to notmuch_config_get

The goal at this point is to remove the dependence on notmuch_config_get_* without breaking any existing functionality. This is a step towards removing notmuch_config_get_* in a future commit. --- notmuch.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/notmuch.c

[PATCH 08/27] lib/config: set default for primary user email

This is mainly copying code from the CLI into the lib. The CLI copy will be deleted in a later commit. --- lib/config.cc | 70 +++--- test/T590-libconfig.sh | 7 ++--- test/test-lib.sh | 6 +++- 3 files changed, 73 insertions(+), 10

[PATCH 05/27] lib/config: add notmuch_config_get_values_string

This is to support the less common (at least in the notmuch codebase) case of accessing a ;-delimited list config value with an arbitrary string key. --- lib/config.cc | 14 ++ lib/notmuch.h | 17 - test/T590-libconfig.sh | 23 +++

[PATCH 06/27] lib/config: add config_pairs iterators

The layer of shims here seems a bit wasteful compared to just calling the corresponding string map functions directly, but it allows control over the API (calling with notmuch_database_t *) and flexibility for future changes. --- lib/config.cc | 39 ++

[PATCH 09/27] lib/open: canonicalize relative path read from config file

This matches functionality in the the CLI function notmuch_config_get_database_path, which was previously used in the CLI code for all calls to open a database. --- lib/open.cc | 13 ++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/lib/open.cc b/lib/open.cc index

[PATCH 22/27] CLI+lib: detect missing database in split configurations.

Eventually we want to do all opening of databases in the top level (main function). This means that detection of missing databases needs to move out of subcommands. It also requires updating the library to use the new NO_DATABASE status code. --- lib/open.cc | 24 ---

[PATCH 27/27] CLI: rename notmuch_config_t to notmuch_conffile_t

The name is a compromise between keeping function names from getting too long, and avoiding confusion with the newer notmuch_config library API. Try to make clear that the residual functionality is only about updating configuration files. --- notmuch-client.h | 56

[PATCH 15/27] CLI/config: migrate notmuch_config_open to new config

notmuch_config_open will be preserved in the medium term for use by the commands that are manipulating the config file directly (config and setup) --- lib/config.cc | 5 +- notmuch-client.h | 2 +- notmuch-config.c

[PATCH 23/27] lib: provide notmuch_config_path

Since the library searches in several locations for a config file, the caller does not know which of these is chosen in the usual case of passing NULL as a config file. This changes provides an API for the caller to retrieve the name of the config file chosen. It will be tested in a following

[PATCH 16/27] CLI/config: use notmuch_database_reopen

This allows `notmuch config` to use the common configuration information carried by the notmuch_database_t opened at the top level. --- notmuch-config.c | 16 +++- test/T750-user-header.sh | 3 ++- 2 files changed, 9 insertions(+), 10 deletions(-) diff --git

[PATCH 20/27] CLI/config: default to storing all config in external files

Previously the fact that some configuration options were only stored in the database (and thus editing the config file had no effect) was a source of user confusion. This was fixed with the series ending at d9af0af1646. On the other hand, the underlying partition of config options into those

[PATCH 04/27] lib: provide notmuch_database_load_config

This is mainly targetted at notmuch-config.c, or similar applications that don't necessarily need both a configuration file and a database to exist. --- lib/notmuch.h | 27 ++ lib/open.cc| 96 ++ test/T590-libconfig.sh | 182

[PATCH 25/27] CLI/config: remove calls to notmuch_config_open from top level

This will allow simplifying the subcommand interface. It also fixes a previously broken test, since notmuch_config_open does not understand the notion of the empty string as a config file name. --- notmuch-client.h | 3 +-- notmuch-config.c | 24 +++-

v1: convert remaining CLI to new configuration

This completes the planned overhaul of configuration handling. It obsoletes [1], and needs to be applied on top of [2]. I will post a rebased version of [2] fairly soon. [1]: id:20210220164448.3956011-1-da...@tethera.net [2]: id:20210208004109.1192719-1-da...@tethera.net