[PATCH 18/36] lib/query: generalize exclude handling to s-expression queries

In fact most of the code path is in common, only the caching of terms in the query needs to be added for s-expression queries. --- lib/query.cc | 34 ++--- test/T081-sexpr-search.sh | 40 +++ 2 files changed, 63

[PATCH 31/36] lib/parse-sexp: thread environment argument through parser

No functionality change, just an extra argument carried everywhere. --- lib/parse-sexp.cc | 47 +-- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc index 291480ca..8f7c26c2 100644 ---

[PATCH 02/36] configure: optional library sfsexp

The configure part is essentially the same as the other checks using pkg-config. Since the optional inclusion of this feature changes what options are available to the user, include it in the "built_with" pseudo-configuration keys. --- configure| 26 +-

[PATCH 14/36] lib/parse-sexp: add term prefix backed fields

We use "boolean" to describe fields that should generate terms literally without stemming or phrase splitting. This terminology might not be ideal but it is already enshrined in notmuch-search-terms(7). --- doc/man7/notmuch-sexp-queries.rst | 18 +- lib/parse-sexp.cc | 49

[PATCH 16/36] lib/parse-sexp: add '*' as syntactic sugar for '(starts-with "")'

Users that insist on using a literal '*' as a tag, can continue to do so by quoting it when searching. --- doc/man7/notmuch-sexp-queries.rst | 19 ++-- lib/parse-sexp.cc | 5 test/T081-sexpr-search.sh | 48 +++ 3 files changed, 70

[PATCH 20/36] lib/parse-sexp: support regular expressions

At least to the degree that the Xapian QueryParser based parser also supports them. Support short alias 'rx' as it seems to make more complex queries nicer to read. --- doc/man7/notmuch-sexp-queries.rst | 8 lib/parse-sexp.cc | 54 ++-

[PATCH 05/36] CLI/search+address: support sexpr queries

Initially support selection of query syntax in two subcommands to enable testing. --- notmuch-search.c | 13 + test/T080-search.sh | 7 +++ test/T095-address.sh | 7 +++ 3 files changed, 23 insertions(+), 4 deletions(-) diff --git a/notmuch-search.c b/notmuch-search.c

[PATCH 11/36] lib/parse-sexp: support subject field

The broken tests are because we do not yet handle phrase searches. --- doc/man7/notmuch-sexp-queries.rst | 62 +-- lib/parse-sexp.cc | 19 +- test/T081-sexpr-search.sh | 57 3 files changed, 133

[PATCH 22/36] lib/query: factor out _notmuch_query_string_to_xapian_query

When dealing with recursive queries (i.e. thread:{foo}) it turns out to be useful just to deal with the underlying Xapian objects, and not wrap them in notmuch objects. --- lib/database-private.h | 7 ++ lib/query.cc | 51 -- 2 files changed,

[PATCH 24/36] lib/parse-sexp: expand queries

The code here is just gluing together _notmuch_query_expand with the existing sexp parser infrastructure. --- doc/man7/notmuch-sexp-queries.rst | 20 +++ lib/parse-sexp.cc | 56 +-- test/T081-sexpr-search.sh | 52

[PATCH 21/36] lib: generate actual Xapian query for "*" and ""

The previous code had the somewhat bizarre effect that the (notmuch specific) query string was "*" (interpreted as MatchAll) and the allegedly parsed xapian_query was "MatchNothing". This commit also reduces code duplication. --- lib/query.cc | 34 ++ 1 file

[PATCH 10/36] lib/parse-sexp: support and, not, and or.

All operations and (Xapian) fields will eventually have an entry in the prefixes table. The flags field is just a placeholder for now, but will eventually distinguish between various kinds of prefixes. --- doc/man7/notmuch-sexp-queries.rst | 16 --- lib/parse-sexp.cc | 76

[PATCH 23/36] lib/thread-fp: factor out query expansion, rewrite in Xapian

It will be convenient not to have to construct a notmuch query object when parsing subqueries, so the commit rewrites the query expansion (currently only used for thread:{} queries) using only Xapian. As a bonus it seems about 15% faster in initial experiments. --- lib/database-private.h | 16

[PATCH 01/36] CLI: make variable n_requested_db_uuid file scope.

It turns out that now that we pass an open database into the subcommands, it is easy to check any requested uuid against the database at the same time as we process the other shared arguments. This results in overall less boilerplate code, as well as making a CLI scope function and variable file

[PATCH 35/36] CLI/tag: enable sexp queries

We have to rewrite _optimize_tag_query here because it is generating a query string in the infix Xapian syntax. Luckily this is easy to do with the sexp query syntax. --- notmuch-tag.c| 43 --- test/T150-tagging.sh | 43

[PATCH 36/36] doc/sexp-queries: update synopsis and description

I chose to go with a somewhat terse synopsis to try to keep the length of the page down. --- doc/man7/notmuch-sexp-queries.rst | 13 - 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/doc/man7/notmuch-sexp-queries.rst b/doc/man7/notmuch-sexp-queries.rst index

[PATCH 32/36] lib/parse-sexp: apply macros

Macros implement lazy evaluation and lexical scope. The former is needed to make certain natural constructs work sensibly (e.g. (tag ,param)) but the latter is mainly future-proofing in case the DSL is is extended to allow local bindings. For technical background, see chapters 6 and 17 of [1]

[PATCH 13/36] lib/parse-sexp: support phrase queries.

Anything that is quoted or not purely word characters is considered a phrase. Phrases are not stemmed, because the stems do not have positional information in the database. It is less efficient to scan the term twice, but it avoids a second pass to add prefixes, so maybe it balances out. In any

[PATCH 04/36] lib: define notmuch_query_create_with_syntax

Set the parsing syntax when the (notmuch) query object is created. Initially the library always returns a trivial query that matches all messages when using s-expression syntax. It seems better to select the syntax at query creation time because the lazy parsing is an implementation detail. ---

[PATCH 08/36] lib: leave stemmer object accessible

This enables using the same stemmer in both query parsers. --- lib/database-private.h | 1 + lib/open.cc| 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/database-private.h b/lib/database-private.h index f206efaf..85d55299 100644 --- a/lib/database-private.h

[PATCH 17/36] lib/parse-sexp: handle unprefixed terms.

This is equivalent to adding the same field name "" for multiple prefixes in the Xapian query parser, but we have to explicitely construct the resulting query. --- lib/parse-sexp.cc | 36 test/T081-sexpr-search.sh | 31 +++

[PATCH 03/36] lib: split notmuch_query_create

Most of the function will be re-usable when creating a query from an s-expression. --- lib/query.cc | 19 --- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/lib/query.cc b/lib/query.cc index 792aba21..39b85e91 100644 --- a/lib/query.cc +++ b/lib/query.cc @@ -84,9

[PATCH 25/36] lib/parse-sexp: support infix subqueries

This is necessary so that programs can take infix syntax queries from a user and use the sexp query syntax to construct e.g. a refinement of that query. --- doc/man7/notmuch-sexp-queries.rst | 7 + lib/parse-sexp.cc | 34 test/T081-sexpr-search.sh

[PATCH 26/36] lib/parse-sexp: parse user headers

One subtle aspect is the replacement of _find_prefix with _notmuch_database_prefix, which understands user headers. Otherwise the code mainly consists of creating a fake prefix record (since the user prefixes are not in the prefix table) and error handling. --- doc/man7/notmuch-sexp-queries.rst |

[PATCH 29/36] CLI/config support saving s-expression queries

This commit does not enable using saved s-expression queries, only saving and retrieving them from the config file or the database. Use in queries will be enabled in a following commit. --- doc/man1/notmuch-config.rst | 5 + notmuch-config.c| 1 + test/T081-sexpr-search.sh |

[PATCH 07/36] lib/parse-sexp: parse single terms and the empty list.

There is not much of a parser here yet, but it already does some useful error reporting. Most functionality sketched in the documentation is not implemented yet; detailed documentation will follow with the implementation. --- doc/conf.py | 4 ++ doc/index.rst

[PATCH 06/36] lib: add new status code for query syntax errors.

This will help provide more meaningful error messages without special casing on the client side. --- bindings/python-cffi/notmuch2/_build.py | 1 + bindings/python-cffi/notmuch2/_errors.py | 3 +++ lib/database.cc | 2 ++ lib/notmuch.h| 4

[PATCH 19/36] lib: factor out query construction from regexp

This will allow re-use of this code outside of the Xapian query parser. --- lib/database-private.h | 5 +++ lib/regexp-fields.cc | 81 +- lib/regexp-fields.h| 6 3 files changed, 68 insertions(+), 24 deletions(-) diff --git

[PATCH 34/36] CLI/{count, dump, reindex, reply, show}: enable sexp queries

The change in each case is to call notmuch_query_create_with_syntax, relying on the already inherited shared options. As a bonus we get improved error handling from the new query creation API. The remaining subcommand is 'tag', which is a bit trickier. --- notmuch-count.c | 10

[PATCH 15/36] lib/parse-sexp: 'starts-with' wildcard searches

The many tests potentially overkill, but they could catch typos in the prefixes table. As a simplifying assumption, for now we assume a single argument to the wildcard operator, as this matches the Xapian semantics. The name 'starts-with' is chosen to emphasize the supported case of wildcards in

[PATCH 12/36] util/unicode: allow calling from C++

The omission of the 'extern "C"' machinery seems like an oversight. --- util/unicode-util.h | 7 +++ 1 file changed, 7 insertions(+) diff --git a/util/unicode-util.h b/util/unicode-util.h index 32d1e6ef..1bb9336a 100644 --- a/util/unicode-util.h +++ b/util/unicode-util.h @@ -4,9 +4,16 @@

[PATCH 30/36] lib/parse-sexp: support saved s-expression queries

It turns out there is not really much code in query-fp.cc useful for supporting the new syntax. The code we could potentially factor out amounts to calling notmuch_database_get_config; both the key construction and the parsing of the results are specific to the query syntax involved. ---

[PATCH 28/36] lib/parse-sexp: handle saved queries

This provides functionality analogous to query: in the Xapian QueryParser based parser. Perhaps counterintuitively, the saved queries currently have to be in the original query syntax (i.e. not s-expressions). --- doc/man7/notmuch-sexp-queries.rst | 6 ++ lib/parse-sexp.cc |

[PATCH 27/36] lib: factor out expansion of saved queries.

This is intended to allow use outside of the Xapian query parser. --- lib/database-private.h | 5 + lib/query-fp.cc| 22 +++--- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/lib/database-private.h b/lib/database-private.h index 9ee3b933..8b9d67fe

[PATCH 09/36] lib/parse-sexp: stem unquoted atoms

This is somewhat less DWIM than the Xapian query parser, but it has the advantage of simplicity. --- doc/man7/notmuch-sexp-queries.rst | 10 -- lib/parse-sexp.cc | 10 +++--- test/T081-sexpr-search.sh | 5 - 3 files changed, 19 insertions(+), 6

[PATCH 33/36] CLI: move query syntax to shared option

This will allow easy addition of a query syntax option to other subcommands. --- notmuch-client.h | 2 ++ notmuch-search.c | 7 ++- notmuch.c| 12 3 files changed, 16 insertions(+), 5 deletions(-) diff --git a/notmuch-client.h b/notmuch-client.h index

v5 sexp query parser

Changes since v4: 1) --query=sexp is now recognized for all notmuch subcommands (and ignored where there is no query argument). 2) cleanup for the (common) case of missing the sfsexp library 3) An updated "SYNOPSIS" and "DESCRIPTION" the notmuch-sexp-queries(7) man page. I don't plan on