Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show

2022-05-18 Thread David Bremner
Alexander Adolf  writes:


>
> Ok, I have added the line as you suggest. I was wondering though, as in
> principle the secret key of test_su...@notmuchmail.org could be
> available (it's in the key-ring at least)?

Yes, it's added to the test suite's temporary keyring. One of the
messages in test/corpora/crypto is not encrypted to that key though.

$ for file in *; do 
echo $file  
gpg < $file |& grep encrypted; echo
done

basic-encrypted.eml
gpg: encrypted with 1024-bit RSA key, ID C44D36DEAD54AB16, created 2011-02-05

encrypted-rfc822-attachment
gpg: encrypted with 4096-bit RSA key, ID 70E3C0DE87068451, created 2019-06-08
gpg: encrypted with 1024-bit RSA key, ID C44D36DEAD54AB16, created 2011-02-05

encrypted-signed.eml
gpg: encrypted with 1024-bit RSA key, ID C44D36DEAD54AB16, created 2011-02-05

simple-encrypted
gpg: encrypted with 4096-bit RSA key, ID 3B7AA7F014E69B5A, created 2016-12-21



> It seems this refers to the very first line of the expected output? I
> have updated the expected file to use an absolute date, and the test
> script to set notmuch-show-relative-dates to nil as you suggest.
>

Yes, it's referring to the first line of output.

>>   Subject: notmuch-show S/MIME test
>>   To: test_su...@notmuchmail.org
>>   Date: Tue, 12 Apr 2022 16:57:30 +0200
>>  @@ -6,10 +6,10 @@
>>   [ smime.p7m: application/pkcs7-mime ]
>>   [ Decryption successful ]
>>   [ multipart/signed ]
>>  -[ Good signature by:  ]
>>  +[ Good signature by: test_su...@notmuchmail.org ]
>>
>> This is exactly the problem we just dealt with for T355-smime.
>
> T357?
>
>> I think the same solution can be applied, but you will need to inline
>> the output so that you can do variable substitution.
>
> I'm lost on what you're suggesting/expecting for this one. Remove the
> angle brackets from the "Good signature" line of the expected file?

Have a look at commit 8723e707c15f7b435f07f5d5ea693496bb9769bb, in
particular the lines

if [ $NOTMUCH_GMIME_EMITS_ANGLE_BRACKETS == 1 ]; then
   EXPECTED_EMAIL_ADDR=''
else
   EXPECTED_EMAIL_ADDR='test_su...@notmuchmail.org'
fi

Then you can use $EXPECTED_EMAIL_ADDR in the expected output. This means
you'll have to generate the expected output rather than hard coding it
in a file. To pick some examples at random, you can see how to do
something similar on lines 157 and 222 of T035-read-config.sh

>
>>   [ multipart/mixed ]
>>   [ multipart/mixed ]
>>   [ text/plain ]
>>   The password is "12345678". But don't tell anyone!
>>  -[ test.dtd: application/octet-stream ]
>>  +[ test.dtd: application/octet-stream (as application/xml-dtd) ]
>>
>> This seems related to the function
>> #'notmuch-show-get-mime-type-of-application/octet-stream, which is
>> calling (mailcap-extension-to-mime "dtd"). Probably this is hard to make
>> reproducible, so just seding away an "(as ...)" string is a reasonable
>> alternative.
>> [...]
>
> I see; unfortunate choice of MIME part. As it seems I will need to
> generate a new test message anyway, what would be a more "portable"
> part? PNG?

PNG should not be worse, but the output might still need postprocessing
to avoid variation.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH v2 1/1] emacs: notmuch-show-header-line: allow format strings and functions

2022-05-18 Thread David Bremner
Jose A Ortega Ruiz  writes:

> On Tue, May 17 2022, Tomi Ollila wrote:
>
>
> [...]
>
>> To me it looks like some lines are indented with tabs and some spaces
>> (noticed as indentation looked weird to me and then started moving cursor
>> at the beginning of line -- there is probably a way to highlight tabs in
>> notmuch show buffer but...)
>
> oh, i've got that highlight on, and i see spaces and tabs mixed all over
> the place, so i didn't pay attention: i have emacs configured to indent
> using spaces, am i supposed to use tabs for notmuch elisp? or just not
> mix the two in the same block?
>

If you run "devel/check-notmuch-commit" (or the embedded emacs command)
it will "do the right thing" for indenting elisp. Basically however
emacs -Q does it.

The only catch is that potentially some files might predate this and/or
have indentation overriding emacs, so you have to pick and choose any
diffs.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [bug] possible condition depending on uninitialized value in _notmuch_message_sync

2022-05-16 Thread David Bremner
Eliza Velasquez  writes:


> It becomes very clear why this error only happens when removing a
> non-existent tag if you look at at message.cc:1570...
>
> --8<---cut here---start->8---
> try {
>   message->doc.remove_term (term);
>   message->modified = true;
> } catch (const Xapian::InvalidArgumentError) {
>   /* We'll let the philosophers try to wrestle with the
>* question of whether failing to remove that which was not
>* there in the first place is failure. For us, we'll silently
>* consider it all good. */
> }
> --8<---cut here---end--->8---

OK, I see why that assignment gets skipped. I think that's not the
actual bug, but rather message->modified should be initialized in
_notmuch_message_create_for_document. I'll have a closer look later
today

d
.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Help with sending mail from notmuch-emacs

2022-05-16 Thread David Bremner
hgv  writes:
>
> I'm still struggling with the Content-Transfer-Encoding header and the
> "charset=utf-8" half of the Content-Type header, both of which I
> cannot get set in my emails, if anyone has any thoughts!

My only (not so helpful) thought is that this almost certainly an issue
outside notmuch in message-mode. If you can duplicate it there (M-x
message-mail) you can get help from the wider community of message-mode
users.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH v2] doc/emacs: document a few notmuch-hello customizable variables.

2022-05-16 Thread David Bremner
A user asked about the thousands separator on IRC, and I had to check
the source.
---

I realized the docstring was getting mangled when auto-translated to
rst, so I did the manual translation.

 doc/conf.py   |  2 +-
 doc/notmuch-emacs.rst | 41 -
 2 files changed, 41 insertions(+), 2 deletions(-)

diff --git a/doc/conf.py b/doc/conf.py
index e46e1d4e..fee52b64 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -45,7 +45,7 @@ if tags.has('WITH_EMACS'):
 # Hacky reimplementation of include to workaround limitations of
 # sphinx-doc
 lines = ['.. include:: /../emacs/rstdoc.rsti\n\n'] # in the source tree
-for file in ('notmuch.rsti', 'notmuch-lib.rsti', 'notmuch-show.rsti', 
'notmuch-tag.rsti', 'notmuch-tree.rsti'):
+for file in ('notmuch.rsti', 'notmuch-lib.rsti', 'notmuch-hello.rsti', 
'notmuch-show.rsti', 'notmuch-tag.rsti', 'notmuch-tree.rsti'):
 lines.extend(open(rsti_dir+'/'+file))
 rst_epilog = ''.join(lines)
 del lines
diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index 67dbfc2b..595f8e2a 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -46,8 +46,47 @@ a mouse or by positioning the cursor and pressing 

 |Customize Notmuch or this page.
 
 You can change the overall appearance of the notmuch-hello screen by
-customizing the variable :index:`notmuch-hello-sections`.
+customizing the variables
 
+:index:`notmuch-hello-sections`
+   |docstring::notmuch-hello-sections|
+
+:index:`notmuch-hello-thousands-separator`
+   |docstring::notmuch-hello-thousands-separator|
+
+:index:`notmuch-show-logo`
+   |docstring::notmuch-show-logo|
+
+:index:`!notmuch-column-control`
+Controls the number of columns for saved searches/tags in notmuch view.
+
+This variable has three potential types of values:
+
+.. describe:: t
+
+   Automatically calculate the number of columns possible based
+   on the tags to be shown and the window width.
+
+.. describe:: integer 
+
+   A lower bound on the number of characters that will
+   be used to display each column.
+
+.. describe:: float 
+
+   A fraction of the window width that is the lower bound on the
+   number of characters that should be used for each column.
+
+So:
+
+- if you would like two columns of tags, set this to 0.5.
+
+- if you would like a single column of tags, set this to 1.0.
+
+- if you would like tags to be 30 characters wide, set this to 30.
+
+- if you don't want to worry about all of this nonsense, leave
+  this set to `t`.
 
 
 notmuch-hello key bindings
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: do not inline application/* in replies

2022-05-16 Thread David Bremner
David Bremner  writes:

> Since notmuch 0.24, notmuch-emacs has overriden the Gnus defaults for
> inlining attachements with type "application/*" when showing
> messages. This series adds some tests and applies the same override
> for reply.

Applied to master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [bug] possible condition depending on uninitialized value in _notmuch_message_sync

2022-05-16 Thread David Bremner
Eliza Velasquez  writes:


> Is it possible then that there's a potential memory error with removing
> a non-existent tag on a message? I wanted to ask about this on the
> mailing list before diving in deeper, since this isn't quite the latest
> version of notmuch and I wasn't sure if it had been fixed in 0.36. I
> searched the mailing list archives for this particular issue, but I
> wasn't able to find anything.

Ideally of course I'd like a reproducer in C.  It would help to have
line numbers in the valgrind output. It might be enough you install the
notmuch debug symbols?

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] doc/emacs: document notmuch-show-toggle-toggle-elide-non-matching

2022-05-15 Thread David Bremner
David Bremner  writes:

> Recently there was a question on the mailing list about the existence
> of this function, so let us try to make it more discoverable.

applied to master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 14/17] CLI/git: change defaults for repo and prefix

2022-05-15 Thread David Bremner
The previous defaults were not suitable for personal (i.e. not
bugtracking for notmuch development) use.

Provide two ways for the user to select nmbug compatible defaults;
command line argument and checking the name of the script.
---
 Makefile.local   |   5 +-
 doc/Makefile.local   |   1 +
 doc/man1/notmuch-git.rst |  21 +++-
 notmuch-git.in   |  62 +++
 test/T850-git.sh | 103 +++
 5 files changed, 160 insertions(+), 32 deletions(-)

diff --git a/Makefile.local b/Makefile.local
index ca2310f4..0fadfb26 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -1,7 +1,7 @@
 # -*- makefile-gmake -*-
 
 .PHONY: all
-all: notmuch notmuch-shared build-man build-info ruby-bindings 
python-cffi-bindings notmuch-git
+all: notmuch notmuch-shared build-man build-info ruby-bindings 
python-cffi-bindings notmuch-git nmbug
 ifeq ($(MAKECMDGOALS),)
 ifeq ($(shell cat .first-build-message 2>/dev/null),)
@NOTMUCH_FIRST_BUILD=1 $(MAKE) --no-print-directory all
@@ -307,9 +307,10 @@ cppcheck:
@echo "No cppcheck found during configure; skipping static checking"
 endif
 
-notmuch-git: notmuch-git.in
+nmbug notmuch-git: notmuch-git.in
sed s/@NOTMUCH_VERSION@/${VERSION}/ < notmuch-git.in > notmuch-git
chmod ugo+rx notmuch-git
+   ln -sf notmuch-git nmbug
 
 DEPS := $(SRCS:%.c=.deps/%.d)
 DEPS := $(DEPS:%.cc=.deps/%.d)
diff --git a/doc/Makefile.local b/doc/Makefile.local
index d43ef269..2f67f4de 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -131,6 +131,7 @@ install-man: ${MAN_GZIP_FILES}
install -m0644 $(filter %.5.gz,$(MAN_GZIP_FILES)) 
$(DESTDIR)/$(mandir)/man5
install -m0644 $(filter %.7.gz,$(MAN_GZIP_FILES)) 
$(DESTDIR)/$(mandir)/man7
cd $(DESTDIR)/$(mandir)/man1 && ln -sf notmuch.1.gz notmuch-setup.1.gz
+   cd $(DESTDIR)/$(mandir)/man1 && ln -sf notmuch-git.1.gz nmbug.1.gz
 endif
 
 ifneq ($(HAVE_SPHINX)$(HAVE_MAKEINFO),11)
diff --git a/doc/man1/notmuch-git.rst b/doc/man1/notmuch-git.rst
index 86f246b6..6a2d7fcd 100644
--- a/doc/man1/notmuch-git.rst
+++ b/doc/man1/notmuch-git.rst
@@ -7,7 +7,9 @@ notmuch-git
 SYNOPSIS
 
 
-**notmuch** **git** [-h] [-C *repo*] [-p *prefix*] [-v] [-l *log level*] 
*subcommand*
+**notmuch** **git** [-h] [-N] [-C *repo*] [-p *prefix*] [-v] [-l *log level*] 
*subcommand*
+
+**nmbug** [-h] [-C *repo*] [-p *prefix*] [-v] [-l *log level*] *subcommand*
 
 DESCRIPTION
 ===
@@ -25,12 +27,17 @@ Supported options for `notmuch git` include
 
show help message and exit
 
+.. option:: -N, --nmbug
+
+   Set defaults for :option:`--tag-prefix` and :option:`--git-dir` suitable 
for the
+   :any:`notmuch` bug tracker
+
 .. option:: -C , --git-dir 
 
Operate on git repository *repo*. See :ref:`repo_location` for
defaults.
 
-.. option::  -p , --tag-prefix 
+.. option:: -p , --tag-prefix 
 
Operate only on tags with prefix *prefix*. See :ref:`prefix_val` for
defaults.
@@ -239,6 +246,10 @@ value to locate the git repository.
 
 - Environment variable :envvar:`NOTMUCH_GIT_DIR`.
 
+- If invoked as `nmbug` or with the :option:`--nmbug` option,
+  :code:`$HOME/.nmbug`; otherwise
+  :code:`$XDG_DATA_HOME/notmuch/$NOTMUCH_PROFILE/git`.
+
 .. _prefix_val:
 
 PREFIX VALUE
@@ -251,9 +262,15 @@ value to define the tag prefix.
 
 - Environment variable :envvar:`NOTMUCH_GIT_PREFIX`.
 
+- If invoked as `nmbug` or with the :option:`--nmbug` option,
+  :code:`notmuch::`, otherwise the empty string.
+
 ENVIRONMENT
 ===
 
+Variable :envvar:`NOTMUCH_PROFILE` influences :ref:`repo_location`.
+If it is unset, 'default' is assumed.
+
 .. envvar:: NOTMUCH_GIT_DIR
 
Default location of git repository. Overriden by :option:`--git-dir`.
diff --git a/notmuch-git.in b/notmuch-git.in
index 709d3d07..35820eb2 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -427,6 +427,13 @@ def init(remote=None):
 This wraps 'git init' with a few extra steps to support subsequent
 status and commit commands.
 """
+from pathlib import Path
+parent = Path(NOTMUCH_GIT_DIR).parent
+try:
+_os.makedirs(parent)
+except FileExistsError:
+pass
+
 _spawn(args=['git', '--git-dir', NOTMUCH_GIT_DIR, 'init',
  '--initial-branch=master', '--quiet', '--bare'], wait=True)
 _git(args=['config', 'core.logallrefupdates', 'true'], wait=True)
@@ -863,9 +870,19 @@ def _notmuch_config_get(key):
 stdout=_subprocess.PIPE, wait=True)
 if status != 0:
 _LOG.error("failed to run notmuch config")
-sys.exit(1)
+_sys.exit(1)
 return stdout.rstrip()
 
+# based on BaseDirectory.save_data_path from pyxdg (LGPL2+)
+def xdg_data_path(profile):
+resource = _os.path.join('notmuch',profile,'git')
+assert not resource.startswith('/')
+_home = _os.path.expanduser('~')
+xdg_data_home = _os.environ.get('XDG_DATA_HOME') or \
+_os.path.join(_home, '.local', 

[PATCH 11/17] CLI/git: add @timed decorator, time a few functions

2022-05-15 Thread David Bremner
Perf will show which binaries are using the CPU cycles, and standard
python profilers will show which python functions, but neither is
great at finding which call to an external binary is taking time, or
locating I/O hotspots.
---
 notmuch-git.in | 19 ++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 1d2dcd53..0aabfb60 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -91,6 +91,20 @@ def _xapian_unquote(string):
 return string
 
 
+def timed(fn):
+"""Timer decorator"""
+from time import perf_counter
+
+def inner(*args, **kwargs):
+start_time = perf_counter()
+rval = fn(*args, **kwargs)
+end_time = perf_counter()
+_LOG.info('{0}: {1:.8f}s elapsed'.format(fn.__name__, end_time - 
start_time))
+return rval
+
+return inner
+
+
 class SubprocessError(RuntimeError):
 "A subprocess exited with a nonzero status"
 def __init__(self, args, status, stdout=None, stderr=None):
@@ -323,6 +337,7 @@ def commit(treeish='HEAD', message=None):
 _git(args=['read-tree', treeish], wait=True)
 raise
 
+@timed
 def _update_index(status):
 with _git(
 args=['update-index', '--index-info'],
@@ -563,6 +578,7 @@ def _is_unmerged(ref='@{upstream}'):
 return base != fetch_head
 
 
+@timed
 def get_status():
 status = {
 'deleted': {},
@@ -583,7 +599,7 @@ def get_status():
 _os.remove(index)
 return status
 
-
+@timed
 def _index_tags():
 "Write notmuch tags to the nmbug.index."
 path = _os.path.join(NOTMUCH_GIT_DIR, 'nmbug.index')
@@ -632,6 +648,7 @@ def _index_tags_for_message(id, status, tags):
 yield '{mode} {hash}\t{path}\n'.format(mode=mode, hash=hash, path=path)
 
 
+@timed
 def _diff_index(index, filter):
 """
 Get an {id: {tag, ...}} dict for a given filter.
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 15/17] CLI/git: support configuration for repo location / prefix

2022-05-15 Thread David Bremner
This is probably more convenient than always passing a command line
argument.

Use notmuch-config for consistency with other notmuch CLI tools.
---
 doc/man1/notmuch-config.rst |  8 +
 doc/man1/notmuch-git.rst|  4 +++
 notmuch-git.in  |  6 ++--
 test/T850-git.sh| 67 -
 4 files changed, 82 insertions(+), 3 deletions(-)

diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 36d48725..e2e9a632 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -107,6 +107,14 @@ paths are presumed relative to `$HOME` for items in section
 
 Default: see :ref:`database`
 
+.. nmconfig:: git.path
+
+Default location for git repository for :any:`notmuch-git`.
+
+.. nmconfig:: git.tag_prefix
+
+Default tag prefix (filter) for :any:`notmuch-git`.
+
 .. nmconfig:: index.decrypt
 
 Policy for decrypting encrypted messages during indexing.  Must be
diff --git a/doc/man1/notmuch-git.rst b/doc/man1/notmuch-git.rst
index 6a2d7fcd..ad859b80 100644
--- a/doc/man1/notmuch-git.rst
+++ b/doc/man1/notmuch-git.rst
@@ -246,6 +246,8 @@ value to locate the git repository.
 
 - Environment variable :envvar:`NOTMUCH_GIT_DIR`.
 
+- Configuration item :nmconfig:`git.path`
+
 - If invoked as `nmbug` or with the :option:`--nmbug` option,
   :code:`$HOME/.nmbug`; otherwise
   :code:`$XDG_DATA_HOME/notmuch/$NOTMUCH_PROFILE/git`.
@@ -262,6 +264,8 @@ value to define the tag prefix.
 
 - Environment variable :envvar:`NOTMUCH_GIT_PREFIX`.
 
+- Configuration item :nmconfig:`git.tag_prefix`.
+
 - If invoked as `nmbug` or with the :option:`--nmbug` option,
   :code:`notmuch::`, otherwise the empty string.
 
diff --git a/notmuch-git.in b/notmuch-git.in
index 35820eb2..3e5205e8 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -1024,7 +1024,9 @@ if __name__ == '__main__':
 if nmbug_mode:
 default = _os.path.join('~', '.nmbug')
 else:
-default = xdg_data_path(notmuch_profile)
+default = _notmuch_config_get ('git.path')
+if default == '':
+default = xdg_data_path(notmuch_profile)
 
 NOTMUCH_GIT_DIR = _os.path.expanduser(_os.getenv('NOTMUCH_GIT_DIR', 
default))
 
@@ -1038,7 +1040,7 @@ if __name__ == '__main__':
 if nmbug_mode:
 prefix = 'notmuch::'
 else:
-prefix = ''
+prefix = _notmuch_config_get ('git.tag_prefix')
 
 TAG_PREFIX =  _os.getenv('NOTMUCH_GIT_PREFIX', prefix)
 
diff --git a/test/T850-git.sh b/test/T850-git.sh
index 98233055..8f91b612 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -15,6 +15,13 @@ git config --global user.name  "Notmuch Test Suite"
 test_begin_subtest "init"
 test_expect_success "notmuch git -p '' -C remote.git init"
 
+test_begin_subtest "init (git.path)"
+notmuch config set git.path configured.git
+notmuch git init
+notmuch config set git.path
+output=$(git -C configured.git rev-parse --is-bare-repository)
+test_expect_equal "$output" "true"
+
 test_begin_subtest "clone"
 test_expect_success "notmuch git -p '' -C tags.git clone remote.git"
 
@@ -175,6 +182,32 @@ repository = CWD/remote.git
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+
+test_begin_subtest "env variable NOTMUCH_GIT_DIR overrides config when invoked 
as 'nmbug'"
+notmuch config set git.path `pwd`/bar
+NOTMUCH_GIT_DIR=`pwd`/remote.git  "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& 
grep '^repository' | notmuch_dir_sanitize > OUTPUT
+notmuch config set git.path
+cat < EXPECTED
+repository = CWD/remote.git
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "env variable NOTMUCH_GIT_DIR overrides config when invoked 
as 'notmuch git'"
+notmuch config set git.path `pwd`/bar
+NOTMUCH_GIT_DIR=`pwd`/remote.git notmuch git -ldebug status |& grep 
'^repository' | notmuch_dir_sanitize > OUTPUT
+notmuch config set git.path
+cat < EXPECTED
+repository = CWD/remote.git
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "env variable NOTMUCH_GIT_PREFIX works when invoked as 
'nmbug'"
+NOTMUCH_GIT_PREFIX=env:: "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep 
'^prefix' | notmuch_dir_sanitize > OUTPUT
+cat < EXPECTED
+prefix = env::
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "env variable NOTMUCH_GIT_PREFIX works when invoked as 
nmbug"
 NOTMUCH_GIT_PREFIX=foo:: "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep 
'^prefix' | notmuch_dir_sanitize > OUTPUT
 cat < EXPECTED
@@ -182,8 +215,19 @@ prefix = foo::
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
-test_begin_subtest "env variable NOTMUCH_GIT_PREFIX works when invoked as 
'notmuch git'"
+test_begin_subtest "env variable NOTMUCH_GIT_PREFIX overrides config when 
invoked as 'nmbug'"
+notmuch config set git.tag_prefix config::
+NOTMUCH_GIT_PREFIX=env:: "$NOTMUCH_BUILDDIR"/nmbug -ldebug status |& grep 
'^prefix' | notmuch_dir_sanitize > OUTPUT
+notmuch config set git.path
+cat < EXPECTED
+prefix = 

[PATCH 06/17] test: initial tests for notmuch-git

2022-05-15 Thread David Bremner
Exercise the main functionality of notmuch-git.  add_git_repos() will
hopefully be simplifed when an init subcommand is added.
---
 notmuch-git.in   |  4 +++
 test/T850-git.sh | 93 
 2 files changed, 97 insertions(+)
 create mode 100755 test/T850-git.sh

diff --git a/notmuch-git.in b/notmuch-git.in
index 71a4df72..3a16914d 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -841,6 +841,10 @@ if __name__ == '__main__':
 level = getattr(_logging, args.log_level.upper())
 _LOG.setLevel(level)
 
+# for test suite
+for var in ['NMBGIT', 'NMBPREFIX', 'NOTMUCH_PROFILE', 'NOTMUCH_CONFIG' ]:
+_LOG.debug('env {:s} = {:s}'.format(var, _os.getenv(var,'%None%')))
+
 if not getattr(args, 'func', None):
 parser.print_usage()
 _sys.exit(1)
diff --git a/test/T850-git.sh b/test/T850-git.sh
new file mode 100755
index ..713b326f
--- /dev/null
+++ b/test/T850-git.sh
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+test_description='"notmuch git" to save and restore tags'
+. $(dirname "$0")/test-lib.sh || exit 1
+
+add_email_corpus
+
+git config --global user.email notm...@example.org
+git config --global user.name  "Notmuch Test Suite"
+
+test_begin_subtest "init"
+test_expect_success "notmuch git -p '' -C remote.git init"
+
+test_begin_subtest "clone"
+test_expect_success "notmuch git -p '' -C tags.git clone remote.git"
+
+test_begin_subtest "commit"
+notmuch git -C tags.git -p '' commit
+git -C tags.git ls-tree -r --name-only HEAD | xargs dirname | sort -u | sed 
s,tags/,id:, > OUTPUT
+notmuch search --output=messages '*' | sort > EXPECTED
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
+test_begin_subtest "checkout"
+notmuch dump > BEFORE
+notmuch tag -inbox '*'
+notmuch git -C tags.git -p '' checkout
+notmuch dump > AFTER
+test_expect_equal_file_nonempty BEFORE AFTER
+
+test_begin_subtest "archive"
+notmuch git -C tags.git -p '' archive | tar tf - | \
+grep 20091117190054.gu3...@dottiness.seas.harvard.edu | sort > OUTPUT
+cat < EXPECTED
+tags/20091117190054.gu3...@dottiness.seas.harvard.edu/
+tags/20091117190054.gu3...@dottiness.seas.harvard.edu/inbox
+tags/20091117190054.gu3...@dottiness.seas.harvard.edu/signed
+tags/20091117190054.gu3...@dottiness.seas.harvard.edu/unread
+EOF
+notmuch git -C tags.git -p '' checkout
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "status"
+notmuch tag +test id:20091117190054.gu3...@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' status > OUTPUT
+cat < EXPECTED
+A  20091117190054.gu3...@dottiness.seas.harvard.edutest
+EOF
+notmuch git -C tags.git -p '' checkout
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "fetch"
+notmuch tag +test2 id:20091117190054.gu3...@dottiness.seas.harvard.edu
+notmuch git -C remote.git -p '' commit
+notmuch tag -test2 id:20091117190054.gu3...@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' fetch
+notmuch git -C tags.git -p '' status > OUTPUT
+cat < EXPECTED
+ a 20091117190054.gu3...@dottiness.seas.harvard.edutest2
+EOF
+notmuch git -C tags.git -p '' checkout
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "merge"
+notmuch git -C tags.git -p '' merge
+notmuch dump id:20091117190054.gu3...@dottiness.seas.harvard.edu | grep -v 
'^#' > OUTPUT
+cat < EXPECTED
++inbox +signed +test2 +unread -- 
id:20091117190054.gu3...@dottiness.seas.harvard.edu
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "push"
+notmuch tag +test3 id:20091117190054.gu3...@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' commit
+notmuch tag -test3 id:20091117190054.gu3...@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' push
+notmuch git -C remote.git -p '' checkout
+notmuch dump id:20091117190054.gu3...@dottiness.seas.harvard.edu | grep -v 
'^#' > OUTPUT
+cat < EXPECTED
++inbox +signed +test2 +test3 +unread -- 
id:20091117190054.gu3...@dottiness.seas.harvard.edu
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "environment passed through when run as 'notmuch git'"
+env NMBGIT=foo NMBPREFIX=bar NOTMUCH_PROFILE=default notmuch git -C tags.git 
-p '' -ldebug status |& \
+grep '^env ' | notmuch_dir_sanitize > OUTPUT
+cat < EXPECTED
+env NMBGIT = foo
+env NMBPREFIX = bar
+env NOTMUCH_PROFILE = default
+env NOTMUCH_CONFIG = CWD/notmuch-config
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_done
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 09/17] test/git: add known broken test for tag with quotes.

2022-05-15 Thread David Bremner
There is current insufficient sanitization and/or escaping of tag names
internally in notmuch-git.
---
 test/T850-git.sh | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/test/T850-git.sh b/test/T850-git.sh
index 994950ed..2badc52d 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -19,6 +19,16 @@ git -C tags.git ls-tree -r --name-only HEAD | xargs dirname 
| sort -u | sed s,ta
 notmuch search --output=messages '*' | sort > EXPECTED
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
+test_begin_subtest "commit, with quoted tag"
+test_subtest_known_broken
+notmuch git -C clone2.git -p '' clone tags.git
+git -C clone2.git ls-tree -r --name-only HEAD | grep /inbox > BEFORE
+notmuch tag '+"quoted tag"' '*'
+notmuch git -C clone2.git -p '' commit
+notmuch tag '-"quoted tag"' '*'
+git -C clone2.git ls-tree -r --name-only HEAD | grep /inbox > AFTER
+test_expect_equal_file_nonempty BEFORE AFTER
+
 test_begin_subtest "checkout"
 notmuch dump > BEFORE
 notmuch tag -inbox '*'
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 13/17] doc/notmuch-git: initial documentation

2022-05-15 Thread David Bremner
This is mainly derived from the various help outputs from the script,
with some massaging of markup and addition of links.

Define notmuch-git as a known external command.  This will allow
"notmuch help git" to invoke man notmuch-git, rather than erroring
out.
---
 doc/conf.py  |   4 +
 doc/index.rst|   1 +
 doc/man1/notmuch-git.rst | 271 +++
 notmuch.c|   2 +
 4 files changed, 278 insertions(+)
 create mode 100644 doc/man1/notmuch-git.rst

diff --git a/doc/conf.py b/doc/conf.py
index 6afeac06..8cc957b5 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -123,6 +123,10 @@ man_pages = [
  u'send mail with notmuch and emacs',
  [notmuch_authors], 1),
 
+('man1/notmuch-git', 'notmuch-git',
+ u'manage notmuch tags with git',
+ [notmuch_authors], 1),
+
 ('man5/notmuch-hooks', 'notmuch-hooks',
  u'hooks for notmuch',
  [notmuch_authors], 5),
diff --git a/doc/index.rst b/doc/index.rst
index fbdcf779..c380ee1d 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -15,6 +15,7 @@ Contents:
man1/notmuch-dump
notmuch-emacs
man1/notmuch-emacs-mua
+   man1/notmuch-git
man5/notmuch-hooks
man1/notmuch-insert
man1/notmuch-new
diff --git a/doc/man1/notmuch-git.rst b/doc/man1/notmuch-git.rst
new file mode 100644
index ..86f246b6
--- /dev/null
+++ b/doc/man1/notmuch-git.rst
@@ -0,0 +1,271 @@
+.. _notmuch-git(1):
+
+===
+notmuch-git
+===
+
+SYNOPSIS
+
+
+**notmuch** **git** [-h] [-C *repo*] [-p *prefix*] [-v] [-l *log level*] 
*subcommand*
+
+DESCRIPTION
+===
+
+Manage notmuch tags with Git.
+
+OPTIONS
+---
+
+Supported options for `notmuch git` include
+
+.. program:: notmuch-git
+
+.. option::  -h, --help
+
+   show help message and exit
+
+.. option:: -C , --git-dir 
+
+   Operate on git repository *repo*. See :ref:`repo_location` for
+   defaults.
+
+.. option::  -p , --tag-prefix 
+
+   Operate only on tags with prefix *prefix*. See :ref:`prefix_val` for
+   defaults.
+
+.. option::   -v, --version
+
+   show notmuch-git's version number and exit
+
+.. option::   -l , --log-level 
+
+   Log verbosity, one of: `critical`, `error`, `warning`, `info`,
+   `debug`. Defaults to `warning`.
+
+SUBCOMMANDS
+---
+
+For help on a particular subcommand, run: 'notmuch-git ...  --help'.
+
+.. program:: notmuch-git
+
+.. option:: archive [tree-ish] [arg ...]
+
+Dump a tar archive of a committed tag set using 'git archive'. See
+:any:`format` for details of the archive contents.
+
+   .. describe:: tree-ish
+
+   The tree or commit to produce an archive for. Defaults to 'HEAD'.
+
+   .. describe:: arg
+
+   If present, any optional arguments are passed through to
+   :manpage:`git-archive(1)`. Arguments to `git-archive` are reordered
+   so that *tree-ish* comes last.
+
+.. option:: checkout
+
+Update the notmuch database from Git.
+
+This is mainly useful to discard your changes in notmuch relative
+to Git.
+
+.. option:: clone 
+
+Create a local `notmuch git` repository from a remote source.
+
+This wraps 'git clone', adding some options to avoid creating a
+working tree while preserving remote-tracking branches and
+upstreams.
+
+.. describe:: repository
+
+The (possibly remote) repository to clone from. See the URLS
+section of :manpage:`git-clone(1)` for more information on
+specifying repositories.
+
+.. option:: commit [message]
+
+Commit prefix-matching tags from the notmuch database to Git.
+
+   .. describe:: message
+
+   Optional text for the commit message.
+
+.. option:: fetch [remote]
+
+Fetch changes from the remote repository.
+
+.. describe:: remote
+
+Override the default configured in `branch..remote` to fetch
+from a particular remote repository (e.g. `origin`).
+
+.. option:: help
+
+Show brief help for an `notmuch git` command.
+
+.. option:: init
+
+Create an empty `notmuch git` repository.
+
+This wraps 'git init' with a few extra steps to support subsequent
+status and commit commands.
+
+.. option:: log [arg ...]
+
+A wrapper for 'git log'.
+
+   .. describe:: arg
+
+   Additional arguments are passed through to 'git log'.
+
+After running `notmuch git fetch`, you can inspect the changes with
+
+::
+
+   $ notmuch git log HEAD..@{upstream}
+
+.. option:: merge [reference]
+
+Merge changes from 'reference' into HEAD and load the result into notmuch.
+
+   .. describe:: reference
+
+   Reference, usually other branch heads, to merge into our
+   branch. Defaults to `@{upstream}`.
+
+.. option:: pull [repository] [refspec ...]
+
+Pull (merge) remote repository changes to notmuch.
+
+**pull** is equivalent to **fetch** followed by **merge**.  We use the
+Git-configured repository for your current branch
+(`branch..repository`, likely `origin`, and `branch..merge`,
+likely `master` or `main`).
+
+   .. describe:: repository
+
+   The "remote" repository that is the source of the pull. This parameter
+   can be 

[PATCH 08/17] CLI/git: suppress warnings about initial branch name

2022-05-15 Thread David Bremner
The canonical nmbug repository still uses "master" as the main branch
name, so defer any potential switch away from that name.
---
 notmuch-git.in | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index f27591c9..78e4b140 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -353,7 +353,8 @@ def init(remote=None):
 This wraps 'git init' with a few extra steps to support subsequent
 status and commit commands.
 """
-_spawn(args=['git', '--git-dir', NOTMUCH_GIT_DIR, 'init', '--bare'], 
wait=True)
+_spawn(args=['git', '--git-dir', NOTMUCH_GIT_DIR, 'init',
+ '--initial-branch=master', '--quiet', '--bare'], wait=True)
 _git(args=['config', 'core.logallrefupdates', 'true'], wait=True)
 # create an empty blob (e69de29bb2d1d6434b8b29ae775ad8c2e48c5391)
 _git(args=['hash-object', '-w', '--stdin'], input='', wait=True)
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 12/17] CLI/git: cache git indices

2022-05-15 Thread David Bremner
If the private index file matches a previously known revision of the
database, we can update the index incrementally using the recorded
lastmod counter. This is typically much faster than a full update,
although it could be slower in the case of large changes to the
database.

The "git-read-tree HEAD" is also a bottleneck, but unfortunately
sometimes is needed. Cache the index checksum and hash to reduce the
number of times the operation is run. The overall design is a
simplified version of the PrivateIndex class.
---
 notmuch-git.in   | 318 ++-
 test/T850-git.sh |  41 ++
 2 files changed, 274 insertions(+), 85 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 0aabfb60..709d3d07 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -38,6 +38,7 @@ import tempfile as _tempfile
 import textwrap as _textwrap
 from urllib.parse import quote as _quote
 from urllib.parse import unquote as _unquote
+import json as _json
 
 __version__ = '@NOTMUCH_VERSION@'
 
@@ -301,41 +302,98 @@ def _is_committed(status):
 return len(status['added']) + len(status['deleted']) == 0
 
 
+class CachedIndex:
+def __init__(self, repo, treeish):
+self.cache_path = _os.path.join(repo, 'notmuch', 'index_cache.json')
+self.index_path = _os.path.join(repo, 'index')
+self.current_treeish = treeish
+# cached values
+self.treeish = None
+self.hash = None
+self.index_checksum = None
+
+self._load_cache_file()
+
+def _load_cache_file(self):
+try:
+with open(self.cache_path) as f:
+data = _json.load(f)
+self.treeish = data['treeish']
+self.hash = data['hash']
+self.index_checksum = data['index_checksum']
+except FileNotFoundError:
+pass
+except _json.JSONDecodeError:
+_LOG.error("Error decoding cache")
+_sys.exit(1)
+
+def __enter__(self):
+self.read_tree()
+return self
+
+def __exit__(self, type, value, traceback):
+checksum = _read_index_checksum(self.index_path)
+(_, hash, _) = _git(
+args=['rev-parse', self.current_treeish],
+stdout=_subprocess.PIPE,
+wait=True)
+
+with open(self.cache_path, "w") as f:
+_json.dump({'treeish': self.current_treeish,
+'hash': hash.rstrip(),  'index_checksum': checksum }, 
f)
+
+@timed
+def read_tree(self):
+current_checksum = _read_index_checksum(self.index_path)
+(_, hash, _) = _git(
+args=['rev-parse', self.current_treeish],
+stdout=_subprocess.PIPE,
+wait=True)
+current_hash = hash.rstrip()
+
+if self.current_treeish == self.treeish and \
+   self.index_checksum and self.index_checksum == current_checksum and 
\
+   self.hash and self.hash == current_hash:
+return
+
+_git(args=['read-tree', self.current_treeish], wait=True)
+
+
 def commit(treeish='HEAD', message=None):
 """
 Commit prefix-matching tags from the notmuch database to Git.
 """
+
 status = get_status()
 
 if _is_committed(status=status):
 _LOG.warning('Nothing to commit')
 return
 
-_git(args=['read-tree', '--empty'], wait=True)
-_git(args=['read-tree', treeish], wait=True)
-try:
-_update_index(status=status)
-(_, tree, _) = _git(
-args=['write-tree'],
-stdout=_subprocess.PIPE,
-wait=True)
-(_, parent, _) = _git(
-args=['rev-parse', treeish],
-stdout=_subprocess.PIPE,
-wait=True)
-(_, commit, _) = _git(
-args=['commit-tree', tree.strip(), '-p', parent.strip()],
-input=message,
-stdout=_subprocess.PIPE,
-wait=True)
-_git(
-args=['update-ref', treeish, commit.strip()],
-stdout=_subprocess.PIPE,
-wait=True)
-except Exception as e:
-_git(args=['read-tree', '--empty'], wait=True)
-_git(args=['read-tree', treeish], wait=True)
-raise
+with CachedIndex(NOTMUCH_GIT_DIR, treeish) as index:
+try:
+_update_index(status=status)
+(_, tree, _) = _git(
+args=['write-tree'],
+stdout=_subprocess.PIPE,
+wait=True)
+(_, parent, _) = _git(
+args=['rev-parse', treeish],
+stdout=_subprocess.PIPE,
+wait=True)
+(_, commit, _) = _git(
+args=['commit-tree', tree.strip(), '-p', parent.strip()],
+input=message,
+stdout=_subprocess.PIPE,
+wait=True)
+_git(
+args=['update-ref', treeish, commit.strip()],
+stdout=_subprocess.PIPE,
+

[PATCH 16/17] CLI/git: add safety checks for checkout and commit

2022-05-15 Thread David Bremner
Commits or checkouts that modify a large fraction of the messages in
the database should be relatively rare (and in some automated process,
probably non-existent). For initial setup, where such operations are
expected, the user can pass --force.
---
 doc/man1/notmuch-config.rst |  7 ++
 doc/man1/notmuch-git.rst| 14 ++--
 notmuch-git.in  | 43 +++--
 test/T850-git.sh| 39 -
 4 files changed, 94 insertions(+), 9 deletions(-)

diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index e2e9a632..388315f6 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -111,6 +111,13 @@ paths are presumed relative to `$HOME` for items in section
 
 Default location for git repository for :any:`notmuch-git`.
 
+.. nmconfig:: git.safe_fraction
+
+   Some :any:`notmuch-git` operations check that the fraction of
+   messages changed (in the database or in git, as appropriate) is not
+   too large. This item controls what fraction of total messages is
+   considered "not too large".
+
 .. nmconfig:: git.tag_prefix
 
 Default tag prefix (filter) for :any:`notmuch-git`.
diff --git a/doc/man1/notmuch-git.rst b/doc/man1/notmuch-git.rst
index ad859b80..fa7a748e 100644
--- a/doc/man1/notmuch-git.rst
+++ b/doc/man1/notmuch-git.rst
@@ -73,13 +73,18 @@ Dump a tar archive of a committed tag set using 'git 
archive'. See
:manpage:`git-archive(1)`. Arguments to `git-archive` are reordered
so that *tree-ish* comes last.
 
-.. option:: checkout
+.. option:: checkout [-f|--force]
 
 Update the notmuch database from Git.
 
 This is mainly useful to discard your changes in notmuch relative
 to Git.
 
+   .. describe:: [-f|--force]
+
+   Override checks that prevent modifying tags for large fractions of
+   messages in the database. See also :nmconfig:`git.safe_fraction`.
+
 .. option:: clone 
 
 Create a local `notmuch git` repository from a remote source.
@@ -94,7 +99,7 @@ upstreams.
 section of :manpage:`git-clone(1)` for more information on
 specifying repositories.
 
-.. option:: commit [message]
+.. option:: commit [-f|--force] [message]
 
 Commit prefix-matching tags from the notmuch database to Git.
 
@@ -102,6 +107,11 @@ Commit prefix-matching tags from the notmuch database to 
Git.
 
Optional text for the commit message.
 
+   .. describe:: -f|--force
+
+   Override checks that prevent modifying tags for large fractions of
+   messages in the database. See also :nmconfig:`git.safe_fraction`.
+
 .. option:: fetch [remote]
 
 Fetch changes from the remote repository.
diff --git a/notmuch-git.in b/notmuch-git.in
index 3e5205e8..6505c2e5 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -241,6 +241,16 @@ def _tag_query(prefix=None):
 prefix = TAG_PREFIX
 return '(tag (starts-with "{:s}"))'.format(prefix.replace('"','\\\"'))
 
+def count_messages(prefix=None):
+"count messages with a given prefix."
+(status, stdout, stderr) = _spawn(
+args=['notmuch', 'count', '--query=sexp', _tag_query(prefix)],
+stdout=_subprocess.PIPE, wait=True)
+if status != 0:
+_LOG.error("failed to run notmuch config")
+sys.exit(1)
+return int(stdout.rstrip())
+
 def get_tags(prefix=None):
 "Get a list of tags with a given prefix."
 (status, stdout, stderr) = _spawn(
@@ -359,7 +369,22 @@ class CachedIndex:
 _git(args=['read-tree', self.current_treeish], wait=True)
 
 
-def commit(treeish='HEAD', message=None):
+def check_safe_fraction(status):
+safe = 0.1
+conf = _notmuch_config_get ('git.safe_fraction')
+if conf and conf != '':
+safe=float(conf)
+
+total = count_messages (TAG_PREFIX)
+change = len(status['added'])+len(status['deleted'])+len(status['missing'])
+fraction = change/total
+_LOG.debug('total messages {:d}, change: {:d}, fraction: 
{:f}'.format(total,change,fraction))
+if fraction > safe:
+_LOG.error('safe fraction {:f} exceeded, stopping.'.format(safe))
+_LOG.error('Use --force to override or reconfigure git.safe_fraction.')
+exit(1)
+
+def commit(treeish='HEAD', message=None, force=False):
 """
 Commit prefix-matching tags from the notmuch database to Git.
 """
@@ -370,6 +395,9 @@ def commit(treeish='HEAD', message=None):
 _LOG.warning('Nothing to commit')
 return
 
+if not force:
+check_safe_fraction (status)
+
 with CachedIndex(NOTMUCH_GIT_DIR, treeish) as index:
 try:
 _update_index(status=status)
@@ -447,7 +475,7 @@ def init(remote=None):
 wait=True)
 
 
-def checkout():
+def checkout(force=None):
 """
 Update the notmuch database from Git.
 
@@ -455,6 +483,10 @@ def checkout():
 to Git.
 """
 status = get_status()
+
+if not force:
+check_safe_fraction(status)
+
 with _spawn(
 args=['notmuch', 'tag', '--batch'], 

[PATCH 07/17] CLI/git: rename environment variables.

2022-05-15 Thread David Bremner
Although the code required to support both new and old environment
variables is small, it complicates the semantics of configuration, and
make the documentation harder to follow.
---
 notmuch-git.in   | 35 ++-
 test/T850-git.sh |  6 +++---
 2 files changed, 17 insertions(+), 24 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 3a16914d..f27591c9 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -18,13 +18,6 @@
 
 """
 Manage notmuch tags with Git
-
-Environment variables:
-
-* NMBGIT specifies the location of the git repository used by nmbug.
-  If not specified $HOME/.nmbug is used.
-* NMBPREFIX specifies the prefix in the notmuch database for tags of
-  interest to nmbug. If not specified 'notmuch::' is used.
 """
 
 from __future__ import print_function
@@ -52,7 +45,7 @@ _LOG = _logging.getLogger('nmbug')
 _LOG.setLevel(_logging.WARNING)
 _LOG.addHandler(_logging.StreamHandler())
 
-NMBGIT = None
+NOTMUCH_GIT_DIR = None
 TAG_PREFIX = None
 
 _HEX_ESCAPE_REGEX = _re.compile('%[0-9A-F]{2}')
@@ -200,7 +193,7 @@ def _spawn(args, input=None, additional_env=None, 
wait=False, stdin=None,
 
 
 def _git(args, **kwargs):
-args = ['git', '--git-dir', NMBGIT] + list(args)
+args = ['git', '--git-dir', NOTMUCH_GIT_DIR] + list(args)
 return _spawn(args=args, **kwargs)
 
 
@@ -268,7 +261,7 @@ def clone(repository):
 with _tempfile.TemporaryDirectory(prefix='nmbug-clone.') as workdir:
 _spawn(
 args=[
-'git', 'clone', '--no-checkout', '--separate-git-dir', NMBGIT,
+'git', 'clone', '--no-checkout', '--separate-git-dir', 
NOTMUCH_GIT_DIR,
 repository, workdir],
 wait=True)
 _git(args=['config', '--unset', 'core.worktree'], wait=True, expect=(0, 5))
@@ -360,7 +353,7 @@ def init(remote=None):
 This wraps 'git init' with a few extra steps to support subsequent
 status and commit commands.
 """
-_spawn(args=['git', '--git-dir', NMBGIT, 'init', '--bare'], wait=True)
+_spawn(args=['git', '--git-dir', NOTMUCH_GIT_DIR, 'init', '--bare'], 
wait=True)
 _git(args=['config', 'core.logallrefupdates', 'true'], wait=True)
 # create an empty blob (e69de29bb2d1d6434b8b29ae775ad8c2e48c5391)
 _git(args=['hash-object', '-w', '--stdin'], input='', wait=True)
@@ -368,7 +361,7 @@ def init(remote=None):
 args=[
 'commit', '--allow-empty', '-m', 'Start a new nmbug repository'
 ],
-additional_env={'GIT_WORK_TREE': NMBGIT},
+additional_env={'GIT_WORK_TREE': NOTMUCH_GIT_DIR},
 wait=True)
 
 
@@ -591,7 +584,7 @@ def get_status():
 
 def _index_tags():
 "Write notmuch tags to the nmbug.index."
-path = _os.path.join(NMBGIT, 'nmbug.index')
+path = _os.path.join(NOTMUCH_GIT_DIR, 'nmbug.index')
 query = ' '.join('tag:"{tag}"'.format(tag=tag) for tag in get_tags())
 prefix = '+{0}'.format(_ENCODED_TAG_PREFIX)
 _git(
@@ -710,7 +703,7 @@ if __name__ == '__main__':
 help='Git repository to operate on.')
 parser.add_argument(
 '-p', '--tag-prefix', metavar='PREFIX',
-default = _os.getenv('NMBPREFIX', 'notmuch::'),
+default = _os.getenv('NOTMUCH_GIT_PREFIX', 'notmuch::'),
 help='Prefix of tags to operate on.')
 parser.add_argument(
 '-v', '--version', action='version',
@@ -826,13 +819,13 @@ if __name__ == '__main__':
 args = parser.parse_args()
 
 if args.git_dir:
-NMBGIT = args.git_dir
+NOTMUCH_GIT_DIR = args.git_dir
 else:
-NMBGIT = _os.path.expanduser(
-_os.getenv('NMBGIT', _os.path.join('~', '.nmbug')))
-_NMBGIT = _os.path.join(NMBGIT, '.git')
-if _os.path.isdir(_NMBGIT):
-NMBGIT = _NMBGIT
+NOTMUCH_GIT_DIR = _os.path.expanduser(
+_os.getenv('NOTMUCH_GIT_DIR', _os.path.join('~', '.nmbug')))
+_NOTMUCH_GIT_DIR = _os.path.join(NOTMUCH_GIT_DIR, '.git')
+if _os.path.isdir(_NOTMUCH_GIT_DIR):
+NOTMUCH_GIT_DIR = _NOTMUCH_GIT_DIR
 
 TAG_PREFIX = args.tag_prefix
 _ENCODED_TAG_PREFIX = _hex_quote(TAG_PREFIX, safe='+@=,')  # quote ':'
@@ -842,7 +835,7 @@ if __name__ == '__main__':
 _LOG.setLevel(level)
 
 # for test suite
-for var in ['NMBGIT', 'NMBPREFIX', 'NOTMUCH_PROFILE', 'NOTMUCH_CONFIG' ]:
+for var in ['NOTMUCH_GIT_DIR', 'NOTMUCH_GIT_PREFIX', 'NOTMUCH_PROFILE', 
'NOTMUCH_CONFIG' ]:
 _LOG.debug('env {:s} = {:s}'.format(var, _os.getenv(var,'%None%')))
 
 if not getattr(args, 'func', None):
diff --git a/test/T850-git.sh b/test/T850-git.sh
index 713b326f..994950ed 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -80,11 +80,11 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "environment passed through when run as 'notmuch git'"
-env NMBGIT=foo NMBPREFIX=bar NOTMUCH_PROFILE=default notmuch git -C tags.git 
-p '' -ldebug status |& \
+env NOTMUCH_GIT_DIR=foo NOTMUCH_GIT_PREFIX=bar 

[PATCH 03/17] notmuch-git: add --git-dir, --tag-prefix arguments

2022-05-15 Thread David Bremner
It is often more convenient to use command line arguments than
environment variables.
---
 notmuch-git.in | 31 +--
 1 file changed, 21 insertions(+), 10 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 5d6e29f0..4b94046d 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -52,13 +52,9 @@ _LOG = _logging.getLogger('nmbug')
 _LOG.setLevel(_logging.WARNING)
 _LOG.addHandler(_logging.StreamHandler())
 
-NMBGIT = _os.path.expanduser(
-_os.getenv('NMBGIT', _os.path.join('~', '.nmbug')))
-_NMBGIT = _os.path.join(NMBGIT, '.git')
-if _os.path.isdir(_NMBGIT):
-NMBGIT = _NMBGIT
+NMBGIT = None
+TAG_PREFIX = None
 
-TAG_PREFIX = _os.getenv('NMBPREFIX', 'notmuch::')
 _HEX_ESCAPE_REGEX = _re.compile('%[0-9A-F]{2}')
 _TAG_DIRECTORY = 'tags/'
 _TAG_FILE_REGEX = _re.compile(_TAG_DIRECTORY + '(?P[^/]*)/(?P[^/]*)')
@@ -79,10 +75,6 @@ def _hex_quote(string, safe='+@=:,'):
 lambda match: match.group(0).lower(),
 uppercase_escapes)
 
-
-_ENCODED_TAG_PREFIX = _hex_quote(TAG_PREFIX, safe='+@=,')  # quote ':'
-
-
 def _xapian_quote(string):
 """
 Quote a string for Xapian's QueryParser.
@@ -688,6 +680,13 @@ if __name__ == '__main__':
 parser = argparse.ArgumentParser(
 description=__doc__.strip(),
 formatter_class=argparse.RawDescriptionHelpFormatter)
+parser.add_argument(
+'-C', '--git-dir', metavar='REPO',
+help='Git repository to operate on.')
+parser.add_argument(
+'-p', '--tag-prefix', metavar='PREFIX',
+default = _os.getenv('NMBPREFIX', 'notmuch::'),
+help='Prefix of tags to operate on.')
 parser.add_argument(
 '-v', '--version', action='version',
 version='%(prog)s {}'.format(__version__))
@@ -800,6 +799,18 @@ if __name__ == '__main__':
 
 args = parser.parse_args()
 
+if args.git_dir:
+NMBGIT = args.git_dir
+else:
+NMBGIT = _os.path.expanduser(
+_os.getenv('NMBGIT', _os.path.join('~', '.nmbug')))
+_NMBGIT = _os.path.join(NMBGIT, '.git')
+if _os.path.isdir(_NMBGIT):
+NMBGIT = _NMBGIT
+
+TAG_PREFIX = args.tag_prefix
+_ENCODED_TAG_PREFIX = _hex_quote(TAG_PREFIX, safe='+@=,')  # quote ':'
+
 if args.log_level:
 level = getattr(_logging, args.log_level.upper())
 _LOG.setLevel(level)
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 02/17] CLI/git: drop support for python < 3.2

2022-05-15 Thread David Bremner
Debian stable had python 3.4.2 3 releases ago (approximately 6 years
ago), so attempting to keep track of the changes in python is probably
no longer worthwhile. We already require python 3.5 for the
python-cffi bindings (although those are not yet used in notmuch-git).
---
 notmuch-git.in | 34 ++
 1 file changed, 2 insertions(+), 32 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index dd30b5be..5d6e29f0 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -43,13 +43,8 @@ import subprocess as _subprocess
 import sys as _sys
 import tempfile as _tempfile
 import textwrap as _textwrap
-try:  # Python 3
-from urllib.parse import quote as _quote
-from urllib.parse import unquote as _unquote
-except ImportError:  # Python 2
-from urllib import quote as _quote
-from urllib import unquote as _unquote
-
+from urllib.parse import quote as _quote
+from urllib.parse import unquote as _unquote
 
 __version__ = '@NOTMUCH_VERSION@'
 
@@ -71,31 +66,6 @@ _TAG_FILE_REGEX = _re.compile(_TAG_DIRECTORY + 
'(?P[^/]*)/(?P[^/]*)')
 # magic hash for Git (git hash-object -t blob /dev/null)
 _EMPTYBLOB = 'e69de29bb2d1d6434b8b29ae775ad8c2e48c5391'
 
-
-try:
-getattr(_tempfile, 'TemporaryDirectory')
-except AttributeError:  # Python < 3.2
-class _TemporaryDirectory(object):
-"""
-Fallback context manager for Python < 3.2
-
-See PEP 343 for details on context managers [1].
-
-[1]: https://www.python.org/dev/peps/pep-0343/
-"""
-def __init__(self, **kwargs):
-self.name = _tempfile.mkdtemp(**kwargs)
-
-def __enter__(self):
-return self.name
-
-def __exit__(self, type, value, traceback):
-_shutil.rmtree(self.name)
-
-
-_tempfile.TemporaryDirectory = _TemporaryDirectory
-
-
 def _hex_quote(string, safe='+@=:,'):
 """
 quote('abc def') -> 'abc%20def'.
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 17/17] debian: install notmuch-git

2022-05-15 Thread David Bremner
Use a separate binary package to avoid dragging in dependencies on
python and git for those that do not want them.
---
 debian/control  | 16 
 debian/notmuch-git.install  |  2 ++
 debian/notmuch-git.manpages |  2 ++
 3 files changed, 20 insertions(+)
 create mode 100644 debian/notmuch-git.install
 create mode 100644 debian/notmuch-git.manpages

diff --git a/debian/control b/debian/control
index a11d4130..9706b0f7 100644
--- a/debian/control
+++ b/debian/control
@@ -66,6 +66,22 @@ Description: thread-based email index, search and tagging
  .
  This package contains the notmuch command-line interface
 
+Package: notmuch-git
+Architecture: all
+Depends:
+ git,
+ notmuch,
+ python3,
+ ${misc:Depends}
+Description: thread-based email index, search and tagging
+ Notmuch is a system for indexing, searching, reading, and tagging
+ large collections of email messages in maildir or mh format. It uses
+ the Xapian library to provide fast, full-text search with a very
+ convenient search syntax.
+ .
+ This package contains a simple tool to save, restore, and synchronize
+ notmuch tags via git repositories.
+
 Package: notmuch-doc
 Architecture: all
 Depends:
diff --git a/debian/notmuch-git.install b/debian/notmuch-git.install
new file mode 100644
index ..2be08276
--- /dev/null
+++ b/debian/notmuch-git.install
@@ -0,0 +1,2 @@
+notmuch-git /usr/bin
+nmbug /usr/bin
diff --git a/debian/notmuch-git.manpages b/debian/notmuch-git.manpages
new file mode 100644
index ..e0895c86
--- /dev/null
+++ b/debian/notmuch-git.manpages
@@ -0,0 +1,2 @@
+usr/share/man/man1/notmuch-git.1.gz
+usr/share/man/man1/nmbug.1.gz
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 10/17] CLI/git: replace enumeration of tags with sexp query.

2022-05-15 Thread David Bremner
Unlike the (current) infix query parser provided by Xapian, the
notmuch specific sexp query parser supports prefixed wildcard queries,
so use those. In addition to being somewhat faster, this avoids
needing to escape all of the user's tags to pass via the shell.
---
 notmuch-git.in   | 26 +++---
 test/T850-git.sh |  6 +-
 2 files changed, 24 insertions(+), 8 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 78e4b140..1d2dcd53 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -221,16 +221,17 @@ def _get_remote():
 stdout=_subprocess.PIPE, wait=True)
 return remote.strip()
 
+def _tag_query(prefix=None):
+if prefix is None:
+prefix = TAG_PREFIX
+return '(tag (starts-with "{:s}"))'.format(prefix.replace('"','\\\"'))
 
 def get_tags(prefix=None):
 "Get a list of tags with a given prefix."
-if prefix is None:
-prefix = TAG_PREFIX
 (status, stdout, stderr) = _spawn(
-args=['notmuch', 'search', '--output=tags', '*'],
+args=['notmuch', 'search', '--query=sexp', '--output=tags', 
_tag_query(prefix)],
 stdout=_subprocess.PIPE, wait=True)
-return [tag for tag in stdout.splitlines() if tag.startswith(prefix)]
-
+return [tag for tag in stdout.splitlines()]
 
 def archive(treeish='HEAD', args=()):
 """
@@ -586,13 +587,12 @@ def get_status():
 def _index_tags():
 "Write notmuch tags to the nmbug.index."
 path = _os.path.join(NOTMUCH_GIT_DIR, 'nmbug.index')
-query = ' '.join('tag:"{tag}"'.format(tag=tag) for tag in get_tags())
 prefix = '+{0}'.format(_ENCODED_TAG_PREFIX)
 _git(
 args=['read-tree', '--empty'],
 additional_env={'GIT_INDEX_FILE': path}, wait=True)
 with _spawn(
-args=['notmuch', 'dump', '--format=batch-tag', '--', query],
+args=['notmuch', 'dump', '--format=batch-tag', '--query=sexp', 
'--', _tag_query()],
 stdout=_subprocess.PIPE) as notmuch:
 with _git(
 args=['update-index', '--index-info'],
@@ -692,6 +692,14 @@ def _help(parser, command=None):
 else:
 parser.parse_args(['--help'])
 
+def _notmuch_config_get(key):
+(status, stdout, stderr) = _spawn(
+args=['notmuch', 'config', 'get', key],
+stdout=_subprocess.PIPE, wait=True)
+if status != 0:
+_LOG.error("failed to run notmuch config")
+sys.exit(1)
+return stdout.rstrip()
 
 if __name__ == '__main__':
 import argparse
@@ -839,6 +847,10 @@ if __name__ == '__main__':
 for var in ['NOTMUCH_GIT_DIR', 'NOTMUCH_GIT_PREFIX', 'NOTMUCH_PROFILE', 
'NOTMUCH_CONFIG' ]:
 _LOG.debug('env {:s} = {:s}'.format(var, _os.getenv(var,'%None%')))
 
+if _notmuch_config_get('built_with.sexp_queries') != 'true':
+_LOG.error("notmuch git needs sexp query support")
+sys.exit(1)
+
 if not getattr(args, 'func', None):
 parser.print_usage()
 _sys.exit(1)
diff --git a/test/T850-git.sh b/test/T850-git.sh
index 2badc52d..72091b56 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -2,6 +2,11 @@
 test_description='"notmuch git" to save and restore tags'
 . $(dirname "$0")/test-lib.sh || exit 1
 
+if [ $NOTMUCH_HAVE_SFSEXP -ne 1 ]; then
+printf "Skipping due to missing sfsexp library\n"
+test_done
+fi
+
 add_email_corpus
 
 git config --global user.email notm...@example.org
@@ -20,7 +25,6 @@ notmuch search --output=messages '*' | sort > EXPECTED
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
 test_begin_subtest "commit, with quoted tag"
-test_subtest_known_broken
 notmuch git -C clone2.git -p '' clone tags.git
 git -C clone2.git ls-tree -r --name-only HEAD | grep /inbox > BEFORE
 notmuch tag '+"quoted tag"' '*'
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 05/17] CLI/git: Add an 'init' command

2022-05-15 Thread David Bremner
From: "W. Trevor King" 

For folks that want to start versioning a new tag-space, instead of
cloning one that someone else has already started.

The empty-blob hash-object call avoids errors like:

  $ nmbug commit
  error: invalid object 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 for
'tags/...'
  fatal: git-write-tree: error building trees
  'git HASH(0x9ef3eb8) write-tree' exited with nonzero value

David Bremner suggested [1]:

  $ git hash-object -w /dev/null

instead of my Python version of:

  $ git hash-object -w --stdin <&-

but I expect that closing stdin is more portable than the /dev/null
path (which doesn't exist on Windows, for example).

The --bare init and use of NMBGIT as the work tree (what could go
wrong with an empty commit?) are suggestions from Michal Sojka [2].

[1]: id:87y4vu6uvf@maritornes.cs.unb.ca
 http://thread.gmane.org/gmane.mail.notmuch.general/18626/focus=18720
[2]: id:87a93a5or2@resox.2x.cz
 http://thread.gmane.org/gmane.mail.notmuch.general/19495/focus=19767
---
 notmuch-git.in | 20 
 1 file changed, 20 insertions(+)

diff --git a/notmuch-git.in b/notmuch-git.in
index 57b5a0ec..71a4df72 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -353,6 +353,25 @@ def fetch(remote=None):
 _git(args=args, wait=True)
 
 
+def init(remote=None):
+"""
+Create an empty nmbug repository.
+
+This wraps 'git init' with a few extra steps to support subsequent
+status and commit commands.
+"""
+_spawn(args=['git', '--git-dir', NMBGIT, 'init', '--bare'], wait=True)
+_git(args=['config', 'core.logallrefupdates', 'true'], wait=True)
+# create an empty blob (e69de29bb2d1d6434b8b29ae775ad8c2e48c5391)
+_git(args=['hash-object', '-w', '--stdin'], input='', wait=True)
+_git(
+args=[
+'commit', '--allow-empty', '-m', 'Start a new nmbug repository'
+],
+additional_env={'GIT_WORK_TREE': NMBGIT},
+wait=True)
+
+
 def checkout():
 """
 Update the notmuch database from Git.
@@ -716,6 +735,7 @@ if __name__ == '__main__':
 'commit',
 'fetch',
 'help',
+'init',
 'log',
 'merge',
 'pull',
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 04/17] CLI/git: make existance of config branch optional on clone

2022-05-15 Thread David Bremner
This branch is actually only used by an associated
utility (notmuch-report), and notmuch-git works fine without it.
---
 notmuch-git.in | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 4b94046d..57b5a0ec 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -273,7 +273,13 @@ def clone(repository):
 wait=True)
 _git(args=['config', '--unset', 'core.worktree'], wait=True, expect=(0, 5))
 _git(args=['config', 'core.bare', 'true'], wait=True)
-_git(args=['branch', 'config', 'origin/config'], wait=True)
+(status, stdout, stderr) = _git(args=['show-ref', '--verify',
+  '--quiet',
+  'refs/remotes/origin/config'],
+expect=(0,1),
+wait=True)
+if status == 0:
+_git(args=['branch', 'config', 'origin/config'], wait=True)
 existing_tags = get_tags()
 if existing_tags:
 _LOG.warning(
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 01/17] nmbug: promote to user tool "notmuch-git"

2022-05-15 Thread David Bremner
So far this is just a rename. Documentation and tests to follow.
---
 Makefile.local  | 7 +--
 devel/nmbug/nmbug => notmuch-git.in | 2 +-
 2 files changed, 6 insertions(+), 3 deletions(-)
 rename devel/nmbug/nmbug => notmuch-git.in (99%)

diff --git a/Makefile.local b/Makefile.local
index d8bbf3e1..ca2310f4 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -1,7 +1,7 @@
 # -*- makefile-gmake -*-
 
 .PHONY: all
-all: notmuch notmuch-shared build-man build-info ruby-bindings 
python-cffi-bindings
+all: notmuch notmuch-shared build-man build-info ruby-bindings 
python-cffi-bindings notmuch-git
 ifeq ($(MAKECMDGOALS),)
 ifeq ($(shell cat .first-build-message 2>/dev/null),)
@NOTMUCH_FIRST_BUILD=1 $(MAKE) --no-print-directory all
@@ -294,7 +294,7 @@ endif
 SRCS  := $(SRCS) $(notmuch_client_srcs)
 CLEAN := $(CLEAN) notmuch notmuch-shared $(notmuch_client_modules)
 CLEAN := $(CLEAN) version.stamp notmuch-*.tar.gz.tmp
-CLEAN := $(CLEAN) .deps
+CLEAN := $(CLEAN) .deps notmuch-git
 
 DISTCLEAN := $(DISTCLEAN) .first-build-message Makefile.config sh.config 
sphinx.config
 
@@ -307,6 +307,9 @@ cppcheck:
@echo "No cppcheck found during configure; skipping static checking"
 endif
 
+notmuch-git: notmuch-git.in
+   sed s/@NOTMUCH_VERSION@/${VERSION}/ < notmuch-git.in > notmuch-git
+   chmod ugo+rx notmuch-git
 
 DEPS := $(SRCS:%.c=.deps/%.d)
 DEPS := $(DEPS:%.cc=.deps/%.d)
diff --git a/devel/nmbug/nmbug b/notmuch-git.in
similarity index 99%
rename from devel/nmbug/nmbug
rename to notmuch-git.in
index 043c1863..dd30b5be 100755
--- a/devel/nmbug/nmbug
+++ b/notmuch-git.in
@@ -51,7 +51,7 @@ except ImportError:  # Python 2
 from urllib import unquote as _unquote
 
 
-__version__ = '0.3'
+__version__ = '@NOTMUCH_VERSION@'
 
 _LOG = _logging.getLogger('nmbug')
 _LOG.setLevel(_logging.WARNING)
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


notmuch-git

2022-05-15 Thread David Bremner
This series obsolates the WIP series at [1]. The rather long interdiff
is given below [2].

At this point I think the tool is feature complete, at least enough
for initial release. The main changes since the last series are the
addition of configuration variables and a "safe_fraction" check that
asks the user to override any operation that would change "lots" of
the database or git repo.  There is also a bunch more documentation
and tests. Finally I dropped support for old python versions (older
than 3.2, including python2). I'll try to get those into master
relatively quickly, to make reviewing this series easier.

This series needs to be applied on top of a couple of patches not yet in master 
[3]

[1]: id:20220423133848.3852688-1-da...@tethera.net

[2]: interdiff

[3]: definitely id:20220513122929.1981190-1-da...@tethera.net and
probably id:20220514122406.2220621-1-da...@tethera.net

diff --git a/Makefile.local b/Makefile.local
index ca2310f4..0fadfb26 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -1,7 +1,7 @@
 # -*- makefile-gmake -*-
 
 .PHONY: all
-all: notmuch notmuch-shared build-man build-info ruby-bindings 
python-cffi-bindings notmuch-git
+all: notmuch notmuch-shared build-man build-info ruby-bindings 
python-cffi-bindings notmuch-git nmbug
 ifeq ($(MAKECMDGOALS),)
 ifeq ($(shell cat .first-build-message 2>/dev/null),)
@NOTMUCH_FIRST_BUILD=1 $(MAKE) --no-print-directory all
@@ -307,9 +307,10 @@ cppcheck:
@echo "No cppcheck found during configure; skipping static checking"
 endif
 
-notmuch-git: notmuch-git.in
+nmbug notmuch-git: notmuch-git.in
sed s/@NOTMUCH_VERSION@/${VERSION}/ < notmuch-git.in > notmuch-git
chmod ugo+rx notmuch-git
+   ln -sf notmuch-git nmbug
 
 DEPS := $(SRCS:%.c=.deps/%.d)
 DEPS := $(DEPS:%.cc=.deps/%.d)
diff --git a/debian/notmuch-git.install b/debian/notmuch-git.install
index ca632418..2be08276 100644
--- a/debian/notmuch-git.install
+++ b/debian/notmuch-git.install
@@ -1 +1,2 @@
 notmuch-git /usr/bin
+nmbug /usr/bin
diff --git a/debian/notmuch-git.manpages b/debian/notmuch-git.manpages
new file mode 100644
index ..e0895c86
--- /dev/null
+++ b/debian/notmuch-git.manpages
@@ -0,0 +1,2 @@
+usr/share/man/man1/notmuch-git.1.gz
+usr/share/man/man1/nmbug.1.gz
diff --git a/doc/Makefile.local b/doc/Makefile.local
index d43ef269..2f67f4de 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -131,6 +131,7 @@ install-man: ${MAN_GZIP_FILES}
install -m0644 $(filter %.5.gz,$(MAN_GZIP_FILES)) 
$(DESTDIR)/$(mandir)/man5
install -m0644 $(filter %.7.gz,$(MAN_GZIP_FILES)) 
$(DESTDIR)/$(mandir)/man7
cd $(DESTDIR)/$(mandir)/man1 && ln -sf notmuch.1.gz notmuch-setup.1.gz
+   cd $(DESTDIR)/$(mandir)/man1 && ln -sf notmuch-git.1.gz nmbug.1.gz
 endif
 
 ifneq ($(HAVE_SPHINX)$(HAVE_MAKEINFO),11)
diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 36d48725..388315f6 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -107,6 +107,21 @@ paths are presumed relative to `$HOME` for items in section
 
 Default: see :ref:`database`
 
+.. nmconfig:: git.path
+
+Default location for git repository for :any:`notmuch-git`.
+
+.. nmconfig:: git.safe_fraction
+
+   Some :any:`notmuch-git` operations check that the fraction of
+   messages changed (in the database or in git, as appropriate) is not
+   too large. This item controls what fraction of total messages is
+   considered "not too large".
+
+.. nmconfig:: git.tag_prefix
+
+Default tag prefix (filter) for :any:`notmuch-git`.
+
 .. nmconfig:: index.decrypt
 
 Policy for decrypting encrypted messages during indexing.  Must be
diff --git a/doc/man1/notmuch-git.rst b/doc/man1/notmuch-git.rst
index 4877f22d..fa7a748e 100644
--- a/doc/man1/notmuch-git.rst
+++ b/doc/man1/notmuch-git.rst
@@ -1,23 +1,25 @@
 .. _notmuch-git(1):
 
-
+===
 notmuch-git
-
+===
 
 SYNOPSIS
 
 
-**notmuch** **git** [-h] [-C REPO] [-p PREFIX] [-v] [-l *log level*] 
*subcommand*
+**notmuch** **git** [-h] [-N] [-C *repo*] [-p *prefix*] [-v] [-l *log level*] 
*subcommand*
+
+**nmbug** [-h] [-C *repo*] [-p *prefix*] [-v] [-l *log level*] *subcommand*
 
 DESCRIPTION
 ===
 
 Manage notmuch tags with Git.
 
-Options
+OPTIONS
 ---
 
-Supported options for **notmuch-git** include
+Supported options for `notmuch git` include
 
 .. program:: notmuch-git
 
@@ -25,77 +27,271 @@ Supported options for **notmuch-git** include
 
show help message and exit
 
-.. option:: -C repo, --git-dir repo
+.. option:: -N, --nmbug
+
+   Set defaults for :option:`--tag-prefix` and :option:`--git-dir` suitable 
for the
+   :any:`notmuch` bug tracker
 
-   Operate on git repository *repo*
+.. option:: -C , --git-dir 
 
-.. option::  -p prefix, --tag-prefix prefix
+   Operate on git repository *repo*. See :ref:`repo_location` for
+   defaults.
 
-   Operate only on tags with prefix *prefix*
+.. option:: -p , 

[PATCH] emacs: document/defcustom notmuch-multipart/alternative-discouraged

2022-05-14 Thread David Bremner
This variable is important for people who want to change the default
behaviour when displaying multipart/alternative messages.  Previously
it was undocumented.  Add a defcustom to help users and copy some
documentation from the wiki. The usual machinery of re-using
docstrings is a bit tricky to use here because it mangles the example
lisp code, and the link to the info node should not be in e.g. the
html page.

Add a simple test to make sure the switch from defvar to defcustom did
not break something obvious.
---
 doc/notmuch-emacs.rst | 21 +++
 emacs/notmuch-lib.el  | 12 +++-
 test/T450-emacs-show.sh   |  4 ++
 .../notmuch-show-multipart-alternative| 62 +++
 4 files changed, 97 insertions(+), 2 deletions(-)
 create mode 100644 
test/emacs-show.expected-output/notmuch-show-multipart-alternative

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index 41f62390..27437f35 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -235,6 +235,27 @@ Display of messages can be controlled by the following 
variables
 :index:`notmuch-show-header-line`
|docstring::notmuch-show-header-line|
 
+:index:`notmuch-multipart/alternative-discouraged`
+
+   Which mime types to hide by default for multipart messages.
+
+   Can either be a list of mime types (as strings) or a function
+   mapping a plist representing the current message to such a list.
+   The following example function would discourage `text/html` and
+   `multipart/related` generally, but discourage `text/plain` should
+   the message be sent from `whate...@example.com`.
+
+   .. code:: lisp
+
+  (defun my--determine-discouraged (msg)
+(let* ((headers (plist-get msg :headers))
+   (from (or (plist-get headers :From) "")))
+  (cond
+   ((string-match "whate...@example.com" from)
+(list "text/plain"))
+   (t
+(list "text/html" "multipart/related")
+
 .. _show-copy:
 
 Copy to kill-ring
diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 6fc71cc7..ad884308 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -567,12 +567,20 @@ Take wildcards into account."
   (string= (downcase t1)
(downcase t2))
 
-(defvar notmuch-multipart/alternative-discouraged
+(defcustom notmuch-multipart/alternative-discouraged
   '(;; Avoid HTML parts.
 "text/html"
 ;; multipart/related usually contain a text/html part and some
 ;; associated graphics.
-"multipart/related"))
+"multipart/related")
+  "Which mime types to hide by default for multipart messages.
+
+Can either be a list of mime types (as strings) or a function
+mapping a plist representing the current message to such a list.
+See Info node `(notmuch-emacs) notmuch-show' for a sample function."
+  :group 'notmuch-show
+  :type '(radio (repeat :tag "MIME Types" string)
+   (function :tag "Function")))
 
 (defun notmuch-multipart/alternative-determine-discouraged (msg)
   "Return the discouraged alternatives for the specified message."
diff --git a/test/T450-emacs-show.sh b/test/T450-emacs-show.sh
index 057ad37e..7a0002c4 100755
--- a/test/T450-emacs-show.sh
+++ b/test/T450-emacs-show.sh
@@ -209,6 +209,10 @@ test_emacs '(notmuch-show "id:'$gen_msg_id'")
 output=$(head -1 OUTPUT.raw|cut -f1-4 -d' ')
 test_expect_equal "$output" "Notmuch Test Suite "
 
+test_begin_subtest "multipart/alternative hides html by default"
+test_emacs '(notmuch-show 
"id:cf0c4d610911171136h1713aa59w9cf9aa31f052a...@mail.gmail.com")
+   (test-visible-output)'
+test_expect_equal_file $EXPECTED/notmuch-show-multipart-alternative OUTPUT
 
 # switching to the crypto corpus, using gpg from here on:
 add_gnupg_home
diff --git a/test/emacs-show.expected-output/notmuch-show-multipart-alternative 
b/test/emacs-show.expected-output/notmuch-show-multipart-alternative
new file mode 100644
index ..e44926bc
--- /dev/null
+++ b/test/emacs-show.expected-output/notmuch-show-multipart-alternative
@@ -0,0 +1,62 @@
+Alex Botero-Lowry  (2009-11-17) (attachment inbox)
+Subject: [notmuch] preliminary FreeBSD support
+To: notmuch@notmuchmail.org
+Date: Tue, 17 Nov 2009 11:36:14 -0800
+
+[ multipart/mixed ]
+[ multipart/alternative ]
+[ text/plain ]
+I saw the announcement this morning, and was very excited, as I had been
+hoping sup would be turned into a library,
+since I like the concept more than the UI (I'd rather an emacs interface).
+
+I did a preliminary compile which worked out fine, but
+sysconf(_SC_SC_GETPW_R_SIZE_MAX) returns -1 on
+FreeBSD, so notmuch_config_open segfaulted.
+
+Attached is a patch that supplies a default buffer size of 64 in cases where
+-1 is returned.
+
+http://www.opengroup.org/austin/docs/austin_328.txt - seems to indicate this
+is acceptable behavior,
+and
+http://mail-index.netbsd.org/pkgsrc-bugs/2006/06/07/msg016808.htmlspecifically
+uses 64 as 

[PATCH] doc: define and use semantic markup for configuration items

2022-05-13 Thread David Bremner
This makes sure each configuration item is cross referenceable without
extra markup, and also adds index entries.
---
 doc/conf.py |  7 +
 doc/man1/notmuch-config.rst | 60 +++--
 doc/man1/notmuch-insert.rst | 15 +-
 doc/man1/notmuch-new.rst|  8 ++---
 doc/man1/notmuch-show.rst   |  2 +-
 5 files changed, 50 insertions(+), 42 deletions(-)

diff --git a/doc/conf.py b/doc/conf.py
index e46e1d4e..6afeac06 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -200,3 +200,10 @@ texinfo_documents += [
 x[2],  # description
 'Miscellaneous'# category
 ) for x in man_pages]
+
+def setup(app):
+import docutils.nodes
+# define nmconfig role and directive for config items.
+app.add_object_type('nmconfig','nmconfig',
+indextemplate='pair: configuration item; %s',
+ref_nodeclass=docutils.nodes.generated)
diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index acc5ecec..36d48725 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -55,13 +55,14 @@ The available configuration items are described below. 
Non-absolute
 paths are presumed relative to `$HOME` for items in section
 **database**.
 
-built_with.
+.. nmconfig:: built_with.
+
 Compile time feature . Current possibilities include
 "retry_lock" (configure option, included by default).
 (since notmuch 0.30, "compact" and "field_processor" are
 always included.)
 
-database.autocommit
+.. nmconfig:: database.autocommit
 
 How often to commit transactions to disk. `0` means wait until
 command completes, otherwise an integer `n` specifies to commit to
@@ -69,7 +70,8 @@ database.autocommit
 
 History: this configuration value was introduced in notmuch 0.33.
 
-database.backup_dir
+.. nmconfig:: database.backup_dir
+
 Directory to store tag dumps when upgrading database.
 
 History: this configuration value was introduced in notmuch 0.32.
@@ -77,7 +79,8 @@ database.backup_dir
 Default: A sibling directory of the Xapian database called
 `backups`.
 
-database.hook_dir
+.. nmconfig:: database.hook_dir
+
 Directory containing hooks run by notmuch commands. See
 :any:`notmuch-hooks(5)`.
 
@@ -85,9 +88,8 @@ database.hook_dir
 
 Default: See HOOKS, below.
 
-.. _database.mail_root:
+.. nmconfig:: database.mail_root
 
-database.mail_root
 The top-level directory where your mail currently exists and to
 where mail will be delivered in the future. Files should be
 individual email messages.
@@ -95,18 +97,18 @@ database.mail_root
 History: this configuration value was introduced in notmuch 0.32.
 
 Default: For compatibility with older configurations, the value of
-database.path is used if **database.mail\_root** is unset.
+database.path is used if :nmconfig:`database.mail_root` is unset.
+
+.. nmconfig:: database.path
 
-database.path
 Notmuch will store its database here, (in
-sub-directory named ``.notmuch`` if **database.mail\_root**
+sub-directory named ``.notmuch`` if :nmconfig:`database.mail_root`
 is unset).
 
 Default: see :ref:`database`
 
-.. _index.decrypt:
+.. nmconfig:: index.decrypt
 
-index.decrypt
 Policy for decrypting encrypted messages during indexing.  Must be
 one of: ``false``, ``auto``, ``nostash``, or ``true``.
 
@@ -161,7 +163,8 @@ index.decrypt
 
 .. _index.header:
 
-index.header.
+.. nmconfig:: index.header.
+
 Define the query prefix , based on a mail header. For
 example ``index.header.List=List-Id`` will add a probabilistic
 prefix ``List:`` that searches the ``List-Id`` field.  User
@@ -170,9 +173,8 @@ index.header.
 supported. See :any:`notmuch-search-terms(7)` for a list of existing
 prefixes, and an explanation of probabilistic prefixes.
 
-.. _maildir.synchronize_flags:
+.. nmconfig:: maildir.synchronize_flags
 
-maildir.synchronize\_flags
 If true, then the following maildir flags (in message filenames)
 will be synchronized with the corresponding notmuch tags:
 
@@ -205,9 +207,8 @@ maildir.synchronize\_flags
 
 Default: ``true``.
 
-.. _new.ignore:
+.. nmconfig:: new.ignore
 
-new.ignore
 A list to specify files and directories that will not be searched
 for messages by :any:`notmuch-new(1)`. Each entry in the list is either:
 
@@ -225,20 +226,21 @@ new.ignore
 
 Default: empty list.
 
-.. _new.tags:
+.. nmconfig:: new.tags
 
-new.tags
 A list of tags that will be added to all messages incorporated by
 **notmuch new**.
 
 Default: ``unread;inbox``.
 
-query.
+.. nmconfig:: query.
+
 Expansion for named query called . See
 :any:`notmuch-search-terms(7)` for more information about named
 queries.
 
-search.exclude\_tags
+.. nmconfig:: search.exclude_tags
+
 A list of tags that will be excluded from search results by
 default. Using 

Re: Help with sending mail from notmuch-emacs

2022-05-12 Thread David Bremner
hgv  writes:

> 1. Mail gets sent but doesn't save in the correct directory via FCC. I
> use msmtpq as my sending program. Upon sending, I receive this
> message: "message-send-mail-with-sendmail: Sending...failed to mail
> for [ -oi -f h...@fastmail.com -t ] : send was successful;" The mail is
> sent but is not saved in the fcc dir nor entered into the notmuch
> database. I have enabled emacs debugger (on entry with different
> functions related to sending) and stepped through the process but
> didn't see any pertinent information in the messages. My fcc line
> reads `(setq notmuch-fcc-dirs "fastmail/Sent +sent +inbox +sent-new")`
> (fuller notmuch-emacs config below). Any guidance on fixing or further
> troubleshooting (what should I look for on debugger) is appreciated.

Did you try setting a break on notmuch-maildir-fcc-with-notmuch-insert?
I'm guessing that it doesn't get called because the output of msmtpq is
misinterpreted as an error. I think this is a bug in msmtpq (or at least
a way in which it differs from a traditional sendmail). For me (using
nullmailer) "/usr/lib/sendmail -oi -f da...@tethera.net -t < foo.eml"
generates no output on success. If you can change / replace the msmtpq
script, you could try commenting out the log line [1] that is printing
"send was successful".

Another test (which would be worthwhile for your second issue as well),
is to test with M-x message-mail (however one calls a command in doom).
This will eliminate the possibility of some setting of notmuch confusing
things. Of course it still won't insert into the notmuch database, but
if it is working you should not see a message about "Sending...failed"

David

[1] Maybe line 503?

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] doc/emacs: document notmuch-show-toggle-toggle-elide-non-matching

2022-05-12 Thread David Bremner
Recently there was a question on the mailing list about the existence
of this function, so let us try to make it more discoverable.
---
 doc/notmuch-emacs.rst | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index 41f62390..67dbfc2b 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -221,6 +221,9 @@ pressing RET after positioning the cursor on a hidden part.
 ``+,-``
 Add or remove arbitrary tags from the current message.
 
+``!``
+|docstring::notmuch-show-toggle-elide-non-matching|
+
 ``?``
 Display full set of key bindings
 
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Change notmuch-show-elide-non-matching-messages on the fly

2022-05-12 Thread David Bremner
alan.schm...@polytechnique.org writes:

> Hello,
>
> I often do searches in long threads with
> notmuch-show-elide-non-matching-messages locally set to t. Sometimes I
> need to show the whole thread. Is there a function to do that, or
> should I write my own?
>

There is notmuch-show-toggle-elide-non-matching, bound to ! in
notmuch-show-mode.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Introducing mujmap, a notmuch/JMAP bridge

2022-05-10 Thread David Bremner
Eliza Velasquez  writes:

> Hello notmuch,
>
> I hope this is an appropriate place to announce this! Apologies if not.
>
> I have just released a tool to synchronize notmuch mail with a JMAP
> server, i.e. synchronizing tags with keywords and mailboxes. It's very
> similar to lieer, which is a tool that does essentially the same but for
> GMail.
>
> https://github.com/elizagamedev/mujmap/
> https://jmap.io/
>

Sounds cool. I added it to the wiki

   https://notmuchmail.org/howto/#index1h2

You can fine tune the wording if you like

   https://notmuchmail.org/wikiwriteaccess/
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Help with sending mail from notmuch-emacs

2022-05-09 Thread David Bremner
hgv  writes:


> 1. Mail gets sent but doesn't save in the correct directory via FCC. I
> use msmtpq as my sending program. Upon sending, I receive this
> message: "message-send-mail-with-sendmail: Sending...failed to mail
> for [ -oi -f h...@fastmail.com -t ] : send was successful;" The mail is
> sent but is not saved in the fcc dir nor entered into the notmuch
> database. I have enabled emacs debugger (on entry with different
> functions related to sending) and stepped through the process but
> didn't see any pertinent information in the messages. My fcc line
> reads `(setq notmuch-fcc-dirs "fastmail/Sent +sent +inbox +sent-new")`
> (fuller notmuch-emacs config below). Any guidance on fixing or further
> troubleshooting (what should I look for on debugger) is appreciated.

What is your setting for the variable

 notmuch-maildir-use-notmuch-insert

It should be t (the default) for that style of notmuch-fcc-dirs setting.

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: public-inbox, nmbug or Debbugs and notmuch sync?

2022-05-09 Thread David Bremner
zimoun  writes:

>
> Somehow, I would like to have a bridge: keep Emacs+Notmuch but index
> emails with another format (backend) than maildir as
> public-inbox-v2-format.  It would avoid data duplication and/or
> conversion.
>
> 1: 
>

I talked to the public-inbox author at some point about using notmuch,
but perl bindings were a hard requirement for that. I don't know of and
other interest in perl bindings, and especially no volunteers to
maintain them, so that idea kind of stalled.

>
>>>  3. Debian or GNU use instances of Debbugs.  Tools as ’mailscripts’ [5]
>>>  using ’bts’ CLI [6] are great.  Personally, I prefer the Emacs packages
>>>  ’piem’ [7] and ’debbugs’ [8].  Well, the point is to be able to inject
>>>  the bug report to my local inboxes – for reading offline or for
>>>  tagging.  How could I bridge the Debbugs meta info with a local ’nmbug’?
>>
>> You can use notmuch-slurp-debbug (from mailscripts) to get the mail
>> messages. I haven't thought about synchronizing metadata with
>> debbugs. In principle I guess it should be possible.

> Somehow, if ’nmbug’ would be able to extract the meta info from Debbugs,
> roughly speaking a map between Message-ID and status, and this map would
> be stored as Notmuch tags.  Then, using “notmuch git” would allow an
> easy way for synchronizing my local copy of the bugs mailing lists with
> the current state of Debbugs.  Bah, I do not know. ;-)

At the moment I am focussed on getting notmuch-git to perform a simpler
task well enough to ship as part of notmuch.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: public-inbox, nmbug or Debbugs and notmuch sync?

2022-05-09 Thread David Bremner
zimoun  writes:

>
> Currently, I synchronize my various Notmuch database using ’muchsync’
> [1] which works well enough for my needs.  Although I am not completely
> satisfied.
>
> Well, I have read this thread, starting at [2]:
>
> WIP: promote nmbug to user sync tool
> id:20220423133848.3852688-1-da...@tethera.net
>
> somehow promoting the subcommand “notmuch git”.
>
>
> All in all, my questions are:
>
>  1. Is it possible to index the Git repo as public-inbox?  From this
>  thread id:608c9185115fd_251d208e0@natae.notmuch [3], I guess no, right?
>
>  Is it planned?  Or any interest?

If I understand your question correctly, the answer is no, and at least
I have no plans for something like that. An nmbug / notmuch-git repo
only contains metadata, not the actual messages. 

>  2. Notmuch is strongly linked to the Maildir format as storage but Git
>  seems an efficient way to exchange and sync across several machines; for
>  instance using the tool grokmirror [4].  Does it make sense to have
>  another backend for Notmuch than Maildir?

I don't know grokmirror, it looks like a layer on top of git. You
can commit maildir to git if you like, notmuch doesn't really care.
As far as having a backend that is not just mail files on disk, that
seems unlikely, but I guess you never know.

>  3. Debian or GNU use instances of Debbugs.  Tools as ’mailscripts’ [5]
>  using ’bts’ CLI [6] are great.  Personally, I prefer the Emacs packages
>  ’piem’ [7] and ’debbugs’ [8].  Well, the point is to be able to inject
>  the bug report to my local inboxes – for reading offline or for
>  tagging.  How could I bridge the Debbugs meta info with a local ’nmbug’?

You can use notmuch-slurp-debbug (from mailscripts) to get the mail
messages. I haven't thought about synchronizing metadata with
debbugs. In principle I guess it should be possible.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: WIP: promote nmbug to user sync tool

2022-05-07 Thread David Bremner
Sean Whitton  writes:

> [please keep me CCed]
>
> Hello David,
>
> On Sat 23 Apr 2022 at 10:38am -03, David Bremner wrote:
>

> Just looking at my current usage, there are two cases where I've wrapped
> nmbug in some additional myrepos scripting.  The first is a status
> command:
>
> status =
> nmbug-spw status | grep -v "^U\s" || true
> # `nmbug status` does not catch committed but unpushed changes
> git --no-pager log --branches \
> --not --remotes \
> --simplify-by-decoration --decorate --oneline
>
> Possibly notmuch-git could learn how to do this?

Perhaps. I think I would prefer something a bit more concise like a
count of unpushed commits. Do you tend to actually have meaningful
commit messages?

> Secondly, I've got this auto-commit command:
>
> autoci =
> nmbug-spw status | perl -ne'/^[AD][ad]?\s/ and $i++ > 500 and exit 1' 
> \
>   && nmbug-spw commit
>

[snip]

> As for the former thing, I wonder if instead there could be some
> mechanism, connected with the new caching, to associate nmbug repos with
> the notmuch database, and refuse to operate unless that association
> already exists?  So, 'nmbug checkout' would mark it as safe to sync back
> and forth between the database and that repo no matter the number of
> changes.

Personally I would be more worried about checkout (e.g. after init)
wiping out my notmuch database, since an errant commit can always be
reverted. Both cases seem to be covered by your heuristic. Perhaps we
could just count the size of the update, and insist on a --force option
if it is too large.

>
> On Sat 23 Apr 2022 at 03:49pm -03, David Bremner wrote:
>
>> Related, the current script does not understand NOTMUCH_PROFILE. That
>> would be a natural way to locate the default git repo.
>
> It would, but it wouldn't help with configuring a default prefix.
> Perhaps an entry in .notmuch-config for that?  Currently I use a tiny
> wrapper script:
>
> #!/bin/sh
>
> NMBGIT="$HOME/lib/nmbug-spw" NMBPREFIX="spw::" nmbug "$@"
>
> but it would be great to just be able to type 'notmuch git ...'.
>

For what it's worth, you can already call

notmuch git -C $HOME/lib/nmbug-spw -p spw:: ...

if that is more convenient.

The defaults have already changed in my latest working branch so the
default repo is under $XDG_DATA_HOME/notmuch/$NOTMUCH_PROFILE/git, and
the default prefix is ''.  But re-reading this, I see see we polled two
people and got two answers for what a reasonable default prefix is, so a
configuration item is definitely needed for prefix. Probably it is also
reasonable to have one for repo location.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: BUG: ruby segfault in notmuch_rb_tags_each

2022-05-05 Thread David Bremner
arcnmx  writes:

> Hi, I've been running into rare crashes via notmuch-vim for a while now and
> finally got around to tracking down the problem. I apologize for the size
> of the reproduction case, but at least it can be reproduced!

Pinging Felipe to see if he has any ideas.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH 1/2] doc/sexp-queries: fix example

2022-05-05 Thread David Bremner
David Bremner  writes:

> Update the explanatory text to match the query.

series applied to master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Forcing a sync of maildir flags?

2022-05-03 Thread David Bremner
Sean Whitton  writes:

> Hello David,
>> # next line is a no-op, because it already doesn't have the unread tag
>> notmuch tag -unread folder:sent
>
> Seems also worth noting that to my mind it ultimately shouldn't be
> necessary to run that command -- notmuch should notice that one copy of
> the message has different maildir flags to the other in a way that's out
> of sync with the notmuch tags it has.
>

Right, that's probably a useful way to think about the problem.

>> The key point is that from notmuch's point of view the message never has
>> the unread tag, so there is no change for "notmuch tag" to sync with
>> maildir flags.
>>
>> It doesn't seem to make a difference if I put the copy in inbox/new or
>> inbox/cur, so I don't think it is related to the previous efforts not to
>> prematurely move files out of new/.
>>
>> As far as I can tell, notmuch-new (unlike notmuch-insert) does not call
>> notmuch_message_tags_to_maildir_flags, so the maildir flags on the newly
>> discovered copy are not updated. Perhaps it should, but that seems like
>> a pretty big change, so I want to proceed with caution.
>
> It makes sense to me for notmuch-new to call that function too, fwiw.

I have the same intuition. Unfortunately it is not as simple as adding
in a couple calls to this function. The complications I am aware of so
far are

1) We need to distinguish between when a newly discovered file should
update tags, and when the newly discovered file should have it's flags
updated.

2) Renaming files in the middle of notmuch new needs to be done
carefully in order that notmuch doesn't lose track of the files. They
are only lost until the next run of notmuch-new, but it's not ideal.

d


___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Forcing a sync of maildir flags?

2022-05-01 Thread David Bremner
Sean Whitton  writes:
>
> Thanks.  Let me record in this thread what I will believe it will take a
> reproduce this in a test:
>
> 1) inbox and sent are Maildirs
>
> 1) Compose mail to a mailing list which will return copies of
>submissions, with `Fcc: sent -unread`

Since you mention Fcc, are you using notmuch insert?

>
> 2) notmuch new, and in post-new hook, notmuch tag -unread -- folder:sent
>
> Expected result: copy of message in both inbox and sent has Seen flag
>
> Actual result: only copy of message in sent has Seen flag

I tried to simulate this in the form of a test. I'm not using notmuch
insert here, but I manage to trigger similar looking behaviour:

test_begin_subtest "duplicate file with seen flag"
test_subtest_known_broken
mkdir -p "${MAIL_DIR}"/sent/{cur,new,tmp}
notmuch config set maildir.synchronize_flags true
add_message '[subject]="seen message"'  '[filename]=fcc-file:2,S' 
'[dir]=sent/cur'
notmuch new
# for debugging
notmuch search subject:"seen message"
mkdir -p "${MAIL_DIR}"/inbox/{cur,new,tmp}
cp "${gen_msg_filename}" "${MAIL_DIR}"/inbox/new/copied-file:2,
notmuch new
# for debugging
notmuch search subject:"seen message"
# next line is a no-op, because it already doesn't have the unread tag
notmuch tag -unread folder:sent
count=$(notmuch search --output=files subject:"seen message" | grep -c ',S$')
# for debugging:
notmuch search --output=files subject:"seen message"
test_expect_equal "${count}" "2"

This has the output

BROKEN duplicate file with seen flag
--- T050-new.41.expected2022-05-01 22:51:45.644747316 +
+++ T050-new.41.output  2022-05-01 22:51:45.644747316 +
@@ -1 +1 @@
-2
+1
No new mail.
thread:0011   2001-01-05 [1/1] Notmuch Test Suite; seen message 
(inbox)
Processed 1 file in almost no time.
No new mail.
thread:0011   2001-01-05 [1/1(2)] Notmuch Test Suite; seen 
message (inbox)

/home/bremner/software/upstream/notmuch/test/tmp.T050-new/mail/sent/cur/fcc-file:2,S

/home/bremner/software/upstream/notmuch/test/tmp.T050-new/mail/inbox/new/copied-file:2,

The key point is that from notmuch's point of view the message never has
the unread tag, so there is no change for "notmuch tag" to sync with
maildir flags.

It doesn't seem to make a difference if I put the copy in inbox/new or
inbox/cur, so I don't think it is related to the previous efforts not to
prematurely move files out of new/.

As far as I can tell, notmuch-new (unlike notmuch-insert) does not call
notmuch_message_tags_to_maildir_flags, so the maildir flags on the newly
discovered copy are not updated. Perhaps it should, but that seems like
a pretty big change, so I want to proceed with caution.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 2/2] doc/sexp-queries: be more consistent about ending example text

2022-04-30 Thread David Bremner
Most, but not all of the the explanatory texts already end in
'.'. This makes the remainder match.
---
 doc/man7/notmuch-sexp-queries.rst | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/doc/man7/notmuch-sexp-queries.rst 
b/doc/man7/notmuch-sexp-queries.rst
index a20943c6..1d7e0ae9 100644
--- a/doc/man7/notmuch-sexp-queries.rst
+++ b/doc/man7/notmuch-sexp-queries.rst
@@ -221,7 +221,7 @@ EXAMPLES
 
 ``(not Bob Marley)``
 
-Match messages containing neither "Bob" nor "Marley", nor their stems,
+Match messages containing neither "Bob" nor "Marley", nor their stems.
 
 ``"quick fox"`` ``quick-fox`` ``quick@fox``
 
@@ -230,11 +230,11 @@ EXAMPLES
 
 ``(folder (of (id 1234@invalid)))``
 
-Match any message in the same folder as the one with Message-Id 
"1234\@invalid"
+Match any message in the same folder as the one with Message-Id 
"1234\@invalid".
 
 ``(id 1234@invalid blah@test)``
 
-Matches Message-Id "1234\@invalid" *or* Message-Id "blah\@test"
+Matches Message-Id "1234\@invalid" *or* Message-Id "blah\@test".
 
 ``(and (infix "date:2009-11-18..2009-11-18") (tag unread))``
 
@@ -260,13 +260,13 @@ EXAMPLES
 
 ``(thread (of (id 1234@invalid)))``
 
-Match any message in the same thread as the one with Message-Id 
"1234\@invalid"
+Match any message in the same thread as the one with Message-Id 
"1234\@invalid".
 
 ``(thread (matching (from b...@example.com) (to b...@example.com)))``
 
 Match any (messages in) a thread containing a message from
 "bob\@example.com" and a (possibly distinct) message to
-"bob\@example.com"
+"bob\@example.com".
 
 ``(to (or b...@example.com mall...@example.org))`` ``(or (to b...@example.com) 
(to mall...@example.org))``
 
@@ -281,7 +281,7 @@ EXAMPLES
 ``(List *)``
 
 Match messages with a non-empty List-Id header, assuming
-configuration ``index.header.List=List-Id``
+configuration ``index.header.List=List-Id``.
 
 .. _macro_examples:
 
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 1/2] doc/sexp-queries: fix example

2022-04-30 Thread David Bremner
Update the explanatory text to match the query.
---
 doc/man7/notmuch-sexp-queries.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/doc/man7/notmuch-sexp-queries.rst 
b/doc/man7/notmuch-sexp-queries.rst
index 0eb86734..a20943c6 100644
--- a/doc/man7/notmuch-sexp-queries.rst
+++ b/doc/man7/notmuch-sexp-queries.rst
@@ -265,8 +265,8 @@ EXAMPLES
 ``(thread (matching (from b...@example.com) (to b...@example.com)))``
 
 Match any (messages in) a thread containing a message from
-"bob\@example.com" and a (possibly distinct) message to "bob at
-example.com")
+"bob\@example.com" and a (possibly distinct) message to
+"bob\@example.com"
 
 ``(to (or b...@example.com mall...@example.org))`` ``(or (to b...@example.com) 
(to mall...@example.org))``
 
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] doc/sexp-queries: escape @ in non-verbatim text

2022-04-30 Thread David Bremner
David Bremner  writes:

> This prevents sphinx-doc from creating spurious mailto: links. Thanks
> to Jakub Wilk for telling me about the fix.

applied to master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH 1/1] emacs: Make notmuch-show-next-thread return nil on failure

2022-04-30 Thread David Bremner
Leo  writes:

> From: Leo Okawa Ericson 
>
> Having notmuch-show-next-thread return non-nil on success and nil on
> failure makes it easier for users to interact with notmuch via elisp.

You talk about notmuch-show-next-thread, but you seem to be modifying
notmuch-search-show-thread. That actually seems like a bit of a strange
function to invoke programatically, since it deduces the argument from
the current buffer.  It might be better for your code to call
notmuch-show directly, since notmuch-search-show-thread is pretty tied
to the internals of notmuch-search-mode.

In any case:

If you change the API to have a meaningful return value, please update
the docstring for the function

We generally need a test for any new feature, and this is arguably a new
feature. If you need help constructing a test, feel free to ask. There
are several examples in  test/T*emacs*.sh.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH 1/1] emacs: Make notmuch-show-next-thread return nil on failure

2022-04-30 Thread David Bremner
Leo Okawa Ericson  writes:

> Tomi Ollila  writes:
>
>>
>> I don't see why this indentation change is happening...
>>
>
> I thought the indentation was inconsistent with the rest of the file.  I
> tried running the auto-indenter on a other more complex functions which
> changed nothing, but it did change it on this function so I assumed it
> was good to include it.  If it's bad etiquette, wrong or unnecessary I'm happy
> to amend the patch.

In general indentation (and whitespace) cleanup is fine, but should be
seperate commits labelled as such.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] doc/sexp-queries: escape @ in non-verbatim text

2022-04-28 Thread David Bremner
This prevents sphinx-doc from creating spurious mailto: links. Thanks
to Jakub Wilk for telling me about the fix.
---
 doc/man7/notmuch-sexp-queries.rst | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/doc/man7/notmuch-sexp-queries.rst 
b/doc/man7/notmuch-sexp-queries.rst
index bc8e5086..0eb86734 100644
--- a/doc/man7/notmuch-sexp-queries.rst
+++ b/doc/man7/notmuch-sexp-queries.rst
@@ -230,11 +230,11 @@ EXAMPLES
 
 ``(folder (of (id 1234@invalid)))``
 
-Match any message in the same folder as the one with Message-Id 
"1234@invalid"
+Match any message in the same folder as the one with Message-Id 
"1234\@invalid"
 
 ``(id 1234@invalid blah@test)``
 
-Matches Message-Id "1234@invalid" *or* Message-Id "blah@test"
+Matches Message-Id "1234\@invalid" *or* Message-Id "blah\@test"
 
 ``(and (infix "date:2009-11-18..2009-11-18") (tag unread))``
 
@@ -260,18 +260,18 @@ EXAMPLES
 
 ``(thread (of (id 1234@invalid)))``
 
-Match any message in the same thread as the one with Message-Id 
"1234@invalid"
+Match any message in the same thread as the one with Message-Id 
"1234\@invalid"
 
 ``(thread (matching (from b...@example.com) (to b...@example.com)))``
 
 Match any (messages in) a thread containing a message from
-"b...@example.com" and a (possibly distinct) message to "bob at
+"bob\@example.com" and a (possibly distinct) message to "bob at
 example.com")
 
 ``(to (or b...@example.com mall...@example.org))`` ``(or (to b...@example.com) 
(to mall...@example.org))``
 
-Match in the "To" or "Cc" headers, "b...@example.com",
-"mall...@example.org", and also "b...@example.com.au" since it
+Match in the "To" or "Cc" headers, "bob\@example.com",
+"mallory\@example.org", and also "bob\@example.com.au" since it
 contains the adjacent triple "bob", "example", "com".
 
 ``(not (to *))``
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


notmuch release 0.36 now available

2022-04-25 Thread David Bremner

Where to obtain notmuch 0.36
===
  https://notmuchmail.org/releases/notmuch-0.36.tar.xz

Which can be verified with:

  https://notmuchmail.org/releases/notmuch-0.36.tar.xz.sha256.asc
  -BEGIN PGP SIGNED MESSAGE-
  Hash: SHA256
  
  130231b830fd980efbd2aab12214392b8841f5d2a5a361aa8c79a79a6035ce40  
notmuch-0.36.tar.xz
  -BEGIN PGP SIGNATURE-
  
  iQIzBAEBCAAdFiEEkiyHYXwaY0SiY6fqA0U5G1WqFSEFAmJmi9kACgkQA0U5G1Wq
  FSFtrA//fQH5HEHv0fIGbSF3RY3n3elDKoNAkUbiE3aCEvONmChueLqC2boptN8M
  +xnIeI2tXtCeWJ0jkdwPeNbg87kGCMO9u0cXY37D6zV0CTUfB1UR5Fl5ihIR525e
  fcFGCO9APf9jp9AX7tUwYrsVK4w+Ho+RKxJOJvOnixu/G8DbAfxqrAtNuER5BhrO
  1pIIy3eI9PtEEO162mTfUdXTkTAgJw0F045GdzBsoswG3c2tXNxgbdau6l/sgA/f
  zXw0wyP6eU5B89Ocn7Ko1PyNbZJSFzmj9IjOmjT0NES6ZKd10g5qi0cY+KJ1lg5G
  /D7bwIYvcpxeLUSRvDx0Xf1Xd2KTdxIn6tCbF6SLG1JkYNr2AGYYc4I0sRVyBLpP
  hnh+l62WxHu9mLJMwJaebtcX0jPetm8OHwzQzPbz3AFoxKmtXke5ok79O+QvYPoX
  blQYppAcivWs8L5i3LD1jQO23LiDfZLQBGvSsN5nTlUMc+K1TwH4K9+Axa1w51ln
  frU0GEtEGsQdHXGhOdKdQ5fQ9shycqwaSD0fkr6d35DSPDU7RAuTq3FcrJLxxzEJ
  OopWrsIi9TDdFOD18GGvrRwfXGp1bhWBFfcgzu3RIfp8JXajEFOj1uwdUDkhiOPH
  Q9EQKyU0djDyKP4uBSUGaj2HUjn1UEwoViV7oPdP6ZNyltOf8Io=
  =ZNnd
  -END PGP SIGNATURE-

  https://notmuchmail.org/releases/notmuch-0.36.tar.xz.asc
  (signed by David Bremner)

What's new in notmuch 0.36
=

Library
---

Add the `sexp` prefix to the infix (traditional) query parser. This
allows specific subqueries to be parsed by the sexp parser (with
appropropriate quoting). See `notmuch-search-terms(7)` for details.

Add another heuristic to regexp fields to prevent phrase parsing of
bracketed sub-expressions.

Command Line Interface
--

Envelope from ("From ") headers are now escaped as X-Envelope-From: in
input to `notmuch-insert`. This prevents creating mbox files when
calling `notmuch-insert` from e.g. `postfix`.

Python (CFFI) Bindings
--

Use the `config_pairs` API in ConfigIterator. This returns all
matching key-value pairs, not just those that happen to be stored in
the database.

Documentation
-

Reorganize documention for `notmuch-config`. Add a few links from
other man pages.

Emacs
-

Bind the usual undo key sequences to new command
"notmuch-tag-undo". This allows transparent undo of tagging
operations.

Tests
-

Fix smime.4 with newer gmime. Unset `XDG_DATA_HOME` and `MAILDIR` for tests.

New add-on tool: notmuch-web
-

The new devel/ tool `notmuch-web` is a very thin web client.  It
supports a full search interface for one user: there is no facility
for multiple users provided today.  See the notmuch-web README file
for more information.

Be careful about running it on a network-connected system: it will
expose a web interface that requires no authentication but exposes
your mail store.

What is notmuch
===
Notmuch is a system for indexing, searching, reading, and tagging
large collections of email messages in maildir or mh format. It uses
the Xapian library to provide fast, full-text search with a convenient
search syntax.

For more about notmuch, see https://notmuchmail.org



signature.asc
Description: PGP signature
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: WIP: promote nmbug to user sync tool

2022-04-23 Thread David Bremner
David Bremner  writes:

> - the (new) man page is inadequate
> - notmuch git doesn't understand the common arguments to notmuch (--config 
> and friends)
> - the defaults for prefix and git repo are wrong for most users. I was 
> thinking about either a
>   --nmbug argument to set the old defaults, or having the script recognize if 
> it is invoked as nmbug.

Related, the current script does not understand NOTMUCH_PROFILE. That
would be a natural way to locate the default git repo.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 02/16] notmuch-git: add --git-dir argument

2022-04-23 Thread David Bremner
The intent is to change the names of the environment variables before
releasing this code, so avoid relying on environment variables in the
test framework.
---
 notmuch-git.in | 18 +-
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index dd30b5be..62889303 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -57,11 +57,7 @@ _LOG = _logging.getLogger('nmbug')
 _LOG.setLevel(_logging.WARNING)
 _LOG.addHandler(_logging.StreamHandler())
 
-NMBGIT = _os.path.expanduser(
-_os.getenv('NMBGIT', _os.path.join('~', '.nmbug')))
-_NMBGIT = _os.path.join(NMBGIT, '.git')
-if _os.path.isdir(_NMBGIT):
-NMBGIT = _NMBGIT
+NMBGIT = None
 
 TAG_PREFIX = _os.getenv('NMBPREFIX', 'notmuch::')
 _HEX_ESCAPE_REGEX = _re.compile('%[0-9A-F]{2}')
@@ -718,6 +714,9 @@ if __name__ == '__main__':
 parser = argparse.ArgumentParser(
 description=__doc__.strip(),
 formatter_class=argparse.RawDescriptionHelpFormatter)
+parser.add_argument(
+'-C', '--git-dir', metavar='REPO',
+help='Git repository to operate on.')
 parser.add_argument(
 '-v', '--version', action='version',
 version='%(prog)s {}'.format(__version__))
@@ -830,6 +829,15 @@ if __name__ == '__main__':
 
 args = parser.parse_args()
 
+if args.git_dir:
+NMBGIT = args.git_dir
+else:
+NMBGIT = _os.path.expanduser(
+_os.getenv('NMBGIT', _os.path.join('~', '.nmbug')))
+_NMBGIT = _os.path.join(NMBGIT, '.git')
+if _os.path.isdir(_NMBGIT):
+NMBGIT = _NMBGIT
+
 if args.log_level:
 level = getattr(_logging, args.log_level.upper())
 _LOG.setLevel(level)
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 05/16] nmbug: Add an 'init' command

2022-04-23 Thread David Bremner
From: "W. Trevor King" 

For folks that want to start versioning a new tag-space, instead of
cloning one that someone else has already started.

The empty-blob hash-object call avoids errors like:

  $ nmbug commit
  error: invalid object 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 for
'tags/...'
  fatal: git-write-tree: error building trees
  'git HASH(0x9ef3eb8) write-tree' exited with nonzero value

David Bremner suggested [1]:

  $ git hash-object -w /dev/null

instead of my Python version of:

  $ git hash-object -w --stdin <&-

but I expect that closing stdin is more portable than the /dev/null
path (which doesn't exist on Windows, for example).

The --bare init and use of NMBGIT as the work tree (what could go
wrong with an empty commit?) are suggestions from Michal Sojka [2].

[1]: id:87y4vu6uvf@maritornes.cs.unb.ca
 http://thread.gmane.org/gmane.mail.notmuch.general/18626/focus=18720
[2]: id:87a93a5or2@resox.2x.cz
 http://thread.gmane.org/gmane.mail.notmuch.general/19495/focus=19767
---
 notmuch-git.in | 20 
 1 file changed, 20 insertions(+)

diff --git a/notmuch-git.in b/notmuch-git.in
index 81d604bb..d574f19e 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -377,6 +377,25 @@ def fetch(remote=None):
 _git(args=args, wait=True)
 
 
+def init(remote=None):
+"""
+Create an empty nmbug repository.
+
+This wraps 'git init' with a few extra steps to support subsequent
+status and commit commands.
+"""
+_spawn(args=['git', '--git-dir', NMBGIT, 'init', '--bare'], wait=True)
+_git(args=['config', 'core.logallrefupdates', 'true'], wait=True)
+# create an empty blob (e69de29bb2d1d6434b8b29ae775ad8c2e48c5391)
+_git(args=['hash-object', '-w', '--stdin'], input='', wait=True)
+_git(
+args=[
+'commit', '--allow-empty', '-m', 'Start a new nmbug repository'
+],
+additional_env={'GIT_WORK_TREE': NMBGIT},
+wait=True)
+
+
 def checkout():
 """
 Update the notmuch database from Git.
@@ -740,6 +759,7 @@ if __name__ == '__main__':
 'commit',
 'fetch',
 'help',
+'init',
 'log',
 'merge',
 'pull',
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 09/16] test/git: add known broken test for tag with quotes.

2022-04-23 Thread David Bremner
There is current insufficient sanitization and/or escaping of tag names
internally in notmuch-git.
---
 test/T850-git.sh | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/test/T850-git.sh b/test/T850-git.sh
index 417692d4..408a6337 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -19,6 +19,16 @@ git -C tags.git ls-tree -r --name-only HEAD | xargs dirname 
| sort -u | sed s,ta
 notmuch search --output=messages '*' | sort > EXPECTED
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
+test_begin_subtest "commit, with quoted tag"
+test_subtest_known_broken
+notmuch git -C clone2.git -p '' clone tags.git
+git -C clone2.git ls-tree -r --name-only HEAD | grep /inbox > BEFORE
+notmuch tag '+"quoted tag"' '*'
+notmuch git -C clone2.git -p '' commit
+notmuch tag '-"quoted tag"' '*'
+git -C clone2.git ls-tree -r --name-only HEAD | grep /inbox > AFTER
+test_expect_equal_file_nonempty BEFORE AFTER
+
 test_begin_subtest "checkout"
 notmuch dump > BEFORE
 notmuch tag -inbox '*'
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 14/16] CLI/git: create CachedIndex class

2022-04-23 Thread David Bremner
The "git-read-tree HEAD" is a bottleneck, but unfortunately sometimes
is needed. Cache the index checksum and hash to reduce the number of
times the operation is run. The overall design is a simplified version
of the PrivateIndex class, which is partially refactored to support
the new class.
---
 notmuch-git.in | 136 +++--
 1 file changed, 97 insertions(+), 39 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index b3f71699..261b3f85 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -342,41 +342,98 @@ def _is_committed(status):
 return len(status['added']) + len(status['deleted']) == 0
 
 
+class CachedIndex:
+def __init__(self, repo, treeish):
+self.cache_path = _os.path.join(repo, 'notmuch', 'index_cache.json')
+self.index_path = _os.path.join(repo, 'index')
+self.current_treeish = treeish
+# cached values
+self.treeish = None
+self.hash = None
+self.index_checksum = None
+
+self._load_cache_file()
+
+def _load_cache_file(self):
+try:
+with open(self.cache_path) as f:
+data = _json.load(f)
+self.treeish = data['treeish']
+self.hash = data['hash']
+self.index_checksum = data['index_checksum']
+except FileNotFoundError:
+pass
+except _json.JSONDecodeError:
+_LOG.error("Error decoding cache")
+_sys.exit(1)
+
+def __enter__(self):
+self.read_tree()
+return self
+
+def __exit__(self, type, value, traceback):
+checksum = _read_index_checksum(self.index_path)
+(_, hash, _) = _git(
+args=['rev-parse', self.current_treeish],
+stdout=_subprocess.PIPE,
+wait=True)
+
+with open(self.cache_path, "w") as f:
+_json.dump({'treeish': self.current_treeish,
+'hash': hash.rstrip(),  'index_checksum': checksum }, 
f)
+
+@timed
+def read_tree(self):
+current_checksum = _read_index_checksum(self.index_path)
+(_, hash, _) = _git(
+args=['rev-parse', self.current_treeish],
+stdout=_subprocess.PIPE,
+wait=True)
+current_hash = hash.rstrip()
+
+if self.current_treeish == self.treeish and \
+   self.index_checksum and self.index_checksum == current_checksum and 
\
+   self.hash and self.hash == current_hash:
+return
+
+_git(args=['read-tree', self.current_treeish], wait=True)
+
+
 def commit(treeish='HEAD', message=None):
 """
 Commit prefix-matching tags from the notmuch database to Git.
 """
+
 status = get_status()
 
 if _is_committed(status=status):
 _LOG.warning('Nothing to commit')
 return
 
-_git(args=['read-tree', '--empty'], wait=True)
-_git(args=['read-tree', treeish], wait=True)
-try:
-_update_index(status=status)
-(_, tree, _) = _git(
-args=['write-tree'],
-stdout=_subprocess.PIPE,
-wait=True)
-(_, parent, _) = _git(
-args=['rev-parse', treeish],
-stdout=_subprocess.PIPE,
-wait=True)
-(_, commit, _) = _git(
-args=['commit-tree', tree.strip(), '-p', parent.strip()],
-input=message,
-stdout=_subprocess.PIPE,
-wait=True)
-_git(
-args=['update-ref', treeish, commit.strip()],
-stdout=_subprocess.PIPE,
-wait=True)
-except Exception as e:
-_git(args=['read-tree', '--empty'], wait=True)
-_git(args=['read-tree', treeish], wait=True)
-raise
+with CachedIndex(NMBGIT, treeish) as index:
+try:
+_update_index(status=status)
+(_, tree, _) = _git(
+args=['write-tree'],
+stdout=_subprocess.PIPE,
+wait=True)
+(_, parent, _) = _git(
+args=['rev-parse', treeish],
+stdout=_subprocess.PIPE,
+wait=True)
+(_, commit, _) = _git(
+args=['commit-tree', tree.strip(), '-p', parent.strip()],
+input=message,
+stdout=_subprocess.PIPE,
+wait=True)
+_git(
+args=['update-ref', treeish, commit.strip()],
+stdout=_subprocess.PIPE,
+wait=True)
+except Exception as e:
+_git(args=['read-tree', '--empty'], wait=True)
+_git(args=['read-tree', treeish], wait=True)
+raise
 
 @timed
 def _update_index(status):
@@ -664,7 +721,7 @@ class PrivateIndex:
 return self
 
 def __exit__(self, type, value, traceback):
-checksum = self._read_index_checksum()
+checksum = _read_index_checksum(self.index_path)
 (count, uuid, lastmod) = _read_database_lastmod()
 

[PATCH 01/16] nmbug: promote to user tool "notmuch-git"

2022-04-23 Thread David Bremner
So far this is just a rename. Documentation and tests to follow.
---
 Makefile.local  | 7 +--
 devel/nmbug/nmbug => notmuch-git.in | 2 +-
 2 files changed, 6 insertions(+), 3 deletions(-)
 rename devel/nmbug/nmbug => notmuch-git.in (99%)

diff --git a/Makefile.local b/Makefile.local
index d8bbf3e1..ca2310f4 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -1,7 +1,7 @@
 # -*- makefile-gmake -*-
 
 .PHONY: all
-all: notmuch notmuch-shared build-man build-info ruby-bindings 
python-cffi-bindings
+all: notmuch notmuch-shared build-man build-info ruby-bindings 
python-cffi-bindings notmuch-git
 ifeq ($(MAKECMDGOALS),)
 ifeq ($(shell cat .first-build-message 2>/dev/null),)
@NOTMUCH_FIRST_BUILD=1 $(MAKE) --no-print-directory all
@@ -294,7 +294,7 @@ endif
 SRCS  := $(SRCS) $(notmuch_client_srcs)
 CLEAN := $(CLEAN) notmuch notmuch-shared $(notmuch_client_modules)
 CLEAN := $(CLEAN) version.stamp notmuch-*.tar.gz.tmp
-CLEAN := $(CLEAN) .deps
+CLEAN := $(CLEAN) .deps notmuch-git
 
 DISTCLEAN := $(DISTCLEAN) .first-build-message Makefile.config sh.config 
sphinx.config
 
@@ -307,6 +307,9 @@ cppcheck:
@echo "No cppcheck found during configure; skipping static checking"
 endif
 
+notmuch-git: notmuch-git.in
+   sed s/@NOTMUCH_VERSION@/${VERSION}/ < notmuch-git.in > notmuch-git
+   chmod ugo+rx notmuch-git
 
 DEPS := $(SRCS:%.c=.deps/%.d)
 DEPS := $(DEPS:%.cc=.deps/%.d)
diff --git a/devel/nmbug/nmbug b/notmuch-git.in
similarity index 99%
rename from devel/nmbug/nmbug
rename to notmuch-git.in
index 043c1863..dd30b5be 100755
--- a/devel/nmbug/nmbug
+++ b/notmuch-git.in
@@ -51,7 +51,7 @@ except ImportError:  # Python 2
 from urllib import unquote as _unquote
 
 
-__version__ = '0.3'
+__version__ = '@NOTMUCH_VERSION@'
 
 _LOG = _logging.getLogger('nmbug')
 _LOG.setLevel(_logging.WARNING)
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 03/16] notmuch-git: add --tag-prefix argument

2022-04-23 Thread David Bremner
The test suite will require setting the tag prefix (at least
initially), and this commit will enable doing that without relying on
environment variables (whose names are planned to change).
---
 notmuch-git.in | 13 -
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 62889303..81d604bb 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -58,8 +58,8 @@ _LOG.setLevel(_logging.WARNING)
 _LOG.addHandler(_logging.StreamHandler())
 
 NMBGIT = None
+TAG_PREFIX = None
 
-TAG_PREFIX = _os.getenv('NMBPREFIX', 'notmuch::')
 _HEX_ESCAPE_REGEX = _re.compile('%[0-9A-F]{2}')
 _TAG_DIRECTORY = 'tags/'
 _TAG_FILE_REGEX = _re.compile(_TAG_DIRECTORY + '(?P[^/]*)/(?P[^/]*)')
@@ -105,10 +105,6 @@ def _hex_quote(string, safe='+@=:,'):
 lambda match: match.group(0).lower(),
 uppercase_escapes)
 
-
-_ENCODED_TAG_PREFIX = _hex_quote(TAG_PREFIX, safe='+@=,')  # quote ':'
-
-
 def _xapian_quote(string):
 """
 Quote a string for Xapian's QueryParser.
@@ -717,6 +713,10 @@ if __name__ == '__main__':
 parser.add_argument(
 '-C', '--git-dir', metavar='REPO',
 help='Git repository to operate on.')
+parser.add_argument(
+'-p', '--tag-prefix', metavar='PREFIX',
+default = _os.getenv('NMBPREFIX', 'notmuch::'),
+help='Prefix of tags to operate on.')
 parser.add_argument(
 '-v', '--version', action='version',
 version='%(prog)s {}'.format(__version__))
@@ -838,6 +838,9 @@ if __name__ == '__main__':
 if _os.path.isdir(_NMBGIT):
 NMBGIT = _NMBGIT
 
+TAG_PREFIX = args.tag_prefix
+_ENCODED_TAG_PREFIX = _hex_quote(TAG_PREFIX, safe='+@=,')  # quote ':'
+
 if args.log_level:
 level = getattr(_logging, args.log_level.upper())
 _LOG.setLevel(level)
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 13/16] CLI/git: create PrivateIndex class

2022-04-23 Thread David Bremner
If the index file matches a previously known revision of the database,
we can update the index incrementally using the recorded lastmod
counter. This is typically much faster than a full update, although it
could be slower in the case of large changes to the database.
---
 notmuch-git.in   | 220 ---
 test/T850-git.sh |  41 +
 2 files changed, 194 insertions(+), 67 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index b69d57e7..b3f71699 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -50,6 +50,10 @@ except ImportError:  # Python 2
 from urllib import quote as _quote
 from urllib import unquote as _unquote
 
+import json as _json
+
+# hopefully big enough, handle 32 bit hosts
+MAX_LASTMOD=2**32
 
 __version__ = '@NOTMUCH_VERSION@'
 
@@ -621,51 +625,159 @@ def get_status():
 'deleted': {},
 'missing': {},
 }
-index = _index_tags()
-maybe_deleted = _diff_index(index=index, filter='D')
-for id, tags in maybe_deleted.items():
-(_, stdout, stderr) = _spawn(
-args=['notmuch', 'search', '--output=files', 'id:{0}'.format(id)],
-stdout=_subprocess.PIPE,
-wait=True)
-if stdout:
-status['deleted'][id] = tags
-else:
-status['missing'][id] = tags
-status['added'] = _diff_index(index=index, filter='A')
-_os.remove(index)
+with PrivateIndex(repo=NMBGIT, prefix=TAG_PREFIX) as index:
+maybe_deleted = index.diff(filter='D')
+for id, tags in maybe_deleted.items():
+(_, stdout, stderr) = _spawn(
+args=['notmuch', 'search', '--output=files', 
'id:{0}'.format(id)],
+stdout=_subprocess.PIPE,
+wait=True)
+if stdout:
+status['deleted'][id] = tags
+else:
+status['missing'][id] = tags
+status['added'] = index.diff(filter='A')
+
 return status
 
-@timed
-def _index_tags():
-"Write notmuch tags to private git index."
-ensure_private_directory(NMBGIT)
-path = _os.path.join(NMBGIT, 'notmuch','index')
-prefix = '+{0}'.format(_ENCODED_TAG_PREFIX)
-_git(
-args=['read-tree', '--empty'],
-additional_env={'GIT_INDEX_FILE': path}, wait=True)
-with _spawn(
-args=['notmuch', 'dump', '--format=batch-tag', '--query=sexp', 
'--', _tag_query()],
-stdout=_subprocess.PIPE) as notmuch:
+class PrivateIndex:
+def __init__(self, repo, prefix):
+try:
+_os.makedirs(_os.path.join(repo, 'notmuch'))
+except FileExistsError:
+pass
+
+file_name = 'notmuch/index'
+self.index_path = _os.path.join(repo, file_name)
+self.cache_path = _os.path.join(repo, 'notmuch', 
'{:s}.json'.format(_hex_quote(file_name)))
+
+self.current_prefix = prefix
+
+self.prefix = None
+self.uuid = None
+self.lastmod = None
+self.checksum = None
+self._load_cache_file()
+self._index_tags()
+
+def __enter__(self):
+return self
+
+def __exit__(self, type, value, traceback):
+checksum = self._read_index_checksum()
+(count, uuid, lastmod) = _read_database_lastmod()
+with open(self.cache_path, "w") as f:
+_json.dump({'prefix': self.current_prefix, 'uuid': uuid, 
'lastmod': lastmod,  'checksum': checksum }, f)
+
+def _load_cache_file(self):
+try:
+with open(self.cache_path) as f:
+data = _json.load(f)
+self.prefix = data['prefix']
+self.uuid = data['uuid']
+self.lastmod = data['lastmod']
+self.checksum = data['checksum']
+except FileNotFoundError:
+return None
+except _json.JSONDecodeError:
+_LOG.error("Error decoding cache")
+_sys.exit(1)
+
+def _read_index_checksum (self):
+"""Read the index checksum, as defined by index-format.txt in the git 
source
+WARNING: assumes SHA1 repo"""
+import binascii
+try:
+with open(self.index_path, 'rb') as f:
+size=_os.path.getsize(self.index_path)
+f.seek(size-20);
+return binascii.hexlify(f.read(20)).decode('ascii')
+except FileNotFoundError:
+return None
+
+@timed
+def _index_tags(self):
+"Write notmuch tags to private git index."
+prefix = '+{0}'.format(_ENCODED_TAG_PREFIX)
+current_checksum = self._read_index_checksum()
+if (self.prefix == None or self.prefix != self.current_prefix
+or self.checksum == None or self.checksum != current_checksum):
+_git(
+args=['read-tree', '--empty'],
+additional_env={'GIT_INDEX_FILE': self.index_path}, wait=True)
+
+query = _tag_query()
+clear_tags = False
+

[PATCH 15/16] debian: install notmuch-git

2022-04-23 Thread David Bremner
Use a separate binary package to avoid dragging in dependencies on
python and git for those that do not want them.
---
 debian/control | 16 
 debian/notmuch-git.install |  1 +
 2 files changed, 17 insertions(+)
 create mode 100644 debian/notmuch-git.install

diff --git a/debian/control b/debian/control
index a11d4130..9706b0f7 100644
--- a/debian/control
+++ b/debian/control
@@ -66,6 +66,22 @@ Description: thread-based email index, search and tagging
  .
  This package contains the notmuch command-line interface
 
+Package: notmuch-git
+Architecture: all
+Depends:
+ git,
+ notmuch,
+ python3,
+ ${misc:Depends}
+Description: thread-based email index, search and tagging
+ Notmuch is a system for indexing, searching, reading, and tagging
+ large collections of email messages in maildir or mh format. It uses
+ the Xapian library to provide fast, full-text search with a very
+ convenient search syntax.
+ .
+ This package contains a simple tool to save, restore, and synchronize
+ notmuch tags via git repositories.
+
 Package: notmuch-doc
 Architecture: all
 Depends:
diff --git a/debian/notmuch-git.install b/debian/notmuch-git.install
new file mode 100644
index ..ca632418
--- /dev/null
+++ b/debian/notmuch-git.install
@@ -0,0 +1 @@
+notmuch-git /usr/bin
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 06/16] CLI/git: suppress warnings about initial branch name

2022-04-23 Thread David Bremner
The canonical nmbug repository still uses "master" as the main branch
name, so defer any potential switch away from that name.
---
 notmuch-git.in | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index d574f19e..3a58fd28 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -384,7 +384,8 @@ def init(remote=None):
 This wraps 'git init' with a few extra steps to support subsequent
 status and commit commands.
 """
-_spawn(args=['git', '--git-dir', NMBGIT, 'init', '--bare'], wait=True)
+_spawn(args=['git', '--git-dir', NMBGIT, 'init',
+ '--initial-branch=master', '--quiet', '--bare'], wait=True)
 _git(args=['config', 'core.logallrefupdates', 'true'], wait=True)
 # create an empty blob (e69de29bb2d1d6434b8b29ae775ad8c2e48c5391)
 _git(args=['hash-object', '-w', '--stdin'], input='', wait=True)
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 11/16] CLI/git: add @timed decorator, time a few functions

2022-04-23 Thread David Bremner
Perf will show which binaries are using the CPU cycles, and standard
python profilers will show which python functions, but neither is
great at finding which call to an external binary is taking time, or
locating I/O hotspots.
---
 notmuch-git.in | 19 ++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index d8b7e45b..0d9f50a1 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -128,6 +128,20 @@ def _xapian_unquote(string):
 return string
 
 
+def timed(fn):
+"""Timer decorator"""
+from time import perf_counter
+
+def inner(*args, **kwargs):
+start_time = perf_counter()
+rval = fn(*args, **kwargs)
+end_time = perf_counter()
+_LOG.info('{0}: {1:.8f}s elapsed'.format(fn.__name__, end_time - 
start_time))
+return rval
+
+return inner
+
+
 class SubprocessError(RuntimeError):
 "A subprocess exited with a nonzero status"
 def __init__(self, args, status, stdout=None, stderr=None):
@@ -360,6 +374,7 @@ def commit(treeish='HEAD', message=None):
 _git(args=['read-tree', treeish], wait=True)
 raise
 
+@timed
 def _update_index(status):
 with _git(
 args=['update-index', '--index-info'],
@@ -600,6 +615,7 @@ def _is_unmerged(ref='@{upstream}'):
 return base != fetch_head
 
 
+@timed
 def get_status():
 status = {
 'deleted': {},
@@ -620,7 +636,7 @@ def get_status():
 _os.remove(index)
 return status
 
-
+@timed
 def _index_tags():
 "Write notmuch tags to the nmbug.index."
 path = _os.path.join(NMBGIT, 'nmbug.index')
@@ -669,6 +685,7 @@ def _index_tags_for_message(id, status, tags):
 yield '{mode} {hash}\t{path}\n'.format(mode=mode, hash=hash, path=path)
 
 
+@timed
 def _diff_index(index, filter):
 """
 Get an {id: {tag, ...}} dict for a given filter.
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 04/16] test: initial tests for notmuch-git

2022-04-23 Thread David Bremner
Exercise the main functionality of notmuch-git.  add_git_repos() will
hopefully be simplifed when an init subcommand is added.
---
 test/T850-git.sh | 95 
 1 file changed, 95 insertions(+)
 create mode 100755 test/T850-git.sh

diff --git a/test/T850-git.sh b/test/T850-git.sh
new file mode 100755
index ..f52dd60d
--- /dev/null
+++ b/test/T850-git.sh
@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+test_description='"notmuch git" to save and restore tags'
+. $(dirname "$0")/test-lib.sh || exit 1
+
+add_git_repos () {
+# first the equivalent of "notmuch git init" (which doesn't exist yet)
+git init --bare --initial-branch=master --quiet remote.git
+git -C remote.git read-tree --empty
+tree=$(git -C remote.git write-tree)
+git -C remote.git hash-object -w /dev/null > /dev/null
+commit=$(echo 'root commit' | git -C remote.git commit-tree $tree)
+git -C remote.git update-ref refs/heads/master $commit
+
+# now make a "local" repo to work on
+git clone --bare --quiet remote.git tags.git
+git -C tags.git config --add remote.origin.fetch 
"+refs/heads/*:refs/remotes/origin/*"
+git -C tags.git fetch origin
+git -C tags.git branch -u origin/master master
+}
+
+add_email_corpus
+add_git_repos
+
+test_begin_subtest "clone"
+# currently broken because of hard-wired requirement for config branch
+test_subtest_known_broken
+test_expect_success "notmuch git -C clone.git clone tags.git"
+
+test_begin_subtest "commit"
+notmuch git -C tags.git -p '' commit
+git -C tags.git ls-tree -r --name-only HEAD | xargs dirname | sort -u | sed 
s,tags/,id:, > OUTPUT
+notmuch search --output=messages '*' | sort > EXPECTED
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
+test_begin_subtest "checkout"
+notmuch dump > BEFORE
+notmuch tag -inbox '*'
+notmuch git -C tags.git -p '' checkout
+notmuch dump > AFTER
+test_expect_equal_file_nonempty BEFORE AFTER
+
+test_begin_subtest "archive"
+notmuch git -C tags.git -p '' archive | tar tf - | \
+grep 20091117190054.gu3...@dottiness.seas.harvard.edu | sort > OUTPUT
+cat < EXPECTED
+tags/20091117190054.gu3...@dottiness.seas.harvard.edu/
+tags/20091117190054.gu3...@dottiness.seas.harvard.edu/inbox
+tags/20091117190054.gu3...@dottiness.seas.harvard.edu/signed
+tags/20091117190054.gu3...@dottiness.seas.harvard.edu/unread
+EOF
+notmuch git -C tags.git -p '' checkout
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "status"
+notmuch tag +test id:20091117190054.gu3...@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' status > OUTPUT
+cat < EXPECTED
+A  20091117190054.gu3...@dottiness.seas.harvard.edutest
+EOF
+notmuch git -C tags.git -p '' checkout
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "fetch"
+notmuch tag +test2 id:20091117190054.gu3...@dottiness.seas.harvard.edu
+notmuch git -C remote.git -p '' commit
+notmuch tag -test2 id:20091117190054.gu3...@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' fetch
+notmuch git -C tags.git -p '' status > OUTPUT
+cat < EXPECTED
+ a 20091117190054.gu3...@dottiness.seas.harvard.edutest2
+EOF
+notmuch git -C tags.git -p '' checkout
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "merge"
+notmuch git -C tags.git -p '' merge
+notmuch dump id:20091117190054.gu3...@dottiness.seas.harvard.edu | grep -v 
'^#' > OUTPUT
+cat < EXPECTED
++inbox +signed +test2 +unread -- 
id:20091117190054.gu3...@dottiness.seas.harvard.edu
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "push"
+notmuch tag +test3 id:20091117190054.gu3...@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' commit
+notmuch tag -test3 id:20091117190054.gu3...@dottiness.seas.harvard.edu
+notmuch git -C tags.git -p '' push
+notmuch git -C remote.git -p '' checkout
+notmuch dump id:20091117190054.gu3...@dottiness.seas.harvard.edu | grep -v 
'^#' > OUTPUT
+cat < EXPECTED
++inbox +signed +test2 +test3 +unread -- 
id:20091117190054.gu3...@dottiness.seas.harvard.edu
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_done
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 07/16] test: use "notmuch git init" for tests.

2022-04-23 Thread David Bremner
Stick with the test-suite specific clone script for now, until clone
no longer insists on the config branch (or init creates a config
branch).
---
 test/T850-git.sh | 8 +---
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/test/T850-git.sh b/test/T850-git.sh
index f52dd60d..d682141f 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -3,13 +3,7 @@ test_description='"notmuch git" to save and restore tags'
 . $(dirname "$0")/test-lib.sh || exit 1
 
 add_git_repos () {
-# first the equivalent of "notmuch git init" (which doesn't exist yet)
-git init --bare --initial-branch=master --quiet remote.git
-git -C remote.git read-tree --empty
-tree=$(git -C remote.git write-tree)
-git -C remote.git hash-object -w /dev/null > /dev/null
-commit=$(echo 'root commit' | git -C remote.git commit-tree $tree)
-git -C remote.git update-ref refs/heads/master $commit
+notmuch git -C remote.git -p '' init
 
 # now make a "local" repo to work on
 git clone --bare --quiet remote.git tags.git
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 12/16] CLI/git: rename private index file.

2022-04-23 Thread David Bremner
All files created by notmuch-git should be in a private directory to
avoid collisions.
---
 notmuch-git.in | 11 +--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 0d9f50a1..b69d57e7 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -638,8 +638,9 @@ def get_status():
 
 @timed
 def _index_tags():
-"Write notmuch tags to the nmbug.index."
-path = _os.path.join(NMBGIT, 'nmbug.index')
+"Write notmuch tags to private git index."
+ensure_private_directory(NMBGIT)
+path = _os.path.join(NMBGIT, 'notmuch','index')
 prefix = '+{0}'.format(_ENCODED_TAG_PREFIX)
 _git(
 args=['read-tree', '--empty'],
@@ -747,6 +748,12 @@ def _help(parser, command=None):
 parser.parse_args(['--help'])
 
 
+def ensure_private_directory(repo):
+try:
+_os.makedirs(_os.path.join(repo, 'notmuch'))
+except FileExistsError:
+pass
+
 if __name__ == '__main__':
 import argparse
 
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 08/16] CLI/git: make existance of config branch optional on clone

2022-04-23 Thread David Bremner
This branch is actually only used by an associated
utility (notmuch-report), and notmuch-git works fine without it.

With this change we can use "notmuch git clone" in the setup for
the tests in T850-git.sh
---
 notmuch-git.in   | 8 +++-
 test/T850-git.sh | 9 +
 2 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 3a58fd28..8b397080 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -303,7 +303,13 @@ def clone(repository):
 wait=True)
 _git(args=['config', '--unset', 'core.worktree'], wait=True, expect=(0, 5))
 _git(args=['config', 'core.bare', 'true'], wait=True)
-_git(args=['branch', 'config', 'origin/config'], wait=True)
+(status, stdout, stderr) = _git(args=['show-ref', '--verify',
+  '--quiet',
+  'refs/remotes/origin/config'],
+expect=(0,1),
+wait=True)
+if status == 0:
+_git(args=['branch', 'config', 'origin/config'], wait=True)
 existing_tags = get_tags()
 if existing_tags:
 _LOG.warning(
diff --git a/test/T850-git.sh b/test/T850-git.sh
index d682141f..417692d4 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -4,20 +4,13 @@ test_description='"notmuch git" to save and restore tags'
 
 add_git_repos () {
 notmuch git -C remote.git -p '' init
-
-# now make a "local" repo to work on
-git clone --bare --quiet remote.git tags.git
-git -C tags.git config --add remote.origin.fetch 
"+refs/heads/*:refs/remotes/origin/*"
-git -C tags.git fetch origin
-git -C tags.git branch -u origin/master master
+notmuch git -C tags.git -p '' clone remote.git
 }
 
 add_email_corpus
 add_git_repos
 
 test_begin_subtest "clone"
-# currently broken because of hard-wired requirement for config branch
-test_subtest_known_broken
 test_expect_success "notmuch git -C clone.git clone tags.git"
 
 test_begin_subtest "commit"
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 10/16] CLI/git: replace enumeration of tags with sexp query.

2022-04-23 Thread David Bremner
Unlike the (current) infix query parser provided by Xapian, the
notmuch specific sexp query parser supports prefixed wildcard queries,
so use those. In addition to being somewhat faster, this avoids
needing to escape all of the user's tags to pass via the shell.
---
 notmuch-git.in   | 23 ---
 test/T850-git.sh |  6 +-
 2 files changed, 21 insertions(+), 8 deletions(-)

diff --git a/notmuch-git.in b/notmuch-git.in
index 8b397080..d8b7e45b 100755
--- a/notmuch-git.in
+++ b/notmuch-git.in
@@ -258,16 +258,17 @@ def _get_remote():
 stdout=_subprocess.PIPE, wait=True)
 return remote.strip()
 
+def _tag_query(prefix=None):
+if prefix is None:
+prefix = TAG_PREFIX
+return '(tag (starts-with "{:s}"))'.format(prefix.replace('"','\\\"'))
 
 def get_tags(prefix=None):
 "Get a list of tags with a given prefix."
-if prefix is None:
-prefix = TAG_PREFIX
 (status, stdout, stderr) = _spawn(
-args=['notmuch', 'search', '--output=tags', '*'],
+args=['notmuch', 'search', '--query=sexp', '--output=tags', 
_tag_query(prefix)],
 stdout=_subprocess.PIPE, wait=True)
-return [tag for tag in stdout.splitlines() if tag.startswith(prefix)]
-
+return [tag for tag in stdout.splitlines()]
 
 def archive(treeish='HEAD', args=()):
 """
@@ -623,13 +624,12 @@ def get_status():
 def _index_tags():
 "Write notmuch tags to the nmbug.index."
 path = _os.path.join(NMBGIT, 'nmbug.index')
-query = ' '.join('tag:"{tag}"'.format(tag=tag) for tag in get_tags())
 prefix = '+{0}'.format(_ENCODED_TAG_PREFIX)
 _git(
 args=['read-tree', '--empty'],
 additional_env={'GIT_INDEX_FILE': path}, wait=True)
 with _spawn(
-args=['notmuch', 'dump', '--format=batch-tag', '--', query],
+args=['notmuch', 'dump', '--format=batch-tag', '--query=sexp', 
'--', _tag_query()],
 stdout=_subprocess.PIPE) as notmuch:
 with _git(
 args=['update-index', '--index-info'],
@@ -872,6 +872,15 @@ if __name__ == '__main__':
 level = getattr(_logging, args.log_level.upper())
 _LOG.setLevel(level)
 
+(status, stdout, stderr) = _spawn(
+args=['notmuch', 'config', 'get', 'built_with.sexp_queries'],
+stdout=_subprocess.PIPE, wait=True)
+if status != 0:
+_LOG.error("failed to run notmuch")
+sys.exit(1)
+if stdout != "true\n":
+_LOG.error("notmuch git needs sexp query support")
+
 if not getattr(args, 'func', None):
 parser.print_usage()
 _sys.exit(1)
diff --git a/test/T850-git.sh b/test/T850-git.sh
index 408a6337..4bf29b20 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -7,6 +7,11 @@ add_git_repos () {
 notmuch git -C tags.git -p '' clone remote.git
 }
 
+if [ $NOTMUCH_HAVE_SFSEXP -ne 1 ]; then
+printf "Skipping due to missing sfsexp library\n"
+test_done
+fi
+
 add_email_corpus
 add_git_repos
 
@@ -20,7 +25,6 @@ notmuch search --output=messages '*' | sort > EXPECTED
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
 test_begin_subtest "commit, with quoted tag"
-test_subtest_known_broken
 notmuch git -C clone2.git -p '' clone tags.git
 git -C clone2.git ls-tree -r --name-only HEAD | grep /inbox > BEFORE
 notmuch tag '+"quoted tag"' '*'
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 16/16] WIP: start manual page for notmuch-git

2022-04-23 Thread David Bremner
---
 doc/conf.py  |   4 ++
 doc/index.rst|   1 +
 doc/man1/notmuch-git.rst | 106 +++
 3 files changed, 111 insertions(+)
 create mode 100644 doc/man1/notmuch-git.rst

diff --git a/doc/conf.py b/doc/conf.py
index e46e1d4e..da0635bb 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -123,6 +123,10 @@ man_pages = [
  u'send mail with notmuch and emacs',
  [notmuch_authors], 1),
 
+('man1/notmuch-git', 'notmuch-git',
+ u'manage notmuch tags with git',
+ [notmuch_authors], 1),
+
 ('man5/notmuch-hooks', 'notmuch-hooks',
  u'hooks for notmuch',
  [notmuch_authors], 5),
diff --git a/doc/index.rst b/doc/index.rst
index fbdcf779..c380ee1d 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -15,6 +15,7 @@ Contents:
man1/notmuch-dump
notmuch-emacs
man1/notmuch-emacs-mua
+   man1/notmuch-git
man5/notmuch-hooks
man1/notmuch-insert
man1/notmuch-new
diff --git a/doc/man1/notmuch-git.rst b/doc/man1/notmuch-git.rst
new file mode 100644
index ..4877f22d
--- /dev/null
+++ b/doc/man1/notmuch-git.rst
@@ -0,0 +1,106 @@
+.. _notmuch-git(1):
+
+
+notmuch-git
+
+
+SYNOPSIS
+
+
+**notmuch** **git** [-h] [-C REPO] [-p PREFIX] [-v] [-l *log level*] 
*subcommand*
+
+DESCRIPTION
+===
+
+Manage notmuch tags with Git.
+
+Options
+---
+
+Supported options for **notmuch-git** include
+
+.. program:: notmuch-git
+
+.. option::  -h, --help
+
+   show help message and exit
+
+.. option:: -C repo, --git-dir repo
+
+   Operate on git repository *repo*
+
+.. option::  -p prefix, --tag-prefix prefix
+
+   Operate only on tags with prefix *prefix*
+
+.. option::   -v, --version
+
+   show notmuch-git's version number and exit
+
+.. option::   -l *level*, --log-level *level* 
{critical,error,warning,info,debug}
+
+   Log verbosity. Defaults to 'warning'.
+
+Subcommands
+---
+
+For help on a particular subcommand, run: 'notmuch-git ...  --help'.
+
+.. option:: archive [TREE-ISH] [ARG ...]
+
+Dump a tar archive of the current nmbug tag set using 'git archive'.
+
+For each tag *tag* for message with Message-Id *id* an empty file
+
+  tags/encode(*id*)/encode(*tag*)
+
+is written to the output.
+
+The encoding preserves alphanumerics, and the characters
+"+-_@=.:," (not the quotes).  All other octets are replaced with
+'%%' followed by a two digit hex number.
+
+positional arguments:
+  TREE-ISHThe tree or commit to produce an archive for. Defaults to
+  'HEAD'.
+  ARG Argument passed through to 'git archive'. Set anything before
+  , see any:`git-archive(1)` for details.
+
+.. option:: checkout
+
+Update the notmuch database from Git.
+
+This is mainly useful to discard your changes in notmuch relative
+to Git.
+Create a local nmbug repository from a remote source.
+
+.. option:: clone repository
+
+This wraps 'git clone', adding some options to avoid creating a
+working tree while preserving remote-tracking branches and
+upstreams.
+
+positional arguments:
+  repository  The (possibly remote) repository to clone from. See the URLS 
section of git-clone(1) for more information on specifying repositories.
+
+clone   Create a local nmbug repository from a remote source.
+commit  Commit prefix-matching tags from the notmuch database 
to Git.
+fetch   Fetch changes from the remote repository.
+helpShow help for an nmbug command.
+initCreate an empty nmbug repository.
+log A simple wrapper for 'git log'.
+merge   Merge changes from 'reference' into HEAD and load the 
result into notmuch.
+pullPull (merge) remote repository changes to notmuch.
+pushPush the local nmbug Git state to a remote repository.
+status  Show pending updates in notmuch or git repo.
+
+
+
+
+SEE ALSO
+
+
+:any:`notmuch(1)`,
+:any:`notmuch-dump(1)`,
+:any:`notmuch-restore(1)`,
+:any:`notmuch-tag(1)`
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


WIP: promote nmbug to user sync tool

2022-04-23 Thread David Bremner
One of the things that new (and old) users of notmuch often miss is a
way to sync tags between hosts. muchsync exists, but it's a third
party tool, and (if I understand correctly) it directly modifies the
notmuch database. Meanwhile we ship a moderately complex python script
called 'nmbug' which can be used to sync arbitrary tags between hosts.
This series is my (first pass at an) attempt to promote nmbug to a
notmuch subcommand "notmuch git". My plan is to replace my own
homebrew "commit notmuch dump output to git" script with this new
command; I think it may appeal to other users as well. As a bonus, it
provides a simple (for the user) way to do incremental backups of the
notmuch database.

I had to add a couple of caching tricks to make the script usable for
my database, but it is now reasonably fast (say 20s) to commit a days
worth of changes on my machine.

There are still some rough edges, mainly due to the heritage of the
script. Some I am aware of include:

- the (new) man page is inadequate
- notmuch git doesn't understand the common arguments to notmuch (--config and 
friends)
- the defaults for prefix and git repo are wrong for most users. I was thinking 
about either a
  --nmbug argument to set the old defaults, or having the script recognize if 
it is invoked as nmbug.
- There are a few too many global variables
- I just remembered, MAX_LASTMOD is unused, leftover from a previous design for 
querying tags.

One thing that not everyone will like is that this requires sexp
queries in your version of notmuch. It's exactly the kind of
application I had in mind when I built the sexp query parser (script
generated queries), and the sexp queries actually allow fixing an
escaping bug in the current nmbug


___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH 2/2] NEWS: user visible changes for 0.36

2022-04-22 Thread David Bremner


Series applied to master and release.

Unless I hear of blockers, I'll probably cut a release this weekend.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Autorefresh buffer at set interval

2022-04-21 Thread David Bremner
Ishe Chinyoka  writes:

> I am sorry if this is in the documentation, but I am failing to pick
> the right spot. Anyway, which variable can I set to have notmuch-poll
> triggered at a set interval?

I don't think this currently exists. You could define your own timer
[1].

> BTW, I am on Arch and am running the Offlineimap as a systemd timer to
> pull my messages every fifteen minutes. I also have a Notmuch script
> under the ~/.mail/.notmuch/hooks/ that runs every time offlineimap
> completes the refresh. My problem is now that after all this, I can't
> seem to have the Notmuch emacs interface update automatically without me
> doing a manual refresh with "G" keystroke.

I suspect you only need "g" rather than "G", if you are already running
notmuch new via a hook.

> This morning I tried putting
> notmuch-poll-and-refresh-this-buffer as a function in the notmuch-hello
> hooks.

You may want the function notmuch-refresh-all-buffers; that could go in
your timer.

[1]: (info "(elisp) Timers")

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] configure: avoid warning with -Wall

2022-04-19 Thread David Bremner
Tomi Ollila  writes:

> On Mon, Apr 18 2022, michaeljgruber wrote:
>
>> From: Michael J Gruber 
>>
>> 7228fe68 ("configure: restructure gmime cert validity checker code",
>> 2022-04-09) restructured generated C code to repurpose it later on. This
>> put usage of `validity` within an `#if`, resulting in an "unused
>> warning" if that `#if` is not executed.
>>
>> Put the variable declariation inside the same if branch and, thus,  quel
>> the warning.
>
> IMO the code could be changed to be even more RAIIy (even so that is not
> common style there), also IMO removing warning is good enough reason to
> deviate from that style (just in one place) so LGTM from me :D
>
> Tomi

applied to master.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[Wishlist] Add equivalent of lastmod:nnn.. to sexp query parser

2022-04-18 Thread David Bremner


This is mainly a reminder to myself, but also an opportunity for query
language bikeshedding.

Currently the sexp query parser parses '(lastmod 1234)' as equivalent to
lastmod:1234..1234. This is reasonable enough, but leaves no easy way to
do the equivalent to lastmod:1234.. (the other one sided range
lastmod:..1234 is easy since 0 works on the left e.g. (lastmod 0 1234).

This could be fixed by

1) changing the semantics of the 1 argument (lastmod 1234)
2) re-using * as in (lastmod 1234 *)
3) introducing a new reserved atom, e.g. "max" (lastmod 1234 max)



___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 2/2] NEWS: user visible changes for 0.36

2022-04-16 Thread David Bremner
---
 NEWS | 42 ++
 1 file changed, 42 insertions(+)

diff --git a/NEWS b/NEWS
index f2a2df0d..63e63f4b 100644
--- a/NEWS
+++ b/NEWS
@@ -1,6 +1,48 @@
 Notmuch 0.36 (UNRELEASED)
 =
 
+Library
+---
+
+Add the `sexp` prefix to the infix (traditional) query parser. This
+allows specific subqueries to be parsed by the sexp parser (with
+appropropriate quoting). See `notmuch-search-terms(7)` for details.
+
+Add another heuristic to regexp fields to prevent phrase parsing of
+bracketed sub-expressions.
+
+Command Line Interface
+--
+
+Envelope from ("From ") headers are now escaped as X-Envelope-From: in
+input to `notmuch-insert`. This prevents creating mbox files when
+calling `notmuch-insert` from e.g. `postfix`.
+
+Python (CFFI) Bindings
+--
+
+Use the `config_pairs` API in ConfigIterator. This returns all
+matching key-value pairs, not just those that happen to be stored in
+the database.
+
+Documentation
+-
+
+Reorganize documention for `notmuch-config`. Add a few links from
+other man pages.
+
+Emacs
+-
+
+Bind the usual undo key sequences to new command
+"notmuch-tag-undo". This allows transparent undo of tagging
+operations.
+
+Tests
+-
+
+Fix smime.4 with newer gmime. Unset XDG_DATA_HOME and MAILDIR for tests.
+
 New add-on tool: notmuch-web
 -
 
-- 
2.35.1

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 1/2] NEWS: fix location for notmuch-web

2022-04-16 Thread David Bremner
It actually lives under 'devel/', not 'contrib/`.
---
 NEWS | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/NEWS b/NEWS
index 23be75de..f2a2df0d 100644
--- a/NEWS
+++ b/NEWS
@@ -4,7 +4,7 @@ Notmuch 0.36 (UNRELEASED)
 New add-on tool: notmuch-web
 -
 
-The new contrib/ tool `notmuch-web` is a very thin web client.  It
+The new devel/ tool `notmuch-web` is a very thin web client.  It
 supports a full search interface for one user: there is no facility
 for multiple users provided today.  See the notmuch-web README file
 for more information.
-- 
2.35.1

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: release process for notmuch 0.36

2022-04-16 Thread David Bremner
David Bremner  writes:

> David Bremner  writes:
>
>> I plan to feature freeze in mid April, and release 0.36 sometime before
>> the end of April.
>
> As promised I have tagged and uploaded the first release candidate for
> 0.36. I will watch for bug reports for a week or so before doing the
> final release.

I have tagged and uploaded a second release candidate, with Michael's
fix for non-sfsexpr build.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] fix build without sfsexp

2022-04-15 Thread David Bremner
michaeljgruber+grubix+...@gmail.com writes:

> From: Michael J Gruber 
>
> a1d139de ("lib: add sexp: prefix to Xapian (infix) query parser.",
> 2022-04-09) introduced sfsexp infix queries. This requires the infix
> preprocessor to be built in in a way which does not require sfsexp when
> notmuch is built without it.
>
> Make the preprocessor throw a Xapian error in this case (and fix the
> build).
>
> Signed-off-by: Michael J Gruber 
> ---
> Maybe something like this - I have not test *with* sfsexp, though.

Either your patch or mine is OK with me. Yours might be slightly nicer
since it gives an actual error message, rather than falling back to the
corresponding phrase (generic Xapian query parser behaviour with unknown
prefixes).

I did test both with and without libsfsexp-dev installed.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] lib: don't compile sexp: prefix if no sfsexp library

2022-04-15 Thread David Bremner
As Michael Gruber pointed out, the build is otherwise broken on
systems without libsfsexp.
---
 lib/prefix.cc  | 4 
 lib/sexp-fp.cc | 3 +++
 lib/sexp-fp.h  | 2 ++
 3 files changed, 9 insertions(+)

diff --git a/lib/prefix.cc b/lib/prefix.cc
index 06e2333a..538a2dd1 100644
--- a/lib/prefix.cc
+++ b/lib/prefix.cc
@@ -61,8 +61,10 @@ prefix_t prefix_table[] = {
   NOTMUCH_FIELD_PROCESSOR },
 { "query",  NULL,   NOTMUCH_FIELD_EXTERNAL |
   NOTMUCH_FIELD_PROCESSOR },
+#if HAVE_SFSEXP
 { "sexp",  NULL,NOTMUCH_FIELD_EXTERNAL |
   NOTMUCH_FIELD_PROCESSOR },
+#endif
 { "from",   "XFROM",NOTMUCH_FIELD_EXTERNAL |
   NOTMUCH_FIELD_PROBABILISTIC |
   NOTMUCH_FIELD_PROCESSOR },
@@ -141,8 +143,10 @@ _setup_query_field (const prefix_t *prefix, 
notmuch_database_t *notmuch)
fp = (new QueryFieldProcessor (*notmuch->query_parser, 
notmuch))->release ();
else if (STRNCMP_LITERAL (prefix->name, "thread") == 0)
fp = (new ThreadFieldProcessor (*notmuch->query_parser, 
notmuch))->release ();
+#if HAVE_SFSEXP
else if (STRNCMP_LITERAL (prefix->name, "sexp") == 0)
fp = (new SexpFieldProcessor (notmuch))->release ();
+#endif
else
fp = (new RegexpFieldProcessor (prefix->name, prefix->flags,
*notmuch->query_parser, 
notmuch))->release ();
diff --git a/lib/sexp-fp.cc b/lib/sexp-fp.cc
index ed26f6ec..eeb8be98 100644
--- a/lib/sexp-fp.cc
+++ b/lib/sexp-fp.cc
@@ -21,6 +21,8 @@
  */
 
 #include "database-private.h"
+
+#if HAVE_SFSEXP
 #include "sexp-fp.h"
 #include 
 
@@ -38,3 +40,4 @@ SexpFieldProcessor::operator() (const std::string & 
query_string)
 return output;
 
 }
+#endif /* HAVE_SFSEXP */
diff --git a/lib/sexp-fp.h b/lib/sexp-fp.h
index 341dfa7e..0e55b961 100644
--- a/lib/sexp-fp.h
+++ b/lib/sexp-fp.h
@@ -23,6 +23,7 @@
 #ifndef NOTMUCH_SEXP_FP_H
 #define NOTMUCH_SEXP_FP_H
 
+#if HAVE_SFSEXP
 #include 
 #include "notmuch.h"
 
@@ -38,4 +39,5 @@ public:
 Xapian::Query operator() (const std::string & query_string);
 };
 
+#endif /* HAVE_SFSEXP */
 #endif /* NOTMUCH_SEXP_FP_H */
-- 
2.35.1

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: release process for notmuch 0.36

2022-04-15 Thread David Bremner
David Bremner  writes:

> I plan to feature freeze in mid April, and release 0.36 sometime before
> the end of April.

As promised I have tagged and uploaded the first release candidate for
0.36. I will watch for bug reports for a week or so before doing the
final release.

d


signature.asc
Description: PGP signature
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH v2 2/2] lib: add sexp: prefix to Xapian (infix) query parser.

2022-04-15 Thread David Bremner
David Bremner  writes:

> This is analogous to the "infix" prefix provided by the s-expression
> based query parser.

series applied to master.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show

2022-04-13 Thread David Bremner
Alexander Adolf  writes:

> Hello David,
>
> David Bremner  writes:
>
>> [...]
>> I'm not sure it's less effort, 
>> [...]
>
> Neither am I... ;-))
>
> It might perhaps seem easier to run the tests I have added to the
> attached, updated patch in your complete environment?
>
>   --alexander


It seems that it is mostly working, but there are a few issues to iron out.

The first is easy, I think. Since we added a message to the crypto
corpus, we need to adjust the tests. I think it is fine to just add
the new message to the failing test output, as in the diff below.

T357-index-decryption: Testing indexing decrypted mail
 FAIL   indexing message fails when secret key not available
--- T357-index-decryption.31.expected   2022-04-13 23:15:02.258922959 
+
+++ T357-index-decryption.31.output 2022-04-13 23:15:02.258922959 
+
@@ -1,5 +1,6 @@
 #= simple-encryp...@crypto.notmuchmail.org index.decryption=failure
 #notmuch-dump batch-tag:3 config,properties,tags
++encrypted +inbox +unread -- 
id:575ddaaf0b234fd85e077cfb4d44d...@notmuchmail.org
 +encrypted +inbox +unread -- id:basic-encryp...@crypto.notmuchmail.org
 +encrypted +inbox +unread -- 
id:encrypted-rfc822-attachm...@crypto.notmuchmail.org
 +encrypted +inbox +unread -- id:encrypted-sig...@crypto.notmuchmail.org


T450-emacs-show: Testing emacs notmuch-show view
 BROKEN show encrypted rfc822 message
!!! Bodypart handler `notmuch-show-insert-part-*/*' threw an error:
!!! Symbol’s value as variable is void: gnus-newsgroup-charset
 FAIL   process cryptographic MIME parts (S/MIME)
--- T450-emacs-show.21.notmuch-show-smime-encrypted-signed-multipart
2022-04-13 23:15:11.267167711 +
+++ T450-emacs-show.21.OUTPUT   2022-04-13 23:15:11.267167711 +
@@ -1,4 +1,4 @@
-test_su...@notmuchmail.org (0 mins. ago) (encrypted inbox)
+test_su...@notmuchmail.org (Yest. 14:57) (encrypted inbox)

If you set notmuch-show-relative-dates to to nil, something like, with
maybe better indentation:

-test_emacs '(let ((notmuch-crypto-process-mime t))
+test_emacs '(let ((notmuch-crypto-process-mime t)
+  (notmuch-show-relative-dates nil))

Then you will get an actual date that you can hard code.

 Subject: notmuch-show S/MIME test
 To: test_su...@notmuchmail.org
 Date: Tue, 12 Apr 2022 16:57:30 +0200
@@ -6,10 +6,10 @@
 [ smime.p7m: application/pkcs7-mime ]
 [ Decryption successful ]
 [ multipart/signed ]
-[ Good signature by:  ]
+[ Good signature by: test_su...@notmuchmail.org ]

This is exactly the problem we just dealt with for T355-smime. I think
the same solution can be applied, but you will need to inline the output
so that you can do variable substitution.

 [ multipart/mixed ]
 [ multipart/mixed ]
 [ text/plain ]
 The password is "12345678". But don't tell anyone!
-[ test.dtd: application/octet-stream ]
+[ test.dtd: application/octet-stream (as application/xml-dtd) ]

This seems related to the function
#'notmuch-show-get-mime-type-of-application/octet-stream, which is
calling (mailcap-extension-to-mime "dtd"). Probably this is hard to make
reproducible, so just seding away an "(as ...)" string is a reasonable
alternative.

 [ smime.p7s: application/pkcs7-signature ]



___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH v2 2/2] test/smime: fix signature verification test with newer gmime.

2022-04-13 Thread David Bremner
Daniel Kahn Gillmor  writes:

> Thanks, Michael--
>
> This LGTM.
>
> It is more narrowly-targeted at permitting this specific variation than
> Bremner's earlier version of the patch (and it doesn't have any tests
> marked BROKEN), which is nice.
>
> It might be marginally cleaner to swap out the LEFT_ANGLE RIGHT_ANGLE
> variables for a single replacement variable like so:
>
> if [ $NOTMUCH_GMIME_EMITS_ANGLE_BRACKETS == 1 ]; then
>EXPECTED_EMAIL_ADDR=''
> else
>EXPECTED_EMAIL_ADDR='test_su...@notmuchmail.org'
> fi

In the end I applied this version to master. Thanks to you both.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Display tags on separate lines on notmuch-hello screen

2022-04-12 Thread David Bremner
Ishe Chinyoka  writes:

> Hi list,
>
> Is it possible to have my tags---whether under saved searches or all
> tagss---displayed as a literal list, that  is each tag along with its
> uncounted number on its own line? Currently I see them all listed on a
> single line.
>

Try setting the variable notmuch-column-control to 1.0 (e.g. with M-x
customize-variable). See the docstring for more options.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show

2022-04-11 Thread David Bremner
Alexander Adolf  writes:

> David Bremner  writes:
>
>> [...]
>> You could also run T355-smime with --debug and inspect
>> tmp.T355-smime/mail afterwards, there should be some signed and
>> encrypted mails.
>> [...]
>
> $ ./T355-smime.sh --debug
> Run tests in a subdir of built notmuch tree.
>
> Any way around the effort of building from source? I'm interested in the
> elisp part only, and I have notmuch installed in /usr/local.
>

I'm not sure it's less effort, but I started making the tests work
without building from source at

  
https://nmbug.notmuchmail.org/nmweb/show/20211025145753.3031094-2-da...@tethera.net

It turns out to be actually slightly tricky (the messages are erased
part way through that file), do I attach the corresponding message.

Hopefully you can figure out how to run your test once it is
written. 



smime-enc.eml:2,S
Description: Binary data
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH v2 2/2] lib: add sexp: prefix to Xapian (infix) query parser.

2022-04-09 Thread David Bremner
This is analogous to the "infix" prefix provided by the s-expression
based query parser.
---
 doc/man7/notmuch-search-terms.rst | 17 -
 lib/Makefile.local|  3 ++-
 lib/prefix.cc |  5 
 lib/sexp-fp.cc| 40 ++
 lib/sexp-fp.h | 41 +++
 test/T081-sexpr-search.sh | 25 +++
 6 files changed, 129 insertions(+), 2 deletions(-)
 create mode 100644 lib/sexp-fp.cc
 create mode 100644 lib/sexp-fp.h

diff --git a/doc/man7/notmuch-search-terms.rst 
b/doc/man7/notmuch-search-terms.rst
index f8ad1edb..4f616b7e 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -169,6 +169,12 @@ property:=
 can be present on a given message with several different values.
 See :any:`notmuch-properties(7)` for more details.
 
+sexp:
+The **sexp:** prefix allows subqueries in the format
+documented in :any:`notmuch-sexp-queries(7)`. Note that subqueries 
containing
+spaces must be quoted, and any embedded double quotes must be escaped
+(see :any:`quoting`).
+
 User defined prefixes are also supported, see :any:`notmuch-config(1)` for
 details.
 
@@ -257,7 +263,7 @@ Boolean
 Probabilistic
   **body:**, **to:**, **attachment:**, **mimetype:**
 Special
-   **from:**, **query:**, **subject:**
+   **from:**, **query:**, **subject:**, **sexp:**
 
 Terms and phrases
 -
@@ -297,6 +303,8 @@ Both of these will match a subject "Free Delicious Pizza" 
while
 
 will not.
 
+.. _quoting:
+
 Quoting
 ---
 
@@ -324,6 +332,13 @@ e.g.
% notmuch search 'folder:"/^.*/(Junk|Spam)$/"'
% notmuch search 'thread:"{from:mallory and date:2009}" and 
thread:{to:mallory}'
 
+Double quotes within query strings need to be doubled to escape them.
+
+::
+
+   % notmuch search 'tag:"""quoted tag"""'
+   % notmuch search 'sexp:"(or ""wizard"" ""php"")"'
+
 DATE AND TIME SEARCH
 
 
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 1378a74b..6d67a2a4 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -64,7 +64,8 @@ libnotmuch_cxx_srcs = \
$(dir)/prefix.cc\
$(dir)/open.cc  \
$(dir)/init.cc  \
-   $(dir)/parse-sexp.cc
+   $(dir)/parse-sexp.cc\
+   $(dir)/sexp-fp.cc
 
 libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
 
diff --git a/lib/prefix.cc b/lib/prefix.cc
index 857c05b9..06e2333a 100644
--- a/lib/prefix.cc
+++ b/lib/prefix.cc
@@ -3,6 +3,7 @@
 #include "thread-fp.h"
 #include "regexp-fields.h"
 #include "parse-time-vrp.h"
+#include "sexp-fp.h"
 
 typedef struct {
 const char *name;
@@ -60,6 +61,8 @@ prefix_t prefix_table[] = {
   NOTMUCH_FIELD_PROCESSOR },
 { "query",  NULL,   NOTMUCH_FIELD_EXTERNAL |
   NOTMUCH_FIELD_PROCESSOR },
+{ "sexp",  NULL,NOTMUCH_FIELD_EXTERNAL |
+  NOTMUCH_FIELD_PROCESSOR },
 { "from",   "XFROM",NOTMUCH_FIELD_EXTERNAL |
   NOTMUCH_FIELD_PROBABILISTIC |
   NOTMUCH_FIELD_PROCESSOR },
@@ -138,6 +141,8 @@ _setup_query_field (const prefix_t *prefix, 
notmuch_database_t *notmuch)
fp = (new QueryFieldProcessor (*notmuch->query_parser, 
notmuch))->release ();
else if (STRNCMP_LITERAL (prefix->name, "thread") == 0)
fp = (new ThreadFieldProcessor (*notmuch->query_parser, 
notmuch))->release ();
+   else if (STRNCMP_LITERAL (prefix->name, "sexp") == 0)
+   fp = (new SexpFieldProcessor (notmuch))->release ();
else
fp = (new RegexpFieldProcessor (prefix->name, prefix->flags,
*notmuch->query_parser, 
notmuch))->release ();
diff --git a/lib/sexp-fp.cc b/lib/sexp-fp.cc
new file mode 100644
index ..ed26f6ec
--- /dev/null
+++ b/lib/sexp-fp.cc
@@ -0,0 +1,40 @@
+/* sexp-fp.cc - "sexp:" field processor glue
+ *
+ * This file is part of notmuch.
+ *
+ * Copyright © 2022 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.

[PATCH v2 1/2] test/sexp: add test for and of stemmed terms.

2022-04-09 Thread David Bremner
Previously only singled stemmed terms were tested.
---
 test/T081-sexpr-search.sh | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index e2936cd7..07b12619 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -31,6 +31,13 @@ thread:XXX   2009-11-18 [1/3] Carl Worth| Jan Janak; 
[notmuch] What a great idea
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "and of stemmed terms"
+notmuch search --query=sexp '(and wonderful wizard)' | notmuch_search_sanitize 
> OUTPUT
+cat < EXPECTED
+thread:XXX   2009-11-18 [1/3] Carl Worth| Jan Janak; [notmuch] What a great 
idea! (inbox unread)
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "or of exact terms"
 notmuch search --query=sexp '(or "php" "wizard")' | notmuch_search_sanitize > 
OUTPUT
 cat < EXPECTED
-- 
2.35.1

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


v2 allow sexp queries in infix parser

2022-04-09 Thread David Bremner
This is a pretty useful feature, so I want to include it 0.36. That
means you have a bit less than a week to give me feedback.

Speaking of feedback, I have no idea why the "and of stemmed terms"
test was included in the first version of this patch. I have split it
out to it's own patch.


___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 2/2] test/smime: fix signature verification test with newer gmime.

2022-04-09 Thread David Bremner
The extra machinery to check for the actual output format is justified
by the possibility that distros may patch this newere output format
into older versions of gmime.
---
 configure  | 17 +
 test/T355-smime.sh |  7 +--
 2 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/configure b/configure
index d6e1200e..922de60d 100755
--- a/configure
+++ b/configure
@@ -588,6 +588,11 @@ int main () {
 #ifdef CHECK_VALIDITY
 validity = g_mime_certificate_get_id_validity (cert);
 if (validity != GMIME_VALIDITY_FULL) return !! fprintf (stderr, "Got 
validity %d, expected %d\n", validity, GMIME_VALIDITY_FULL);
+#endif
+#ifdef CHECK_EMAIL
+const char *email = g_mime_certificate_get_email (cert);
+if (! email) return !! fprintf (stderr, "no email returned");
+if (email[0] == '<') return 2;
 #endif
 return 0;
 }
@@ -622,6 +627,15 @@ EOF
errors=$((errors + 1))
fi
fi
+   printf "Checking for GMime new email format... "
+   if ${CC} -DCHECK_EMAIL ${CFLAGS} ${gmime_cflags} _check_gmime_cert.c 
${gmime_ldflags} -o _check_email &&
+   GNUPGHOME=${TEMP_GPG} ./_check_email; then
+   gmime_new_email_format=1
+   printf "Yes.\n"
+   else
+   gmime_new_email_format=0
+   printf "No (some tests will be skipped).\n"
+   fi
 else
printf 'No.\nFailed to set up gpgsm for testing X.509 certificate 
validity support.\n'
errors=$((errors + 1))
@@ -1559,6 +1573,9 @@ NOTMUCH_HAVE_XAPIAN_DB_RETRY_LOCK=${WITH_RETRY_LOCK}
 # Whether GMime can verify X.509 certificate validity
 NOTMUCH_GMIME_X509_CERT_VALIDITY=${gmime_x509_cert_validity}
 
+# Whether GMime returns bare emails (without <>)
+NOTMUCH_GMIME_NEW_EMAIL_FORMAT=${gmime_new_email_format}
+
 # Whether GMime can verify signatures when decrypting with a session key:
 NOTMUCH_GMIME_VERIFY_WITH_SESSION_KEY=${gmime_verify_with_session_key}
 
diff --git a/test/T355-smime.sh b/test/T355-smime.sh
index 31fa4b4e..2905263b 100755
--- a/test/T355-smime.sh
+++ b/test/T355-smime.sh
@@ -35,6 +35,9 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "signature verification (notmuch CLI)"
+if [ $NOTMUCH_GMIME_NEW_EMAIL_FORMAT != 1 ]; then
+test_subtest_known_broken
+fi
 output=$(notmuch show --format=json --verify subject:"test signed message 001" 
\
 | notmuch_json_show_sanitize \
 | sed -e 's|"created": [-1234567890]*|"created": 946728000|g' \
@@ -46,7 +49,7 @@ expected='[[[{"id": "X",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
  "tags": ["inbox","signed"],
- "crypto": {"signed": {"status": [{"fingerprint": "'$FINGERPRINT'", "status": 
"good","userid": "CN=Notmuch Test Suite", "email": 
"", "expires": 424242424, "created": 946728000}]}},
+ "crypto": {"signed": {"status": [{"fingerprint": "'$FINGERPRINT'", "status": 
"good","userid": "CN=Notmuch Test Suite", "email": 
"test_su...@notmuchmail.org", "expires": 424242424, "created": 946728000}]}},
  "headers": {"Subject": "test signed message 001",
  "From": "Notmuch Test Suite ",
  "To": "test_su...@notmuchmail.org",
@@ -55,7 +58,7 @@ expected='[[[{"id": "X",
  "sigstatus": [{"fingerprint": "'$FINGERPRINT'",
  "status": "good",
  "userid": "CN=Notmuch Test Suite",
- "email": "",
+ "email": "test_su...@notmuchmail.org",
  "expires": 424242424,
  "created": 946728000}],
  "content-type": "multipart/signed",
-- 
2.35.1

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH 1/2] configure: restructure gmime cert validity checker code

2022-04-09 Thread David Bremner
The goal is to generalize this to also check the output format of
g_mime_certificate_get_email.
---
 configure | 16 +---
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/configure b/configure
index 36f3f606..d6e1200e 100755
--- a/configure
+++ b/configure
@@ -552,11 +552,7 @@ EOF
rm -rf "$TEMP_GPG"
 fi
 
-# see https://github.com/jstedfast/gmime/pull/90
-# should be fixed in GMime in 3.2.7, but some distros might patch
-printf "Checking for GMime X.509 certificate validity... "
-
-cat > _check_x509_validity.c < _check_gmime_cert.c <
 #include 
 
@@ -589,16 +585,22 @@ int main () {
 if (sig == NULL) return !! fprintf (stderr, "no GMimeSignature found at 
position 0\n");
 cert = g_mime_signature_get_certificate (sig);
 if (cert == NULL) return !! fprintf (stderr, "no GMimeCertificate 
found\n");
+#ifdef CHECK_VALIDITY
 validity = g_mime_certificate_get_id_validity (cert);
 if (validity != GMIME_VALIDITY_FULL) return !! fprintf (stderr, "Got 
validity %d, expected %d\n", validity, GMIME_VALIDITY_FULL);
-
+#endif
 return 0;
 }
 EOF
+
+# see https://github.com/jstedfast/gmime/pull/90
+# should be fixed in GMime in 3.2.7, but some distros might patch
+printf "Checking for GMime X.509 certificate validity... "
+
 if ! TEMP_GPG=$(mktemp -d "${TMPDIR:-/tmp}/notmuch.XX"); then
printf 'No.\nCould not make tempdir for testing X.509 certificate 
validity support.\n'
errors=$((errors + 1))
-elif ${CC} ${CFLAGS} ${gmime_cflags} _check_x509_validity.c 
${gmime_ldflags} -o _check_x509_validity \
+elif ${CC} -DCHECK_VALIDITY ${CFLAGS} ${gmime_cflags} _check_gmime_cert.c 
${gmime_ldflags} -o _check_x509_validity \
&& echo disable-crl-checks > "$TEMP_GPG/gpgsm.conf" \
&& echo 
"4D:E0:FF:63:C0:E9:EC:01:29:11:C8:7A:EE:DA:3A:9A:7F:6E:C1:0D S" >> 
"$TEMP_GPG/trustlist.txt" \
&& GNUPGHOME=${TEMP_GPG} gpgsm --batch --quiet --import < 
"$srcdir"/test/smime/ca.crt
-- 
2.35.1

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] doc/emacs: document a few notmuch-hello customizable variables.

2022-04-08 Thread David Bremner
A user asked about the thousands separator on IRC, and I had to check
the source.
---
 doc/conf.py   |  2 +-
 doc/notmuch-emacs.rst | 12 +++-
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/doc/conf.py b/doc/conf.py
index e46e1d4e..fee52b64 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -45,7 +45,7 @@ if tags.has('WITH_EMACS'):
 # Hacky reimplementation of include to workaround limitations of
 # sphinx-doc
 lines = ['.. include:: /../emacs/rstdoc.rsti\n\n'] # in the source tree
-for file in ('notmuch.rsti', 'notmuch-lib.rsti', 'notmuch-show.rsti', 
'notmuch-tag.rsti', 'notmuch-tree.rsti'):
+for file in ('notmuch.rsti', 'notmuch-lib.rsti', 'notmuch-hello.rsti', 
'notmuch-show.rsti', 'notmuch-tag.rsti', 'notmuch-tree.rsti'):
 lines.extend(open(rsti_dir+'/'+file))
 rst_epilog = ''.join(lines)
 del lines
diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index 41f62390..346d0342 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -46,9 +46,19 @@ a mouse or by positioning the cursor and pressing 

 |Customize Notmuch or this page.
 
 You can change the overall appearance of the notmuch-hello screen by
-customizing the variable :index:`notmuch-hello-sections`.
+customizing the variables
 
+:index:`notmuch-hello-sections`
+   |docstring::notmuch-hello-sections|
 
+:index:`notmuch-hello-thousands-separator`
+   |docstring::notmuch-hello-thousands-separator|
+
+:index:`notmuch-show-logo`
+   |docstring::notmuch-show-logo|
+
+:index:`notmuch-column-control`
+   |docstring::notmuch-column-control|
 
 notmuch-hello key bindings
 --
-- 
2.35.1

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show

2022-04-07 Thread David Bremner
Alexander Adolf  writes:

>
> Where can I find the S/MIME test key? test/smime directory?

test/smime/0xE0972A47.p12

> Is there a better way for generating the test messages than importing
> the test key into my "production" keychain? Look to T355-smime for
> inspiration?

The former is probably the simplest. You could also run T355-smime with
--debug and inspect tmp.T355-smime/mail afterwards, there should be some
signed and encrypted mails.

> Where do I put the generated test messages to be able to use them in the
> test?

It should be fine to add to corpora/crypto, then call

add_email_corpus crypto

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show

2022-04-07 Thread David Bremner
Alexander Adolf  writes:

>
> I have had a look to both files, and I _think_ T450-emacs-show would
> seem more appropriate. It seems that T355-smime is more aimed at the
> pure mechanics of en-/decryption, and the handling of that in
> libnotmuch. Neither of these have been a problem for me.
>
> Further looking at T450-emacs-show, I find that it uses a somewhat
> extensive framework to implement its test cases. Any chances of giving
> me a fast start with this, or will it be down to "use the source, Luke"?
>

I think something like the following test (but with an smime encrypted
message instead of the existing PGP/MIME encrypted message.

test_begin_subtest "process cryptographic MIME parts"
;; additional variables can be dynamically bound as needed, but this one
;; in particular is needed for decryption
test_emacs '(let ((notmuch-crypto-process-mime t))
;; simulate the user action
(notmuch-show
"id:20091117203301.gv3...@dottiness.seas.harvard.edu")
;; this is the test framework, write the terminal display to a file
;; called OUTPUT
(test-visible-output))'
test_expect_equal_file $EXPECTED/notmuch-show-process-crypto-mime-parts-on 
OUTPUT

you would also need to create a file with a "screen display" in
$EXPECTED, and add that as part of your patch.

I hope that helps,

David
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] emacs: Add more front ends for address completion

2022-04-06 Thread David Bremner
Utkarsh Singh  writes:

>>
>> Can you be more precise about what you are asking / proposing here?
>> Assume I only skimmed the thread.
>
> Currently, notmuch-address.el, the library used to generate completion
> candidates for recipient's addresses in Notmuch's message compostion
> mode uses non-standard Emacs API's for in-buffer completion, namely
> completing-read and Company.  Now these API's makes it difficult to
> utilize alternative in-buffer completion UI such as Corfu(1).

#'completing-read is of course standard since forever. We might be using
 it in some strange way, or it might be superceded by better things, but
completing-read is not non-standard.

>
> These are the proposed solutions for the given problem:
>
> 1. Add completion-at-point to the existing sets of frontend.  As noted
> by Tomi, this is a "messy" solution as it unnecessarily obfuscate the
> user options `notmuch-address-selection-function' and
> `notmuch-address-command'.
>
> 2. Make notmuch-address.el itself a backend for EUDC(2).  As stated by
> Alexander, this will not only remove any frontend related code from
> notmuch-address.el, but will also unify completion condidates for other
> sources such as BBDB, LDAP, MacOS Contacts, etc.
>

As a reviewer / release manager, I care about the amount of code
changes. As a user I hope my existing setup will keep working with no,
or (less good) minimal changes. Other than that I am open to ideas.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show

2022-03-29 Thread David Bremner
Alexander Adolf  writes:

> The issue I saw with S/MIME encrypted messages was, that the processing
> of the MIME tree stopped after decryption. It turned out there was no
> handler for application/(x-)?pkcs-mime type entities in notmuch-show.el,
> and the "handler of last resort" notmuch-show-insert-part-*/* was
> called. Thus, this patch just adds a notmuch-show handler for
> application/(x-)?pkcs-mime type entities.

Hi Alexander;

Thanks for sending that. It's a bit tricky to review SMIME stuff as I
believe none of the devs is really using SMIME regularly. One thing that
would help is adding one or more tests. Basically the test suite already
has a SMIME certificate, so if you can encrypt a message to that cert
(or use one of the existing ones), you can test that the message
displays OK in Emacs. The corresponding test set would probably be
T450-emacs-show, but it might be easier to add to T355-smime. Eventually
we'll also need a short update to doc/notmuch-emacs.rst.

I hope that helps move review forward,

David
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] emacs: add new option notmuch-search-exclude

2022-03-25 Thread David Bremner
Mohsin Kaleem  writes:

> David Bremner  writes:
>
>> Try adding a line
>> "unset XDG_DATA_HOME"
>>
>> after
>>
>> unset XDG_CONFIG_HOME
>>
>> on line 108 of test/test-lib.sh
>
> Doesn't seem to be such a line in test/test-lib.sh, but I added it to
> test/test-vars.sh and we're down from 42 failing tests to 4 :-).

Ah right, I was looking at an old version.

>
> T391-python-cffi: Testing python bindings (pytest)
>  FAIL   python cffi tests (NOTMUCH_CONFIG set)
>   (cd /home/mohkale/prog/repos/notmuch/bindings/python-cffi/build/stage 
> && python3 -m pytest --verbose 
> --log-file=/home/mohkale/prog/repos/notmuch/test/tmp.T391-python-cffi/test.output)
> /usr/bin/python3: No module named pytest
>  FAIL   python cffi tests (NOTMUCH_CONFIG unset)
>   (cd /home/mohkale/prog/repos/notmuch/bindings/python-cffi/build/stage 
> && python3 -m pytest --verbose 
> --log-file=/home/mohkale/prog/repos/notmuch/test/tmp.T391-python-cffi/test.output)

So does "python3 -m pytest" work for you? Assuming yes, is there some
special environment needed? We don't explicitly unset PYTHONPATH in
T391, so in if you need PYTHONPATH to locate pytest, that should work
here as well. What output do you get from "python3 -c 'import pytest;
print(pytest.__file__)'"

> T560-lib-error: Testing error reporting for library
>  FAIL   Open null pointer
>   --- T560-lib-error.2.EXPECTED   2022-03-22 20:25:30.496475869 +
>   +++ T560-lib-error.2.OUTPUT 2022-03-22 20:25:30.496475869 +
>   @@ -1,3 +1,2 @@
>== stdout ==
>== stderr ==
>   -Error: could not locate database.
> test1.c: In function 'main':
> test1.c:7:5: warning: 'notmuch_database_open' is deprecated: function 
> deprecated as of libnotmuch 5.4 [-Wdeprecated-declarations]
> 7 | stat = notmuch_database_open (NULL, 0, 0);
>   | ^~~~
> In file included from test1.c:2:
> /home/mohkale/prog/repos/notmuch/lib/notmuch.h:332:1: note: declared here
>   332 | notmuch_database_open (const char *path,
>   | ^
>  FAIL   create NULL path
>   --- T560-lib-error.6.EXPECTED   2022-03-22 20:25:31.093160154 +
>   +++ T560-lib-error.6.OUTPUT 2022-03-22 20:25:31.093160154 +
>   @@ -1,3 +1,3 @@
>== stdout ==
>== stderr ==
>   -Error: could not locate database.
>   +Error: Cannot open database at CWD/home/.local/share/notmuch/default: 
> No such file or directory.

I can duplicate this if I have a setting for MAILDIR in the environment.

I will send a patch clearing XDG_DATA_HOME and MAILDIR, but for now you
could add "unset MAILDIR" to test/test-vars.sh
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] test: unset XDG_DATA_HOME and MAILDIR for tests.

2022-03-25 Thread David Bremner
The some of the tests (and the library functions they exercise) that
rely on XDG_CONFIG_HOME also check XDG_DATA_HOME and MAILDIR.
---
 test/test-vars.sh | 4 
 1 file changed, 4 insertions(+)

this obsoletes the version that only unset XDG_DATA_HOME

diff --git a/test/test-vars.sh b/test/test-vars.sh
index 812bcf81..02d60f89 100644
--- a/test/test-vars.sh
+++ b/test/test-vars.sh
@@ -44,6 +44,10 @@ unset GREP_OPTIONS
 # For lib/open.cc:_load_key_file
 unset XDG_CONFIG_HOME
 
+# for lib/open.cc:_choose_database_path
+unset XDG_DATA_HOME
+unset MAILDIR
+
 # For emacsclient
 unset ALTERNATE_EDITOR
 
-- 
2.35.1

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] test: unset XDG_DATA_HOME for tests.

2022-03-25 Thread David Bremner
The some of the the tests that rely on XDG_CONFIG_HOME also check
XDG_DATA_HOME.
---
 test/test-vars.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/test/test-vars.sh b/test/test-vars.sh
index 812bcf81..cfd3905b 100644
--- a/test/test-vars.sh
+++ b/test/test-vars.sh
@@ -43,6 +43,7 @@ unset GREP_OPTIONS
 
 # For lib/open.cc:_load_key_file
 unset XDG_CONFIG_HOME
+unset XDG_DATA_HOME
 
 # For emacsclient
 unset ALTERNATE_EDITOR
-- 
2.35.1

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] emacs: add new option notmuch-search-exclude

2022-03-22 Thread David Bremner
Mohsin Kaleem  writes:

> David Bremner  writes:
>
>> It's just a guess, but do you by chance override XDG_DATA_HOME? I can
>> imagine that causing problems for the test suite.
>
> I do export an environment variable $XDG_DATA_HOME, but it points to the
> default location "/home/mohkale/.local/share" so I'm not sure why that
> would be causing this to fail.
>
> I tried running `XDG_DATA_HOME= make test` and it's still failing. I
> also tried moving my notmuch repository copy outside of any XDG*
> directories but the tests are still failing :-(.
>
> -- 
> Mohsin Kaleem

Try adding a line
"unset XDG_DATA_HOME"

after

unset XDG_CONFIG_HOME

on line 108 of test/test-lib.sh
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] emacs: add new option notmuch-search-exclude

2022-03-22 Thread David Bremner
Mohsin Kaleem  writes:

> David Bremner  writes:
>
>> Can you be more specific about what tests are failing?
>
> Pretty much all of them, although I'm beginning to suspect it may be an
> environment issue on my part.
>
> I've attached the output of running `make test` in the root of the
> project from master, do you have any idea why so much seems to be
> failing. Some python tests fail with no module named pytest but I do
> have it installed, others seem to be failing because they cannot find
> (or maybe create) a database file. I thought it might be because the
> directory for the database doesn't exist so I made it manually yet the
> tests are still failing :/.

The test suite creates many databases, so manual creation should not be needed.q

Running the tests under .cache is a bit unusual, but it seems to work ok
for me. This makes me think the problem is some environment variable. 

It's just a guess, but do you by chance override XDG_DATA_HOME? I can
imagine that causing problems for the test suite.

d


___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] emacs: add new option notmuch-search-exclude

2022-03-22 Thread David Bremner
Mohsin Kaleem  writes:

>
> I'm more than happy to add some new tests but at least for me the test
> suite appears to be failing on the master branch. I think it'd be easier
> to wait until the existing tests are fixed before adding or changing new
> ones.
>
Can you be more specific about what tests are failing? Currently I know
that the latest gmime broke smime.4

For the moment, you can use the following to skip that test

% make NOTMUCH_SKIP_TESTS=smime.4 test

(in general a space separated list of tests to skip is supported; see
test/README)

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: Failing notmuch/SMIME test

2022-03-22 Thread David Bremner
Daniel Kahn Gillmor  writes:

> I suppose the right way to fix this generically is a test which
> abstracts out whether gmime reports an angle-addr or a addr-spec for
> x.509 certs, and then adjust the tests to match.
>
> I can try to send a patch for this, but it'll take me a while to swap
> it all back in.
>
> If anyone wants to propose a patch in the meantime, i'd also be happy to
> review.
>
> The simplest thing in the short term is probably to switch the test to
> matching based on the bare e-mail address and assert a build-dep on
> gmime 3.2.8 (see attached), but that seems a little bit extreme, since
> gmime only released 3.2.9 recently (and 3.2.8 never made it out the door
> via any formal channels, if i understand the history correctly).

What do you think about accepting both with < > and without, via some
"sanitize" function? Or just marking the test broken for old gmime. I
think we do something like the latter in some places.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


release process for notmuch 0.36

2022-03-20 Thread David Bremner

I plan to feature freeze in mid April, and release 0.36 sometime before
the end of April.

d


signature.asc
Description: PGP signature
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: [PATCH] emacs: Add more front ends for address completion

2022-03-20 Thread David Bremner
Utkarsh Singh  writes:

> Thank you for working on this issue.  But from now, its up to Notmuch
> maintainers to decide whether or not they want any improvements in
> `notmuch-address.el'.

Can you be more precise about what you are asking / proposing here?
Assume I only skimmed the thread.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


[PATCH] lib: add sexp: prefix to Xapian (infix) query parser.

2022-03-19 Thread David Bremner
This is analogous to the "infix" prefix provided by the s-expression
based query parser.
---

One of the motivations for the s-expression parser is to provide some
features that the infix parser lacks. In particular the wildcard
support is more general in the s-expression parser. This patch allows
users to use those features from the s-expression parser without
converting their entire query.

 doc/man7/notmuch-search-terms.rst | 17 -
 lib/Makefile.local|  3 ++-
 lib/prefix.cc |  5 
 lib/sexp-fp.cc| 40 ++
 lib/sexp-fp.h | 41 +++
 test/T081-sexpr-search.sh | 32 
 6 files changed, 136 insertions(+), 2 deletions(-)
 create mode 100644 lib/sexp-fp.cc
 create mode 100644 lib/sexp-fp.h

diff --git a/doc/man7/notmuch-search-terms.rst 
b/doc/man7/notmuch-search-terms.rst
index f8ad1edb..4f616b7e 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -169,6 +169,12 @@ property:=
 can be present on a given message with several different values.
 See :any:`notmuch-properties(7)` for more details.
 
+sexp:
+The **sexp:** prefix allows subqueries in the format
+documented in :any:`notmuch-sexp-queries(7)`. Note that subqueries 
containing
+spaces must be quoted, and any embedded double quotes must be escaped
+(see :any:`quoting`).
+
 User defined prefixes are also supported, see :any:`notmuch-config(1)` for
 details.
 
@@ -257,7 +263,7 @@ Boolean
 Probabilistic
   **body:**, **to:**, **attachment:**, **mimetype:**
 Special
-   **from:**, **query:**, **subject:**
+   **from:**, **query:**, **subject:**, **sexp:**
 
 Terms and phrases
 -
@@ -297,6 +303,8 @@ Both of these will match a subject "Free Delicious Pizza" 
while
 
 will not.
 
+.. _quoting:
+
 Quoting
 ---
 
@@ -324,6 +332,13 @@ e.g.
% notmuch search 'folder:"/^.*/(Junk|Spam)$/"'
% notmuch search 'thread:"{from:mallory and date:2009}" and 
thread:{to:mallory}'
 
+Double quotes within query strings need to be doubled to escape them.
+
+::
+
+   % notmuch search 'tag:"""quoted tag"""'
+   % notmuch search 'sexp:"(or ""wizard"" ""php"")"'
+
 DATE AND TIME SEARCH
 
 
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 1378a74b..6d67a2a4 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -64,7 +64,8 @@ libnotmuch_cxx_srcs = \
$(dir)/prefix.cc\
$(dir)/open.cc  \
$(dir)/init.cc  \
-   $(dir)/parse-sexp.cc
+   $(dir)/parse-sexp.cc\
+   $(dir)/sexp-fp.cc
 
 libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
 
diff --git a/lib/prefix.cc b/lib/prefix.cc
index 857c05b9..06e2333a 100644
--- a/lib/prefix.cc
+++ b/lib/prefix.cc
@@ -3,6 +3,7 @@
 #include "thread-fp.h"
 #include "regexp-fields.h"
 #include "parse-time-vrp.h"
+#include "sexp-fp.h"
 
 typedef struct {
 const char *name;
@@ -60,6 +61,8 @@ prefix_t prefix_table[] = {
   NOTMUCH_FIELD_PROCESSOR },
 { "query",  NULL,   NOTMUCH_FIELD_EXTERNAL |
   NOTMUCH_FIELD_PROCESSOR },
+{ "sexp",  NULL,NOTMUCH_FIELD_EXTERNAL |
+  NOTMUCH_FIELD_PROCESSOR },
 { "from",   "XFROM",NOTMUCH_FIELD_EXTERNAL |
   NOTMUCH_FIELD_PROBABILISTIC |
   NOTMUCH_FIELD_PROCESSOR },
@@ -138,6 +141,8 @@ _setup_query_field (const prefix_t *prefix, 
notmuch_database_t *notmuch)
fp = (new QueryFieldProcessor (*notmuch->query_parser, 
notmuch))->release ();
else if (STRNCMP_LITERAL (prefix->name, "thread") == 0)
fp = (new ThreadFieldProcessor (*notmuch->query_parser, 
notmuch))->release ();
+   else if (STRNCMP_LITERAL (prefix->name, "sexp") == 0)
+   fp = (new SexpFieldProcessor (notmuch))->release ();
else
fp = (new RegexpFieldProcessor (prefix->name, prefix->flags,
*notmuch->query_parser, 
notmuch))->release ();
diff --git a/lib/sexp-fp.cc b/lib/sexp-fp.cc
new file mode 100644
index ..ed26f6ec
--- /dev/null
+++ b/lib/sexp-fp.cc
@@ -0,0 +1,40 @@
+/* sexp-fp.cc - "sexp:" field processor glue
+ *
+ * This file is part of notmuch.
+ *
+ * Copyright © 2022 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it 

  1   2   3   4   5   6   7   8   9   10   >