[PATCH 3/4] test/emacs-tree: add known broken test for subject with CR/NL

[David Bremner]

The test is intentionally vague as it's hard to pin down the correct
output before the code is fixed.
---
 test/T460-emacs-tree.sh | 9 +
 1 file changed, 9 insertions(+)

diff --git a/test/T460-emacs-tree.sh b/test/T460-emacs-tree.sh
index 9388a8ed..69a9df74 100755
--- a/test/T460-emacs-tree.sh
+++ b/test/T460-emacs-tree.sh
@@ -222,6 +222,15 @@ test_emacs '(let ((notmuch-tree-outline-enabled t))
 # folding all messages by height or depth should look the same
 test_expect_equal_file $EXPECTED/notmuch-tree-tag-inbox OUTPUT
 
+test_begin_subtest "notmuch-tree for message with subject with embedded CRNL"
+test_subtest_known_broken
+add_message "[subject]=\"=?UTF-8?B?8J+Pi++4jw==?= A SALE to boost your 
=?UTF-8?Q?workout=0D=0A?=\" [body]=the-message-body" 
+test_emacs "(notmuch-tree \"id:${gen_msg_id}\")
+   (notmuch-test-wait)
+   (test-output)"
+# one line of output, plus "End of search results."
+test_expect_equal "$(wc -l < OUTPUT)" 2
+
 add_email_corpus duplicate
 
 ID3=87r2ecrr6x@zephyr.silentflame.com
-- 
2.43.0

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Carriage returns in subject line cause problems in unthreaded and tree mode

[Tomi Ollila]

On Thu, May 16 2024, Richard Stanton wrote:

> Today I received an email with (raw) subject line
>
> Subject: =?UTF-8?B?8J+Pi++4jw==?= A SALE to boost your
>  =?UTF-8?Q?workout=0D=0A?=
>
> When displayed in Emacs in either unthreaded or tree mode, “^M” appears after 
> the word “workout”, and the displayed line is split into two, like this:
>
>  Today 07:12  Jabra️ A SALE to boost your workout^M
>(important inbox)
>
> It would probably be a good idea to check for, and remove, carriage return 
> characters in the subject line.

It depends how common it is to see those (ugly?) ^M characters there?

... and is it so that the Newline(=0A) splits the line in two.

-- and is the behavior different in some other modes (I (mostly) just use 
notmuch-search and notmuch-show modes so cannot recall how other views
look like...

The problem would be worse if carriage return (=0D) moved the cursor
to the beginning of line and following text ovewrote what were there.

Tomi

> (Note that the message displays fine in the default threaded mode.)
>
> Richard Stanton
>
> ___
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-le...@notmuchmail.org
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Carriage returns in subject line cause problems in unthreaded and tree mode

[Richard Stanton]

Today I received an email with (raw) subject line

Subject: =?UTF-8?B?8J+Pi++4jw==?= A SALE to boost your
 =?UTF-8?Q?workout=0D=0A?=

When displayed in Emacs in either unthreaded or tree mode, “^M” appears after 
the word “workout”, and the displayed line is split into two, like this:

 Today 07:12  Jabra️ A SALE to boost your workout^M
   (important inbox)

It would probably be a good idea to check for, and remove, carriage return 
characters in the subject line.

(Note that the message displays fine in the default threaded mode.)

Richard Stanton

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

[PATCH 6/6] WIP/tests: (count ...) tests for subject

[David Bremner]

---
 test/T083-sexpr-count.sh | 8 
 1 file changed, 8 insertions(+)

diff --git a/test/T083-sexpr-count.sh b/test/T083-sexpr-count.sh
index 1be3f62d..b1c0a3ac 100755
--- a/test/T083-sexpr-count.sh
+++ b/test/T083-sexpr-count.sh
@@ -141,4 +141,12 @@ test_begin_subtest "no tag is used less than 4 times"
 output=$(notmuch count --output=messages --query=sexp '(tag (count * 3))')
 test_expect_equal "${output}" "0"
 
+test_begin_subtest "subjects with unique words"
+notmuch search --query=sexp '(and (from gusarov) (subject (count 1)))' | 
notmuch_search_sanitize > OUTPUT
+cat <EXPECTED
+thread:XXX   2009-11-17 [1/1] Mikhail Gusarov; [notmuch] [PATCH] Handle rename 
of message file (inbox unread)
+thread:XXX   2009-11-17 [1/5] Mikhail Gusarov| Carl Worth, Keith Packard; 
[notmuch] [PATCH 2/2] Include  to get uint32_t in C++ file with gcc 
4.4 (inbox unread)
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_done
-- 
2.39.1

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: [PATCH] test: add regression test for Subject with newline.

[David Bremner]

David Bremner  writes:

> David Bremner  writes:
>
>> This tests the issue reported by Thibault in id:87wn9w4xus@thb.lt
>> ---
>>
>> I could not duplicate the problem here. Maybe it depends on the version of 
>> gmime?
>> I have 3.2.9 here.
>
> Now that I have gmime 3.2.13 I can confirm your bug (my previously
> posted test fails).  It seems the purely a gmime issue (the following
> reproducer is thanks to Jakub Wilk; on debian it needs the package
> gir1.2-gmime-3.0).

I think I won't go through the hassle of marking this broken for a
narrow range of gmime versions. Dropping the patch from the review queue
for now.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: [Bug] Notmuch Emacs Message Subject Can Be Truncated

[David Bremner]

Bogdan Ruslanovich Drozd  writes:

> In `notmuch-message-mode' subject can be truncated.
>
> How to reproduce:
>
> 1. Send an email message to yourself with the subject “test abcdf fdcba
>12345 54321 qwerty ytrewq 09876 67890 =- -= zxcv vcxz m,./ /.,m asdf
>fdsa ';lk kl;' qaz zaq ]'/ /'] йцукен некуцй ъхзщш шщзхъ” via Emacs
>notmuch.
>
> 2. Get the message sent to yourself and view the message in Emacs
>notmuch.  (In my case subject is truncated to “test abcdf fdcba 12345
>54321 qwerty ytrewq 09876 67890 =- -= zxcv vcxz m,./ /.,m asdf fdsa
>';lk kl;' qaz zaq ]'/ /'] йцукен некуцй ъх”, i. e. “зщш шщзхъ” part
>is lost.)
>
> If you couldn’t reproduce the bug, try eval (`M-:') `(setq
> rfc2047-encode-encoded-words nil)' and try to reproduce again.

I guess it is most likely the same bug discussed at

  https://nmbug.notmuchmail.org/nmweb/show/87wn9w4xus@thb.lt
  https://nmbug.notmuchmail.org/nmweb/show/87r0x8es8e@tethera.net
  
That one turns out to be a bug in the GMime library, fixed upstream:

https://github.com/jstedfast/gmime/commit/1a33a55baafc73b0

As far as I know that's not part of any GMime release yet. 

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

[Bug] Notmuch Emacs Message Subject Can Be Truncated

[Bogdan Ruslanovich Drozd]

In `notmuch-message-mode' subject can be truncated.

How to reproduce:

1. Send an email message to yourself with the subject “test abcdf fdcba
   12345 54321 qwerty ytrewq 09876 67890 =- -= zxcv vcxz m,./ /.,m asdf
   fdsa ';lk kl;' qaz zaq ]'/ /'] йцукен некуцй ъхзщш шщзхъ” via Emacs
   notmuch.

2. Get the message sent to yourself and view the message in Emacs
   notmuch.  (In my case subject is truncated to “test abcdf fdcba 12345
   54321 qwerty ytrewq 09876 67890 =- -= zxcv vcxz m,./ /.,m asdf fdsa
   ';lk kl;' qaz zaq ]'/ /'] йцукен некуцй ъх”, i. e. “зщш шщзхъ” part
   is lost.)

If you couldn’t reproduce the bug, try eval (`M-:') `(setq
rfc2047-encode-encoded-words nil)' and try to reproduce again.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: bug: subject line with several chunks of base64

[David Bremner]

Emmanuel Beffara  writes:

> Hello,
>
> I am stumbling upon what looks to be a bug.
>
> Sometimes I receive messages where a header entry consists of several chunks
> of base64-encoded quoted utf-8 text. For instance:
>
> Subject: 
> =?UTF-8?B?dGhpcyBpcyBqdXN0IGFuIGV4YW1wbGUgZm9yIGRlbW9uc3RyYXRpb24gcHVycA==?=  
> =?UTF-8?B?b3Nlcw==?=
>
> For such input, `notmuch search` and `notmuch show` (whatever the output
> format) consider only the first block as part of the subject, the second block
> is ignored. Apparently, this happens when the first block is padded with =
> signs as in my example, not when the base64 encoded part is full.
>

This looks like the same bug as discussed at

 https://nmbug.notmuchmail.org/nmweb/show/87wn9w4xus@thb.lt

It turns out to be a bug in the GMime library, fixed upstream:

   https://github.com/jstedfast/gmime/commit/1a33a55baafc73b0

As far as I know that's not part of any GMime release yet. 
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

bug: subject line with several chunks of base64

[Emmanuel Beffara]

Hello,

I am stumbling upon what looks to be a bug.

Sometimes I receive messages where a header entry consists of several chunks
of base64-encoded quoted utf-8 text. For instance:

Subject: 
=?UTF-8?B?dGhpcyBpcyBqdXN0IGFuIGV4YW1wbGUgZm9yIGRlbW9uc3RyYXRpb24gcHVycA==?=  
=?UTF-8?B?b3Nlcw==?=

For such input, `notmuch search` and `notmuch show` (whatever the output
format) consider only the first block as part of the subject, the second block
is ignored. Apparently, this happens when the first block is padded with =
signs as in my example, not when the base64 encoded part is full.

Note that other MUAs treat this header by concatenating the blocks and
ignoring the two spaces between them. At least, mutt does that (as well as
neomutt accessing the message through notmuch) and so do the webmails I could
test with.

This should be fixed, because truncating header lines is certainly an
important problem.

-- 
Emmanuel
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Possible bug: incorrect processing of multipart, base64-encoded subject lines.

[Jakub Wilk]

* Thibault Polge , 2022-09-21 16:15:

Subject: =?UTF-8?B?TGl2cmFpc29uIHByw6l2dWUgcG91ciBhdWpvdXJk4oCZaHU=?=
=?UTF-8?B?aTogUkVTVFJBUCBTYWRkbGUgQmFnIFNhY2NvY2hlLi4u?=

Notmuch displays the subject as only the decoded contents of the first 
fragment


I believe this is now fixed in GMime upstream:
https://github.com/jstedfast/gmime/commit/1a33a55baafc73b0

--
Jakub Wilk
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: [PATCH] test: add regression test for Subject with newline.

[David Bremner]

David Bremner  writes:

> This tests the issue reported by Thibault in id:87wn9w4xus@thb.lt
> ---
>
> I could not duplicate the problem here. Maybe it depends on the version of 
> gmime?
> I have 3.2.9 here.

Now that I have gmime 3.2.13 I can confirm your bug (my previously
posted test fails).  It seems the purely a gmime issue (the following
reproducer is thanks to Jakub Wilk; on debian it needs the package
gir1.2-gmime-3.0).

#!/usr/bin/python3

import pathlib
import tempfile

import gi
gi.require_version('GMime', '3.0')
from gi.repository import GMime
GMime.init()

msg = b'''\
Subject: =?UTF-8?B?SGVsbG8=?= =?UTF-8?B?IHdvcmxk?=

.
'''

with tempfile.NamedTemporaryFile() as tmpfile:
tmpfile.write(msg)
tmpfile.flush()
fp = GMime.StreamFile.open(tmpfile.name, 'r')
parser = GMime.Parser.new_with_stream(fp)
msg = parser.construct_message()
subject = msg.subject
expected = 'Hello world'
assert subject == expected, f'{subject!r} != {expected!r}'


___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

[PATCH] test: add regression test for Subject with newline.

[David Bremner]

This tests the issue reported by Thibault in id:87wn9w4xus@thb.lt
---

I could not duplicate the problem here. Maybe it depends on the version of 
gmime?
I have 3.2.9 here.

 test/T050-new.sh |  4 
 test/corpora/indexing/subject-newline:2, | 10 ++
 2 files changed, 14 insertions(+)
 create mode 100644 test/corpora/indexing/subject-newline:2,

diff --git a/test/T050-new.sh b/test/T050-new.sh
index cb67889c..0ac9fd7c 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -463,4 +463,8 @@ notmuch search 
id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain > EXPECT
 notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and 
ersatz > OUTPUT
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
+test_begin_subtest "base64 subject with newline"
+output=$(notmuch show id:subject-with-newl...@shopping.com | grep ^Subject:)
+test_expect_equal "${output}" \
+ "Subject: Livraison prévue pour aujourd’hui: RESTRAP Saddle 
Bag Saccoche..."
 test_done
diff --git a/test/corpora/indexing/subject-newline:2, 
b/test/corpora/indexing/subject-newline:2,
new file mode 100644
index ..bb2dde55
--- /dev/null
+++ b/test/corpora/indexing/subject-newline:2,
@@ -0,0 +1,10 @@
+From: "Thibault" 
+To: notmuch@notmuchmail.org
+Date: Wed, 18 Nov 2009 02:08:10 -0800
+Subject: =?UTF-8?B?TGl2cmFpc29uIHByw6l2dWUgcG91ciBhdWpvdXJk4oCZaHU=?=
+ =?UTF-8?B?aTogUkVTVFJBUCBTYWRkbGUgQmFnIFNhY2NvY2hlLi4u?=
+Message-ID: 
+
+The subject should be
+
+> Livraison prévue pour aujourd’hui: RESTRAP Saddle Bag Saccoche...
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: [PATCH] nmweb: escape subject in search view

[David Bremner]

David Bremner  writes:

> Fix a bug reported by Jakub Wilk [1].
>
> [1]: id:20220822064717.qftn4tr7cs4r2...@jwilk.net

applied to master

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Possible bug: incorrect processing of multipart, base64-encoded subject lines.

[Thibault Polge]

Hi all,

Notmuch doesn't parse correctly some Subject lines consisting of
multiple base64-encoded snippets.  As an example, this is the raw
subject line of an e-mail from Amazon:

Subject: =?UTF-8?B?TGl2cmFpc29uIHByw6l2dWUgcG91ciBhdWpvdXJk4oCZaHU=?=
 =?UTF-8?B?aTogUkVTVFJBUCBTYWRkbGUgQmFnIFNhY2NvY2hlLi4u?=

Notmuch displays the subject as only the decoded contents of the first
fragment (or first line), "Livraison prévue pour aujourd’hu", fully
ignoring the second base64 fragment, "i: RESTRAP Saddle Bag
Saccoche...".  The complete subject line should thus read:

> Livraison prévue pour aujourd’hui: RESTRAP Saddle Bag Saccoche...

Regards,
Thibault
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: [PATCH] nmweb: escape subject in search view

[David Bremner]

David Bremner  writes:

> Fix a bug reported by Jakub Wilk [1].
>
> [1]: id:20220822064717.qftn4tr7cs4r2...@jwilk.net
> ---
>  devel/notmuch-web/nmweb.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/devel/notmuch-web/nmweb.py b/devel/notmuch-web/nmweb.py
> index 928e4863..7b555c62 100755
> --- a/devel/notmuch-web/nmweb.py
> +++ b/devel/notmuch-web/nmweb.py
> @@ -131,7 +131,7 @@ env.globals['mailto_addrs'] = mailto_addrs
>  def link_msg(msg):
>lnk = quote_plus(msg.messageid.encode('utf8'))
>try:
> -subj = msg.header('Subject')
> +subj = html.escape(msg.header('Subject'))
>except LookupError:
>  subj = ""
>out = '%s' % (prefix, lnk, subj)
> -- 
> 2.35.2

I've deployed this patch on nmbug.notmuchmail.org. It seems to do the
right thing, at least for Jakub's original reported message.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

[PATCH] nmweb: escape subject in search view

[David Bremner]

Fix a bug reported by Jakub Wilk [1].

[1]: id:20220822064717.qftn4tr7cs4r2...@jwilk.net
---
 devel/notmuch-web/nmweb.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/devel/notmuch-web/nmweb.py b/devel/notmuch-web/nmweb.py
index 928e4863..7b555c62 100755
--- a/devel/notmuch-web/nmweb.py
+++ b/devel/notmuch-web/nmweb.py
@@ -131,7 +131,7 @@ env.globals['mailto_addrs'] = mailto_addrs
 def link_msg(msg):
   lnk = quote_plus(msg.messageid.encode('utf8'))
   try:
-subj = msg.header('Subject')
+subj = html.escape(msg.header('Subject'))
   except LookupError:
 subj = ""
   out = '%s' % (prefix, lnk, subj)
-- 
2.35.2

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

[no subject]

[David Bremner]

This series is against branch "next", which is currently ahead of
master by the following 7 commits

9c1ed5ab CLI: document handling of --config for external commands
0d33392f CLI: pass --config to external commands via NOTMUCH_CONFIG.
163dae81 test: initial tests for external commands
a5a3ed90 CLI: set NOTMUCH_CONFIG in hooks.
cd050fd7 test: add known broken tests for setting NOTMUCH_CONFIG in hooks
e7732d21 CLI: mention sexp-queries in help topics, alphabetize
383d9db2 CLI: simplify help command

These are all on the list, and will probably be applied to master
within the next week or so.

The changes since last version are not too huge (interdiff at the end).
Functional changes are as follows

- drop --version command for notmuch commit (people can use notmuch
  --version to get the same info)

- check that there are tags with the given prefix in the database
  already; if not require --force to proceed.

- enable --config handling (this is actually in the prerequisite
  patches in branch next).


diff --git a/Makefile.local b/Makefile.local
index 0fadfb26..7699c208 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -45,6 +45,15 @@ $(SHA256_FILE): $(TAR_FILE)
 $(DETACHED_SIG_FILE): $(TAR_FILE)
gpg --armor --detach-sign $^
 
+CLEAN := $(CLEAN) notmuch-git
+notmuch-git: notmuch-git.py
+   cp $< $@
+   chmod ugo+x $@
+
+CLEAN := $(CLEAN) nmbug
+nmbug: notmuch-git
+   ln -s $< $@
+
 .PHONY: dist
 dist: $(TAR_FILE)
 
@@ -294,7 +303,7 @@ endif
 SRCS  := $(SRCS) $(notmuch_client_srcs)
 CLEAN := $(CLEAN) notmuch notmuch-shared $(notmuch_client_modules)
 CLEAN := $(CLEAN) version.stamp notmuch-*.tar.gz.tmp
-CLEAN := $(CLEAN) .deps notmuch-git
+CLEAN := $(CLEAN) .deps
 
 DISTCLEAN := $(DISTCLEAN) .first-build-message Makefile.config sh.config 
sphinx.config
 
@@ -307,10 +316,6 @@ cppcheck:
@echo "No cppcheck found during configure; skipping static checking"
 endif
 
-nmbug notmuch-git: notmuch-git.in
-   sed s/@NOTMUCH_VERSION@/${VERSION}/ < notmuch-git.in > notmuch-git
-   chmod ugo+rx notmuch-git
-   ln -sf notmuch-git nmbug
 
 DEPS := $(SRCS:%.c=.deps/%.d)
 DEPS := $(DEPS:%.cc=.deps/%.d)
diff --git a/notmuch-git.in b/notmuch-git.py
old mode 100755
new mode 100644
similarity index 99%
rename from notmuch-git.in
rename to notmuch-git.py
index 6505c2e5..24ab3e5c
--- a/notmuch-git.in
+++ b/notmuch-git.py
@@ -40,8 +40,6 @@ from urllib.parse import quote as _quote
 from urllib.parse import unquote as _unquote
 import json as _json
 
-__version__ = '@NOTMUCH_VERSION@'
-
 _LOG = _logging.getLogger('nmbug')
 _LOG.setLevel(_logging.WARNING)
 _LOG.addHandler(_logging.StreamHandler())
@@ -376,6 +374,10 @@ def check_safe_fraction(status):
 safe=float(conf)
 
 total = count_messages (TAG_PREFIX)
+if total == 0:
+_LOG.error('No existing tags with given prefix, 
stopping.'.format(safe))
+_LOG.error('Use --force to override.')
+exit(1)
 change = len(status['added'])+len(status['deleted'])+len(status['missing'])
 fraction = change/total
 _LOG.debug('total messages {:d}, change: {:d}, fraction: 
{:f}'.format(total,change,fraction))
@@ -931,9 +933,6 @@ if __name__ == '__main__':
 parser.add_argument(
 '-N', '--nmbug', action='store_true',
 help='Set defaults for --tag-prefix and --git-dir for the notmuch bug 
tracker')
-parser.add_argument(
-'-v', '--version', action='version',
-version='%(prog)s {}'.format(__version__))
 parser.add_argument(
 '-l', '--log-level',
 choices=['critical', 'error', 'warning', 'info', 'debug'],
diff --git a/notmuch.c b/notmuch.c
index a6c49bcb..c75b0188 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -201,8 +201,6 @@ static const command_t commands[] = {
 { "emacs-mua", NULL, 0,
   "send mail with notmuch and emacs." },
 #endif
-{ "git", NULL, 0,
-  "manage notmuch tags with git" },
 { "help", notmuch_help_command, NOTMUCH_COMMAND_CONFIG_CREATE, /* create 
but don't save config */
   "This message, or more detailed help for the named command." }
 };
diff --git a/test/T850-git.sh b/test/T850-git.sh
index 508615e1..7ea50939 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -7,6 +7,9 @@ if [ $NOTMUCH_HAVE_SFSEXP -ne 1 ]; then
 test_done
 fi
 
+# be very careful using backup_database / restore_database in this
+# file, as they fool the cache invalidation checks in notmuch-git.
+
 add_email_corpus
 
 git config --global user.email notm...@example.org
@@ -28,6 +31,35 @@ test_expect_success "notmuch git -p '' -C tags.git clone 
remote.git"
 test_begin_subtest "initial commit needs force"
 test_expect_code 1 "notmuch git -C tags.git commit"
 
+test_begin_subtest "committing new prefix requires force"
+notmuch git -C force-prefix.git init
+notmuch tag +new-prefix::foo 
id:20091117190054.gu3...@dottiness.seas.harvard.edu
+test_expect_code 1 "notmuch git -l debug -p 'new-prefix::' -C force-prefix.git 
commit"
+notmuch tag 

Re: [PATCH 1/2] test: known broken tests for bracketed terms in subject

[Sean Whitton]

Hello,

On Sat 19 Mar 2022 at 07:38am -03, David Bremner wrote:

> David Bremner  writes:
>
>> The heuristics in the field processor currently incorrectly trigger
>> phrase parsing.
>
> I have applied this series to master

Nice, thanks!

-- 
Sean Whitton
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: [PATCH 1/2] test: known broken tests for bracketed terms in subject

[David Bremner]

David Bremner  writes:

> The heuristics in the field processor currently incorrectly trigger
> phrase parsing.

I have applied this series to master

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

[PATCH 1/2] test: known broken tests for bracketed terms in subject

[David Bremner]

The heuristics in the field processor currently incorrectly trigger
phrase parsing.
---
 test/T650-regexp-query.sh | 18 ++
 1 file changed, 18 insertions(+)

diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index 55dc6c88..4ee6b171 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -65,6 +65,24 @@ thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; - (inbox 
unread)
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "bracketed subject search (with dquotes)"
+test_subtest_known_broken
+notmuch search subject:notmuch and subject:show > EXPECTED
+notmuch search 'subject:"(show notmuch)"' > OUTPUT
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
+test_begin_subtest "bracketed subject search (with dquotes and operator 'or')"
+test_subtest_known_broken
+notmuch search subject:notmuch or subject:show > EXPECTED
+notmuch search 'subject:"(notmuch or show)"' > OUTPUT
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
+test_begin_subtest "bracketed subject search (with dquotes and operator 'and')"
+test_subtest_known_broken
+notmuch search subject:notmuch and subject:show > EXPECTED
+notmuch search 'subject:"(notmuch and show)"' > OUTPUT
+test_expect_equal_file_nonempty EXPECTED OUTPUT
+
 test_begin_subtest "xapian wildcard search for from:"
 notmuch search --output=messages 'from:cwo*' > OUTPUT
 test_expect_equal_file cworth.msg-ids OUTPUT
-- 
2.34.1

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

[no subject]

[David Bremner]

Floris wrote: 
> FWIW having spaces between the function name and parentheses is rather
>uncommon for python style.  Though of course complaining about style
>without using an auto-formatter is pretty meh these days :)
>

Yeah fair enough, it's the default in the C code, but we pretty
clearly have different styles going on in different languages.

>> +val_p = capi.lib.notmuch_config_pairs_value (super()._iter_p)
>> +key_p = capi.lib.notmuch_config_pairs_key (super()._iter_p)
>> +key = base.BinString.from_cffi(key_p)
>
>does key_p need a NULL check first or can it never be NULL?

I think it can never be NULL, but if it is, better to raise an exception I 
think.

>>  def test_iter(self, db):
>> -assert list(db.config) == []
>> -db.config['spam'] = 'ham'
>> -db.config['eggs'] = 'bacon'
>> -assert set(db.config) == {'spam', 'eggs'}
>> -assert set(db.config.keys()) == {'spam', 'eggs'}
>> -assert set(db.config.values()) == {'ham', 'bacon'}
>> -assert set(db.config.items()) == {('spam', 'ham'), ('eggs', 
>> 'bacon')}
>> +import re
>> +prefix = re.compile(r"^TEST[.]")
>> +assert [ x for x in list(db.config) if prefix.match(x) ] == []
>
>`x.startswith('TEST.')` is probably lighter weight here, maybe easier to
>read too that's subjective i guess
>
>You can even try something like this to further make it more readable:
>
>has_prefix = lambda x: x.startswith('TEST.')

I did variation on this, defining an inner function instead of using a lambda.

>
>> +db.config['TEST.spam'] = 'TEST.ham'
>> +db.config['TEST.eggs'] = 'TEST.bacon'
>> +assert { x for x in set(db.config) if prefix.match(x) } == 
>> {'TEST.spam', 'TEST.eggs'}
>> +assert { x for x in set(db.config.keys()) if prefix.match (x) } == 
>> {'TEST.spam', 'TEST.eggs'}
>
>I'm not sure why you need to wrap `db.config.keys()` in `set()`?  This
>explicitly creates a set out of things before turning it back into an
>interator while you're fine with the original iterator I think?

Good question. That seems to apply to list(db.config) above also? 

___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Variable used to detect reply prefixes in Subject

[David Bremner]

Jonas Bernoulli  writes:

> Hello,
>
> Please tell me what variable I have to customize to recognize additional
> unusual prefixes that some people add to the subject when they send a
> reply.
>
> I searched using some strings I would expect to appear in the value of
> that variable but either got no results or way too many.

There is some simple (-minded?) processing in notmuch-reply.c, which
just adds Re: to the front of the subject if it is not already present.

The original headers are also returned, so in principle the emacs front
end could do something smarter.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Variable used to detect reply prefixes in Subject

[Teemu Likonen]

* 2022-01-23 15:13:18+0100, Jonas Bernoulli wrote:

> Please tell me what variable I have to customize to recognize
> additional unusual prefixes that some people add to the subject when
> they send a reply.

There is message-subject-re-regexp. 

> Ps: Since (I assume) many notmuch users also frequent emacs-devel,
> where a frequent commentator uses the prefix "RE: [External] : ",
> maybe notmuch could even recognize that by default? (I asked him
> to consider not doing that anymore, but even if he agrees, that
> would not affect his numerous past replies.)

Yes. That is annoying. I have written a custom Subject cleaner for my
replies:


(add-hook 'message-setup-hook 'my-message-subject-clean)

(defun my-message-subject-clean ()
  ;; Delete first \( ... \) group from Subject header.
  (my-with-message-headers
(let ((case-fold-search nil))
  (dolist (re '(".*\\(\\[ ?SPAM\\(?: UNSURE\\|\\?\\)? ?\\] \\(?:Re: \\)?\\)"
".*\\(\\(?:Re: \\)* *\\[External\\][ :]*\\(?:Re: *\\)*\\)"))
(goto-char (point-min))
(when (re-search-forward (concat "^Subject: " re) nil t)
  (delete-region (match-beginning 1) (match-end 1)))

(defmacro my-with-message-headers ( body)
  (declare (indent 0))
  `(save-excursion
 (save-restriction
   (save-match-data
 (message-narrow-to-head)
 ,@body

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 6965F03973F0D4CA22B9410F0F2CAE0E07608462


signature.asc
Description: PGP signature
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Variable used to detect reply prefixes in Subject

[Jonas Bernoulli]

Hello,

Please tell me what variable I have to customize to recognize additional
unusual prefixes that some people add to the subject when they send a
reply.

I searched using some strings I would expect to appear in the value of
that variable but either got no results or way too many.

 Thanks!
 Jonas


Ps: Since (I assume) many notmuch users also frequent emacs-devel,
where a frequent commentator uses the prefix "RE: [External] : ",
maybe notmuch could even recognize that by default? (I asked him
to consider not doing that anymore, but even if he agrees, that
would not affect his numerous past replies.)
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Provide an option to make thread summaries keep initial subject

[David Bremner]

Thomas Schwinge  writes:

> Hi!
>
> Regarding the following ideas -- from almost a decade ago ;-) -- is
> anyone aware of any work in that area?

[snip]

> Austin wrote:
>> I think this would be fantastic.  I've proposed unconditionally
>> showing the earliest subject before and it seems that people who
>> correspond mostly with those who have good threading etiquette would
>> prefer this change, but those who correspond with more people who use
>> 'reply' like an address book prefer the current behavior.

So just to be clear, when viewing threads with sort-order=oldest-first,
the initial subject is in fact show. So if I understand the request
correctly, the request is to decouple which subject is shown from the
sort-order?

> And then, especially, the following one would be very useful for me:
>
>> Another option, which I'd like to experiment with but haven't found
>> the time, is to show *all* distinct subjects for matched messages in a
>> thread (modulo "Re:", etc) in the summary buffer, probably on multiple
>> lines.  Since most threads only have a single unique subject, they
>> would appear just as they do now, but it would be clear when someone
>> (or something, like git) changed the subject mid-thread.  This
>> approach would be far more robust while retaining good usability, but
>> it would require more code than just changing our subject-picking
>> heuristic.
>
> I'm aware of notmuch Emacs UI 'notmuch-tree' and 'notmuch-unthreaded',
> but these are not quite what is desired here: too verbose, compared to
> the concise display variant of 'notmuch-search'.

I don't think anyone picked up on this.  It would probably require
augmenting the json / sexpr to list all the subjects. I guess the
existing machinery to collect authors (lib/thread.cc) could be copied /
generalized to collect subjects. There is also some question of what
constitutes a change in subject, but there are already some (probably
inadequate) heuristics in the library for that. Another issue is how to
return the subjects from the library. Currently the authors are returned
as a comma separated list, but I doubt that's what you want for
subjects. C being C, we'd probably have to settle for some char ** null
terminated array of pointers.


___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Provide an option to make thread summaries keep initial subject

[Thomas Schwinge]

Hi!

Regarding the following ideas -- from almost a decade ago ;-) -- is
anyone aware of any work in that area?

On 2012-09-25T15:31:37-0400, Austin Clements  wrote:
> Quoth Olivier Berger on Sep 25 at  6:03 pm:
>> Whenever a participant changes the subject in the middle of a thread,
>> the summary reported by notmuch search will change.
>> 
>> However, the result is that some mails tend to "disappear" from search
>> results, when (bad) participants reply instead of composing a new mail,
>> and change a subject (see
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=688699 for more details
>> on that problem, and a discussion). Of course, they're still there, but
>> their subject being masked, one may then grep or C-s for a particular
>> subject and miss them.
>> 
>> I think it would be interesting to allow notmuch to potentially keep the
>> original subject and not the most recent one for the thread summaries.
>> 
>> What do you think ?
>
> I think this would be fantastic.  I've proposed unconditionally
> showing the earliest subject before and it seems that people who
> correspond mostly with those who have good threading etiquette would
> prefer this change, but those who correspond with more people who use
> 'reply' like an address book prefer the current behavior.

And then, especially, the following one would be very useful for me:

> Another option, which I'd like to experiment with but haven't found
> the time, is to show *all* distinct subjects for matched messages in a
> thread (modulo "Re:", etc) in the summary buffer, probably on multiple
> lines.  Since most threads only have a single unique subject, they
> would appear just as they do now, but it would be clear when someone
> (or something, like git) changed the subject mid-thread.  This
> approach would be far more robust while retaining good usability, but
> it would require more code than just changing our subject-picking
> heuristic.

I'm aware of notmuch Emacs UI 'notmuch-tree' and 'notmuch-unthreaded',
but these are not quite what is desired here: too verbose, compared to
the concise display variant of 'notmuch-search'.


Grüße
 Thomas
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

[no subject]

[Jon Fineman]

I was curious if anybody had any ideas of how to add unread messages
along with total messages to the hello view for each saved search?

I am aware of the count-query parameter to the notmuch-saved-searches
call that shows a count based on a search term. However when all
messages are read the folder is hidden from view. I would really like
to see both unread and total counts.

Thanks.

Jon
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

[PATCH 11/36] lib/parse-sexp: support subject field

[David Bremner]

The broken tests are because we do not yet handle phrase searches.
---
 doc/man7/notmuch-sexp-queries.rst | 62 +--
 lib/parse-sexp.cc | 19 +-
 test/T081-sexpr-search.sh | 57 
 3 files changed, 133 insertions(+), 5 deletions(-)

diff --git a/doc/man7/notmuch-sexp-queries.rst 
b/doc/man7/notmuch-sexp-queries.rst
index 0304759e..08e97cc3 100644
--- a/doc/man7/notmuch-sexp-queries.rst
+++ b/doc/man7/notmuch-sexp-queries.rst
@@ -36,9 +36,8 @@ An s-expression query is either an atom, the empty list, or a
 a *field*, *logical operation*, or *modifier*, and 0 or more
 subqueries.
 
-``*``
-``()``
-The empty list matches all messages
+``*`` ``()``
+Match all messages.
 
 *term*
 Match all messages containing *term*, possibly after
@@ -64,6 +63,59 @@ subqueries.
 FIELDS
 ``
 
+*Fields* (also called *prefixes* in notmuch documentation)
+correspond to attributes of mail messages. Some are inherent (and
+immutable) like ``subject``, while others ``tag`` and ``property`` are
+settable by the user.  Each concrete field in
+:any:`the table below `
+is discussed further under "Search prefixes" in
+:any:`notmuch-search-terms(7)`. The row *user* refers to user defined
+fields, described in :any:`notmuch-config(1)`.
+
+.. _field-table:
+
+.. table:: Fields with supported modifiers
+
+  ++---+---+---+---+--+
+  |   field|  combine  |   type|  expand   | wildcard  |  regex   |
+  ++===+===+===+===+==+
+  |   *none*   |and|   |no |yes|no|
+  ++---+---+---+---+--+
+  |   *user*   |and|  phrase   |no |yes|no|
+  ++---+---+---+---+--+
+  | attachment |and|  phrase   |yes|yes|no|
+  ++---+---+---+---+--+
+  |body|and|  phrase   |no |no |no|
+  ++---+---+---+---+--+
+  |date|   |   range   |no |no |no|
+  ++---+---+---+---+--+
+  |   folder   |or |  phrase   |yes|yes|   yes|
+  ++---+---+---+---+--+
+  |from|and|  phrase   |yes|yes|   yes|
+  ++---+---+---+---+--+
+  | id |or |   term|no |yes|   yes|
+  ++---+---+---+---+--+
+  | is |and|   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  |  lastmod   |   |   range   |no |no |no|
+  ++---+---+---+---+--+
+  |mid |or |   term|no |yes|   yes|
+  ++---+---+---+---+--+
+  |  mimetype  |or |  phrase   |yes|yes|no|
+  ++---+---+---+---+--+
+  |path|or |   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  |  property  |and|   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  |  subject   |and|  phrase   |yes|yes|   yes|
+  ++---+---+---+---+--+
+  |tag |and|   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  |   thread   |or |   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  | to |and|  phrase   |yes|yes|no|
+  ++---+---+---+---+--+
+
 .. _modifiers:
 
 MODIFIERS
@@ -86,6 +138,10 @@ EXAMPLES
 ``(not Bob Marley)``
 Match messages containing neither "Bob" nor "Marley", nor their stems,
 
+``(subject quick "brown fox")``
+Match messages whose subject contains "quick" (anywhere, stemmed) and
+the phrase "brown fox".
+
 .. |q1| replace:: :math:`q_1`
 .. |q2| replace:: :math:`q_2`
 .. |qn| replace:: :math:`q_n`
diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 0d2c0ba8..25556058 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -8,7 +8,8 @@
  * defi

[PATCH 10/31] lib/parse-sexp: support subject field

[David Bremner]

The broken tests are because we do not yet handle phrase searches.
---
 doc/man7/notmuch-sexp-queries.rst | 62 +--
 lib/parse-sexp.cc | 19 +-
 test/T081-sexpr-search.sh | 57 
 3 files changed, 133 insertions(+), 5 deletions(-)

diff --git a/doc/man7/notmuch-sexp-queries.rst 
b/doc/man7/notmuch-sexp-queries.rst
index fef6beaf..ae1619dc 100644
--- a/doc/man7/notmuch-sexp-queries.rst
+++ b/doc/man7/notmuch-sexp-queries.rst
@@ -36,9 +36,8 @@ An s-expression query is either an atom, the empty list, or a
 a *field*, *logical operation*, or *modifier*, and 0 or more
 subqueries.
 
-``*``
-``()``
-The empty list matches all messages
+``*`` ``()``
+Match all messages.
 
 *term*
 Match all messages containing *term*, possibly after
@@ -64,6 +63,59 @@ subqueries.
 FIELDS
 ``
 
+*Fields* (also called *prefixes* in notmuch documentation)
+correspond to attributes of mail messages. Some are inherent (and
+immutable) like ``subject``, while others ``tag`` and ``property`` are
+settable by the user.  Each concrete field in
+:any:`the table below `
+is discussed further under "Search prefixes" in
+:any:`notmuch-search-terms(7)`. The row *user* refers to user defined
+fields, described in :any:`notmuch-config(1)`.
+
+.. _field-table:
+
+.. table:: Fields with supported modifiers
+
+  ++---+---+---+---+--+
+  |   field|  combine  |   type|  expand   | wildcard  |  regex   |
+  ++===+===+===+===+==+
+  |   *none*   |and|   |no |yes|no|
+  ++---+---+---+---+--+
+  |   *user*   |and|  phrase   |no |yes|no|
+  ++---+---+---+---+--+
+  | attachment |and|  phrase   |yes|yes|no|
+  ++---+---+---+---+--+
+  |body|and|  phrase   |no |no |no|
+  ++---+---+---+---+--+
+  |date|   |   range   |no |no |no|
+  ++---+---+---+---+--+
+  |   folder   |or |  phrase   |yes|yes|   yes|
+  ++---+---+---+---+--+
+  |from|and|  phrase   |yes|yes|   yes|
+  ++---+---+---+---+--+
+  | id |or |   term|no |yes|   yes|
+  ++---+---+---+---+--+
+  | is |and|   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  |  lastmod   |   |   range   |no |no |no|
+  ++---+---+---+---+--+
+  |mid |or |   term|no |yes|   yes|
+  ++---+---+---+---+--+
+  |  mimetype  |or |  phrase   |yes|yes|no|
+  ++---+---+---+---+--+
+  |path|or |   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  |  property  |and|   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  |  subject   |and|  phrase   |yes|yes|   yes|
+  ++---+---+---+---+--+
+  |tag |and|   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  |   thread   |or |   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  | to |and|  phrase   |yes|yes|no|
+  ++---+---+---+---+--+
+
 .. _modifiers:
 
 MODIFIERS
@@ -86,6 +138,10 @@ EXAMPLES
 ``(not Bob Marley)``
 Match messages containing neither "Bob" nor "Marley", nor their stems,
 
+``(subject quick "brown fox")``
+Match messages whose subject contains "quick" (anywhere, stemmed) and
+the phrase "brown fox".
+
 .. |q1| replace:: :math:`q_1`
 .. |q2| replace:: :math:`q_2`
 .. |qn| replace:: :math:`q_n`
diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 97bfecbd..a5d41f30 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -7,7 +7,8 @@
  * defi

[PATCH 10/27] lib/parse-sexp: support subject field

[David Bremner]

The broken tests are because we do not yet handle phrase searches.
---
 doc/man7/notmuch-sexp-queries.rst | 57 +++
 lib/parse-sexp.cc | 19 +--
 test/T081-sexpr-search.sh | 56 ++
 3 files changed, 130 insertions(+), 2 deletions(-)

diff --git a/doc/man7/notmuch-sexp-queries.rst 
b/doc/man7/notmuch-sexp-queries.rst
index 1594d147..5427bb0a 100644
--- a/doc/man7/notmuch-sexp-queries.rst
+++ b/doc/man7/notmuch-sexp-queries.rst
@@ -64,6 +64,59 @@ subqueries.
 FIELDS
 ``
 
+*Fields* (also called *prefixes* in notmuch documentation)
+correspond to attributes of mail messages. Some are inherent (and
+immutable) like ``subject``, while others ``tag`` and ``property`` are
+settable by the user.  Each concrete field in
+:any:`the table below `
+is discussed further under "Search prefixes" in
+:any:`notmuch-search-terms(7)`. The row *user* refers to user defined
+fields, described in :any:`notmuch-config(1)`.
+
+.. _field-table:
+
+.. table:: Fields with supported modifiers
+
+  ++---+---+---+---+--+
+  |   field|  combine  |   type|  expand   | wildcard  |  regex   |
+  ++===+===+===+===+==+
+  |   *none*   |and|   |no |yes|no|
+  ++---+---+---+---+--+
+  |   *user*   |and|  phrase   |no |yes|no|
+  ++---+---+---+---+--+
+  | attachment |and|  phrase   |yes|yes|no|
+  ++---+---+---+---+--+
+  |body|and|  phrase   |no |no |no|
+  ++---+---+---+---+--+
+  |date|   |   range   |no |no |no|
+  ++---+---+---+---+--+
+  |   folder   |or |  phrase   |yes|yes|   yes|
+  ++---+---+---+---+--+
+  |from|and|  phrase   |yes|yes|   yes|
+  ++---+---+---+---+--+
+  | id |or |   term|no |yes|   yes|
+  ++---+---+---+---+--+
+  | is |and|   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  |  lastmod   |   |   range   |no |no |no|
+  ++---+---+---+---+--+
+  |mid |or |   term|no |yes|   yes|
+  ++---+---+---+---+--+
+  |  mimetype  |or |  phrase   |yes|yes|no|
+  ++---+---+---+---+--+
+  |path|or |   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  |  property  |and|   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  |  subject   |and|  phrase   |yes|yes|   yes|
+  ++---+---+---+---+--+
+  |tag |and|   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  |   thread   |or |   term|yes|yes|   yes|
+  ++---+---+---+---+--+
+  | to |and|  phrase   |yes|yes|no|
+  ++---+---+---+---+--+
+
 .. _modifiers:
 
 MODIFIERS
@@ -86,6 +139,10 @@ EXAMPLES
 ``(not Bob Marley)``
 Match messages containing neither "Bob" nor "Marley", nor their stems,
 
+``(subject quick "brown fox")``
+Match messages whose subject contains "quick" (anywhere, stemmed) and
+the phrase "brown fox".
+
 .. |q1| replace:: :math:`q_1`
 .. |q2| replace:: :math:`q_2`
 .. |qn| replace:: :math:`q_n`
diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 97bfecbd..a5d41f30 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -7,7 +7,8 @@
  * definitions from sexp.h */
 
 typedef enum {
-SEXP_FLAG_NONE = 0,
+SEXP_FLAG_NONE = 0,
+SEXP_FLAG_FIELD= 1 << 0,
 } _sexp_flag_t;
 
 typedef struct  {
@@ -25,6 +26,8 @@ static _sexp_prefix_t prefixes[] =
   SEXP_FLAG_NONE },
 { "or", Xapian::Query::OP_OR, 

[PATCH 07/25] lib/parse-sexp: parse 'subject'

[David Bremner]

Initially this only works for lists of individual terms, which is a
bit inconvenient. This will be improved in a following commit.
---
 lib/parse-sexp.cc | 37 +
 test/T081-sexpr-search.sh | 15 +++
 2 files changed, 52 insertions(+)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index dfbebf2b..898cfdd0 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -16,6 +16,17 @@ static _sexp_op_t operations[] =
 { }
 };
 
+typedef struct  {
+const char *name;
+Xapian::Query::op xapian_op;
+} _sexp_field_t;
+
+static _sexp_field_t fields[] =
+{
+{ "subject",Xapian::Query::OP_PHRASE },
+{ }
+};
+
 static notmuch_status_t _sexp_to_xapian_query (notmuch_database_t *notmuch, 
const sexp_t *sx,
   Xapian::Query );
 
@@ -61,6 +72,27 @@ _notmuch_sexp_string_to_xapian_query (notmuch_database_t 
*notmuch, const char *q
 return _sexp_to_xapian_query (notmuch, sx, output);
 }
 
+static notmuch_status_t
+_sexp_combine_field (const char *prefix,
+Xapian::Query::op operation,
+const sexp_t *sx,
+Xapian::Query )
+{
+std::vector terms;
+
+for (const sexp_t *cur = sx; cur; cur = cur->next) {
+   std::string pref_str = prefix;
+   std::string word = cur->val;
+
+   if (operation == Xapian::Query::OP_PHRASE)
+   word = Xapian::Unicode::tolower (word);
+
+   terms.push_back (pref_str + word);
+}
+output = Xapian::Query (operation, terms.begin (), terms.end ());
+return NOTMUCH_STATUS_SUCCESS;
+}
+
 /* Here we expect the s-expression to be a proper list, with first
  * element defining and operation, or as a special case the empty
  * list */
@@ -84,7 +116,12 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const 
sexp_t *sx, Xapian::Qu
if (strcasecmp (op->name, hd_sexp (sx)->val) == 0) {
return _sexp_combine_query (notmuch, op->xapian_op, op->initial, 
sx->list->next, output);
}
+}
 
+for (const _sexp_field_t *field = fields; field && field->name; field++) {
+   if (strcasecmp (field->name, hd_sexp (sx)->val) == 0)
+   return _sexp_combine_field (_find_prefix (field->name), 
field->xapian_op, sx->list->next,
+   output);
 }
 
 _notmuch_database_log_append (notmuch, "unimplemented prefix %s\n", 
sx->list->val);
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 41a82886..872f2603 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -19,6 +19,21 @@ for query in '(or)' '(not ())' '(not (not))' '(not (and))' \
 test_expect_equal_file /dev/null OUTPUT
 done
 
+test_begin_subtest "Search by subject"
+add_message [subject]=subjectsearchtest '[date]="Sat, 01 Jan 2000 12:00:00 
-"'
+output=$(notmuch search --query-syntax=sexp '(subject subjectsearchtest)' | 
notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; 
subjectsearchtest (inbox unread)"
+
+test_begin_subtest "Search by subject (case insensitive)"
+notmuch search tag:inbox and subject:maildir | notmuch_search_sanitize > 
EXPECTED
+notmuch search --query-syntax=sexp '(subject Maildir)' | 
notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by subject (utf-8):"
+add_message [subject]=utf8-sübjéct '[date]="Sat, 01 Jan 2000 12:00:00 -"'
+output=$(notmuch search --query-syntax=sexp '(subject utf8 sübjéct)' | 
notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; 
utf8-sübjéct (inbox unread)"
+
 test_begin_subtest "Unbalanced parens"
 # A code 1 indicates the error was handled (a crash will return e.g. 139).
 test_expect_code 1 "notmuch search --query-syntax=sexp '('"
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

[PATCH 06/11] lib/parse-sexp: parse 'subject'

[David Bremner]

Initially this only works for lists of individual terms, which is a
bit inconvenient. This will be improved in a following commit.
---
 lib/parse-sexp.cc | 36 
 test/T081-sexpr-search.sh | 15 +++
 2 files changed, 51 insertions(+)

diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index eded98e7..4a2fac8b 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -16,6 +16,17 @@ static _sexp_op_t operations[] =
 { }
 };
 
+typedef struct  {
+const char *name;
+Xapian::Query::op xapian_op;
+} _sexp_field_t;
+
+static _sexp_field_t fields[] =
+{
+{ "subject",Xapian::Query::OP_PHRASE },
+{ }
+};
+
 static Xapian::Query _sexp_to_xapian_query (sexp_t *sx);
 
 static Xapian::Query
@@ -46,6 +57,26 @@ _notmuch_sexp_string_to_xapian_query (notmuch_database_t 
*notmuch, const char *q
 return _sexp_to_xapian_query (sx);
 }
 
+static Xapian::Query
+_sexp_combine_field (const char *prefix,
+Xapian::Query::op operation,
+sexp_t *sx)
+{
+std::vector terms;
+
+for (sexp_t *cur = sx; cur; cur = cur->next) {
+   std::string pref_str = prefix;
+   std::string word = cur->val;
+
+   if (operation == Xapian::Query::OP_PHRASE)
+   word = Xapian::Unicode::tolower (word);
+
+
+   terms.push_back (pref_str + word);
+}
+return Xapian::Query (operation, terms.begin (), terms.end ());
+}
+
 /* Here we expect the s-expression to be a proper list, with first
  * element defining and operation, or as a special case the empty
  * list */
@@ -68,5 +99,10 @@ _sexp_to_xapian_query (sexp_t *sx)
return _sexp_combine_query (op->xapian_op, op->initial, 
sx->list->next);
 }
 
+for (const _sexp_field_t *field = fields; field && field->name; field++) {
+   if (strcasecmp (field->name, hd_sexp (sx)->val) == 0)
+   return _sexp_combine_field (_find_prefix (field->name), 
field->xapian_op, sx->list->next);
+}
+
 INTERNAL_ERROR ("unimplemented prefix %s\n", sx->list->val);
 }
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 0b75334a..1a80a133 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -19,4 +19,19 @@ for query in '(or)' '(not ())' '(not (not))' '(not (and))' \
 test_expect_equal_file /dev/null OUTPUT
 done
 
+test_begin_subtest "Search by subject"
+add_message [subject]=subjectsearchtest '[date]="Sat, 01 Jan 2000 12:00:00 
-"'
+output=$(notmuch search --query-syntax=sexp '(subject subjectsearchtest)' | 
notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; 
subjectsearchtest (inbox unread)"
+
+test_begin_subtest "Search by subject (case insensitive)"
+notmuch search tag:inbox and subject:maildir | notmuch_search_sanitize > 
EXPECTED
+notmuch search --query-syntax=sexp '(subject Maildir)' | 
notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "Search by subject (utf-8):"
+add_message [subject]=utf8-sübjéct '[date]="Sat, 01 Jan 2000 12:00:00 -"'
+output=$(notmuch search --query-syntax=sexp '(subject utf8 sübjéct)' | 
notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; 
utf8-sübjéct (inbox unread)"
+
 test_done
-- 
2.30.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

[PATCH 5/8] test: add regression tests for n_t_get_{authors, subject}

[David Bremner]

This is returning explicitely cached data, so no database access is needed.
---
 test/T568-lib-thread.sh | 33 +
 1 file changed, 33 insertions(+)

diff --git a/test/T568-lib-thread.sh b/test/T568-lib-thread.sh
index c7d4f26b..0f9fa443 100755
--- a/test/T568-lib-thread.sh
+++ b/test/T568-lib-thread.sh
@@ -198,4 +198,37 @@ yunaayketfm@aiko.keithp.com
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest "get authors from closed database"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+const char *authors;
+authors = notmuch_thread_get_authors (thread);
+printf("%d\n%s\n", thread != NULL, authors);
+}
+EOF
+cat < EXPECTED
+== stdout ==
+1
+Lars Kellogg-Stedman, Mikhail Gusarov, Keith Packard, Carl Worth
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "get subject from closed database"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+const char *subject;
+subject = notmuch_thread_get_subject (thread);
+    printf("%d\n%s\n", thread != NULL, subject);
+}
+EOF
+cat < EXPECTED
+== stdout ==
+1
+[notmuch] Working with Maildir storage?
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+
 test_done
-- 
2.27.0
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

[no subject]

[David Bremner]

Here's a not too ambitious attempt to clean up the error handling on
zlib output. It's bit gross to treat any error reported by zlib as
fatal, but it's a step up from ignoring them, and it's in the client
code, not in the library.

The first patch splits out the first fix of
id:20200410173039.yextmxk4wtgpl...@siegel.lan

The last patch is a fix for the bug reported by qsx.


___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 1/3] test: add known broken tests for from: and subject:

[Tomi Ollila]

On Wed, Aug 21 2019, David Bremner wrote:

> Given we want 'a b' to parse as 'a AND b', then for any
> probabilistic (free text) prefix foo:, we should also get 'foo:a
> foo:b' expanding to 'foo:a AND foo:b'. Currently this is not true due
> to the implimentation of regex fields.

implementation =D

> ---
>  test/T760-implicit-operators.sh | 28 
>  1 file changed, 28 insertions(+)
>  create mode 100755 test/T760-implicit-operators.sh
>
> diff --git a/test/T760-implicit-operators.sh b/test/T760-implicit-operators.sh
> new file mode 100755
> index ..b79673df
> --- /dev/null
> +++ b/test/T760-implicit-operators.sh
> @@ -0,0 +1,28 @@
> +#!/usr/bin/env bash
> +test_description='implicit operators in query parser'
> +. $(dirname "$0")/test-lib.sh || exit 1
> +
> +test_AND() {
> +add_message  "[$1]=a@b"
> +add_message  "[$1]=b@c"
> +
> +test_begin_subtest "$1: implicitly joined by AND"
> +$2
> +notmuch count $1:a@b > OUTPUT
> +notmuch count $1:a $1:b >> OUTPUT
> +notmuch count $1:a@b OR $1:b@c >> OUTPUT
> +notmuch count $1:a@b $1:b@c >> OUTPUT
> +cat < EXPECTED
> +1
> +1
> +2
> +0
> +EOF

the above could be done  printf %s\\n  1  1  2  0  > EXPECTED

(whichever way is "clearer" -- using '%s\n' or even "%s\n" is 
also possible (just increasingly harder to write ;D))

Tomi

> +test_expect_equal_file EXPECTED OUTPUT
> +}
> +
> +test_AND from test_subtest_known_broken
> +test_AND subject test_subtest_known_broken
> +test_AND to
> +
> +test_done
> -- 
> 2.23.0.rc1
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/3] test: add known broken tests for from: and subject:

[David Bremner]

Given we want 'a b' to parse as 'a AND b', then for any
probabilistic (free text) prefix foo:, we should also get 'foo:a
foo:b' expanding to 'foo:a AND foo:b'. Currently this is not true due
to the implimentation of regex fields.
---
 test/T760-implicit-operators.sh | 28 
 1 file changed, 28 insertions(+)
 create mode 100755 test/T760-implicit-operators.sh

diff --git a/test/T760-implicit-operators.sh b/test/T760-implicit-operators.sh
new file mode 100755
index ..b79673df
--- /dev/null
+++ b/test/T760-implicit-operators.sh
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+test_description='implicit operators in query parser'
+. $(dirname "$0")/test-lib.sh || exit 1
+
+test_AND() {
+add_message  "[$1]=a@b"
+add_message  "[$1]=b@c"
+
+test_begin_subtest "$1: implicitly joined by AND"
+$2
+notmuch count $1:a@b > OUTPUT
+notmuch count $1:a $1:b >> OUTPUT
+notmuch count $1:a@b OR $1:b@c >> OUTPUT
+notmuch count $1:a@b $1:b@c >> OUTPUT
+cat < EXPECTED
+1
+1
+2
+0
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+}
+
+test_AND from test_subtest_known_broken
+test_AND subject test_subtest_known_broken
+test_AND to
+
+test_done
-- 
2.23.0.rc1

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v4 10/17] indexing: record protected subject when indexing cleartext

[Daniel Kahn Gillmor]

When indexing the cleartext of an encrypted message, record any
protected subject in the database, which should make it findable and
visible in search.

Signed-off-by: Daniel Kahn Gillmor 
---
 lib/index.cc   | 42 ++
 lib/message.cc |  8 +++
 lib/notmuch-private.h  |  4 
 test/T356-protected-headers.sh | 16 +
 4 files changed, 61 insertions(+), 9 deletions(-)

diff --git a/lib/index.cc b/lib/index.cc
index f216ae5d..1fd9e67e 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -367,13 +367,15 @@ _index_content_type (notmuch_message_t *message, 
GMimeObject *part)
 
 static void
 _index_encrypted_mime_part (notmuch_message_t *message, notmuch_indexopts_t 
*indexopts,
-   GMimeMultipartEncrypted *part);
+   GMimeMultipartEncrypted *part,
+   _notmuch_message_crypto_t *msg_crypto);
 
 /* Callback to generate terms for each mime part of a message. */
 static void
 _index_mime_part (notmuch_message_t *message,
  notmuch_indexopts_t *indexopts,
- GMimeObject *part)
+ GMimeObject *part,
+ _notmuch_message_crypto_t *msg_crypto)
 {
 GMimeStream *stream, *filter;
 GMimeFilter *discard_non_term_filter;
@@ -403,6 +405,8 @@ _index_mime_part (notmuch_message_t *message,
  _notmuch_message_add_term (message, "tag", "encrypted");
 
for (i = 0; i < g_mime_multipart_get_count (multipart); i++) {
+   notmuch_status_t status;
+   GMimeObject *child;
if (GMIME_IS_MULTIPART_SIGNED (multipart)) {
/* Don't index the signature, but index its content type. */
if (i == GMIME_MULTIPART_SIGNED_SIGNATURE) {
@@ -419,7 +423,8 @@ _index_mime_part (notmuch_message_t *message,
 g_mime_multipart_get_part (multipart, i));
if (i == GMIME_MULTIPART_ENCRYPTED_CONTENT) {
_index_encrypted_mime_part(message, indexopts,
-  GMIME_MULTIPART_ENCRYPTED 
(part));
+  GMIME_MULTIPART_ENCRYPTED (part),
+  msg_crypto);
} else {
if (i != GMIME_MULTIPART_ENCRYPTED_VERSION) {
_notmuch_database_log (notmuch_message_get_database 
(message),
@@ -428,8 +433,13 @@ _index_mime_part (notmuch_message_t *message,
}
continue;
}
-   _index_mime_part (message, indexopts,
- g_mime_multipart_get_part (multipart, i));
+   child = g_mime_multipart_get_part (multipart, i);
+   status = _notmuch_message_crypto_potential_payload (msg_crypto, 
child, part, i);
+   if (status)
+   _notmuch_database_log (notmuch_message_get_database (message),
+  "Warning: failed to mark the potential 
cryptographic payload (%s).\n",
+  notmuch_status_to_string (status));
+   _index_mime_part (message, indexopts, child, msg_crypto);
}
return;
 }
@@ -439,7 +449,7 @@ _index_mime_part (notmuch_message_t *message,
 
mime_message = g_mime_message_part_get_message (GMIME_MESSAGE_PART 
(part));
 
-   _index_mime_part (message, indexopts, g_mime_message_get_mime_part 
(mime_message));
+   _index_mime_part (message, indexopts, g_mime_message_get_mime_part 
(mime_message), msg_crypto);
 
return;
 }
@@ -516,7 +526,8 @@ _index_mime_part (notmuch_message_t *message,
 static void
 _index_encrypted_mime_part (notmuch_message_t *message,
notmuch_indexopts_t *indexopts,
-   GMimeMultipartEncrypted *encrypted_data)
+   GMimeMultipartEncrypted *encrypted_data,
+   _notmuch_message_crypto_t *msg_crypto)
 {
 notmuch_status_t status;
 GError *err = NULL;
@@ -553,6 +564,10 @@ _index_encrypted_mime_part (notmuch_message_t *message,
return;
 }
 if (decrypt_result) {
+   status = _notmuch_message_crypto_successful_decryption (msg_crypto);
+   if (status)
+   _notmuch_database_log_append (notmuch, "failed to mark the message 
as decrypted (%s)\n",
+ notmuch_status_to_string (status));
if (get_sk) {
status = notmuch_message_add_property (message, "session-key",
   
g_mime_decrypt_result_get_session_key (decrypt_result));
@@ -562,7 +577,8 @@ _index_encrypted_mime_part (notmuch_message_t *message,
}
g_object_unref (decrypt_result);
 }
-_index_mime_part (message, indexopts, clear);
+status = _notmuc

Re: [PATCH v3 10/17] indexing: record protected subject when indexing cleartext

[Daniel Kahn Gillmor]

On Mon 2019-05-27 17:17:20 -0400, Daniel Kahn Gillmor wrote:
> When indexing the cleartext of an encrypted message, record any
> protected subject in the database, which should make it findable and
> visible in search.

ugh, please ignore v3 of this patch (10/17) as well.  v4 should be
coming shortly.

   --dkg, juggling too many branches


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

_notmuch_database_log vs _notmuch_database_log_append [was: Re: [PATCH v2 10/17] indexing: record protected subject when indexing cleartext]

[Daniel Kahn Gillmor]

On Mon 2019-05-27 07:24:41 -0300, David Bremner wrote:
> Daniel Kahn Gillmor  writes:
>
>> +status = _notmuch_message_crypto_potential_payload (msg_crypto, clear, 
>> GMIME_OBJECT (encrypted_data), GMIME_MULTIPART_ENCRYPTED_CONTENT);
>> +_index_mime_part (message, indexopts, clear, msg_crypto);
>>  g_object_unref (clear);
>
> If you're going to ignore the return value here (not sure if that's a
> good idea?)  please explicitly cast to void rather than putting in
> status to ignore.

Good catch, thanks.  I've logged the error with
_notmuch_database_log_append in v3 of this patch, i hope that makes
sense to you!

i note that _index_encrypted_mime_part() itself uses an odd mixture of
_notmuch_database_log_append and _notmuch_database_log, which maybe is a
sign that more cleanup is due there.

I confess i always get a bit confused about when to use one vs. the
other, though.  _log resets the database's status_string, while
_log_append just appends to it.

On IRC, you wrote "I'd say it makes sense to append for warnings", which
is a plausible rule of thumb, but seems like it might not map to the
intent of how we expect the status_string to be used -- is it for the
caller of the library?  or something else?

For example, should every call into the library reset the status string,
and then there would only be one _database_log() function?  (i don't
know whether that's feasible, or what it would mean for internal code
that already calls exported functions directly).

It'd probably be worthwhile for someone to do an audit of those uses and
come up with some normalized way of handling this that we can clearly
explain, because i think it's a bit unwieldy now.  I'm writing this
report with the intent of tagging this e-mail with notmuch::bug, in the
hopes that someone interested in doing maintenance work will take this
on as a future project :)

   --dkg


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 10/17] indexing: record protected subject when indexing cleartext

[Daniel Kahn Gillmor]

When indexing the cleartext of an encrypted message, record any
protected subject in the database, which should make it findable and
visible in search.

Signed-off-by: Daniel Kahn Gillmor 
---
 lib/index.cc   | 46 +++---
 lib/message.cc |  8 ++
 lib/notmuch-private.h  |  4 +++
 test/T356-protected-headers.sh | 16 
 4 files changed, 65 insertions(+), 9 deletions(-)

diff --git a/lib/index.cc b/lib/index.cc
index f216ae5d..2e0bacb1 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -367,13 +367,15 @@ _index_content_type (notmuch_message_t *message, 
GMimeObject *part)
 
 static void
 _index_encrypted_mime_part (notmuch_message_t *message, notmuch_indexopts_t 
*indexopts,
-   GMimeMultipartEncrypted *part);
+   GMimeMultipartEncrypted *part,
+   _notmuch_message_crypto_t *msg_crypto);
 
 /* Callback to generate terms for each mime part of a message. */
 static void
 _index_mime_part (notmuch_message_t *message,
  notmuch_indexopts_t *indexopts,
- GMimeObject *part)
+ GMimeObject *part,
+ _notmuch_message_crypto_t *msg_crypto)
 {
 GMimeStream *stream, *filter;
 GMimeFilter *discard_non_term_filter;
@@ -403,6 +405,8 @@ _index_mime_part (notmuch_message_t *message,
  _notmuch_message_add_term (message, "tag", "encrypted");
 
for (i = 0; i < g_mime_multipart_get_count (multipart); i++) {
+   notmuch_status_t status;
+   GMimeObject *child;
if (GMIME_IS_MULTIPART_SIGNED (multipart)) {
/* Don't index the signature, but index its content type. */
if (i == GMIME_MULTIPART_SIGNED_SIGNATURE) {
@@ -419,7 +423,8 @@ _index_mime_part (notmuch_message_t *message,
 g_mime_multipart_get_part (multipart, i));
if (i == GMIME_MULTIPART_ENCRYPTED_CONTENT) {
_index_encrypted_mime_part(message, indexopts,
-  GMIME_MULTIPART_ENCRYPTED 
(part));
+  GMIME_MULTIPART_ENCRYPTED (part),
+  msg_crypto);
} else {
if (i != GMIME_MULTIPART_ENCRYPTED_VERSION) {
_notmuch_database_log (notmuch_message_get_database 
(message),
@@ -428,8 +433,13 @@ _index_mime_part (notmuch_message_t *message,
}
continue;
}
-   _index_mime_part (message, indexopts,
- g_mime_multipart_get_part (multipart, i));
+   child = g_mime_multipart_get_part (multipart, i);
+   status = _notmuch_message_crypto_potential_payload (msg_crypto, 
child, part, i);
+   if (status)
+   _notmuch_database_log (notmuch_message_get_database (message),
+  "Warning: failed to mark the potential 
cryptographic payload (%s).\n",
+  notmuch_status_to_string (status));
+   _index_mime_part (message, indexopts, child, msg_crypto);
}
return;
 }
@@ -439,7 +449,7 @@ _index_mime_part (notmuch_message_t *message,
 
mime_message = g_mime_message_part_get_message (GMIME_MESSAGE_PART 
(part));
 
-   _index_mime_part (message, indexopts, g_mime_message_get_mime_part 
(mime_message));
+   _index_mime_part (message, indexopts, g_mime_message_get_mime_part 
(mime_message), msg_crypto);
 
return;
 }
@@ -516,7 +526,8 @@ _index_mime_part (notmuch_message_t *message,
 static void
 _index_encrypted_mime_part (notmuch_message_t *message,
notmuch_indexopts_t *indexopts,
-   GMimeMultipartEncrypted *encrypted_data)
+   GMimeMultipartEncrypted *encrypted_data,
+   _notmuch_message_crypto_t *msg_crypto)
 {
 notmuch_status_t status;
 GError *err = NULL;
@@ -553,6 +564,10 @@ _index_encrypted_mime_part (notmuch_message_t *message,
return;
 }
 if (decrypt_result) {
+   status = _notmuch_message_crypto_successful_decryption (msg_crypto);
+   if (status)
+   _notmuch_database_log_append (notmuch, "failed to mark the message 
as decrypted (%s)\n",
+ notmuch_status_to_string (status));
if (get_sk) {
status = notmuch_message_add_property (message, "session-key",
   
g_mime_decrypt_result_get_session_key (decrypt_result));
@@ -562,7 +577,12 @@ _index_encrypted_mime_part (notmuch_message_t *message,
}
g_object_unref (decrypt_result);
 }
-_index_mime_part (message, indexopts, clear);
+status = _notmuc

Re: [PATCH v2 10/17] indexing: record protected subject when indexing cleartext

[David Bremner]

Daniel Kahn Gillmor  writes:

> +status = _notmuch_message_crypto_potential_payload (msg_crypto, clear, 
> GMIME_OBJECT (encrypted_data), GMIME_MULTIPART_ENCRYPTED_CONTENT);
> +_index_mime_part (message, indexopts, clear, msg_crypto);
>  g_object_unref (clear);

If you're going to ignore the return value here (not sure if that's a
good idea?)  please explicitly cast to void rather than putting in
status to ignore.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2 17/17] cli/reply: pull proposed subject line from the message, not the index

[Daniel Kahn Gillmor]

Protected subject lines were being emitted in reply when the cleartext
of documents was indexed.  create_reply_message() was pulling the
subject line from the index, rather than pulling it from the
GMimeMessage object that it already has on hand.

This one-line fix to notmuch-reply.c solves that problem, and doesn't
cause any additional tests to fail.

Signed-off-by: Daniel Kahn Gillmor 
---
 notmuch-reply.c  | 2 +-
 test/T356-protected-headers.sh   | 1 -
 test/T358-emacs-protected-headers.sh | 1 -
 3 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/notmuch-reply.c b/notmuch-reply.c
index 2689b247..46bab434 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -591,7 +591,7 @@ create_reply_message(void *ctx,
 from_addr);
 g_mime_object_set_header (GMIME_OBJECT (reply), "From", from_addr, NULL);
 
-    subject = notmuch_message_get_header (message, "subject");
+subject = g_mime_message_get_subject (mime_message);
 if (subject) {
    if (strncasecmp (subject, "Re:", 3))
subject = talloc_asprintf (ctx, "Re: %s", subject);
diff --git a/test/T356-protected-headers.sh b/test/T356-protected-headers.sh
index 746c4760..4af018f3 100755
--- a/test/T356-protected-headers.sh
+++ b/test/T356-protected-headers.sh
@@ -100,7 +100,6 @@ test_json_nodes <<<"$output" \
 'subject:[0]["subject"]="This is a protected header"'
 
 test_begin_subtest "indexed protected subject is not visible in reply header"
-test_subtest_known_broken
 output=$(notmuch reply --format=json 
'id:protected-hea...@crypto.notmuchmail.org')
 test_json_nodes <<<"$output" \
 'subject:["original"]["headers"]["Subject"]="This is a 
protected header"' \
diff --git a/test/T358-emacs-protected-headers.sh 
b/test/T358-emacs-protected-headers.sh
index 765511d4..c195d5c2 100755
--- a/test/T358-emacs-protected-headers.sh
+++ b/test/T358-emacs-protected-headers.sh
@@ -92,7 +92,6 @@ test_expect_equal_file EXPECTED OUTPUT
 
 # notmuch-emacs still leaks the subject line:
 test_begin_subtest "don't leak protected subject during reply, even if indexed"
-test_subtest_known_broken
 test_emacs "(let ((message-hidden-headers '()))
(notmuch-show \"id:protected-hea...@crypto.notmuchmail.org\")
(notmuch-show-reply)
-- 
2.20.1

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2 07/17] test: add test for missing external subject

[Daniel Kahn Gillmor]

Adding another test to ensure that we handle protected headers
gracefully when no external subject is present.

Signed-off-by: Daniel Kahn Gillmor 
---
 test/T356-protected-headers.sh|  6 
 .../subjectless-protected-header.eml  | 29 +++
 2 files changed, 35 insertions(+)
 create mode 100644 
test/corpora/protected-headers/subjectless-protected-header.eml

diff --git a/test/T356-protected-headers.sh b/test/T356-protected-headers.sh
index 68d431e9..59ab58d7 100755
--- a/test/T356-protected-headers.sh
+++ b/test/T356-protected-headers.sh
@@ -25,6 +25,12 @@ test_json_nodes <<<"$output" \
 'crypto:[0][0][0]["crypto"]={"decrypted": {"status": "full", 
"header-mask": {"Subject": "Subject Unavailable"}}}' \
 'subject:[0][0][0]["headers"]["Subject"]="This is a protected 
header"'
 
+test_begin_subtest "when no external header is present, show masked subject as 
null"
+output=$(notmuch show --decrypt=true --format=json 
id:subjectless-protected-hea...@crypto.notmuchmail.org)
+test_json_nodes <<<"$output" \
+'crypto:[0][0][0]["crypto"]={"decrypted": {"status": "full", 
"header-mask": {"Subject": null}}}' \
+'subject:[0][0][0]["headers"]["Subject"]="This is a protected 
header"'
+
 test_begin_subtest "misplaced protected headers should not be made visible 
during decryption"
 output=$(notmuch show --decrypt=true --format=json 
id:misplaced-protected-hea...@crypto.notmuchmail.org)
 test_json_nodes <<<"$output" \
diff --git a/test/corpora/protected-headers/subjectless-protected-header.eml 
b/test/corpora/protected-headers/subjectless-protected-header.eml
new file mode 100644
index ..7163b9ae
--- /dev/null
+++ b/test/corpora/protected-headers/subjectless-protected-header.eml
@@ -0,0 +1,29 @@
+From: test_su...@notmuchmail.org
+To: test_su...@notmuchmail.org
+Date: Sat, 01 Jan 2000 12:00:00 +
+Message-ID: 
+MIME-Version: 1.0
+Content-Type: multipart/encrypted; boundary="=-=-=";
+   protocol="application/pgp-encrypted"
+
+--=-=-=
+Content-Type: application/pgp-encrypted
+
+Version: 1
+
+--=-=-=
+Content-Type: application/octet-stream
+Subject: this should not show up as a protected header
+
+-BEGIN PGP MESSAGE-
+
+hIwDxE023q1UqxYBA/9ZaOuxGtLVWiA7KQfB+4td1AILd1uy039UDb+9YwlhmJTq
+mNqVJu+ZkFniZPMliM0z1QRBkBeL2Q7MrHAdYxYBKrDHKVja4O7jwqeKjy5BzQCW
+fnyT+sb2Mh+dz5P2voF3XJHgqzhFY1rtVEatXSZADwwIVU6oZqGZ8GOELNGSd9KX
+ASNElH7WGZB/TQ5X+MktzOLExx5QWaRK9skogI2RRoOquS7KpMcjzb2FWaJDjr1s
+RGboX7NG3xCvNUV2ByFTvLOeo7eO1GfUsabTUbMMvh3AE1UvHgCu8VJiRrMdmPln
+BM2xnwCYec6wYJ46fHukTgv+286nSQcV0XT6a+qM5GMgV5DMHW2vSyl6kTszJ3EP
+xvQBfPCItA==
+=Gkxz
+-END PGP MESSAGE-
+--=-=-=--
-- 
2.20.1

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2 10/17] indexing: record protected subject when indexing cleartext

[Daniel Kahn Gillmor]

When indexing the cleartext of an encrypted message, record any
protected subject in the database, which should make it findable and
visible in search.

Signed-off-by: Daniel Kahn Gillmor 
---
 lib/index.cc   | 42 ++
 lib/message.cc |  8 +++
 lib/notmuch-private.h  |  4 
 test/T356-protected-headers.sh | 16 +
 4 files changed, 61 insertions(+), 9 deletions(-)

diff --git a/lib/index.cc b/lib/index.cc
index f216ae5d..1fd9e67e 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -367,13 +367,15 @@ _index_content_type (notmuch_message_t *message, 
GMimeObject *part)
 
 static void
 _index_encrypted_mime_part (notmuch_message_t *message, notmuch_indexopts_t 
*indexopts,
-   GMimeMultipartEncrypted *part);
+   GMimeMultipartEncrypted *part,
+   _notmuch_message_crypto_t *msg_crypto);
 
 /* Callback to generate terms for each mime part of a message. */
 static void
 _index_mime_part (notmuch_message_t *message,
  notmuch_indexopts_t *indexopts,
- GMimeObject *part)
+ GMimeObject *part,
+ _notmuch_message_crypto_t *msg_crypto)
 {
 GMimeStream *stream, *filter;
 GMimeFilter *discard_non_term_filter;
@@ -403,6 +405,8 @@ _index_mime_part (notmuch_message_t *message,
  _notmuch_message_add_term (message, "tag", "encrypted");
 
for (i = 0; i < g_mime_multipart_get_count (multipart); i++) {
+   notmuch_status_t status;
+   GMimeObject *child;
if (GMIME_IS_MULTIPART_SIGNED (multipart)) {
/* Don't index the signature, but index its content type. */
if (i == GMIME_MULTIPART_SIGNED_SIGNATURE) {
@@ -419,7 +423,8 @@ _index_mime_part (notmuch_message_t *message,
 g_mime_multipart_get_part (multipart, i));
if (i == GMIME_MULTIPART_ENCRYPTED_CONTENT) {
_index_encrypted_mime_part(message, indexopts,
-  GMIME_MULTIPART_ENCRYPTED 
(part));
+  GMIME_MULTIPART_ENCRYPTED (part),
+  msg_crypto);
} else {
if (i != GMIME_MULTIPART_ENCRYPTED_VERSION) {
_notmuch_database_log (notmuch_message_get_database 
(message),
@@ -428,8 +433,13 @@ _index_mime_part (notmuch_message_t *message,
}
continue;
}
-   _index_mime_part (message, indexopts,
- g_mime_multipart_get_part (multipart, i));
+   child = g_mime_multipart_get_part (multipart, i);
+   status = _notmuch_message_crypto_potential_payload (msg_crypto, 
child, part, i);
+   if (status)
+   _notmuch_database_log (notmuch_message_get_database (message),
+  "Warning: failed to mark the potential 
cryptographic payload (%s).\n",
+  notmuch_status_to_string (status));
+   _index_mime_part (message, indexopts, child, msg_crypto);
}
return;
 }
@@ -439,7 +449,7 @@ _index_mime_part (notmuch_message_t *message,
 
mime_message = g_mime_message_part_get_message (GMIME_MESSAGE_PART 
(part));
 
-   _index_mime_part (message, indexopts, g_mime_message_get_mime_part 
(mime_message));
+   _index_mime_part (message, indexopts, g_mime_message_get_mime_part 
(mime_message), msg_crypto);
 
return;
 }
@@ -516,7 +526,8 @@ _index_mime_part (notmuch_message_t *message,
 static void
 _index_encrypted_mime_part (notmuch_message_t *message,
notmuch_indexopts_t *indexopts,
-   GMimeMultipartEncrypted *encrypted_data)
+   GMimeMultipartEncrypted *encrypted_data,
+   _notmuch_message_crypto_t *msg_crypto)
 {
 notmuch_status_t status;
 GError *err = NULL;
@@ -553,6 +564,10 @@ _index_encrypted_mime_part (notmuch_message_t *message,
return;
 }
 if (decrypt_result) {
+   status = _notmuch_message_crypto_successful_decryption (msg_crypto);
+   if (status)
+   _notmuch_database_log_append (notmuch, "failed to mark the message 
as decrypted (%s)\n",
+ notmuch_status_to_string (status));
if (get_sk) {
status = notmuch_message_add_property (message, "session-key",
   
g_mime_decrypt_result_get_session_key (decrypt_result));
@@ -562,7 +577,8 @@ _index_encrypted_mime_part (notmuch_message_t *message,
}
g_object_unref (decrypt_result);
 }
-_index_mime_part (message, indexopts, clear);
+status = _notmuc

[PATCH v2 05/17] cli/show: emit payload subject instead of outside subject

[Daniel Kahn Gillmor]

Correctly fix the two outstanding tests so that the protected (hidden)
subject is properly reported.

Signed-off-by: Daniel Kahn Gillmor 
---
 notmuch-client.h   |  2 +-
 notmuch-reply.c|  4 +++-
 notmuch-show.c | 14 +-
 test/T356-protected-headers.sh |  2 --
 4 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index a82cb431..39e26f2e 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -232,7 +232,7 @@ format_part_sprinter (const void *ctx, struct sprinter *sp, 
mime_node_t *node,
 
 void
 format_headers_sprinter (struct sprinter *sp, GMimeMessage *message,
-bool reply);
+bool reply, const _notmuch_message_crypto_t 
*msg_crypto);
 
 typedef enum {
 NOTMUCH_SHOW_TEXT_PART_REPLY = 1 << 0,
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 7f284229..2689b247 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -663,7 +663,9 @@ static int do_reply(notmuch_config_t *config,
 
/* The headers of the reply message we've created */
sp->map_key (sp, "reply-headers");
-   format_headers_sprinter (sp, reply, true);
+   /* FIXME: send msg_crypto here to avoid killing the
+    * subject line on reply to encrypted messages! */
+   format_headers_sprinter (sp, reply, true, NULL);
 
/* Start the original */
sp->map_key (sp, "original");
diff --git a/notmuch-show.c b/notmuch-show.c
index 0816a5e1..b1f6a4bb 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -197,7 +197,7 @@ _is_from_line (const char *line)
 
 void
 format_headers_sprinter (sprinter_t *sp, GMimeMessage *message,
-bool reply)
+bool reply, const _notmuch_message_crypto_t 
*msg_crypto)
 {
 /* Any changes to the JSON or S-Expression format should be
  * reflected in the file devel/schemata. */
@@ -209,7 +209,10 @@ format_headers_sprinter (sprinter_t *sp, GMimeMessage 
*message,
 sp->begin_map (sp);
 
 sp->map_key (sp, "Subject");
-sp->string (sp, g_mime_message_get_subject (message));
+if (msg_crypto && msg_crypto->payload_subject) {
+   sp->string (sp, msg_crypto->payload_subject);
+} else
+   sp->string (sp, g_mime_message_get_subject (message));
 
 sp->map_key (sp, "From");
 sp->string (sp, g_mime_message_get_from_string (message));
@@ -616,6 +619,7 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, 
mime_node_t *node,
  * reflected in the file devel/schemata. */
 
 if (node->envelope_file) {
+   const _notmuch_message_crypto_t *msg_crypto = NULL;
sp->begin_map (sp);
format_message_sprinter (sp, node->envelope_file);
 
@@ -626,8 +630,8 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, 
mime_node_t *node,
sp->end (sp);
}
 
+   msg_crypto = mime_node_get_message_crypto_status (node);
if (notmuch_format_version >= 4) {
-   const _notmuch_message_crypto_t *msg_crypto = 
mime_node_get_message_crypto_status (node);
sp->map_key (sp, "crypto");
sp->begin_map (sp);
if (msg_crypto->sig_list ||
@@ -655,7 +659,7 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, 
mime_node_t *node,
}
 
sp->map_key (sp, "headers");
-   format_headers_sprinter (sp, GMIME_MESSAGE (node->part), false);
+   format_headers_sprinter (sp, GMIME_MESSAGE (node->part), false, 
msg_crypto);
 
sp->end (sp);
return;
@@ -748,7 +752,7 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, 
mime_node_t *node,
sp->begin_map (sp);
 
sp->map_key (sp, "headers");
-   format_headers_sprinter (sp, GMIME_MESSAGE (node->part), false);
+   format_headers_sprinter (sp, GMIME_MESSAGE (node->part), false, NULL);
 
sp->map_key (sp, "body");
sp->begin_list (sp);
diff --git a/test/T356-protected-headers.sh b/test/T356-protected-headers.sh
index 599ff1ed..8a8fef6a 100755
--- a/test/T356-protected-headers.sh
+++ b/test/T356-protected-headers.sh
@@ -21,7 +21,6 @@ test_json_nodes <<<"$output" \
 
 test_begin_subtest "verify protected header is visible with decryption"
 output=$(notmuch show --decrypt=true --format=json 
id:protected-hea...@crypto.notmuchmail.org)
-test_subtest_known_broken
 test_json_nodes <<<"$output" \
 'crypto:[0][0][0]["crypto"]={"decrypted": {"status": "full"}}' 
\
 'subject:[0][0][0]["headers"]["Subject"]="This is a protected 
header"'
@@ -58,7 +57,6 @@ test_json_nodes <<<"$output" \
 
 test_begin_subtest "verify nes

Re: [PATCH 11/20] cli/show: emit payload subject instead of outside subject

[Daniel Kahn Gillmor]

On Thu 2018-06-28 21:40:04 -0300, David Bremner wrote:
> Daniel Kahn Gillmor  writes:
>
>>  
>>  sp->map_key (sp, "Subject");
>> -sp->string (sp, g_mime_message_get_subject (message));
>> +if (msg_crypto && msg_crypto->payload_subject) {
>> +sp->string (sp, msg_crypto->payload_subject);
>> +} else
>> +sp->string (sp, g_mime_message_get_subject (message));
>
> This is not really an issue with your patch per se, but do we actually
> use this code for anything other than top level messages? I'm wondering
> because of my experiments with storing message-document level headers
> [1]. It seems difficult to look at the database when things are done at
> the GMime level like this.
>
> [1]  id:20180623014247.17834-1-da...@tethera.net

hm, we might be also using this code (format_headers_sprinter() in
notmuch-show.c) for attached/forwarded messages.  what i wouldn't want
to happen is for an internal message to get its subject swapped out for
the top-level message's subject!

In the new draft of this series, i'm creating an
encrypted-message-with-forwarded-attachment to try to ensure that this
isn't a problem.  thanks for raising the issue.

  --dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[no subject]

[David Bremner]

Thanks to Gregor for the test data. I can now duplicate the problem
with a regression test using (small) synthetic data.


___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 17/20] cli/reply: add --protected-subject boolean flag

[David Bremner]

Daniel Kahn Gillmor  writes:

> This flag indicates the intent of the client to protect the subject
> line, which allows "notmuch reply" to safely emit the earlier
> message's encrypted subject without risking leaking it in the clear in
> the reply.
>
> Obviously, it should only be used by a client that *will* protect the
> subject line.  This feels clumsier than i'd like, but we really don't
> want to be the ones who leak data on the wire that had been protected
> otherwise, and this seems like a safe way to ensure that the MUA is
> capable.
> ---
>  doc/man1/notmuch-reply.rst | 12 
>  notmuch-client.h   |  4 +++-
>  notmuch-reply.c| 20 
>  notmuch-show.c |  9 -
>  test/T356-protected-headers.sh |  7 +++
>  5 files changed, 42 insertions(+), 10 deletions(-)
>
> diff --git a/doc/man1/notmuch-reply.rst b/doc/man1/notmuch-reply.rst
> index c893ba04..08aadba6 100644
> --- a/doc/man1/notmuch-reply.rst
> +++ b/doc/man1/notmuch-reply.rst
> @@ -70,6 +70,18 @@ Supported options for **reply** include
>  order, and copy values from the first that contains something
>  other than only the user's addresses.
>  
> +``--protected-subject=(true|false)``
> +
> +Indicates that the replying client plans to protect (hide) the
> +subject in the subsequent reply.  When replying to an encrypted
> +    message that itself has an encrypted subject, **notmuch**
> +**reply** needs to propose a subject for the new reply e-mail.  If
> +the client can handle protected subjects safely (if this flag is
> +    set to ``true``), then the cleartext subject will be proposed.
> +Otherwise, the external (dummy) subject is proposed, to avoid
> +leaking the previously protected subject on reply. Defaults to
> +``false``.
> +
>  ``--decrypt=(false|auto|true)``

What about using a keyword argument like --protected=subject ? that
would allow easy future addition of protected headers by specifying e.g.

  --protected=subject --protected=references

>  If ``true``, decrypt any MIME encrypted parts found in the
> diff --git a/notmuch-client.h b/notmuch-client.h
> index 0af96986..014fa064 100644
> --- a/notmuch-client.h
> +++ b/notmuch-client.h
> @@ -235,7 +235,9 @@ typedef enum {
>  /* typical "notmuch show" or other standard output: */
>  HEADERS_FORMAT_NORMAL = 0,
>  /* set only if this is being generated as a reply: */
> -HEADERS_FORMAT_REPLY = 1 << 0
> +HEADERS_FORMAT_REPLY = 1 << 0,
> +/* set only if the invoking MUA will responsibly protect the subject 
> line */
> +HEADERS_FORMAT_PROTECTED_SUBJECT = 1 << 1
>  } notmuch_headers_format_flags;

a minor improvement might be to end the list with a ',' from the start
to minimize diff.

>  
> diff --git a/notmuch-reply.c b/notmuch-reply.c
> index 749eac6d..d1092ce9 100644
> --- a/notmuch-reply.c
> +++ b/notmuch-reply.c
> @@ -612,7 +612,8 @@ static int do_reply(notmuch_config_t *config,
>   notmuch_query_t *query,
>   notmuch_show_params_t *params,
>   int format,
> - bool reply_all)
> + bool reply_all,
> + bool protected_subject)

similarly, maybe use the already defined flag enum rather than a boolean
here.

d


___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 11/20] cli/show: emit payload subject instead of outside subject

[David Bremner]

Daniel Kahn Gillmor  writes:

>  
>  sp->map_key (sp, "Subject");
> -sp->string (sp, g_mime_message_get_subject (message));
> +if (msg_crypto && msg_crypto->payload_subject) {
> + sp->string (sp, msg_crypto->payload_subject);
> +} else
> + sp->string (sp, g_mime_message_get_subject (message));

This is not really an issue with your patch per se, but do we actually
use this code for anything other than top level messages? I'm wondering
because of my experiments with storing message-document level headers
[1]. It seems difficult to look at the database when things are done at
the GMime level like this.

[1]  id:20180623014247.17834-1-da...@tethera.net

>  test_json_nodes <<<"$output" \
>  'crypto:[0][0][0]["crypto"]={"decrypted": {"status": 
> "full"}}' \
>  'subject:[0][0][0]["headers"]["Subject"]="This is a 
> protected header"'
>  
> -

naughty white space change. Time to update your pre-commit hook?

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 18/20] indexing: record protected subject when indexing cleartext

[Jameson Graef Rollins]

On Fri, May 11 2018, Daniel Kahn Gillmor  wrote:
> When indexing the cleartext of an encrypted message, record any
> protected subject in the database, which should make it findable and
> visible in search.
> ---
>  lib/index.cc   | 42 ++
...
> diff --git a/lib/index.cc b/lib/index.cc
> index 0ad683fa..db16b6f8 100644
> --- a/lib/index.cc
> +++ b/lib/index.cc
...
> @@ -430,8 +435,13 @@ _index_mime_part (notmuch_message_t *message,
>   }
>   continue;
>   }
> - _index_mime_part (message, indexopts,
> -   g_mime_multipart_get_part (multipart, i));
> + child = g_mime_multipart_get_part (multipart, i);
> + status = _notmuch_message_crypto_potential_payload (msg_crypto, 
> child, part, i);
> + if (status)
> + _notmuch_database_log (_notmuch_message_database (message),
> +"Warning: failed to mark the potential 
> cryptographic payload (%s).\n",
> +notmuch_status_to_string (status));
> + _index_mime_part (message, indexopts, child, msg_crypto);
>   }
>   return;
>  }

In 9088db76d89264b733f6b45e776d8952da237921 _notmuch_message_database
was exposed as notmuch_message_get_database, so this patch needs to be
updated:

diff --git a/lib/index.cc b/lib/index.cc
index 74e1ba43..18e712b1 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -438,7 +438,7 @@ _index_mime_part (notmuch_message_t *message,
child = g_mime_multipart_get_part (multipart, i);
status = _notmuch_message_crypto_potential_payload (msg_crypto, 
child, part, i);
if (status)
-   _notmuch_database_log (_notmuch_message_database (message),
+   _notmuch_database_log (notmuch_message_get_database (message),
   "Warning: failed to mark the potential 
cryptographic payload (%s).\n",
   notmuch_status_to_string (status));
_index_mime_part (message, indexopts, child, msg_crypto);

This fix doesn't affect any of the subsequent patches in this series.

jamie.


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 15/20] cli/reply: ensure encrypted Subject: line does not leak in the clear

[Daniel Kahn Gillmor]

Now that we can decrypt headers, we want to make sure that clients
using "notmuch reply" to prepare a reply don't leak cleartext in their
subject lines.  In particular, the ["reply-headers"]["Subject"] should
by default show the external Subject.
---
 test/T356-protected-headers.sh | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/test/T356-protected-headers.sh b/test/T356-protected-headers.sh
index 67d2e0cb..687681ff 100755
--- a/test/T356-protected-headers.sh
+++ b/test/T356-protected-headers.sh
@@ -78,4 +78,11 @@ output=$(notmuch show --verify --format=json 
id:signed-protected-header@crypto.n
 test_json_nodes <<<"$output" \
 'crypto:[0][0][0]["crypto"]={"signed": {"status": [{"created": 
1525350527, "fingerprint": "5AEAB11F5E33DCE875DDB75B6D92612D94E46381", 
"status": "good"}], "headers": ["Subject"]}}'
 
+test_begin_subtest "protected subject does not leak by default in replies"
+output=$(notmuch reply --decrypt=true --format=json 
id:protected-hea...@crypto.notmuchmail.org)
+test_json_nodes <<<"$output" \
+    'crypto:["original"]["crypto"]={"decrypted": {"status": 
"full", "masked-headers": {"Subject": "encrypted message"}}}' \
+'subject:["original"]["headers"]["Subject"]="This is a 
protected header"' \
+'reply-subject:["reply-headers"]["Subject"]="Re: encrypted 
message"'
+
 test_done
-- 
2.17.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 17/20] cli/reply: add --protected-subject boolean flag

[Daniel Kahn Gillmor]

This flag indicates the intent of the client to protect the subject
line, which allows "notmuch reply" to safely emit the earlier
message's encrypted subject without risking leaking it in the clear in
the reply.

Obviously, it should only be used by a client that *will* protect the
subject line.  This feels clumsier than i'd like, but we really don't
want to be the ones who leak data on the wire that had been protected
otherwise, and this seems like a safe way to ensure that the MUA is
capable.
---
 doc/man1/notmuch-reply.rst | 12 
 notmuch-client.h   |  4 +++-
 notmuch-reply.c| 20 
 notmuch-show.c |  9 -
 test/T356-protected-headers.sh |  7 +++
 5 files changed, 42 insertions(+), 10 deletions(-)

diff --git a/doc/man1/notmuch-reply.rst b/doc/man1/notmuch-reply.rst
index c893ba04..08aadba6 100644
--- a/doc/man1/notmuch-reply.rst
+++ b/doc/man1/notmuch-reply.rst
@@ -70,6 +70,18 @@ Supported options for **reply** include
 order, and copy values from the first that contains something
 other than only the user's addresses.
 
+``--protected-subject=(true|false)``
+
+Indicates that the replying client plans to protect (hide) the
+    subject in the subsequent reply.  When replying to an encrypted
+message that itself has an encrypted subject, **notmuch**
+**reply** needs to propose a subject for the new reply e-mail.  If
+the client can handle protected subjects safely (if this flag is
+set to ``true``), then the cleartext subject will be proposed.
+Otherwise, the external (dummy) subject is proposed, to avoid
+leaking the previously protected subject on reply. Defaults to
+``false``.
+
 ``--decrypt=(false|auto|true)``
 
 If ``true``, decrypt any MIME encrypted parts found in the
diff --git a/notmuch-client.h b/notmuch-client.h
index 0af96986..014fa064 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -235,7 +235,9 @@ typedef enum {
 /* typical "notmuch show" or other standard output: */
 HEADERS_FORMAT_NORMAL = 0,
 /* set only if this is being generated as a reply: */
-HEADERS_FORMAT_REPLY = 1 << 0
+HEADERS_FORMAT_REPLY = 1 << 0,
+/* set only if the invoking MUA will responsibly protect the subject line 
*/
+HEADERS_FORMAT_PROTECTED_SUBJECT = 1 << 1
 } notmuch_headers_format_flags;
 
 
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 749eac6d..d1092ce9 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -612,7 +612,8 @@ static int do_reply(notmuch_config_t *config,
notmuch_query_t *query,
notmuch_show_params_t *params,
int format,
-   bool reply_all)
+   bool reply_all,
+   bool protected_subject)
 {
 GMimeMessage *reply;
 mime_node_t *node;
@@ -659,18 +660,19 @@ static int do_reply(notmuch_config_t *config,
return 1;
 
if (format == FORMAT_JSON || format == FORMAT_SEXP) {
+   notmuch_headers_format_flags flags = HEADERS_FORMAT_REPLY;
sp->begin_map (sp);
 
-   /* The headers of the reply message we've created */
-   sp->map_key (sp, "reply-headers");
-   /* FIXME: send msg_crypto here to avoid killing the
-* subject line on reply to encrypted messages! */
-   format_headers_sprinter (sp, reply, HEADERS_FORMAT_REPLY, NULL);
-
/* Start the original */
sp->map_key (sp, "original");
format_part_sprinter (config, sp, node, true, false);
 
+   /* The headers of the reply message we've created */
+   sp->map_key (sp, "reply-headers");
+   if (protected_subject)
+   flags |= HEADERS_FORMAT_PROTECTED_SUBJECT;
+   format_headers_sprinter (sp, reply, flags, 
mime_node_get_message_crypto_status (node));
+
/* End */
sp->end (sp);
} else {
@@ -699,6 +701,7 @@ notmuch_reply_command (notmuch_config_t *config, int argc, 
char *argv[])
 notmuch_database_t *notmuch;
 notmuch_query_t *query;
 char *query_string;
+bool protected_subject = false;
 int opt_index;
 notmuch_show_params_t params = {
.part = -1,
@@ -715,6 +718,7 @@ notmuch_reply_command (notmuch_config_t *config, int argc, 
char *argv[])
  { "headers-only", FORMAT_HEADERS_ONLY },
  { 0, 0 } } },
{ .opt_int = _format_version, .name = "format-version" },
+   { .opt_bool = _subject, .name = "protected-subject" },
{ .opt_keyword = _all, .name = "reply-to", .keywords =
  (notmuch_keyword_t []){ { "all", true },
  { "sender", false },
@@ -764,7 +768,7 @@ notmuch_reply_command (notmu

[PATCH 18/20] indexing: record protected subject when indexing cleartext

[Daniel Kahn Gillmor]

When indexing the cleartext of an encrypted message, record any
protected subject in the database, which should make it findable and
visible in search.
---
 lib/index.cc   | 42 ++
 lib/message.cc |  8 +++
 lib/notmuch-private.h  |  4 
 test/T356-protected-headers.sh | 20 
 4 files changed, 65 insertions(+), 9 deletions(-)

diff --git a/lib/index.cc b/lib/index.cc
index 0ad683fa..db16b6f8 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -367,13 +367,15 @@ _index_content_type (notmuch_message_t *message, 
GMimeObject *part)
 static void
 _index_encrypted_mime_part (notmuch_message_t *message, notmuch_indexopts_t 
*indexopts,
GMimeContentType *content_type,
-   GMimeMultipartEncrypted *part);
+   GMimeMultipartEncrypted *part,
+   _notmuch_message_crypto_t *msg_crypto);
 
 /* Callback to generate terms for each mime part of a message. */
 static void
 _index_mime_part (notmuch_message_t *message,
  notmuch_indexopts_t *indexopts,
- GMimeObject *part)
+ GMimeObject *part,
+ _notmuch_message_crypto_t *msg_crypto)
 {
 GMimeStream *stream, *filter;
 GMimeFilter *discard_non_term_filter;
@@ -404,6 +406,8 @@ _index_mime_part (notmuch_message_t *message,
  _notmuch_message_add_term (message, "tag", "encrypted");
 
for (i = 0; i < g_mime_multipart_get_count (multipart); i++) {
+   notmuch_status_t status;
+   GMimeObject *child;
if (GMIME_IS_MULTIPART_SIGNED (multipart)) {
/* Don't index the signature, but index its content type. */
if (i == GMIME_MULTIPART_SIGNED_SIGNATURE) {
@@ -421,7 +425,8 @@ _index_mime_part (notmuch_message_t *message,
if (i == GMIME_MULTIPART_ENCRYPTED_CONTENT) {
_index_encrypted_mime_part(message, indexopts,
   content_type,
-  GMIME_MULTIPART_ENCRYPTED 
(part));
+  GMIME_MULTIPART_ENCRYPTED (part),
+  msg_crypto);
} else {
if (i != GMIME_MULTIPART_ENCRYPTED_VERSION) {
_notmuch_database_log (_notmuch_message_database 
(message),
@@ -430,8 +435,13 @@ _index_mime_part (notmuch_message_t *message,
}
continue;
}
-   _index_mime_part (message, indexopts,
- g_mime_multipart_get_part (multipart, i));
+   child = g_mime_multipart_get_part (multipart, i);
+   status = _notmuch_message_crypto_potential_payload (msg_crypto, 
child, part, i);
+   if (status)
+   _notmuch_database_log (_notmuch_message_database (message),
+  "Warning: failed to mark the potential 
cryptographic payload (%s).\n",
+  notmuch_status_to_string (status));
+   _index_mime_part (message, indexopts, child, msg_crypto);
}
return;
 }
@@ -441,7 +451,7 @@ _index_mime_part (notmuch_message_t *message,
 
mime_message = g_mime_message_part_get_message (GMIME_MESSAGE_PART 
(part));
 
-   _index_mime_part (message, indexopts, g_mime_message_get_mime_part 
(mime_message));
+   _index_mime_part (message, indexopts, g_mime_message_get_mime_part 
(mime_message), msg_crypto);
 
return;
 }
@@ -518,7 +528,8 @@ static void
 _index_encrypted_mime_part (notmuch_message_t *message,
notmuch_indexopts_t *indexopts,
g_mime_3_unused(GMimeContentType *content_type),
-   GMimeMultipartEncrypted *encrypted_data)
+   GMimeMultipartEncrypted *encrypted_data,
+   _notmuch_message_crypto_t *msg_crypto)
 {
 notmuch_status_t status;
 GError *err = NULL;
@@ -573,6 +584,10 @@ _index_encrypted_mime_part (notmuch_message_t *message,
return;
 }
 if (decrypt_result) {
+   status = _notmuch_message_crypto_successful_decryption (msg_crypto);
+   if (status)
+   _notmuch_database_log_append (notmuch, "failed to mark the message 
as decrypted (%s)\n",
+ notmuch_status_to_string (status));
 #if HAVE_GMIME_SESSION_KEYS
if (get_sk) {
status = notmuch_message_add_property (message, "session-key",
@@ -584,7 +599,8 @@ _index_encrypted_mime_part (notmuch_message_t *message,
 #endif
g_object_unref (decrypt_result);
 }
-_index_mime_part (message, indexopts, clear);
+status = _notmuch_message_crypto_potential_payl

[PATCH 13/20] test: add test for missing external subject

[Daniel Kahn Gillmor]

Adding another test to ensure that we handle it gracefully when no
external subject is present.
---
 test/T356-protected-headers.sh|  6 
 .../subjectless-protected-header.eml  | 29 +++
 2 files changed, 35 insertions(+)
 create mode 100644 
test/corpora/protected-headers/subjectless-protected-header.eml

diff --git a/test/T356-protected-headers.sh b/test/T356-protected-headers.sh
index c6838995..901ee60c 100755
--- a/test/T356-protected-headers.sh
+++ b/test/T356-protected-headers.sh
@@ -27,6 +27,12 @@ test_json_nodes <<<"$output" \
 'crypto:[0][0][0]["crypto"]={"decrypted": {"status": "full", 
"masked-headers": {"Subject": "encrypted message"}}}' \
 'subject:[0][0][0]["headers"]["Subject"]="This is a protected 
header"'
 
+test_begin_subtest "when no external header is present, show masked subject as 
null"
+output=$(notmuch show --decrypt=true --format=json 
id:subjectless-protected-hea...@crypto.notmuchmail.org)
+test_json_nodes <<<"$output" \
+'crypto:[0][0][0]["crypto"]={"decrypted": {"status": "full", 
"masked-headers": {"Subject": null}}}' \
+'subject:[0][0][0]["headers"]["Subject"]="This is a protected 
header"'
+
 test_begin_subtest "misplaced protected headers should not be made visible 
during decryption"
 output=$(notmuch show --decrypt=true --format=json 
id:misplaced-protected-hea...@crypto.notmuchmail.org)
 test_json_nodes <<<"$output" \
diff --git a/test/corpora/protected-headers/subjectless-protected-header.eml 
b/test/corpora/protected-headers/subjectless-protected-header.eml
new file mode 100644
index ..7163b9ae
--- /dev/null
+++ b/test/corpora/protected-headers/subjectless-protected-header.eml
@@ -0,0 +1,29 @@
+From: test_su...@notmuchmail.org
+To: test_su...@notmuchmail.org
+Date: Sat, 01 Jan 2000 12:00:00 +
+Message-ID: <subjectless-protected-hea...@crypto.notmuchmail.org>
+MIME-Version: 1.0
+Content-Type: multipart/encrypted; boundary="=-=-=";
+   protocol="application/pgp-encrypted"
+
+--=-=-=
+Content-Type: application/pgp-encrypted
+
+Version: 1
+
+--=-=-=
+Content-Type: application/octet-stream
+Subject: this should not show up as a protected header
+
+-BEGIN PGP MESSAGE-
+
+hIwDxE023q1UqxYBA/9ZaOuxGtLVWiA7KQfB+4td1AILd1uy039UDb+9YwlhmJTq
+mNqVJu+ZkFniZPMliM0z1QRBkBeL2Q7MrHAdYxYBKrDHKVja4O7jwqeKjy5BzQCW
+fnyT+sb2Mh+dz5P2voF3XJHgqzhFY1rtVEatXSZADwwIVU6oZqGZ8GOELNGSd9KX
+ASNElH7WGZB/TQ5X+MktzOLExx5QWaRK9skogI2RRoOquS7KpMcjzb2FWaJDjr1s
+RGboX7NG3xCvNUV2ByFTvLOeo7eO1GfUsabTUbMMvh3AE1UvHgCu8VJiRrMdmPln
+BM2xnwCYec6wYJ46fHukTgv+286nSQcV0XT6a+qM5GMgV5DMHW2vSyl6kTszJ3EP
+xvQBfPCItA==
+=Gkxz
+-END PGP MESSAGE-
+--=-=-=--
-- 
2.17.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 11/20] cli/show: emit payload subject instead of outside subject

[Daniel Kahn Gillmor]

Correctly fix the two outstanding tests so that the protected (hidden)
subject is properly reported.
---
 notmuch-client.h   |  2 +-
 notmuch-reply.c|  4 +++-
 notmuch-show.c | 11 +++
 test/T356-protected-headers.sh |  3 ---
 4 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index dfc7c047..73c8a163 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -232,7 +232,7 @@ format_part_sprinter (const void *ctx, struct sprinter *sp, 
mime_node_t *node,
 
 void
 format_headers_sprinter (struct sprinter *sp, GMimeMessage *message,
-bool reply);
+bool reply, const _notmuch_message_crypto_t 
*msg_crypto);
 
 typedef enum {
 NOTMUCH_SHOW_TEXT_PART_REPLY = 1 << 0,
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 75cf7ecb..fe02c590 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -663,7 +663,9 @@ static int do_reply(notmuch_config_t *config,
 
/* The headers of the reply message we've created */
sp->map_key (sp, "reply-headers");
-   format_headers_sprinter (sp, reply, true);
+   /* FIXME: send msg_crypto here to avoid killing the
+    * subject line on reply to encrypted messages! */
+   format_headers_sprinter (sp, reply, true, NULL);
 
/* Start the original */
sp->map_key (sp, "original");
diff --git a/notmuch-show.c b/notmuch-show.c
index 4e918461..190e9128 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -196,7 +196,7 @@ _is_from_line (const char *line)
 
 void
 format_headers_sprinter (sprinter_t *sp, GMimeMessage *message,
-bool reply)
+bool reply, const _notmuch_message_crypto_t 
*msg_crypto)
 {
 /* Any changes to the JSON or S-Expression format should be
  * reflected in the file devel/schemata. */
@@ -208,7 +208,10 @@ format_headers_sprinter (sprinter_t *sp, GMimeMessage 
*message,
 sp->begin_map (sp);
 
 sp->map_key (sp, "Subject");
-sp->string (sp, g_mime_message_get_subject (message));
+if (msg_crypto && msg_crypto->payload_subject) {
+   sp->string (sp, msg_crypto->payload_subject);
+} else
+   sp->string (sp, g_mime_message_get_subject (message));
 
 sp->map_key (sp, "From");
 sp->string (sp, g_mime_message_get_from_string (message));
@@ -641,7 +644,7 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, 
mime_node_t *node,
}
 
sp->map_key (sp, "headers");
-   format_headers_sprinter (sp, GMIME_MESSAGE (node->part), false);
+   format_headers_sprinter (sp, GMIME_MESSAGE (node->part), false, 
msg_crypto);
 
sp->end (sp);
return;
@@ -734,7 +737,7 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, 
mime_node_t *node,
sp->begin_map (sp);
 
sp->map_key (sp, "headers");
-   format_headers_sprinter (sp, GMIME_MESSAGE (node->part), false);
+   format_headers_sprinter (sp, GMIME_MESSAGE (node->part), false, NULL);
 
sp->map_key (sp, "body");
sp->begin_list (sp);
diff --git a/test/T356-protected-headers.sh b/test/T356-protected-headers.sh
index 9c6fe467..242ad105 100755
--- a/test/T356-protected-headers.sh
+++ b/test/T356-protected-headers.sh
@@ -23,12 +23,10 @@ test_json_nodes <<<"$output" \
 
 test_begin_subtest "verify protected header is visible with decryption"
 output=$(notmuch show --decrypt=true --format=json 
id:protected-hea...@crypto.notmuchmail.org)
-test_subtest_known_broken
 test_json_nodes <<<"$output" \
 'crypto:[0][0][0]["crypto"]={"decrypted": {"status": "full"}}' 
\
 'subject:[0][0][0]["headers"]["Subject"]="This is a protected 
header"'
 
-
 test_begin_subtest "misplaced protected headers should not be made visible 
during decryption"
 output=$(notmuch show --decrypt=true --format=json 
id:misplaced-protected-hea...@crypto.notmuchmail.org)
 test_json_nodes <<<"$output" \
@@ -61,7 +59,6 @@ test_json_nodes <<<"$output" \
 
 test_begin_subtest "verify nested message/rfc822 protected header is visible"
 output=$(notmuch show --decrypt=true --format=json 
id:nested-rfc822-mess...@crypto.notmuchmail.org)
-test_subtest_known_broken
 test_json_nodes <<<"$output" \
 'crypto:[0][0][0]["crypto"]={"decrypted": {"status": "full"}}' 
\
 'subject:[0][0][0]["headers"]["Subject"]="This is a message 
using draft-melnikov-smime-header-signing"'
-- 
2.17.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] test: add known broken test for regexp search of second subject

[David Bremner]

David Bremner <da...@tethera.net> writes:

> We expect this to give the same answer as the non-regexp subject
> search. It does not because the regexp search relies on the value
> slot, which currently contains only one subject.

Pushed to master. I still need to revisit the fix.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

wish: notmuch-emacs: wash From: / To: and Subject: in notmuch search / show

[Gregor Zattler]

Dear notmuch-emacs developers,

it would be nice if there was a facility to wash (= shorten)
the displayed To:, From:  and Subject headers at least for
the display of notmuch search results.

>From the users perspective this could be done via a list of
regular expressions with corresponding short forms.

I assume this would be best achieved in a frontend, not in notmuch
the command line utility.


Example use case:
Some of my emails are from a request tracker and have
extraordinary long From: and Subject: suffixes which add up to 49
characters.  The only way to see the relevant parts of both From:
and Subject: in this cases is to use notmuch-emacs in a single,
maximized window.  I would like to radically shorten this
boilerplate.


Ciao; Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[no subject]

[Matthew Lear]

From: Matthew Lear <m...@bubblegen.co.uk>
To: notmuch@notmuchmail.org
Cc: Matthew Lear <m...@bubblegen.co.uk>
Subject: [PATCH] Update date search syntax.
Date: Thu,  1 Feb 2018 20:52:18 +
Message-Id: <20180201205218.4368-1-m...@bubblegen.co.uk>
X-Mailer: git-send-email 2.14.1

If searching using the date prefix and timestamps, each timestamp
is required to be prefixed with an @
Legacy syntax of .. without the
date prefix is still honoured, only without the @ specifiers.
---
 doc/man7/notmuch-search-terms.rst | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/doc/man7/notmuch-search-terms.rst 
b/doc/man7/notmuch-search-terms.rst
index 6d2bf62a..b6e7079a 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -124,10 +124,13 @@ date:.. or date:
 The time range can also be specified using timestamps with a
 syntax of:
 
-..
+@..@
 
 Each timestamp is a number representing the number of seconds
-since 1970-01-01 00:00:00 UTC.
+since 1970-01-01 00:00:00 UTC. A date range search using
+timestamps is also permitted without using the date prefix and
+@ specifiers, although this is considered legacy and pre-dates
+the date prefix.
 
 lastmod:..
 The **lastmod:** prefix can be used to restrict the result by the
-- 
2.14.1

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] test: add known broken test for regexp search of second subject

[David Bremner]

We expect this to give the same answer as the non-regexp subject
search. It does not because the regexp search relies on the value
slot, which currently contains only one subject.
---
 test/T670-duplicate-mid.sh | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/test/T670-duplicate-mid.sh b/test/T670-duplicate-mid.sh
index c198c506..bf8cc3a8 100755
--- a/test/T670-duplicate-mid.sh
+++ b/test/T670-duplicate-mid.sh
@@ -47,6 +47,16 @@ EOF
 notmuch search --output=files subject:'"message 2"' | notmuch_dir_sanitize > 
OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest 'Regexp search for second subject'
+test_subtest_known_broken
+cat <EXPECTED
+MAIL_DIR/copy0
+MAIL_DIR/copy1
+MAIL_DIR/copy2
+EOF
+notmuch search --output=files 'subject:"/message 2/"' | notmuch_dir_sanitize > 
OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 add_message '[id]="duplicate"' '[body]="sekrit" [filename]=copy3'
 test_begin_subtest 'search for body in duplicate file'
 cat <EXPECTED
-- 
2.15.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 0/6] Sort by from and subject

[Mark Walters]

Hi

On Sat, 30 Sep 2017, Jani Nikula <j...@nikula.org> wrote:
> On Sat, 30 Sep 2017, William Casarin <j...@jb55.com> wrote:
>> Jani Nikula <j...@nikula.org> writes:
>>
>>> I think there are two considerations here:
>>>
>>> First, is this something we want to have? Is this generally useful?
>>
>> Sorting by from and subject are in most mail clients (mutt, gnus, outlook...)
>
> Which of those display results as threads, and of those that do, how do
> they sort the threads? In the notmuch case, the threads would be sorted
> based on one of the matching messages. Which one should it be? For
> current date based searches, the message used for sorting is also
> selected based on date.

I agree with Jani that for thread based views sorting by from is a
little odd -- sorting by subject less so as that is mostly constant in a
thread.

But allowing sorting by from in message based views could be useful. If
we are looking at the notmuch-emacs frontend then that would be in tree
view. This calls notmuch-show.c from the CLI, so that is the place I
would suggest we do this. If notmuch-show.c returns each message as "if
it were in its own thread" in the sense of the sexp output (see
devel/schemata) I think notmuch-tree would just work.

>>> There's still the issue of From: and Subject: needing more heuristic for
>>> useful sorting that I mentioned in id:87efrm70ai@nikula.org.
>>
>> I think I understand what you mean in id:87efrm70ai@nikula.org but I
>> don't have enough knowledge of notmuch to implement what you're asking
>> :(. I believe these are rare cases because I haven't ran into the issue
>> you described?
>
> Look at the subject line of this message. Should it be sorted starting
> at "Re:", "[PATCH", or "Sort"? You could argue for and against any one
> of them. Contrast that with the thread sorting above: If the matching
> message in this thread changes from one with vs. without "Re:", the sort
> placement of the thread could change considerably.
>
> It's common for some corporate mail systems to switch "Firstname
> Lastname" in messages to "Lastname, Firstname". Should we do something
> about that?
>
> Arguably we could do the sorting first, and think of ways to improve it
> afterwards.

These are concerns, but I agree that we try some form of sorting first
and then think about improving it. I think the above as an interface to
notmuch-tree should work -- I had something similar in the very early
days of notmuch-tree (when it was notmuch-pick) but dropped it when
trying to keep the patch set small enough to be even vaguely manageable.

Best wishes

Mark

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 0/6] Sort by from and subject

[Jani Nikula]

On Sat, 30 Sep 2017, William Casarin <j...@jb55.com> wrote:
> Jani Nikula <j...@nikula.org> writes:
>
>> I think there are two considerations here:
>>
>> First, is this something we want to have? Is this generally useful?
>
> Sorting by from and subject are in most mail clients (mutt, gnus, outlook...)

Which of those display results as threads, and of those that do, how do
they sort the threads? In the notmuch case, the threads would be sorted
based on one of the matching messages. Which one should it be? For
current date based searches, the message used for sorting is also
selected based on date.

If we expand the sorting, perhaps we should also think about sorting by
relevance provided by Xapian. Not saying you'd have to do this, but I'd
like to figure out how all of this should work. The implementation
doesn't look particularly difficult, it's the design that's harder to
get right.

>> There's still the issue of From: and Subject: needing more heuristic for
>> useful sorting that I mentioned in id:87efrm70ai@nikula.org.
>
> I think I understand what you mean in id:87efrm70ai@nikula.org but I
> don't have enough knowledge of notmuch to implement what you're asking
> :(. I believe these are rare cases because I haven't ran into the issue
> you described?

Look at the subject line of this message. Should it be sorted starting
at "Re:", "[PATCH", or "Sort"? You could argue for and against any one
of them. Contrast that with the thread sorting above: If the matching
message in this thread changes from one with vs. without "Re:", the sort
placement of the thread could change considerably.

It's common for some corporate mail systems to switch "Firstname
Lastname" in messages to "Lastname, Firstname". Should we do something
about that?

Arguably we could do the sorting first, and think of ways to improve it
afterwards.

>> Second, if we decide we want this, IMHO the interfaces (both human and
>> the lib) need to split the sort key and sort order from the
>> start. Fixing it later on is not going to be fun.
>
> I agree, I figured that would have been a larger refactor so I decided
> not to mix it in with this one. I'll start working on a branch that
> addresses this.

Please wait for feedback from others first, to not waste effort based on
just my opinions. I don't make the decisions here. :)


BR,
Jani.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 0/6] Sort by from and subject

[William Casarin]

Hey Jani,

Jani Nikula <j...@nikula.org> writes:

> I think there are two considerations here:
>
> First, is this something we want to have? Is this generally useful?

Sorting by from and subject are in most mail clients (mutt, gnus, outlook...)

> There's still the issue of From: and Subject: needing more heuristic for
> useful sorting that I mentioned in id:87efrm70ai@nikula.org.

I think I understand what you mean in id:87efrm70ai@nikula.org but I
don't have enough knowledge of notmuch to implement what you're asking
:(. I believe these are rare cases because I haven't ran into the issue
you described?

> Second, if we decide we want this, IMHO the interfaces (both human and
> the lib) need to split the sort key and sort order from the
> start. Fixing it later on is not going to be fun.

I agree, I figured that would have been a larger refactor so I decided
not to mix it in with this one. I'll start working on a branch that
addresses this.


Thanks!
William

-- 
https://jb55.com
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 0/6] Sort by from and subject

[Jani Nikula]

On Mon, 25 Sep 2017, William Casarin <j...@jb55.com> wrote:
> This patch series replaces my original set[1]. I've been using this
> extensivly for about 3 weeks now and I'm pretty happy with it. I've
> added the ability to change sort-order on the fly with the O key
> binding.

I think there are two considerations here:

First, is this something we want to have? Is this generally useful?
There's still the issue of From: and Subject: needing more heuristic for
useful sorting that I mentioned in id:87efrm70ai@nikula.org.

Second, if we decide we want this, IMHO the interfaces (both human and
the lib) need to split the sort key and sort order from the
start. Fixing it later on is not going to be fun.


BR,
Jani.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: emacs: reply subject sanitization

[David Bremner]

Jani Nikula  writes:

> v2 of id:20170915155716.19597-1-j...@nikula.org, now with test.
>
> BR,
> Jani.

pushed.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: emacs: reply subject sanitization

[David Edmondson]

On Tuesday, 2017-09-26 at 21:26:06 +0300, Jani Nikula wrote:

> v2 of id:20170915155716.19597-1-j...@nikula.org, now with test.

Looks good.

dme.
-- 
I can't explain, you would not understand. This is not how I am.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 2/2] emacs: sanitize subject in replies

[Jani Nikula]

Commit a7964c86d125 ("emacs: Sanitize authors and subjects in search
and show") added sanitization of header information for display. Do
the same for reply subjects.

This fixes the long-standing annoying artefact of certain versions of
mailman using tab as folding whitespace, leading to tabs in reply
subjects.
---
 emacs/notmuch-mua.el | 2 +-
 test/T310-emacs.sh   | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
index fd64b362b542..7a341ebf0588 100644
--- a/emacs/notmuch-mua.el
+++ b/emacs/notmuch-mua.el
@@ -218,7 +218,7 @@ mutiple parts get a header."
 else
 collect pair)))
  (notmuch-mua-mail (plist-get reply-headers :To)
-   (plist-get reply-headers :Subject)
+   (notmuch-sanitize (plist-get reply-headers 
:Subject))
(notmuch-headers-plist-to-alist reply-headers)
nil (notmuch-mua-get-switch-function
 
diff --git a/test/T310-emacs.sh b/test/T310-emacs.sh
index 2ef566bac490..4456bc659158 100755
--- a/test/T310-emacs.sh
+++ b/test/T310-emacs.sh
@@ -402,7 +402,6 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "Reply within emacs to a message with TAB in subject"
-test_subtest_known_broken
 test_emacs '(let ((message-hidden-headers ''()))
(notmuch-search 
"id:1258471718-6781-1-git-send-email-dotted...@dottedmag.net")
(notmuch-test-wait)
-- 
2.11.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

emacs: reply subject sanitization

[Jani Nikula]

v2 of id:20170915155716.19597-1-j...@nikula.org, now with test.

BR,
Jani.

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 0/6] Sort by from and subject

[William Casarin]

This patch series replaces my original set[1]. I've been using this
extensivly for about 3 weeks now and I'm pretty happy with it. I've
added the ability to change sort-order on the fly with the O key
binding.

Main use cases
--

* subject sorting: github subscriptions

Sorting through your subscribed github repos is now easier than ever!

before: https://jb55.com/s/7889d50bc848b089.png
after:  https://jb55.com/s/c8b5055939d0482f.png

* from sorting: rss feeds

https://jb55.com/s/68a5aa6ecdebbbfe.png


Cheers,
William


[1] id:20170904160040.23642-1-j...@jb55.com

William Casarin (6):
  sorting: add the ability to sort by from and subject
  sorting: update ruby bindings for from and subject
  sorting: update man page
  emacs: replace oldest-first with sort-order
  emacs: notmuch-search-orders
  emacs: add notmuch-search-change-order

 bindings/ruby/init.c | 24 
 doc/man1/notmuch-address.rst |  6 +++-
 emacs/notmuch-hello.el   | 18 ++--
 emacs/notmuch-jump.el| 13 -
 emacs/notmuch-lib.el | 15 --
 emacs/notmuch-tree.el|  2 +-
 emacs/notmuch.el | 66 +---
 lib/notmuch.h| 16 +++
 lib/query.cc | 12 
 notmuch-search.c |  4 +++
 10 files changed, 133 insertions(+), 43 deletions(-)

-- 
2.13.2

https://jb55.com
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/6] sorting: add the ability to sort by from and subject

[William Casarin]

* add {from,subject}-{ascending,descending} sort options
---
 lib/notmuch.h| 16 
 lib/query.cc | 12 
 notmuch-search.c |  4 
 3 files changed, 32 insertions(+)

diff --git a/lib/notmuch.h b/lib/notmuch.h
index f26565f3..071bfe4d 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -765,6 +765,22 @@ typedef enum {
  */
 NOTMUCH_SORT_NEWEST_FIRST,
 /**
+ * Sort by from: in ascending order
+ */
+NOTMUCH_SORT_FROM_ASC,
+/**
+ * Sort by from: in descending order
+ */
+NOTMUCH_SORT_FROM_DESC,
+/**
+ * Sort by subject: in ascending order
+ */
+NOTMUCH_SORT_SUBJECT_ASC,
+/**
+ * Sort by subject: in descending order
+ */
+NOTMUCH_SORT_SUBJECT_DESC,
+/**
  * Sort by message-id.
  */
 NOTMUCH_SORT_MESSAGE_ID,
diff --git a/lib/query.cc b/lib/query.cc
index 9c6ecc8d..106814a8 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -327,6 +327,18 @@ _notmuch_query_search_documents (notmuch_query_t *query,
case NOTMUCH_SORT_NEWEST_FIRST:
enquire.set_sort_by_value (NOTMUCH_VALUE_TIMESTAMP, TRUE);
break;
+   case NOTMUCH_SORT_FROM_ASC:
+   enquire.set_sort_by_value (NOTMUCH_VALUE_FROM, FALSE);
+   break;
+   case NOTMUCH_SORT_FROM_DESC:
+   enquire.set_sort_by_value (NOTMUCH_VALUE_FROM, TRUE);
+   break;
+   case NOTMUCH_SORT_SUBJECT_ASC:
+   enquire.set_sort_by_value (NOTMUCH_VALUE_SUBJECT, FALSE);
+   break;
+   case NOTMUCH_SORT_SUBJECT_DESC:
+   enquire.set_sort_by_value (NOTMUCH_VALUE_SUBJECT, TRUE);
+   break;
case NOTMUCH_SORT_MESSAGE_ID:
enquire.set_sort_by_value (NOTMUCH_VALUE_MESSAGE_ID, FALSE);
break;
diff --git a/notmuch-search.c b/notmuch-search.c
index 380e9d8f..b80647e9 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -789,6 +789,10 @@ static const notmuch_opt_desc_t common_options[] = {
 { NOTMUCH_OPT_KEYWORD, _context.sort, "sort", 's',
   (notmuch_keyword_t []){ { "oldest-first", NOTMUCH_SORT_OLDEST_FIRST },
  { "newest-first", NOTMUCH_SORT_NEWEST_FIRST },
+ { "from-ascending", NOTMUCH_SORT_FROM_ASC },
+ { "from-descending", NOTMUCH_SORT_FROM_DESC },
+ { "subject-ascending", NOTMUCH_SORT_SUBJECT_ASC },
+ { "subject-descending", NOTMUCH_SORT_SUBJECT_DESC 
},
  { 0, 0 } } },
 { NOTMUCH_OPT_KEYWORD, _context.format_sel, "format", 'f',
   (notmuch_keyword_t []){ { "json", NOTMUCH_FORMAT_JSON },
-- 
2.13.2

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 2/6] sorting: update ruby bindings for from and subject

[William Casarin]

---
 bindings/ruby/init.c | 24 
 1 file changed, 24 insertions(+)

diff --git a/bindings/ruby/init.c b/bindings/ruby/init.c
index 5556b43e..ace8f666 100644
--- a/bindings/ruby/init.c
+++ b/bindings/ruby/init.c
@@ -104,6 +104,30 @@ Init_notmuch (void)
  */
 rb_define_const (mod, "SORT_NEWEST_FIRST", INT2FIX 
(NOTMUCH_SORT_NEWEST_FIRST));
 /*
+ * Document-const: Notmuch::SORT_FROM_ASC
+ *
+ * Sort query results by from in ascending order
+ */
+rb_define_const (mod, "SORT_FROM_ASC", INT2FIX (NOTMUCH_SORT_FROM_ASC));
+/*
+ * Document-const: Notmuch::SORT_FROM_DESC
+ *
+ * Sort query results by from in descending order
+ */
+rb_define_const (mod, "SORT_FROM_DESC", INT2FIX (NOTMUCH_SORT_FROM_DESC));
+/*
+ * Document-const: Notmuch::SORT_SUBJECT_ASC
+ *
+ * Sort query results by subject in ascending order
+ */
+rb_define_const (mod, "SORT_SUBJECT_ASC", INT2FIX 
(NOTMUCH_SORT_SUBJECT_ASC));
+/*
+ * Document-const: Notmuch::SORT_SUBJECT_DESC
+ *
+ * Sort query results by from in descending order
+ */
+rb_define_const (mod, "SORT_SUBJECT_DESC", INT2FIX 
(NOTMUCH_SORT_SUBJECT_DESC));
+/*
  * Document-const: Notmuch::SORT_MESSAGE_ID
  *
  * Sort query results by message id
-- 
2.13.2

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] emacs: sanitize subject in replies

[Jani Nikula]

Commit a7964c86d125 ("emacs: Sanitize authors and subjects in search
and show") added sanitization of header information for display. Do
the same for reply subjects.

This fixes the long-standing annoying artefact of certain versions of
mailman using tab as folding whitespace, leading to tabs in reply
subjects.
---
 emacs/notmuch-mua.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
index fd64b362b542..7a341ebf0588 100644
--- a/emacs/notmuch-mua.el
+++ b/emacs/notmuch-mua.el
@@ -218,7 +218,7 @@ mutiple parts get a header."
 else
 collect pair)))
  (notmuch-mua-mail (plist-get reply-headers :To)
-   (plist-get reply-headers :Subject)
+   (notmuch-sanitize (plist-get reply-headers 
:Subject))
(notmuch-headers-plist-to-alist reply-headers)
nil (notmuch-mua-get-switch-function
 
-- 
2.11.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 1/3] sorting: add the ability to sort by from and subject

[William Casarin]

David Bremner  writes:

> It seems worth mentioning that it's possible to preprocess values into
> keys (see Xapian::Enquire::set_sort_by_key). So things like Re:
> etc... could be stripped.

Hmm looks like I need to create a KeyMaker class which appears to be a
glorified (Xapian::Document -> String) callback. Shouldn't be too
difficult other than the choice of algorithm. It would have to strip:

  Re: Fwd: ... etc?

Sounds like the choice algorithm could be tricky and/or error prone?
Would need lots of tests, and like Jani said, questionable worth in
these scenarios, considering I was mainly dealing with scenarios like
these:

  From: Top News - MIT Technology Review
  From: Newest questions tagged haskell - Stack Overflow

So the current implementation would only be useful for RSS feeds,
automated bots, etc. that are non-mailing-list-email use cases.

I'm working on updates to notmuch-mode right now. I'll post a patch for
that soon for people who are interested.

- William
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 1/3] sorting: add the ability to sort by from and subject

[David Bremner]

William Casarin <j...@jb55.com> writes:

> William Casarin <j...@jb55.com> writes:
>
>> Jani Nikula <j...@nikula.org> writes:
>>
>>> The implementation seems simple enough, but what's the use case, really?
>>
>> I get all of my rss feeds sent to my inbox, I wanted to be able to group
>> similar feeds (mainly by from, sometimes subject). Alternatively if
>> there was a way to group by tags I could do it that way, but I don't tag
>> all of my individual feeds.
>>
>> If this is too obscure of a use case please feel free to ignore!
>
> A use case for the subject sort is my github tag. Since the repo name is
> at the start of the subject line, it sorts by repo so I can quickly skim
> across all the issues/prs in all the github repos I follow

It seems worth mentioning that it's possible to preprocess values into
keys (see Xapian::Enquire::set_sort_by_key). So things like Re:
etc... could be stripped.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 1/3] sorting: add the ability to sort by from and subject

[William Casarin]

William Casarin <j...@jb55.com> writes:

> Jani Nikula <j...@nikula.org> writes:
>
>> The implementation seems simple enough, but what's the use case, really?
>
> I get all of my rss feeds sent to my inbox, I wanted to be able to group
> similar feeds (mainly by from, sometimes subject). Alternatively if
> there was a way to group by tags I could do it that way, but I don't tag
> all of my individual feeds.
>
> If this is too obscure of a use case please feel free to ignore!

A use case for the subject sort is my github tag. Since the repo name is
at the start of the subject line, it sorts by repo so I can quickly skim
across all the issues/prs in all the github repos I follow

Cheers,
William

-- 
https://jb55.com
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 1/3] sorting: add the ability to sort by from and subject

[Jani Nikula]

On Mon, 04 Sep 2017, William Casarin <j...@jb55.com> wrote:
> * add {from,subject}-{ascending,descending} sort options

The implementation seems simple enough, but what's the use case, really?

When thinking about the usefulness of the feature, you have to think
about what gets indexed for from: and subject: prefixes, and how they
would get sorted. (See lib/index.cc for details.)

From: J. Random Hacker <hac...@example.com>
From: "Hacker, Random J." <hac...@example.com>
From: hac...@example.com

Subject: [notmuch] [PATCH 1/3] sorting
Subject: [PATCH 1/3] sorting
Subject: Re: [PATCH 1/3] sorting
Subject: Vast: [PATCH 1/3] sorting

With those in mind, does it still seem useful to sort by from or
subject?

If yes, consider thread sorting with some search terms, and all of the
above present in a thread. The placement of a thread in the result list
depends on which messages in the thread match the query. Even if the
matching messages are supposedly the same.

I'm not convinced.


BR,
Jani.

> ---
> I'm not sure if we want to eventually refactor ascending and descending
> into a separate option, but I decided to keep it this way for now.
>
>  lib/notmuch.h| 16 
>  lib/query.cc | 12 
>  notmuch-search.c |  4 
>  3 files changed, 32 insertions(+)
>
> diff --git a/lib/notmuch.h b/lib/notmuch.h
> index 66ecb5fc..f5764683 100644
> --- a/lib/notmuch.h
> +++ b/lib/notmuch.h
> @@ -764,6 +764,22 @@ typedef enum {
>   */
>  NOTMUCH_SORT_NEWEST_FIRST,
>  /**
> + * Sort by from: in ascending order
> + */
> +NOTMUCH_SORT_FROM_ASC,
> +/**
> + * Sort by from: in descending order
> + */
> +NOTMUCH_SORT_FROM_DESC,
> +/**
> + * Sort by subject: in ascending order
> + */
> +NOTMUCH_SORT_SUBJECT_ASC,
> +/**
> + * Sort by subject: in descending order
> + */
> +NOTMUCH_SORT_SUBJECT_DESC,
> +/**
>   * Sort by message-id.
>   */
>  NOTMUCH_SORT_MESSAGE_ID,
> diff --git a/lib/query.cc b/lib/query.cc
> index 9c6ecc8d..106814a8 100644
> --- a/lib/query.cc
> +++ b/lib/query.cc
> @@ -327,6 +327,18 @@ _notmuch_query_search_documents (notmuch_query_t *query,
>   case NOTMUCH_SORT_NEWEST_FIRST:
>   enquire.set_sort_by_value (NOTMUCH_VALUE_TIMESTAMP, TRUE);
>   break;
> + case NOTMUCH_SORT_FROM_ASC:
> + enquire.set_sort_by_value (NOTMUCH_VALUE_FROM, FALSE);
> + break;
> + case NOTMUCH_SORT_FROM_DESC:
> + enquire.set_sort_by_value (NOTMUCH_VALUE_FROM, TRUE);
> + break;
> + case NOTMUCH_SORT_SUBJECT_ASC:
> + enquire.set_sort_by_value (NOTMUCH_VALUE_SUBJECT, FALSE);
> + break;
> + case NOTMUCH_SORT_SUBJECT_DESC:
> + enquire.set_sort_by_value (NOTMUCH_VALUE_SUBJECT, TRUE);
> + break;
>   case NOTMUCH_SORT_MESSAGE_ID:
>   enquire.set_sort_by_value (NOTMUCH_VALUE_MESSAGE_ID, FALSE);
>   break;
> diff --git a/notmuch-search.c b/notmuch-search.c
> index 380e9d8f..b80647e9 100644
> --- a/notmuch-search.c
> +++ b/notmuch-search.c
> @@ -789,6 +789,10 @@ static const notmuch_opt_desc_t common_options[] = {
>  { NOTMUCH_OPT_KEYWORD, _context.sort, "sort", 's',
>(notmuch_keyword_t []){ { "oldest-first", NOTMUCH_SORT_OLDEST_FIRST },
>     { "newest-first", NOTMUCH_SORT_NEWEST_FIRST },
> +   { "from-ascending", NOTMUCH_SORT_FROM_ASC },
> +   { "from-descending", NOTMUCH_SORT_FROM_DESC },
> +   { "subject-ascending", NOTMUCH_SORT_SUBJECT_ASC },
> +   { "subject-descending", NOTMUCH_SORT_SUBJECT_DESC 
> },
> { 0, 0 } } },
>  { NOTMUCH_OPT_KEYWORD, _context.format_sel, "format", 'f',
>(notmuch_keyword_t []){ { "json", NOTMUCH_FORMAT_JSON },
> -- 
> 2.13.2
>
> ___
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 2/3] sorting: update ruby bindings for from and subject

[William Casarin]

---
 bindings/ruby/init.c | 24 
 1 file changed, 24 insertions(+)

diff --git a/bindings/ruby/init.c b/bindings/ruby/init.c
index 5556b43e..ace8f666 100644
--- a/bindings/ruby/init.c
+++ b/bindings/ruby/init.c
@@ -104,6 +104,30 @@ Init_notmuch (void)
  */
 rb_define_const (mod, "SORT_NEWEST_FIRST", INT2FIX 
(NOTMUCH_SORT_NEWEST_FIRST));
 /*
+ * Document-const: Notmuch::SORT_FROM_ASC
+ *
+ * Sort query results by from in ascending order
+ */
+rb_define_const (mod, "SORT_FROM_ASC", INT2FIX (NOTMUCH_SORT_FROM_ASC));
+/*
+ * Document-const: Notmuch::SORT_FROM_DESC
+ *
+ * Sort query results by from in descending order
+ */
+rb_define_const (mod, "SORT_FROM_DESC", INT2FIX (NOTMUCH_SORT_FROM_DESC));
+/*
+ * Document-const: Notmuch::SORT_SUBJECT_ASC
+ *
+ * Sort query results by subject in ascending order
+ */
+rb_define_const (mod, "SORT_SUBJECT_ASC", INT2FIX 
(NOTMUCH_SORT_SUBJECT_ASC));
+/*
+ * Document-const: Notmuch::SORT_SUBJECT_DESC
+ *
+ * Sort query results by from in descending order
+ */
+rb_define_const (mod, "SORT_SUBJECT_DESC", INT2FIX 
(NOTMUCH_SORT_SUBJECT_DESC));
+/*
  * Document-const: Notmuch::SORT_MESSAGE_ID
  *
  * Sort query results by message id
-- 
2.13.2

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/3] sorting: add the ability to sort by from and subject

[William Casarin]

* add {from,subject}-{ascending,descending} sort options
---
I'm not sure if we want to eventually refactor ascending and descending
into a separate option, but I decided to keep it this way for now.

 lib/notmuch.h| 16 
 lib/query.cc | 12 
 notmuch-search.c |  4 
 3 files changed, 32 insertions(+)

diff --git a/lib/notmuch.h b/lib/notmuch.h
index 66ecb5fc..f5764683 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -764,6 +764,22 @@ typedef enum {
  */
 NOTMUCH_SORT_NEWEST_FIRST,
 /**
+ * Sort by from: in ascending order
+ */
+NOTMUCH_SORT_FROM_ASC,
+/**
+ * Sort by from: in descending order
+ */
+NOTMUCH_SORT_FROM_DESC,
+/**
+ * Sort by subject: in ascending order
+ */
+NOTMUCH_SORT_SUBJECT_ASC,
+/**
+ * Sort by subject: in descending order
+ */
+NOTMUCH_SORT_SUBJECT_DESC,
+/**
  * Sort by message-id.
  */
 NOTMUCH_SORT_MESSAGE_ID,
diff --git a/lib/query.cc b/lib/query.cc
index 9c6ecc8d..106814a8 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -327,6 +327,18 @@ _notmuch_query_search_documents (notmuch_query_t *query,
case NOTMUCH_SORT_NEWEST_FIRST:
enquire.set_sort_by_value (NOTMUCH_VALUE_TIMESTAMP, TRUE);
break;
+   case NOTMUCH_SORT_FROM_ASC:
+   enquire.set_sort_by_value (NOTMUCH_VALUE_FROM, FALSE);
+   break;
+   case NOTMUCH_SORT_FROM_DESC:
+   enquire.set_sort_by_value (NOTMUCH_VALUE_FROM, TRUE);
+   break;
+   case NOTMUCH_SORT_SUBJECT_ASC:
+   enquire.set_sort_by_value (NOTMUCH_VALUE_SUBJECT, FALSE);
+   break;
+   case NOTMUCH_SORT_SUBJECT_DESC:
+   enquire.set_sort_by_value (NOTMUCH_VALUE_SUBJECT, TRUE);
+   break;
case NOTMUCH_SORT_MESSAGE_ID:
enquire.set_sort_by_value (NOTMUCH_VALUE_MESSAGE_ID, FALSE);
break;
diff --git a/notmuch-search.c b/notmuch-search.c
index 380e9d8f..b80647e9 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -789,6 +789,10 @@ static const notmuch_opt_desc_t common_options[] = {
 { NOTMUCH_OPT_KEYWORD, _context.sort, "sort", 's',
   (notmuch_keyword_t []){ { "oldest-first", NOTMUCH_SORT_OLDEST_FIRST },
  { "newest-first", NOTMUCH_SORT_NEWEST_FIRST },
+ { "from-ascending", NOTMUCH_SORT_FROM_ASC },
+ { "from-descending", NOTMUCH_SORT_FROM_DESC },
+ { "subject-ascending", NOTMUCH_SORT_SUBJECT_ASC },
+ { "subject-descending", NOTMUCH_SORT_SUBJECT_DESC 
},
  { 0, 0 } } },
 { NOTMUCH_OPT_KEYWORD, _context.format_sel, "format", 'f',
   (notmuch_keyword_t []){ { "json", NOTMUCH_FORMAT_JSON },
-- 
2.13.2

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch-search: sort by subject or author?

[William Casarin]

David Bremner <da...@tethera.net> writes:

> I think it's not really possible at the moment. If you want this to work
> with large searches then it probably needs to be done at the CLI level
> (see [1] for work in progress adding sorting by file size).
>
> Luckily 'From' and 'Subject' are already value slots in the database,
> which means that nothing should have to change in the database to add
> those sort orders (i.e. it should be less work than [1]).
>
> [1] id:20170518222708.30032-1-...@adirat.com

I have a patch that is working, I'll send it over soon. Thanks!

- William

-- 
https://jb55.com
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch-search: sort by subject or author?

[David Bremner]

William Casarin <j...@jb55.com> writes:

> Hey there,
>
> Is there a way to sort by subject or author in emacs/notmuch-search? I
> find myself wanting to do this a lot. My particular use case is rss
> feeds, where I have many different feeds in my rss tag that I would like
> to group together.
>
> If not, I am interested in adding this feature.
>
> Thanks,
> William

I think it's not really possible at the moment. If you want this to work
with large searches then it probably needs to be done at the CLI level
(see [1] for work in progress adding sorting by file size).

Luckily 'From' and 'Subject' are already value slots in the database,
which means that nothing should have to change in the database to add
those sort orders (i.e. it should be less work than [1]).

[1] id:20170518222708.30032-1-...@adirat.com
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

notmuch-search: sort by subject or author?

[William Casarin]

Hey there,

Is there a way to sort by subject or author in emacs/notmuch-search? I
find myself wanting to do this a lot. My particular use case is rss
feeds, where I have many different feeds in my rss tag that I would like
to group together.

If not, I am interested in adding this feature.

Thanks,
William
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 3/5] test: known broken test for subject after reindexing

[David Bremner]

In [1], Mark gave a test that was behaving strangly. This turns out to
be specific to reindexing.  I suppose one could argue that picking the
lexicographically last file name is a defensible choice, but it's
almost as easy to take the first, which seems more intuitive. So mark
the current situation as broken.

[1]: id:1503859703-2973-1-git-send-email-markwalters1...@gmail.com
---
 test/T670-duplicate-mid.sh | 8 
 1 file changed, 8 insertions(+)

diff --git a/test/T670-duplicate-mid.sh b/test/T670-duplicate-mid.sh
index d2f89432..d4c1d1c2 100755
--- a/test/T670-duplicate-mid.sh
+++ b/test/T670-duplicate-mid.sh
@@ -40,6 +40,14 @@ notmuch reindex '*'
 notmuch search --output=files "sekrit" | notmuch_dir_sanitize > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest 'reindex choses subject from first filename'
+test_subtest_known_broken
+cat < EXPECTED
+thread:XXX   2001-01-05 [1/1(3)] Notmuch Test Suite; message 0 (inbox unread)
+EOF
+notmuch search id:duplicate | notmuch_search_sanitize > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
 rm ${MAIL_DIR}/copy0
 test_begin_subtest 'Deleted first duplicate file does not stop notmuch show 
from working'
 output=$(notmuch show --body=false --format=json id:duplicate |
-- 
2.13.2

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 5/5] test/duplicate-mid: check for subject with notmuch-show

[David Bremner]

In [1] Mark showed that the the current code (d7a49e81) is not
consistent in it's handling of subjects of messages with duplicate
message-ids (or in notmuch-speak, of messages with multiple files).
notmuch-search uses indexing order and explicitedly preserves the
first. notmuch-show (apparently) uses alphabetical (or at least xapian
term order) of filenames. In a perfect world we would probably report
all subjects in the json output; at the very least we should be
consistent.

[1]: id:87378dny3d@qmul.ac.uk
---
 test/T670-duplicate-mid.sh | 25 +
 1 file changed, 25 insertions(+)

diff --git a/test/T670-duplicate-mid.sh b/test/T670-duplicate-mid.sh
index ce010cf7..21a9689a 100755
--- a/test/T670-duplicate-mid.sh
+++ b/test/T670-duplicate-mid.sh
@@ -13,6 +13,31 @@ EOF
 notmuch search id:duplicate | notmuch_search_sanitize > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest 'First subject preserved in notmuch-show (json)'
+test_subtest_known_broken
+output=$(notmuch show --body=false --format=json id:duplicate | 
notmuch_json_show_sanitize)
+expected='[[[{
+"id": "X",
+"match": true,
+"excluded": false,
+"filename": [
+"'"${MAIL_DIR}"/copy0'",
+"'"${MAIL_DIR}"/copy1'",
+"'"${MAIL_DIR}"/copy2'"
+],
+"timestamp": 42,
+"date_relative": "2001-01-05",
+"tags": ["inbox","unread"],
+"headers": {
+"Subject": "message 1",
+"From": "Notmuch Test Suite <test_su...@notmuchmail.org>",
+"To": "Notmuch Test Suite <test_su...@notmuchmail.org>",
+"Date": "GENERATED_DATE"
+}
+ },
+['
+test_expect_equal_json "$output" "$expected"
+
 test_begin_subtest 'Search for second subject'
 cat <EXPECTED
 MAIL_DIR/copy0
-- 
2.13.2

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] test: check for subject consistency between search and show

[David Bremner]

In [1] Mark showed that the the current code (d7a49e81) is not
consistent in it's handling of subjects of messages with duplicate
message-ids (or in notmuch-speak, of messages with multiple files).
notmuch-search uses indexing order and explicitedly preserves the
first. notmuch-show (apparently) uses alphabetical (or at least xapian
term order) of filenames. In a perfect world we would probably report
all subjects in the json output; at the very least we should be
consistent.

[1]: id:87378dny3d@qmul.ac.uk
---
 test/T670-duplicate-mid.sh | 62 +-
 1 file changed, 56 insertions(+), 6 deletions(-)

diff --git a/test/T670-duplicate-mid.sh b/test/T670-duplicate-mid.sh
index decbc0a4..55ebde38 100755
--- a/test/T670-duplicate-mid.sh
+++ b/test/T670-duplicate-mid.sh
@@ -12,6 +12,29 @@ EOF
 notmuch search id:duplicate | notmuch_search_sanitize > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+test_begin_subtest 'First subject preserved in notmuch-show (json)'
+output=$(notmuch show --body=false --format=json id:duplicate | 
notmuch_json_show_sanitize)
+expected='[[[{
+"id": "X",
+"match": true,
+"excluded": false,
+"filename": [
+"'"${MAIL_DIR}"/copy1'",
+"'"${MAIL_DIR}"/copy2'"
+],
+"timestamp": 42,
+"date_relative": "2001-01-05",
+"tags": ["inbox","unread"],
+"headers": {
+"Subject": "message 1",
+"From": "Notmuch Test Suite <test_su...@notmuchmail.org>",
+"To": "Notmuch Test Suite <test_su...@notmuchmail.org>",
+"Date": "GENERATED_DATE"
+}
+ },
+['
+test_expect_equal_json "$output" "$expected"
+
 test_begin_subtest 'Search for second subject'
 cat <EXPECTED
 MAIL_DIR/copy1
@@ -37,25 +60,52 @@ notmuch reindex '*'
 notmuch search --output=files "sekrit" | notmuch_dir_sanitize > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
-rm ${MAIL_DIR}/copy1
+add_message '[id]="duplicate"' '[subject]="reorder file names"' 
'[filename]=00-copy4'
+test_begin_subtest 'First subject preserved in notmuch-show (json), file order'
+test_subtest_known_broken
+output=$(notmuch show --body=false --format=json id:duplicate | 
notmuch_json_show_sanitize)
+expected='[[[{
+    "id": "X",
+"match": true,
+"excluded": false,
+"filename": [
+"'"${MAIL_DIR}"/00-copy4'",
+"'"${MAIL_DIR}"/copy1'",
+"'"${MAIL_DIR}"/copy2'"
+],
+"timestamp": 42,
+"date_relative": "2001-01-05",
+"tags": ["inbox","unread"],
+"headers": {
+"Subject": "message 1",
+"From": "Notmuch Test Suite <test_su...@notmuchmail.org>",
+"To": "Notmuch Test Suite <test_su...@notmuchmail.org>",
+"Date": "GENERATED_DATE"
+}
+ },
+['
+test_expect_equal_json "$output" "$expected"
+
+rm ${MAIL_DIR}/00-copy4
 test_begin_subtest 'Deleted first duplicate file does not stop notmuch show 
from working'
-output=$(notmuch show --body=false --format=json id:duplicate)
+output=$(notmuch show --body=false --format=json id:duplicate | 
notmuch_json_show_sanitize)
 expected='[[[{
-"id": "'duplicate'",
+"id": "X",
 "match": true,
 "excluded": false,
 "filename": [
+"'"${MAIL_DIR}"/00-copy4'",
 "'"${MAIL_DIR}"/copy1'",
 "'"${MAIL_DIR}"/copy2'"
 ],
-"timestamp": 978709435,
+"timestamp": 42,
 "date_relative": "2001-01-05",
 "tags": ["inbox","unread"],
 "headers": {
-"Subject": "message 2",
+"Subject": "message 1",
 "From": "Notmuch Test Suite <test_su...@notmuchmail.org>",
 "To": "Notmuch Test Suite <test_su...@notmuchmail.org>",
-"Date": "Fri, 05 Jan 2001 15:43:55 +"
+"Date": "GENERATED_DATE"
 }
  },
 ['
-- 
2.14.1

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[no subject]

[David Bremner]

This is the first allegedly complete version of support for gmime 3

It obsoletes several partial series [1][2]

   - id:20170602022232.17264-1-da...@tethera.net
   - patches 9-11, starting at id:20170527165121.9654-10-da...@tethera.net

There still remains the question of whether we should include

  [PATCH 10/23] test: test parsing of malformed from addresses

I'm not very optimistic about a fix coming along, and it's not
obviously a big issue for users.

The following are crypto related, and could particularly use review

[PATCH 01/23] emacs: convert to use format-version 3
[PATCH 02/23] devel/schemata: describe version 4
[PATCH 03/23] cli: implement structured output version 4
[PATCH 07/23] test/crypto: mark extra space in userid as a bug in
[PATCH 08/23] test: add test for modified pgp/mime signed message
[PATCH 09/23] test/crypto: add test for corrupted signatures
[PATCH 16/23] cli: generalize use of GMIME_SIGNATURE_{ERROR,STATUS}
[PATCH 17/23] cli: hide rename of GMimeCertificateTrust
[PATCH 22/23] cli: make keyid from fingerprint in gmime 3.0
[PATCH 23/23] cli: wrap getting uid
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[no subject]

[Tomi Ollila]

This implementation adds add_exit_function (and rm_exit_function)
which can also be used for other things in the future.

Now that I did this simpler way would be to just check for
existence of $GNUPGHOME for indication to exit gpg processes.

If that path is taken this series can be used for future reference
if need for atexit functionality arises.

From Tomi Ollila <tomi.oll...@iki.fi> # This line is ignored.
From: Tomi Ollila <tomi.oll...@iki.fi>
Subject: stop gpg-agent (among other) processes at test module exit
In-Reply-To: 

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 3/4] test: add known broken tests wildcard search in from and subject

[David Bremner]

David Bremner  writes:

> This was broken by the addition of regexp searching. The detection of
> wildcards is not currently done in the recursive call to parse_query,
> because of quoting issues.

Patches 3 and 4 pushed to release and master.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/2] test: add known broken test for null from: and subject: query

[David Bremner]

These queries currently fail with field processors enabled because the
code expects a non-empty string.
---
 test/T650-regexp-query.sh | 20 
 1 file changed, 20 insertions(+)

diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index 61739e87..f2ae1387 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -11,6 +11,26 @@ fi
 
 notmuch search --output=messages from:cworth > cworth.msg-ids
 
+# these headers will generate no document terms
+add_message '[from]="-" [subject]="empty from"'
+add_message '[subject]="-"'
+
+test_begin_subtest "null from: search"
+test_subtest_known_broken
+notmuch search 'from:""' | notmuch_search_sanitize > OUTPUT
+cat < EXPECTED
+thread:XXX   2001-01-05 [1/1] -; empty from (inbox unread)
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "null subject: search"
+test_subtest_known_broken
+notmuch search 'subject:""' | notmuch_search_sanitize > OUTPUT
+cat < EXPECTED
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; - (inbox unread)
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 test_begin_subtest "xapian wildcard search for from:"
 notmuch search --output=messages 'from:cwo*' > OUTPUT
 test_expect_equal_file cworth.msg-ids OUTPUT
-- 
2.11.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/4] test: add known broken test for empty from: and subject: query

[David Bremner]

These queries currently fail with field processors enabled because the
code expects a non-empty string.
---
 test/T650-regexp-query.sh | 9 +
 1 file changed, 9 insertions(+)

diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index df48ab82..b1d84439 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -11,6 +11,15 @@ fi
 
 notmuch search --output=messages from:cworth > cworth.msg-ids
 
+test_begin_subtest "empty from: search"
+test_subtest_known_broken
+notmuch search --output=messages 'from:""' and from:cworth > OUTPUT
+test_expect_equal_file cworth.msg-ids OUTPUT
+
+test_begin_subtest "empty subject: search"
+test_subtest_known_broken
+notmuch search --output=messages 'subject:""' and from:cworth > OUTPUT
+test_expect_equal_file cworth.msg-ids OUTPUT
 test_begin_subtest "regexp from search, case sensitive"
 notmuch search --output=messages from:/carl/ > OUTPUT
 test_expect_equal_file /dev/null OUTPUT
-- 
2.11.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 3/4] test: add known broken tests wildcard search in from and subject

[David Bremner]

This was broken by the addition of regexp searching. The detection of
wildcards is not currently done in the recursive call to parse_query,
because of quoting issues.
---
 test/T650-regexp-query.sh | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index 049477b4..ba4a64e0 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -18,6 +18,16 @@ test_expect_equal_file cworth.msg-ids OUTPUT
 test_begin_subtest "empty subject: search"
 notmuch search --output=messages 'subject:""' and from:cworth > OUTPUT
 test_expect_equal_file cworth.msg-ids OUTPUT
+
+test_begin_subtest "xapian wildcard search for from:"
+test_subtest_known_broken
+notmuch search --output=messages 'from:cwo*' > OUTPUT
+test_expect_equal_file cworth.msg-ids OUTPUT
+
+test_begin_subtest "xapian wildcard search for subject:"
+test_subtest_known_broken
+test_expect_equal $(notmuch count 'subject:count*') 1
+
 test_begin_subtest "regexp from search, case sensitive"
 notmuch search --output=messages from:/carl/ > OUTPUT
 test_expect_equal_file /dev/null OUTPUT
-- 
2.11.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] fixup! lib: regexp matching in 'subject' and 'from'

[David Bremner]

---
 lib/regexp-fields.cc | 5 +
 1 file changed, 5 insertions(+)

diff --git a/lib/regexp-fields.cc b/lib/regexp-fields.cc
index b2b39504..65108e38 100644
--- a/lib/regexp-fields.cc
+++ b/lib/regexp-fields.cc
@@ -62,6 +62,11 @@ RegexpPostingSource::init (const Xapian::Database )
 it_ = db_.valuestream_begin (slot_);
 end_ = db.valuestream_end (slot_);
 started_ = false;
+
+/* make sure we start on a matching value */
+while (!at_end() && regexec (_, (*it_).c_str (), 0, NULL, 0) != 0) {
+   ++it_;
+}
 }
 
 Xapian::doccount
-- 
2.11.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 2/4] lib: regexp matching in 'subject' and 'from'

[David Bremner]

the idea is that you can run

% notmuch search subject://
% notmuch search from://

or

% notmuch search subject:"your usual phrase search"
% notmuch search from:"usual phrase search"

This feature is only available with recent Xapian, specifically
support for field processors is needed.

It should work with bindings, since it extends the query parser.

This is easy to extend for other value slots, but currently the only
value slots are date, message_id, from, subject, and last_mod. Date is
already searchable;  message_id is left for a followup commit.

This was originally written by Austin Clements, and ported to Xapian
field processors (from Austin's custom query parser) by yours truly.
---
 doc/man7/notmuch-search-terms.rst |  25 ++-
 lib/Makefile.local|   1 +
 lib/database.cc   |   9 ++-
 lib/regexp-fields.cc  | 144 ++
 lib/regexp-fields.h   |  77 
 test/T650-regexp-query.sh |  82 ++
 6 files changed, 332 insertions(+), 6 deletions(-)
 create mode 100644 lib/regexp-fields.cc
 create mode 100644 lib/regexp-fields.h
 create mode 100755 test/T650-regexp-query.sh

diff --git a/doc/man7/notmuch-search-terms.rst 
b/doc/man7/notmuch-search-terms.rst
index de93d733..47cab48d 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -34,10 +34,14 @@ indicate user-supplied values):
 
 -  from:
 
+-  from://
+
 -  to:
 
 -  subject:
 
+-  subject://
+
 -  attachment:
 
 -  mimetype:
@@ -71,6 +75,15 @@ subject of an email. Searching for a phrase in the subject 
is supported
 by including quotation marks around the phrase, immediately following
 **subject:**.
 
+If notmuch is built with **Xapian Field Processors** (see below) the
+**from:** and **subject** prefix can be also used to restrict the
+results to those whose from/subject value matches a regular expression
+(see **regex(7)**) delimited with //.
+
+::
+
+   notmuch search 'from:/bob@.*[.]example[.]com/'
+
 The **attachment:** prefix can be used to search for specific filenames
 (or extensions) of attachments to email messages.
 
@@ -220,13 +233,18 @@ Boolean and Probabilistic Prefixes
 --
 
 Xapian (and hence notmuch) prefixes are either **boolean**, supporting
-exact matches like "tag:inbox"  or **probabilistic**, supporting a more 
flexible **term** based searching. The prefixes currently supported by notmuch 
are as follows.
-
+exact matches like "tag:inbox" or **probabilistic**, supporting a more
+flexible **term** based searching. Certain **special** prefixes are
+processed by notmuch in a way not stricly fitting either of Xapian's
+built in styles. The prefixes currently supported by notmuch are as
+follows.
 
 Boolean
**tag:**, **id:**, **thread:**, **folder:**, **path:**, **property:**
 Probabilistic
-   **from:**, **to:**, **subject:**, **attachment:**, **mimetype:**
+  **to:**, **attachment:**, **mimetype:**
+Special
+   **from:**, **query:**, **subject:**
 
 Terms and phrases
 -
@@ -396,6 +414,7 @@ Currently the following features require field processor 
support:
 
 - non-range date queries, e.g. "date:today"
 - named queries e.g. "query:my_special_query"
+- regular expression searches, e.g. "subject:/^\\[SPAM\\]/"
 
 SEE ALSO
 
diff --git a/lib/Makefile.local b/lib/Makefile.local
index b77e5780..cd92fc79 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -52,6 +52,7 @@ libnotmuch_cxx_srcs = \
$(dir)/query.cc \
$(dir)/query-fp.cc  \
$(dir)/config.cc\
+   $(dir)/regexp-fields.cc \
$(dir)/thread.cc
 
 libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
diff --git a/lib/database.cc b/lib/database.cc
index fa4c3116..573c9fe0 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -21,6 +21,7 @@
 #include "database-private.h"
 #include "parse-time-vrp.h"
 #include "query-fp.h"
+#include "regexp-fields.h"
 #include "string-util.h"
 
 #include 
@@ -277,7 +278,8 @@ prefix_t prefix_table[] = {
NOTMUCH_FIELD_PROCESSOR },
 #endif
 { "from",  "XFROM",NOTMUCH_FIELD_EXTERNAL |
-   NOTMUCH_FIELD_PROBABILISTIC },
+   NOTMUCH_FIELD_PROBABILISTIC |
+   NOTMUCH_FIELD_PROCESSOR },
 { "to","XTO",  NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC },
 { "attachment","XATTACHMENT",  NOTMUCH_FIELD_EXTERNAL |
@@ -285,7 +287,8 @@ prefix_t prefix

Re: [patch v5 4/6] lib: regexp matching in 'subject' and 'from'

[Jani Nikula]

On Thu, 16 Feb 2017, David Bremner <da...@tethera.net> wrote:
> the idea is that you can run
>
> % notmuch search subject://
> % notmuch search from://
>
> or
>
> % notmuch search subject:"your usual phrase search"
> % notmuch search from:"usual phrase search"
>
> This feature is only available with recent Xapian, specifically
> support for field processors is needed.
>
> It should work with bindings, since it extends the query parser.
>
> This is easy to extend for other value slots, but currently the only
> value slots are date, message_id, from, subject, and last_mod. Date is
> already searchable;  message_id is left for a followup commit.
>
> This was originally written by Austin Clements, and ported to Xapian
> field processors (from Austin's custom query parser) by yours truly.
> ---
>  doc/man7/notmuch-search-terms.rst |  25 ++-
>  lib/Makefile.local|   1 +
>  lib/database.cc   |  11 +--
>  lib/regexp-fields.cc  | 144 
> ++
>  lib/regexp-fields.h   |  77 
>  test/T630-regexp-query.sh |  81 +
>  6 files changed, 332 insertions(+), 7 deletions(-)
>  create mode 100644 lib/regexp-fields.cc
>  create mode 100644 lib/regexp-fields.h
>  create mode 100755 test/T630-regexp-query.sh
>
> diff --git a/doc/man7/notmuch-search-terms.rst 
> b/doc/man7/notmuch-search-terms.rst
> index de93d733..47cab48d 100644
> --- a/doc/man7/notmuch-search-terms.rst
> +++ b/doc/man7/notmuch-search-terms.rst
> @@ -34,10 +34,14 @@ indicate user-supplied values):
>  
>  -  from:
>  
> +-  from://
> +
>  -  to:
>  
>  -  subject:
>  
> +-  subject://
> +
>  -  attachment:
>  
>  -  mimetype:
> @@ -71,6 +75,15 @@ subject of an email. Searching for a phrase in the subject 
> is supported
>  by including quotation marks around the phrase, immediately following
>  **subject:**.
>  
> +If notmuch is built with **Xapian Field Processors** (see below) the
> +**from:** and **subject** prefix can be also used to restrict the
> +results to those whose from/subject value matches a regular expression
> +(see **regex(7)**) delimited with //.
> +
> +::
> +
> +   notmuch search 'from:/bob@.*[.]example[.]com/'
> +
>  The **attachment:** prefix can be used to search for specific filenames
>  (or extensions) of attachments to email messages.
>  
> @@ -220,13 +233,18 @@ Boolean and Probabilistic Prefixes
>  --
>  
>  Xapian (and hence notmuch) prefixes are either **boolean**, supporting
> -exact matches like "tag:inbox"  or **probabilistic**, supporting a more 
> flexible **term** based searching. The prefixes currently supported by 
> notmuch are as follows.
> -
> +exact matches like "tag:inbox" or **probabilistic**, supporting a more
> +flexible **term** based searching. Certain **special** prefixes are
> +processed by notmuch in a way not stricly fitting either of Xapian's
> +built in styles. The prefixes currently supported by notmuch are as
> +follows.
>  
>  Boolean
> **tag:**, **id:**, **thread:**, **folder:**, **path:**, **property:**
>  Probabilistic
> -   **from:**, **to:**, **subject:**, **attachment:**, **mimetype:**
> +  **to:**, **attachment:**, **mimetype:**
> +Special
> +   **from:**, **query:**, **subject:**
>  
>  Terms and phrases
>  -
> @@ -396,6 +414,7 @@ Currently the following features require field processor 
> support:
>  
>  - non-range date queries, e.g. "date:today"
>  - named queries e.g. "query:my_special_query"
> +- regular expression searches, e.g. "subject:/^\\[SPAM\\]/"
>  
>  SEE ALSO
>  
> diff --git a/lib/Makefile.local b/lib/Makefile.local
> index b77e5780..cd92fc79 100644
> --- a/lib/Makefile.local
> +++ b/lib/Makefile.local
> @@ -52,6 +52,7 @@ libnotmuch_cxx_srcs =   \
>   $(dir)/query.cc \
>   $(dir)/query-fp.cc  \
>   $(dir)/config.cc\
> + $(dir)/regexp-fields.cc \
>   $(dir)/thread.cc
>  
>  libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) 
> $(libnotmuch_cxx_srcs:.cc=.o)
> diff --git a/lib/database.cc b/lib/database.cc
> index 450ee295..ee971f32 100644
> --- a/lib/database.cc
> +++ b/lib/database.cc
> @@ -21,6 +21,7 @@
>  #include "database-private.h"
>  #include "parse-time-vrp.h"
>  #include "query-fp.h"
> +#include "regexp-fields.h"
>  #include "string-util.h"
>  
>  #include 
> 

[patch v5 4/6] lib: regexp matching in 'subject' and 'from'

[David Bremner]

the idea is that you can run

% notmuch search subject://
% notmuch search from://

or

% notmuch search subject:"your usual phrase search"
% notmuch search from:"usual phrase search"

This feature is only available with recent Xapian, specifically
support for field processors is needed.

It should work with bindings, since it extends the query parser.

This is easy to extend for other value slots, but currently the only
value slots are date, message_id, from, subject, and last_mod. Date is
already searchable;  message_id is left for a followup commit.

This was originally written by Austin Clements, and ported to Xapian
field processors (from Austin's custom query parser) by yours truly.
---
 doc/man7/notmuch-search-terms.rst |  25 ++-
 lib/Makefile.local|   1 +
 lib/database.cc   |  11 +--
 lib/regexp-fields.cc  | 144 ++
 lib/regexp-fields.h   |  77 
 test/T630-regexp-query.sh |  81 +
 6 files changed, 332 insertions(+), 7 deletions(-)
 create mode 100644 lib/regexp-fields.cc
 create mode 100644 lib/regexp-fields.h
 create mode 100755 test/T630-regexp-query.sh

diff --git a/doc/man7/notmuch-search-terms.rst 
b/doc/man7/notmuch-search-terms.rst
index de93d733..47cab48d 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -34,10 +34,14 @@ indicate user-supplied values):
 
 -  from:
 
+-  from://
+
 -  to:
 
 -  subject:
 
+-  subject://
+
 -  attachment:
 
 -  mimetype:
@@ -71,6 +75,15 @@ subject of an email. Searching for a phrase in the subject 
is supported
 by including quotation marks around the phrase, immediately following
 **subject:**.
 
+If notmuch is built with **Xapian Field Processors** (see below) the
+**from:** and **subject** prefix can be also used to restrict the
+results to those whose from/subject value matches a regular expression
+(see **regex(7)**) delimited with //.
+
+::
+
+   notmuch search 'from:/bob@.*[.]example[.]com/'
+
 The **attachment:** prefix can be used to search for specific filenames
 (or extensions) of attachments to email messages.
 
@@ -220,13 +233,18 @@ Boolean and Probabilistic Prefixes
 --
 
 Xapian (and hence notmuch) prefixes are either **boolean**, supporting
-exact matches like "tag:inbox"  or **probabilistic**, supporting a more 
flexible **term** based searching. The prefixes currently supported by notmuch 
are as follows.
-
+exact matches like "tag:inbox" or **probabilistic**, supporting a more
+flexible **term** based searching. Certain **special** prefixes are
+processed by notmuch in a way not stricly fitting either of Xapian's
+built in styles. The prefixes currently supported by notmuch are as
+follows.
 
 Boolean
**tag:**, **id:**, **thread:**, **folder:**, **path:**, **property:**
 Probabilistic
-   **from:**, **to:**, **subject:**, **attachment:**, **mimetype:**
+  **to:**, **attachment:**, **mimetype:**
+Special
+   **from:**, **query:**, **subject:**
 
 Terms and phrases
 -
@@ -396,6 +414,7 @@ Currently the following features require field processor 
support:
 
 - non-range date queries, e.g. "date:today"
 - named queries e.g. "query:my_special_query"
+- regular expression searches, e.g. "subject:/^\\[SPAM\\]/"
 
 SEE ALSO
 
diff --git a/lib/Makefile.local b/lib/Makefile.local
index b77e5780..cd92fc79 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -52,6 +52,7 @@ libnotmuch_cxx_srcs = \
$(dir)/query.cc \
$(dir)/query-fp.cc  \
$(dir)/config.cc\
+   $(dir)/regexp-fields.cc \
$(dir)/thread.cc
 
 libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
diff --git a/lib/database.cc b/lib/database.cc
index 450ee295..ee971f32 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -21,6 +21,7 @@
 #include "database-private.h"
 #include "parse-time-vrp.h"
 #include "query-fp.h"
+#include "regexp-fields.h"
 #include "string-util.h"
 
 #include 
@@ -277,7 +278,8 @@ prefix_t prefix_table[] = {
NOTMUCH_FIELD_PROCESSOR },
 #endif
 { "from",  "XFROM",NOTMUCH_FIELD_EXTERNAL |
-   NOTMUCH_FIELD_PROBABILISTIC },
+   NOTMUCH_FIELD_PROBABILISTIC |
+   NOTMUCH_FIELD_PROCESSOR },
 { "to","XTO",  NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC },
 { "attachment","XATTACHMENT",  NOTMUCH_FIELD_EXTERNAL |
@@ -285,7 +287,8 @@ prefix_t prefix

Re: [Patch v4] lib: regexp matching in 'subject' and 'from'

[David Bremner]

Mark Walters <markwalters1...@gmail.com> writes:

>
> Hi
>
> Broadly I like the backslash escaping option. Two thoughts: can any
> fields (from/subject/message-id) start with a "\" anyway? I think not
> but thought it worth checking.

From and subject are probablistic xapian fields, so punctuation is
essentially ignored by the query parser. That being said, nothing
prevents subjects from starting with /.  According to my reading of
rfc5322, conforming message ids cannot contain any of '()<>[]:;@\,."'

> Secondly, message-id is often round-tripped, that is output from notmuch
> and then fed back to notmuch. Do we want to escape the output as above
> before printing in any cases? My view is that if we output the
> message-id prefixed with "id:" then we should escape it (which applies
> with --output=messages --format=text), but if we don't print the "id:"
> part then we shouldn't (eg with --format=json). A similar thing would
> apply to emacs: if it is a normal stash then escape the id, but if it is
> a "bare stash" then do not.

Yes that sounds about right. Do we actually output from and subject with
prefixes attached to them? I have the feeling not.

>
> Actually, one more thing: it would be a shame to block or significantly
> delay the series for such a corner case.
>

If it's _only_ the output of notmuch search --output=messages, then I
guess it's doable.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [Patch v4] lib: regexp matching in 'subject' and 'from'

[Mark Walters]

On Thu, 09 Feb 2017, David Bremner <da...@tethera.net> wrote:
> Jani Nikula <j...@nikula.org> writes:
>
>>
>> Theoretically "/" is an acceptable character in message-ids [1]. Rare,
>> unlikely, but acceptable. Searching for message-id's beginning with "/"
>> would have to use regexps, which would break in all sorts of ways
>> throughout the stack. I don't think there are handy alternatives to
>> "//", given the characters that are acceptable in message-ids,
>> but this is something to think about.
>
> Would telling the user to \ escape ( or double /) the initial / be good
> enough there? This would disable regex processing.  I guess this goes
> back to someone's earlier suggestion.  A third option would be to use
> single quotes there ("id:'/foo'"), but that isn't really consistent with 
> either Xapian
> or usual regex conventions.
>
> So I guess my favourite idea ATM is to use id:\/some/crazy/message-id
> FWIW, I don't have any such message ids.
>
>> For example, could the regexp matcher for message-ids first check if the
>> "regexp" is a strict match with "/" and all, and accept those? This
>> might be a reasonable workaround if it can be made to work.
>
> We're building a query, so I think the equivalent is to make an OR, with
> the exact match and the regex posting source. That could be done,
> although I'm a bit uneasy about how this makes the syntax for id:
> different, so id:/foo would be legit, but from:/foo would be an error.
> Maybe the dwim-factor is worth it.

Hi

Broadly I like the backslash escaping option. Two thoughts: can any
fields (from/subject/message-id) start with a "\" anyway? I think not
but thought it worth checking.

Secondly, message-id is often round-tripped, that is output from notmuch
and then fed back to notmuch. Do we want to escape the output as above
before printing in any cases? My view is that if we output the
message-id prefixed with "id:" then we should escape it (which applies
with --output=messages --format=text), but if we don't print the "id:"
part then we shouldn't (eg with --format=json). A similar thing would
apply to emacs: if it is a normal stash then escape the id, but if it is
a "bare stash" then do not.

Actually, one more thing: it would be a shame to block or significantly
delay the series for such a corner case.

Best wishes

Mark



___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [Patch v4] lib: regexp matching in 'subject' and 'from'

[Tomi Ollila]

On Thu, Feb 09 2017, David Bremner  wrote:

> Jani Nikula  writes:
>
>>
>> Theoretically "/" is an acceptable character in message-ids [1]. Rare,
>> unlikely, but acceptable. Searching for message-id's beginning with "/"
>> would have to use regexps, which would break in all sorts of ways
>> throughout the stack. I don't think there are handy alternatives to
>> "//", given the characters that are acceptable in message-ids,
>> but this is something to think about.
>
> Would telling the user to \ escape ( or double /) the initial / be good

a while ago I thought this double // but dismissed it quickly 
(re-searching just for single quote can be useful...) In the rare
cases anyone needs to disable regex processing, imo this \ is the
best idea i've (not) come up with.

some command line testing with(and -out) quoting:

$ printf %s\\n  id:\/some/crazy/message-id
id:/some/crazy/message-id

$ printf %s\\n  "id:\/some/crazy/message-id"
id:\/some/crazy/message-id

$ printf %s\\n  'id:\/some/crazy/message-id'
id:\/some/crazy/message-id


$ printf %s\\n  id:\\/some/crazy/message-id
id:\/some/crazy/message-id

$ printf %s\\n  "id:\\/some/crazy/message-id"
id:\/some/crazy/message-id

$ printf %s\\n  'id:\\/some/crazy/message-id'
id:\\/some/crazy/message-id

so:
$ printf %s\\n  'id:"\/some/crazy/message-id with spaces"'
id:"\/some/crazy/message-id with spaces"


> enough there? This would disable regex processing.  I guess this goes
> back to someone's earlier suggestion.  A third option would be to use
> single quotes there ("id:'/foo'"), but that isn't really consistent with
> either Xapian 
> or usual regex conventions.

$ printf %s\\n 'id:"'\''/foo with spaces ;D'\''"'
id:"'/foo with spaces ;D'"

or, perhaps this is clearer >;)

$ printf %s\\n 'id:"'"'"'/foo with spaces ;D'"'"'"'
id:"'/foo with spaces ;D'"

>
> So I guess my favourite idea ATM is to use id:\/some/crazy/message-id
> FWIW, I don't have any such message ids.
>
>> For example, could the regexp matcher for message-ids first check if the
>> "regexp" is a strict match with "/" and all, and accept those? This
>> might be a reasonable workaround if it can be made to work.
>
> We're building a query, so I think the equivalent is to make an OR, with
> the exact match and the regex posting source. That could be done,
> although I'm a bit uneasy about how this makes the syntax for id:
> different, so id:/foo would be legit, but from:/foo would be an error.
> Maybe the dwim-factor is worth it.
>
> d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [Patch v4] lib: regexp matching in 'subject' and 'from'

[David Bremner]

Jani Nikula  writes:

>
> Theoretically "/" is an acceptable character in message-ids [1]. Rare,
> unlikely, but acceptable. Searching for message-id's beginning with "/"
> would have to use regexps, which would break in all sorts of ways
> throughout the stack. I don't think there are handy alternatives to
> "//", given the characters that are acceptable in message-ids,
> but this is something to think about.

Would telling the user to \ escape ( or double /) the initial / be good
enough there? This would disable regex processing.  I guess this goes
back to someone's earlier suggestion.  A third option would be to use
single quotes there ("id:'/foo'"), but that isn't really consistent with either 
Xapian
or usual regex conventions.

So I guess my favourite idea ATM is to use id:\/some/crazy/message-id
FWIW, I don't have any such message ids.

> For example, could the regexp matcher for message-ids first check if the
> "regexp" is a strict match with "/" and all, and accept those? This
> might be a reasonable workaround if it can be made to work.

We're building a query, so I think the equivalent is to make an OR, with
the exact match and the regex posting source. That could be done,
although I'm a bit uneasy about how this makes the syntax for id:
different, so id:/foo would be legit, but from:/foo would be an error.
Maybe the dwim-factor is worth it.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [Patch v4] lib: regexp matching in 'subject' and 'from'

[David Bremner]

Tomi Ollila <tomi.oll...@iki.fi> writes:

> TOn Sun, Jan 29 2017, Jani Nikula <j...@nikula.org> wrote:
>
>>
>> I'd go with an error. It's easy to loosen the rules later on if we
>> decide that's a good idea. Much harder to accept loose rules now, let
>> users get used to it, and try to tighten the rules if we realize we'd
>> need that for some reason.
>
> I agree -- should we allow trailing slash ('/') without first char also
> being '/' (e.g. subject:blah/)
>

I'd say that should also be an error. it doesn't add anything useful to
subject search. Even for path search, (which is non-trivial to add regex
search for, I think) the trailing / doesn't add anything does it?

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [Patch v4] lib: regexp matching in 'subject' and 'from'

[Tomi Ollila]

TOn Sun, Jan 29 2017, Jani Nikula <j...@nikula.org> wrote:

> On Wed, 25 Jan 2017, David Bremner <da...@tethera.net> wrote:
>> Tomi Ollila <tomi.oll...@iki.fi> writes:
>>
>>>
>>> Why would not mesasge_id not be useful to regex match. I can come up quite
>>> a few use cases... but if there are techinal difficulties... then that
>>> should be mentioned instead.
>>
>> I'll have a look. Since the first version of this patch (when that
>> message was written), people have actually asked for some kind of
>> wildcard matching of message-ids.
>
> Theoretically "/" is an acceptable character in message-ids [1]. Rare,
> unlikely, but acceptable. Searching for message-id's beginning with "/"
> would have to use regexps, which would break in all sorts of ways
> throughout the stack. I don't think there are handy alternatives to
> "//", given the characters that are acceptable in message-ids,
> but this is something to think about.
>
> For example, could the regexp matcher for message-ids first check if the
> "regexp" is a strict match with "/" and all, and accept those? This
> might be a reasonable workaround if it can be made to work.
>
> [1] https://tools.ietf.org/html/rfc2822#section-3.2.4
>
>>> maybe this commit message should inform that xapian with field processors
>>> (1.4.x) is required for this feature -- and emphasize it a bit better in
>>> manual page ?
>>>
>>> Probably '//' is used to escape '/' -- should such a character ever needed
>>> in regex search.
>>>
>>
>> Currently no escaping is needed because it only looks at the first and
>> last characters of the string (the usual xapian/shell rules mean that "" 
>> might
>> be needed).
>>
>> The following seem to work as hoped
>>
>> # match a / with a space before it
>>
>> % notmuch search 'subject:"/ //"'
>>
>> # just a slash
>>
>> % notmuch search subject:///
>>
>> # anchored slash
>>
>> % notmuch search subject:/^//
>>
>> The trailing slash is actually decorative, we could drop it. Actually
>> *blush* I just noticed the current code is missing something from this line
>>
>>  if (str.at (0) == '/' && str.at (str.size () - 1)){
>>
>> _if_ that line is fixed, then it will have the slightly odd behaviour of
>>
>> subject:/blah
>>
>> doing a non-regex search
>>
>> We could also throw an error for that case, maybe that's the best option.
>
> I'd go with an error. It's easy to loosen the rules later on if we
> decide that's a good idea. Much harder to accept loose rules now, let
> users get used to it, and try to tighten the rules if we realize we'd
> need that for some reason.

I agree -- should we allow trailing slash ('/') without first char also
being '/' (e.g. subject:blah/)

Tomi

>
> BR,
> Jani.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [Patch v4] lib: regexp matching in 'subject' and 'from'

[Jani Nikula]

On Wed, 25 Jan 2017, David Bremner <da...@tethera.net> wrote:
> Tomi Ollila <tomi.oll...@iki.fi> writes:
>
>>
>> Why would not mesasge_id not be useful to regex match. I can come up quite
>> a few use cases... but if there are techinal difficulties... then that
>> should be mentioned instead.
>
> I'll have a look. Since the first version of this patch (when that
> message was written), people have actually asked for some kind of
> wildcard matching of message-ids.

Theoretically "/" is an acceptable character in message-ids [1]. Rare,
unlikely, but acceptable. Searching for message-id's beginning with "/"
would have to use regexps, which would break in all sorts of ways
throughout the stack. I don't think there are handy alternatives to
"//", given the characters that are acceptable in message-ids,
but this is something to think about.

For example, could the regexp matcher for message-ids first check if the
"regexp" is a strict match with "/" and all, and accept those? This
might be a reasonable workaround if it can be made to work.

[1] https://tools.ietf.org/html/rfc2822#section-3.2.4

>> maybe this commit message should inform that xapian with field processors
>> (1.4.x) is required for this feature -- and emphasize it a bit better in
>> manual page ?
>>
>> Probably '//' is used to escape '/' -- should such a character ever needed
>> in regex search.
>>
>
> Currently no escaping is needed because it only looks at the first and
> last characters of the string (the usual xapian/shell rules mean that "" might
> be needed).
>
> The following seem to work as hoped
>
> # match a / with a space before it
>
> % notmuch search 'subject:"/ //"'
>
> # just a slash
>
> % notmuch search subject:///
>
> # anchored slash
>
> % notmuch search subject:/^//
>
> The trailing slash is actually decorative, we could drop it. Actually
> *blush* I just noticed the current code is missing something from this line
>
>  if (str.at (0) == '/' && str.at (str.size () - 1)){
>
> _if_ that line is fixed, then it will have the slightly odd behaviour of
>
> subject:/blah
>
> doing a non-regex search
>
> We could also throw an error for that case, maybe that's the best option.

I'd go with an error. It's easy to loosen the rules later on if we
decide that's a good idea. Much harder to accept loose rules now, let
users get used to it, and try to tighten the rules if we realize we'd
need that for some reason.

BR,
Jani.

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch