[PATCH v5 02/12] crypto: make shared crypto code behave library-like

2017-10-15 Thread Daniel Kahn Gillmor
If we're going to reuse the crypto code across both the library and the client, then it needs to report error states properly and not write to stderr. --- lib/database.cc | 6 lib/notmuch.h | 17 +++ mime-node.c | 7 - util/crypto.c | 92

[PATCH v5 12/12] cli/reindex: add --try-decrypt=(true|false)

2017-10-15 Thread Daniel Kahn Gillmor
Try to decrypt any encrypted parts of newly-discovered messages while re-indexing them. The cleartext of any successfully-decrypted messages will be indexed, with tags applied in the same form as from notmuch insert --try-decrypt=true. Note: if the deprecated crypto.gpg_path configuration option

[PATCH v5 09/12] cli: set up shared command-line arguments for indexing

2017-10-15 Thread Daniel Kahn Gillmor
We have an indexopts structure for manipulating indexing in different ways, but we also have three command-line invocations that can trigger indexing: new, insert, and reindex. This changeset prepares a common parser that these subcommands can share. At the moment, it's just --try-decrypt, but

[PATCH v5 06/12] crypto: index encrypted parts when indexopts try_decrypt is set.

2017-10-15 Thread Daniel Kahn Gillmor
If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property

[PATCH v5 10/12] cli/new: add --try-decrypt=(true|false)

2017-10-15 Thread Daniel Kahn Gillmor
Try to decrypt any encrypted parts of newly-discovered messages while indexing them. The cleartext of any successfully-decrypted messages will be indexed, with tags applied in the same form as from notmuch insert --try-decrypt=true. Note: if the deprecated crypto.gpg_path configuration option is

[PATCH v5 07/12] config: indexing defaults will be stored in the database.

2017-10-15 Thread Daniel Kahn Gillmor
At indexing time, the database needs to know its internal defaults. It shouldn't be contingent on an external config file (since that can't be retrieved from the database object itself). This behaves the same as the query.* configurations, which are also stored in the database itself, so we're

[PATCH v5 08/12] config: define new option index.try_decrypt

2017-10-15 Thread Daniel Kahn Gillmor
By default, notmuch won't try to decrypt on indexing. With this patch, we make it possible to indicate a per-database preference using the config variable "index.try_decrypt", which by default will be false. --- doc/man1/notmuch-config.rst | 12 lib/indexopts.c | 18

[PATCH v5 04/12] properties: add notmuch_message_remove_all_properties_with_prefix()

2017-10-15 Thread Daniel Kahn Gillmor
Subsequent patches will introduce a convention that properties whose name starts with "index." will be stripped (and possibly re-added) during re-indexing. This patch lays the groundwork for doing that. --- lib/message-property.cc | 18 -- lib/notmuch.h | 16

cleartext indexing version 5

2017-10-15 Thread Daniel Kahn Gillmor
Thanks to everyone for their reviews of this series that provides indexing of the cleartext of encrypted messages (and thanks to bremner for already pushing some of the less controversial patches of the previous revisions of this series). I hope you'll agree that your suggestions have improved

[PATCH v5 11/12] cli/insert: add --try-decrypt=(true|false)

2017-10-15 Thread Daniel Kahn Gillmor
Allow an incoming message to be delivered while indexing the cleartext, on a per-message basis. This requires the secret keys for the message to be available. For the moment, the most functional approach is to ensure that gpg-agent is running and knows about any secret keys that might be useful

Re: [PATCH v3 11/15] config: indexing defaults will be stored in the database.

2017-10-15 Thread Daniel Kahn Gillmor
On Sat 2017-10-14 15:08:51 -0300, David Bremner wrote: > Daniel Kahn Gillmor <d...@fifthhorseman.net> writes: > >> static int >> _print_db_config(notmuch_config_t *config, const char *name) >> @@ -859,6 +860,8 @@ notmuch_config_command_get (notmuch_co

[PATCH v5 05/12] reindex: drop all properties named with prefix "index."

2017-10-15 Thread Daniel Kahn Gillmor
This allows us to create new properties that will be automatically set during indexing, and cleared during re-indexing, just by choice of property name. --- lib/message.cc | 6 ++ lib/notmuch.h | 6 ++ 2 files changed, 12 insertions(+) diff --git a/lib/message.cc b/lib/message.cc index

[PATCH v5 01/12] crypto: move into libnotmuch_util

2017-10-15 Thread Daniel Kahn Gillmor
This prepares us for using the crypto object in both libnotmuch and the client. --- Makefile.local| 1 - notmuch-client.h | 22 +- util/Makefile.local | 2 +- crypto.c => util/crypto.c | 7 ++- util/crypto.h | 28

[PATCH v5 03/12] index: implement notmuch_indexopts_t with try_decrypt

2017-10-15 Thread Daniel Kahn Gillmor
This is currently mostly a wrapper around _notmuch_crypto_t that keeps its internals private and doesn't expose any of the GMime API. However, non-crypto indexing options might also be added later (e.g. filters or other transformations). --- lib/add-message.cc| 11 ++- lib/indexopts.c

Re: [PATCH v3 10/15] crypto: index encrypted parts when indexopts try_decrypt is set.

2017-10-13 Thread Daniel Kahn Gillmor
On Thu 2017-10-12 22:08:26 -0300, David Bremner wrote: > The second _notmuch_database_log will override the first > here. You can use _notmuch_database_log_append if you don't want to > clear the existing log (e.g. at least for the second _log > here). good catch, thanks! I'll include that in my

Re: [Łukasz Stelmach] Bug#864493: notmuch: bash completion eats typed option

2017-09-07 Thread Daniel Kahn Gillmor
On Thu 2017-08-31 10:50:57 -0300, David Bremner wrote: > This eventually turned out to be a bug in the "npm" debian package, > since fixed upstream there. only if you actually purge npm itself :( on a system where npm is in the "config-files" state, it's entirely possible that the old/bad npm

[PATCH 08/10] add --try-decrypt=(true|false) to notmuch new

2017-09-12 Thread Daniel Kahn Gillmor
Try to decrypt any encrypted parts of newly-discovered messages while indexing them. The cleartext of any successfully-decrypted messages will be indexed, with tags applied in the same form as from notmuch insert --try-decrypt=true. Note: if the deprecated crypto.gpg_path configuration option is

Re: [PATCH 07/10] Define new config option index.try_decrypt

2017-09-12 Thread Daniel Kahn Gillmor
On Tue 2017-09-12 19:01:50 -0400, Daniel Kahn Gillmor wrote: > By default, notmuch won't try to decrypt on indexing. With this > patch, we make it possible to indicate a per-database preference using > the config variable "index.try_decrypt", which by default will be

[PATCH 06/10] crypto: index encrypted parts when indexopts try_decrypt is set.

2017-09-12 Thread Daniel Kahn Gillmor
If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index-decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property

[PATCH 10/10] add --try-decrypt=(true|false) to notmuch reindex

2017-09-12 Thread Daniel Kahn Gillmor
Try to decrypt any encrypted parts of newly-discovered messages while re-indexing them. The cleartext of any successfully-decrypted messages will be indexed, with tags applied in the same form as from notmuch insert --try-decrypt=true. Note: if the deprecated crypto.gpg_path configuration option

[PATCH 09/10] add --try-decrypt=(true|false) to notmuch insert

2017-09-12 Thread Daniel Kahn Gillmor
Allow an incoming message to be delivered while indexing the cleartext, on a per-message basis. This requires the secret keys for the message to be available. For the moment, the most functional approach is to ensure that gpg-agent is running and knows about any secret keys that might be useful

[PATCH 02/10] crypto: Move crypto.c into libutil

2017-09-12 Thread Daniel Kahn Gillmor
ypto.c b/util/crypto.c similarity index 79% rename from crypto.c rename to util/crypto.c index cc45b885..97e8c8f4 100644 --- a/crypto.c +++ b/util/crypto.c @@ -16,18 +16,26 @@ * along with this program. If not, see https://www.gnu.org/licenses/ . * * Authors: Jameson Rollins <jroll...@fines

[PATCH 07/10] Define new config option index.try_decrypt

2017-09-12 Thread Daniel Kahn Gillmor
By default, notmuch won't try to decrypt on indexing. With this patch, we make it possible to indicate a per-database preference using the config variable "index.try_decrypt", which by default will be false. --- doc/man1/notmuch-config.rst | 12 lib/indexopts.c | 16

[PATCH 03/10] crypto: make shared crypto code behave library-like

2017-09-12 Thread Daniel Kahn Gillmor
If we're going to reuse the crypto code across both the library and the client, then it needs to report error states properly and not write to stderr. --- lib/database.cc | 6 lib/notmuch.h | 17 +++ mime-node.c | 7 - util/crypto.c | 89

[PATCH 01/10] reorganize indexing of multipart/signed and multipart/encrypted

2017-09-12 Thread Daniel Kahn Gillmor
This makes no functional changes, but prepares the codebase for a cleaner changeset for dealing with indexing some encrypted messages in the clear. --- lib/index.cc | 42 ++ 1 file changed, 22 insertions(+), 20 deletions(-) diff --git a/lib/index.cc

[PATCH 04/10] tests: prepare for more crypto tests (using add_gnupg_home)

2017-09-12 Thread Daniel Kahn Gillmor
Move add_gnupg_home to test-lib.sh to prepare it for reuse. --- test/T350-crypto.sh | 17 - test/test-lib.sh| 15 +++ 2 files changed, 15 insertions(+), 17 deletions(-) diff --git a/test/T350-crypto.sh b/test/T350-crypto.sh index 1d408af7..e1b8fd83 100755 ---

[PATCH 05/10] index: implement notmuch_indexopts_t with try_decrypt

2017-09-12 Thread Daniel Kahn Gillmor
This is currently mostly a wrapper around _notmuch_crypto_t that keeps its internals private and doesn't expose any of the GMime API. However, non-crypto indexing options might also be added later (e.g. filters or other transformations). --- lib/add-message.cc| 9 - lib/indexopts.c

Re: [PATCH v2] python: deprecated add_message calls index_file correctly and returns result

2017-09-12 Thread Daniel Kahn Gillmor
On Wed 2017-08-30 10:16:33 +0200, Gaute Hope wrote: > The deprecated Database.add_message now calls the new index_file with > correct number of arguments (without an extra `self`), and returns the > tuple from index_file - as it used to do before. > > This change also adds a DeprecationWarning to

Re: [PATCH 0/9] argument parsing fixes and improvements

2017-09-25 Thread Daniel Kahn Gillmor
On Mon 2017-09-25 08:34:13 -0300, David Bremner wrote: > I think there is two different discussions one could be having here; one > about the UI, the other about the implimentation. > > From the UI point of view, Are you using the term "UI" to mean "API" here? i tend to think of "UI" as the CLI

Re: [PATCH] doc: fix typos

2017-09-27 Thread Daniel Kahn Gillmor
On Tue 2017-09-26 19:23:09 +0200, Jakub Wilk wrote: > --- > doc/man7/notmuch-search-terms.rst | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/doc/man7/notmuch-search-terms.rst > b/doc/man7/notmuch-search-terms.rst > index dd76972e..b27f31f7 100644 > ---

[PATCH] fix documentation bug (leading quotes break documentation)

2017-09-05 Thread Daniel Kahn Gillmor
--- lib/notmuch.h | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/notmuch.h b/lib/notmuch.h index 66ecb5fc..f26565f3 100644 --- a/lib/notmuch.h +++ b/lib/notmuch.h @@ -562,9 +562,10 @@ notmuch_database_get_directory (notmuch_database_t *database, * terms from the

Re: [PATCH 1/2] cli: convert notmuch_bool_t to stdbool

2017-10-09 Thread Daniel Kahn Gillmor
On Sat 2017-10-07 11:44:04 +0300, Jani Nikula wrote: > C99 stdbool turned 18 this year. There really is no reason to use our > own, except in the library interface for backward > compatibility. Convert the cli and test binaries to stdbool. +1 LGTM. I'd also be interested in contemplating a

[PATCH v3 14/15] cli/insert: add --try-decrypt=(true|false)

2017-10-09 Thread Daniel Kahn Gillmor
Allow an incoming message to be delivered while indexing the cleartext, on a per-message basis. This requires the secret keys for the message to be available. For the moment, the most functional approach is to ensure that gpg-agent is running and knows about any secret keys that might be useful

[PATCH v3 09/15] gmime-extra: drop compat layer for g_mime_multipart_encrypted_decrypt

2017-10-09 Thread Daniel Kahn Gillmor
In practice, we're going to see this function invoked differently depending on which gmime we build against. The compatibility layer forces our code into the lowest-common-denominator -- unable to make use of new features even when built against a newer version. Dropping the compatibility layer

cleartext indexing, round 3

2017-10-09 Thread Daniel Kahn Gillmor
What follows is the third round of the latest revision of cleartext indexing patches. This series is rebased against current master, with the improved option-handling code, and takes into consideration the review that Jani did on round 2 of this series. I welcome additional review and feedback.

[PATCH v3 06/15] crypto: make shared crypto code behave library-like

2017-10-09 Thread Daniel Kahn Gillmor
If we're going to reuse the crypto code across both the library and the client, then it needs to report error states properly and not write to stderr. --- lib/database.cc | 6 lib/notmuch.h | 17 +++ mime-node.c | 7 - util/crypto.c | 92

[PATCH v3 15/15] cli/reindex: add --try-decrypt=(true|false)

2017-10-09 Thread Daniel Kahn Gillmor
Try to decrypt any encrypted parts of newly-discovered messages while re-indexing them. The cleartext of any successfully-decrypted messages will be indexed, with tags applied in the same form as from notmuch insert --try-decrypt=true. Note: if the deprecated crypto.gpg_path configuration option

[PATCH v3 07/15] tests: prepare for more crypto tests (using add_gnupg_home)

2017-10-09 Thread Daniel Kahn Gillmor
Move add_gnupg_home to test-lib.sh to prepare it for reuse. --- test/T350-crypto.sh | 17 - test/test-lib.sh| 17 + 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/test/T350-crypto.sh b/test/T350-crypto.sh index 53bf4113..401a22c9 100755 ---

[PATCH v3 04/15] crypto: move into libutil

2017-10-09 Thread Daniel Kahn Gillmor
This prepares us for using the crypto object in both the library and the client. --- Makefile.local| 1 - notmuch-client.h | 22 +- util/Makefile.local | 2 +- crypto.c => util/crypto.c | 6 +- util/crypto.h | 28

[PATCH v3 02/15] crypto: drop pretense of notmuch_crypto_context_t

2017-10-09 Thread Daniel Kahn Gillmor
notmuch_crypto_context_t was introduced (i think) as some sort of abstraction layer to make notmuch somewhat independent of GMime. But it isn't even useful for GMime 3.0 or later -- we can drop the pretense that it's some sort of abstraction in this case, and just call it what it is,

[PATCH v3 11/15] config: indexing defaults will be stored in the database.

2017-10-09 Thread Daniel Kahn Gillmor
At indexing time, the database needs to know its internal defaults. It shouldn't be contingent on an external config file (since that can't be retrieved from the database object itself). This behaves the same as the query.* configurations, which are also stored in the database itself, so we're

[PATCH v3 01/15] crypto: rename notmuch_crypto_t to _notmuch_crypto_t

2017-10-09 Thread Daniel Kahn Gillmor
The notmuch_crypto_t struct isn't used externally, and we have no plans to explicitly export it. Prefix its name (and associated functions) with _ to make that intent clear. --- crypto.c | 12 ++-- mime-node.c | 6 +++--- notmuch-client.h | 12 ++-- notmuch-reply.c

Re: [PATCH v2 04/10] index: implement notmuch_indexopts_t with try_decrypt

2017-10-09 Thread Daniel Kahn Gillmor
On Sat 2017-09-23 19:10:18 +0300, Jani Nikula wrote: >> --- a/lib/indexopts.c >> +++ b/lib/indexopts.c >> @@ -21,9 +21,27 @@ >> #include "notmuch-private.h" >> >> notmuch_indexopts_t * >> -notmuch_database_get_default_indexopts (notmuch_database_t unused (*db)) >>

Re: [PATCH v2 05/10] crypto: index encrypted parts when indexopts try_decrypt is set.

2017-10-09 Thread Daniel Kahn Gillmor
On Sat 2017-09-23 19:05:40 +0300, Jani Nikula wrote: > I'd like this #if block to be abstracted to separate functions for gmime > 2 vs. 3. Adding conditional compilation within functions is ugly and > hard to track for both branches. That abstraction has already happened -- it's in util/crypto.c;

[PATCH v3 12/15] config: define new option index.try_decrypt

2017-10-09 Thread Daniel Kahn Gillmor
By default, notmuch won't try to decrypt on indexing. With this patch, we make it possible to indicate a per-database preference using the config variable "index.try_decrypt", which by default will be false. --- doc/man1/notmuch-config.rst | 12 lib/indexopts.c | 18

[PATCH v3 03/15] crypto: _notmuch_crypto_cleanup should return void

2017-10-09 Thread Daniel Kahn Gillmor
There's no chance that _notmuch_crypto_cleanup() will ever return anything other than 0, and no one ever checks its return value anyway. So make it return void instead of int. --- crypto.c | 7 ++- notmuch-client.h | 2 +- 2 files changed, 3 insertions(+), 6 deletions(-) diff --git

[PATCH v3 13/15] cli/new: add --try-decrypt=(true|false)

2017-10-09 Thread Daniel Kahn Gillmor
Try to decrypt any encrypted parts of newly-discovered messages while indexing them. The cleartext of any successfully-decrypted messages will be indexed, with tags applied in the same form as from notmuch insert --try-decrypt=true. Note: if the deprecated crypto.gpg_path configuration option is

[PATCH v3 08/15] index: implement notmuch_indexopts_t with try_decrypt

2017-10-09 Thread Daniel Kahn Gillmor
This is currently mostly a wrapper around _notmuch_crypto_t that keeps its internals private and doesn't expose any of the GMime API. However, non-crypto indexing options might also be added later (e.g. filters or other transformations). --- lib/add-message.cc| 9 - lib/indexopts.c

Re: [PATCH v2 02/10] crypto: make shared crypto code behave library-like

2017-10-09 Thread Daniel Kahn Gillmor
On Sat 2017-09-23 18:36:18 +0300, Jani Nikula wrote: >> if (!cryptoctx) >> return NULL; > > I guess this will work because we initialize cryptoctx to NULL, but if > we return the status, I think we should trust status == success means > cryptoctx is fine, and otherwise we shouldn't

[PATCH v3 05/15] gmime-extra: remove duplicate GMimeAddressType typedef

2017-10-09 Thread Daniel Kahn Gillmor
"typedef GMimeAddressType GMimeRecipientType" is already present further down in the compatibility wrapper (with other typedefs). We don't need it twice. --- util/gmime-extra.h | 1 - 1 file changed, 1 deletion(-) diff --git a/util/gmime-extra.h b/util/gmime-extra.h index de275bc1..e060bcc2

[PATCH v3 10/15] crypto: index encrypted parts when indexopts try_decrypt is set.

2017-10-09 Thread Daniel Kahn Gillmor
If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index-decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property

Re: Public notmuch.mbox missing some messages which are tagged in nmbug

2017-10-11 Thread Daniel Kahn Gillmor
On Wed 2017-10-11 08:04:05 -0300, David Bremner wrote: > "W. Trevor King" writes: > >> If somebody has copies of those messages, can they push them up into >> the mbox? Maybe they went to the freelists list [3]? Maybe they were >> off-list? Maybe they just missed inclusion in

Re: [PATCH 0/3] nmbug:

2017-10-10 Thread Daniel Kahn Gillmor
On Tue 2017-10-10 15:49:48 -0700, W. Trevor King wrote: > Two changes and a bugfix spun off from today's IRC disussion. This series looks reasonable to me, from what little i understand of nmbug. Thanks for proposing the changes, Trevor. --dkg signature.asc Description: PGP signature

[PATCH v4 08/15] index: implement notmuch_indexopts_t with try_decrypt

2017-10-10 Thread Daniel Kahn Gillmor
This is currently mostly a wrapper around _notmuch_crypto_t that keeps its internals private and doesn't expose any of the GMime API. However, non-crypto indexing options might also be added later (e.g. filters or other transformations). --- lib/add-message.cc| 9 - lib/indexopts.c

avoid double typedef

2017-10-10 Thread Daniel Kahn Gillmor
On IRC, Domo pointed out that older gcc complains when a typedef gets repeated. So i'm updating patch 8 in this series to avoid double typedefs. The rest of the series should be unchanged, so i'm avoiding re-flooding the lst with them, but i'm happy to send along a full round of v4 if folks

Re: [PATCH v3 04/15] crypto: move into libutil

2017-10-12 Thread Daniel Kahn Gillmor
Hi Bremner-- Thanks for the review! On Thu 2017-10-12 07:54:33 -0300, David Bremner wrote: > Daniel Kahn Gillmor <d...@fifthhorseman.net> writes: > >> This prepares us for using the crypto object in both the library and >> the client. > > I think we could be

Re: [PATCH v4 08/15] index: implement notmuch_indexopts_t with try_decrypt

2017-10-12 Thread Daniel Kahn Gillmor
On Thu 2017-10-12 08:18:15 -0300, David Bremner wrote: > Daniel Kahn Gillmor <d...@fifthhorseman.net> writes: > >> >> +if (!indexopts) >> +indexopts = def_indexopts = notmuch_database_get_default_indexopts >> (notmuch); >> + > &

[PATCH 2/2] doc: make SEE ALSO references one-per-line

2017-10-17 Thread Daniel Kahn Gillmor
This will make future diffs cleaner, make it easier to keep them alphabetical, and make it easier to scan and search the documentation sources. --- doc/man1/notmuch-address.rst | 16 doc/man1/notmuch-compact.rst | 16 doc/man1/notmuch-config.rst |

[PATCH 1/2] doc: fix documentation typos and copy/paste-os

2017-10-17 Thread Daniel Kahn Gillmor
--- doc/man1/notmuch-dump.rst| 2 +- doc/man1/notmuch-restore.rst | 8 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/man1/notmuch-dump.rst b/doc/man1/notmuch-dump.rst index f3f2b394..1e4468e6 100644 --- a/doc/man1/notmuch-dump.rst +++ b/doc/man1/notmuch-dump.rst

[PATCH v6 09/14] config: define new option index.try_decrypt

2017-10-17 Thread Daniel Kahn Gillmor
By default, notmuch won't try to decrypt on indexing. With this patch, we make it possible to indicate a per-database preference using the config variable "index.try_decrypt", which by default will be false. --- doc/man1/notmuch-config.rst | 12 doc/man7/notmuch-properties.rst |

[PATCH v6 07/14] crypto: index encrypted parts when indexopts try_decrypt is set.

2017-10-17 Thread Daniel Kahn Gillmor
If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property

[PATCH v6 06/14] reindex: drop all properties named with prefix "index."

2017-10-17 Thread Daniel Kahn Gillmor
This allows us to create new properties that will be automatically set during indexing, and cleared during re-indexing, just by choice of property name. --- doc/man7/notmuch-properties.rst | 6 ++ lib/message.cc | 6 ++ lib/notmuch.h | 6 ++ 3 files

[PATCH v6 02/14] crypto: make shared crypto code behave library-like

2017-10-17 Thread Daniel Kahn Gillmor
If we're going to reuse the crypto code across both the library and the client, then it needs to report error states properly and not write to stderr. --- lib/database.cc | 6 lib/notmuch.h | 17 +++ mime-node.c | 7 - util/crypto.c | 92

cleartext indexing, revision 6

2017-10-17 Thread Daniel Kahn Gillmor
This is the 6th revision of the cleartext indexing series. It differs from the 5th mainly in minor edits: fixing typos and improving comments, documentation, and commit messages. It also has the following (slightly more) significant changes: * improvement of the test suite to permit passing

[PATCH v6 08/14] config: store index.try_default in the database.

2017-10-17 Thread Daniel Kahn Gillmor
At indexing time, the database needs some way to know its internal defaults for how to index encrypted parts. It shouldn't be contingent on an external config file (since that can't be retrieved from the database object itself). This behaves similarly to the query.* configurations, which are

[PATCH v6 10/14] cli: set up shared command-line arguments for indexing

2017-10-17 Thread Daniel Kahn Gillmor
We have an indexopts structure for manipulating indexing in different ways, but we also have three command-line invocations that can trigger indexing: new, insert, and reindex. This changeset prepares a common parser that these subcommands can share. Note: if the deprecated crypto.gpg_path

[PATCH v6 11/14] tests: emacs_fcc_message: allow passing --arguments to notmuch new

2017-10-17 Thread Daniel Kahn Gillmor
Subsequent patches may want to send GNU-style --long-arguments to notmuch new in the test suite, in particular when invoking emacs_fcc_message. This changeset makes that possible. --- test/test-lib.sh | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git

[PATCH v6 03/14] index: implement notmuch_indexopts_t with try_decrypt

2017-10-17 Thread Daniel Kahn Gillmor
This is currently mostly a wrapper around _notmuch_crypto_t that keeps its internals private and doesn't expose any of the GMime API. However, non-crypto indexing options might also be added later (e.g. filters or other transformations). --- lib/add-message.cc| 11 ++- lib/indexopts.c

[PATCH v6 04/14] doc: add notmuch-properties(7)

2017-10-17 Thread Daniel Kahn Gillmor
We will want a user-facing place to record details about the use of notmuch properties shortly. This establishes a new manual page for that purpose. --- doc/conf.py | 4 +++ doc/index.rst | 1 + doc/man1/notmuch-dump.rst | 5 ++--

[PATCH v6 05/14] properties: add notmuch_message_remove_all_properties_with_prefix()

2017-10-17 Thread Daniel Kahn Gillmor
Subsequent patches will introduce a convention that properties whose name starts with "index." will be stripped (and possibly re-added) during re-indexing. This patch lays the groundwork for doing that. --- lib/message-property.cc | 18 -- lib/notmuch.h | 16

[PATCH v6 12/14] cli/new: add --try-decrypt=(true|false)

2017-10-17 Thread Daniel Kahn Gillmor
Enable override of the index.try_decrypt setting during "notmuch new" on a per-invocation basis. We update the documentation and tab completion, and also add a test. --- completion/notmuch-completion.bash | 13 -- doc/man1/notmuch-new.rst | 12 + notmuch-new.c

[PATCH v6 01/14] crypto: move into libnotmuch_util

2017-10-17 Thread Daniel Kahn Gillmor
This prepares us for using the crypto object in both libnotmuch and the client. --- Makefile.local| 1 - notmuch-client.h | 22 +- util/Makefile.local | 2 +- crypto.c => util/crypto.c | 7 ++- util/crypto.h | 28

[PATCH v6 13/14] cli/insert: add --try-decrypt=(true|false)

2017-10-17 Thread Daniel Kahn Gillmor
Enable override of the index.try_decrypt setting on a per-message basis when invoking "notmuch insert". We also update the documentation and tab completion. --- completion/notmuch-completion.bash | 6 +- doc/man1/notmuch-insert.rst| 14 ++ notmuch-insert.c

[PATCH v6 14/14] cli/reindex: add --try-decrypt=(true|false)

2017-10-17 Thread Daniel Kahn Gillmor
Enable override of the index.try_decrypt setting on a per-run basis when invoking "notmuch reindex". This allows the possibility of (for example) an emacs keybinding that adds the cleartext of the currently shown decrypted message to the index, making it searchable in the future. It also enables

Re: [PATCH] lib: add support for thread: queries

2017-10-17 Thread Daniel Kahn Gillmor
On Mon 2017-10-16 22:07:54 +0300, Jani Nikula wrote: > Add support for querying threads using message-ids in addition to > thread-ids. The main benefit is that thread queries via message-ids > are portable across databases, re-indexing, and thread joining, while > thread ids can be somewhat

Re: [PATCH] cli/crypto: fix segfault on failed gmime2 crypto context creation

2017-10-17 Thread Daniel Kahn Gillmor
On Mon 2017-10-16 18:40:44 +0300, Jani Nikula wrote: > Commit 1fdc08d0ffab ("cli/crypto: treat failure to create a crypto > context as fatal.") started treating crypto context creation failures > "as fatal", returning NULL from _mime_node_create(). > > Unfortunately, we do not have NULL checks for

Re: web interface to notmuch

2017-10-19 Thread Daniel Kahn Gillmor
On Thu 2017-10-19 11:01:53 -0400, Brian Sniffen wrote: > I put together something like this, visible at > https://github.com/briansniffen/notmuch/tree/nmweb/contrib/notmuch-web > > It's not much of a service. I am pretty sure it is exploitable---that > content in text/html parts of messages can

Re: web interface to notmuch

2017-10-19 Thread Daniel Kahn Gillmor
On Thu 2017-10-19 16:00:33 -0400, Brian Sniffen wrote: > I don’t think they can be sanitized. Web tech moves so fast. well, there are at least a handful of python modules that claim to do some sort of sanitization. in debian alone, we have at least: python3-django-html-sanitizer

Re: [PATCH v6 04/14] doc: add notmuch-properties(7)

2017-10-19 Thread Daniel Kahn Gillmor
On Tue 2017-10-17 15:09:58 -0400, Daniel Kahn Gillmor wrote: > diff --git a/doc/man7/notmuch-properties.rst b/doc/man7/notmuch-properties.rst > new file mode 100644 [...] > +**notmuch** **count** **property:**<*key*>=<*value*> > + > +**notmuch** **search** *

[PATCH] NEWS: cleartext indexing

2017-10-22 Thread Daniel Kahn Gillmor
give a short summary of cleartext indexing of encrypted e-mail messages. --- NEWS | 18 ++ 1 file changed, 18 insertions(+) diff --git a/NEWS b/NEWS index c2ebb644..895b7627 100644 --- a/NEWS +++ b/NEWS @@ -11,6 +11,24 @@ Out-of-tree builds has been dropped. The same can now

Re: web interface to notmuch

2017-10-22 Thread Daniel Kahn Gillmor
On Sat 2017-10-21 23:00:00 +0300, Jani Nikula wrote: > For the list archive, we could restrict to displaying text/plain only. and text/x-diff, surely :) But yeah, good point. Brian, what do you think about such a constraint? would that make your implementation safe enough to put on the public

[PATCH] NEWS: document notmuch reindex

2017-10-22 Thread Daniel Kahn Gillmor
Describe the introduction of the "reindex" subcommand. This blurb acknowledges Subject: instability under reindexing when multiple copies exist (suggesting that this is something that needn't necessarily hold up a release). --- NEWS | 13 + 1 file changed, 13 insertions(+) diff

Re: [PATCH] python: add bindings for notmuch_message_get_propert(y/ies)

2017-11-28 Thread Daniel Kahn Gillmor
On Tue 2017-11-28 23:46:11 +0100, Ruben Pollan wrote: > Message.get_property (prop) returns a string with the value of the property > and > Message.get_properties (prop, exact=False) returns a list [(key, value)] This looks like a sensible approach to me. I'd be curious to hear what others

Re: [PATCH 15/18] crypto: actually stash session keys when try-decrypt=true

2017-11-30 Thread Daniel Kahn Gillmor
On Thu 2017-11-16 08:53:14 -0400, David Bremner wrote: > I'd be happier if we didn't further entrench the text format in the test > suite. How hard would it be to use json output (+maybe python?) here? json output seems clunkier to me, and i don't think it's necessary for the purposes of these

Re: [PATCH 18/18] python: add try_decrypt argument to Database.index_file()

2017-11-30 Thread Daniel Kahn Gillmor
On Thu 2017-11-16 09:06:09 -0400, David Bremner wrote: > I think this new bindings functionality needs a test. agreed, the python bindings do need to be added to the test suite (this is also true in the newer version of the series). I'm happy to add those tests as a condition of getting the

[PATCH] crypto: signature verification reports valid User IDs

2017-11-29 Thread Daniel Kahn Gillmor
When i'm trying to understand a message signature, i care that i know who it came from (the "validity" of the identity associated with the key), *not* whether i'm willing to accept the keyholder's other identity assertions (the "trust" associated with the certificate). We've been reporting User

[PATCH 3/5] debian/control: Priority: extra is deprecated

2017-11-30 Thread Daniel Kahn Gillmor
Move transitional package to to "oldlibs/optional" This resolves two lintian warnings: W: notmuch-emacs: transitional-package-should-be-oldlibs-optional oldlibs/extra W: notmuch-emacs: priority-extra-is-replaced-by-priority-optional --- debian/control | 1 - 1 file changed, 1 deletion(-) diff

[PATCH 2/5] debian/changelog: remove trailing whitespace

2017-11-30 Thread Daniel Kahn Gillmor
--- debian/changelog | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/debian/changelog b/debian/changelog index 67282a07..decef1e9 100644 --- a/debian/changelog +++ b/debian/changelog @@ -836,7 +836,7 @@ notmuch (0.9~rc2-1) experimental; urgency=low - New

Re: [PATCH] cli/help: give a hint about notmuch-emacs-mua

2017-11-29 Thread Daniel Kahn Gillmor
On Thu 2017-10-26 18:27:51 -0400, Daniel Kahn Gillmor wrote: > "notmuch help" doesn't mention "notmuch-emacs-mua" even though we > support it through the try_external_command() mechanism. > > In addition, "notmuch help emacs-mua" doesn't work, eve

[PATCH v2 19/21] crypto: add --decrypt=nostash to avoid stashing session keys

2017-11-30 Thread Daniel Kahn Gillmor
Here's the configuration choice for people who want a cleartext index, but don't want stashed session keys. Interestingly, this "nostash" decryption policy is actually the same policy that should be used by "notmuch show" and "notmuch reply", since they never modify the index or database when

[PATCH v2 12/21] cli/show: use decryption policy "auto" by default.

2017-11-30 Thread Daniel Kahn Gillmor
When showing a message, if the user doesn't specify --decrypt= at all, but a stashed session key is known to notmuch, notmuch should just go ahead and try to decrypt the message with the session key (without bothering the user for access to their asymmetric secret key). The user can disable this

[PATCH v2 16/21] cli/new, insert, reindex: change index.decrypt to "auto" by default

2017-11-30 Thread Daniel Kahn Gillmor
The new "auto" decryption policy is not only good for "notmuch show" and "notmuch reindex". It's also useful for indexing messages -- there's no good reason to not try to go ahead and index the cleartext of a message that we have a stashed session key for. This change updates the defaults and

[PATCH v2 08/21] indexopts: change _try_decrypt to _decrypt_policy

2017-11-30 Thread Daniel Kahn Gillmor
This terminology makes it clearer what's going on at the API layer, and paves the way for future changesets that offer more nuanced decryption policy. --- lib/index.cc | 2 +- lib/indexopts.c | 10 +- lib/notmuch.h| 8 notmuch-client.h | 4 ++-- notmuch.c| 12

[PATCH v2 20/21] docs: clean up documentation about decryption policies

2017-11-30 Thread Daniel Kahn Gillmor
Now that the range of sensible decryption policies has come into full view, we take a bit of space to document the distinctions. Most people will use either "auto" or "true" -- but we provide "false" and "nostash" to handle use cases that might reasonably be requested. Note also that these can

[PATCH v2 11/21] cli/reply: use decryption policy "auto" by default.

2017-11-30 Thread Daniel Kahn Gillmor
If the user doesn't specify --decrypt= at all, but a stashed session key is known to notmuch, when replying to an encrypted message, notmuch should just go ahead and decrypt. The user can disable this at the command line with --decrypt=false, though it's not clear why they would ever want to do

[PATCH v2 09/21] lib: convert notmuch decryption policy to an enum

2017-11-30 Thread Daniel Kahn Gillmor
Future patches in this series will introduce new policies; this merely readies the way for them. We also convert --try-decrypt to a keyword argument instead of a boolean. --- lib/index.cc | 2 +- lib/indexopts.c | 21 +++-- lib/notmuch.h

[PATCH v2 14/21] cli/new, insert, reindex: update documentation for --decrypt=auto

2017-11-30 Thread Daniel Kahn Gillmor
we also include --decrypt=auto in the tab completion. --- completion/notmuch-completion.bash | 6 +++--- doc/man1/notmuch-insert.rst| 16 ++-- doc/man1/notmuch-new.rst | 10 +++--- doc/man1/notmuch-reindex.rst | 23 ++- 4 files changed,

[PATCH v2 10/21] crypto: new decryption policy "auto"

2017-11-30 Thread Daniel Kahn Gillmor
This new automatic decryption policy should make it possible to decrypt messages that we have stashed session keys for, without incurring a call to the user's asymmetric keys. --- doc/man1/notmuch-config.rst | 11 --- lib/index.cc | 3 ++- lib/indexopts.c

session keys, version 2

2017-11-30 Thread Daniel Kahn Gillmor
This is the second revision of the session keys series. the earlier version of this series can be found following id:20171025065203.24403-1-...@fifthhorseman.net. This version addresses the ideas and critiques raised on list about the first version. In particular: * ./configure now detects

[PATCH v2 13/21] cli/show, reply: document use of stashed session keys in notmuch-properties

2017-11-30 Thread Daniel Kahn Gillmor
The stashed session keys are stored internally as notmuch properties. So a user or developer who is reading about those properties might want to understand how they fit into the bigger picture. Note here that decrypting with a stored session key no longer needs -decrypt for "notmuch show" and

<    1   2   3   4   5   6   7   8   9   10   >