Re: [PATCH v2 06/21] crypto: Test restore of cleartext index from stashed session keys

[Daniel Kahn Gillmor]

On Thu 2017-12-07 00:20:12 -0800, Jameson Graef Rollins wrote:
> On Mon, Dec 04 2017, David Bremner  wrote:
>> Pushed patches 1 to 6. I seem to recall 7 and 8 basically adressed
>> concerns/suggestions Jamie had, so I'm hoping he can have a quick look
>> at those.
>
> Yes, this new series is great and definitely addresses all my concerns.
> I'm stoked to see that the first part of it has been pushed, and looking
> forward to the full series!
>
> This is really great progress, Daniel.  Thanks for pushing on this.

Thanks for the review!  I've just pushed v3 of what remains of this
series, which is basically the same as the remaining patches here, with
a couple minor cleanups.

you can find it starting at
id:20171208062404.17269-1-...@fifthhorseman.net

--dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 11/15] cli/reindex: destroy stashed session keys when --decrypt=false

[Daniel Kahn Gillmor]

There are some situations where the user wants to get rid of the
cleartext index of a message.  For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.

In that case, the natural thing to do is:

   notmuch reindex --decrypt=false id:whate...@example.biz

But of course, clearing the cleartext index without clearing the
stashed session key is just silly.  So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.

Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).

As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist.  This problem really needs to be addressed within
xapian, though, if we want it fixed right.
---
 doc/man1/notmuch-reindex.rst  |  3 +++
 lib/message.cc|  5 +
 test/T357-index-decryption.sh | 17 +
 3 files changed, 25 insertions(+)

diff --git a/doc/man1/notmuch-reindex.rst b/doc/man1/notmuch-reindex.rst
index d87e9d85..e8174f39 100644
--- a/doc/man1/notmuch-reindex.rst
+++ b/doc/man1/notmuch-reindex.rst
@@ -30,6 +30,9 @@ Supported options for **reindex** include
 the user's secret keys.  If decryption is successful, index
 the cleartext itself.
 
+If ``false``, notmuch reindex will also delete any stashed
+    session keys for all messages matching the search terms.
+
 Be aware that the index is likely sufficient to reconstruct
 the cleartext of the message itself, so please ensure that the
 notmuch message index is adequately protected. DO NOT USE
diff --git a/lib/message.cc b/lib/message.cc
index 12743460..d5db89b6 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -2002,6 +2002,11 @@ notmuch_message_reindex (notmuch_message_t *message,
 ret = notmuch_message_remove_all_properties_with_prefix (message, 
"index.");
 if (ret)
goto DONE; /* XXX TODO: distinguish from other error returns above? */
+if (indexopts && notmuch_indexopts_get_decrypt_policy (indexopts) == 
NOTMUCH_DECRYPT_FALSE) {
+   ret = notmuch_message_remove_all_properties (message, "session-key");
+   if (ret)
+   goto DONE;
+}
 
 /* re-add the filenames with the associated indexopts */
 for (; notmuch_filenames_valid (orig_filenames);
diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh
index bd213415..9f46a01b 100755
--- a/test/T357-index-decryption.sh
+++ b/test/T357-index-decryption.sh
@@ -227,6 +227,23 @@ test_expect_equal \
 "$output" \
 "$expected"
 
+test_begin_subtest "purging stashed session keys should lose access to the 
cleartext"
+notmuch reindex --decrypt=false id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
+test_begin_subtest "and cleartext should be unrecoverable now that there are 
no stashed session keys"
+notmuch dump
+notmuch reindex --decrypt=true id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
 
 # TODO: test removal of a message from the message store between
 # indexing and reindexing.
-- 
2.15.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 07/15] cli/show, reply: document use of stashed session keys in notmuch-properties

[Daniel Kahn Gillmor]

The stashed session keys are stored internally as notmuch properties.
So a user or developer who is reading about those properties might
want to understand how they fit into the bigger picture.

Note here that decrypting with a stored session key no longer needs
-decrypt for "notmuch show" and "notmuch reply".
---
 doc/man7/notmuch-properties.rst | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/doc/man7/notmuch-properties.rst b/doc/man7/notmuch-properties.rst
index 05444f6c..1a3f690e 100644
--- a/doc/man7/notmuch-properties.rst
+++ b/doc/man7/notmuch-properties.rst
@@ -77,9 +77,12 @@ of its normal activity.
 **session-key**
 
 When **notmuch-show(1)** or **nomtuch-reply** encounters a message
-with an encrypted part and ``--decrypt`` is set, if notmuch finds a
-``session-key`` property associated with the message, it will try
-that stashed session key for decryption.
+with an encrypted part, if notmuch finds a ``session-key``
+property associated with the message, it will try that stashed
+session key for decryption.
+
+If you do not want to use any stashed session keys that might be
+present, you should pass those programs ``--decrypt=false``.
 
 Using a stashed session key with "notmuch show" will speed up
 rendering of long encrypted threads.  It also allows the user to
-- 
2.15.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v2 06/21] crypto: Test restore of cleartext index from stashed session keys

[Jameson Graef Rollins]

On Mon, Dec 04 2017, David Bremner  wrote:
> Pushed patches 1 to 6. I seem to recall 7 and 8 basically adressed
> concerns/suggestions Jamie had, so I'm hoping he can have a quick look
> at those.

Yes, this new series is great and definitely addresses all my concerns.
I'm stoked to see that the first part of it has been pushed, and looking
forward to the full series!

This is really great progress, Daniel.  Thanks for pushing on this.

jamie.


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v2 06/21] crypto: Test restore of cleartext index from stashed session keys

[Daniel Kahn Gillmor]

On Mon 2017-12-04 21:59:18 -0400, David Bremner wrote:
> Pushed patches 1 to 6. I seem to recall 7 and 8 basically adressed
> concerns/suggestions Jamie had, so I'm hoping he can have a quick look
> at those.

to be fair, i thought Jamie's concerns were correct -- the normalized
interface is better.  i was only a bit leery about the change
introducing an extra delay.  But Jamie's concerns were also corroborated
by id:87r2szgvik@tethera.net -- maybe you could ask the author of
that message to review?  :P I think Jamie is currently slammed by
non-notmuch work, unfortunately.

   --dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v2 06/21] crypto: Test restore of cleartext index from stashed session keys

[David Bremner]

Pushed patches 1 to 6. I seem to recall 7 and 8 basically adressed
concerns/suggestions Jamie had, so I'm hoping he can have a quick look
at those.

d



___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2 13/21] cli/show, reply: document use of stashed session keys in notmuch-properties

[Daniel Kahn Gillmor]

The stashed session keys are stored internally as notmuch properties.
So a user or developer who is reading about those properties might
want to understand how they fit into the bigger picture.

Note here that decrypting with a stored session key no longer needs
-decrypt for "notmuch show" and "notmuch reply".
---
 doc/man7/notmuch-properties.rst | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/doc/man7/notmuch-properties.rst b/doc/man7/notmuch-properties.rst
index f27dd607..1bb7b1ca 100644
--- a/doc/man7/notmuch-properties.rst
+++ b/doc/man7/notmuch-properties.rst
@@ -77,9 +77,12 @@ of its normal activity.
 **session-key**
 
 When **notmuch-show(1)** or **nomtuch-reply** encounters a message
-with an encrypted part and ``--decrypt`` is set, if notmuch finds a
-``session-key`` property associated with the message, it will try
-that stashed session key for decryption.
+with an encrypted part, if notmuch finds a ``session-key``
+property associated with the message, it will try that stashed
+session key for decryption.
+
+If you do not want to use any stashed session keys that might be
+present, you should pass those programs ``--decrypt=false``.
 
 Using a stashed session key with "notmuch show" will speed up
 rendering of long encrypted threads.  It also allows the user to
-- 
2.15.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2 06/21] crypto: Test restore of cleartext index from stashed session keys

[Daniel Kahn Gillmor]

If you've got a notmuch dump that includes stashed session keys for
every decrypted message, and you've got your message archive, you
should be able to get back to the same index that you had before.

Here we add a simple test that give some flavor of how that works.
---
 test/T357-index-decryption.sh | 31 +++
 1 file changed, 31 insertions(+)

diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh
index 22e716c6..11ea2074 100755
--- a/test/T357-index-decryption.sh
+++ b/test/T357-index-decryption.sh
@@ -156,6 +156,37 @@ test_expect_equal \
 "$output" \
 "$expected"
 
+add_email_corpus crypto
+
+test_begin_subtest "indexing message fails when secret key not available"
+notmuch reindex --try-decrypt id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch dump )
+expected='#notmuch-dump batch-tag:3 config,properties,tags
++encrypted +inbox +unread -- id:simple-encryp...@crypto.notmuchmail.org
+#= simple-encryp...@crypto.notmuchmail.org index.decryption=failure'
+test_expect_equal \
+"$output" \
+"$expected"
+
+test_begin_subtest "cannot find cleartext index"
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
+test_begin_subtest "cleartext index recovery on reindexing with stashed 
session keys"
+notmuch restore <https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2 17/21] cli/reindex: destroy stashed session keys when --decrypt=false

[Daniel Kahn Gillmor]

There are some situations where the user wants to get rid of the
cleartext index of a message.  For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.

In that case, the natural thing to do is:

   notmuch reindex --decrypt=false id:whate...@example.biz

But of course, clearing the cleartext index without clearing the
stashed session key is just silly.  So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.

Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).

As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist.  This problem really needs to be addressed within
xapian, though, if we want it fixed right.
---
 doc/man1/notmuch-reindex.rst  |  3 +++
 lib/message.cc|  5 +
 test/T357-index-decryption.sh | 17 +
 3 files changed, 25 insertions(+)

diff --git a/doc/man1/notmuch-reindex.rst b/doc/man1/notmuch-reindex.rst
index d87e9d85..e8174f39 100644
--- a/doc/man1/notmuch-reindex.rst
+++ b/doc/man1/notmuch-reindex.rst
@@ -30,6 +30,9 @@ Supported options for **reindex** include
 the user's secret keys.  If decryption is successful, index
 the cleartext itself.
 
+If ``false``, notmuch reindex will also delete any stashed
+    session keys for all messages matching the search terms.
+
 Be aware that the index is likely sufficient to reconstruct
 the cleartext of the message itself, so please ensure that the
 notmuch message index is adequately protected. DO NOT USE
diff --git a/lib/message.cc b/lib/message.cc
index 12743460..d5db89b6 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -2002,6 +2002,11 @@ notmuch_message_reindex (notmuch_message_t *message,
 ret = notmuch_message_remove_all_properties_with_prefix (message, 
"index.");
 if (ret)
goto DONE; /* XXX TODO: distinguish from other error returns above? */
+if (indexopts && notmuch_indexopts_get_decrypt_policy (indexopts) == 
NOTMUCH_DECRYPT_FALSE) {
+   ret = notmuch_message_remove_all_properties (message, "session-key");
+   if (ret)
+   goto DONE;
+}
 
 /* re-add the filenames with the associated indexopts */
 for (; notmuch_filenames_valid (orig_filenames);
diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh
index b2717a7a..64317c64 100755
--- a/test/T357-index-decryption.sh
+++ b/test/T357-index-decryption.sh
@@ -218,6 +218,23 @@ test_expect_equal \
 "$output" \
 "$expected"
 
+test_begin_subtest "purging stashed session keys should lose access to the 
cleartext"
+notmuch reindex --decrypt=false id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
+test_begin_subtest "and cleartext should be unrecoverable now that there are 
no stashed session keys"
+notmuch dump
+notmuch reindex --decrypt=true id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
 
 # TODO: test removal of a message from the message store between
 # indexing and reindexing.
-- 
2.15.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Stashed session keys

[Daniel Kahn Gillmor]

Hi meskio--

On Wed 2017-11-15 23:41:28 +0100, meskio wrote:
> Nice feature. I'm using it and it works fine. I notice some speed up, 
> improving 
> the painfulness of reading long encrypted threads in alot. And I like to 
> don't 
> be able to have around my old private keys.

cool, i'm glad it's working for you!

> I implemented some support for it in alot (using the patch I just sent adding 
> notmuch_message_get_property to the python binding):
> https://github.com/meskio/alot/tree/session-key

very nice :)

 --dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Stashed session keys

[meskio]

Quoting Daniel Kahn Gillmor (2017-10-25 08:51:45)
> Now that cleartext indexing is merged, let's add the ability to stash
> session keys!

Nice feature. I'm using it and it works fine. I notice some speed up, improving 
the painfulness of reading long encrypted threads in alot. And I like to don't 
be able to have around my old private keys.

I implemented some support for it in alot (using the patch I just sent adding 
notmuch_message_get_property to the python binding):
https://github.com/meskio/alot/tree/session-key

Thanks for the work.

-- 
meskio | http://meskio.net/
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
 My contact info: http://meskio.net/crypto.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Nos vamos a Croatan.


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 05/18] crypto: Test restore of cleartext index from stashed session keys

[David Bremner]

Daniel Kahn Gillmor <d...@fifthhorseman.net> writes:

> On Tue 2017-11-14 09:13:52 -0400, David Bremner wrote:
>> Daniel Kahn Gillmor <d...@fifthhorseman.net> writes:
>>
>>> If you've got a notmuch dump that includes stashed session keys for
>>> every decrypted message, and you've got your message archive, you
>>> should be able to get back to the same index that you had before.
>>
>> Out of curiousity, have you given any thought to what happens when
>> someone sends a message with the same message-id but a different
>> session-key? it seems like the user can potentially lose access to the
>> encrypted message.
>
> yep!  I even have that case in my own mailbox due to messages i've sent
> to schleuder encrypted mailing lists to which i'm also subscribed.
>
> It works fine.  notmuch stashes both session keys against the message-id
> (you can have multiple properties with the same name as long as they
> have different values).  And upon decryption, it tries each session-key
> in succession.  This is a little bit sloppy (maybe it would be less
> sloppy to associate each message key with each version of the message
> somehow?), but it's significantly simpler and basically unnoticeable
> compared to the speedup gains provided by the rest of the series.
>
>  --dkg

Great!

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 05/18] crypto: Test restore of cleartext index from stashed session keys

[Daniel Kahn Gillmor]

On Tue 2017-11-14 09:13:52 -0400, David Bremner wrote:
> Daniel Kahn Gillmor <d...@fifthhorseman.net> writes:
>
>> If you've got a notmuch dump that includes stashed session keys for
>> every decrypted message, and you've got your message archive, you
>> should be able to get back to the same index that you had before.
>
> Out of curiousity, have you given any thought to what happens when
> someone sends a message with the same message-id but a different
> session-key? it seems like the user can potentially lose access to the
> encrypted message.

yep!  I even have that case in my own mailbox due to messages i've sent
to schleuder encrypted mailing lists to which i'm also subscribed.

It works fine.  notmuch stashes both session keys against the message-id
(you can have multiple properties with the same name as long as they
have different values).  And upon decryption, it tries each session-key
in succession.  This is a little bit sloppy (maybe it would be less
sloppy to associate each message key with each version of the message
somehow?), but it's significantly simpler and basically unnoticeable
compared to the speedup gains provided by the rest of the series.

 --dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 05/18] crypto: Test restore of cleartext index from stashed session keys

[David Bremner]

Daniel Kahn Gillmor <d...@fifthhorseman.net> writes:

> If you've got a notmuch dump that includes stashed session keys for
> every decrypted message, and you've got your message archive, you
> should be able to get back to the same index that you had before.
>

Out of curiousity, have you given any thought to what happens when
someone sends a message with the same message-id but a different
session-key? it seems like the user can potentially lose access to the
encrypted message.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Stashed session keys

[Daniel Kahn Gillmor]

On Sun 2017-11-12 11:51:11 +0800, Daniel Kahn Gillmor wrote:
> On Sat 2017-11-11 15:31:36 -0800, Jameson Graef Rollins wrote:
>> I haven't decided what's the best way to do that yet, but something
>> like the following happening automatically at inbox view might do the
>> trick:
>>
>>   notmuch reindex --try-decrypt=true (tag:inbox AND tag:encrypted)
>
> This seems like a reasonable way to ensure that your long-term, personal
> secret keys only get accessed when you are interactively working with
> your mail user agent.
>
> You might be able to target the reindex even more narrowly by adding
> something like "AND not property:index-decryption=success"

Sorry, this should be "AND not property:index.decryption=success"

   --dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Stashed session keys

[Jameson Graef Rollins]

On Sun, Nov 12 2017, Daniel Kahn Gillmor  wrote:
>> Finally, I think it would be worthwhile to resolve the disparity between
>> the usage of "decrypt" and "try-decrypt" in the CLI and config options.
>> I'm not sure why we're using different terms in different contexts, even
>> though the meanings are essentially the same.  A follow-up patch series
>> changing "try-decrypt" -> "decrypt" would probably be in order.
>
> If this series lands, i'd be happy to supply such an term-normalizing
> series for subsequent consideration.
>
> If people feel that this term normalization is the main blocker to
> landing this series, i could try to rebase it with a different UI terms,
> but rebasing the series for this change feels like busy-work to me (and
> would be more effort than a simple normalization patch on top).  i'd
> rather spend my limited notmuch hacking+reviewing time providing useful
> features.

I don't think it's a blocker at all, but I do think the term
normalization should happen before the next release, since I think we
should change "try-decrypt" -> "decrypt", which would affect things
already in master.  But I very much hope this series will land before
the next release, in which case I think it's fine to wait to do the
normalization after this series lands.

jamie.


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Stashed session keys

[Daniel Kahn Gillmor]

On Sat 2017-11-11 15:31:36 -0800, Jameson Graef Rollins wrote:
> I have reviewed and tested this series, and it seems solidly
> implemented and very well motivated.  I have been using it regularly
> for a couple weeks now and have found no issues with it's usage, and
> have noticed the considerable speed up when viewing encrypted threads
> (as much as x8 for show on a thread of just 8 encrypted messages).  I
> fully support it's integration unconditionally.

thanks for the review, the testing, and the reportback.  I'm glad to
hear that it's giving you the same sort of speedups that it gives me.

> Daniel likes to think of this in terms of being able to "delete"
> encrypted messages in the wild (via deletion of the original
> encryption key) whereas I like to think of it in terms of preserving
> access to received encrypted messages after key rotation.  Both
> benefits hold, though, obviously.

yes, these are different ways of looking at the same key management
strategy.

> I think these policies cover all potential use cases that I can see.
> However, there will need to be further work on the UX to make things
> flow more smoothly.

Agreed.  The goal of this series is to provide the framework that can be
used to build smoother UX, but it doesn't get all the way to providing
the smoothest possible UX.  Such is the nature of toolkit development.

> I haven't decided what's the best way to do that yet, but something
> like the following happening automatically at inbox view might do the
> trick:
>
>   notmuch reindex --try-decrypt=true (tag:inbox AND tag:encrypted)

This seems like a reasonable way to ensure that your long-term, personal
secret keys only get accessed when you are interactively working with
your mail user agent.

You might be able to target the reindex even more narrowly by adding
something like "AND not property:index-decryption=success"

> Finally, I think it would be worthwhile to resolve the disparity between
> the usage of "decrypt" and "try-decrypt" in the CLI and config options.
> I'm not sure why we're using different terms in different contexts, even
> though the meanings are essentially the same.  A follow-up patch series
> changing "try-decrypt" -> "decrypt" would probably be in order.

If this series lands, i'd be happy to supply such an term-normalizing
series for subsequent consideration.

If people feel that this term normalization is the main blocker to
landing this series, i could try to rebase it with a different UI terms,
but rebasing the series for this change feels like busy-work to me (and
would be more effort than a simple normalization patch on top).  i'd
rather spend my limited notmuch hacking+reviewing time providing useful
features.

  --dkg


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Stashed session keys

[Daniel Kahn Gillmor]

On Wed 2017-10-25 02:51:45 -0400, Daniel Kahn Gillmor wrote:
> Now that cleartext indexing is merged, let's add the ability to stash
> session keys!

I'd really appreciate feedback on this series.  It works for me, and i'm
using it.  I don't want this to take another two years to land, if
possible.  And it blocks additional improvements that i would like to
eventually land as well.

So any review of any part of this series would be appreciated.  I want
to make notmuch a mail client that provides actually usable encrypted
mail.

Regards,

  --dkg


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 05/18] crypto: Test restore of cleartext index from stashed session keys

[Daniel Kahn Gillmor]

If you've got a notmuch dump that includes stashed session keys for
every decrypted message, and you've got your message archive, you
should be able to get back to the same index that you had before.

Here we add a simple test that give some flavor of how that works.
---
 test/T357-index-decryption.sh | 31 +++
 1 file changed, 31 insertions(+)

diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh
index 22e716c6..11ea2074 100755
--- a/test/T357-index-decryption.sh
+++ b/test/T357-index-decryption.sh
@@ -156,6 +156,37 @@ test_expect_equal \
 "$output" \
 "$expected"
 
+add_email_corpus crypto
+
+test_begin_subtest "indexing message fails when secret key not available"
+notmuch reindex --try-decrypt id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch dump )
+expected='#notmuch-dump batch-tag:3 config,properties,tags
++encrypted +inbox +unread -- id:simple-encryp...@crypto.notmuchmail.org
+#= simple-encryp...@crypto.notmuchmail.org index.decryption=failure'
+test_expect_equal \
+"$output" \
+"$expected"
+
+test_begin_subtest "cannot find cleartext index"
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
+test_begin_subtest "cleartext index recovery on reindexing with stashed 
session keys"
+notmuch restore <https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 14/18] cli/reindex: destroy stashed session keys when --try-decrypt=false

[Daniel Kahn Gillmor]

There are some situations where the user wants to get rid of the
cleartext index of a message.  For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.

In that case, the natural thing to do is:

   notmuch reindex --try-decrypt=false id:whate...@example.biz

But of course, clearing the cleartext index without clearing the
stashed session key is just silly.  So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.

Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).

As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist.  This problem really needs to be addressed within
xapian, though, if we want it fixed right.
---
 doc/man1/notmuch-reindex.rst  |  3 +++
 lib/message.cc|  5 +
 test/T357-index-decryption.sh | 17 +
 3 files changed, 25 insertions(+)

diff --git a/doc/man1/notmuch-reindex.rst b/doc/man1/notmuch-reindex.rst
index b15981a2..6f5e48a9 100644
--- a/doc/man1/notmuch-reindex.rst
+++ b/doc/man1/notmuch-reindex.rst
@@ -30,6 +30,9 @@ Supported options for **reindex** include
 the user's secret keys.  If decryption is successful, index
 the cleartext itself.
 
+If ``false``, notmuch reindex will also delete any stashed
+    session keys for all messages matching the search terms.
+
 Be aware that the index is likely sufficient to reconstruct
 the cleartext of the message itself, so please ensure that the
 notmuch message index is adequately protected. DO NOT USE
diff --git a/lib/message.cc b/lib/message.cc
index 12743460..cfd99130 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -2002,6 +2002,11 @@ notmuch_message_reindex (notmuch_message_t *message,
 ret = notmuch_message_remove_all_properties_with_prefix (message, 
"index.");
 if (ret)
goto DONE; /* XXX TODO: distinguish from other error returns above? */
+if (indexopts && notmuch_indexopts_get_try_decrypt (indexopts) == 
NOTMUCH_DECRYPT_FALSE) {
+   ret = notmuch_message_remove_all_properties (message, "session-key");
+   if (ret)
+   goto DONE;
+}
 
 /* re-add the filenames with the associated indexopts */
 for (; notmuch_filenames_valid (orig_filenames);
diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh
index 7fc59f1e..4f5a501a 100755
--- a/test/T357-index-decryption.sh
+++ b/test/T357-index-decryption.sh
@@ -218,6 +218,23 @@ test_expect_equal \
 "$output" \
 "$expected"
 
+test_begin_subtest "purging stashed session keys should lose access to the 
cleartext"
+notmuch reindex --try-decrypt=false id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
+test_begin_subtest "and cleartext should be unrecoverable now that there are 
no stashed session keys"
+notmuch dump
+notmuch reindex --try-decrypt=true id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
 
 # TODO: test removal of a message from the message store between
 # indexing and reindexing.
-- 
2.14.2

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 10/18] cli/show, reply: document use of stashed session keys in notmuch-properties

[Daniel Kahn Gillmor]

The stashed session keys are stored internally as notmuch properties.
So a user or developer who is reading about those properties might
want to understand how they fit into the bigger picture.

Note here that decrypting with a stored session key no longer needs
-decrypt for "notmuch show" and "notmuch reply".
---
 doc/man7/notmuch-properties.rst | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/doc/man7/notmuch-properties.rst b/doc/man7/notmuch-properties.rst
index 31d7a104..e2ab43be 100644
--- a/doc/man7/notmuch-properties.rst
+++ b/doc/man7/notmuch-properties.rst
@@ -77,9 +77,12 @@ of its normal activity.
 **session-key**
 
 When **notmuch-show(1)** or **nomtuch-reply** encounters a message
-with an encrypted part and ``--decrypt`` is set, if notmuch finds a
-``session-key=`` property associated with the message, it will try
-that stashed session key for decryption.
+with an encrypted part, if notmuch finds a ``session-key=``
+property associated with the message, it will try that stashed
+session key for decryption.
+
+If you do not want to use any stashed session keys that might be
+present, you should pass those programs ``--decrypt=false``.
 
 Using a stashed session key with "notmuch show" will speed up
 rendering of long encrypted threads.  It also allows the user to
-- 
2.14.2

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch