date:20161123

Re: Bug#842291: notmuch processes frequently stuck in select()

2016-11-23 Thread David Bremner

Daniel Kahn Gillmor  writes:
>
>  0) turn off CRL updates entirely during s/mime signature verification
>
>  1) do s/mime signature verification without CRL updates, but schedule
> CRL checks to happen in the background for dirmngr, so that future
> verifications will reflect the cert validity
>
>  2) have dirmngr avoid checking CRLs that it knows it has already
> updated recently
>
>  3) tell dirmngr to use much shorter CRL fetch timeouts
>

>
> Any thoughts on the best way to pursue this?
>
> --dkg

Maybe the issue is in gmime's usage of gpgme. If I understand correctly
(which is far from a sure thing), pkcs7_verify calls gpgme_op_verify
which is synchronous, and (apparently) does not support timeouts. An
alternate strategy would be to call gpgme_op_verify_start, and then call
gpgme_wait, which has a nonblocking mode. I don't really understand the
S/MIME model, but naively it seems OK for signature verification to fail
if the CRL check doesn't finish quickly.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] tests: account for varying-size OpenPGP signatures

2016-11-23 Thread Daniel Kahn Gillmor

On Wed 2016-11-23 11:59:38 -0500, David Bremner wrote:
> there are a bunch of sed oneliners (or perl equivalent) collected in
> test-lib.sh as functions test_*_sanitize. I wonder if that would be
> worthwhile here, to have one place to update regexps etc...  Something
> in the style of notmuch_show_sanitize wrapping a call to
> notmuch_json_show_sanitize

Thanks for the suggestion, please see v2 of this patch.

 --dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2] tests: account for varying-size cryptographic signatures

2016-11-23 Thread Daniel Kahn Gillmor

GnuPG 2.1.16 is now injecting the full issuer fingerprint in its
signatures, which makes them about 32 octets larger when
ascii-armored.

This change in size means that the size of the MIME parts will vary
depending on the version of gpg that the user has installed.  at any
rate, the signature part should be non-zero (this is true for
basically any MIME part), so we just test for that instead of an exact
size.
---
 test/T350-crypto.sh | 22 ++
 test/T355-smime.sh  |  2 +-
 test/test-lib.sh|  3 ++-
 3 files changed, 13 insertions(+), 14 deletions(-)

diff --git a/test/T350-crypto.sh b/test/T350-crypto.sh
index df2dc74..a1e5e20 100755
--- a/test/T350-crypto.sh
+++ b/test/T350-crypto.sh
@@ -59,7 +59,7 @@ expected='[[[{"id": "X",
  "content": "This is a test signed message.\n"},
  {"id": 3,
  "content-type": "application/pgp-signature",
- "content-length": 280}]}]},
+ "content-length": "NONZERO"}]}]},
  ['
 test_expect_equal_json \
 "$output" \
@@ -94,7 +94,7 @@ expected='[[[{"id": "X",
  "content": "This is a test signed message.\n"},
  {"id": 3,
  "content-type": "application/pgp-signature",
- "content-length": 280}]}]},
+ "content-length": "NONZERO"}]}]},
  ['
 test_expect_equal_json \
 "$output" \
@@ -127,7 +127,7 @@ expected='[[[{"id": "X",
  "content": "This is a test signed message.\n"},
  {"id": 3,
  "content-type": "application/pgp-signature",
- "content-length": 280}]}]},
+ "content-length": "NONZERO"}]}]},
  ['
 test_expect_equal_json \
 "$output" \
@@ -197,7 +197,7 @@ expected='[[[{"id": "X",
  "content-type": "multipart/encrypted",
  "content": [{"id": 2,
  "content-type": "application/pgp-encrypted",
- "content-length": 11},
+ "content-length": "NONZERO"},
  {"id": 3,
  "content-type": "multipart/mixed",
  "content": [{"id": 4,
@@ -205,7 +205,7 @@ expected='[[[{"id": "X",
  "content": "This is a test encrypted message.\n"},
  {"id": 5,
  "content-type": "application/octet-stream",
- "content-length": 28,
+ "content-length": "NONZERO",
  "content-transfer-encoding": "base64",
  "filename": "TESTATTACHMENT"}]}]}]},
  ['
@@ -234,11 +234,9 @@ test_expect_equal_file OUTPUT TESTATTACHMENT
 
 test_begin_subtest "decryption failure with missing key"
 mv "${GNUPGHOME}"{,.bak}
-# The length of the encrypted attachment varies so must be normalized.
 output=$(notmuch show --format=json --decrypt subject:"test encrypted message 
001" \
 | notmuch_json_show_sanitize \
-| sed -e 's|"created": [1234567890]*|"created": 946728000|' \
-| sed -e 's|"content-length": 6[1234567890]*|"content-length": 652|')
+| sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "X",
  "match": true,
  "excluded": false,
@@ -255,10 +253,10 @@ expected='[[[{"id": "X",
  "content-type": "multipart/encrypted",
  "content": [{"id": 2,
  "content-type": "application/pgp-encrypted",
- "content-length": 11},
+ "content-length": "NONZERO"},
  {"id": 3,
  "content-type": "application/octet-stream",
- "content-length": 652}]}]},
+ "content-length": "NONZERO"}]}]},
  ['
 test_expect_equal_json \
 "$output" \
@@ -295,7 +293,7 @@ expected='[[[{"id": "X",
  "content-type": "multipart/encrypted",
  "content": [{"id": 2,
  "content-type": "application/pgp-encrypted",
- "content-length": 11},
+ "content-length": "NONZERO"},
  {"id": 3,
  "content-type": "text/plain",
  "content": "This is another test encrypted message.\n"}]}]},
@@ -370,7 +368,7 @@ expected='[[[{"id": "X",
  "content": "This is a test signed message.\n"},
  {"id": 3,
  "content-type": "application/pgp-signature",
- "content-length": 280}]}]},
+ "content-length": "NONZERO"}]}]},
  ['
 test_expect_equal_json \
 "$output" \
diff --git a/test/T355-smime.sh b/test/T355-smime.sh
index d942412..a8be45e 100755
--- a/test/T355-smime.sh
+++ b/test/T355-smime.sh
@@ -69,7 +69,7 @@ expected='[[[{"id": "X",
  "content-type": "text/plain",
  "content": "This is a test signed message.\n"},
  {"id": 3,
-  "content-length": 1922,
+  "content-length": "NONZERO",
   "content-transfer-encoding": "base64",
   "content-type": "application/x-pkcs7-signature",
   "filename": "smime.p7s"}]}]},
diff --git a/test/test-lib.sh b/test/test-lib.sh
index a12c6d0..f55d2c6 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -736,7 +736,8 @@ notmuch_json_show_sanitize ()
-e 's|"Date": "Fri, 05 Jan 2001 [^"]*"|"Date": "GENERATED_DATE"|g' \
-e 's|"filename": "signature.asc",||g' \
-e 's|"filename": "/[^"]*",|"filename": "Y",|g' \
-   -e 's|"timestamp": 97...|"timestamp": 42|g'
+   -e 's|"timestamp": 97...|"timestamp": 42|g' \
+-e 's|"content-length": [1-9][0-9]*|"content-length": "NONZERO"|g'
 }
 
 notmuch_emacs_error_sanitize ()
-- 
2.10.2

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug#842291: notmuch processes frequently stuck in select()

2016-11-23 Thread Daniel Kahn Gillmor

Control: affects 842291 + gpgsm dirmngr

On Wed 2016-11-23 03:50:40 -0500, David Bremner wrote:
> David Bremner  writes:
>
>> Brian May  writes:
>>> strace shows notmuch looping in select.
>>>
>>> select(10, [9], [], NULL, {1, 0})   = 0 (Timeout)
>>> select(10, [9], [], NULL, {1, 0})   = 0 (Timeout)
>>> select(10, [9], [], NULL, {1, 0})   = 0 (Timeout)
>>> select(10, [9], [], NULL, {1, 0})   = 0 (Timeout)
>>> etc
>>>
>>
>> a backtrace would be helpful.
>>
>> d
>
> Nevermind, I managed to download the list archive for debian-devel and
> replicate the bug.
>
> The bug seems to be related to smime signature verification. After
> adding the attached mail message (and running notmuch-new), to replicate
> the hang it suffices to run
>
> % notmuch show --decrypt id:20161116t143809.ga.21721.s...@fsing.rootsland.net 
>  
>
> As far as workarounds, turning off decryption / signature verification
> should allow you to at least view the thread.

I've noticed similar behavior, and it seems to correlate with gpgsm
asking dirmngr for an update to CRLs related to S/MIME certs.

some CRLs simply do not respond at all (resulting in a timeout), and
some do not respond, or are laggy, when accessed over tor specifically.

I see a couple possible ways to consider resolving this, none of them
great, and i don't know exactly how to implement any of them:

 0) turn off CRL updates entirely during s/mime signature verification

 1) do s/mime signature verification without CRL updates, but schedule
CRL checks to happen in the background for dirmngr, so that future
verifications will reflect the cert validity

 2) have dirmngr avoid checking CRLs that it knows it has already
updated recently

 3) tell dirmngr to use much shorter CRL fetch timeouts


Some example traffic from my dirmngr that uses tor (complete with
timestamps indicating just how bad the delays can be):

Nov 22 14:08:24 alice dirmngr[11976]: no CRL available for issuer id 
770B4DA5913F2572B9F679AE0819FB7D77572689
Nov 22 14:08:24 alice dirmngr[11976]: fetching CRL from 
'http://crl.ll.mit.edu/getcrl/LLCA3'
Nov 22 14:08:44 alice dirmngr[11976]: resolving 'crl.ll.mit.edu' failed: No data
Nov 22 14:08:44 alice dirmngr[11976]: can't connect to 'crl.ll.mit.edu': host 
not found
Nov 22 14:08:44 alice dirmngr[11976]: error retrieving 
'http://crl.ll.mit.edu/getcrl/LLCA3': Unknown host
Nov 22 14:08:44 alice dirmngr[11976]: crl_fetch via DP failed: Unknown host
Nov 22 14:08:45 alice dirmngr[11976]: no CRL available for issuer id 
770B4DA5913F2572B9F679AE0819FB7D77572689
Nov 22 14:08:45 alice dirmngr[11976]: fetching CRL from 
'http://crl.ll.mit.edu/getcrl/LLCA3'
Nov 22 14:09:05 alice dirmngr[11976]: resolving 'crl.ll.mit.edu' failed: No data
Nov 22 14:09:05 alice dirmngr[11976]: can't connect to 'crl.ll.mit.edu': host 
not found
Nov 22 14:09:05 alice dirmngr[11976]: error retrieving 
'http://crl.ll.mit.edu/getcrl/LLCA3': Unknown host
Nov 22 14:09:05 alice dirmngr[11976]: crl_fetch via DP failed: Unknown host
Nov 22 14:09:05 alice dirmngr[11976]: no CRL available for issuer id 
26FD002905277B015EE9B2A3C092A348F28A4C6B
Nov 22 14:09:05 alice dirmngr[11976]: fetching CRL from 
'http://crl.startssl.com/sca-client1.crl'
Nov 22 14:09:25 alice dirmngr[11976]: resolving 'crl.startssl.com' failed: No 
data
Nov 22 14:09:25 alice dirmngr[11976]: can't connect to 'crl.startssl.com': host 
not found
Nov 22 14:09:25 alice dirmngr[11976]: error retrieving 
'http://crl.startssl.com/sca-client1.crl': Unknown host
Nov 22 14:09:25 alice dirmngr[11976]: crl_fetch via DP failed: Unknown host
Nov 22 14:09:25 alice dirmngr[11976]: no CRL available for issuer id 
26FD002905277B015EE9B2A3C092A348F28A4C6B
Nov 22 14:09:25 alice dirmngr[11976]: fetching CRL from 
'http://crl.startssl.com/sca-client1.crl'
Nov 22 14:09:45 alice dirmngr[11976]: resolving 'crl.startssl.com' failed: No 
data
Nov 22 14:09:45 alice dirmngr[11976]: can't connect to 'crl.startssl.com': host 
not found
Nov 22 14:09:45 alice dirmngr[11976]: error retrieving 
'http://crl.startssl.com/sca-client1.crl': Unknown host
Nov 22 14:09:45 alice dirmngr[11976]: crl_fetch via DP failed: Unknown host

that's a 20-second lag between each failed check, adding up to 80
seconds delay in rendering a single thread where 4 messages were signed
by S/MIME keys signed by two different authorities.

Fwiw, crl.ll.mit.edu doesn't seem to respond over tor on port 80 at all
in some cases, and in other cases takes nearly a minute to reply:

0 dkg@alice:/tmp/cdtemp.Ue45bu$ time wget -q 
'http://crl.ll.mit.edu/getcrl/LLCA3'

real0m0.694s
user0m0.008s
sys 0m0.008s
0 dkg@alice:/tmp/cdtemp.Ue45bu$ time torsocks wget -q 
'http://crl.ll.mit.edu/getcrl/LLCA3'

real0m58.828s
user0m0.008s
sys 0m0.008s
0 dkg@alice:/tmp/cdtemp.Ue45bu$ 


Any thoughts on the best way to pursue this?

--dkg


signature.asc
Description: PGP signature
___

Re: [PATCH] tests: account for varying-size OpenPGP signatures

2016-11-23 Thread David Bremner

Daniel Kahn Gillmor  writes:

> @@ -37,7 +37,8 @@ test_expect_success 'emacs delivery of signed message' \
>  test_begin_subtest "signature verification"
>  output=$(notmuch show --format=json --verify subject:"test signed message 
> 001" \
>  | notmuch_json_show_sanitize \
> -| sed -e 's|"created": [1234567890]*|"created": 946728000|')
> +| sed -e 's|"created": [1234567890]*|"created": 946728000|' \
> +  -e 's/"content-length": [1-9][0-9]*/"content-length": "NONZERO"/')

there are a bunch of sed oneliners (or perl equivalent) collected in
test-lib.sh as functions test_*_sanitize. I wonder if that would be
worthwhile here, to have one place to update regexps etc...  Something
in the style of notmuch_show_sanitize wrapping a call to
notmuch_json_show_sanitize
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] tests: account for varying-size OpenPGP signatures

2016-11-23 Thread Daniel Kahn Gillmor

GnuPG 2.1.16 is now injecting the full issuer fingerprint in its
signatures, which makes them about 32 octets larger when
ascii-armored.

This change in size means that the size of the MIME parts will vary
depending on the version of gpg that the user has installed.  at any
rate, the signature part should be non-zero, so we just test for that
instead of an exact size.
---
 test/T350-crypto.sh | 20 
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/test/T350-crypto.sh b/test/T350-crypto.sh
index df2dc74..38f984f 100755
--- a/test/T350-crypto.sh
+++ b/test/T350-crypto.sh
@@ -37,7 +37,8 @@ test_expect_success 'emacs delivery of signed message' \
 test_begin_subtest "signature verification"
 output=$(notmuch show --format=json --verify subject:"test signed message 001" 
\
 | notmuch_json_show_sanitize \
-| sed -e 's|"created": [1234567890]*|"created": 946728000|')
+| sed -e 's|"created": [1234567890]*|"created": 946728000|' \
+  -e 's/"content-length": [1-9][0-9]*/"content-length": "NONZERO"/')
 expected='[[[{"id": "X",
  "match": true,
  "excluded": false,
@@ -59,7 +60,7 @@ expected='[[[{"id": "X",
  "content": "This is a test signed message.\n"},
  {"id": 3,
  "content-type": "application/pgp-signature",
- "content-length": 280}]}]},
+ "content-length": "NONZERO"}]}]},
  ['
 test_expect_equal_json \
 "$output" \
@@ -71,7 +72,8 @@ echo "${FINGERPRINT}:6:" | gpg --no-tty --import-ownertrust 
>>"$GNUPGHOME"/trust
 gpg --no-tty --check-trustdb >>"$GNUPGHOME"/trust.log 2>&1
 output=$(notmuch show --format=json --verify subject:"test signed message 001" 
\
 | notmuch_json_show_sanitize \
-| sed -e 's|"created": [1234567890]*|"created": 946728000|')
+| sed -e 's|"created": [1234567890]*|"created": 946728000|'\
+  -e 's/"content-length": [1-9][0-9]*/"content-length": "NONZERO"/')
 expected='[[[{"id": "X",
  "match": true,
  "excluded": false,
@@ -94,7 +96,7 @@ expected='[[[{"id": "X",
  "content": "This is a test signed message.\n"},
  {"id": 3,
  "content-type": "application/pgp-signature",
- "content-length": 280}]}]},
+ "content-length": "NONZERO"}]}]},
  ['
 test_expect_equal_json \
 "$output" \
@@ -105,7 +107,8 @@ test_begin_subtest "signature verification with signer key 
unavailable"
 mv "${GNUPGHOME}"{,.bak}
 output=$(notmuch show --format=json --verify subject:"test signed message 001" 
\
 | notmuch_json_show_sanitize \
-| sed -e 's|"created": [1234567890]*|"created": 946728000|')
+| sed -e 's|"created": [1234567890]*|"created": 946728000|' \
+  -e 's/"content-length": [1-9][0-9]*/"content-length": "NONZERO"/')
 expected='[[[{"id": "X",
  "match": true,
  "excluded": false,
@@ -127,7 +130,7 @@ expected='[[[{"id": "X",
  "content": "This is a test signed message.\n"},
  {"id": 3,
  "content-type": "application/pgp-signature",
- "content-length": 280}]}]},
+ "content-length": "NONZERO"}]}]},
  ['
 test_expect_equal_json \
 "$output" \
@@ -348,7 +351,8 @@ y
 | gpg --no-tty --quiet --import
 output=$(notmuch show --format=json --verify subject:"test signed message 001" 
\
 | notmuch_json_show_sanitize \
-| sed -e 's|"created": [1234567890]*|"created": 946728000|')
+| sed -e 's|"created": [1234567890]*|"created": 946728000|' \
+  -e 's/"content-length": [1-9][0-9]*/"content-length": "NONZERO"/')
 expected='[[[{"id": "X",
  "match": true,
  "excluded": false,
@@ -370,7 +374,7 @@ expected='[[[{"id": "X",
  "content": "This is a test signed message.\n"},
  {"id": 3,
  "content-type": "application/pgp-signature",
- "content-length": 280}]}]},
+ "content-length": "NONZERO"}]}]},
  ['
 test_expect_equal_json \
 "$output" \
-- 
2.10.2

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: some issues with emacs 25

2016-11-23 Thread Matthew Lear

On 23/11/16 11:57, David Bremner wrote:
> In a fresh emacs, can you generate an error using the recipe in
> 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24950 ?

No. I can evaluate the recipe ok and eval of the last line yields:
10 (#o12, #xa, ?\C-j)


___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: some issues with emacs 25

2016-11-23 Thread David Bremner

Matthew Lear  writes:


> I'd happily post some debug info to help analyse this if somebody could
> give me a few pointers about what would be needed and the emacs-fu
> needed to generate this.
>
> Cheers,
> --  Matt


In a fresh emacs, can you generate an error using the recipe in

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24950 ?

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: some issues with emacs 25

2016-11-23 Thread Matthew Lear

On 22/11/16 14:24, Matthew Lear wrote:
> I can confirm that it is indeed fixed. I'm not able to reproduce it with
> the latest emacs 25 from git.

Unfortunately, it seems I was too hasty in confirming this.
For some reason unknown to me, replying to the html in some
multipart/alternative works fine, but others cause the same error to be
thrown. I've also seen the error occur when replying to a
multipart/mixed message, too.

Even more bizarrely, I'm currently looking at a thread with two emails
where both are multipart/alternative and contain text/plain and
text/html parts, and replying to the first message triggers the
error/bug but replying to the second message does not. I can't see any
formatting difference between the two.

Sorry, but it would appear as though there is still some work needed to
resolve this in it's entirety. I suppose it's likely that this could be
an emacs-25 issue, but what I mentioned above makes no sense to me so I
can't say for sure.

I'd happily post some debug info to help analyse this if somebody could
give me a few pointers about what would be needed and the emacs-fu
needed to generate this.

Cheers,
--  Matt

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug#842291: notmuch processes frequently stuck in select()

2016-11-23 Thread David Bremner

David Bremner  writes:

> Brian May  writes:
>> strace shows notmuch looping in select.
>>
>> select(10, [9], [], NULL, {1, 0})   = 0 (Timeout)
>> select(10, [9], [], NULL, {1, 0})   = 0 (Timeout)
>> select(10, [9], [], NULL, {1, 0})   = 0 (Timeout)
>> select(10, [9], [], NULL, {1, 0})   = 0 (Timeout)
>> etc
>>
>
> a backtrace would be helpful.
>
> d

Nevermind, I managed to download the list archive for debian-devel and
replicate the bug.

The bug seems to be related to smime signature verification. After
adding the attached mail message (and running notmuch-new), to replicate
the hang it suffices to run

% notmuch show --decrypt id:20161116t143809.ga.21721.s...@fsing.rootsland.net  

As far as workarounds, turning off decryption / signature verification
should allow you to at least view the thread.

20161116T143809.GA.21721.stse@fsing.rootsland.net.eml:2,
Description: Binary data
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug#842291: notmuch processes frequently stuck in select()

Re: [PATCH] tests: account for varying-size OpenPGP signatures

[PATCH v2] tests: account for varying-size cryptographic signatures

Re: Bug#842291: notmuch processes frequently stuck in select()

Re: [PATCH] tests: account for varying-size OpenPGP signatures

[PATCH] tests: account for varying-size OpenPGP signatures

Re: some issues with emacs 25

Re: some issues with emacs 25

Re: some issues with emacs 25

Re: Bug#842291: notmuch processes frequently stuck in select()

10 matches

Site Navigation

Mail list logo

Footer information