Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show
Alexander Adolf writes: > > Ok, I have added the line as you suggest. I was wondering though, as in > principle the secret key of test_su...@notmuchmail.org could be > available (it's in the key-ring at least)? Yes, it's added to the test suite's temporary keyring. One of the messages in test/corpora/crypto is not encrypted to that key though. $ for file in *; do echo $file gpg < $file |& grep encrypted; echo done basic-encrypted.eml gpg: encrypted with 1024-bit RSA key, ID C44D36DEAD54AB16, created 2011-02-05 encrypted-rfc822-attachment gpg: encrypted with 4096-bit RSA key, ID 70E3C0DE87068451, created 2019-06-08 gpg: encrypted with 1024-bit RSA key, ID C44D36DEAD54AB16, created 2011-02-05 encrypted-signed.eml gpg: encrypted with 1024-bit RSA key, ID C44D36DEAD54AB16, created 2011-02-05 simple-encrypted gpg: encrypted with 4096-bit RSA key, ID 3B7AA7F014E69B5A, created 2016-12-21 > It seems this refers to the very first line of the expected output? I > have updated the expected file to use an absolute date, and the test > script to set notmuch-show-relative-dates to nil as you suggest. > Yes, it's referring to the first line of output. >> Subject: notmuch-show S/MIME test >> To: test_su...@notmuchmail.org >> Date: Tue, 12 Apr 2022 16:57:30 +0200 >> @@ -6,10 +6,10 @@ >> [ smime.p7m: application/pkcs7-mime ] >> [ Decryption successful ] >> [ multipart/signed ] >> -[ Good signature by: ] >> +[ Good signature by: test_su...@notmuchmail.org ] >> >> This is exactly the problem we just dealt with for T355-smime. > > T357? > >> I think the same solution can be applied, but you will need to inline >> the output so that you can do variable substitution. > > I'm lost on what you're suggesting/expecting for this one. Remove the > angle brackets from the "Good signature" line of the expected file? Have a look at commit 8723e707c15f7b435f07f5d5ea693496bb9769bb, in particular the lines if [ $NOTMUCH_GMIME_EMITS_ANGLE_BRACKETS == 1 ]; then EXPECTED_EMAIL_ADDR='' else EXPECTED_EMAIL_ADDR='test_su...@notmuchmail.org' fi Then you can use $EXPECTED_EMAIL_ADDR in the expected output. This means you'll have to generate the expected output rather than hard coding it in a file. To pick some examples at random, you can see how to do something similar on lines 157 and 222 of T035-read-config.sh > >> [ multipart/mixed ] >> [ multipart/mixed ] >> [ text/plain ] >> The password is "12345678". But don't tell anyone! >> -[ test.dtd: application/octet-stream ] >> +[ test.dtd: application/octet-stream (as application/xml-dtd) ] >> >> This seems related to the function >> #'notmuch-show-get-mime-type-of-application/octet-stream, which is >> calling (mailcap-extension-to-mime "dtd"). Probably this is hard to make >> reproducible, so just seding away an "(as ...)" string is a reasonable >> alternative. >> [...] > > I see; unfortunate choice of MIME part. As it seems I will need to > generate a new test message anyway, what would be a more "portable" > part? PNG? PNG should not be worse, but the output might still need postprocessing to avoid variation. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show
Hello David, admittedly being unfamiliar with the test suite, and not being able to run the tests at my end, this is a bit of a head scratcher for me. Thus up-front apologies, and thanks for bearing with me! David Bremner writes: > [...] > It seems that it is mostly working, but there are a few issues to iron out. > > The first is easy, I think. Since we added a message to the crypto > corpus, we need to adjust the tests. I think it is fine to just add > the new message to the failing test output, as in the diff below. > > T357-index-decryption: Testing indexing decrypted mail > FAIL indexing message fails when secret key not available > --- T357-index-decryption.31.expected 2022-04-13 23:15:02.258922959 > + > +++ T357-index-decryption.31.output 2022-04-13 23:15:02.258922959 > + > @@ -1,5 +1,6 @@ >#= simple-encryp...@crypto.notmuchmail.org index.decryption=failure >#notmuch-dump batch-tag:3 config,properties,tags > ++encrypted +inbox +unread -- > id:575ddaaf0b234fd85e077cfb4d44d...@notmuchmail.org >+encrypted +inbox +unread -- id:basic-encryp...@crypto.notmuchmail.org >+encrypted +inbox +unread -- > id:encrypted-rfc822-attachm...@crypto.notmuchmail.org >+encrypted +inbox +unread -- id:encrypted-sig...@crypto.notmuchmail.org Ok, I have added the line as you suggest. I was wondering though, as in principle the secret key of test_su...@notmuchmail.org could be available (it's in the key-ring at least)? > T450-emacs-show: Testing emacs notmuch-show view > BROKEN show encrypted rfc822 message > !!! Bodypart handler `notmuch-show-insert-part-*/*' threw an error: > !!! Symbol’s value as variable is void: gnus-newsgroup-charset > FAIL process cryptographic MIME parts (S/MIME) > --- T450-emacs-show.21.notmuch-show-smime-encrypted-signed-multipart > 2022-04-13 23:15:11.267167711 + > +++ T450-emacs-show.21.OUTPUT 2022-04-13 23:15:11.267167711 + > @@ -1,4 +1,4 @@ > -test_su...@notmuchmail.org (0 mins. ago) (encrypted inbox) > +test_su...@notmuchmail.org (Yest. 14:57) (encrypted inbox) > > If you set notmuch-show-relative-dates to to nil, something like, with > maybe better indentation: > > -test_emacs '(let ((notmuch-crypto-process-mime t)) > +test_emacs '(let ((notmuch-crypto-process-mime t) > + (notmuch-show-relative-dates nil)) > > Then you will get an actual date that you can hard code. It seems this refers to the very first line of the expected output? I have updated the expected file to use an absolute date, and the test script to set notmuch-show-relative-dates to nil as you suggest. >Subject: notmuch-show S/MIME test >To: test_su...@notmuchmail.org >Date: Tue, 12 Apr 2022 16:57:30 +0200 > @@ -6,10 +6,10 @@ >[ smime.p7m: application/pkcs7-mime ] >[ Decryption successful ] >[ multipart/signed ] > -[ Good signature by: ] > +[ Good signature by: test_su...@notmuchmail.org ] > > This is exactly the problem we just dealt with for T355-smime. T357? > I think the same solution can be applied, but you will need to inline > the output so that you can do variable substitution. I'm lost on what you're suggesting/expecting for this one. Remove the angle brackets from the "Good signature" line of the expected file? >[ multipart/mixed ] >[ multipart/mixed ] >[ text/plain ] >The password is "12345678". But don't tell anyone! > -[ test.dtd: application/octet-stream ] > +[ test.dtd: application/octet-stream (as application/xml-dtd) ] > > This seems related to the function > #'notmuch-show-get-mime-type-of-application/octet-stream, which is > calling (mailcap-extension-to-mime "dtd"). Probably this is hard to make > reproducible, so just seding away an "(as ...)" string is a reasonable > alternative. > [...] I see; unfortunate choice of MIME part. As it seems I will need to generate a new test message anyway, what would be a more "portable" part? PNG? ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show
David Bremner writes: > [...] > It seems that it is mostly working, but there are a few issues to iron out. > [...] Hello David, Apologies for the delay in getting back to you. I was busy with other stuff, and will tend to this shortly. Cheers, --alexander ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show
Alexander Adolf writes: > Hello David, > > David Bremner writes: > >> [...] >> I'm not sure it's less effort, >> [...] > > Neither am I... ;-)) > > It might perhaps seem easier to run the tests I have added to the > attached, updated patch in your complete environment? > > --alexander It seems that it is mostly working, but there are a few issues to iron out. The first is easy, I think. Since we added a message to the crypto corpus, we need to adjust the tests. I think it is fine to just add the new message to the failing test output, as in the diff below. T357-index-decryption: Testing indexing decrypted mail FAIL indexing message fails when secret key not available --- T357-index-decryption.31.expected 2022-04-13 23:15:02.258922959 + +++ T357-index-decryption.31.output 2022-04-13 23:15:02.258922959 + @@ -1,5 +1,6 @@ #= simple-encryp...@crypto.notmuchmail.org index.decryption=failure #notmuch-dump batch-tag:3 config,properties,tags ++encrypted +inbox +unread -- id:575ddaaf0b234fd85e077cfb4d44d...@notmuchmail.org +encrypted +inbox +unread -- id:basic-encryp...@crypto.notmuchmail.org +encrypted +inbox +unread -- id:encrypted-rfc822-attachm...@crypto.notmuchmail.org +encrypted +inbox +unread -- id:encrypted-sig...@crypto.notmuchmail.org T450-emacs-show: Testing emacs notmuch-show view BROKEN show encrypted rfc822 message !!! Bodypart handler `notmuch-show-insert-part-*/*' threw an error: !!! Symbol’s value as variable is void: gnus-newsgroup-charset FAIL process cryptographic MIME parts (S/MIME) --- T450-emacs-show.21.notmuch-show-smime-encrypted-signed-multipart 2022-04-13 23:15:11.267167711 + +++ T450-emacs-show.21.OUTPUT 2022-04-13 23:15:11.267167711 + @@ -1,4 +1,4 @@ -test_su...@notmuchmail.org (0 mins. ago) (encrypted inbox) +test_su...@notmuchmail.org (Yest. 14:57) (encrypted inbox) If you set notmuch-show-relative-dates to to nil, something like, with maybe better indentation: -test_emacs '(let ((notmuch-crypto-process-mime t)) +test_emacs '(let ((notmuch-crypto-process-mime t) + (notmuch-show-relative-dates nil)) Then you will get an actual date that you can hard code. Subject: notmuch-show S/MIME test To: test_su...@notmuchmail.org Date: Tue, 12 Apr 2022 16:57:30 +0200 @@ -6,10 +6,10 @@ [ smime.p7m: application/pkcs7-mime ] [ Decryption successful ] [ multipart/signed ] -[ Good signature by: ] +[ Good signature by: test_su...@notmuchmail.org ] This is exactly the problem we just dealt with for T355-smime. I think the same solution can be applied, but you will need to inline the output so that you can do variable substitution. [ multipart/mixed ] [ multipart/mixed ] [ text/plain ] The password is "12345678". But don't tell anyone! -[ test.dtd: application/octet-stream ] +[ test.dtd: application/octet-stream (as application/xml-dtd) ] This seems related to the function #'notmuch-show-get-mime-type-of-application/octet-stream, which is calling (mailcap-extension-to-mime "dtd"). Probably this is hard to make reproducible, so just seding away an "(as ...)" string is a reasonable alternative. [ smime.p7s: application/pkcs7-signature ] ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show
Hello David, David Bremner writes: > [...] > I'm not sure it's less effort, > [...] Neither am I... ;-)) It might perhaps seem easier to run the tests I have added to the attached, updated patch in your complete environment? --alexander >From ba6f00bbd1803f5cccfafdb262f17b79b6c95252 Mon Sep 17 00:00:00 2001 From: Alexander Adolf Date: Fri, 25 Mar 2022 14:13:28 +0100 Subject: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show When processing encrypted S/MIME messages, after decryption the "last resort" handler notmuch-show-insert-part-*/* was called, because there was no application/pkcs7-mime handler, resulting in the decrypted contents not being displayed. This commit adds a new function notmuch-show-insert-part-application/pkcs7-mime (and an alias notmuch-show-insert-part-application/x-pkcs7-mime for the legacy MIME type) to render the S/MIME protected part after decryption. --- emacs/notmuch-show.el | 17 +++ test/T450-emacs-show.sh | 11 +++- .../crypto/smime-encrypted-signed-multipart | 51 +++ ...much-show-smime-encrypted-signed-multipart | 15 ++ 4 files changed, 93 insertions(+), 1 deletion(-) create mode 100644 test/corpora/crypto/smime-encrypted-signed-multipart create mode 100644 test/emacs-show.expected-output/notmuch-show-smime-encrypted-signed-multipart diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el index 7c1f02c9..b7edfc98 100644 --- a/emacs/notmuch-show.el +++ b/emacs/notmuch-show.el @@ -712,6 +712,23 @@ will return nil if the CID is unknown or cannot be retrieved." (defun notmuch-show-insert-part-application/pgp-encrypted (_msg _part _content-type _nth _depth _button) t) +(defun notmuch-show-insert-part-application/pkcs7-mime (msg part _content-type _nth depth _button) + "Render S/MIME protected content after decryption. + +An alias for this function is also defined to handle entities +using the legacy application/x-pkcs7-mime MIME type." + (let* ((encstatus (car (plist-get part :encstatus))) + (inner-part (car (plist-get part :content +;; Insert a button detailing the encryption status. +(notmuch-crypto-insert-encstatus-button encstatus) +(if (not (string= (plist-get encstatus :status) "bad")) +;; Show all decrypted parts. +(notmuch-show-insert-bodypart msg inner-part depth + +;; Support for the legacy "x-" type. +(fset 'notmuch-show-insert-part-application/x-pkcs7-mime + 'notmuch-show-insert-part-application/pkcs7-mime) + (defun notmuch-show-insert-part-multipart/* (msg part _content-type _nth depth _button) (let ((inner-parts (plist-get part :content)) (start (point))) diff --git a/test/T450-emacs-show.sh b/test/T450-emacs-show.sh index 057ad37e..5c4bdd7e 100755 --- a/test/T450-emacs-show.sh +++ b/test/T450-emacs-show.sh @@ -49,7 +49,7 @@ test_emacs '(let ((notmuch-crypto-process-mime nil)) (test-visible-output))' test_expect_equal_file $EXPECTED/notmuch-show-process-crypto-mime-parts-off OUTPUT -test_begin_subtest "process cryptographic MIME parts" +test_begin_subtest "process cryptographic MIME parts (PGP)" test_emacs '(let ((notmuch-crypto-process-mime t)) (notmuch-show "id:20091117203301.gv3...@dottiness.seas.harvard.edu") (test-visible-output))' @@ -245,4 +245,13 @@ test_emacs "(test-log-error (notmuch-show \"$tid\")))" test_expect_equal "$(cat MESSAGES)" "COMPLETE" +# more crypto tests, using S/MIME from here on: +add_gpgsm_home + +test_begin_subtest "process cryptographic MIME parts (S/MIME)" +test_emacs '(let ((notmuch-crypto-process-mime t)) + (notmuch-show "id:575ddaaf0b234fd85e077cfb4d44d...@notmuchmail.org") + (test-visible-output))' +test_expect_equal_file $EXPECTED/notmuch-show-smime-encrypted-signed-multipart OUTPUT + test_done diff --git a/test/corpora/crypto/smime-encrypted-signed-multipart b/test/corpora/crypto/smime-encrypted-signed-multipart new file mode 100644 index ..f0bd6c51 --- /dev/null +++ b/test/corpora/crypto/smime-encrypted-signed-multipart @@ -0,0 +1,51 @@ +Message-Id: <575ddaaf0b234fd85e077cfb4d44d...@notmuchmail.org> +From: test_su...@notmuchmail.org +To: test_su...@notmuchmail.org +Subject: notmuch-show S/MIME test +Date: Tue, 12 Apr 2022 16:57:30 +0200 +MIME-Version: 1.0 +Content-Type: application/pkcs7-mime; + smime-type=enveloped-data; + name=smime.p7m +Content-Transfer-Encoding: base64 +Content-Disposition: attachment; filename=smime.p7m + +MIAGCSqGSIb3DQEHA6CAMIACAQAxggFFMIIBQQIBADApMB0xGzAZBgNVBAMTEk5vdG11Y2ggVGVz +dCBTdWl0ZQIIb3SMlL0MZ6kwDQYJKoZIhvcNAQEBBQAEggEAeCEItxJpxL5frDmEeuMRpi8TcCCw +WHraQ//IDMkA6fcbDIA8hfJpRNpiL5AvqQBipPELtb95rHKxOes7fUPMbT5FVA7dm72hur5N9VRj +kN8Jgs6BjpXRKBA4S5eEzu9J2DJYnTWueZUItKlUzXUXg9AWwyEOKtlXfpOEGZ8FSTNQaE4thipO +hmElscsz1tGmw2+8E1dFeXZyHArruqMAxzqOtiM6G3Y5dj1i8V+s6BSRLzep0JQZ0T/Jq5LE5T+E +rYpgjopj1IT7IOwOP6B+YuhkalXjX7ursH9CNsDg+YjvWPn8RblAH5BRKLHNo1jMm9JLPFH9/0qJ +IJAD0U2q4jCABgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBbgb
Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show
Alexander Adolf writes: > David Bremner writes: > >> [...] >> You could also run T355-smime with --debug and inspect >> tmp.T355-smime/mail afterwards, there should be some signed and >> encrypted mails. >> [...] > > $ ./T355-smime.sh --debug > Run tests in a subdir of built notmuch tree. > > Any way around the effort of building from source? I'm interested in the > elisp part only, and I have notmuch installed in /usr/local. > I'm not sure it's less effort, but I started making the tests work without building from source at https://nmbug.notmuchmail.org/nmweb/show/20211025145753.3031094-2-da...@tethera.net It turns out to be actually slightly tricky (the messages are erased part way through that file), do I attach the corresponding message. Hopefully you can figure out how to run your test once it is written. smime-enc.eml:2,S Description: Binary data ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show
David Bremner writes: > [...] > You could also run T355-smime with --debug and inspect > tmp.T355-smime/mail afterwards, there should be some signed and > encrypted mails. > [...] $ ./T355-smime.sh --debug Run tests in a subdir of built notmuch tree. Any way around the effort of building from source? I'm interested in the elisp part only, and I have notmuch installed in /usr/local. Cheers, --alexander ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show
Alexander Adolf writes: > > Where can I find the S/MIME test key? test/smime directory? test/smime/0xE0972A47.p12 > Is there a better way for generating the test messages than importing > the test key into my "production" keychain? Look to T355-smime for > inspiration? The former is probably the simplest. You could also run T355-smime with --debug and inspect tmp.T355-smime/mail afterwards, there should be some signed and encrypted mails. > Where do I put the generated test messages to be able to use them in the > test? It should be fine to add to corpora/crypto, then call add_email_corpus crypto ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show
David Bremner writes: > [...] > I think something like the following test (but with an smime encrypted > message instead of the existing PGP/MIME encrypted message. > > test_begin_subtest "process cryptographic MIME parts" > ;; additional variables can be dynamically bound as needed, but this one > ;; in particular is needed for decryption > test_emacs '(let ((notmuch-crypto-process-mime t)) > ;; simulate the user action > (notmuch-show > "id:20091117203301.gv3...@dottiness.seas.harvard.edu") > ;; this is the test framework, write the terminal display to a file > ;; called OUTPUT > (test-visible-output))' > test_expect_equal_file > $EXPECTED/notmuch-show-process-crypto-mime-parts-on OUTPUT > > you would also need to create a file with a "screen display" in > $EXPECTED, and add that as part of your patch. > > I hope that helps, > [...] To make this work, I will need to generate two S/MIME test messages. One encrypted only, and one encrypted and signed. Where can I find the S/MIME test key? test/smime directory? Is there a better way for generating the test messages than importing the test key into my "production" keychain? Look to T355-smime for inspiration? Where do I put the generated test messages to be able to use them in the test? Many thanks in advance and cheers, --alexander ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show
Alexander Adolf writes: > > I have had a look to both files, and I _think_ T450-emacs-show would > seem more appropriate. It seems that T355-smime is more aimed at the > pure mechanics of en-/decryption, and the handling of that in > libnotmuch. Neither of these have been a problem for me. > > Further looking at T450-emacs-show, I find that it uses a somewhat > extensive framework to implement its test cases. Any chances of giving > me a fast start with this, or will it be down to "use the source, Luke"? > I think something like the following test (but with an smime encrypted message instead of the existing PGP/MIME encrypted message. test_begin_subtest "process cryptographic MIME parts" ;; additional variables can be dynamically bound as needed, but this one ;; in particular is needed for decryption test_emacs '(let ((notmuch-crypto-process-mime t)) ;; simulate the user action (notmuch-show "id:20091117203301.gv3...@dottiness.seas.harvard.edu") ;; this is the test framework, write the terminal display to a file ;; called OUTPUT (test-visible-output))' test_expect_equal_file $EXPECTED/notmuch-show-process-crypto-mime-parts-on OUTPUT you would also need to create a file with a "screen display" in $EXPECTED, and add that as part of your patch. I hope that helps, David ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show
Hello David, David Bremner writes: > [...] > Thanks for sending that. It's a bit tricky to review SMIME stuff as I > believe none of the devs is really using SMIME regularly. One thing that > would help is adding one or more tests. Basically the test suite already > has a SMIME certificate, so if you can encrypt a message to that cert > (or use one of the existing ones), you can test that the message > displays OK in Emacs. The corresponding test set would probably be > T450-emacs-show, but it might be easier to add to T355-smime. Eventually > we'll also need a short update to doc/notmuch-emacs.rst. > [...] Apologies for the delay in getting back to you, and thanks for the pointers. I have had a look to both files, and I _think_ T450-emacs-show would seem more appropriate. It seems that T355-smime is more aimed at the pure mechanics of en-/decryption, and the handling of that in libnotmuch. Neither of these have been a problem for me. Further looking at T450-emacs-show, I find that it uses a somewhat extensive framework to implement its test cases. Any chances of giving me a fast start with this, or will it be down to "use the source, Luke"? Many thanks in advance and looking forward to your thoughts, --alexander ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs/smime: render decrypted MIME entities in notmuch-show
Alexander Adolf writes: > The issue I saw with S/MIME encrypted messages was, that the processing > of the MIME tree stopped after decryption. It turned out there was no > handler for application/(x-)?pkcs-mime type entities in notmuch-show.el, > and the "handler of last resort" notmuch-show-insert-part-*/* was > called. Thus, this patch just adds a notmuch-show handler for > application/(x-)?pkcs-mime type entities. Hi Alexander; Thanks for sending that. It's a bit tricky to review SMIME stuff as I believe none of the devs is really using SMIME regularly. One thing that would help is adding one or more tests. Basically the test suite already has a SMIME certificate, so if you can encrypt a message to that cert (or use one of the existing ones), you can test that the message displays OK in Emacs. The corresponding test set would probably be T450-emacs-show, but it might be easier to add to T355-smime. Eventually we'll also need a short update to doc/notmuch-emacs.rst. I hope that helps move review forward, David ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
