Re: version 4: Retrieve GPG keys asynchronously.
On Wed, Sep 11 2019, Daniel Kahn Gillmor wrote: > In id:87blw1rldy@tethera.net, bremner identified two of my edits > breaking dme's series that permits asynchronous retrieval of OpenPGP > keys. > > This is a revision of that series that should apply to the current > master, taking into account my earlier fixes. > > dme, if you could review i would appreciate it. I think i've done > what you would have done, but as usual my elisp skills are weaker than > yours, so i welcome your feedback. The test suite passes for me at > least :) > > Sorry for contributing to the collision earlier! This series looks good to me. I did not (take time to look how to) test, but I trust dkg has basically executed all changed code :D. Some of the code was just style change w/o functional change (one let* context puzzled me for a while) -- for better which is good! > > --dkg > Tomi ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v5 2/4] util/repair: identify and repair "Mixed Up" mangled messages
On Sun, Sep 15 2019, Daniel Kahn Gillmor wrote:
> This patch implements a functional identification and repair process
If there is going to be more versions, then the above could be changed
to either
1) This commit implements...
or just
2) Implement a functional ...
> for "Mixed Up" MIME messages as described in
> https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling-00#section-4.1
>
> The detection test is not entirely complete, in that it does not
> verify the contents of the latter two message subparts, but this is
> probably safe to skip, because those two parts are unlikely to be
> readable anyway, and the only part we are effectively omitting (the
> first subpart) is guaranteed to be empty anyway, so its removal can be
> reversed if you want to do so. I've left FIXMEs in the code so that
> anyone excited about adding these additional checks can see where to
> put them in.
>
> I'll use this functionality in the next two patches.
If there is going to be more versions, then the above line could be removed...
(as committed changes are not "patches" ...)
Tomi
>
> Signed-off-by: Daniel Kahn Gillmor
> ---
> util/repair.c | 84 +++
> util/repair.h | 10 ++
> 2 files changed, 94 insertions(+)
>
> diff --git a/util/repair.c b/util/repair.c
> index 629e6f23..9fba97b7 100644
> --- a/util/repair.c
> +++ b/util/repair.c
> @@ -120,3 +120,87 @@ _notmuch_repair_crypto_payload_skip_legacy_display
> (GMimeObject *payload)
> return payload;
> }
> }
> +
> +/* see
> + *
> https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling-00#section-4.1.1
> */
> +static bool
> +_notmuch_is_mixed_up_mangled (GMimeObject *part)
> +{
> +GMimeMultipart *mpart = NULL;
> +GMimeObject *parts[3] = {NULL, NULL, NULL};
> +GMimeContentType *type = NULL;
> +char *prelude_string = NULL;
> +bool prelude_is_empty;
> +
> +if (part == NULL)
> + return false;
> +type = g_mime_object_get_content_type (part);
> +if (type == NULL)
> + return false;
> +if (! g_mime_content_type_is_type (type, "multipart", "mixed"))
> + return false;
> +if (! GMIME_IS_MULTIPART (part)) /* probably impossible */
> + return false;
> +mpart = GMIME_MULTIPART (part);
> +if (mpart == NULL)
> + return false;
> +if (g_mime_multipart_get_count (mpart) != 3)
> + return false;
> +parts[0] = g_mime_multipart_get_part (mpart, 0);
> +if (! g_mime_content_type_is_type (g_mime_object_get_content_type
> (parts[0]),
> +"text", "plain"))
> + return false;
> +if (! GMIME_IS_TEXT_PART (parts[0]))
> + return false;
> +parts[1] = g_mime_multipart_get_part (mpart, 1);
> +if (! g_mime_content_type_is_type (g_mime_object_get_content_type
> (parts[1]),
> +"application", "pgp-encrypted"))
> + return false;
> +parts[2] = g_mime_multipart_get_part (mpart, 2);
> +if (! g_mime_content_type_is_type (g_mime_object_get_content_type
> (parts[2]),
> +"application", "octet-stream"))
> + return false;
> +
> +/* Is parts[0] length 0? */
> +prelude_string = g_mime_text_part_get_text (GMIME_TEXT_PART (parts[0]));
> +prelude_is_empty = (prelude_string[0] == '\0');
> +g_free (prelude_string);
> +if (! prelude_is_empty)
> + return false;
> +
> +/* FIXME: after decoding and stripping whitespace, is parts[1]
> + * subpart just "Version: 1" ? */
> +
> +/* FIXME: can we determine that parts[2] subpart is *only* PGP
> + * encrypted data? I tried g_mime_part_get_openpgp_data () but
> + * found https://github.com/jstedfast/gmime/issues/60 */
> +
> +return true;
> +}
> +
> +
> +/* see
> + *
> https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling-00#section-4.1.2
> */
> +GMimeObject *
> +_notmuch_repair_mixed_up_mangled (GMimeObject *part)
> +{
> +GMimeMultipart *mpart = NULL, *mpart_ret = NULL;
> +GMimeObject *ret = NULL;
> +
> +if (! _notmuch_is_mixed_up_mangled (part))
> + return NULL;
> +mpart = GMIME_MULTIPART (part);
> +ret = GMIME_OBJECT (g_mime_multipart_encrypted_new ());
> +if (ret == NULL)
> + return NULL;
> +mpart_ret = GMIME_MULTIPART (ret);
> +if (mpart_ret == NULL) {
> + g_object_unref (ret);
> + return NULL;
> +}
> +g_mime_object_set_content_type_parameter (ret, "protocol",
> "application/pgp-encrypted");
> +
> +g_mime_multipart_insert (mpart_ret, 0, g_mime_multipart_get_part (mpart,
> 1));
> +g_mime_multipart_insert (mpart_ret, 1, g_mime_multipart_get_part (mpart,
> 2));
> +return ret;
> +}
> diff --git a/util/repair.h b/util/repair.h
> index 9974d693..492f5a20 100644
> --- a/util/repair.h
> +++ b/util/repair.h
> @@ -25,9 +25,19 @@ extern "C" {
> * returned object will only be released when the original part is
> * disposed of.
>
[PATCH] Drop devel/printmimestructure (it is in mailscripts 0.11)
mailscripts 0.11 now ships a derivative of devel/printmimestructure
called email-print-mime-structure. Maintenance for that utility will
happen in mailscripts from now on, so we should not track an
independent copy of it in notmuch's source tree.
See https://bugs.debian.org/939993 for more details about the
adoption.
Signed-off-by: Daniel Kahn Gillmor
---
devel/printmimestructure | 69
1 file changed, 69 deletions(-)
delete mode 100755 devel/printmimestructure
diff --git a/devel/printmimestructure b/devel/printmimestructure
deleted file mode 100755
index 70e0a5c0..
--- a/devel/printmimestructure
+++ /dev/null
@@ -1,69 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
-# Author: Daniel Kahn Gillmor
-# License: GPLv3+
-
-# This script reads a MIME message from stdin and produces a treelike
-# representation on it stdout.
-
-# Example:
-#
-# 0 dkg@alice:~$ printmimestructure <
'Maildir/cur/1269025522.M338697P12023.monkey,S=6459,W=6963:2,Sa'
-# └┬╴multipart/signed 6546 bytes
-# ├─╴text/plain inline 895 bytes
-# └─╴application/pgp-signature inline [signature.asc] 836 bytes
-# 0 dkg@alice:~$
-
-
-# If you want to number the parts, i suggest piping the output through
-# something like "cat -n"
-
-from __future__ import print_function
-
-import email
-import sys
-
-def print_part(z, prefix):
-fname = '' if z.get_filename() is None else ' [' + z.get_filename() + ']'
-cset = '' if z.get_charset() is None else ' (' + z.get_charset() + ')'
-disp = z.get_params(None, header='Content-Disposition')
-if (disp is None):
-disposition = ''
-else:
-disposition = ''
-for d in disp:
-if d[0] in [ 'attachment', 'inline' ]:
-disposition = ' ' + d[0]
-if z.is_multipart():
-nbytes = len(z.as_string())
-else:
-nbytes = len(z.get_payload())
-
-print('{}{}{}{}{} {:d} bytes'.format(
-prefix,
-z.get_content_type(),
-cset,
-disposition,
-fname,
-nbytes,
-))
-
-def test(z, prefix=''):
-if (z.is_multipart()):
-print_part(z, prefix+'┬╴')
-if prefix.endswith('└'):
-prefix = prefix.rpartition('└')[0] + ' '
-if prefix.endswith('├'):
-prefix = prefix.rpartition('├')[0] + '│'
-parts = z.get_payload()
-i = 0
-while (i < parts.__len__()-1):
-test(parts[i], prefix + '├')
-i += 1
-test(parts[i], prefix + '└')
-# FIXME: show epilogue?
-else:
-print_part(z, prefix+'─╴')
-
-test(email.message_from_file(sys.stdin), '└')
--
2.23.0
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] Drop devel/printmimestructure (it is in mailscripts 0.11)
On Sun, Sep 15 2019, Daniel Kahn Gillmor wrote:
> mailscripts 0.11 now ships a derivative of devel/printmimestructure
> called email-print-mime-structure. Maintenance for that utility will
> happen in mailscripts from now on, so we should not track an
> independent copy of it in notmuch's source tree.
So only for debian users >;)
(I downloaded the package from packages.debian.org couple of hours ago,
manually, as package damager in this system did not find it.)
I'm fine dropping this, just that helping users to download the tool
might be a bit harder now ?
Tomi
>
> See https://bugs.debian.org/939993 for more details about the
> adoption.
>
> Signed-off-by: Daniel Kahn Gillmor
> ---
> devel/printmimestructure | 69
> 1 file changed, 69 deletions(-)
> delete mode 100755 devel/printmimestructure
>
> diff --git a/devel/printmimestructure b/devel/printmimestructure
> deleted file mode 100755
> index 70e0a5c0..
> --- a/devel/printmimestructure
> +++ /dev/null
> @@ -1,69 +0,0 @@
> -#!/usr/bin/env python
> -# -*- coding: utf-8 -*-
> -
> -# Author: Daniel Kahn Gillmor
> -# License: GPLv3+
> -
> -# This script reads a MIME message from stdin and produces a treelike
> -# representation on it stdout.
> -
> -# Example:
> -#
> -# 0 dkg@alice:~$ printmimestructure <
> 'Maildir/cur/1269025522.M338697P12023.monkey,S=6459,W=6963:2,Sa'
> -# └┬╴multipart/signed 6546 bytes
> -# ├─╴text/plain inline 895 bytes
> -# └─╴application/pgp-signature inline [signature.asc] 836 bytes
> -# 0 dkg@alice:~$
> -
> -
> -# If you want to number the parts, i suggest piping the output through
> -# something like "cat -n"
> -
> -from __future__ import print_function
> -
> -import email
> -import sys
> -
> -def print_part(z, prefix):
> -fname = '' if z.get_filename() is None else ' [' + z.get_filename() + ']'
> -cset = '' if z.get_charset() is None else ' (' + z.get_charset() + ')'
> -disp = z.get_params(None, header='Content-Disposition')
> -if (disp is None):
> -disposition = ''
> -else:
> -disposition = ''
> -for d in disp:
> -if d[0] in [ 'attachment', 'inline' ]:
> -disposition = ' ' + d[0]
> -if z.is_multipart():
> -nbytes = len(z.as_string())
> -else:
> -nbytes = len(z.get_payload())
> -
> -print('{}{}{}{}{} {:d} bytes'.format(
> -prefix,
> -z.get_content_type(),
> -cset,
> -disposition,
> -fname,
> -nbytes,
> -))
> -
> -def test(z, prefix=''):
> -if (z.is_multipart()):
> -print_part(z, prefix+'┬╴')
> -if prefix.endswith('└'):
> -prefix = prefix.rpartition('└')[0] + ' '
> -if prefix.endswith('├'):
> -prefix = prefix.rpartition('├')[0] + '│'
> -parts = z.get_payload()
> -i = 0
> -while (i < parts.__len__()-1):
> -test(parts[i], prefix + '├')
> -i += 1
> -test(parts[i], prefix + '└')
> -# FIXME: show epilogue?
> -else:
> -print_part(z, prefix+'─╴')
> -
> -test(email.message_from_file(sys.stdin), '└')
> --
> 2.23.0
>
> ___
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH v5 2/4] util/repair: identify and repair "Mixed Up" mangled messages
This patch implements a functional identification and repair process
for "Mixed Up" MIME messages as described in
https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling-00#section-4.1
The detection test is not entirely complete, in that it does not
verify the contents of the latter two message subparts, but this is
probably safe to skip, because those two parts are unlikely to be
readable anyway, and the only part we are effectively omitting (the
first subpart) is guaranteed to be empty anyway, so its removal can be
reversed if you want to do so. I've left FIXMEs in the code so that
anyone excited about adding these additional checks can see where to
put them in.
I'll use this functionality in the next two patches.
Signed-off-by: Daniel Kahn Gillmor
---
util/repair.c | 84 +++
util/repair.h | 10 ++
2 files changed, 94 insertions(+)
diff --git a/util/repair.c b/util/repair.c
index 629e6f23..9fba97b7 100644
--- a/util/repair.c
+++ b/util/repair.c
@@ -120,3 +120,87 @@ _notmuch_repair_crypto_payload_skip_legacy_display
(GMimeObject *payload)
return payload;
}
}
+
+/* see
+ *
https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling-00#section-4.1.1
*/
+static bool
+_notmuch_is_mixed_up_mangled (GMimeObject *part)
+{
+GMimeMultipart *mpart = NULL;
+GMimeObject *parts[3] = {NULL, NULL, NULL};
+GMimeContentType *type = NULL;
+char *prelude_string = NULL;
+bool prelude_is_empty;
+
+if (part == NULL)
+ return false;
+type = g_mime_object_get_content_type (part);
+if (type == NULL)
+ return false;
+if (! g_mime_content_type_is_type (type, "multipart", "mixed"))
+ return false;
+if (! GMIME_IS_MULTIPART (part)) /* probably impossible */
+ return false;
+mpart = GMIME_MULTIPART (part);
+if (mpart == NULL)
+ return false;
+if (g_mime_multipart_get_count (mpart) != 3)
+ return false;
+parts[0] = g_mime_multipart_get_part (mpart, 0);
+if (! g_mime_content_type_is_type (g_mime_object_get_content_type
(parts[0]),
+ "text", "plain"))
+ return false;
+if (! GMIME_IS_TEXT_PART (parts[0]))
+ return false;
+parts[1] = g_mime_multipart_get_part (mpart, 1);
+if (! g_mime_content_type_is_type (g_mime_object_get_content_type
(parts[1]),
+ "application", "pgp-encrypted"))
+ return false;
+parts[2] = g_mime_multipart_get_part (mpart, 2);
+if (! g_mime_content_type_is_type (g_mime_object_get_content_type
(parts[2]),
+ "application", "octet-stream"))
+ return false;
+
+/* Is parts[0] length 0? */
+prelude_string = g_mime_text_part_get_text (GMIME_TEXT_PART (parts[0]));
+prelude_is_empty = (prelude_string[0] == '\0');
+g_free (prelude_string);
+if (! prelude_is_empty)
+ return false;
+
+/* FIXME: after decoding and stripping whitespace, is parts[1]
+ * subpart just "Version: 1" ? */
+
+/* FIXME: can we determine that parts[2] subpart is *only* PGP
+ * encrypted data? I tried g_mime_part_get_openpgp_data () but
+ * found https://github.com/jstedfast/gmime/issues/60 */
+
+return true;
+}
+
+
+/* see
+ *
https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling-00#section-4.1.2
*/
+GMimeObject *
+_notmuch_repair_mixed_up_mangled (GMimeObject *part)
+{
+GMimeMultipart *mpart = NULL, *mpart_ret = NULL;
+GMimeObject *ret = NULL;
+
+if (! _notmuch_is_mixed_up_mangled (part))
+ return NULL;
+mpart = GMIME_MULTIPART (part);
+ret = GMIME_OBJECT (g_mime_multipart_encrypted_new ());
+if (ret == NULL)
+ return NULL;
+mpart_ret = GMIME_MULTIPART (ret);
+if (mpart_ret == NULL) {
+ g_object_unref (ret);
+ return NULL;
+}
+g_mime_object_set_content_type_parameter (ret, "protocol",
"application/pgp-encrypted");
+
+g_mime_multipart_insert (mpart_ret, 0, g_mime_multipart_get_part (mpart,
1));
+g_mime_multipart_insert (mpart_ret, 1, g_mime_multipart_get_part (mpart,
2));
+return ret;
+}
diff --git a/util/repair.h b/util/repair.h
index 9974d693..492f5a20 100644
--- a/util/repair.h
+++ b/util/repair.h
@@ -25,9 +25,19 @@ extern "C" {
* returned object will only be released when the original part is
* disposed of.
*/
+
GMimeObject *
_notmuch_repair_crypto_payload_skip_legacy_display (GMimeObject *payload);
+/* Detecting and repairing "Mixed-Up MIME mangling". see
+ *
https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling-00#section-4.1
+ * If this returns NULL, the message was probably not "Mixed up". If
+ * it returns non-NULL, then there is a newly-allocated MIME part that
+ * represents the repaired version. The caller is responsible for
+ * ensuring that any returned object is freed with g_object_unref. */
+GMimeObject *
Re: [PATCH v4 2/4] util/repair: identify and repair "Mixed Up" mangled messages
Thanks for the review, David.
On Fri 2019-09-13 22:58:27 -0300, David Bremner wrote:
> Daniel Kahn Gillmor writes:
>
>> +/* see
>> + *
>> https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling-00#section-4.1.1
>> */
>> +static bool
>> +_notmuch_is_mixed_up_mangled (GMimeObject *part)
>> +{
>> +GMimeMultipart *mpart = NULL;
>> +GMimeObject *first, *second, *third = NULL;
>> +char *prelude_string = NULL;
>> +bool prelude_is_empty;
>> +
>> +if (! g_mime_content_type_is_type (g_mime_object_get_content_type
>> (part),
>> + "multipart", "mixed"))
>> +return false;
>
> Can g_mime_object_get_content_type plausibly fail (and return NULL) here?
hm, yes, it can -- if it's passed NULL, for example. It's not an
external function, but I can program this more defensively. i'll do
that in the next revision.
>> +if (! GMIME_IS_MULTIPART (part))
>> +return false;
>
> I guess this happens if the mime structure does not match the content
> type declaration? Not sure if this needs a comment or if it's clear
> enough.
yeah, this is unlikely, maybe even impossible given the current gmime
implementation, but it's a defensive step.
>> +mpart = GMIME_MULTIPART (part);
>> +if (mpart == NULL)
>> +return false;
>> +if (g_mime_multipart_get_count (mpart) != 3)
>> +return false;
>> +first = g_mime_multipart_get_part (mpart, 0);
>
> there's a slight cognitive dissonance for me between the zero and one
> based indexing schemes here. part0, part1, and part2? or maybe an
> GMimeObject *part[3]
sure, i'll make it *parts[3] to avoid the dissonance. ("part" is already
taken -- it's the argument for the function.
>> +if (! g_mime_content_type_is_type (g_mime_object_get_content_type
>> (first),
>> + "text", "plain"))
>> +return false;
>> +if (! GMIME_IS_TEXT_PART (first))
>> +return false;
>> +second = g_mime_multipart_get_part (mpart, 1);
>> +if (! g_mime_content_type_is_type (g_mime_object_get_content_type
>> (second),
>> + "application", "pgp-encrypted"))
>> +return false;
>> +third = g_mime_multipart_get_part (mpart, 2);
>> +if (! g_mime_content_type_is_type (g_mime_object_get_content_type
>> (third),
>> + "application", "octet-stream"))
>> +return false;
>> +
>> +/* Is first subpart length 0? */
>> +prelude_string = g_mime_text_part_get_text (GMIME_TEXT_PART (first));
>> +prelude_is_empty = ! (strcmp ("", prelude_string));
>> +g_free (prelude_string);
>
> It might make sense to use the EMPTY_STRING macro here, although
> currently it's only accesible via notmuch-private.h
hm, notmuch-private.h is in lib/, which seems inappropriate for use
outside the library. however, i agree that this seems like a
superfluous call to strcmp -- i'll change it for a simple test.
I've updated this 2/4 patch on my mixed-up-mangling branch (visible at
https://gitlab.com/dkg/notmuch), and i'll send it to this thread
shortly.
--dkg
signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v5 2/4] util/repair: identify and repair "Mixed Up" mangled messages
On Sun 2019-09-15 23:26:59 +0300, Tomi Ollila wrote: > On Sun, Sep 15 2019, Daniel Kahn Gillmor wrote: > >> This patch implements a functional identification and repair process > > If there is going to be more versions, then the above could be changed > to either > > 1) This commit implements... > > or just > > 2) Implement a functional ... > >> for "Mixed Up" MIME messages as described in >> https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling-00#section-4.1 >> >> The detection test is not entirely complete, in that it does not >> verify the contents of the latter two message subparts, but this is >> probably safe to skip, because those two parts are unlikely to be >> readable anyway, and the only part we are effectively omitting (the >> first subpart) is guaranteed to be empty anyway, so its removal can be >> reversed if you want to do so. I've left FIXMEs in the code so that >> anyone excited about adding these additional checks can see where to >> put them in. >> >> I'll use this functionality in the next two patches. > > If there is going to be more versions, then the above line could be removed... > > (as committed changes are not "patches" ...) I concur with (and appreciate) your terminological pedantry. the revision on my gitlab branch has adopted your suggestions, but i don't want to burden the list with another revision right now. hopefully this change to the commit message won't block adoption -- anyone merging the series should feel free to adjust the commit message as Tomi recommends. --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] Drop devel/printmimestructure (it is in mailscripts 0.11)
Tomi Ollila writes: > On Sun, Sep 15 2019, Daniel Kahn Gillmor wrote: > >> mailscripts 0.11 now ships a derivative of devel/printmimestructure >> called email-print-mime-structure. Maintenance for that utility will >> happen in mailscripts from now on, so we should not track an >> independent copy of it in notmuch's source tree. > > So only for debian users >;) > > (I downloaded the package from packages.debian.org couple of hours ago, > manually, as package damager in this system did not find it.) Typo of the day, even if intentional. > I'm fine dropping this, just that helping users to download the tool > might be a bit harder now ? maybe we can link to https://git.spwhitton.name/mailscripts e.g. in some wiki page about troubleshooting? d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
