Re: [PATCH 13/18] cli/new, insert, reindex: change index.try_decrypt to "auto" by default

[Daniel Kahn Gillmor]

On Thu 2017-11-16 08:40:41 -0400, David Bremner wrote:
> Daniel Kahn Gillmor  writes:
>
>> The new "auto" decryption policy is not only good for "notmuch show"
>> and "notmuch reindex".  It's also useful for indexing messages --
>> there's no good reason to not try to go ahead and index the cleartext
>> of a message that we have a stashed session key for.
>
> I'm confused here. You talk about indexing other than reindex, but the
> only tests that change are reindex? Is this meant to change "notmuch
> new" behaviour?

the "auto" policy won't change the behavior of notmuch upon seeing a new
message (new, insert) from "false" -- all it does differently from
"false" is try to use session keys when they are available (and there's
no way for them to be available on a never-before-seen message).

   --dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] cli/help: give a hint about notmuch-emacs-mua

[Daniel Kahn Gillmor]

On Thu 2017-10-26 18:27:51 -0400, Daniel Kahn Gillmor wrote:
> "notmuch help" doesn't mention "notmuch-emacs-mua" even though we
> support it through the try_external_command() mechanism.
>
> In addition, "notmuch help emacs-mua" doesn't work, even though we
> ship the appropriate manpage.
>
> This changeset fixes both of these problems.

Ping!  i don't think this patch is complex or objectionable.  Can it be
merged?

--dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] cli/help, completion: added pointers to notmuch-properties(7)

[Daniel Kahn Gillmor]

On Thu 2017-10-26 18:28:12 -0400, Daniel Kahn Gillmor wrote:
> ---
>  completion/notmuch-completion.bash | 2 +-
>  notmuch.c  | 2 ++
>  2 files changed, 3 insertions(+), 1 deletion(-)

Ping on this patch as well.  I think this should be safe and simple to
merge.

--dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] crypto: signature verification reports valid User IDs

[Daniel Kahn Gillmor]

When i'm trying to understand a message signature, i care that i know
who it came from (the "validity" of the identity associated with the
key), *not* whether i'm willing to accept the keyholder's other
identity assertions (the "trust" associated with the certificate).

We've been reporting User ID information based on the "trust"
associated with the certificate, because GMime didn't clearly expose
the validity of the User IDs.

This change relies on fixes made in GMime 3.0.3 and later which
include https://github.com/jstedfast/gmime/pull/18.
---
 configure  |  3 ++-
 notmuch-show.c | 36 +---
 test/T355-smime.sh |  8 +++-
 util/gmime-extra.h | 11 ---
 4 files changed, 38 insertions(+), 20 deletions(-)

diff --git a/configure b/configure
index d3e30b53..cfbf827c 100755
--- a/configure
+++ b/configure
@@ -478,9 +478,10 @@ fi
 # we need to have a version >= 2.6.5 to avoid a crypto bug. We need
 # 2.6.7 for permissive "From " header handling.
 GMIME_MINVER=2.6.7
+GMIME3_MINVER=3.0.3
 
 printf "Checking for GMime development files... "
-if pkg-config --exists "gmime-3.0"; then
+if pkg-config --exists "gmime-3.0 > $GMIME3_MINVER"; then
 printf "Yes (3.0).\n"
 have_gmime=1
 gmime_cflags=$(pkg-config --cflags gmime-3.0)
diff --git a/notmuch-show.c b/notmuch-show.c
index 7afd3947..4b89be0a 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -401,6 +401,32 @@ format_signature_errors (sprinter_t *sp, GMimeSignature 
*signature)
 }
 #endif
 
+
+static const char*
+_get_certificate_valid_userid (GMimeCertificate *cert)
+{
+/* output user id only if validity is FULL or ULTIMATE. */
+#if (GMIME_MAJOR_VERSION < 3)
+/* note that gmime 2.6 is using the term "trust" here, which
+ * is WRONG.  It's actually user id "validity". */
+const char *name = g_mime_certificate_get_name (cert);
+if (name == NULL)
+   return name;
+GMimeCertificateTrust trust = g_mime_certificate_get_trust (cert);
+if (trust == GMIME_CERTIFICATE_TRUST_FULLY || trust == 
GMIME_CERTIFICATE_TRUST_ULTIMATE)
+   return name;
+return NULL;
+#else
+const char *uid = g_mime_certificate_get_user_id (cert);
+if (uid == NULL)
+   return uid;
+GMimeValidity validity = g_mime_certificate_get_id_validity (cert);
+if (validity == GMIME_VALIDITY_FULL || validity == GMIME_VALIDITY_ULTIMATE)
+   return uid;
+return NULL;
+#endif
+}
+
 /* Signature status sprinter (GMime 2.6) */
 static void
 format_part_sigstatus_sprinter (sprinter_t *sp, mime_node_t *node)
@@ -446,15 +472,11 @@ format_part_sigstatus_sprinter (sprinter_t *sp, 
mime_node_t *node)
sp->map_key (sp, "expires");
sp->integer (sp, expires);
}
-   /* output user id only if validity is FULL or ULTIMATE. */
-   /* note that gmime is using the term "trust" here, which
-* is WRONG.  It's actually user id "validity". */
if (certificate) {
-   const char *name = g_mime_certificate_get_uid (certificate);
-   GMimeCertificateTrust trust = g_mime_certificate_get_trust 
(certificate);
-   if (name && (trust == GMIME_CERTIFICATE_TRUST_FULLY || trust == 
GMIME_CERTIFICATE_TRUST_ULTIMATE)) {
+   const char *uid = _get_certificate_valid_userid (certificate);
+   if (uid) {
sp->map_key (sp, "userid");
-   sp->string (sp, name);
+   sp->string (sp, uid);
}
}
} else if (certificate) {
diff --git a/test/T355-smime.sh b/test/T355-smime.sh
index 1523f17b..be45e3b1 100755
--- a/test/T355-smime.sh
+++ b/test/T355-smime.sh
@@ -48,6 +48,12 @@ EOF
 test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "signature verification (notmuch CLI)"
+if [ "${NOTMUCH_GMIME_MAJOR}" -lt 3 ]; then
+# gmime 2 can't report User IDs properly for S/MIME
+USERID=''
+else
+USERID='"userid": "CN=Notmuch Test Suite",'
+fi
 output=$(notmuch show --format=json --verify subject:"test signed message 001" 
\
 | notmuch_json_show_sanitize \
 | sed -e 's|"created": [-1234567890]*|"created": 946728000|' \
@@ -65,7 +71,7 @@ expected='[[[{"id": "X",
  "Date": "Sat, 01 Jan 2000 12:00:00 +"},
  "body": [{"id": 1,
  "sigstatus": [{"fingerprint": "'$FINGERPRINT'",
- "status": "good",
+ "status": "good",'$USERID'
  "expires": 424242424,
  "created": 946728000}],
  "content-type": "multipart/signed",
diff --git a/util/gmime-extra.h b/util/gmime-extra.h
index 40bf1454..ec4ca186 100644
--- a/util/gmime-extra.h
+++ b/util/gmime-extra.h
@@ -16,11 +16,9 @@ GMimeStream *g_mime_stream_stdout_new(void);
 #define g_mime_2_6_unref(obj) g_object_unref (obj)
 #define g_mime_3_unused(arg) arg
 #define g_mime_certificate_get_fpr16(cert) g_mime_certificate_get_key_id (cert)
-#define g_mime_certificate_get_uid(cert) g_mime_certificate_get_name (cert);
 #else /* GMime >= 3.0 */
 
 #define 

Re: [PATCH] python: add bindings for notmuch_message_get_propert(y/ies)

[Floris Bruynooghe]

Daniel Kahn Gillmor  writes:

> On Tue 2017-11-28 23:46:11 +0100, Ruben Pollan wrote:
>> Message.get_property (prop) returns a string with the value of the property 
>> and
>> Message.get_properties (prop, exact=False) returns a list [(key, value)]
>
> This looks like a sensible approach to me.  I'd be curious to hear what
> others think of this.
>
> In considering the API design space here, it occurs to me that it might
> be more pythonic for get_properties to return a dict like:

I would probably model properties as a dictionary in notdb, making this
a collections.abc.MutableMapping implementation with a .get_all(prop)
method inspired from the stdlib email.message package.  This kind of
also implies making properties access a property rather then a method
call:

msg.properties['prop'] = 'foo'
msg.properties['prop'] = 'bar'
msg.properties['prop'] == 'foo'  # pot luck
msg.properties.get_all('prop') == {'foo', 'bar'}  # properties are unsorted are 
they?

This also has the binary question problem, is this returned as bytes or
as str?  Current Python bindings seem to go for .decode('utf-8',
errors='ignore') afaik which is somewhat lossy.


Cheers,
Floris
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: DRAFT Introduce CFFI-based Python bindings

[Floris Bruynooghe]

Patrick Totzke  writes:

> Quoting David Bremner (2017-11-28 23:59:26)
>> Floris Bruynooghe  writes:
>> 
>> >
>> > Lastly there are some downsides to the choices I made:
>> > - I ended up going squarely for CPython 3.6+.  Choosing Python
>> >   3 allowed better API design, e.g. with keyword-only parameters
>> >   etc.  Choosing CPython 3.4+ restricts the madness that can
>> >   happen with __del__ and gives some newer (tho now unused)
>> >   features in weakref.finalizer.
>> > - This is no longer drop-in compatible.
>> > - I haven't got to a stage where my initial goal of speed has
>> >   been proven yet.
>> 
>> I guess you'll have to convince the maintainers / users of alot and afew
>> that this makes sense before we go much further. I'd point out that
>> Debian stable is only at python 3.5, so that makes me a bit wary of this
>> (being able to run the test suite on debian stable and similar aged
>> distros useful for me, and I suspect other developers).
>> 
>> I know there are issues with memory management in the current bindings,
>> so that may be a strong reason to push to python 3.6; it seems to need
>> more investigation at the moment.
>> 
>> d
>
>
> I am generally in favour of modernizing the notmuch python bindings,
> especially when it comes to memory management and exception handling.
>
> At the moment, the alot interface officially only supports python v2.7
> but our dependencies have now mostly been updated and we are working on
> port to python 3, see here: https://github.com/pazz/alot/pull/1055
>
> @Floris, you are welcome to join #alot on freenode if you want to
> discuss details on that.
>
> You mention that your new API breaks compatibility with the existing
> ones. Do you have some demo code that uses the new API for reference?

Short, untested, example which works with what's posted:

db = notdb.Database.create()
# or
db = notdb.Database(path=None, mode=notdb.Database.MODE.READ_WRITE)
print(db.path) -> pathlib.Path (a py34 dependency)
if 'unread' in db.tags:  # tags behaves like a set
print('unread mail!')
with db.atomic():
msg = db.add('/path/to/file')
mdg = db.get('/path/to/file')
msg = db.find('some-msgid')
db.remove('path/to/other/file')
# sorry, don't have a query interface yet
assert 'unread' in msg.tags
for tag in msg.tags:
print(f'a tag: {tag}')
with msg.frozen():  # Message.frozen() not yet implemented
msg.tags.clear()  # all set operations supported
msg.tags.add('atag')
msg.tags_to_flags()

I imagine the query interface would be something like:

with db.query('tag:atag') as query:
print(f'results: {query.count}')
for msg in query:
print(msg.path)

But to be honest I've been spending most time on getting the
memory-safety figured (which I hope I finally did) so far and I think
the tag handling is so far the nicest thing to show off.  They're
completely normal Python sets with no special behaviour at all (well,
that's not true - there's the binary interface, see the code posted for
this).

Actually, this last point is kind of important and I failed to mention
it before too.  The existing Python bindings convert many bytes from
libnotmuch to Python strings, that is unicode on Python 3.  For many it
uses b'bytes'.decode('utf-8', errors='ignore') which is a sane default
if you want to display things.  But if you need to round-trip a tag and
store it again you might be changing the tag.  I've not found the right
way to handle binary data (e.g. also needed for messageid) everywhere
yet but for tags I've gone with:

for tag in iter(msg.tags):  # iter() normally called implicitly by for loop
print(f'All unicode this: {tag}')
for tag in msg.tags.iter(encoding=None):
other_msg.tags.add(tag)  # This passes pure bytes around, loses nothing
# What I've used for message ID for now is a "BinString" type
print(f'All just unicode: {msg.messageid}')
binary_msgid = bytes(msg.messageid)  # No lossy conversion

This BinString stuff is somewhat hacky, not sure how sane that is.  The
second iterator on tags feels somewhat cleaner.  Likewise tags could be
BinString as well instead of plain str.

Cheers,
Floris
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: DRAFT Introduce CFFI-based Python bindings

[Floris Bruynooghe]

David Bremner  writes:

> Floris Bruynooghe  writes:
>
>>
>> Lastly there are some downsides to the choices I made:
>> - I ended up going squarely for CPython 3.6+.  Choosing Python
>>   3 allowed better API design, e.g. with keyword-only parameters
>>   etc.  Choosing CPython 3.4+ restricts the madness that can
>>   happen with __del__ and gives some newer (tho now unused)
>>   features in weakref.finalizer.
>> - This is no longer drop-in compatible.
>> - I haven't got to a stage where my initial goal of speed has
>>   been proven yet.
>
> I guess you'll have to convince the maintainers / users of alot and afew
> that this makes sense before we go much further. I'd point out that
> Debian stable is only at python 3.5, so that makes me a bit wary of this
> (being able to run the test suite on debian stable and similar aged
> distros useful for me, and I suspect other developers).
>
> I know there are issues with memory management in the current bindings,
> so that may be a strong reason to push to python 3.6; it seems to need
> more investigation at the moment.

So on earlier Python versions, sure this is possible at not too much
cost.

- Python 3.4+ would just cost the use of some f-strings.  Not major, was
  just nice to use.
- Python <3.4 afaik would only need a tweak to the Database.tags and
  Message.tags properties.  I *think* swapping the caching of these
  using a weakref should suffice and not break the brittle
  Python-libnotmuch memory management.
  Mind you I think Python 3.0-3.3 are pretty old and not much point in
  supporting them.  But this would also apply for 2.7 support.
- Python 2.7 is probably the worst, in that keyword-only arguments would
  be gone.  If python 2.7 is required I'd be much keener to have another
  go at a drop-in replacement with the memory safety features and then
  build the "notdb" API on top off it.  But for that to be worth it
  people need to be convinced enough that maintaining a CFFI version is
  nicer than a ctypes version I guess.

Kind regards,
Floris
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: DRAFT Introduce CFFI-based Python bindings

[Patrick Totzke]

Quoting David Bremner (2017-11-28 23:59:26)
> Floris Bruynooghe  writes:
> 
> >
> > Lastly there are some downsides to the choices I made:
> > - I ended up going squarely for CPython 3.6+.  Choosing Python
> >   3 allowed better API design, e.g. with keyword-only parameters
> >   etc.  Choosing CPython 3.4+ restricts the madness that can
> >   happen with __del__ and gives some newer (tho now unused)
> >   features in weakref.finalizer.
> > - This is no longer drop-in compatible.
> > - I haven't got to a stage where my initial goal of speed has
> >   been proven yet.
> 
> I guess you'll have to convince the maintainers / users of alot and afew
> that this makes sense before we go much further. I'd point out that
> Debian stable is only at python 3.5, so that makes me a bit wary of this
> (being able to run the test suite on debian stable and similar aged
> distros useful for me, and I suspect other developers).
> 
> I know there are issues with memory management in the current bindings,
> so that may be a strong reason to push to python 3.6; it seems to need
> more investigation at the moment.
> 
> d


I am generally in favour of modernizing the notmuch python bindings,
especially when it comes to memory management and exception handling.

At the moment, the alot interface officially only supports python v2.7
but our dependencies have now mostly been updated and we are working on
port to python 3, see here: https://github.com/pazz/alot/pull/1055

@Floris, you are welcome to join #alot on freenode if you want to
discuss details on that.

You mention that your new API breaks compatibility with the existing
ones. Do you have some demo code that uses the new API for reference?

Cheers,
P


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] python: add bindings for notmuch_message_get_propert(y/ies)

[Ruben Pollan]

Message.get_property (prop) returns a string with the value of the property and
Message.get_properties (prop, exact=False) returns a list [(key, value)]
---
 bindings/python/notmuch/globals.py |  5 +++
 bindings/python/notmuch/message.py | 81 +-
 2 files changed, 85 insertions(+), 1 deletion(-)

diff --git a/bindings/python/notmuch/globals.py 
b/bindings/python/notmuch/globals.py
index 71426c84..801062dc 100644
--- a/bindings/python/notmuch/globals.py
+++ b/bindings/python/notmuch/globals.py
@@ -75,6 +75,11 @@ class NotmuchMessageS(Structure):
 NotmuchMessageP = POINTER(NotmuchMessageS)
 
 
+class NotmuchMessagePropertiesS(Structure):
+pass
+NotmuchMessagePropertiesP = POINTER(NotmuchMessagePropertiesS)
+
+
 class NotmuchTagsS(Structure):
 pass
 NotmuchTagsP = POINTER(NotmuchTagsS)
diff --git a/bindings/python/notmuch/message.py 
b/bindings/python/notmuch/message.py
index d5b98e4f..2025f979 100644
--- a/bindings/python/notmuch/message.py
+++ b/bindings/python/notmuch/message.py
@@ -19,7 +19,7 @@ Copyright 2010 Sebastian Spaeth 
 """
 
 
-from ctypes import c_char_p, c_long, c_uint, c_int
+from ctypes import c_char_p, c_long, c_uint, c_int, POINTER, byref
 from datetime import date
 from .globals import (
 nmlib,
@@ -29,6 +29,7 @@ from .globals import (
 NotmuchTagsP,
 NotmuchMessageP,
 NotmuchMessagesP,
+NotmuchMessagePropertiesP,
 NotmuchFilenamesP,
 )
 from .errors import (
@@ -113,6 +114,36 @@ class Message(Python3StringMixIn):
 _maildir_flags_to_tags.argtypes = [NotmuchMessageP]
 _maildir_flags_to_tags.restype = c_int
 
+"""notmuch_message_get_property"""
+_get_property = nmlib.notmuch_message_get_property
+_get_property.argtypes = [NotmuchMessageP, c_char_p, POINTER(c_char_p)]
+_get_property.restype = c_int
+
+"""notmuch_message_get_properties"""
+_get_properties = nmlib.notmuch_message_get_properties
+_get_properties.argtypes = [NotmuchMessageP, c_char_p, c_int]
+_get_properties.restype = NotmuchMessagePropertiesP
+
+"""notmuch_message_properties_valid"""
+_properties_valid = nmlib.notmuch_message_properties_valid
+_properties_valid.argtypes = [NotmuchMessagePropertiesP]
+_properties_valid.restype = bool
+
+"""notmuch_message_properties_value"""
+_properties_value = nmlib.notmuch_message_properties_value
+_properties_value.argtypes = [NotmuchMessagePropertiesP]
+_properties_value.restype = c_char_p
+
+"""notmuch_message_properties_key"""
+_properties_key = nmlib.notmuch_message_properties_key
+_properties_key.argtypes = [NotmuchMessagePropertiesP]
+_properties_key.restype = c_char_p
+
+"""notmuch_message_properties_move_to_next"""
+_properties_move_to_next = nmlib.notmuch_message_properties_move_to_next
+_properties_move_to_next.argtypes = [NotmuchMessagePropertiesP]
+_properties_move_to_next.restype = None
+
 #Constants: Flags that can be set/get with set_flag
 FLAG = Enum(['MATCH'])
 
@@ -433,6 +464,54 @@ class Message(Python3StringMixIn):
 _freeze.argtypes = [NotmuchMessageP]
 _freeze.restype = c_uint
 
+def get_property(self, prop):
+""" Retrieve the value for a single property key
+
+:param prop: The name of the property to get.
+:returns: String with the property value or None if there is no such
+  key. In the case of multiple values for the given key, the
+  first one is retrieved.
+:raises: :exc:`NotInitializedError` if message has not been
+ initialized
+"""
+if not self._msg:
+raise NotInitializedError()
+
+value = c_char_p("")
+status = Message._get_property(self._msg, prop, byref(value))
+if status != 0:
+raise NotmuchError(status)
+
+return value.value
+
+def get_properties(self, prop="", exact=False):
+""" Get the properties for *message*, returning
+notmuch_message_properties_t object which can be used to iterate
+over all properties.
+
+:param prop: The name of the property to get. Otherwise it will return
+ the full list of properties of the message.
+:param exact: if True, require exact match with key. Otherwise
+  treat as prefix.
+:returns: A dictionary with the property names and values {key: value}
+:raises: :exc:`NotInitializedError` if message has not been
+ initialized
+"""
+if not self._msg:
+raise NotInitializedError()
+
+properties_dict = {}
+properties = Message._get_properties(self._msg, prop, exact)
+while Message._properties_valid(properties):
+key = Message._properties_key(properties)
+value = Message._properties_value(properties)
+if key not in properties_dict:
+properties_dict[key] = []
+  

Re: [PATCH] python: add bindings for notmuch_message_get_propert(y/ies)

[meskio]

Quoting Daniel Kahn Gillmor (2017-11-29 02:57:24)
> On Tue 2017-11-28 23:46:11 +0100, Ruben Pollan wrote:
> > Message.get_property (prop) returns a string with the value of the property 
> > and
> > Message.get_properties (prop, exact=False) returns a list [(key, value)]
> 
> This looks like a sensible approach to me.  I'd be curious to hear what
> others think of this.
> 
> In considering the API design space here, it occurs to me that it might
> be more pythonic for get_properties to return a dict like:
> 
>{ key: [ value, … ], key: [ value, … ] }
> 
> Any reason you chose one over the other?  My python-fu is shallow, so
> please don't take my aesthetic guesswork as authoritative; but i'm
> imagining a user wanting to grab a bunch of properties and then easily
> access them by key, and the dict seems like the simple way to do that.

Yes, the dict is more pythonic. I thought about it, I went for the tuples it 
was 
simpler to implement (and use in my use case). But giving a second thought it 
makes more sense to do a dict.

> Also, does get_properties() work with prop=None to fetch all properties?
> if so, maybe that should be the default?

I didn't thought about that, but you are right, with prop="" you get the full 
list of properties of the message. Nice, let's put it as default value.

-- 
meskio | http://meskio.net/
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
 My contact info: http://meskio.net/crypto.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Nos vamos a Croatan.


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch