Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 9 Jun 2017, at 20:54, Donald Stufft  wrote:
> 
> 
> All that being said, if someone *does* want pip to use WinHTTP, requests 
> provides a mechanism where you can plug in your own network handling code, so 
> someone could write a requests-winhttp adapter that did that, and we could 
> possibly even expose the ability to ask pip to use that. However that’s 
> entirely in the domain of pip/requests feature requests and not really 
> related to python-dev/this PEP itself.

As an FYI I played with this on Mac far enough to prove that it’d work: 
https://github.com/Lukasa/requests-darwin 


It’s not anywhere near feature complete, but it basically works pretty ok. Of 
course, ideally Requests would just get entirely out of the way at this point 
because it duplicates a lot of NSURLSession’s logic, but it’s an approach that 
can work.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 9 Jun 2017, at 20:41, Steve Dower  wrote:
> 
> These are totally fine for implementing a requests-like API that relies on 
> system configuration for HTTPS connections. They are not sufficient for 
> building a server, but they should be sufficient for downloading the packages 
> you need to build a server.
> 
> This is starting to feel to me like we're stuck on "building the thing right" 
> and have skipped over "building the right thing". But maybe it's just me...

For the purposes of this PEP I think we should exclude this. The only way this 
works on a decent number of platforms (hi there BSDs and Linuxes) is if the 
option on those platforms is libcurl. Linux does not ship a high level HTTP 
client library: libcurl is basically the option. That would require pip binding 
libcurl, NSURLSession, and the Windows API. It’s not clear to me that the 
solution to “we don’t want to backport some SSL code” is “write a bunch of 
bindings to other third-party libraries”.

Cory 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 8 Jun 2017, at 21:17, Donald Stufft  wrote:
> 
>> 
>> On Jun 8, 2017, at 3:57 PM, Steve Dower > > wrote:
>> 
>> Awesome, this is exactly what I needed to see.
>> 
>> What if Python 2.7 just exposed the OpenSSL primitives necessary so that 
>> ctypes could use them? Is that a viable approach here? Presumably then a 
>> MemoryBIO backport could be pure-Python.
>> 
>> It doesn't help the other *ythons, but unless they have MemoryBIO 
>> implementations ready to backport then I can't think of anything that will 
>> help them short of a completely new API.
>> 
> 
> 
> I would have to let Cory answer the question about feasibility here since 
> he’s much more familiar with OpenSSL’s API (and even binding something like 
> this with ctypes) than I am. The first thing that really stands out to me 
> though is it just feels a bit like shuffling deckchairs?

The short answer is that, while it’s do-able, we have some problems with ABI 
compatibility. OpenSSL 1.1 and 1.0 are ABI incompatible, so I have to write 
divergent ctypes code to handle each case. It may also be relevant to support 
OpenSSL 0.9.x so we roll into the same ABI compatibility concern all over 
again. Doubly annoyingly a bunch of OpenSSL code in 1.0 is actually macros 
which don’t work in ctypes so there’ll be a lot of futzing about in structures 
to get what I need to do done.

This also doesn’t get into the difficulty of some operating systems shipping a 
LibreSSL masquerading as an OpenSSL, which is subtly incompatible in ways I 
don’t fully understand at this time.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 7 Jun 2017, at 20:06, Jim Baker  wrote:
> 
> Sorry, I wish this were true, but PyOpenSSL is not available on Jython, 
> because we do not yet support CFFI for Jython. CFFI support is something we 
> have looked at, but we have not implemented. (There is a related and far more 
> ambitious project to implement support for the C Extension API, 
> http://jyni.org/ , which Stefan Richthofer is working on 
> with me under the Google Summer of Code.)

This is what I was worried about. Moving to require PyOpenSSL *also* locks us 
out of Jython support, at least for the time being. That’s another point in the 
“con” column for making PyOpenSSL a mandatory dependency.

> I should mention that we sometimes see undocumented functionality leak out. 
> For example, 
> https://docs.python.org/3/library/socket.html#socket.socket.listen 
>  doesn't 
> say anything about backlog=0, but the requests test suite (last time I looked 
> on Jython) now depends on it. We assumed it was something like 
> http://pubs.opengroup.org/onlinepubs/009695399/functions/listen.html 
> , but 
> as described in http://bugs.python.org/issue8498 
> , it now means that "a server application 
> in python that accepts exactly 1 connection", by passing to the underlying C. 
> More tests, more docs, please, even though of course that's a lot of dev 
> effort.

Ah, yes, we do. In our defense, this is the semantic of the listen syscall, and 
the socket module is generally a pretty thin wrapper around most of the socket 
syscalls. But in hindsight this is definitely the kind of thing that gets 
tricky for someone trying to reimplement the socket module in terms of a 
different abstraction. I don’t want to dive down this rabbit hole because if we 
do I’ll have to start talking about how the complexity of the socket API is one 
of the other implementation hurdles for PEP 543, but I think that’s a 
conversation for another time.

Cory___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 7 Jun 2017, at 14:29, Victor Stinner  wrote:
> 
> 2017-06-07 10:56 GMT+02:00 Nathaniel Smith :
>> Another testing challenge is that the stdlib ssl module has no way to
>> trigger a renegotiation, and therefore there's no way to write tests
>> to check that it properly handles a renegotiation, even though
>> renegotiation is by far the trickiest part of the protocol to get
>> right. (In particular, renegotiation is the only case where attempting
>> to read can give WantWrite and vice-versa.)
> 
> Renegociation was the source of a vulnerability in SSL/TLS protocols,
> so maybe it's a good thing that it's not implemented :-)
> https://www.rapid7.com/db/vulnerabilities/tls-sess-renegotiation
> 
> Renegociation was removed from the new TLS 1.3 protocol:
> https://tlswg.github.io/tls13-spec/
> "TLS 1.3 forbids renegotiation"

Renegotiation remains extremely widely deployed with TLS client certificates in 
enterprise environments, sadly.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 6 Jun 2017, at 18:49, Jim Baker  wrote:
> 
> With Nick's suggestion of _tls_bootstrap, this has certainly become more 
> complicated. I'm still thinking of the ramifications for future Jython 2.7 
> support, if Python dev goes down this path. It still seems easier to me to 
> avoid exposing the SSLObject/MemoryBIO model to 2.7 and instead concentrate 
> on just having native TLS support. Yes, this might require more work than a 
> simple backport, but everyone is resource constrained, not just the Requests 
> or Jython teams.

Oh. This hadn’t occurred to me until just now, but doesn’t PyOpenSSL (or 
anything built on CFFI) just basically not run on Jython? Or am I mistaken?

> My concrete suggestion is that any work on PEP 543 will benefit from 
> improving the testing currently found in test_ssl as part of its 
> implementation. For a variety of reasons, test functions like ssl_io_loop 
> (https://github.com/python/cpython/blob/master/Lib/test/test_ssl.py#L1691 
> ) 
> avoid asserting more than it can properly manage the framing of 
> wrapped/unwrapped data, so for example one doesn't want to state explicitly 
> when SSL_ERROR_WANT_READ would be called (too much white box at that point). 
> On the other hand, the lack of unit testing of, for example, SSLObject.read, 
> but instead only doing at the functional test level, means that there's 
> nothing explicit testing "will raise an SSLWantReadError if it needs more 
> data than the incoming BIO has available" (per the docs), or similar use of 
> signaling (SSLEOFError comes to mind).

Yeah, PEP 543 just basically assumes the stdlib’s testing of TLS doesn’t exist 
and that it’ll have to manufacture its own. Unfortunately, because it is 
attempting to abstract across many different implementations the tests will 
need to be fairly high-level: for example, there is no consistent way to 
discuss the actual size of the buffers in the buffer-based TLS implementation, 
and they are allowed to be unbounded buffers, so tests cannot validate that 
TLSWantReadError and TLSWantWriteError are ever actually raised: all they can 
do is run tests that will handle them in the even that they are raised and 
confirm the data is transferred appropriately.

Cory___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 5 Jun 2017, at 12:00, Nick Coghlan  wrote:
> 
> Would you be OK with the notion of a "just for pip bootstrapping"
> private backend in _ensurepip_ssl?
> 
> That is, the only officially supported async-backed requests
> configuration on Py2 would be with the PyOpenSSL dependency installed,
> but in collaboration with the pip devs we'd also plumb in the pieces
> to let a new async-backed requests work without any extension modules
> other than those in the standard library.
> 
> That way, the only thing truly gated on the backport would be *pip*
> updating its bundled version of requests to the async-backed version -
> for normal third party use, the change would be "you need PyOpenSSL",
> rather than "you need a newer version of Python".
> 
> We'd still effectively end up with two different code execution paths
> (one backed by PyOpenSSL, one backed by the new private _ensurepip_ssl
> extension module), but the level of divergence would be much lower
> (it's just a question of where MemoryBIO and SSLObject are coming
> from) and the support scope for the less frequently used path would be
> much narrower (i.e. if a problem report isn't related to pip
> bootstrapping, it can be closed as "won't fix”

It’s not clear to me what the structure of that looks like, or what work is 
required to achieve it.

Right now Twisted never uses MemoryBIO and SSLObject: it always uses PyOpenSSL. 
That seems like we’d need to add MemoryBIO and SSLObject support to Twisted 
which can be enabled in some way other than feature detection (that is, so it 
can be installed). Without PEP 543 this is pretty gross. With PEP 543 it sucks 
less, but it also gates this work behind PEP 543 being successfully implemented 
and landed.

I guess we are *open* to that approach? It’s not clear to me how beneficial 
that is, and it doesn’t gain any of the ecosystem benefits (no-one else on Py 2 
can ever use this chunk of tested, understood code), but it’s certainly an 
option. The indirection gives me pause though.

Cory___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 3 Jun 2017, at 07:25, Nick Coghlan  wrote:
> 
> As a result, as awkward and annoying as it would be, I'm wondering if
> the structure of the requests migration may need to be something like:
> 
> - switch unconditionally to an async backend on Py3
> - switch conditionally to an async backend on Py2 when PyOpenSSL is available
> - retain sufficient sync-only support on Py2 to allow pip to disable
> the PyOpenSSL dependency

Ultimately we don’t have the resources to do this.

The requests core development team is 4 people, 3 of whom are part time 
best-effort maintainers and one of whom is me, who has foolishly decided to 
also work on PEP 543 as well as take over the lead maintenance role on urllib3. 
urllib3 has another two additional best-effort maintainers.

We simply do not have the development resources to support two parallel 
branches of code. Even if we did, doing so is a recipe for adding bugs and 
inconsistencies between the two, leading to increased workload as we break our 
users and fail to keep the two branches in sync. It also makes it much harder 
for users to migrate from our synchronous backend to our async one, as those 
two no longer use identical underlying implementations and so will have subtle 
inconsistencies.

The TL;DR there is: no, we’d rather stay sync only than do that.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 2 Jun 2017, at 17:59, Donald Stufft  wrote:
> 
> I suspect (though I’d let him speak for himself) that Cory would rather 
> continue to be sync only than require pip to go back to not using requests.

We are not wedded to supporting pip, but I think the interaction between the 
two tools is positive. I think pip gains a lot from depending on us, and we get 
a lot of value from having pip as a dependency. So the cost of supporting pip 
would have to be pretty darn high for us to want to stop doing that, and so on 
this issue I think Donald is right.

Cory___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 2 Jun 2017, at 17:39, Nick Coghlan  wrote:
> 
> On 3 June 2017 at 02:22, Donald Stufft  wrote:
>> It’s not just bootstrapping that pip has a problem with for C extensions, it
>> also prevents upgrading PyOpenSSL on Windows because having pip import
>> PyOpenSSL locks the .dll, and we can’t delete it or overwrite it until the
>> pip process exits and no longer imports PyOpenSSL. This isn’t a problem on
>> Linux or macOS or the other *nix clients though. We patch requests as it is
>> today to prevent it from importing simplejson and cryptography for this
>> reason.
> 
> Would requests be loading PyOpenSSL on Windows, though? If the aim is
> to facilitate PEP 543, then I'd expect it to be using the SChannel
> backend in that case.

Only assuming the SChannel backend is available. This would also likely be a 
third-party Python library so you can just search and replace “PyOpenSSL” above 
with “SChannel” and hit the exact same problem.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 2 Jun 2017, at 10:42, Victor Stinner  wrote:
> 
> Writing new code seems more risky and error-prone than backporting
> already "battle-tested" MemoryBIO from master. I also expect that
> writing code to validate certificate will be longer than the "100
> lines of C code in (probably)" expected by Steve Dower.
> 
> rust-certitude counts around 700 lines of Rust and 80 lines of Python
> code. But maybe I misunderstood the purpose of certitude: Steve Dower
> asked to only validate a certificate, not load or export CA.

That’s all certitude does. The docs of certitude are from an older version of 
the project when I was considering just doing a live-export to PEM file, before 
I realised the security concerns of that approach.

We’d also require some other code that lives outside certitude. In particular, 
code needs to be written to handle the OpenSSL verify callback to save off the 
cert chain and to translate errors appropriately. There’s also a follow-on 
problem: the ssl module allows you to call SSLContext.load_default_certs and 
then SSLContext.load_verify_locations. If you do that, those two behave 
*additively*: both the default certs and custom verify locations are trusted. 
Certitude doesn’t allow you to do that: it says that if you choose to use the 
system certs then darn it that’s all you get. Working out how to do that 
without just importing random stuff into the user’s keychain would be…tricky. 
Do-able, for sure, but would require code I haven’t written for Certitude (I 
may have written it using ctypes elsewhere though).

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 20:59, Steve Dower  wrote:
> 
> On 01Jun2017 1010, Nathaniel Smith wrote:
>> I believe that for answering this question about the ssl module, it's really 
>> only Linux users that matter, since pip/requests/everyone else pushing for 
>> this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS 
>> (IIUC) is to stop using the ssl module entirely in favor of new ctypes 
>> bindings for their respective native TLS libraries.
>> (And yes, in principle it might be possible to write new ctypes-based 
>> bindings for openssl, but (a) this whole project is already teetering on the 
>> verge of being impossible given the resources available, so adding any major 
>> extra deliverable is likely to sink the whole thing, and (b) compared to the 
>> proprietary libraries, openssl is *much* harder and riskier to wrap at the 
>> ctypes level because it has different/incompatible ABIs depending on its 
>> micro version and the vendor who distributed it. This is why manylinux 
>> packages that need openssl have to ship their own, but pip can't and 
>> shouldn't ship its own openssl for many hopefully obvious reasons.)
> 
> How much of a stop-gap would it be (for Windows at least) to override 
> OpenSSL's certificate validation with a call into the OS? This leaves most of 
> the work with OpenSSL, but lets the OS say yes/no to the certificates based 
> on its own configuration.
> 
> For Windows, this is under 100 lines of C code in (probably) _ssl, and while 
> I think an SChannel based approach is the better way to go long-term,[1] 
> offering platform-specific certificate validation as the default in 2.7 is 
> far more palatable than backporting new public API.

It’s entirely do-able. This is where I reveal just how long I’ve been fretting 
over this problem: https://pypi.python.org/pypi/certitude 
. Ignore the description, it’s wildly 
out-of-date: let me summarise the library instead.

Certitude is a Python library that uses CFFI and Rust to call into the system 
certificate validation libraries on macOS and Windows using a single unified 
API. Under the covers it has a whole bunch of Rust code that translates from 
what OpenSSL can give you (a list of certificates in the peer cert chain in DER 
format) and into what those two operating systems expect. The Rust code for 
Windows is here[1] and is about as horrifying a chunk of Rust as you can 
imagine seeing (the Windows API does not translate very neatly into Rust so the 
word “unsafe” appears a lot), but it does appear to work, at least in the 
mainline cases and in the few tests I have. The macOS code is here[2] and is 
moderately less horrifying, containing no instances of the word “unsafe”.

I lifted this approach from Chrome, because at the moment this is what they do: 
they use their custom fork of OpenSSL (BoringSSL) to do the actual TLS protocol 
manipulation, but hand the cert chain verification off to platform-native 
libraries on Windows and macOS.

I have never productised this library because ultimately I never had the time 
to spend writing a sufficiently broad test-case to confirm to me that it worked 
in all cases. There are very real risks in calling these APIs directly because 
if you get it wrong it’s easy to fail open.

It should be noted that right now certitude only works with, surprise, 
PyOpenSSL. Partly this is because the standard library does not expose 
SSL_get_peer_cert_chain, but even if it did that wouldn’t be enough as OpenSSL 
with VERIFY_NONE does not actually *save* the peer cert chain anywhere. That 
means even with PyOpenSSL the only way to get the peer cert chain is to hook 
into the verify callback and save off the certs as they come in, a gloriously 
absurd solution that is impossible with pure-Python code from the ssl module.

While this approach could work with _ssl.c, it ultimately doesn’t resolve the 
issue. It involves writing a substantial amount of new code which needs to be 
maintained by the ssl module maintainers. All of this code needs to be tested 
*thoroughly*, because python-dev would be accepting responsibility for the 
incredibly damaging potential CVEs in that code. And it doesn’t get python-dev 
out of the business of shipping OpenSSL on macOS and Windows, meaning that 
python-dev continues to bear the burden of OpenSSL CVEs, as well as the brand 
new CVEs that it is at risk of introducing.

Oh, and it can’t be backported to Python 2.7 or any of the bugfix-only Python 3 
releases, and as I just noted the ssl module has never made it possible to use 
this approach from outside CPython. So it’s strictly just as bad as the 
situation PEP 543 is in, but with more C code. Doesn’t sound like a winning 
description to me. ;)

Cory

[1]: 
https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/windows.rs
 

[2]: 

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 17:14, Chris Angelico  wrote:
> 
> 
> Do you have figures for how many people use pip on Windows vs Linux vs Mac OS?


I have figures for the download numbers, which are an awkward proxy because 
most people don’t CI on Windows and macOS, but they’re the best we have. Linux 
has approximately 20x the download numbers of either Windows or macOS, and both 
Windows and macOS are pretty close together. These numbers are a bit confounded 
due to the fact that 1/4 of Linux’s downloads are made up of systems that don’t 
report their platform, so the actual ratio could be anywhere from about 25:1 to 
3:1 in favour of Linux for either Windows or macOS. All of this is based on the 
downloads made in the last month.

Again, an enormous number of these downloads are going to be CI downloads which 
overwhelmingly favour Linux systems.

For some extra perspective, the next highest platform by download count is 
FreeBSD, with 0.04% of the downloads of Linux.

HTH,

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 15:10, David Wilson  wrote:

> Finding someone to audit the signature checking capabilities of [0] will
> have vastly lower net cost than getting the world into a situation where
> pip no longer runs on the >1e6 EC2 instances that will be running Ubuntu
> 14.04/16.04 LTS until the turn of the next decade.

So for the record I’m assuming most of the previous email was a joke: certainly 
it’s not going to happen. ;)

But this is a real concern that does need to be addressed: Requests can’t 
meaningfully use this as its only TLS backend until it propagates to the wider 
2.7 ecosystem, at least far enough such that pip can drop Python 2.7 releases 
lower than 2.7.14 (or wherever MemoryBIO ends up, if backported). So a concern 
emerges: if you grant my other premises about the utility of the backport, is 
it worth backporting at all?

The answer to that is honestly not clear to me. I chatted with the pip 
developers, and they have 90%+ of their users currently on Python 2, but more 
than half of those are on 2.7.9 or later. This shows some interest in upgrading 
to newer Python 2s. The question, I think, is: do we end up in a position where 
a good number of developers are on 2.7.14 or later and only a very small 
fraction on 2.7.13 or earlier before the absolute number of Python 2 devs drops 
low enough to just drop Python 2?

I don’t have an answer to that question. I have a gut instinct that says yes, 
probably, but a lack of certainty. My suspicion is that most of the core dev 
community believe the answer to that is “no”. But I’d say that from my 
perspective this is the crux of the problem. We can hedge against this by just 
choosing to backport and accepting that it may never become useful, but a 
reasonable person can disagree and say that it’s just not worth the effort.

Frankly, I think that amidst all the other arguments this is the one that most 
concretely needs answering, because if we don’t think Requests can ever 
meaningfully rely on the presence of MemoryBIO on 2.7 (where “rely on” can be 
approximated to 90%+ of 2.7 users having access to it AND 2.7 still having 
non-trivial usage numbers) then ultimately this PEP doesn’t grant me much 
benefit.

There are others who believe there are a few other benefits we could get from 
it (helping out Twisted etc.), but I don’t know that I’m well placed to make 
those arguments. (I also suspect I’d get accused of moving the goalposts.)

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 14:53, Antoine Pitrou <solip...@pitrou.net> wrote:
> 
> On Thu, 1 Jun 2017 14:37:55 +0100
> Cory Benfield <c...@lukasa.co.uk> wrote:
>>> 
>>> And indeed it doesn't.  Unless the target user base for pip is widely
>>> different than Python's, it shouldn't cause you any problems either.  
>> 
>> Maybe not now, but I think it’s fair to say that it did, right?
> 
> Until Python 3.2 and perhaps 3.3, yes. Since 3.4, definitely not.  For
> example asyncio quickly grew a sizable community around it, even though
> it had established Python 2-compatible competitors.

Sure, but “until 3.2” covers a long enough time to take us from now to 
“deprecation of Python 2”. Given that the Requests team is 4 people, unlike 
python-dev’s much larger number, I suspect we’d have at least as much pain 
proportionally as Python did. I’m not wild about signing up for that.

>>> Then the PEP is really wrong or misleading in the way it states its own
>>> motivations.  
>> 
>> How so?
> 
> In the sentence "There are plans afoot to look at moving Requests to a
> more event-loop-y model, and doing so basically mandates a MemoryBIO",
> and also in the general feeling it gives that the backport is motivated
> by security reasons primarily.

Ok, let’s address those together.

There are security reasons to do the backport, but they are “it helps us build 
a pathway to PEP 543”. Right now there are a lot of people interested in seeing 
PEP 543 happen, but vastly fewer in a position to do the work. I am, but only 
if I can actually use it for the things that are in my job. If I can’t, then 
PEP 543 becomes an “evenings and weekends” activity for me *at best*, and 
something I have to drop entirely at worst.

Adopting PEP 543 *would* be a security benefit, so while this PEP itself is not 
directly in and of itself a security benefit, it builds a pathway to something 
that is.

As to the plans to move Requests to a more event loop-y model, I think that it 
does stand in the way of this, but only insomuch as, again, we want our event 
loopy model to be as bug-free as possible. But I can concede that rewording on 
that point would be valuable.

*However*, it’s my understanding that even if I did that rewording, you’d still 
be against it. Is that correct? 

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 14:21, Antoine Pitrou <anto...@python.org> wrote:
> 
> 
> Le 01/06/2017 à 15:12, Cory Benfield a écrit :
>> 
>> I don’t know what to do with that answer, really. I gave you some data (80%+ 
>> of requests downloads over the last month were Python 2), and you responded 
>> with “it doesn’t cause us problems”.
> 
> And indeed it doesn't.  Unless the target user base for pip is widely
> different than Python's, it shouldn't cause you any problems either.

Maybe not now, but I think it’s fair to say that it did, right? As I recall, 
Python spent a long time with two fully supported Python versions, and then an 
even longer time with a version that was getting bugfixes. Tell me, which did 
you get more feedback on during that time?

Generally speaking it is fair to say that at this point *every line of code in 
Requests* is exercised or depended on by one of our users. If we write new code 
available to a small fraction of them, and it is in any way sizeable, then that 
stops being true. Again, we should look at the fact that most libraries that 
successfully support Python 2 and Python 3 do so through having codebases that 
share as much code as possible between the two implementations. Each line of 
code that is exercised in only one implementation becomes a vector for a long, 
lingering bug.

Anyway, all I know is that the last big project to do this kind of hard cut was 
Python, and while many of us are glad that Python 3 is real and glad that we 
pushed through the pain, I don’t think anyone would argue that the move was 
painless. A lesson can be learned there, especially for Requests which is not 
currently nursing a problem as fundamental to it as Python was.

>> As a final note, because I think we’re getting into the weeds here: this is 
>> not *necessary*. None of this is *necessary*. Requests exists, and works 
>> today.
> 
> And pip could even bundle a frozen 2.7-compatible version of Requests if
> it wanted/needed to…

Sure, if pip wants to internalise supporting and maintaining that version. One 
of the advantages of the pip/Requests relationship is that pip gets to stop 
worrying about HTTP: if there’s a HTTP problem, that’s on someone else to fix. 
Bundling that would remove that advantage.

> 
>> Let me be clear that there is no intention to use either Tornado or
> Twisted’s HTTP/1.1 parsers or engines. [...] Requests very much intends
> to use our own HTTP logic, not least because we’re sick of relying on
> someone else’s.
> 
> Then the PEP is really wrong or misleading in the way it states its own
> motivations.

How so? TLS is not a part of the HTTP parser. It’s an intermediary layer 
between the transport (resolutely owned by the network layer in 
Twisted/Tornado) and the parsing layer (resolutely owned by Requests). Ideally 
we would not roll our own.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 12:20, Antoine Pitrou <solip...@pitrou.net> wrote:
> 
> On Thu, 1 Jun 2017 12:01:41 +0100
> Cory Benfield <c...@lukasa.co.uk> wrote:
>> In principle, sure. In practice, that means most of our users don’t use 
>> those features and so we don’t get any feedback on whether they’re good 
>> solutions to the problem.
> 
> On bugs.python.org we get plenty of feedback from people using Python
> 3's features, and we have been for years.
> 
> Your concern would have been very valid in the Python 3.2 timeframe,
> but I don't think it is anymore.

Ok? I guess?

I don’t know what to do with that answer, really. I gave you some data (80%+ of 
requests downloads over the last month were Python 2), and you responded with 
“it doesn’t cause us problems”. That’s good for you, I suppose, and well done, 
but it doesn’t seem immediately applicable to the concern I have.

>> All of this is related. I wrote a very, very long email initially and 
>> deleted it all because it was just too long to expect any normal human being 
>> to read it, but the TL;DR here is that we also want to support async/await, 
>> and doing so requires a memory BIO object.
> 
> async/await doesn't require a memory BIO object.  For example, Tornado
> supports async/await (*) even though it doesn't use a memory BIO object
> for its SSL layer.  And asyncio started with a non-memory BIO SSL
> implementation while still using "yield from".
> 
> (*) Despite the fact that Tornado's own coroutines are yield-based
> generators.

You are right, sorry. I should not have used the word “require”. Allow me to 
rephrase.

MemoryBIO objects are vastly, vastly more predictable and tractable than 
wrapped sockets when combined with non-blocking I/O. Using wrapped sockets and 
select/poll/epoll/kqueue, while possible, requires extremely subtle code that 
is easy to get wrong, and can nonetheless still have awkward bugs in it. I 
would be extremely loathe to use such an implementation, but you are correct, 
such an implementation can exist.

>> As to Tornado, the biggest concern there is that there is no support for 
>> composing the TLS over non-TCP sockets as far as I am aware. The wrapped 
>> socket approach is not suitable for some kinds of stream-based I/O that 
>> users really should be able to use with Requests (e.g. UNIX pipes).
> 
> Hmm, why would you use TLS on UNIX pipes except as an academic
> experiment?  Tornado is far from a full-fledged networking package like
> Twisted, but its HTTP(S) support should be very sufficient
> (understandably, since it is the core use case for it).

Let me be clear that there is no intention to use either Tornado or Twisted’s 
HTTP/1.1 parsers or engines. With all due respect to both projects, I have 
concerns about both their client implementations. Tornado’s default is 
definitely not suitable for use in Requests, and the curl backend is but, 
surprise surprise, requires a C extension and oh god we’re back here again. I 
have similar concerns about Twisted’s default HTTP/1.1 client. Tornado’s 
HTTP/1.1 server is certainly sufficient, but also not of much use to Requests. 
Requests very much intends to use our own HTTP logic, not least because we’re 
sick of relying on someone else’s.

Literally what we want is to have an event loop backing us that we can 
integrate with async/await and that requires us to reinvent as few wheels as 
possible while giving an overall better end-user experience. If I were to use 
Tornado, because I would want to integrate PEP 543 support into Tornado I’d 
ultimately have to rewrite Tornado’s TLS implementation *anyway* to replace it 
with a PEP 543 version. If I’m doing that, I’d much rather do it with MemoryBIO 
than wrapped sockets, for all of the reasons above.

As a final note, because I think we’re getting into the weeds here: this is not 
*necessary*. None of this is *necessary*. Requests exists, and works today. 
We’ll get Windows TLS support regardless of anything that’s done here, because 
I’ll just shim it into urllib3 like we did for macOS. What I am pushing for 
with PEP 543 is an improvement that would benefit the whole ecosystem: all I 
want to do is to make it possible for me to actually use it and ship it to 
users in the tools I maintain.

It is reasonable and coherent for python-dev to say “well, good luck, but no 
backports to help you out”. The result of that is that I put PEP 543 on the 
backburner (because it doesn’t solve Requests/urllib3’s problems, and 
ultimately my day job is about resolving those issues), and probably that we 
shutter the async discussion for Requests until we drop Python 2 support. 
That’s fine, Python is your project, not mine. But I don’t see that there’s any 
reason for us not to ask for this backport. After all, the worst you can do is 
say no, and my problems remain the 

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 12:09, David Wilson <dw+python-...@hmmz.org> wrote:
> 
> On Thu, Jun 01, 2017 at 11:47:31AM +0100, Cory Benfield wrote:
> 
>> I have, but discarded the idea.
> 
> I'm glad to hear it was given sufficent thought. :)
> 
> I have one final 'crazy' idea, and actually it does not seem to bad at
> all: can't you just fork a subprocess or spawn threads to handle the
> blocking SSL APIs?
> 
> Sure it wouldn't be beautiful, but it is more appealing than forcing an
> upgrade on all 2.7 users just so they can continue to use pip. (Which,
> ironically, seems to resonate strongly with the motivation behind all of
> this work -- allowing users to continue with their old environments
> without forcing an upgrade to 3.x!)

So, this will work, but at a performance and code cleanliness cost. This 
essentially becomes a Python-2-only code-path, and a very large and complex one 
at that. This has the combined unfortunate effects of meaning a) a 
proportionally small fraction of our users get access to the code path we want 
to take forward into the future, and b) the majority of our users get an 
inferior experience of having a library either spawn threads or processes under 
their feet, which in Python has a tendency to get nasty fast (I for one have 
experienced the joy of having to ctrl+c multiple times to get a program using 
paramiko to actually die).

Again, it’s worth noting that this change will not just affect pip but also the 
millions of Python 2 applications using Requests. I am ok with giving those 
users access to only part of the functionality that the Python 3 users get, but 
I’m not ok with that smaller part also being objectively worse than what we do 
today.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 11:51, Antoine Pitrou <solip...@pitrou.net> wrote:
> 
> On Thu, 1 Jun 2017 11:45:14 +0100
> Cory Benfield <c...@lukasa.co.uk> wrote:
>> 
>> I am claiming that using OpenSSL certificate validation with root stores 
>> that are not intended for OpenSSL can be. This is because trust of a 
>> certificate is non-binary. For example, consider WoSign. The Windows TLS 
>> implementation will distrust certificates that chain up to WoSign as a root 
>> certificate that were issued after October 21 2016. This is not something 
>> that can currently be represented as a PEM file. Therefore, the person 
>> exporting the certs needs to choose: should that be exported or not? If it 
>> is, then OpenSSL will happily trust it even in situations where the system 
>> trust store would not.
> 
> I was not talking about exporting the whole system CA as a PEM file, I
> was talking about adding an option for system adminstrators to
> configure an extra CA certificate to be recognized by pip.

Generally speaking system administrators aren’t wild about this option, as it 
means that they can only add to the trust store, not remove from it. So, while 
possible, it’s not a complete solution to this issue. I say this because the 
option *already* exists, at least in part, via the REQUESTS_CA_BUNDLE 
environment variable, and we nonetheless still get many complaints from system 
administrators.

>> More generally, macOS allows the administrator to configure graduated trust: 
>> that is, to override whether or not a root should be trusted for certificate 
>> validation in some circumstances. Again, exporting this to a PEM does not 
>> persist this information.
> 
> How much of this is relevant to pip?

Depends. If the design goal is “pip respects the system administrator”, then 
the answer is “all of it”. An administrator wants to be able to configure their 
system trust settings. Ideally they want to do this once, and once only, such 
that all applications on their system respect it.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 11:39, Antoine Pitrou <solip...@pitrou.net> wrote:
> 
> On Thu, 1 Jun 2017 11:22:21 +0100
> Cory Benfield <c...@lukasa.co.uk> wrote:
>> 
>> Who is the “we” that should move on? Python core dev? Or the Python 
>> ecosystem?
> 
> Sorry.  Python core dev certainly.  As for the rest of the ecosystem, it
> is moving on as well.

Moving, sure, but slowly. Again, I point to the 80% download number.

>> Requests is stuck in a place from which it cannot move.
>> We feel we cannot drop 2.7 support. We want to support as many TLS
>> backends as possible.
> 
> Well, certain features could be 3.x-only, couldn't they?

In principle, sure. In practice, that means most of our users don’t use those 
features and so we don’t get any feedback on whether they’re good solutions to 
the problem. This is not great. Ideally we want features to be available across 
as wide a deploy base as possible, otherwise we risk shipping features that 
don’t solve the actual problem very well. Good software comes, in part, from 
getting user feedback.

>> We want to enable the pip developers to focus on
>> their features, rather than worrying about HTTP and TLS. And we want
>> people to adopt the async/await keywords as much as possible.
> 
> I don't get what async/await keywords have to do with this.  We're
> talking about backporting the ssl memory BIO object…

All of this is related. I wrote a very, very long email initially and deleted 
it all because it was just too long to expect any normal human being to read 
it, but the TL;DR here is that we also want to support async/await, and doing 
so requires a memory BIO object.

>> I want to move on, but I want to bring that 80% of our userbase with us when 
>> we do. My reading of your post is that you would rather Requests not adopt 
>> the async/await paradigm than backport MemoryBIO: is my understanding 
>> correct?
> 
> Well you cannot use async/await on 2.7 in any case, and you cannot use
> asyncio on 2.7 (Trollius, which was maintained by Victor, has been
> abandoned AFAIK).  If you want to use coroutines in 2.7, you need to
> use Tornado or Twisted.  Twisted may not, but Tornado works fine with
> the stdlib ssl module.

I can use Twisted on 2.7, and Twisted has great integration with async/await 
and asyncio when they are available. Great and getting greater, in fact, thanks 
to the work of the Twisted and asyncio teams.

As to Tornado, the biggest concern there is that there is no support for 
composing the TLS over non-TCP sockets as far as I am aware. The wrapped socket 
approach is not suitable for some kinds of stream-based I/O that users really 
should be able to use with Requests (e.g. UNIX pipes). Not a complete 
non-starter, but also not something I’d like to forego.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 11:39, David Wilson <dw+python-...@hmmz.org> wrote:
> 
> Hi Cory,
> 
> On Thu, Jun 01, 2017 at 11:22:21AM +0100, Cory Benfield wrote:
> 
>> We want to support as many TLS backends as possible.
> 
> Just a wild idea, but have you investigated a pure-Python fallback for
> 2.7 such as TLSlite? Of course the fallback need only be used during
> bootstrapping, and the solution would be compatible with every stable
> LTS Linux distribution release that was not shipping the latest and
> greatest 2.7.

I have, but discarded the idea. There are no pure-Python TLS implementations 
that are both feature-complete and actively maintained. Additionally, doing 
crypto operations in pure-Python is a bad idea, so any implementation that did 
crypto in Python code would be ruled out immediately (which rules out TLSLite), 
so I’d need what amounts to a custom library: pure-Python TLS with crypto from 
OpenSSL, which is not currently exposed by any Python module. Ultimately it’s 
just not a winner.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 11:28, Antoine Pitrou <anto...@python.org> wrote:
> 
> 
> Le 01/06/2017 à 12:23, Cory Benfield a écrit :
>> 
>> No it can’t.
>> 
>> OpenSSL builds chains differently, and disregards some metadata that Windows 
>> and macOS store, which means that cert validation will work differently than 
>> in the system store. This can lead to pip accepting a cert marked as 
>> “untrusted for SSL”, for example, which would be pretty bad.
> 
> Are you claiming that OpenSSL certificate validation is insecure and
> shouldn't be used at all?  I have never heard that claim before.

Of course I’m not.

I am claiming that using OpenSSL certificate validation with root stores that 
are not intended for OpenSSL can be. This is because trust of a certificate is 
non-binary. For example, consider WoSign. The Windows TLS implementation will 
distrust certificates that chain up to WoSign as a root certificate that were 
issued after October 21 2016. This is not something that can currently be 
represented as a PEM file. Therefore, the person exporting the certs needs to 
choose: should that be exported or not? If it is, then OpenSSL will happily 
trust it even in situations where the system trust store would not.

More generally, macOS allows the administrator to configure graduated trust: 
that is, to override whether or not a root should be trusted for certificate 
validation in some circumstances. Again, exporting this to a PEM does not 
persist this information.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 11:18, Antoine Pitrou  wrote:
> 
> On Thu, 1 Jun 2017 20:05:48 +1000
> Chris Angelico  wrote:
>> 
>> As stated in this thread, OS-provided certificates are not handled by
>> that. For instance, if a local administrator distributes a self-signed
>> cert for the intranet server, web browsers will use it, but pip will
>> not.
> 
> That's true.  But:
> 1) pip could grow a config entry to set an alternative or additional CA
> path

No it can’t.

Exporting the Windows or macOS security store to a big file of PEM is a 
security vulnerability because the macOS and Windows security stores expect to 
work with their own certificate chain building algorithms. OpenSSL builds 
chains differently, and disregards some metadata that Windows and macOS store, 
which means that cert validation will work differently than in the system 
store. This can lead to pip accepting a cert marked as “untrusted for SSL”, for 
example, which would be pretty bad.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 10:23, Antoine Pitrou  wrote:
> 
> Yes, I disagree.  We needn't backport that new API to Python 2.7.
> Perhaps it's time to be reasonable: Python 2.7 has been in bugfix-only
> mode for a very long time.  Python 3.6 is out.  We should move on.

Who is the “we” that should move on? Python core dev? Or the Python ecosystem? 
Because if it’s the latter, then I’m going to tell you right now that the 
ecosystem did not get the memo. If you check the pip download numbers for 
Requests in the last month you’ll see that 80% of our downloads (9.4 million) 
come from Python 2. That is an enormous proportion: far too many to consider 
not supporting that user-base. So Requests is basically bound to support that 
userbase.

Requests is stuck in a place from which it cannot move. We feel we cannot drop 
2.7 support. We want to support as many TLS backends as possible. We want to 
enable the pip developers to focus on their features, rather than worrying 
about HTTP and TLS. And we want people to adopt the async/await keywords as 
much as possible. It turns out that we cannot satisfy all of those desires with 
the status quo, so we proposed an alternative that involves backporting 
MemoryBIO.

So, to the notion of “we need to move on”, I say this: we’re trying. We really, 
genuinely, are. I don’t know how much stronger of a signal I can give about how 
much Requests cares about Python 3 than to signal that we’re trying to adopt 
async/await and be compatible with asyncio. I believe that Python 3 is the 
present and future of this language. But right now, we can’t properly adopt it 
because we have a userbase that you want to leave behind, and we don’t.

I want to move on, but I want to bring that 80% of our userbase with us when we 
do. My reading of your post is that you would rather Requests not adopt the 
async/await paradigm than backport MemoryBIO: is my understanding correct? If 
so, fair enough. If not, I’d like to try to work with you to a place where we 
can all get what we want.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 31 May 2017, at 08:42, Victor Stinner <victor.stin...@gmail.com> wrote:
> 
> Hi,
> 
> I wrote a PEP based on the previous thread "Backport ssl.MemoryBIO on
> Python 2.7?". Thanks for Cory Benfield, Alex Gaynor and Nick Coghlan
> who helped me to write it!

It probably goes without saying, given that I helped with the drafting for the 
PEP, but I’m strongly in favour of this PEP. Just in case it helps to get that 
reaffirmation here. ;)

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?

[Cory Benfield]

> On 23 May 2017, at 20:34, David Wilson <dw+python-...@hmmz.org> wrote:
> 
> On Tue, May 23, 2017 at 06:13:17PM -0700, Cory Benfield wrote:
> 
>> There are discussions around Requests unvendoring its dependencies
>> thanks to the improved nature of pip. This might be a year of pretty
>> big changes for Requests.
> 
> In which case, what is to prevent Requests from just depending on
> pyOpenSSL as usual?

Requests currently does not depend on PyOpenSSL on 2.7: it’s simply an optional 
dependency for those who need it.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?

[Cory Benfield]

> On 23 May 2017, at 17:58, Victor Stinner <victor.stin...@gmail.com> wrote:
> 
> 2017-05-23 19:54 GMT-05:00 Cory Benfield <c...@lukasa.co.uk>:
>> In the absence of a Python 2.7 backport, Requests is required to basically 
>> use the same solution that Twisted currently does: namely, a mandatory 
>> dependency on PyOpenSSL.
> 
> Last time I looked at requests, it embedded all its dependencies. I
> don't like the idea of embedding PyOpenSSL…

There are discussions around Requests unvendoring its dependencies thanks to 
the improved nature of pip. This might be a year of pretty big changes for 
Requests.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?

[Cory Benfield]

> On 23 May 2017, at 17:46, Victor Stinner <victor.stin...@gmail.com> wrote:
> 
> 
> He described me his PEP and I strongly support it (sorry, I missed it
> when he posted it on python-dev), but we decided (Guido van Rossum,
> Christian Heimes, Cory Benfield and me, see the tweet below) to not
> put this in the stdlib right now, but spend more time on testing it on
> Twisted, asyncio, requests, etc. So publishing an implementation on
> PyPI was proposed instead. It seems like we agreed on a smooth plan
> (or am I wrong, Cory?).

Yes, this is correct. There are plans afoot to look at moving Requests to a 
more event-loop-y model, and doing so basically mandates a MemoryBIO. In the 
absence of a Python 2.7 backport, Requests is required to basically use the 
same solution that Twisted currently does: namely, a mandatory dependency on 
PyOpenSSL.

I’m very much +1 on this.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 543: A Unified TLS API for Python

[Cory Benfield]

> On 24 Feb 2017, at 09:55, Nick Coghlan  wrote:
> 
> Heh, I guess you must have already caught everyone with a strong opinion 
> about this by way of security-sig :)

So it seems. =D

> On the procedural front, the main open question is whether or not Guido wants 
> to review and approve this PEP himself, or if he'd prefer to delegate that 
> task.
> 
> Assuming the latter case, I think it may make sense for Christian to handle 
> the BDFL-Delegate responsibilities as well - I know he's a co-author of the 
> PEP, but I also think he's the most qualified to make the final decision on 
> the suitability of this design.

Either way, I think I’d like to confirm this works by writing two more modules. 
First, I’d like to actually write most of the tls module into something that 
can live on PyPI, in no small part because backporting it would be useful for 
tools like urllib3. Secondly, I’d like to write a shim for the standard library 
ssl module so that I have a clearer picture of whether we want to shim it in 
Python code or whether we should take this opportunity to write new bindings to 
the C code.

The goal here, essentially, is to have a PoC that the API is at least suitable 
for tools like urllib3/requests before we commit to it too much, and the best 
way to do that is to prove that the stdlib can meet that API and that at least 
one other implementation can do as well. I’ll use SecureTransport for that 
proof point, but I’ll make available a branch of urllib3 that has patches 
applied to use the new API so that anyone who wants to investigate shimming a 
different TLS implementation has got a test suite they can try to run.

 Obviously that’s slightly orthogonal to PEP acceptance, but I wanted to keep 
people up-to-speed with my thinking.

Cory___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 543: A Unified TLS API for Python

[Cory Benfield]

All,

Python has never exposed the ability to use TLS backends other than OpenSSL in 
the standard library. As Christian and I discussed back at the Developer Summit 
at PyCon US 2016, the ssl module would more properly be called the openssl 
module, due to exposing many OpenSSL-specific concepts and behaviours in its 
API. This has meant that the Python ecosystem is overwhelmingly also an OpenSSL 
ecosystem, which is problematic on Windows and macOS (and Unices that for 
whatever reason aren’t interested in shipping an OpenSSL), as it has meant that 
Python needs to bring its own OpenSSL, and that it is troublesome to interact 
with the system trust database.

The first step to resolving this would be to provide a new module that exposes 
TLS concepts in a much more generic manner. There are two possible approaches 
to this. The first is to have this module be a generic concrete implementation 
that can be compiled against multiple TLS backends (like curl). This would 
require that all the relevant bindings be built into the standard library from 
the get-go, which provides a substantial maintenance burden on a team of people 
that are already understaffed maintaining the ssl module alone. The second 
approach is to define a generic high-level TLS interface that provides a 
minimal usable interface that can be implemented by both first- and third-party 
modules. This would allow the standard library to continue to ship with only 
exactly what it needs (for example, OpenSSL, SecureTransport and SChannel), but 
would give those who want to use more esoteric TLS choices (NSS, GnuTLS, 
mbedTLS, and BoringSSL are some examples) an API that they can implement that 
will guarantee that complying modules can use the appropriate TLS backend.

To that end, I’ve composed a draft PEP that would define this API: PEP 543. 
This draft can be found online here[1], and I have reproduced it below for 
those who want to reply inline. I should apologise in advance that this PEP is 
quite large: it is this large because it lays out a substantial proportion of 
the new code that would be added directly in the PEP. This is to help de-risk 
the work by showing as much of it up-front during the PEP stage, rather than 
having it emerge over the implementation process.

Please note that this proposal is not intended to resolve the current problem 
pip has with TLSv1.2, and so is not subject to the tight time constraints 
associated with that problem. That problem will be solved by building a 
temporary shim into urllib3 (and therefore pip) that can use SecureTransport 
bindings on the Mac. This proposal is intended as a solution to the 
more-general problem of supporting different TLS backends in Python, and so is 
larger in scope and less urgent.

I should also mention that there will be a tendency to want to make this API 
all things to all people from the get-go. I’m going to strongly resist attempts 
to extend this API too much, because each additional bit of API surface makes 
it harder for us to encourage module authors to conform to this API. I will 
encourage people to extend this API over time as needed, but to begin with I 
think it is most important that basic TLS clients and servers can be 
constructed with this API. More specialised features should be considered 
future enhancements, rather than being tacked on to this initial PEP.

Please let me know what you think. I’d like significant feedback from the 
community before I ask the BDFL to pronounce on this PEP, as this is the kind 
of change that will affect a large amount of the Python ecosystem. It’s 
important to me that we minimise the risk to ourselves and the community by 
getting this as right as possible, and while we can help with that by limiting 
scope, it’s also true that we need as many eyes as possible.

Please let me know what you think.

Cory

[1]: https://www.python.org/dev/peps/pep-0543/

—

PEP: 543
Title: A Unified TLS API for Python
Version: $Revision$
Last-Modified: $Date$
Author: Cory Benfield <c...@lukasa.co.uk>,
Christian Heimes <christ...@python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 17-Oct-2016
Python-Version: 3.7
Post-History: 11-Jan-2017, 19-Jan-2017, 02-Feb-2017, 09-Feb-2017


Abstract


This PEP would define a standard TLS interface in the form of a collection of
abstract base classes. This interface would allow Python implementations and
third-party libraries to provide bindings to TLS libraries other than OpenSSL
that can be used by tools that expect the interface provided by the Python
standard library, with the goal of reducing the dependence of the Python
ecosystem on OpenSSL.


Rationale
=

In the 21st century it has become increasingly clear that robust and
user-friendly TLS support is an extremely important part of the ecosystem of
any popular programming language. For most of its lifetime, this role in the
Python ecosystem has primarily been served by the

Re: [Python-Dev] SSL certificates recommendations for downstreampython packagers

[Cory Benfield]

> On 2 Feb 2017, at 03:38, Stephen J. Turnbull 
> <turnbull.stephen...@u.tsukuba.ac.jp> wrote:
> 
> Cory Benfield writes:
> 
>> The TL;DR is: I understand Christian’s concern, but I don’t think
>> it’s important if you’re very, very careful.
> 
> But AIUI, the "you" above is the end-user or admin of end-user's
> system, no?  We know that they aren't very careful (or perhaps more
> accurate, this is too fsckin' complicated for anybody but an infosec
> expert to do very well).

I think "you" is the coder of the interface.

From a security perspective I think we have to discount the possibility of 
administrator error from our threat model. A threat model that includes “defend 
the system against intrusions that the administrator incorrectly allows” is an 
insanely difficult one to respond to, given that it basically requires psychic 
powers to determine what the administrator *meant* instead of what they 
configured. Now, where we allow configuration we have a duty to ensure that 
it’s as easy as possible to configure correctly, but when using the system 
trust store most of the configuration is actually provided by the OS tools, 
rather than by the above-mentioned “you”, so that’s not in our control.

The risk, and the need to be very very careful, comes from ensuring that the 
semantics of the OS configuration are preserved through to the behaviour of the 
program. This is definitely a place with razor-blades all around, which is why 
I have tended to defer to the Chrome security team on this issue. In 
particular, the BoringSSL developers are razor-sharp people who have their 
heads screwed on when it comes to practical security decisions, and I’ve found 
that emulating them is usually a safe bet in the face of ambiguity.

However, it’s unquestionable that the *safest* route to go down in terms of 
preserving the expectations of users is to use the platform-native TLS 
implementation wholesale, rather than do a hybrid model like Chrome does where 
OpenSSL does the protocol bits and the system does the X509 bits. That way 
Python ends up behaving basically like Edge or Safari on the relevant 
platforms, or perhaps more importantly behaving like .NET on Windows and like 
CoreFoundation on macOS, which is a much better place to be in terms of user 
and administrator expectations.

As a side benefit, that approach helps take Python a bit closer to feeling 
“platform-native” on many platforms, which can only be a good thing for those 
of us who want to see more Python on the desktop (or indeed on the mobile 
device).

> I[1] still agree with you that it's *unlikely* that end-users/admins
> will need to worry about it.  But we need to be really careful about
> what we say here, or at least where the responsible parties will be
> looking.

I agree. In an ideal world I’d say to Steve that he should shelve his current 
work and wait for the TLS ABC PEP that is incoming (hopefully I’ll send a 
first-draft to python-dev today). However, I’m nothing if not pragmatic, and 
having Steve continue his current work in parallel to the TLS ABC PEP is 
probably a good idea so that we can avoid having all our eggs in one basket. 
Perhaps we can get the TLS ABC stuff in place in time for Steve to just swap 
over to using SChannel altogether, but if that doesn’t work out and Steve can 
get a halfway-house out the door earlier then that’s fine by me.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstreampython packagers

[Cory Benfield]

> On 1 Feb 2017, at 14:20, Steve Dower  wrote:
> 
> Sorry, I misspoke when I said "certificate validation callback", I meant the 
> same callback Cory uses below (name escapes me now, but it's unfortunately 
> similar to what I said). There are two callbacks in OpenSSL, one that allows 
> you to verify each certificate in the chain individually, and one that 
> requires you to validate the entire chain.
> 
> I do indeed take the entire chain in one go and pass it to the OS API. 
> Christian also didn't like that I was bypassing *all* of OpenSSL's 
> certificate handling here, but maybe there's a way to make it reliable if 
> Chrome has done it?

So, my understanding is that bypassing OpenSSL’s cert handling is basically 
fine. The risks are only in cases where OpenSSL’s cert handling would be a 
supplement to what the OS provides, which is not really very common and I don’t 
think is a major risk for Python.

So in general, it is not unreasonable to ask your OS “are these certificates 
valid for this connection based on your trust DB” and circumventing OpenSSL 
entirely there. Please do bear in mind you need to ask your OS the right 
question. For Windows this stuff is actually kinda hard because the API is 
somewhat opaque, but you have to worry about setting correct certificate 
usages, building up chain policies, and then doing appropriate error handling 
(AFAIK the crypto API can “fail validation” for some reasons that have nothing 
to do with validation itself, so worth bearing that in mind).

The TL;DR is: I understand Christian’s concern, but I don’t think it’s 
important if you’re very, very careful.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Cory Benfield]

> On 31 Jan 2017, at 18:26, Steve Dower  wrote:
> 
> In short, I want to allow Python code to set OpenSSL's certificate validation 
> callback. Basically, given a raw certificate, return True/False based on 
> whether it should be trusted. I then have separate code (yet to be published) 
> implementing the callback on Windows by calling into the WinVerifyTrust API 
> with the HTTPS provider, which (allegedly) behaves identically to browsers 
> using the same API (again, allegedly - I have absolutely no evidence to 
> support this other than some manual testing).

For context here Steve, this is not quite what Chrome does (and I cannot stress 
enough that the Chrome approach is the best one I’ve seen, the folks working on 
it really do know what they’re doing). The reason here is a bit tricky, but 
essentially the validation callback is called incrementally for each step up 
the chain. This is not normally what a platform validation API actually wants: 
generally they want the entire cert chain the remote peer sent at once.

Chrome, instead, essentially disables the OpenSSL cert validation entirely: 
they still require the certificate be presented, but override the verification 
callback to always say “yeah that’s cool, no big deal”. They then take the 
complete cert chain provided by the remote peer and pass that to the platform 
validation code in one shot after the handshake is complete, but before they 
send/receive any data on the connection. This is still safe: so long as you 
don’t actually expose any data before you have validated the certificates you 
aren’t at risk.

I have actually prototyped this approach for Requests/urllib3 in the past. I 
wrote a small Rust extension to call into the platform-native code, and then 
wrapped it in a CFFI library that exposed a single callable to validate a cert 
chain for a specific hostname (library is here: 
https://github.com/python-hyper/certitude 
). This could then be called from 
urllib3 code that used PyOpenSSL using this patch here: 
https://github.com/shazow/urllib3/pull/802/files 


PLEASE DON’T ACTUALLY USE THIS CODE. I have not validated that certitude does 
entirely the right things with the platform APIs. This is just an example of a 
stripped-down version of what Chrome does, as a potential example of how to get 
something working for your Python use-case.

Cory___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Cory Benfield]

> On 31 Jan 2017, at 09:56, Paul Moore <p.f.mo...@gmail.com> wrote:
> 
> On 31 January 2017 at 09:19, Cory Benfield <c...@lukasa.co.uk> wrote:
>> 
>> In general, it is unwise to mix trust stores. If you want to use your OS’s
>> trust store, the best approach is to use the OS’s TLS stack as well. At
>> least that way when a user says “It works in my browser”, you know it should
>> work for you too.
> 
> As a bystander (and an "end user" of this stuff) the message I'm
> getting here is a bit worrying. To take a step back from the sysadmin
> issues here, is the statement
> 
>It's safe to use Python (either via the stdlib, or various 3rd
> party libraries like requests) to access https URLs
> 
> correct? I understand that "safe" is a complex concept here, but in
> terms of promoting Python, I'd be using the term in the sense of "at
> least as acceptable as using something like C# or Java" - in other
> words I'm not introducing any new vulnerabilities if I argue for
> Python over one of those languages?

My answer would be “Yes*”.

The asterisk is that it works best when you understand how you are rooting your 
trust stores. On Linux, if you use your distro provided OpenSSL and its trust 
store it is as safe as the rest of your system. On macOS, if you use the system 
Python it is unsafe. If you use a Python compiled against Homebrew’s OpenSSL, 
it is safe, though it is only safe based on a snapshot in time (this is for 
boring Homebrew-related reasons that are nothing to do with Python). On 
Windows, it is safe, though potentially less functional. Requests is basically 
safe, though it is definitely possible.

If you are trying to reach websites on the open web using Requests, that’s 
safe, subject to the criteria below.

The answer gets murkier if you do substantial configuration of a trust 
database. If you have added or removed root certificates from a system trust 
store, or you have configured the trust database on a Windows or Apple machine, 
then it all gets murky fast. If you’ve only added or expanded trust then Python 
is still safe on all those platforms, it is just less functional. If you’ve 
removed trust then it is possible that a Python process will trust something 
that other processes on your machine will not.

However, you should be aware that that’s basically the default state of being 
for TLS. Java is basically in the same place as Python today, but with the 
added fun of having a hand-rolled TLS implementation (JSSE). C# is a different 
story: the stdlib uses SSPI for certificate validation, which basically means 
SChannel *on Windows*. In the wider .NET ecosystem I have no idea how this is 
being done.

So C# applications are Windows-native safe on Windows, and are a crapshoot 
elsewhere. For Java vs Python, I’d say we’re slightly ahead right now.

Again, the long-term solution to this fix is to allow us to use SChannel and 
SecureTransport to provide TLS on the relevant platforms. This will also let 
people use GnuTLS or NSS or whatever other TLS implementations float their boat 
on other Unices. I’ll be bringing a draft PEP in front of python-dev sometime 
in the next month to start this work: if you’re interested, I recommend helping 
out with that process! 

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Cory Benfield]

> On 31 Jan 2017, at 09:33, Christian Heimes  wrote:
> 
> One small correction, it is possible to export some of the trust
> settings to a TRUSTED CERTIFICATE and import them into OpenSSL. It works
> correctly in 1.0.1 and since 1.0.2e or f. Trust settings are stored in
> X509_AUX extension after the actual certificate and signature. OpenSSL's
> default loaders for cert dir and cert file do load auxiliary trust
> information.

Ah, good spot.

I suspect the code you’d need to write to safely extract that functionality is 
pretty subtle. I definitely don’t trust myself to get it right.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Cory Benfield]

> On 30 Jan 2017, at 21:00, David Cournapeau <courn...@gmail.com> wrote:
> 
> 
> 
> On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <c...@lukasa.co.uk 
> <mailto:c...@lukasa.co.uk>> wrote:
> 
> 
> > On 30 Jan 2017, at 13:53, David Cournapeau <courn...@gmail.com 
> > <mailto:courn...@gmail.com>> wrote:
> >
> > Are there any official recommendations for downstream packagers beyond PEP 
> > 476 ? Is it "acceptable" for downstream packagers to patch python's default 
> > cert locations ?
> 
> There *are* no default cert locations on Windows or macOS that can be 
> accessed by OpenSSL.
> 
> I cannot stress this strongly enough: you cannot provide a platform-native 
> certificate validation logic for Python *and* use OpenSSL for certificate 
> validation on Windows or macOS. (macOS can technically do this when you link 
> against the system OpenSSL, at the cost of using a catastrophically insecure 
> version of OpenSSL.)
> 
> Ah, thanks, that's already useful information.
> 
> Just making sure I understand: this means there is no way to use python's SSL 
> library to use the system store on windows, in particular private 
> certifications that are often deployed by internal ITs in large orgs ?

If only it were that simple!

No, you absolutely *can* do that. You can extract the trust roots from the 
system trust store, convert them into PEM/DER-encoded files, and load them into 
OpenSSL. That will work.

The problem is that both SecureTransport and SChannel have got a number of 
differences from OpenSSL. In no particular order:

1. Their chain building logic is different. This means that, given a collection 
of certificates presented by a server and a bundle of already-trusted certs, 
each implementation may build a different trust chain. This may cause one 
implementation to refuse to validate where the others do, or vice versa. This 
is very common with older OpenSSLs.
2. SecureTransport and SChannel both use the system trust DB, which on both 
Windows and mac allows the setting of custom policies. OpenSSL won’t respect 
these policies, which means you can fail-open (that is, export and use a root 
certificate that the OS believes should not be trusted for a given use case). 
There is no way to export these trust policies into OpenSSL.
3. SecureTransport, SChannel, and OpenSSL all support different X.509 
extensions and understand them differently. This means that some certs may be 
untrusted for certain uses by Windows but trusted for those uses by OpenSSL, 
for example.

In general, it is unwise to mix trust stores. If you want to use your OS’s 
trust store, the best approach is to use the OS’s TLS stack as well. At least 
that way when a user says “It works in my browser”, you know it should work for 
you too.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Cory Benfield]

> On 30 Jan 2017, at 13:53, David Cournapeau  wrote:
> 
> Are there any official recommendations for downstream packagers beyond PEP 
> 476 ? Is it "acceptable" for downstream packagers to patch python's default 
> cert locations ?

There *are* no default cert locations on Windows or macOS that can be accessed 
by OpenSSL.

I cannot stress this strongly enough: you cannot provide a platform-native 
certificate validation logic for Python *and* use OpenSSL for certificate 
validation on Windows or macOS. (macOS can technically do this when you link 
against the system OpenSSL, at the cost of using a catastrophically insecure 
version of OpenSSL.) 

The only program I am aware of that does platform-native certificate validation 
on all three major desktop OS platforms is Chrome. It does this using a fork of 
OpenSSL to do the actual TLS, but the platform-native crypto library to do the 
certificate validation. This is the only acceptable way to do this, and Python 
does not expose the appropriate hooks to do it from within Python code. This 
would require that you carry substantial patches to the standard library to 
achieve this, all of which would be custom code. I strongly recommend you don't 
undertake to do this unless you are very confident of your ability to write 
this code correctly.

The best long term solution to this is to stop using OpenSSL on platforms that 
don't consider it the 'blessed' approach. If you're interested in following 
that work, we're currently discussing it on the security-SIG, and you'd be 
welcome to join. 

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecate `from __future__ import unicode_literals`?

[Cory Benfield]

> On 16 Dec 2016, at 16:07, Ethan Furman  wrote:
> 
> On 12/16/2016 11:24 AM, Guido van Rossum wrote:
> 
>> I am beginning to think that `from __future__ import unicode_literals` does
>> more harm than good. I don't recall exactly why we introduced it, but with
>> the restoration of u"" literals in Python 3.3 we have a much better story
>> for writing straddling code that is unicode-correct.
> 
> So cross-version code would be primarily 2.7 and 3.3+ ?  I can live with that.

Speaking for third-party library authors, almost all cross-version code that 
does anything remotely close to a network is 2.7 and 3.3+. Requests dropped 3.2 
support basically as soon as we could once 3.3’s unicode literals were 
restored, and since then I haven’t written anything that targets 3.2. It’s just 
too frustrating.

And while I’m shoving my oar in, I’ve never seen anyone be happy with using 
“from __future__ import unicode_literals”. As others in this thread have said, 
it just points a loaded gun at your foot and lets you wait for it to go off.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Cory Benfield]

> On 30 Aug 2016, at 16:07, M.-A. Lemburg  wrote:
> 
> That was not my point. It's unfortunate that Python depends on
> a library which is inevitably going to need updates frequently,
> and which then may have the implication that Python won't compile on
> systems which don't ship with more recent OpenSSL libs - even
> if your application doesn't even need ssl at all.
> 
> Crypto is important to have, but at the same time it's not
> essentially for everything you do in Python, e.g. you can
> easily run data analysis scripts or applications without ever
> touching the ssl module.
> 
> Yet, a move to require OpenSSL 1.0.2 for Python 3.7 will make
> it impossible to run such apps on systems that still use OpenSSL
> 1.0.1, e.g. Ubuntu 14.04 or CentOS 7.

If your application doesn’t need SSL, then you can compile without OpenSSL. I 
just downloaded and compiled the current tip of the CPython repository on a 
system with no OpenSSL, and the world didn’t explode, it just printed this:

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2  _curses   _curses_panel  
_dbm  _gdbm _lzma  
_sqlite3  _ssl  _tkinter   
readline  zlib 
To find the necessary bits, look in setup.py in detect_modules() for the 
module's name.

So this user you have considered, who needs Python but not the ssl module, is 
still well served. The ssl module is not mandatory in CPython, and no-one is 
proposing that it should be.

But the real question is this: who *is* this hypothetical user? This user 
apparently needs the latest CPython, but is entirely unwilling to update 
literally anything else, including moving to a more recent release of their 
operating system. They are equipped to compile Python from source, but are 
apparently unwilling or unable to install a more recent OpenSSL from source. 
I’m not entirely certain that python-dev should be supporting that user: that 
user should be contacting their LTS supplier.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Cory Benfield]

> On 29 Aug 2016, at 15:31, M.-A. Lemburg  wrote:
> 
> Ubuntu 14.04 is a widely deployed system and newer Python version
> should run on such widely deployed systems without having to
> replace important vendor maintained system libraries such as
> OpenSSL.

That's quite the non-sequitur. You never answered my question: what does 
"widely deployed" mean? At what level of deployment do we need to support the 
system configuration with no changes?

Do we need to support compiling out of box with Windows 10? Because we don't: 
if they want SSL, we need them to compile and install an OpenSSL. Do we need to 
support compiling out of the box on macOS 10.12 Sierra? Because we don't: if 
they want SSL they need to install their own OpenSSL.

At a certain point we need to give up on systems that do not provide up to date 
copies of important libraries, or say that if you want to use Python on them 
you need to compile without our support libraries. 

> Python 3.7 starts shipping around June 2018 (assuming the 18 month
> release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
> be able to use Python 3.7 on such a system, you'd have to upgrade
> to a more recent LTS version 10 months before the EOL date (with
> all the associated issues) or lose vendor maintenance support and
> run with your own copy of OpenSSL.

Yes, that's true. But on the other hand, that LTS release is *already out* at 
this time, and has been for four months. By the time of the release of Python 
3.7 it will have been released for two years and two months. The request to 
upgrade is not unreasonable. 

> Sure, but Ubuntu will continue to support OpenSSL 1.0.1
> until 2019, backporting important security fixes as necessary and
> that's what's important.

Then Ubuntu can ship us an engineer who is willing to support the SSL module 
with OpenSSL 1.0.1 going forward. The one engineer we have has said he is 
unwilling to do it.

> This doesn't sound like a feature worth breaking compatibility
> to me.

Does the compatibility guarantee apply to libraries that Python will link 
against?

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Cory Benfield]

> On 29 Aug 2016, at 04:09, M.-A. Lemburg  wrote:
> 
> On 28.08.2016 22:40, Christian Heimes wrote:
>> ...
>> I like to reduce the maintenance burden and list of supported OpenSSL
>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>> will reach EOL by the end of this year,
>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>> 0.9.8 is still required for some platforms (OSX).
>> ...
>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>> 1.0.2 features for 3.7.
>> ...
> 
> Hmm, that last part would mean that Python 3.7 will no longer compile
> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> Since 14.04 LTS is supported until 2019, I think it would be better
> to only start requiring 1.0.2 in Python 3.8.

Can someone explain to me why this is a use-case we care about?

The nature of a stable distribution is that all the packages are guaranteed to 
work together. Once you start compiling your own software, you are out in the 
cold: you are no longer covered by that guarantee. Why, then, should someone 
compiling a version of Python that was barely conceived when Ubuntu 14.04 was 
released expect that they can compile it from source using only dependencies 
available in their mainline distribution?

I absolutely understand wanting to support Ubuntu 14.04 for any release of 
Python that already compiles on Ubuntu 14.04: that makes sense. But new 
releases should not be shackled to what is in LTS releases of operating 
systems. If it were, a more coherent argument would be that we cannot drop 
0.9.8 support in any Python released before the middle of 2017 because RHEL 5 
ships it, and we will not be able to require OpenSSL 1.0.2 until almost 2021 
because RHEL 6 ships 1.0.1. After all, why does Ubuntu 14.04 get privileged 
over RHEL? Both are supported LTS releases, after all.

I don’t believe that the Python dev team has ever promised that future versions 
of Python will compile on a given LTS release. Am I mistaken?

While I’m here, I should note that Ubuntu 14.04 goes EOL in 2019, while OpenSSL 
1.0.1 loses support from upstream at the end of this year, which is probably a 
good reason to stop supporting it in new Python versions. OpenSSL 1.0.0 and 
0.9.8 are already unsupported by upstream.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

[Cory Benfield]

> On 28 Aug 2016, at 16:40, Christian Heimes  wrote:
> 
> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> 1.0.2 features for 3.7. What is the status of Python.org's OSX builds?
> Is it possible to drop 0.9.8?

I strongly support this change. Python with old OpenSSL versions is an enormous 
source of difficult-to-debug problems for users attempting to work with the 
web, and puts our users at totally unnecessary risk. Let’s move on.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] security SIG? (was: Discussion overload)

[Cory Benfield]

> On 18 Jun 2016, at 04:06, Brett Cannon  wrote:
> 
> Do we need a security SIG? E.g. would people like Christian and Cory like to 
> have a separate place to talk about the ssl stuff brought up at the language 
> summit?


Honestly, I’m not sure what we would gain.

Unless that SIG is empowered to take action, all it will be is a factory for 
generating arguments like this one. It will inevitably be either a toxic 
environment in itself, or a source of toxic threads on python-dev as the 
security SIG brings new threads like this one to the table.

It should be noted that of the three developers that originally stepped forward 
on the security side of things here (myself, Donald, and Christian), only I am 
left subscribed to python-dev and nosy’d on the relevant issues. Put another 
way: each time we do this, several people on the security side burn themselves 
out in the thread and walk away (it’s possible that those on the other side of 
the threads do too, I just don’t know those people so well). It’s hard to get 
enthusiastic about signing people up for that. =)

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] security SIG? (was: Discussion overload)

[Cory Benfield]

> On 18 Jun 2016, at 11:38, Stephen J. Turnbull  wrote:
> 
> I see the security issue as a backyard swimming pool.  The law may say
> you must put a fence around it, but even 6 year olds can climb the
> fence, fall in the pool, and drown.  The hard-line security advocate
> position then is "the risk is a *kid's life*, backyard pools must be
> banned".  You have to sympathize with their honest and deep concern,
> but the community accepts that risk in the case of swimming pools.  I
> suspect the Python community at large is going to be happy with
> Larry's decision and the strategy of emphasizing the secrets module
> starting with 3.6.

I don’t think that’s really an accurate representation of any of the arguments 
put forward here. A better analogy is this:

- Right now, we have a fence around the neighbourhood swimming pool. This fence 
has a gate with a guard, and the guard prevents 6 year olds from getting 
through the gate.
- Kids can climb the fence (by using random.choice or something like it). The 
security community is mostly in agreement with the stdlib folks: we’re happy to 
say that this problem is best dealt with by educating children to not climb the 
fence (don’t use random.choice in a security context).
- In Python 3.4 and earlier, the guard on this gate will, in some 
circumstances, turn up to work drunk. The guard is very good at pretending to 
be sober, so you cannot tell just by looking that he’s drunk, but in this state 
he will let anyone through the gate. He sobers up fast, so it only matters if a 
child tries to get in very shortly after you open the swimming pool.
- In Python 3.5 we included a patch that, by accident, installed a breathalyser 
on the gate. Now the guard can only open the gate when he’s sober.
- The problem is that he cannot open the gate *for anyone* while he’s drunk. 
All entry to the pool stops if he shows up drunk.
- The security folks want to say “yes, this breathalyser is awesome, leave it 
in place, it should always have been there”.
- The compat folks want to say “the gate hasn’t had a breathalyser for years, 
and it represents a genuine inconvenience to adults who want to swim, so we 
need to remove it”.

We are not trying to take non-CSPRNGs away from you. We are not trying to 
remove the random module, we are not trying to say that everyone must use 
urandom for all cases. We totally agree with the consenting adults policy. We 
just believe that the number of people who have used os.urandom and actively 
wanted the Linux behaviour may well be zero, and is certainly a tiny fraction 
of the user base of os.urandom, whereas the number of people who have used 
os.urandom and expected it to produce safe random bytes is dramatically larger.

We believe that invisible risks are bad. We believe that it is difficult to 
meaningfully consent to behaviour you do not know about. And we believe that 
when your expectations are violated, it is better for the system to fail hard 
than to subtly degrade to a behaviour that puts you at risk, because one of 
these provides a trigger for action and one does not.

In the case of ‘consenting adults’: users cannot be said to have meaningfully 
consented to behaviours they do not understand. Consider urllib2 and PEP 476. 
Prior to Python 2.7.9, urllib2 did not validate TLS certificates. It *could*, 
if a user was willing to configure it to do so, but by default it did not. We 
could defend that behaviour under ‘consenting adults’: users *technically* 
consented to the behaviour by using the code. However, most of these users 
*passively* consented: their consent is inferred by their use of the API, but 
they have written no code that actively asserts their consent.

In Requests, we allow consenting adults to turn off cert verification if they 
want to, but they have to *actively* consent: they *have* to say verify=False. 
And they have to say it every time: we deliberately provide no global switch to 
turn off cert validation in Requests, you have to set verify=False every single 
time. This is very deliberate. If a user wants to shoot themselves in the foot 
they are welcome to do so, but we don’t hand the gun to the user pointed at 
their foot.

We’re arguing for the same here with os.urandom(). If you want the Linux 
default urandom behaviour, that’s fine, but we think that it’s surprising to 
people and that they should be forced to *actively ask* for that behaviour, 
rather than passively be given it.

The TL;DR is: consent is not the absence of saying no, it’s the presence of 
saying yes.

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 16 Jun 2016, at 09:19, Stefan Krah  wrote:
> 
> This should only concern code that a) was specifically written for
> 3.5.0/3.5.1 and b) implements a serious cryptographic application
> in Python.
> 
> I think b) is not a good idea anyway due to timing and side channel
> attacks and the lack of secure wiping of memory. Such applications
> should be written in C, where one does not have to predict the
> behavior of multiple layers of abstractions.

No, it concerns code that generates its random numbers from Python. For 
example, you may want to use AES GCM to encrypt a file at rest. AES GCM 
requires the use of an nonce, and has only one rule about this nonce: you MUST 
NOT, under any circumstances, re-use an nonce/key combination. If you do, AES 
GCM fails catastrophically (I cannot emphasise this enough, re-using a 
nonce/key combination in AES GCM totally destroys all the properties the 
algorithm provides)[0].

You can use a C implementation of all of the AES logic, including offload to 
your x86 CPU with its fancy AES GCM instruction set. However, you *need* to 
provide an nonce: AES GCM can’t magically guess what it is, and it needs to be 
communicated in some way for the decryption[1]. In situations where you do not 
have an easily available nonce (you do have it for TLS, for example), you will 
need to provide one, and the logical and obvious thing to do is to use a random 
number. Your Python application needs to obtain that random number, and the 
safest way to do it is via os.urandom().

This is the problem with this argument: we cannot wave our hands and say 
“os.urandom can be as unsafe as we want because crypto code must not be written 
in Python”. Even if we never implement an algorithm in Python (and I agree with 
you that crypto primitives in general should not be implemented in Python for 
the exact reasons you suggest), most algorithms require the ability to be 
provided with good random numbers by their callers. As long as crypto 
algorithms require good nonces, Python needs access to a secure CSPRNG. Kernel 
CSPRNGs are *strongly* favoured for many reasons that I won’t go into here, so 
os.urandom is our winner.

python-dev cannot wash its hands of the security decision here. As I’ve said 
many times, I’m pleased to see the decision makers have not done that: while I 
don’t agree with their decision, I totally respect that it was theirs to make, 
and they made it with all of the facts.

Cory


[0]: Someone will *inevitably* point out that other algorithms resist nonce 
misuse somewhat better than this. While that’s true, it’s a) not relevant, 
because some standards require use of the non-NMR algorithms, and b) unhelpful, 
because even if we could switch, we’d need access to the better primitives, 
which we don’t have.

[1]: Again, to head off some questions at the pass: the reason nonces are 
usually provided by the user of the algorithm is that sometimes they’re 
generated semi-deterministically. For example, TLS generates a unique key for 
each session (again, requiring randomness, but that’s neither here nor there), 
and so TLS can use deterministic *but non-repeated* nonces, which in practice 
it derives from record numbers. Because you have two options (re-use keys with 
random nonces, or random keys with deterministic nonces), a generic algorithm 
implementation does not constrain your choice of nonce.



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 12 Jun 2016, at 14:43, Theodore Ts'o  wrote:
> 
> Well, it can only happen on Linux because you insist on falling back
> to /dev/urandom --- and because other OS's have the good taste not to
> use systemd and/or Python very early in the boot process.  If someone
> tried to run a python script in early FreeBSD init scripts, it would
> block just as you were seeing on Linux --- you just haven't seen that
> yet, because arguably the FreeBSD developers have better taste in
> their choice of init scripts than Red Hat and Debian.  :-)

Heh, yes, so to be clear, I said “this can only happen on Linux” because I’m 
talking about the world that we live in: the one where I lost this debate. =D

Certainly right now the codebase as it stands could encounter the same problems 
on FreeBSD. That’s a problem for Python to deal with.

> So the question is whether I should do what FreeBSD did, which will
> statisfy those people who are freaking out and whinging about how
> Linux could allow stupidly written or deployed Python scripts get
> cryptographically insecure bytes, by removing that option from Python
> developers.  Or should I remove that one line from changes in the
> random.git patch series, and allow /dev/urandom to be used even when
> it might be insecure, so as to satisfy all of the people who are
> freaking out and whinging about the fact that a stupildly written
> and/or deployed Python script might block during early boot and hang a
> system?
> 
> Note that I've tried to do what I can to make the time that
> /dev/urandom might block as small as possible, but at the end of the
> day, there is still the question of whether I should remove the choice
> re: blocking from userspace, ala FreeBSD, or not.  And either way,
> some number of people will be whinging and freaking out.  Which is why
> I completely sympathetic to how Guido might be getting a little
> exasperated over this whole thread.  :-)

I don’t know that we need to talk about removing the choice. I understand the 
desire to commit to backwards compatibility, of course I do. My problem with 
/dev/urandom is not that it *exists*, per se: all kinds of stupid stuff exists 
for the sake of backward compatibility.

My problem with /dev/urandom is that it’s a trap, lying in wait for someone who 
doesn’t know enough about the problem they’re solving to step into it. And it’s 
the worst kind of trap: it’s one you don’t know you’ve stepped in. Nothing 
about the failure mode of /dev/urandom is obvious. Worse, well-written apps 
that try their best to do the right thing can still step into that failure mode 
if they’re run in a situation that they weren’t expecting (e.g. on an embedded 
device without hardware RNG or early in the boot process).

So my real problem with /dev/urandom is that the man page doesn’t say, in 
gigantic letters, “this device has a really nasty failure mode that you cannot 
possibly detect by just running the code in the dangerous mode”. It’s 
understandable to have insecure weak stuff available to users: Python has loads 
of it. But where possible, the documentation marks it as such. It’d be good to 
have /dev/urandom’s man page say “hey, by the way, you almost certainly don’t 
want this: try using getrandom() instead”.

Anyway, regarding changing the behaviour of /dev/urandom: as you’ve correctly 
highlighted, at this point you’re damned if you do and damned if you don’t. If 
you don’t change, you’ll forever have people like me saying that /dev/urandom 
is dangerous, and that its behaviour in the unseeded/poorly-seeded state is a 
misfeature. I trust you’ll understand when I tell you that that opinion has 
nothing to do with *you* or the Linux kernel maintainership. This is all about 
the way software security evolves: things that used to be ok start to become 
not ok over time. We learn, we improve.

Of course, if you do change the behaviour, you’ll rightly have programmers 
stumble onto this exact problem. They’ll be unhappy too. And the worst part of 
all of this is that neither side of that debate is *wrong*: they just 
prioritise different things. Guido, Larry, and friends aren’t wrong, any more 
than I am: we just rate the different concerns differently. That’s fine: after 
all, it’s probably why Guido invented and maintains an extremely popular 
programming language and I haven’t and never will! I have absolutely no problem 
with breaking “working” code if I believe that that code is exposing users to 
risks they aren’t aware of (you can check my OSS record to prove it, and I’m 
happy to provide references).

The best advice I can give anyone in this debate, on either side, is to make 
decisions that you can live with. Consider the consequences, consider the 
promises you’ve made to users, and then do what you think is right. Guido and 
Larry have decided to go with backward-compatibility: fine. They’re 
responsible, the buck stops with them, they know that. The same is true for 
you, Ted, with the 

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 12 Jun 2016, at 07:11, Theodore Ts'o  wrote:
> 
> On Sat, Jun 11, 2016 at 05:46:29PM -0400, Donald Stufft wrote:
>> 
>> It was a RaspberryPI that ran a shell script on boot that called
>> ssh-keygen.  That shell script could have just as easily been a
>> Python script that called os.urandom via
>> https://github.com/sybrenstuvel/python-rsa instead of a shell script
>> that called ssh-keygen.
> 
> So I'm going to argue that the primary bug was in the how the systemd
> init scripts were configured.  In generally, creating keypairs at boot
> time is just a bad idea.  They should be created lazily, in a
> just-in-time paradigm.

Agreed. I hope that if there is only one thing every participant has learned 
from this (extremely painful for all concerned) discussion, it’s that doing 
anything that requires really good random numbers should be delayed as long as 
possible on all systems, and should absolutely not be done during the boot 
process on Linux. Don’t generate key pairs, don’t make TLS connections, just 
don’t perform any action that requires really good randomness at all.

> So some people will freak out when the keygen systemd unit hangs,
> blocking the boot --- and other people will freak out of the systemd
> unit doesn't hang, and you get predictable SSH keys --- and some wiser
> folks will be asking the question, why the *heck* is it not
> openssh/systemd's fault for trying to generate keys this early,
> instead of after the first time sshd needs host ssh keys?  If you wait
> until the first time the host ssh keys are needed, then the system is
> fully booted, so it's likely that the entropy will be collected -- and
> even if it isn't, networking will already be brought up, and the
> system will be in multi-user mode, so entropy will be collected very
> quickly.

As far as I know we still only have three programs that were encountering this 
problem: Debian’s autopkgtest (which patched with PYTHONHASHSEED=0), 
systemd-cron (which is moving from Python to Rust anyway), and cloud-init (not 
formally reported but mentioned to me by a third-party). It remains unclear to 
me why the systemd-cron service files can’t simply request to be delayed until 
the kernel CSPRNG is seeded: I guess systemd doesn’t have any way to express 
that constraint? Perhaps it should.

Of this set, only cloud-init worries me, and it worries me for the *opposite* 
reason that Guido and Larry are worried. Guido and Larry are worried that 
programs like cloud-init will be delayed by two minutes while they wait for 
entropy: that’s an understandable concern. I’m much more worried that programs 
like cloud-init may attempt to establish TLS connections or create keys during 
this two minute window, leaving them staring down the possibility of performing 
“secure” actions with insecure keys.

This is why I advocate, like Donald does, for having *some* tool in Python that 
allows Python programs to crash if they attempt to generate cryptographically 
secure random bytes on a system that is incapable of providing them (which, in 
practice, can only happen on Linux systems). I don’t care how it’s spelled, I 
just care that programs that want to use a properly-seeded CSPRNG can error out 
effectively when one is not available. That allows us to ensure that Python 
programs that want to do TLS or build key pairs correctly refuse to do so when 
used in this state, *and* that they provide a clearly debuggable reason for why 
they refused. That allows the savvy application developers that Ted talked 
about to make their own decisions about whether their rapid startup is 
sufficiently important to take the risk.

Cory


[0]: 
https://github.com/systemd-cron/systemd-cron/issues/43#issuecomment-160343989



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 11 Jun 2016, at 09:24, Larry Hastings  wrote:
> Only Linux and OS X have never-blocking /dev/urandom.  On Linux, you can 
> choose to block by calling getrandom().  On OS X you have no choice, you can 
> only use the never-blocking /dev/urandom.  (OS X also has a /dev/random but 
> it behaves identically to /dev/urandom.)  OS X's man page reassuringly claims 
> blocking is never necessary; the blogosphere disagrees.
> If I were writing the function for the secrets module, I'd write it like you 
> have above: call os.getrandom() if it's present, and os.urandom() if it 
> isn't.  I believe that achieves current-best-practice everywhere: it does the 
> right thing on Linux, it does the right thing on Solaris, it does the right 
> thing on all the other OSes where reading from /dev/urandom can block, and it 
> uses the only facility available to us on OS X.

Sorry Larry, but as far as I know this is misleading (it’s not *wrong*, but it 
suggests that OS X’s /dev/urandom is the same as Linux’s, which is emphatically 
not true).

I’ve found the discussion around OS X’s random devices to be weirdly abstract, 
given that the source code for it is public, so I went and took a look. My 
initial reading of it (and, to be clear, this is a high-level read of a 
codebase I don’t know well, so please take this with the grain of salt that is 
intended) is that the operating system literally will not boot without at least 
128 bits of entropy to read from the EFI boot loader. In the absence of 128 
bits of entropy the kernel will panic, rather than continue to boot.

Generally speaking that entropy will come from RDRAND, given the restrictions 
on where OS X can be run (Intel CPUs for real OS X, virtualised on top of OS X, 
and so on top of Intel CPUs, for VMs), which imposes a baseline on the quality 
of the entropy you can get. Assuming that OS X is being run in a manner that is 
acceptable from the perspective of its license agreement (and we can all agree 
that no-one would violate the terms of OS X’s license agreement, right?), I 
think it’s reasonable to assume that OS X, either virtualised or not, is 
getting 128 bits of somewhat sensible entropy from the boot loader/CPU before 
it boots.

That means we can say this about OS X’s /dev/urandom: the reason it never 
blocks is because the situation of “not enough entropy to generate good random 
numbers” is synonymous with “not enough entropy to boot the OS”. So maybe we 
can stop casting aspersions on OS X’s RNG now.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 9 Jun 2016, at 14:48, Doug Hellmann  wrote:
> 
> I agree those are the two options. I want the application developer to make 
> the choice, not us.

Right, but right now those two options aren’t available: only one of them is. 
And one way or another we’re taking an action here: either we’re leaving 
os.urandom() as it stands now, or reverting it back to the way it was in 3.4.0.

This means that you *do* want python-dev to make a choice: specifically, you 
want python-dev to make the choice that was made in 3.4.0, rather than the one 
that was made in 3.5.0. That’s fine, but we shouldn’t be pretending that either 
side is arguing for inaction or the status quo for Python 3.5 a choice was made 
with insufficient knowledge of the outcomes, and now we’re arguing about 
whether we can revert that choice. The difference is, now we *do* know about 
both outcomes, which means we are consciously choosing between them.

> All of which fails to be backwards compatible (new exceptions and hanging 
> behavior), which means you’re breaking apps.

Backwards compatible with what? Python 3.5.0 and 3.5.1 both have this 
behaviour, so I assume you mean “backward compatible with 3.4”. However, part 
of the point of a major release is that it doesn’t have to be backward 
compatible in this manner: Python breaks backward compatibility all the time in 
major releases.

I should point out that as far as I'm aware there are exactly two applications 
that suffer from this problem. One of them is Debian’s autopkgtest, which has 
resolved this problem by invoking Python with PYTHONHASHSEED=0. The other is 
systemd-cron, and frankly it does not seem at all unreasonable to suggest that 
perhaps systemd-cron should *maybe* hold off until the system’s CSPRNG gets 
seeded before it starts executing.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 9 Jun 2016, at 13:53, Doug Hellmann  wrote:
> 
> I agree with David. We shouldn't break existing behavior in a way
> that might lead to someone else's software being unusable.

What does ‘usable’ mean? Does it mean “the code must execute from beginning to 
end”? Or does it mean “the code must maintain the expected invariants”? If it’s 
the second, what reasonably counts as “the expected invariants”?

The problem here is that both definitions of ‘broken’ are unclear. If we leave 
os.urandom() as it is, there is a small-but-nonzero change that your program 
will hang, potentially indefinitely. If we change it back, there is a 
small-but-nonzero chance your program will generate you bad random numbers.

If we assume, for a moment, that os.urandom() doesn’t get called during Python 
startup (that is that we adopt Christian’s approach to deal with random and 
SipHash as separate concerns), what we’ve boiled down to is: your application 
called os.urandom() so early that you’ve got weak random numbers, does it hang 
or proceed? Those are literally our two options.

These two options can be described a different way. If you didn’t actually need 
strong random numbers but were affected by the hang, that program failed 
obviously, and it failed closed. You *will* notice that your program didn’t 
start up, you’ll investigate, and you’ll take action. On the other hand, if you 
need strong random numbers but were affected by os.urandom() returning bad 
random numbers, you almost certainly will *not* notice, and your program will 
have failed *open*: that is, you are exposed to a security risk, and you have 
no way to be alerted to that fact.

For my part, I think the first failure mode is *vastly* better than the second, 
even if the first failure mode affects vastly more people than the second one 
does. Failing early, obviously, and safely is IMO much, much better than 
failing late, silently, and dangerously.

I’d argue that all the security disagreements that happen in this list boil 
down to weighting that differently. For my part, I want code that expects to be 
used in a secure context to fail *as loudly as possible* if it is unable to 
operate securely. And for that reason:

> Adding a new API that does block allows anyone to call that when
> they want guaranteed random values, and the decision about whether
> to block or not can be placed in the application developer's hands.

I’d rather flip this around. Add a new API that *does not* block. Right now, 
os.urandom() is trying to fill two niches, one of which is security focused. 
I’d much rather decide that os.urandom() is the secure API and fail as loudly 
as possible when people are using it insecurely than to decide that 
os.urandom() is the *insecure* API and require changes.

This is because, again, people very rarely notice this kind of new API 
introduction unless their code explodes when they migrate. If you think you can 
find a way to blow up the secure crypto code only, I’m willing to have that 
patch too, but otherwise I really think that those who expect this code to be 
safe should be prioritised over those who expect it to be 100% available.

My ideal solution: change os.urandom() to throw an exception if the kernel 
CSPRNG is not seeded, and add a new function for saying you don’t care if the 
CSPRNG isn’t seeded, with all the appropriate “don’t use this unless you’re 
sure” warnings on it.

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 9 Jun 2016, at 12:54, Christian Heimes  wrote:
> 
> Therefore I propose to fix problem 2 and 3:
> 
> - add a new random_seed member to _Py_HashSecret and use it to derive an
> initial Mersenne-Twister state for the default random instance of the
> random module.
> 
> - try CPRNG for _Py_HashSecret first, fall back to a user space RNG when
> the Kernel's CPRNG would block.
> 
> For some operating systems like Windows and OSX, we can assume that
> Kernel CPRNG is always available. For Linux we can use getrandom() in
> non-blocking mode and handle EWOULDBLOCK. On BSD the seed state can be
> queried from /proc.
> 

I am in agreement with Christian here.

Let me add: Larry has suggested that it’s ok that os.urandom() can degrade to 
weak random numbers in part because "os.urandom() doesn't actually guarantee 
it's suitable for cryptography.” That’s true, that is what the documentation 
says.

However, that documentation has been emphatically disagreed with by the entire 
Python ecosystem *including* the Python standard library. Both 
random.SystemRandom and the secrets module use os.urandom() to generate their 
random numbers. The secrets module says this right at the top:

"The secrets module is used for generating cryptographically strong random 
numbers suitable for managing data such as passwords, account authentication, 
security tokens, and related secrets.”

Regressing the behaviour in os.urandom() would mean that this statement is not 
unequivocally true but only situationally true. It would be more accurate to 
say “The secrets module should generate cryptographically strong random numbers 
most of the time”. So I’d argue that while os.urandom() does not make these 
promises, the rest of the standard library behaves like it does. While we’re 
here I should note that the cryptography project unequivocally recommends 
os.urandom[0], and that this aspect of Linux’s /dev/urandom behaviour is 
considered to be a dangerous misfeature by almost everyone in the crypto 
community.

The Linux kernel can’t change this stuff easily because they mustn’t break 
userspace. Python *is* userspace, we can do what we like, and we should be 
aiming to make sure that doing the obvious thing in Python amounts to doing the 
*right* thing. *Obviously* this shouldn’t block startup, and obviously we 
should fix that, but I disagree that we should be reverting the change to 
os.urandom().

Cory

[0]: https://cryptography.io/en/latest/random-numbers/



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New OpenSSL - has anyone ever looked at (in)compatibility with LibreSSL

[Cory Benfield]

> On 15 Mar 2016, at 01:08, Jim Baker  wrote:
> 
> I have no vested interest in this, other than the continuing work we have 
> done to make Jython compatible with OpenSSL's model, warts and all.
> 
> But the fact that BoringSSL cleans up the OpenSSL API 
> (https://boringssl.googlesource.com/boringssl/+/HEAD/PORTING.md), at the cost 
> of possible backwards breaking API changes looks reasonable. I suppose there 
> is some risk - perhaps the maintainers will decide that returning 1 should 
> mean OK, but that's not going to happen, is it. The real issue here is that 
> no direct exposure of BoringSSL to other packages. I don't think that happens 
> with CPython. (Ironically it happens with Jython, due to how signed jars 
> poorly interact with shading/Java namespace remapping.)
> 
> Maintaining security means dealing with the inevitable churn. Did I mention 
> Jython's support of Python-compatible SSL? I think I did :p

It is *possible* to support BoringSSL: curl does. However, the BoringSSL 
developers *really* only target Chromium when they consider the possibility of 
breakage, so it costs curl quite a bit of development time[0]. curl accepts 
that cost because it supports every TLS stack under the sun: given that CPython 
currently supports exactly one, widening it to two is a very big risk indeed.

Cory


[0]: See https://github.com/curl/curl/issues/275, 
https://github.com/curl/curl/pull/524, https://github.com/curl/curl/pull/640



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 493: HTTPS verification migration tools for Python 2.7

[Cory Benfield]

> On 24 Feb 2016, at 12:19, M.-A. Lemburg <m...@egenix.com> wrote:
> 
> On 24.02.2016 12:28, Cory Benfield wrote:
>> 
>>> On 24 Feb 2016, at 10:32, Nick Coghlan <ncogh...@gmail.com> wrote:
>>> 
>>> Security Considerations
>>> ---
>>> 
>>> Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11, this
>>> approach does introduce a new downgrade attack against the default security
>>> settings that potentially allows a sufficiently determined attacker to 
>>> revert
>>> Python to the default behaviour used in CPython 2.7.8 and earlier releases.
>>> However, such an attack requires the ability to modify the execution
>>> environment of a Python process prior to the import of the ``ssl`` module,
>>> and any attacker with such access would already be able to modify the
>>> behaviour of the underlying OpenSSL implementation.
>>> 
>> 
>> I’m not entirely sure this is accurate. Specifically, an attacker that is 
>> able to set environment variables but nothing else (no filesystem access) 
>> would be able to disable hostname validation. To my knowledge this is the 
>> only environment variable that could be set that would do that.
> 
> An attacker with access to the OS environment of a process would
> be able to do lots of things. I think disabling certificate checks
> is not one of the highest ranked attack vectors you'd use, given
> such capabilities :-)
> 
> Think of LD_PRELOAD attacks, LD_LIBRARY_PATH manipulations, shell PATH
> manipulations (think spawned processes), compiler flag manipulations
> (think "pip install sourcepkg"), OpenSSL reconfiguration, etc.
> 
> Probably much easier than an active attack would be to simply extract
> sensitive information from the environ and use this for more direct
> attacks, e.g. accessing databases, payment services, etc.

To be clear, I’m not suggesting that this represents a reason not to do any of 
this, just that we should not suggest that there is no risk here: there is, and 
it is a new attack vector.

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 493: HTTPS verification migration tools for Python 2.7

[Cory Benfield]

> On 24 Feb 2016, at 10:32, Nick Coghlan  wrote:
> 
> Security Considerations
> ---
> 
> Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11, this
> approach does introduce a new downgrade attack against the default security
> settings that potentially allows a sufficiently determined attacker to revert
> Python to the default behaviour used in CPython 2.7.8 and earlier releases.
> However, such an attack requires the ability to modify the execution
> environment of a Python process prior to the import of the ``ssl`` module,
> and any attacker with such access would already be able to modify the
> behaviour of the underlying OpenSSL implementation.
> 

I’m not entirely sure this is accurate. Specifically, an attacker that is able 
to set environment variables but nothing else (no filesystem access) would be 
able to disable hostname validation. To my knowledge this is the only 
environment variable that could be set that would do that.

It’s just worth noting here that this potentially opens a little crack in 
Python’s armour.

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] dynamic linking, libssl.1.0.0.dylib, libcrypto.1.0.0.dylib and Mac OS X

[Cory Benfield]

> On 24 Dec 2015, at 11:17, Chris Withers  wrote:
> 
> Hi All,
> 
> Here's a couple of examples of this problem in the wild:
> 
> https://github.com/alekstorm/backports.ssl/issues/9
> http://stackoverflow.com/questions/32978365/how-do-i-run-psycopg2-on-el-capitan-without-hitting-a-libssl-error
> https://github.com/psycopg/psycopg2/issues/385
> 
> I'm well out of my depth here, I just want to use these libraries, but I'm 
> happy to try and do the work to make the world a better place for Mac users 
> of these libraries...
> 
> Chris

Chris,

I think this is actually nothing to do with Python itself, and everything to do 
with Mac OS X and the neutered way it ships OpenSSL. Given that the library 
you’re actually having difficulty with is cryptography, I recommend using their 
mailing list[0] to ask your question again. I happen to know that there have 
been a few problems with OS X and OpenSSL since El Capitan, so you’re probably 
not the first to encounter them.

Cory

[0]: https://mail.python.org/mailman/listinfo/cryptography-dev


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Cory Benfield]

> On 27 Nov 2015, at 06:04, Nick Coghlan  wrote:
> 
> Feature: Configuration API
> ==
> 
> This change is proposed for inclusion in CPython 2.7.12 and later CPython 
> 2.7.x
> releases. It consists of a new ``ssl._verify_https_certificates()`` to specify
> the default handling of HTTPS certificates in standard library client 
> libraries.
> 
> It is not proposed to forward port this change to Python 3, so Python 3
> applications that need to support skipping certificate verification will still
> need to define their own suitable security context.
> 
> Feature detection
> -
> 
> The marker attribute on the ``ssl`` module related to this feature is the
> ``ssl._verify_https_certificates`` function itself.
> 
> Specification
> -
> 
> The ``ssl._verify_https_certificates`` function will work as follows::
> 
>def _verify_https_certificates(enable=True):
>"""Verify server HTTPS certificates by default?"""
>global _create_default_https_context
>if enable:
>_create_default_https_context = create_default_context
>else:
>_create_default_https_context = _create_unverified_context
> 
> If called without arguments, or with ``enable`` set to a true value, then
> standard library client modules will subsequently verify HTTPS
> certificates by default, otherwise they will skip verification.

Perhaps I missed this, Nick, but what happens if multiple third party libraries 
apply updates to call this function in incompatible ways? For example, if you 
depend on libfoo which calls ssl._verify_https_certificates(False) and libbar 
which calls ssl._verify_https_certificates(True)? Is it…last import wins?


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Cory Benfield]

> On 24 Nov 2015, at 12:53, Christian Heimes  wrote:
> Right, with Antoine and Alex out of scope and you, Victor and me working
> for Red Hat, the air is getting thin. Benjamin is familiar with the ssl
> module. Or we can follow Alex's advice and ask somebody from the PyCA
> group (Donald, Paul, lvh) or requests (Cory) to get some outside
> perspective.

I’m not a CPython core developer, so I think that excludes me.

For what it’s worth, I think that this is a good PEP, and I’m strongly in 
favour of it being approved in some form.


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability

[Cory Benfield]

> On 19 Nov 2015, at 03:53, Ben Bangert  wrote:
> 
> In Python 2 and 3, the ssl module's SSLContext object has a way to set
> SSL options, but not to set SSL modes.
> 
> The set_mode command and some of the available modes:
> https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html
> 
> The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the
> SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl
> library allows the setting of SSLContext modes, it seems very odd that
> the Python 2/3 ssl modules do not. Though I could understand that
> perhaps not all SSL libraries Python might build against would have
> this mode thing available.
> 

Ben,

Do we need the ability to set arbitrary modes? Most of the modes mentioned in 
the OpenSSL documentation are things we actively don’t want the user to set 
because stuff will randomly break. With that in mind, and with the fact that 
SSL_MODE_RELEASE_BUFFERS is so obviously better than the standard, should we 
just instead have the ssl module automatically set SSL_MODE_RELEASE_BUFFERS 
unconditionally?

If so, I’m happy to submit a bug/patch to get that to happen.

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability

[Cory Benfield]

> On 19 Nov 2015, at 15:26, Ben Bangert  wrote:
> 
> I can't think of any other mode to set, setting this with the
> condition cited for that vulnerability looks like a good idea.
> 
> Cheers,
> Ben

Ok, we’re agreed. The work can be tracked under Issue 25672: 
https://bugs.python.org/issue25672

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] typeshed for 3rd party packages

[Cory Benfield]

On 24 April 2015 at 15:21, Steven D'Aprano st...@pearwood.info wrote:

 If the type hints are wrong, there are two errors: false positives, when
 code which should be allowed is flagged as a type error; and false
 negatives, when code which should be flagged as an error is not.
 Ideally, there should be no false positives. But false negatives are not
 so important, since you will still be doing runtime checks. All that
 means is that the static type-checker will be a little less capable of
 picking up type errors at compile time.

I think that's a rational view that will not be shared as widely as I'd like.

Given that the purpose of a type checker is to catch bugs caused by
passing incorrectly typed objects to a function, it seems entirely
reasonable to me to raise a bug against a type hint that allows code
that was of an incorrect type where that incorrectness *could* have
been caught by the type hint. Extending from that into the general
ratio of reports that are actually bugs versus reports that are
errors on the part of the reporter, I can assume that plenty of
people will raise bug reports for incorrect cases as well.

From the perspective of sustainable long-term maintenance, I think the
only way to do type hints is to have them be sufficiently exhaustive
that a user would have to actively *try* to hit an edge case false
negative. I believe that requests' API is too dynamically-typed to fit
into that category at this time.

PS: I should mention that, as Gary Bernhardt pointed out at PyCon,
people often believe (incorrectly) that types are a replacement for
tests. For that reason I feel like underspecified type hints are
something of an attractive nuisance. Again, I really think this is a
case of do it properly or not at all.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Cory Benfield]

On 21 April 2015 at 18:12, Steven D'Aprano st...@pearwood.info wrote:
 I expect that dealing with duck typing will be very high on the list
 of priorities for the future. In the meantime, for this specific use-case,
 you're probably not going to be able to statically check this type hint.
 Your choices would be:

 - don't type check anything;

To be clear, for the moment this is what requests will do, unless the
other maintainers strongly disagree with me (they don't). I am quite
convinced that PEP 484 is insufficiently powerful to make it
worthwhile for requests to provide 'official' type hints.

I suspect someone will provide hints to typeshed, and I certainly hope
they're good, because if they're bad we'll definitely field bug
reports about them (more on this in a different thread I think).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Cory Benfield]

On 21 April 2015 at 17:59, Guido van Rossum gu...@python.org wrote:
 For me, PEP 484 is a stepping stone. Among the authors of PEP 484 there was
 much discussion about duck typing, and mypy even has some limited support
 for duck typing (I think you can still find it by searching the mypy code
 for protocol). But we ran out of time getting all the details written up
 and agreed upon, so we decided to punt -- for now. But duck typing still
 needs to have a way to talk about things like seek method with this type
 signature (something like `def seek(self, offset: int, whence:
 int=SEEK_SET) - int`) so the current proposal gets us part of the way
 there.

 The hope is that once 3.5 is out (with PEP 484's typing.py included
 *provisional* mode) we can start working on the duck typing specification.
 The alternative would have been to wait until 3.6, but we didn't think that
 there would be much of an advantage to postponing the more basic type
 hinting syntax (it would be like refusing to include import until you've
 sorted out packages). During the run of 3.5 we'll hopefully get feedback on
 where duck typing is most needed and how to specify it -- valuable input
 that would be much harder to obtain of *no* part of the type hints notation
 were standardized.

This makes a lot of sense.

If PEP 484 is intended to be a stepping stone (or compromise, or beta,
or whatever word one wants to use), then it is easy to forgive it its
limitations, and I'm looking forward to seeing it improve.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] typeshed for 3rd party packages

[Cory Benfield]

On 22 April 2015 at 11:46, M.-A. Lemburg m...@egenix.com wrote:
 Unlike with translations, where missing or poor ones don't have
 much effect on the usefulness of the software, a type checker
 would complain loudly and probably show lots of false positives
 (if you read a type bug as positive), causing the usual complaints
 by users to the software authors.

 I don't really think that users would instead complain to the type
 checker authors or find the actual source of the problem which are
 the broken stub files.

This is my expectation as well.

Requests receives bug reports for bugs that were introduced by
downstream packagers, or for bugs that are outright in unrelated
projects. I field IRC questions about 'requests bugs' that are
actually bugs in the web application on the other end of the HTTP
connection! I can *guarantee* that if a stub file is bad, I'll get
told, not the author of the stub file.


 OTOH, if the type checkers are written in a way where they can
 detect authoritative stubs compared to non-authoritative ones
 and point users to the possible type stub file problem, this
 could be resolved, I guess.

 The stub files would then need an authoritative flag and
 probably also be versioned to get this working.

This would be great: +1.

 As I've explained above, in my experience, people (*) often first go
 to the authors of the software and not do research to find out
 that the tool they were using has a problem (via the non-authoritative
 stub files it's using).

 (*) More experienced users of pylint like tools will probably think
 twice due to the many false positives these tools tend to generate.
 I'm not sure whether people using type checkers would have the same
 approach, though, esp. not if they are coming from the land of
 statically typed languages.

I can back this up, as can a search through Requests' past GitHub
issues. pylint in particular has caused me pain due to at least one
GitHub issue that was nothing more than a dump of pylint output when
run over Requests', where *every single line* was a false positive.

This ends up having negative network effects as well, because the next
time someone opens a GitHub issue with the word 'pylint' in it I'm
simply going to close it, rather than waste another 45 minutes
visually confirming that every line is a false positive. I worry that
stub files will have the same problems.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Cory Benfield]

On 21 April 2015 at 01:45, Chris Angelico ros...@gmail.com wrote:
 When you're writing a library, it can be a great help to provide type
 annotations, because every application that uses your library can
 benefit.

It can be a great help to whom? Not to me (the library author),
because I can't use them in my library code, because I have to support
2.7. That's by no means a bad thing (after all, most libraries are
written to help others), but I found it really unclear who was being
advantaged here.

To show the downside, I decided to annotate requests' API a little
bit. Then I stopped, because I really couldn't work out how to do it.

For example, requests.get() takes a URL parameter. This can be an
instance of basestring, or anything that provides a __str__ or
__unicode__ method that provides us a string that looks like a URL
(such as a URL abstraction class). We don't know ahead of time what
those objects may be, so there's no type signature I can provide here
that is of any use beyond 'Any'.

We also provide headers/params options, whose type signature would be
(as far as I can tell) Union[Iterable[Tuple[basestring, basestring]],
Mapping[basestring, basestring]], which is a fairly unpleasant type
(and also limiting, as we totally accept two-element lists here as
well as two-tuples. That's got *nothing* on the type of the `files`
argument, which is the most incredibly polymorphic argument I've ever
seen: the best I can work out it would be:

Optional[
Union[
Mapping[
basestring,
Union[
Tuple[basestring, Optional[Union[basestring, file]]],
Tuple[basestring, Optional[Union[basestring, file]],
Optional[basestring]],
Tuple[basestring, Optional[Union[basestring, file]],
Optional[basestring], Optional[Headers]]
]
],
Iterable[
Tuple[
basestring,
Union[
Tuple[basestring, Optional[Union[basestring, file]]],
Tuple[basestring, Optional[Union[basestring,
file]], Optional[basestring]],
Tuple[basestring, Optional[Union[basestring,
file]], Optional[basestring], Optional[Headers]]
]
]
]
]

Headers = Union[
Mapping[basestring, basestring],
Iterable[Tuple[basestring, basestring]],
]

This includes me factoring out the Headers type for my own sanity, and
does not include the fact that in practice almost all instances of
'basestring' above actually mean 'anything that can be safely cast to
basestring', and 'file' means 'any file-like object including BytesIO
and StringIO' (speaking of which, the PEP doesn't even begin to go
into how to handle that kind of requirement: do we have some kind of
Readable interface that can be implemented? Or do I have to manually
union together all types someone might want to use?).

Now, this can be somewhat improved: the three types of tuple (again,
not just tuples, often lists!) can be factored out, as can the
Union[basestring, file]. However, there's still really no way around
the fact that this is a seriously complex type signature!

Python has had years of people writing APIs that feel natural, even if
that means weird contortions around 'types'. Writing an argument as
'clever' as the 'files' argument in requests in a statically typed
language is an absolute nightmare, but Python makes it an easy and
natural thing to do.

Further, Python's type system is not sufficiently flexible to allow
library authors to adequately specify the types their code actually
works on. I need to be able to talk about interfaces, because
interfaces are the contract around which APIs are build in Python, but
I cannot do that with this system in a way that makes any sense at
all. To even begin to be useful for library authors this PEP would
need to allow some kind of type hierarchy that is *not* defined by
inheritance, but instead by interfaces. We've been discouraging use of
'type' and 'isinstance' for years because they break duck typing, but
that has *nothing* on what this PEP appears to do to duck typing.

I suppose the TLDR of this email is that I think that libraries with
'pythonic' APIs are the least likely to take up this typing system
because it will provide the least value to them. Maintaining
signatures will be a pain (stub files are necessary), writing clear
signatures will be a pain (see above), and writing signatures that are
both sufficiently open to be back-compatible and sufficiently
restrictive to be able to catch bugs seems like it might be very
nearly impossible.

I'd love for someone to tell me I'm wrong, though. I want to like this
PEP, I really do.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Cory Benfield]

On 21 April 2015 at 12:23, Gustavo Carneiro gjcarne...@gmail.com wrote:
 Documentation is not checked.  It often loses sync with the actual code.
 Docs say one thing, code does another.

Agreed. I don't think anyone would disagree here. I'm talking from the
position of being a library author, where supporting versions of
Python lower than 3.5 will be a reality for at least 5 more years. I
will not be able to inline my type hints, so they'll have to go in
stub files, and now we've got the same problem: type hints can go out
of step just as easily as documentation can.

 Documenting types in docstrings forces you to repeat yourself.  You have to
 write the parameter names twice, once in the function definition line,
 another in the docstring.  And you need to understand the syntax of whatever
 docstring format will be used to check the code.

Yes, but I'm repeating myself O(small integer) lines away from where I
first wrote it. The maintenance burden of keeping those things in step
is not very high. As for your last point, my code is not checked
against my docstrings, primarily for the reasons I mentioned in my
original email. My docstrings are merely rendered into documentation.

 I'm sure I'm not the only one that thinks that type hints actually *improve
 readability*.

In some cases, sure. No contest.

 but I am certain that most of the time the type hints make the
 code easier to read because you don't have to spend so much mental effort
 trying to figure out gee, I wonder if this parameter is supposed to be
 bool, string, or int?

Yes. If you spend a lot of your time doing this, then type hints will
help you, *assuming* the programmer who originally wrote the ambiguous
code diligently maintains the type hints. I have no reason to believe
that a programmer whose code is that ambiguous is going to do that, or
even write type hints at all.

The bigger problem is that, previously, Python functions never took
bool, string, or int. They took 'object that can be evaluated in a
boolean context', 'object that behaves like strings in some undefined
way', and 'objects that behave like ints in some undefined way'. The
type hint you provide is *not helpful*. What I *need* is a type hint
that says 'object that has the .endswith() method' or 'object that can
be safely cast to a string', so I can tell whether I can pass you a
custom URL class that happens to render into a textual URL or whether
it definitely has to be a string object. But you can't express that
relationship in this model, so you will just say 'string', and now I
will turn off type checking because the type hinter keeps shouting at
me for using my URLBuilder object.

If you only work with primitive types and only want your users to use
primitive types, this proposal almost certainly works brilliantly. If
you *don't*, this proposal begins to become extremely uncomfortable
extremely fast.

 Even if it only helps in the simplest cases, it saves the reader of the code
 a lot of effort if you add up all those little cases together.

 My suggestion, wherever an annotated type is too complex, don't annotate it.
 If you do /only/ the simple cases, the end result is easier to read.

If that's genuinely the bar for success for this PEP, then fine, I
wish it the best. I'd *like* a PEP that can deal with the complex
cases too, but I freely admit to having no idea how such a thing may
be implemented. All I can say is that I think this PEP will have
limited uptake amongst many third-party libraries, particularly the
complex ones.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Cory Benfield]

On 21 April 2015 at 10:10, Chris Angelico ros...@gmail.com wrote:
 At this point, you may want to just stop caring about the exact type.
 Part of the point of gradual typing is that you can short-cut a lot of
 this. And quite frankly, this isn't really helping anything. Just skip
 it and say that it's Union[Mapping, Iterable, None].

I think this is my problem with the proposal. This change doesn't
catch any of the bugs anyone was going to hit (no-one is passing a
callable to this parameter), and it doesn't catch any of the bugs
someone might actually hit (getting the tuple elements in the wrong
order, for example). At this point the type signature is worse than
useless.

It seems like the only place the type annotations will get used is in
relatively trivial cases where the types are obvious anyway. I don't
deny that *some* bugs will be caught, but I suspect they'll
overwhelmingly be crass ones that would have been caught quickly
anyway. The minute a type signature might actually help prevent a
subtle bug we can't use them anymore because the toolchain isn't
powerful enough to do it so we just put 'Any' and call it a day. And
even then we haven't helped the trivial case much because anyone who
wants to provide a duck-typed object that should be perfectly suitable
in this case no longer can.

What this proposal *needs* to be generally useful, IMO, is the ability
to specify types that actually match the way we have encouraged people
to write code: duck typed, favouring composition over inheritance,
etc. At the very least that means 'type constraints' (see Haskell,
Rust, really any language with a powerful and robust type system:
hell, even see Go's interfaces) need to be something we can specify.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Cory Benfield]

On 21 April 2015 at 15:31, Chris Angelico ros...@gmail.com wrote:
 Granted, there are some
 vague areas - how many functions take a file-like object, and are
 they all the same? - but between MyPy types and the abstract base
 types that already exist, there are plenty of ways to formalize duck
 typing.

Are there? Can I have a link or an example, please? I feel like I
don't know how I'm supposed to do this, and I'd like to see how that
works. I'll even give a concrete use-case: I want to be able to take a
file-like object that has a .read() method and a .seek() method.

 And frankly, even with the uncertainties, I'd still rather
 have a function declared as taking a file-like object than an
 object with a .read() method that takes an integer and returns up to
 that many bytes of data, and a .seek() method that blah blah blah
 blah. Sometimes, the precision is completely useless.

It is completely useless. Happily, this is a strawman, and no-one was
asking for it, so we can all live happily ever after!

The correct specification is read method with this type signature
and seek method with this type signature. I would even be prepared
to waive the type signatures on read and seek, given that enforcing
the type hinting on others is not a good idea.

I suspect I have a mismatch with several others in this discussion. My
position is that if I'm going to have a type system, I'd like to have
a powerful one: Haskell, not Java. Otherwise I'll get by with the duck
typing that has worked just fine for us over the last 20 years. I
suspect, however, that many others in this conversation want any type
system at all, so long as they can have one.

Is that an accurate characterisation of your position, Chris?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Cory Benfield]

On 21 April 2015 at 16:09, Chris Angelico ros...@gmail.com wrote:
 Pretty accurate, yeah. Here's how I see it:

 def incremental_parser(input: FileLike) - List[Token]:
 tokens = []
 data = 
 while True:
 if not data:
 data = input.read(64)
 token = Token(data[0]); data = data[1:]
 while token.wants_more():
 token.give_more(data[0]); data = data[1:]
 tokens.append(token)
 if token.is_end_of_stream(): break
 input.seek(-len(data), 1)
 return tokens

 If you were to exhaustively stipulate the requirements on the
 file-like object, you'd have to say:

 * Provides a read() method which takes an integer and returns up to
 that many bytes
 * Provides a seek() method which takes two integers
 * Is capable of relative seeking by at least 63 bytes backward
 * Is open for reading
 * Etcetera

 That's not the type system's job. Not in Python. Maybe in Haskell, but
 not in Python. So how much _should_ go into the type hint? I'm happy
 with FileLike or however it's to be spelled; maybe separate readable
 files from writable ones, as those are two fairly clear variants, but
 that's about all you really need. If you provide incremental_parser()
 with an input file that's non-seekable, it's going to have problems -
 and your editor may or may not even be able to detect that (some files
 are seekable but only in the forward direction, but they'll have the
 exact same seek() method).

Ok, that makes sense to me. =) Looking at mypy's source code, I see
shutil.copyfileobj has the following signature:

def copyfileobj(fsrc: IO[AnyStr], fdst: IO[AnyStr], length: int =
16*1024) - None:

so it seems that mypy defines a general IO datatype here.

I continue to be worried about the lack of a *general* solution to
this problem, but I'm glad that some specific solutions exist. I still
think library authors will not do much to take up this proposal. =)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] http.client: Determining Content-Length

[Cory Benfield]

On 31 March 2015 at 17:55, Demian Brecht demianbre...@gmail.com wrote:
 Hi all,

 I'm not sure whether this should be python-list or here, but given it's a 
 premature code review for http.client, I figured I'd post here first.

 I'm in the process of adding support for chunked transfer encoding to 
 http.client (issue #12319). One of the bits of functionality that I'm working 
 on in is ironing out some of the kinks out in determining the content length 
 of the request bodies. Given the number of data types allowed as bodies it 
 does get a little messy, so I wanted to get some feedback here and see if 
 anyone can shoot holes through it prior to updating the current patch. The 
 tests are passing, but there may be use cases not accounted for in the new 
 implementation.

 https://gist.github.com/demianbrecht/f94be5a51e32bb9c81e1

 The above is intended to replace _set_content_length (current state of the 
 patch can be seen here: 
 http://bugs.python.org/review/12319/diff/14331/Lib/http/client.py). There is 
 a comprehensive list of data types currently supported can be found here: 
 http://bugs.python.org/issue23740. Comments and feedback are much appreciated.

Nothing problematic leaps out at me here, but only if I look at your
proposal in the context of the full patch.

Your full patch does seem to have an edge case: for HTTP/1.1, if I
pass a generator and no content-length you'll blow up by indexing into
it. That's probably fine (your only option in this case would be to
frame this request by half-closing the TCP stream, which I seem to
recall you ruling out elsewhere), but you might want to wrap the
exception in something more helpful: not sure.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backwards compatibility after certificate autovalidation

[Cory Benfield]

On 8 September 2014 18:23, Jim J. Jewett jimjjew...@gmail.com wrote:
 Summary:  There needs to be a simple way to opt out at install time.
 It would be far better to offer more fine-grained control, but leaving
 that better solution to downstream is acceptable.

Does this argument apply to a hypothetical 2.7 backport of this
change, or does it apply to making the change in 3.5? (Or of course
both.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Cory Benfield]

On 3 September 2014 01:19, Antoine Pitrou solip...@pitrou.net wrote:
 Keeping saying it doesn't make it magically true.

Sure, but it *is* true, at the very least for HTTP.

RFC 2818 (HTTP over TLS) has the following language in section 3.1:

 If the hostname is available, the client MUST check it against the
 server's identity as presented in the server's Certificate message,
 in order to prevent man-in-the-middle attacks.

 If the client has external information as to the expected identity of
 the server, the hostname check MAY be omitted.

The default behaviour of httplib is in contravention of the normative
language of this specification and is therefore bugged. (For those
unclear about the relevance of RFC 2818, it's normatively referenced
by RFC 7230, which is the HTTP/1.1 spec.)

This should silence the debate about whether or not httplib's
behaviour is a bug or not.

 Besides, it can perfectly well be a bug fix *as well as* a break in
 backwards compatibility.

This is definitely true, and this change is both. The only question
that matters is whether we believe we're doing users a service by
breaking their code. I'd argue, along with Glyph, Alex and Donald,
that we are. I've been on the losing side of this debate a number of
times though, and I expect I will be again.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Cory Benfield]

On 31 August 2014 07:45, Nick Coghlan ncogh...@gmail.com wrote:
 There's also the fact that most corporate Python users are
 unlikely to know that PyPI exists, let alone that it contains a module
 called requests that does SSL certificate validation by default.
 Those of us in the corporate world that interact directly with
 upstream are still the exception rather than the rule)

I think this point deserves just a little bit more emphasis. This is
why any solution that begins with 'use PyPI' is insufficient. I've
worked on requests for 3 years now and most of my colleagues have
never heard of it, and it's not because I don't talk about it (I talk
about it all the time!).

When building internal tools, corporate environments frequently
restrict themselves to the standard library. This is because it's hard
enough to get adoption of a tool when it requires a new language
runtime, let alone if you have to get people ramped up on package
distribution as well! I have enough trouble getting people to upgrade
Python versions at work: trying to get them up to speed on pip and
PyPI is worse.

It is no longer tenable in the long term for Python to trust the
network: you're right in this regard Nick. In the past, on this very
list, I've been bullish about fixing up Python's network security
position. I was an aggressive supporter of PEP 466 (and there are some
corners of PEP 466 that I think didn't go far enough). However, I'm
with you here: we should do this once and do it right. Corporate users
*will* bump into it, and they will look to the docs to fix it. That
fix needs to be easy and painless.

A user-level cert store is a good start, and if cryptography.io aren't
interested in it I might take a look at implementing it under the
certifi.io umbrella instead.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 21671: CVE-2014-0224 OpenSSL upgrade to 1.0.1h on Windows required

[Cory Benfield]

On 17 June 2014 17:41, Yates, Andy (CS Houston, TX) aya...@hp.com wrote:
 Is it possible to drop in new OpenSSL versions
 on Windows without rebuilding Python?

If you think this is a problem you're going to have more than once,
you'll want to look  hard at whether it's worth using pyOpenSSL
(either the egenix version or the PyCA one[1]) instead, and delivering
binary releases with a bundled copy of OpenSSL. PyOpenSSL from PyCA is
actually considering bundling OpenSSL on Windows anyway[2], so you
might find this problem goes away.

[1] https://github.com/pyca/pyopenssl
[2] https://github.com/pyca/cryptography/issues/1121
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7

[Cory Benfield]

Nick,

On 26 March 2014 12:00, Nick Coghlan ncogh...@gmail.com wrote:
 As one example, the Python 2 ``ssl`` module does not support the Server
 Name Identification standard.

Tiny note: SNI is 'Server Name Indication', not 'Identification'. =)

Otherwise, I'm +1 on this as well.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements

[Cory Benfield]

On 25 March 2014 08:26, Chris Angelico ros...@gmail.com wrote:
 Exactly the same. If someone wants to distribute SEPython (that
 someone might be python.org itself, or ActiveState, or anyone else who
 has an interest in it), they're welcome to do so; and it could be done
 either as an all-in-one that packages all of CPython, or as an add-on;
 either way would work just as well, but the former would be cleaner.

Reading this I suspect we're mostly in agreement but having trouble
communicating. My understanding of your point is simply that you don't
want python-dev to 'bless' either of the 2.7 releases proposed as
_the_ 2.7 release, instead pushing that choice on to people
distributing Python. I can get behind that plan so long as the source
code releases are named in such a way that people are either a) forced
to make a choice; or b) defaulted to secure 2.7.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements

[Cory Benfield]

On 24 March 2014 19:37, Chris Angelico ros...@gmail.com wrote:
 The opting in could be done at the distro level. Red Hat could ship a
 thing called /usr/bin/python that runs SEPython, and that package
 could be identified and numbered in such a way that it's clearly a
 drop-in replacement for vanilla Python. If backward compatibility is
 done carefully (which, from everything I'm seeing here, is the way
 it's to be done), there should be absolutely no downside to using
 SEPython, except for portability (because now you're depending on it
 for security), and corner cases like testing.

What's your solution for OS X, Windows et al? My concern is that if
you have a release called 'Python' and a release called 'Python with
security stuff', a surprisingly large number of people will download
the first, especially if the notes for the security release say 'this
may cause some minor compatibility problems'. IMHO, I'd rather have
good security be the default for everyone, and require an explicit
opt-out to get the bad security release.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements

[Cory Benfield]

On 25 March 2014 09:01, Chris Angelico ros...@gmail.com wrote:
 So by that model, current 2.7 is fully compliant, and anything that
 doesn't actively conflict with that is also compliant. Any script that
 is written for the current 2.7 is guaranteed also to run on any
 compliant SEPython; and anything written for SEPython has to
 gracefully handle (which might mean cleanly bombing) anything down to
 and including current 2.7. Does that make sense?

Absolutely. =) My additional concern on top of that is wanting users
to fall into a pit of success by making it overwhelmingly more likely
that users will accidentally end up with the safe version if they
aren't paying attention. I'm not hugely bothered about how that's
done: I'd just like not to have to field Requests bug reports about
lack of security that boil down to a user having grabbed the insecure
version by accident.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements

[Cory Benfield]

On 24 March 2014 08:44, Nick Coghlan ncogh...@gmail.com wrote:
 The position I am coming to is that the enhanced security release should
 be the default one that we publish binary installers for, but we should also
 ensure that downstream redistributors can easily do Python 2.7 with legacy
 SSL releases if they so choose. I'm happier forcing end users to rely on a
 redistributor to opt in to a lower security option than I am to knowingly
 publish too many additional releases with network security infrastructure
 that is (at best) rapidly approaching its use by date.

I'm with you here Nick, the default should really be Python 2.7 with
working TLS, not Python 2.7 with broken TLS (my only-slightly
tongue-in-cheek proposed names for the releases). My concern with
having Chris' proposal of an opt-in SEPython release is about
discovery. I suspect most people will never find out about the
SEPython release, meaning that most people will remain on insecure
Python 2.7 distributions. If that happens, requests is essentially
required to continue to support our current TLS model for 2.7 for as
long as we support 2.7, and this PEP would give us no benefit at all.
We're obviously not the only stakeholder here, but it's worth noting.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements

[Cory Benfield]

On 23 March 2014 at 04:32:17, Terry Reedy 
(tjre...@udel.edu(mailto:tjre...@udel.edu)) wrote:
 Instead, I think the PEP should propose a special series of server
 enhancement releases that are based on the final 2.7 maintenance release
 (2.7.8 or 2.7.9) but which have have a different application-specific
 enhancement policy.

This is an interesting idea. My biggest problem with it is that, at least
with the ssl library, these aren’t server-only problems. If we suggest that
they are, we end up in the same position we’re in right now (that is, hurting
the internet).

For example, Python 2.7’s ssl module lacks the OP_NO_COMPRESSION option for
OpenSSL, meaning that the application is at the mercy of the server to determine
whether it’s vulnerable to the CRIME attack. Given that all modern browsers
already disable TLS compression, we can assume that lots of server admins 
haven’t
bothered disabling it on their end. This leaves pretty much anyone using HTTPS,
client or server, on Python 2.7 at risk of the CRIME attack. This isn’t a
server-only problem, so I feel like limiting the fixes to a ‘server’ release
is not good enough.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements

[Cory Benfield]

On 22 March 2014 at 21:11:24, Nick Coghlan 
(ncogh...@gmail.com(mailto:ncogh...@gmail.com)) wrote:

 Folks,
  
 I have just posted a proposal to change the way we treat enhancements
 that relate to Python's support for network security enhancements. I
 now see these as categorically different from most other enhancements,
 as they have implications for the evolution of internet security as a
 whole, rather than being limited to affecting the security of
 individual applications written in Python.

I am 100%, overwhelmingly in favour of this. Without this PEP, Python 2.7
is a security liability, any it becomes nothing short of irresponsible to
use Python 2.7 for any form of networking code that hits the open
internet.

On top of that, the current state of the ssl module means that Python 2.7
and earlier cannot ever support a standard-compliant implementation of
things like HTTP/2. That’s a fairly tragic state of affairs for 2.7,
especially if we’re supposed to claim with a straight face that it’s
acceptable to still use Python 2.7.

Treat this as my strong +1. Additionally, I’m happy to volunteer my time
and limited expertise to help make this happen. I’ll help work on
back porting things, review code, write docs: whatever it takes to get
this to happen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enable Hostname and Certificate Chain Validation

[Cory Benfield]

On 24 January 2014 03:06, Stephen J. Turnbull step...@xemacs.org wrote:
 Are you kidding?  These *aren't* the apps that I care about breaking,
 and I know that the PHBs won't pay attention to what I say about
 fixing their sites and cert chains (believe me, I've tried, and the
 answer is as Paul Moore says: the users can punch the go ahead anyway
 button, what's the big deal here?), they'll just deprecate Python.

Surely the solution here isn't to say well then, let's be insecure by
default, it's to provide a go ahead anyway button. That at least lets us
push the choice to be insecure by default onto someone else. The idea that
an enterprise will deprecate Python instead of adding a single environment
variable or one line at the top of their scripts seems hugely unlikely.

As an example, Requests provides a stop verifying certs button, and
that works fine for us. (I know that Requests is outside the stdlib and so
it's not a perfect analogy, but it's serviceable.) I suspect most people who
want this change don't care if users have an easy way to circumvent it,
we just want to have the user/enterprise make that choice for themselves.
Put another way, we want the average user to fall into a pit of success.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enable Hostname and Certificate Chain Validation

[Cory Benfield]

On 23 January 2014 08:37, Stephen J. Turnbull step...@xemacs.org wrote:

 I don't know what the right answer is, but this needs careful
 discussion and amelioration, not just you're broken, so take the
 consequences!

Absolutely. =)

With that said, having a great big button to turn the change off
(e.g. environment variable, global setting) feels like an acceptable
mitigation to me, but then I'm probably still more aggressive than the
core developers! Either way, it looks like the deprecation dance is
going to be the consensus decision here.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enable Hostname and Certificate Chain Validation

[Cory Benfield]

Donald Stufft donald at stufft.io writes:

 
 I would like to propose that a backwards incompatible change be
 made to Python to make verification of hostname and certificate
 chain the default instead of requiring it to be opt in.

I'm overwhelmingly, dramatically +1 on this. There's no good
architectural reason to not use the built-in certificate chains by
default. I'd like to be in favour of backporting this change to earlier
Python versions as well, but it feels too aggressive, even to me.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enable Hostname and Certificate Chain Validation

[Cory Benfield]

On 22 January 2014 10:30, Donald Stufft don...@stufft.io wrote:
 I would like to propose that a backwards incompatible change be
 made to Python to make verification of hostname and certificate
 chain the default instead of requiring it to be opt in.

I'm overwhelmingly, dramatically +1 on this. There's no good
architectural reason to not use the built-in certificate chains by
default. I'd like to be in favour of backporting this change to earlier
Python versions as well, but it feels just a bit too aggressive.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com