Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 01.09.2014 10:09, Nick Coghlan wrote: On 1 September 2014 17:13, Christian Heimes christ...@python.org wrote: On 01.09.2014 08:44, Nick Coghlan wrote: Yes, it would have exactly the same security failure modes as sitecustomize, except it would only fire if the application imported the ssl module. The -S and -I switches would need to disable the implied sslcustomize, just as they disable import site. A malicious package can already play havoc with your installation with a custom ssl module. If somebody is able to sneak in a ssl.py then you are screwed anyway. sslcustomize is not going to make the situation worse. That's not quite true - we're fairly careful about putting the standard library before userspace directories, so aside from the current directory problem, shadowing ssl itself can be tricky to arrange. It's really easy to modify sys.modules to override any module that has already been loaded or add new ones bypassing sys.path entirely, so the sys.path layout doesn't provide any protection against such hacks. If you gain access to one of the dirs on sys.path, you can play such tricks in sitecustomize.py. Any 3rd party package can do the same. We'd have to add digital API signatures to the ssl module to prevent such stunts :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 08 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ...11 days to go 2014-09-27: PyDDF Sprint 2014 ... 19 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Wed, Sep 3, 2014 at 3:48 PM, Stephen J. Turnbull step...@xemacs.org wrote: Guido van Rossum writes: lot: five years ago (when I worked at Google!) it was common to find internal services that required SSL but had a misconfigured certificate, and the only way to access those services was to override the browser complaints. Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- I would tend to discount your recent experience, then. Smaller (and possibly even more important in this fast-developing area, younger) organizations are a lot more nimble about things like this. As a defensive data point: I don't remember a single instance of this happening for Google internal services, at least since I arrived in 2007. I'm not doubting that Guido remembers some thing(s) but in general people here at Google would not stand for that, then or now. I would not call it common, especially five years ago. Common things I _have_ encountered over the years everywhere I've been both internal and external: services that listen on the https port 443 but don't have a valid cert as they are intended only for http port 80 access. Those are becoming somewhat less common, the only thing I regularly see that on anymore is random home router web config UIs as issuing a signed server certificate for security hole ridden commodity embedded box is... a challenge. (I'm not commenting on the PEP plans as it seems the right things are happening for now) -gps @ Google ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Thu, 4 Sep 2014 13:11:38 +1000 Nick Coghlan ncogh...@gmail.com wrote: That leaves Python 2.7, and I have to say I'm now persuaded that a backport (including any required httplib and urllib features) is the right way to go. One of the tasks I'd been dreading as a follow-on from PEP 466 was organising the code audit to make sure our existing Python 2 applications are properly configuring SSL. If we instead change Python 2.7.9 to validate certificates by default, then the need to do that audit *goes away*, replaced by the far more mundane tasking of doing integration testing on 2.7.9, which we'd have to do *anyway*. What are our existing Python 2 applications? Is it a Red Hat-specific statement? What is the code audit you are talking about? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 4 September 2014 22:39, Antoine Pitrou solip...@pitrou.net wrote: On Thu, 4 Sep 2014 13:11:38 +1000 Nick Coghlan ncogh...@gmail.com wrote: That leaves Python 2.7, and I have to say I'm now persuaded that a backport (including any required httplib and urllib features) is the right way to go. One of the tasks I'd been dreading as a follow-on from PEP 466 was organising the code audit to make sure our existing Python 2 applications are properly configuring SSL. If we instead change Python 2.7.9 to validate certificates by default, then the need to do that audit *goes away*, replaced by the far more mundane tasking of doing integration testing on 2.7.9, which we'd have to do *anyway*. What are our existing Python 2 applications? Is it a Red Hat-specific statement? What is the code audit you are talking about? Yes, that's a Red Hat specific statement. In addition to OpenStack, many of our other products include Python components. We mostly rely on pycurl, pyopenssl, python-nss, etc for SSL/TLS for historical reasons (since ssl wasn't available in the Python standard library until RHEL 6), which was why these concerns got picked up via the OpenStack community rather than the Fedora community. However, being aware of the problem creates that element of doubt as to whether there are *other* cases where someone used the stdlib ssl module rather than one of the alternatives that verifies HTTPS by default. I'd previously been looking at the question while wearing my upstream core developer CPython redistributor hats. Glyph et al finally prompted me to look at it while wearing my automated quality assurance toolchain architect for an application vendor hat (i.e. my actual day job), and that's a *radically* different perspective when it comes to a language runtime that fails to verify HTTPS connections correctly by default. That's the core reason I changed my mind: this conversation forced me to think through how I could ensure correct HTTPS handling in all our Python code, and the only workable solution I now see is to fix it at the standard library level. The alternative is to prepare for the likelihood of future CVEs in Python based tools based on failure to properly verify SSL certificates, as I don't see a practical way of automatically detecting tried to use HTTPS without verifying the SSL certificate properly. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 3 September 2014 01:19, Antoine Pitrou solip...@pitrou.net wrote: Keeping saying it doesn't make it magically true. Sure, but it *is* true, at the very least for HTTP. RFC 2818 (HTTP over TLS) has the following language in section 3.1: If the hostname is available, the client MUST check it against the server's identity as presented in the server's Certificate message, in order to prevent man-in-the-middle attacks. If the client has external information as to the expected identity of the server, the hostname check MAY be omitted. The default behaviour of httplib is in contravention of the normative language of this specification and is therefore bugged. (For those unclear about the relevance of RFC 2818, it's normatively referenced by RFC 7230, which is the HTTP/1.1 spec.) This should silence the debate about whether or not httplib's behaviour is a bug or not. Besides, it can perfectly well be a bug fix *as well as* a break in backwards compatibility. This is definitely true, and this change is both. The only question that matters is whether we believe we're doing users a service by breaking their code. I'd argue, along with Glyph, Alex and Donald, that we are. I've been on the losing side of this debate a number of times though, and I expect I will be again. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 3 Sep 2014 18:28, Cory Benfield c...@lukasa.co.uk wrote: This is definitely true, and this change is both. The only question that matters is whether we believe we're doing users a service by breaking their code. I'd argue, along with Glyph, Alex and Donald, that we are. I've been on the losing side of this debate a number of times though, and I expect I will be again. The default stdlib behaviour will change in 3.5, I don't think anyone is disputing that. While I earlier said that should depend on the sslcustomize PEP, I now think they should be made orthogonal so the SSL customisation PEP can focus on its potential for *increasing* security in properly configured environments rather than deliberately decreasing it after upgrading to Python 3.5 in improperly configured ones. The backwards compatibility argument only applies to Python 2 maintenance releases (where dreid indicated an intention to request backporting the change), and there I'm quite happy to take the position of use requests, Twisted or Python 3.5+ to get HTTPS done right. There are a variety of reasons not to use the Python 2 stdlib for modern networking, and making better tools more readily accessible to Python 2 users by backporting ensurepip is my preferred answer. Regards, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Wed, 3 Sep 2014 20:34:32 +1000 Nick Coghlan ncogh...@gmail.com wrote: The backwards compatibility argument only applies to Python 2 maintenance releases (where dreid indicated an intention to request backporting the change), and there I'm quite happy to take the position of use requests, Twisted or Python 3.5+ to get HTTPS done right. or asyncio. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Tue, 02 Sep 2014 21:29:16 -0400 R. David Murray rdmur...@bitdance.com wrote: The top proposal so far is an sslcustomize.py file that could be used to either decrease or increase the default security. This is a much less handy solution than application options (eg, curl, wget) that allow disabling security for this cert or this CLI session. It also is more prone to unthinking abuse since it is persistent. So perhaps it is indeed not worth it. (That's why I suggested an environment variable...something you could specify on the command line for a one-off.) I'll be fine with not adding any hooks at all, and letting people configure their application code correctly :-) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Wed, 03 Sep 2014 16:31:13 +0200, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 02 Sep 2014 21:29:16 -0400 R. David Murray rdmur...@bitdance.com wrote: The top proposal so far is an sslcustomize.py file that could be used to either decrease or increase the default security. This is a much less handy solution than application options (eg, curl, wget) that allow disabling security for this cert or this CLI session. It also is more prone to unthinking abuse since it is persistent. So perhaps it is indeed not worth it. (That's why I suggested an environment variable...something you could specify on the command line for a one-off.) I'll be fine with not adding any hooks at all, and letting people configure their application code correctly :-) Again, the problem arises when it is not *their* application code, but a third party tool that hasn't been ported to 3.5. I'm OK with letting go of this invalid-cert issue myself, given the lack of negative feedback Twisted got. I'll just keep my fingers crossed. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 09/03/2014 08:58 AM, R. David Murray wrote: I'm OK with letting go of this invalid-cert issue myself, given the lack of negative feedback Twisted got. I'll just keep my fingers crossed. I apologize if I missed this point, but if we have the source code then it is possible to go in and directly modify the application/utility to be able to talk over https to a router with an invalid certificate? This is an option when creating the ssl_context? -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Ethan Furman ethan at stoneleaf.us writes: I apologize if I missed this point, but if we have the source code then it is possible to go in and directly modify the application/utility to be able to talk over https to a router with an invalid certificate? This is an option when creating the ssl_context? -- ~Ethan~ Yes, it's totally possible to create (and pass to ``http.client``) an ``SSLContext`` which doesn't verify various things. My proposal is only about changing what happens when you don't explicitly pass a context. Cheers, Alex ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 09/03/2014 10:15 AM, Alex Gaynor wrote: Ethan Furman writes: I apologize if I missed this point, but if we have the source code then it is possible to go in and directly modify the application/utility to be able to talk over https to a router with an invalid certificate? This is an option when creating the ssl_context? Yes, it's totally possible to create (and pass to ``http.client``) an ``SSLContext`` which doesn't verify various things. My proposal is only about changing what happens when you don't explicitly pass a context. Excellent. Last question (I hope): it is possible to (easily) create an SSLContext that will verify against a self-signed certificate? -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Wed, Sep 3, 2014 at 8:58 AM, R. David Murray rdmur...@bitdance.com wrote: I'm OK with letting go of this invalid-cert issue myself, given the lack of negative feedback Twisted got. I'll just keep my fingers crossed. I'm with this sentiment (cautiously +1) -- and not just because of Twisted's experience or Glyph's passion. Network security is much more important now than it was five years ago -- and yet Python 2.7 is at least that old. My own experience has changed a lot: five years ago (when I worked at Google!) it was common to find internal services that required SSL but had a misconfigured certificate, and the only way to access those services was to override the browser complaints. Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- internal services here all redirect to SSL, and not a browser that can find fault with their certs. If I did get a complaint about a certificate I would fire off an email to a sysadmin alerting them to the issue. Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal). Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back). But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all. I don't want to start preaching security doom and gloom (the experts are doing enough of that :-), but the scale and sophistication of attacks (whether publicized or not) is constantly increasing, and routine maintenance checks on old software are just one of the small ways that we can help the internet become more secure. (And please let the PSF sysadmin team beef up *.python.org -- sooner or later some forgotten part of our infrastructure *will* come under attack.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Wed, 3 Sep 2014 10:54:55 -0700 Guido van Rossum gu...@python.org wrote: Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal). I'm entirely against this. Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back). Python is routinely updated to bugfix releases by Linux distributions and other distribution channels, you usually have no say over what's shipped in those updates. This is not like changing the major version used for executing the script, which is normally a manual change. Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- internal services here all redirect to SSL, and not a browser that can find fault with their certs. Good for you. I still sometimes get warnings about expired certificates - and sometimes ones that don't exactly match the domain being fetched (for example, the certificate wouldn't be valid for that specific subdomain - note that CAs often charge a premium for multiple subdomains, which why small or non-profit Web sites sometimes skimp on them). You shouldn't assume that the experience of well-connected people in the Silicon Valley is representative of what people over the world encounter. Yes, where there's a lot of money and a lot of accumulated domain competence, security procedures are updated and followed more scrupulously... But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all. Only if you are actually the one managing that certificate and the machine it's installed one... Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sep 3, 2014, at 1:54 PM, Guido van Rossum gu...@python.org wrote: On Wed, Sep 3, 2014 at 8:58 AM, R. David Murray rdmur...@bitdance.com mailto:rdmur...@bitdance.com wrote: I'm OK with letting go of this invalid-cert issue myself, given the lack of negative feedback Twisted got. I'll just keep my fingers crossed. I'm with this sentiment (cautiously +1) -- and not just because of Twisted's experience or Glyph's passion. Network security is much more important now than it was five years ago -- and yet Python 2.7 is at least that old. My own experience has changed a lot: five years ago (when I worked at Google!) it was common to find internal services that required SSL but had a misconfigured certificate, and the only way to access those services was to override the browser complaints. Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- internal services here all redirect to SSL, and not a browser that can find fault with their certs. If I did get a complaint about a certificate I would fire off an email to a sysadmin alerting them to the issue. Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal). Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back). But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all. +1, this makes me unreasonably happy. I don't want to start preaching security doom and gloom (the experts are doing enough of that :-), but the scale and sophistication of attacks (whether publicized or not) is constantly increasing, and routine maintenance checks on old software are just one of the small ways that we can help the internet become more secure. (And please let the PSF sysadmin team beef up *.python.org http://python.org/ -- sooner or later some forgotten part of our infrastructure *will* come under attack.) This is an ongoing effort amongst the Infra team, part of the process is moving infrastructure away from hand crafted servers towards servers managed by config management as well as making sure all our services are behind TLS as well. -- --Guido van Rossum (python.org/~guido http://python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Wed, 03 Sep 2014 20:37:38 +0200, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 3 Sep 2014 10:54:55 -0700 Guido van Rossum gu...@python.org wrote: Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- internal services here all redirect to SSL, and not a browser that can find fault with their certs. Good for you. I still sometimes get warnings about expired certificates - and sometimes ones that don't exactly match the domain being fetched (for example, the certificate wouldn't be valid for that specific subdomain - note that CAs often charge a premium for multiple subdomains, which why small or non-profit Web sites sometimes skimp on them). You shouldn't assume that the experience of well-connected people in the Silicon Valley is representative of what people over the world encounter. Yes, where there's a lot of money and a lot of accumulated domain competence, security procedures are updated and followed more scrupulously... Heck, yesterday I got invalid certs from...I think it was roku.com, but in any case not some obscure little company...the actual cert was an akamai cert, which means something is configured wrong somewhere. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 03.09.2014 19:29, Ethan Furman wrote: Excellent. Last question (I hope): it is possible to (easily) create an SSLContext that will verify against a self-signed certificate? Yes: context = ssl.create_default_context(cafile=/path/to/selfsigned.pem) That works iff the certificate is valid, not expired and its CN or SAN matches the hostname of the service. When the hostname doesn't match then you have to set context.check_hostname = False Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Wed, 03 Sep 2014 10:09:36 -0700, Ethan Furman et...@stoneleaf.us wrote: On 09/03/2014 08:58 AM, R. David Murray wrote: I'm OK with letting go of this invalid-cert issue myself, given the lack of negative feedback Twisted got. I'll just keep my fingers crossed. I apologize if I missed this point, but if we have the source code then it is possible to go in and directly modify the application/utility to be able to talk over https to a router with an invalid certificate? This is an option when creating the ssl_context? The immediately preceding paragraph that you didn't quote said that the context was 3rd party applications, not source code under your control. Yes, you can (usually) still hack the source, but there are good reasons to prefer to not do that, unfamiliarity with the codebase being just one of them. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Antoine, I think we are well past the point where arguments can sway positions. There clearly is no agreement on this issue. So please treat my post as a BDFL tie-breaker. I will just give you one thing to ponder -- those small/non-profit websites that can't afford proper certs are exactly the ones that will be hosting malware soon. Sorry for them, and the certificate vendors certainly aren't in it for charity, but they must fix their certificate issues (and probably improve many other sysadmin practices). --Guido On Wed, Sep 3, 2014 at 11:37 AM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 3 Sep 2014 10:54:55 -0700 Guido van Rossum gu...@python.org wrote: Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal). I'm entirely against this. Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back). Python is routinely updated to bugfix releases by Linux distributions and other distribution channels, you usually have no say over what's shipped in those updates. This is not like changing the major version used for executing the script, which is normally a manual change. Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- internal services here all redirect to SSL, and not a browser that can find fault with their certs. Good for you. I still sometimes get warnings about expired certificates - and sometimes ones that don't exactly match the domain being fetched (for example, the certificate wouldn't be valid for that specific subdomain - note that CAs often charge a premium for multiple subdomains, which why small or non-profit Web sites sometimes skimp on them). You shouldn't assume that the experience of well-connected people in the Silicon Valley is representative of what people over the world encounter. Yes, where there's a lot of money and a lot of accumulated domain competence, security procedures are updated and followed more scrupulously... But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all. Only if you are actually the one managing that certificate and the machine it's installed one... Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 03.09.2014 19:54, Guido van Rossum wrote: Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal). Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back). But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all. I'm +1 for Python 3.5 but -1 for Python 2.7. The SSLContext backport will landed in Python 2.7.9 (to be released). No Python 2 user is familiar with the feature yet. But more importantly: None of the stdlib modules support the new feature, too. httplib, imaplib ... they all don't take a SSLContext object as an argument. PEP-466 does not include the backport for the network modules. Without the context argument there is simply no clean way to configure the SSL handshake properly. The default settings must stay until we decide to backport the context argument and have a way to configure the default behavior. Nick and me are planing a PEP. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
2014-09-03 21:26 GMT+02:00 Christian Heimes christ...@python.org: On 03.09.2014 19:54, Guido van Rossum wrote: I'm +1 for Python 3.5 but -1 for Python 2.7. The SSLContext backport will landed in Python 2.7.9 (to be released). No Python 2 user is familiar with the feature yet. But more importantly: None of the stdlib modules support the new feature, too. httplib, imaplib ... they all don't take a SSLContext object as an argument. PEP-466 does not include the backport for the network modules. Without the context argument there is simply no clean way to configure the SSL handshake properly. Thanks, you replied before I asked the question :-) (If certificates are validated by default, how do you disable the checks?) Sorry, I didn't follow the whole discussion and Python 2.7 changes related to security. Does Python 2.7 support loading (automatically) system certificate authorities? Like the Windows certificate store: https://docs.python.org/dev/whatsnew/3.4.html#whatsnew34-win-cert-store Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
OK, that changes my position for 2.7 (but not for 3.5). I had assumed there was a way to disable the cert check by changing one parameter to the urlopen() call. (And I had wanted to add that there should be a clear FAQ about the subject.) If this isn't possible that changes the situation. (But I still think that once we do have that simple change option we should do it, in a later 2.7 upgrade.) I apologize for speaking before I had read all facts, and I'll await what you and Nick come up with. --Guido On Wed, Sep 3, 2014 at 12:26 PM, Christian Heimes christ...@python.org wrote: On 03.09.2014 19:54, Guido van Rossum wrote: Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal). Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back). But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all. I'm +1 for Python 3.5 but -1 for Python 2.7. The SSLContext backport will landed in Python 2.7.9 (to be released). No Python 2 user is familiar with the feature yet. But more importantly: None of the stdlib modules support the new feature, too. httplib, imaplib ... they all don't take a SSLContext object as an argument. PEP-466 does not include the backport for the network modules. Without the context argument there is simply no clean way to configure the SSL handshake properly. The default settings must stay until we decide to backport the context argument and have a way to configure the default behavior. Nick and me are planing a PEP. Christian -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 09/03/2014 12:10 PM, R. David Murray wrote: On Wed, 03 Sep 2014 10:09:36 -0700, Ethan Furman et...@stoneleaf.us wrote: On 09/03/2014 08:58 AM, R. David Murray wrote: I'm OK with letting go of this invalid-cert issue myself, given the lack of negative feedback Twisted got. I'll just keep my fingers crossed. I apologize if I missed this point, but if we have the source code then it is possible to go in and directly modify the application/utility to be able to talk over https to a router with an invalid certificate? This is an option when creating the ssl_context? The immediately preceding paragraph that you didn't quote said that the context was 3rd party applications, not source code under your control. Yes, you can (usually) still hack the source, but there are good reasons to prefer to not do that, unfamiliarity with the codebase being just one of them. I appreciate that there is a distinction, yet in most cases we have the source code available (it is the nature of Python) and if push comes to shove (and a bunch of other colloquialisms) then modifying that source code can get you up and running again. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 03.09.2014 21:37, Victor Stinner wrote: Thanks, you replied before I asked the question :-) (If certificates are validated by default, how do you disable the checks?) Sorry, I didn't follow the whole discussion and Python 2.7 changes related to security. Does Python 2.7 support loading (automatically) system certificate authorities? Like the Windows certificate store: https://docs.python.org/dev/whatsnew/3.4.html#whatsnew34-win-cert-store Yes! Alex and others have ported all SSL features back to Python 2.7. The SSL module in Python 2.7.9 will have the same feature set as Python 3.4. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 03.09.2014 21:37, Guido van Rossum wrote: OK, that changes my position for 2.7 (but not for 3.5). I had assumed there was a way to disable the cert check by changing one parameter to the urlopen() call. (And I had wanted to add that there should be a clear FAQ about the subject.) If this isn't possible that changes the situation. (But I still think that once we do have that simple change option we should do it, in a later 2.7 upgrade.) I apologize for speaking before I had read all facts, and I'll await what you and Nick come up with. You are welcome! :) I like to see cert checks in Python 2.7, too. Eventually Python 2.7 should have them enabled by default or at least have one very simple way to enable them globally. Right now we aren't there yet. Perhaps Python 2.7.10 or 2.7.11 will have the necessary bits and backports. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Guido van Rossum guido at python.org writes: OK, that changes my position for 2.7 (but not for 3.5). I had assumed there was a way to disable the cert check by changing one parameter to the urlopen() call. (And I had wanted to add that there should be a clear FAQ about the subject.) If this isn't possible that changes the situation. (But I still think that once we do have that simple change option we should do it, in a later 2.7 upgrade.) I apologize for speaking before I had read all facts, and I'll await what you and Nick come up with. --Guido This probably doesn't surprise anyone, but I'm more than happy to do the back- porting work for httplib, and any other modules which need SSLContext support; does this require an additional PEP, or does it fit under PEP466 or PEP476? Alex ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Guido van Rossum writes: lot: five years ago (when I worked at Google!) it was common to find internal services that required SSL but had a misconfigured certificate, and the only way to access those services was to override the browser complaints. Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- I would tend to discount your recent experience, then. Smaller (and possibly even more important in this fast-developing area, younger) organizations are a lot more nimble about things like this. That is not intended to express an opinion about a backport, though. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Wed, Sep 3, 2014, at 13:37, Alex Gaynor wrote: Guido van Rossum guido at python.org writes: OK, that changes my position for 2.7 (but not for 3.5). I had assumed there was a way to disable the cert check by changing one parameter to the urlopen() call. (And I had wanted to add that there should be a clear FAQ about the subject.) If this isn't possible that changes the situation. (But I still think that once we do have that simple change option we should do it, in a later 2.7 upgrade.) I apologize for speaking before I had read all facts, and I'll await what you and Nick come up with. --Guido This probably doesn't surprise anyone, but I'm more than happy to do the back- porting work for httplib, and any other modules which need SSLContext support; does this require an additional PEP, or does it fit under PEP466 or PEP476? I imagine you could put it into 476. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 4 Sep 2014 04:39, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 3 Sep 2014 10:54:55 -0700 Guido van Rossum gu...@python.org wrote: Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal). I'm entirely against this. Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back). Python is routinely updated to bugfix releases by Linux distributions and other distribution channels, you usually have no say over what's shipped in those updates. This is not like changing the major version used for executing the script, which is normally a manual change. We can potentially deal with the more conservative part of the user base on the redistributor side - so long as the PEP says it's OK for us to not apply this particular change if we deem it appropriate to do so. That would make this a case of upstream asking us to do the kind of risk assessment that people pay us for, which is a kind of request I'm generally happy to get :) That way, if downstream users get upset, we can point them at their vendor support department, rather than having them take out their ire on upstream volunteers. Also, after thinking through the implications a bit more, my guess is that Fedora Software Collections will accept the change without any fuss, but the CentOS/RHEL side could be a more involved discussion. On the other hand, orgs with these kinds of problems aren't likely to have rolled out RHEL 7 or CentOS 7 yet either, so they're probably still back on Python 2.6 (RHEL 6) or even 2.4 (RHEL 5). 2.7.9 is going to be a somewhat interesting release that requires careful attention anyway (due to the completion of the PEP 466 backports), so if Guido's OK with it, sure, let's kill the HTTPS isn't problem for Python 2 as well. One additional wrinkle in that case: we will likely need to backport the SSLContext related changes to httplib as well. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 4 Sep 2014 06:39, Alex Gaynor alex.gay...@gmail.com wrote: Guido van Rossum guido at python.org writes: OK, that changes my position for 2.7 (but not for 3.5). I had assumed there was a way to disable the cert check by changing one parameter to the urlopen() call. (And I had wanted to add that there should be a clear FAQ about the subject.) If this isn't possible that changes the situation. (But I still think that once we do have that simple change option we should do it, in a later 2.7 upgrade.) I apologize for speaking before I had read all facts, and I'll await what you and Nick come up with. --Guido This probably doesn't surprise anyone, but I'm more than happy to do the back- porting work for httplib, and any other modules which need SSLContext support; does this require an additional PEP, or does it fit under PEP466 or PEP476? I suggest writing up a separate PEP for 2.7 that covers exactly what would need to be backported in order to make the same HTTPS handling change in Python 2. For 476, I suggest taking my list of modules that call _create_stdlib_cert and being completely explicit as to which ones are *not* changing (as that will help clarify the scope of the backport proposal). I learned that lesson with PEP 453 - it's well worth making the Python 3 PEP easier to accept by making it independent of the inevitably more controversial Python 2 backport PEP :) Cheers, Nick. Alex ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Thu, 4 Sep 2014 09:19:56 +1000 Nick Coghlan ncogh...@gmail.com wrote: Python is routinely updated to bugfix releases by Linux distributions and other distribution channels, you usually have no say over what's shipped in those updates. This is not like changing the major version used for executing the script, which is normally a manual change. We can potentially deal with the more conservative part of the user base on the redistributor side - so long as the PEP says it's OK for us to not apply this particular change if we deem it appropriate to do so. So people would believe python.org that they would get HTTPS cert validation by default, but their upstream distributor would have disabled it for them? That's even worse... Of course, people could read distribution-specific package changelogs, but nobody does that. 2.7.9 is going to be a somewhat interesting release that requires careful attention anyway (due to the completion of the PEP 466 backports), so if Guido's OK with it, sure, let's kill the HTTPS isn't problem for Python 2 as well. Possible unvoluntary breakage due to a large backport is one thing. Deliberate breakage is another. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 09/03/2014 04:36 PM, Antoine Pitrou wrote: On Thu, 4 Sep 2014 09:19:56 +1000 Nick Coghlan ncogh...@gmail.com wrote: Python is routinely updated to bugfix releases by Linux distributions and other distribution channels, you usually have no say over what's shipped in those updates. This is not like changing the major version used for executing the script, which is normally a manual change. We can potentially deal with the more conservative part of the user base on the redistributor side - so long as the PEP says it's OK for us to not apply this particular change if we deem it appropriate to do so. So people would believe python.org that they would get HTTPS cert validation by default, but their upstream distributor would have disabled it for them? That's even worse... I agree. If the vendors don't want to have validation by default, they should stick with 2.7.8. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 09/03/2014 05:00 PM, Ethan Furman wrote: On 09/03/2014 04:36 PM, Antoine Pitrou wrote: On Thu, 4 Sep 2014 09:19:56 +1000 Nick Coghlan ncogh...@gmail.com wrote: Python is routinely updated to bugfix releases by Linux distributions and other distribution channels, you usually have no say over what's shipped in those updates. This is not like changing the major version used for executing the script, which is normally a manual change. We can potentially deal with the more conservative part of the user base on the redistributor side - so long as the PEP says it's OK for us to not apply this particular change if we deem it appropriate to do so. So people would believe python.org that they would get HTTPS cert validation by default, but their upstream distributor would have disabled it for them? That's even worse... I agree. If the vendors don't want to have validation by default, they should stick with 2.7.8. If good argument can be made for why we should make validation by default optional, then that point should be well made in Python's release notes, and some easy programmatic way to tell if validation is on or off (which may just be more docs saying call SSLContext and examine the results: xxx means you're validating, yyy means you are not). -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 4 September 2014 10:00, Ethan Furman et...@stoneleaf.us wrote: On 09/03/2014 04:36 PM, Antoine Pitrou wrote: On Thu, 4 Sep 2014 09:19:56 +1000 Nick Coghlan ncogh...@gmail.com wrote: Python is routinely updated to bugfix releases by Linux distributions and other distribution channels, you usually have no say over what's shipped in those updates. This is not like changing the major version used for executing the script, which is normally a manual change. We can potentially deal with the more conservative part of the user base on the redistributor side - so long as the PEP says it's OK for us to not apply this particular change if we deem it appropriate to do so. So people would believe python.org that they would get HTTPS cert validation by default, but their upstream distributor would have disabled it for them? That's even worse... I agree. If the vendors don't want to have validation by default, they should stick with 2.7.8. Yes, that's the way it would work in practice - we'd call it 2.7.8, and backport fixes from upstream 2.7.x as needed (as someone put it to me recently, a useful way to think of component version numbers in RHEL is as referring to the *oldest* pieces of code in that component). I've spent quite a bit more time considering the proposal, and I'm now satisfied that making this change upstream isn't likely to cause any major problems, due to the fact that folks who are likely to suffer from this change aren't likely to even be on 2.7 yet. While we managed not to be last this time, the RHEL/CentOS ecosystem is still a useful benchmark for the roll out of Python versions into conservative enterprise organisations, and the more conservative users within *that* ecosystem are likely to wait for the x.1 release (at the earliest) rather than upgrading as soon as x.0 is out. RHEL 7.0 only came out in June, so most of those conservative environments are still going to be on Python 2.6 in RHEL 6. While we shipped 2.7 support well before the release of RHEL 7 as part of Software Collections and OpenShift, the kinds of environments where properly validating SSL by default may cause problems aren't likely to be on the leading edge of adopting new approaches to software deployment like SCL and PaaS. Fixing the HTTPS validation defaults would have several significant positive consequences: - lowers a major barrier to Python adoption (and continued usage) for public internet focused services - fixes a latent security defect for Python applications operating over the public internet - fixes a latent security defect for Python applications operating in a properly configured intranet environment - reveals a latent security defect for Python applications communicating with improperly configured public internet services - reveals a latent security defect for Python applications operating in improperly configured intranet environments The debate is thus solely over how to manage the consequences of the last two, since going from silent failure to noisy failure looks like going from working to broken to someone that isn't already intimately familiar with the underlying issues. That question needs to be considered separately for 3 different versions: - 3.5 - 3.4 - 2.7 3.5 is not controversial, the answer is that the standard library's handling of HTTPS URLs should change to verify certificates properly. No ifs, buts, or maybes - Python 3.5 should automatically verify all HTTPS connections, with explicit developer action required to skip (or otherwise relax) the validation check. So far, we have assumed that 3.4 will get at most a warning. However, I have changed my mind on that, because Python 3 is still largely an early adopter driven technology (it's making inroads into more conservative environments, but it's still at least a few years away from catching up to Python 2 on that front). As a result, the kinds of environments RDM and I are worried about will generally *not* be using Python 3, or if they are, it will be for custom scripts that they can fix. I wouldn't suggest actually making that change without getting an explicit +1 from the Canonical folks (since 3.4 is in Ubuntu LTS 14.04), but I would now personally be +1 on just *fixing it* in 3.4.2, rather than doing a bunch of additional development work purely so we can make folks wait another year for the Python 3 standard library to correctly handle HTTPS. That leaves Python 2.7, and I have to say I'm now persuaded that a backport (including any required httplib and urllib features) is the right way to go. One of the tasks I'd been dreading as a follow-on from PEP 466 was organising the code audit to make sure our existing Python 2 applications are properly configuring SSL. If we instead change Python 2.7.9 to validate certificates by default, then the need to do that audit *goes away*, replaced by the far more mundane tasking of doing integration testing on 2.7.9, which we'd have to do *anyway*. Systematically
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Aug 29, 2014, at 7:44 PM, Alex Gaynor alex.gay...@gmail.com wrote: Disabling verification entirely externally to the program, through a CLI flag or environment variable. I'm pretty down on this idea, the problem you hit is that it's a pretty blunt instrument to swing, and it's almost impossible to imagine it not hitting things it shouldn't; it's far too likely to be used in applications that make two sets of outbound connections: 1) to some internal service which you want to disable verification on, and 2) some external service which needs strong validation. A global flag causes the latter to fail silently when subjected to a MITM attack, and that's exactly what we're trying to avoid. It also makes things much harder for library authors: I write an API client for some API, and make TLS connections to it. I want those to be verified by default. I can't even rely on the httplib defaults, because someone might disable them from the outside. I would strongly recommend against such a mechanism. For what it's worth, Twisted simply unconditionally started verifying certificates in 14.0 with no disable switch, and (to my knowledge) literally no users have complained. Twisted has a very, very strict backwards compatibility policy. For example, I once refused to accept the deletion of a class that raised an exception upon construction, on the grounds that someone might have been inadvertently importing that class, and they shouldn't see an exception until they've seen a deprecation for one release. Despite that, we classified failing to verify certificates as a security bug, and fixed it with no deprecation period. When users type the 's' after the 'p' and before the ':' in a URL, they implicitly expect browser-like certificate verification. The lack of complaints is despite the fact that 14.0 has been out for several months now, and, thanks to the aforementioned strict policy, users tend to upgrade fairly often (since they know they can almost always do so without fear of application-breaking consequences). According to PyPI metadata, 14.0.0 has had 273283 downloads so far. Furthermore, disable verification is a nonsensical thing to do with TLS. select a trust root is a valid configuration option, and OpenSSL already provides it via the SSL_CERT_DIR environment variable, so there's no need for Python to provide anything beyond that. -glyph ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Tue, 2 Sep 2014 14:00:02 -0700 Glyph Lefkowitz gl...@twistedmatrix.com wrote: I would strongly recommend against such a mechanism. For what it's worth, Twisted simply unconditionally started verifying certificates in 14.0 with no disable switch, and (to my knowledge) literally no users have complained. And how many people are using Twisted as an HTTPS client? (compared to e.g. Python's httplib, and all the third-party libraries building on it?) Furthermore, disable verification is a nonsensical thing to do with TLS. It's not. For example, if you have an expired cert, all you can do AFAIK is to disable verification. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 1 Sep 2014 16:05, Nick Coghlan ncogh...@gmail.com wrote: The final change would be to seed the context factory map appropriately for the standard library modules where we wanted to keep the *old* default: for modname in (nntplib, poplib, imaplib, ftplib, smtplib, asyncio.selector_events, urllib.request, http.client): named_contexts[modname] = create_legacy_context The list I have above is for *all* current uses of sss._create_stdlib_context. The backwards incompatible part of PEP 476 would then just be about removing names from that list (currently just http.client, but I'd suggest asyncio.selector_events as another candidate, taking advantage of asyncio's provisional API status). Update on this: Christian I both like the SSL named context and customisation idea independently of the proposal to change the default behaviour, so we're going to write it up as a separate PEP. Changing the default behaviour would then be reduced to a proposal to take a couple of module names off the uses legacy SSL settings list. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Antoine Pitrou solipsis at pitrou.net writes: And how many people are using Twisted as an HTTPS client? (compared to e.g. Python's httplib, and all the third-party libraries building on it?) I don't think anyone could give an honest estimate of these counts, however there's two factors to bare in mind: a) It's extremely strongly recommended to use requests to make any HTTP requests precisely because httplib is negligent in certificate and hostname checking by default, b) We're talking about Python3, which has fewer users than Python2. Furthermore, disable verification is a nonsensical thing to do with TLS. It's not. For example, if you have an expired cert, all you can do AFAIK is to disable verification. It really is a nonsensical operation, accepting any random TLS certificate without pinning or use of a certificate authorities makes a valid connection completely indistinguishable from a MITM attack. If I were the emperor of the universe (or even just Python ;-)) I wouldn't allow this operation at all, however, I'm not and you can still disable any and all verification. It just requires you to pass a different argument, which doesn't seem overly burdensome. This whole scenario seems to be predicated on a siutation where: You have a peer whose certificate you can't change, and you have a piece of code you can't change, and you're going to upgrade your Python installation, and you want to talk to this peer, and you need to use an encrypted channel, but don't really care if it's being MITM'd. It doesn't seem to me that this is reasonably Python's responsibility to deal with the fact that you have no ability to upgrade any of your infrastructure, except your Python version. Alex ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Tue, 2 Sep 2014 22:16:18 + (UTC) Alex Gaynor alex.gay...@gmail.com wrote: Furthermore, disable verification is a nonsensical thing to do with TLS. It's not. For example, if you have an expired cert, all you can do AFAIK is to disable verification. It really is a nonsensical operation, accepting any random TLS certificate without pinning or use of a certificate authorities makes a valid connection completely indistinguishable from a MITM attack. It's still distinguishable from a passively-eavesdroppable clear-text connection, though. This whole scenario seems to be predicated on a siutation where: You have a peer whose certificate you can't change, and you have a piece of code you can't change, and you're going to upgrade your Python installation, and you want to talk to this peer, and you need to use an encrypted channel, but don't really care if it's being MITM'd. It doesn't seem to me that this is reasonably Python's responsibility to deal with the fact that you have no ability to upgrade any of your infrastructure, except your Python version. Oh, I agree that whoever upgrades their Python version should be able to fix any of their applications should they start failing. That's why I don't want to let new command-line options and environment variables proliferate in the name of damage control. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 02.09.2014 23:32, Antoine Pitrou wrote:
Furthermore, disable verification is a nonsensical thing to do with TLS.
It's not. For example, if you have an expired cert, all you can do
AFAIK is to disable verification.
It's possible to ignore or just warn about expired certs with simple
verify callback. The callback looks like this:
int verify_callback(int ok, X509_STORE_CTX *ctx)
{
if (X509_STORE_CTX_get_error(ctx) == X509_V_ERR_CERT_HAS_EXPIRED)
return 1;
return ok;
}
It's installed like this:
PySSLContext *self;
X509_STORE *store = SSL_CTX_get_cert_store(self-ctx);
X509_STORE_set_verify_cb(store, verify_callback);
The X509_STORE_CTX struct is created when a certificate chain is
verified. It holds all sorts of states like chain, leaf cert, current
cert that is tested, validation depth, error flags and more. In order to
write useful verify callbacks me or somebody else has to write a
X509_STORE_CTX type and X509 cert type. It's something I want to do for
more than a year but I don't find any spare time. :(
Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 3 Sep 2014 08:18, Alex Gaynor alex.gay...@gmail.com wrote: Antoine Pitrou solipsis at pitrou.net writes: And how many people are using Twisted as an HTTPS client? (compared to e.g. Python's httplib, and all the third-party libraries building on it?) I don't think anyone could give an honest estimate of these counts, however there's two factors to bare in mind: a) It's extremely strongly recommended to use requests to make any HTTP requests precisely because httplib is negligent in certificate and hostname checking by default, b) We're talking about Python3, which has fewer users than Python2. Creating *new* incompatibilities between Python 2 Python 3 is a major point of concern. One key focus of 3.5 is *reducing* barriers to migration, and this PEP would be raising a new one. It's a change worth making, but we have time to ensure there are easy ways to do things like skipping cert validation, or tolerate expired certificates. Regards, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Nick Coghlan ncoghlan at gmail.com writes: Creating *new* incompatibilities between Python 2 Python 3 is a major point of concern. Clearly this change should be backported to Python2. -David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Tue, 02 Sep 2014 22:16:18 -, Alex Gaynor alex.gay...@gmail.com wrote: This whole scenario seems to be predicated on a siutation where: You have a peer whose certificate you can't change, and you have a piece of code you can't change, and you're going to upgrade your Python installation, and you want to talk to this peer, and you need to use an encrypted channel, but don't really care if it's being MITM'd. It doesn't seem to me that this is reasonably Python's responsibility to deal with the fact that you have no ability to upgrade any of your infrastructure, except your Python version. I would say that is an excellent summary, except that you are sometimes *forced* to use an encrypted channel (the device only opens the https port). That is, in the real messy world of network devices, accepting an invalid cert is not a nonsensical operation. (Of course, what I really want is to be able to say I know this cert has invalid fields, accept it anyway, but warn me if it changes when I connect again.) It's a good point that the actual number of people who will be hit by this and can't hack the code in question will be (possibly vanishingly) small, especially if you only consider python3. But we've already had one call for a backport :) --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sep 2, 2014, at 4:01 PM, Nick Coghlan ncogh...@gmail.com wrote: On 3 Sep 2014 08:18, Alex Gaynor alex.gay...@gmail.com wrote: Antoine Pitrou solipsis at pitrou.net writes: And how many people are using Twisted as an HTTPS client? (compared to e.g. Python's httplib, and all the third-party libraries building on it?) I don't think anyone could give an honest estimate of these counts, however there's two factors to bare in mind: a) It's extremely strongly recommended to use requests to make any HTTP requests precisely because httplib is negligent in certificate and hostname checking by default, b) We're talking about Python3, which has fewer users than Python2. Creating *new* incompatibilities between Python 2 Python 3 is a major point of concern. One key focus of 3.5 is *reducing* barriers to migration, and this PEP would be raising a new one. No. Providing the security that the user originally asked for is not a backwards incompatible change. It is a bug fix. And believe me: I care a _LOT_ about reducing barriers to migration. This would not be on my list of the top 1000 things that make migration difficult. It's a change worth making, but we have time to ensure there are easy ways to do things like skipping cert validation, or tolerate expired certificates. The API already supports both of these things. What I believe you're implicitly saying is that there needs to be a way to do this without editing code, and... no, there really doesn't. Not to mention the fact that you could already craft a horrific monkeypatch to allow operators to cause the ssl module to malfunction by 'pip install'ing a separate package, which is about as supported as this should be. -glyph ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 3 Sep 2014 09:08, David Reid dr...@dreid.org wrote: Nick Coghlan ncoghlan at gmail.com writes: Creating *new* incompatibilities between Python 2 Python 3 is a major point of concern. Clearly this change should be backported to Python2. Proposing to break backwards compatibility in a maintenance release would require an *even better* migration story for handling legacy infrastructure. There's a reason the Py3 compatibility break was accompanied by a long term support commitment for Python 2 (and why that commitment was extended another 5 years when it became clear the implications of the binary and text handling were more significant than we originally realised). It's important to remember that the tech industry in general, and open source focused companies in particular, tend to be amongst the best of the best when it comes to infrastructure management (and we're still pretty terrible at it). The vast majority of infrastructure in the world *isn't* in the hands of those companies though - it's in the hands of companies where having any kind of computing infrastructure at all is merely a cost of doing business. Alex's ridiculous scenario doesn't sound ridiculous to me - it sounds like a typical corporate computing environment. Regards, Nick. -David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sep 2, 2014, at 4:28 PM, Nick Coghlan ncogh...@gmail.com wrote: On 3 Sep 2014 09:08, David Reid dr...@dreid.org wrote: Nick Coghlan ncoghlan at gmail.com writes: Creating *new* incompatibilities between Python 2 Python 3 is a major point of concern. Clearly this change should be backported to Python2. Proposing to break backwards compatibility in a maintenance release (...) As we keep saying, this is not a break in backwards compatibility, it's a bug fix. Yes, systems might break, but that breakage represents an increase in security which may well be operationally important. Not everyone with a working application has the relevant understanding an expertise to know that Python's HTTP client is exposing them to surveillance. These applications should break. That is the very nature of the fix. It is not a compatibility break that the system starts correctly rejecting invalid connections. By way of analogy, here's another kind of breach in security: an arbitrary remote code execution vulnerability in XML-RPC. I think we all agree that any 0day RCE vulnerabilities in Python really ought to be fixed and could be legitimately included without worrying about backwards compatibility breaks. (At least... gosh, I hope so.) Perhaps this arbitrary remote execution looks harmless; the use of an eval() instead of an int() someplace. Perhaps someone discovered that they can do 3 + 4 in their XML-RPC and the server does the computation for them. Great! They start relying on this in their applications to use symbolic values in their requests instead of having explicit enumerations. This can save you quite a bit of code! When the RCE is fixed, this application will break, and that's fine. In fact that's the whole point of issuing the fix, that people will no longer be able to make arbitrary computation requests of your server any more. If that server's maintainer has the relevant context and actually wants the XML-RPC endpoint to enable arbitrary RCE, they can easily modify their application to start doing eval() on the data that they received, just as someone can easily modify their application to intentionally disable all connection security. (Let's stop calling it certificate verification because that sounds like some kind of clerical detail: if you disable certificate verification, TLS connections are unauthenticated and unidentified and therefore insecure.) For what it's worth, on the equivalent Twisted change, I originally had just these concerns, but my mind was changed when I considered what exactly the user-interface ramifications were for people typing that 's' for 'secure' in URLs. I was convinced, and we made the change, and there have been no ill effects that I'm aware of as a result. In fact, there has been a renewed interest in Twisted for HTTP client work, because we finally made security work more or less like it's supposed to, and the standard library is so broken. I care about the health of the broader Python community, so I will passionately argue that this change should be made, but for me personally it's a lot easier to justify that everyone should use Twisted (at least since 14+) because transport security in the stdlib is such a wreck and even if it gets fixed it's going to have easy options to turn it off unilaterally so your application can never really be sure if it's getting transport security when it's requesting transport security. -glyph ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sep 2, 2014, at 7:47 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: On Sep 2, 2014, at 4:28 PM, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: On 3 Sep 2014 09:08, David Reid dr...@dreid.org mailto:dr...@dreid.org wrote: Nick Coghlan ncoghlan at gmail.com http://gmail.com/ writes: Creating *new* incompatibilities between Python 2 Python 3 is a major point of concern. Clearly this change should be backported to Python2. Proposing to break backwards compatibility in a maintenance release (...) As we keep saying, this is not a break in backwards compatibility, it's a bug fix. Yes, systems might break, but that breakage represents an increase in security which may well be operationally important. Not everyone with a working application has the relevant understanding an expertise to know that Python's HTTP client is exposing them to surveillance. These applications should break. That is the very nature of the fix. It is not a compatibility break that the system starts correctly rejecting invalid connections. By way of analogy, here's another kind of breach in security: an arbitrary remote code execution vulnerability in XML-RPC. I think we all agree that any 0day RCE vulnerabilities in Python really ought to be fixed and could be legitimately included without worrying about backwards compatibility breaks. (At least... gosh, I hope so.) Perhaps this arbitrary remote execution looks harmless; the use of an eval() instead of an int() someplace. Perhaps someone discovered that they can do 3 + 4 in their XML-RPC and the server does the computation for them. Great! They start relying on this in their applications to use symbolic values in their requests instead of having explicit enumerations. This can save you quite a bit of code! When the RCE is fixed, this application will break, and that's fine. In fact that's the whole point of issuing the fix, that people will no longer be able to make arbitrary computation requests of your server any more. If that server's maintainer has the relevant context and actually wants the XML-RPC endpoint to enable arbitrary RCE, they can easily modify their application to start doing eval() on the data that they received, just as someone can easily modify their application to intentionally disable all connection security. (Let's stop calling it certificate verification because that sounds like some kind of clerical detail: if you disable certificate verification, TLS connections are unauthenticated and unidentified and therefore insecure.) For what it's worth, on the equivalent Twisted change, I originally had just these concerns, but my mind was changed when I considered what exactly the user-interface ramifications were for people typing that 's' for 'secure' in URLs. I was convinced, and we made the change, and there have been no ill effects that I'm aware of as a result. In fact, there has been a renewed interest in Twisted for HTTP client work, because we finally made security work more or less like it's supposed to, and the standard library is so broken. I care about the health of the broader Python community, so I will passionately argue that this change should be made, but for me personally it's a lot easier to justify that everyone should use Twisted (at least since 14+) because transport security in the stdlib is such a wreck and even if it gets fixed it's going to have easy options to turn it off unilaterally so your application can never really be sure if it's getting transport security when it's requesting transport security. I completely agree with everything Glyph has said in this post. (To the shock of everyone involved I’m sure!). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Tue, 2 Sep 2014 16:47:35 -0700 Glyph Lefkowitz gl...@twistedmatrix.com wrote: On Sep 2, 2014, at 4:28 PM, Nick Coghlan ncogh...@gmail.com wrote: On 3 Sep 2014 09:08, David Reid dr...@dreid.org wrote: Nick Coghlan ncoghlan at gmail.com writes: Creating *new* incompatibilities between Python 2 Python 3 is a major point of concern. Clearly this change should be backported to Python2. Proposing to break backwards compatibility in a maintenance release (...) As we keep saying, this is not a break in backwards compatibility, it's a bug fix. Keeping saying it doesn't make it magically true. Besides, it can perfectly well be a bug fix *as well as* a break in backwards compatibility. Which is why we sometimes choose to fix bugs only in the feature development branch. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 9/2/2014 7:47 PM, Glyph Lefkowitz wrote: On Sep 2, 2014, at 4:28 PM, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: On 3 Sep 2014 09:08, David Reid dr...@dreid.org mailto:dr...@dreid.org wrote: Clearly this change should be backported to Python2. Proposing to break backwards compatibility in a maintenance release (...) For code that depends on a bug, fixing the bug breaks compatibility between releases without and with the fix. Nothing new. We occasionally don't backpart disruptive bugfixes because of that. We also backport some security fixes to otherwise unmaintained versions. As we keep saying, this is not a break in backwards compatibility, it's a bug fix. Disagreement on whether changing the default for certificate validation is a 'bugfix' or 'enhancement' (and in the colloquial or python tracker sense) the seems like the crux of the disagreement here. The case for 'bug', at least in the colloquial meaning, is at least plausible. On the other hand, the tracker meaning of 'behavior' issue (contrary to intention and doc) is more strict in including implementation bugs but not necessarily design bugs. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Tue, 02 Sep 2014 20:59:54 -0400, Terry Reedy tjre...@udel.edu wrote: On 9/2/2014 7:47 PM, Glyph Lefkowitz wrote: On Sep 2, 2014, at 4:28 PM, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: On 3 Sep 2014 09:08, David Reid dr...@dreid.org mailto:dr...@dreid.org wrote: Clearly this change should be backported to Python2. Proposing to break backwards compatibility in a maintenance release (...) For code that depends on a bug, fixing the bug breaks compatibility between releases without and with the fix. Nothing new. We occasionally don't backpart disruptive bugfixes because of that. We also backport some security fixes to otherwise unmaintained versions. As we keep saying, this is not a break in backwards compatibility, it's a bug fix. Disagreement on whether changing the default for certificate validation is a 'bugfix' or 'enhancement' (and in the colloquial or python tracker sense) the seems like the crux of the disagreement here. The case for 'bug', at least in the colloquial meaning, is at least plausible. On the other hand, the tracker meaning of 'behavior' issue (contrary to intention and doc) is more strict in including implementation bugs but not necessarily design bugs. Up to this point the suggestion has been to change the default only in 3.5, and what we are arguing about is what kind of and how much aid to give to those for whom 3.5 breaks backward compatibility. So arguing about feature versus bug is irrelevant. The top proposal so far is an sslcustomize.py file that could be used to either decrease or increase the default security. This is a much less handy solution than application options (eg, curl, wget) that allow disabling security for this cert or this CLI session. It also is more prone to unthinking abuse since it is persistent. So perhaps it is indeed not worth it. (That's why I suggested an environment variable...something you could specify on the command line for a one-off.) The more I think about it the less I like sslcustomize. I think I'd rather do without the ability to turn off checking without modifying code rather than introduce an intentional vector for a system-wide decrease in security. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Antoine Pitrou writes: On Tue, 2 Sep 2014 16:47:35 -0700 Glyph Lefkowitz gl...@twistedmatrix.com wrote: As we keep saying, this is not a break in backwards compatibility, it's a bug fix. Keeping saying it doesn't make it magically true. It's not magically true, it is just true. What the hardliners fail to acknowledge is that this is *not a bug in Python, it's a bug in the whole system*, and *mostly* in the environment. Changing Python will not change the environment, and applications will fail, with unknown consequences. Saying they should fail *right* now is bogus when you don't even know what those applications are, or what other security measures may be in place: Now is better than never. Although never is often better than *right* now. On the other hand, I commend the Twisted developers for putting their values into their code with their reputation on the line. I hope they win big with this move! Shouldn't we all hope for that? Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 1 September 2014 11:10, R. David Murray rdmur...@bitdance.com wrote: It sounds like this would address my concerns as well (I don't really care *how* it is implemented as long as I don't have to touch the code of a third party application when I upgrade my python version to 3.5...remember, the context here is backward compatibility concerns). Does it address the issue of accepting an invalid cert, though? That's actually an interesting question, as the PEP doesn't currently propose adding any new global configuration knobs to the ssl or httplib modules - it just proposes switching httplib from the legacy (but fully backwards compatible) ssl._create_stdlib_context() API to the newer (but potentially backwards incompatible in some environments) ssl.create_default_context() API. Having the ssl module import an sslcustomize module at the end wouldn't be enough unless the appropriate APIs were put in place to allow things to be configured at a process global level. One possible way to do that would be to provide a central context factory mapping that provide a module specific SSL context creator. We'd seed it appropriately for the stdlib modules where we wanted to use the legacy context definition, but it would default to using ssl.create_default_context. Under that kind of model, the first change we would actually make is to make ssl._create_stdlib_context() public under a suitable name, let's say ssl.create_legacy_context() Independenting of any other changes, exposing ssl.create_legacy_context() like that would also make it straightforward for folks to opt in to the old behaviour as an interim hack in a way that is easy to grep for and fix later (it's also something a linter can easily disallow). The second change would be to provide a mapping from arbitrary names to context factories in the ssl module that defaults to ssl.create_default_context: named_contexts = defaultdict((lambda name: create_default_context)) (A more accurate name would be named_context_factory, but I think named_contexts reads better. Folks will learn quickly enough that it actually stores context factories rather than prebuilt context objects) The third change would be to replace all calls to ssl._create_stdlib_context() with calls to ssl.named_contexts[__name__]() instead. The final change would be to seed the context factory map appropriately for the standard library modules where we wanted to keep the *old* default: for modname in (nntplib, poplib, imaplib, ftplib, smtplib, asyncio.selector_events, urllib.request, http.client): named_contexts[modname] = create_legacy_context The list I have above is for *all* current uses of sss._create_stdlib_context. The backwards incompatible part of PEP 476 would then just be about removing names from that list (currently just http.client, but I'd suggest asyncio.selector_events as another candidate, taking advantage of asyncio's provisional API status). The revert to 3.4 behaviour content for sslcustomize.py would then just be: import ssl ssl.named_contexts[http.client] = ssl.create_legacy_context However, someone that wanted to also enforce SSL properly for other standard library modules could go the other way: import ssl for modname in (nntplib, poplib, imaplib, ftplib, smtplib, urllib.request): ssl.named_contexts[modname] = ssl.create_default_context Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 23:10, Nick Coghlan ncogh...@gmail.com wrote: Assuming sslcustomize was in site-packages rather than the standard library directories, you would also be able to use virtual environments with an appropriate sslcustomize module to disable cert checking even if the application you were running didn't support direct configuration. Would this mean that a malicious package could install a custom sslcustomize.py and so add unwanted certs to the system? I guess we have to assume that installed packages are trusted, but I just wanted to be explicit. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 1 September 2014 16:07, Paul Moore p.f.mo...@gmail.com wrote: On 31 August 2014 23:10, Nick Coghlan ncogh...@gmail.com wrote: Assuming sslcustomize was in site-packages rather than the standard library directories, you would also be able to use virtual environments with an appropriate sslcustomize module to disable cert checking even if the application you were running didn't support direct configuration. Would this mean that a malicious package could install a custom sslcustomize.py and so add unwanted certs to the system? I guess we have to assume that installed packages are trusted, but I just wanted to be explicit. Yes, it would have exactly the same security failure modes as sitecustomize, except it would only fire if the application imported the ssl module. The -S and -I switches would need to disable the implied sslcustomize, just as they disable import site. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 01.09.2014 08:44, Nick Coghlan wrote: Yes, it would have exactly the same security failure modes as sitecustomize, except it would only fire if the application imported the ssl module. The -S and -I switches would need to disable the implied sslcustomize, just as they disable import site. A malicious package can already play havoc with your installation with a custom ssl module. If somebody is able to sneak in a ssl.py then you are screwed anyway. sslcustomize is not going to make the situation worse. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 1 September 2014 17:13, Christian Heimes christ...@python.org wrote: On 01.09.2014 08:44, Nick Coghlan wrote: Yes, it would have exactly the same security failure modes as sitecustomize, except it would only fire if the application imported the ssl module. The -S and -I switches would need to disable the implied sslcustomize, just as they disable import site. A malicious package can already play havoc with your installation with a custom ssl module. If somebody is able to sneak in a ssl.py then you are screwed anyway. sslcustomize is not going to make the situation worse. That's not quite true - we're fairly careful about putting the standard library before userspace directories, so aside from the current directory problem, shadowing ssl itself can be tricky to arrange. sslcustomize would be more like sitecustomize - since it wouldn't normally be in the standard library, it can appear anywhere on sys.path, rather than having to be injected ahead of the standard library. I think that's OK though - compared to the security nightmare that is downloading modules from PyPI and running ./setup.py install (or, even worse, sudo ./setup.py install), this would be a rather esoteric attack vector, and the existing -S and -I mechanisms could be used to defend against it :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Le 01/09/2014 10:09, Nick Coghlan a écrit : On 1 September 2014 17:13, Christian Heimes christ...@python.org wrote: On 01.09.2014 08:44, Nick Coghlan wrote: Yes, it would have exactly the same security failure modes as sitecustomize, except it would only fire if the application imported the ssl module. The -S and -I switches would need to disable the implied sslcustomize, just as they disable import site. A malicious package can already play havoc with your installation with a custom ssl module. If somebody is able to sneak in a ssl.py then you are screwed anyway. sslcustomize is not going to make the situation worse. That's not quite true - we're fairly careful about putting the standard library before userspace directories, so aside from the current directory problem, shadowing ssl itself can be tricky to arrange. Not sure why. Just put another module named ssl in sys.modules directly. You can also monkeypatch the genuine ssl module. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou anto...@python.org wrote: Not sure why. Just put another module named ssl in sys.modules directly. You can also monkeypatch the genuine ssl module. That has to be done inside the same process. But imagine this scenario: You have a program that gets invoked as root (or some other user than yourself), and you're trying to fiddle with what it sees. You don't have root access, but you can manipulate the file system, to the extent that your userid has access. What can you do to affect this other program? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Mon, 1 Sep 2014 23:24:39 +1000 Chris Angelico ros...@gmail.com wrote: On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou anto...@python.org wrote: Not sure why. Just put another module named ssl in sys.modules directly. You can also monkeypatch the genuine ssl module. That has to be done inside the same process. But imagine this scenario: You have a program that gets invoked as root (or some other user than yourself), and you're trying to fiddle with what it sees. You don't have root access, but you can manipulate the file system, to the extent that your userid has access. What can you do to affect this other program? If you're root you shouldn't run untrusted code. See https://docs.python.org/3/using/cmdline.html#cmdoption-I Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Mon, Sep 1, 2014 at 11:34 PM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 1 Sep 2014 23:24:39 +1000 Chris Angelico ros...@gmail.com wrote: On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou anto...@python.org wrote: Not sure why. Just put another module named ssl in sys.modules directly. You can also monkeypatch the genuine ssl module. That has to be done inside the same process. But imagine this scenario: You have a program that gets invoked as root (or some other user than yourself), and you're trying to fiddle with what it sees. You don't have root access, but you can manipulate the file system, to the extent that your userid has access. What can you do to affect this other program? If you're root you shouldn't run untrusted code. See https://docs.python.org/3/using/cmdline.html#cmdoption-I Right, which is why sslcustomize has to be controlled by that, but the possibility of patching (or monkeypatching) ssl.py isn't as big a deal. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Mon, 1 Sep 2014 23:42:10 +1000 Chris Angelico ros...@gmail.com wrote: On Mon, Sep 1, 2014 at 11:34 PM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 1 Sep 2014 23:24:39 +1000 Chris Angelico ros...@gmail.com wrote: On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou anto...@python.org wrote: Not sure why. Just put another module named ssl in sys.modules directly. You can also monkeypatch the genuine ssl module. That has to be done inside the same process. But imagine this scenario: You have a program that gets invoked as root (or some other user than yourself), and you're trying to fiddle with what it sees. You don't have root access, but you can manipulate the file system, to the extent that your userid has access. What can you do to affect this other program? If you're root you shouldn't run untrusted code. See https://docs.python.org/3/using/cmdline.html#cmdoption-I Right, which is why sslcustomize has to be controlled by that, but the possibility of patching (or monkeypatching) ssl.py isn't as big a deal. To be frank I don't understand what you're arguing about. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 2 Sep 2014 00:08, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 1 Sep 2014 23:42:10 +1000 Chris Angelico ros...@gmail.com wrote: That has to be done inside the same process. But imagine this scenario: You have a program that gets invoked as root (or some other user than yourself), and you're trying to fiddle with what it sees. You don't have root access, but you can manipulate the file system, to the extent that your userid has access. What can you do to affect this other program? If you're root you shouldn't run untrusted code. See https://docs.python.org/3/using/cmdline.html#cmdoption-I Right, which is why sslcustomize has to be controlled by that, but the possibility of patching (or monkeypatching) ssl.py isn't as big a deal. To be frank I don't understand what you're arguing about. When I said shadowing ssl can be tricky to arrange, Chris correctly interpreted it as referring to the filesystem based privilege escalation scenario that isolated mode handles, not to normal in-process monkeypatching or module injection. I don't consider the latter cases to be interesting attack scenarios, as they imply the attacker is *already* running arbitrary Python code inside your CPython process, so you've already lost. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Tue, 2 Sep 2014 00:53:11 +1000 Nick Coghlan ncogh...@gmail.com wrote: On 2 Sep 2014 00:08, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 1 Sep 2014 23:42:10 +1000 Chris Angelico ros...@gmail.com wrote: That has to be done inside the same process. But imagine this scenario: You have a program that gets invoked as root (or some other user than yourself), and you're trying to fiddle with what it sees. You don't have root access, but you can manipulate the file system, to the extent that your userid has access. What can you do to affect this other program? If you're root you shouldn't run untrusted code. See https://docs.python.org/3/using/cmdline.html#cmdoption-I Right, which is why sslcustomize has to be controlled by that, but the possibility of patching (or monkeypatching) ssl.py isn't as big a deal. To be frank I don't understand what you're arguing about. When I said shadowing ssl can be tricky to arrange, Chris correctly interpreted it as referring to the filesystem based privilege escalation scenario that isolated mode handles, not to normal in-process monkeypatching or module injection. There's no actual difference. You can have a sitecustomize.py that does the monkeypatching or the shadowing. There doesn't seem to be anything tricky about that. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 2 Sep 2014 00:59, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 2 Sep 2014 00:53:11 +1000 Nick Coghlan ncogh...@gmail.com wrote: To be frank I don't understand what you're arguing about. When I said shadowing ssl can be tricky to arrange, Chris correctly interpreted it as referring to the filesystem based privilege escalation scenario that isolated mode handles, not to normal in-process monkeypatching or module injection. There's no actual difference. You can have a sitecustomize.py that does the monkeypatching or the shadowing. There doesn't seem to be anything tricky about that. Oh, now I get what you mean - yes, sitecustomize already poses the same kind of problem as the proposed sslcustomize (hence the existence of the related command line options). I missed that you had switched to talking about using that attack vector, rather than trying to shadow stdlib modules directly through the filesystem (which is the only tricky thing I was referring to). Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sep 1, 2014, at 11:35 AM, Nick Coghlan ncogh...@gmail.com wrote: On 2 Sep 2014 00:59, Antoine Pitrou solip...@pitrou.net mailto:solip...@pitrou.net wrote: On Tue, 2 Sep 2014 00:53:11 +1000 Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: To be frank I don't understand what you're arguing about. When I said shadowing ssl can be tricky to arrange, Chris correctly interpreted it as referring to the filesystem based privilege escalation scenario that isolated mode handles, not to normal in-process monkeypatching or module injection. There's no actual difference. You can have a sitecustomize.py that does the monkeypatching or the shadowing. There doesn't seem to be anything tricky about that. Oh, now I get what you mean - yes, sitecustomize already poses the same kind of problem as the proposed sslcustomize (hence the existence of the related command line options). I missed that you had switched to talking about using that attack vector, rather than trying to shadow stdlib modules directly through the filesystem (which is the only tricky thing I was referring to). Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Or you can just install something with easy_install, or you can drop a .pth file and monkey patch there. You can’t stop people from overriding modules, it’s trivial to do. The sys.path ordering just makes it slightly less trivial. — Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 01.09.2014 17:35, Nick Coghlan wrote: Oh, now I get what you mean - yes, sitecustomize already poses the same kind of problem as the proposed sslcustomize (hence the existence of the related command line options). If an attacker is able to place a module like sitecustomize.py in an import directory or any .pth file in a site-packages directory than this Python installation is compromised. .pth files are insidious because they are always loaded and their code is always executed. I don't see how sslcustomize is going to make a difference here. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sep 1, 2014, at 1:01 PM, Christian Heimes christ...@python.org wrote: On 01.09.2014 17:35, Nick Coghlan wrote: Oh, now I get what you mean - yes, sitecustomize already poses the same kind of problem as the proposed sslcustomize (hence the existence of the related command line options). If an attacker is able to place a module like sitecustomize.py in an import directory or any .pth file in a site-packages directory than this Python installation is compromised. .pth files are insidious because they are always loaded and their code is always executed. I don't see how sslcustomize is going to make a difference here. Right, this is the point I was trying to make. If you’ve installed a malicious package it’s game over. There’s nothing Python can do to help you. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 2 Sep 2014 03:08, Donald Stufft don...@stufft.io wrote: On Sep 1, 2014, at 1:01 PM, Christian Heimes christ...@python.org wrote: On 01.09.2014 17:35, Nick Coghlan wrote: Oh, now I get what you mean - yes, sitecustomize already poses the same kind of problem as the proposed sslcustomize (hence the existence of the related command line options). If an attacker is able to place a module like sitecustomize.py in an import directory or any .pth file in a site-packages directory than this Python installation is compromised. .pth files are insidious because they are always loaded and their code is always executed. I don't see how sslcustomize is going to make a difference here. Right, this is the point I was trying to make. If you’ve installed a malicious package it’s game over. There’s nothing Python can do to help you. Yes, that's what I said originally when pointing out that isolated mode and the switch to disable site module processing would need to disable sslcustomize processing as well. Antoine was replying to a side comment about it being tricky to shadow stdlib modules. I left out the qualifier directly in my original comment, and he left out indirectly through sitecustomize in his initial reply, so we were talking past each for a while. Cheers, Nick. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 12:21, R. David Murray rdmur...@bitdance.com wrote: On Sun, 31 Aug 2014 03:25:25 +0200, Antoine Pitrou solip...@pitrou.net wrote: On Sun, 31 Aug 2014 09:26:30 +1000 Nick Coghlan ncogh...@gmail.com wrote: In relation to changing the Python CLI API to offer some of the wget/curl style command line options, I like the idea of providing recipes in the docs for implementing them at the application layer, but postponing making the *default* behaviour configurable that way. I'm against any additional environment variables and command-line options. It will only complicate and obscure the security parameters of certificate validation. As Antoine says here, I'm also opposed to adding more Python specific configuration options. However, I think there may be something worthwhile we can do that's closer to the way browsers work, and has the significant benefit of being implementable as a PyPI module first (more on that in a separate reply). The existing knobs have already been mentioned in this thread, I won't mention them here again. Do those knobs allow one to instruct urllib to accept an invalid certificate without changing the program code? Only if you add the specific certificate concerned to the certificate store that Python is using (which PEP 476 currently suggests will be the platform wide certificate store). Whether or not that is an adequate solution is the point currently in dispute. My view is that the core problem/concern we need to address here is how we manage the migration away from a network communication model that trusts the network by default. That transition will happen regardless of whether or not we adapt Python as a platform - the challenge for us is how we can address it in a way that minimises the impact on existing users, while still ensuring future users are protected by default. This would be relatively easy if we only had to worry about the public internet (since we're followers rather than leaders in that environment), but we don't. Python made the leap into enterprise environments long ago, so we not only need to cope with corporate intranets, we need to cope with corporate intranets that aren't necessarily being well managed. That's what makes this a harder problem for us than it is for a new language like Go that was created by a public internet utility, specifically for use over the public internet - they didn't *have* an installed base to manage, they could just build a language specifically tailored for the task of running network services on Linux, without needing to account for any other use cases. The reason our existing installed base creates a problem is because corporate network security has historically focused on perimeter defence: carving out a trusted island behind the corporate firewall where users and other computer systems could be safely assumed not to be malicious. As an industry, we have learned though harsh experience that *this model doesn't work*. You can't trust the network, period. A corporate intranet is *less* dangerous than the public internet, but you still can't trust it. This don't trust the network ethos is also reinforced by the broad shift to utility computing where more and more companies are running distributed networks, where some of their systems are actually running on vendor provided servers. The network perimeter is evaporating, as corporate intranets start to look a lot more like recreations of the internet in miniature, with the only difference being the existence of more formal contractual relationships than typically exist between internet peers. Unfortunately, far too many organisations (especially those outside the tech industry) still trust in perimeter defence for their internal network security, and hence tolerate the use of unsecured connections, or skipping certificate validation internally. This is actually a really terrible idea, but it's still incredibly common due to the general failure of the technology industry to take usability issues seriously when we design security systems - doing the wrong unsafe thing is genuinely easier than doing things right. We have enough evidence now to be able to say (as Alex does in PEP 476) that it has been comprehensively demonstrated that opt-in security really just means security failures are common and silent by default. We've seen it with C buffer overflow vulnerabilities, we've seen it with plain text communication links, we've seen it with SSL certificate validation - the vast majority of users and developers will just run with the default behaviour of the platform or application they're using, even if those defaults have serious problems. As the saying goes, you can't document your way out of a usability problem - uncovered connections, or that are vulnerable to a man-in-the-middle attack appear to work for all functional purposes, they're just vulnerable to monitoring and subversion. It turns out opt-out security with a global off switch
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Aug 31, 2014, at 2:09 AM, Nick Coghlan ncogh...@gmail.com wrote: At the same time, we need to account for the fact that most existing organisations still trust in perimeter defence for their internal network security, and hence tolerate (or even actively encourage) the use of unsecured connections, or skipping certificate validation, internally. This is actually a really terrible idea, but it's still incredibly common due to the general failure of the technology industry to take usability issues seriously when we design security systems (at least until recently) - doing the wrong unsafe thing is genuinely easier than doing things right. Just a quick clarification in order to be a little clearer, this change will (obviously) only effect those who trust perimeter security *and* decided to install an invalid certificate instead of just using HTTP. I'm not saying that this doesn't happen, just being specific (I'm not actually sure why they would install a TLS certificate at all if they are trusting perimeter security, but I'm sure folks do). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 12:21, R. David Murray rdmur...@bitdance.com wrote: Do those knobs allow one to instruct urllib to accept an invalid certificate without changing the program code? My first reply ended up being a context dump of the challenges created by legacy corporate intranets that may not be immediately obvious to folks that spend most of their time working on or with the public internet. I decided to split these more technical details out to a new reply for the benefit of folks that already know all that history :) To answer David's specific question, the existing knobs at the OpenSSL level (SSL_CERT_DIR and SSL_CERT_FILE ) let people add an internal CA, opt out of the default CA system, and trust *specific* self-signed certs. What they don't allow is a global trust any cert setting - exceptions need to be added at the individual cert level or at the CA level, or the application needs to offer an option to not do cert validation at all. That trust anything option at the platform level is the setting that is a really bad idea - if an organisation thinks it needs that (because they have a lot of self-signed certs, but aren't verifying their HTTPS connections to those servers), then what they really need is an internal CA, where their systems just need to be set up to trust the internal CA in addition to the platform CA certs. With Alex's proposal, organisations that are already running an internal CA should be just fine - Python 3.5 will see the CA cert in the platform cert store and accept certs signed by it as valid. (Note: the Python 3.4 warning should take this into account, which could be a problem since we don't currently do validity checks against the platform store by default. The PEP needs to cover the mechanics of that in more detail, as I think it means we'll need to make *some* changes to the default configuration even in Python 3.4 to get accurate validity data back from OpenSSL) However, we also need to accept that there's a reason browser vendors still offer click through insecurity for sites with self-signed certificates, and tools like wget/curl offer the option to say don't check the certificate: these are necessary compromises to make SSL based network connections actually work on many current corporate intranets. It is corporate environments that also make it desirable to be able to address this potential problem at a *user* level, since many Python users in a large organisations are actually running Python entirely out of their home directories, rather than as a system installation (they may not even have admin access to their own systems). My suggestion at this point is that we take a leaf from both browser vendors and the design of SSH: make it easy to *add* a specific self-signed cert to the set a *particular user* trusts by default (preferably *only* for a particular host, to limit the power of such certs). python -m ssl doesn't currently do anything interesting, so it could be used to provide an API for managing that user level certificate store. A Python-specific user level cert store is something that could be developed as a PyPI library for Python 2.7.9+ and 3.4+ (Is cert management considered in scope for cryptography.io? If so, that could be a good home). So while I agree with the intent of PEP 476, and like the suggested end state, I'm back to thinking that the transition plan for existing corporate users needs more work before it can be accepted. This is especially true since it becomes another barrier to migrating from Python 2.7 to Python 3.5+ (a warning in Python 3.4 doesn't help with that aspect, although a new -3 warning might). A third party module that offers a user level certificate store, and a gevent.monkey style way of opting in to this behaviour for existing Python versions would be one way to provide a more compelling transition plan. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 16:16, Donald Stufft don...@stufft.io wrote: On Aug 31, 2014, at 2:09 AM, Nick Coghlan ncogh...@gmail.com wrote: At the same time, we need to account for the fact that most existing organisations still trust in perimeter defence for their internal network security, and hence tolerate (or even actively encourage) the use of unsecured connections, or skipping certificate validation, internally. This is actually a really terrible idea, but it's still incredibly common due to the general failure of the technology industry to take usability issues seriously when we design security systems (at least until recently) - doing the wrong unsafe thing is genuinely easier than doing things right. Just a quick clarification in order to be a little clearer, this change will (obviously) only effect those who trust perimeter security *and* decided to install an invalid certificate instead of just using HTTP. I'm not saying that this doesn't happen, just being specific (I'm not actually sure why they would install a TLS certificate at all if they are trusting perimeter security, but I'm sure folks do). It's the end result when a company wide edict to use HTTPS isn't backed up by the necessary documentation and training on how to get a properly signed cert from your internal CA (or, even better, when such an edict comes down without setting up an internal CA first). Folks hit the internet instead, find instructions on creating a self-signed cert, install that, and tell their users to ignore the security warning and accept the cert. Historically, Python clients have just worked in environments that required a click-through on the browser side, since you had to opt in to checking the certificates properly. Self-signed certificates can also be really handy for doing local testing - you're not really aiming to authenticate the connection in that case, you're just aiming to test that the secure connection machinery is all working properly. (As far as the what about requests? question goes - that's in a similar situation to Go, where being new allows it to choose different defaults, and folks for whom those defaults don't work just won't use it. There's also the fact that most corporate Python users are unlikely to know that PyPI exists, let alone that it contains a module called requests that does SSL certificate validation by default. Those of us in the corporate world that interact directly with upstream are still the exception rather than the rule) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 07:45, Nick Coghlan ncogh...@gmail.com wrote: There's also the fact that most corporate Python users are unlikely to know that PyPI exists, let alone that it contains a module called requests that does SSL certificate validation by default. Those of us in the corporate world that interact directly with upstream are still the exception rather than the rule) I think this point deserves just a little bit more emphasis. This is why any solution that begins with 'use PyPI' is insufficient. I've worked on requests for 3 years now and most of my colleagues have never heard of it, and it's not because I don't talk about it (I talk about it all the time!). When building internal tools, corporate environments frequently restrict themselves to the standard library. This is because it's hard enough to get adoption of a tool when it requires a new language runtime, let alone if you have to get people ramped up on package distribution as well! I have enough trouble getting people to upgrade Python versions at work: trying to get them up to speed on pip and PyPI is worse. It is no longer tenable in the long term for Python to trust the network: you're right in this regard Nick. In the past, on this very list, I've been bullish about fixing up Python's network security position. I was an aggressive supporter of PEP 466 (and there are some corners of PEP 466 that I think didn't go far enough). However, I'm with you here: we should do this once and do it right. Corporate users *will* bump into it, and they will look to the docs to fix it. That fix needs to be easy and painless. A user-level cert store is a good start, and if cryptography.io aren't interested in it I might take a look at implementing it under the certifi.io umbrella instead. Cory ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 30.08.2014 00:22, Antoine Pitrou wrote: SSL_CERT_DIR and SSL_CERT_FILE are used, if set, when SSLContext.load_verify_locations() is called. Actually, come to think of it, this allows us to write a better test for that method. Patch welcome! The environment vars are used only when SSLContext.set_default_verify_paths() is called. load_verify_locations() loads certificates from a given file, directory or memory but it doesn't look at the env vars. create_default_context() calls SSLContext.load_default_certs() when neither cafile, capath nor cadata is given as an argument. SSLContext.load_default_certs() then calls SSLContext.set_default_verify_paths(). However there is a catch: SSLContext.set_default_verify_paths() is not called on Windows. In retrospective it was a bad decision by me to omit the call. http://hg.python.org/cpython/file/164a17eca081/Lib/ssl.py#l376 Christian PS: SSL_CERT_DIR and SSL_CERT_FILE are the default names. It's possible to change the names in OpenSSL. ssl.get_default_verify_paths() returns the names and paths to the default verify locations. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sun, 31 Aug 2014 16:45:42 +1000, Nick Coghlan ncogh...@gmail.com wrote: On 31 August 2014 16:16, Donald Stufft don...@stufft.io wrote: On Aug 31, 2014, at 2:09 AM, Nick Coghlan ncogh...@gmail.com wrote: At the same time, we need to account for the fact that most existing organisations still trust in perimeter defence for their internal network security, and hence tolerate (or even actively encourage) the use of unsecured connections, or skipping certificate validation, internally. This is actually a really terrible idea, but it's still incredibly common due to the general failure of the technology industry to take usability issues seriously when we design security systems (at least until recently) - doing the wrong unsafe thing is genuinely easier than doing things right. Just a quick clarification in order to be a little clearer, this change will (obviously) only effect those who trust perimeter security *and* decided to install an invalid certificate instead of just using HTTP. I'm not saying that this doesn't happen, just being specific (I'm not actually sure why they would install a TLS certificate at all if they are trusting perimeter security, but I'm sure folks do). It's the end result when a company wide edict to use HTTPS isn't backed up by the necessary documentation and training on how to get a properly signed cert from your internal CA (or, even better, when such an edict comes down without setting up an internal CA first). Folks hit the internet instead, find instructions on creating a self-signed cert, install that, and tell their users to ignore the security warning and accept the cert. Historically, Python clients have just worked in environments that required a click-through on the browser side, since you had to opt in to checking the certificates properly. Self-signed certificates can also be really handy for doing local testing - you're not really aiming to authenticate the connection in that case, you're just aiming to test that the secure connection machinery is all working properly. Self-signed certificates are not crazy in an internal corporate environment even when properly playing the defense in depth game. Once you've acked the cert the first time, you will be warned if it changes (like an ssh host key). Sure, as Nick says the corp could set up an internal signing authority and make sure everyone has their CA...and they *should*...but realistically, that is probably relatively rare at the moment, because it is not particularly easy to accomplish (distributing the CA everywhere it needs to go is still a Hard Problem, though it has gotten a lot better). Given the reality of human nature, even when the documentation accompanying the HTTPS initiative is good, there will *still* be someone who hasn't followed the internal rules, yet you really need to talk to the piece of infrastructure they are maintaining. At least that one is short term problem (for some definition of short that may be several months long), but it does exist. In addition, as has been mentioned before, self-signed certs are often embedded in *devices* from vendors (I'm looking at you, Cisco). This is another area where security conciousness has gotten better (the cert exists) but isn't good yet (the cert is self-signed and replacing it isn't trivial when it is even possible; and, because the self-signed cert happens by defaultit gets left in place). And in the case of those embedded certs, the cert can wind up *invalid* (expired) as well as self-signed. (This last item is where my concern about being able to talk to invalid certs comes from.) And yes, I have encountered all of this in the wild. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 16:16, R. David Murray wrote: Self -signed certificates are not crazy in an internal corporate environment even when properly playing the defense in depth game. Once you've acked the cert the first time, you will be warned if it changes (like an ssh host key). Sure, as Nick says the corp could set up an internal signing authority and make sure everyone has their CA...and they *should*...but realistically, that is probably relatively rare at the moment, because it is not particularly easy to accomplish (distributing the CA everywhere it needs to go is still a Hard Problem, though it has gotten a lot better). It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store. That's all. A self-signed certificate acts as its own root CA (so to speak). But there is a downside, too. The certificate is trusted for any and all connections. Python's SSL module has no way to trust a specific certificate for a host. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 17:27, Christian Heimes christ...@python.org wrote: It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store. Stuff it into the trust store is the hard bit, though. I have honestly no idea how to do that. Or if it's temporary (which it likely is) how to manage it - delete it when I no longer need it, list what junk I've added over time, etc. And equally relevantly, no idea how to do that in a way that won't clash with my company's policies... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 08:24, Nick Coghlan wrote: To answer David's specific question, the existing knobs at the OpenSSL level (SSL_CERT_DIR and SSL_CERT_FILE ) let people add an internal CA, opt out of the default CA system, and trust *specific* self-signed certs. This works only on Unix platforms iff SSL_CERT_DIR and SSL_CERT_FILE are both set to a non-empty string that points to non-existing files or something like /dev/null. On Windows my enhancement will always cause the system trust store to kick in. There is currently no way to disable the Windows system store for ssl.create_default_context() and ssl._create_stdlib_context() with the functions' default arguments. On Mac OS X the situation is even more obscure. Apple's OpenSSL binaries are using Apple's Trust Evaluation Agent. You have to set OPENSSL_X509_TEA_DISABLE=1 in order to prevent the agent from adding trusted certs from OSX key chain. Hynek Schlawack did a deep dive into it. https://hynek.me/articles/apple-openssl-verification-surprises/ A Python-specific user level cert store is something that could be developed as a PyPI library for Python 2.7.9+ and 3.4+ (Is cert management considered in scope for cryptography.io? If so, that could be a good home). Python's SSL module is lacking some functionalities in order to implement a fully functional cert store. * no verify hook to verify each certificate in the chain like https://www.openssl.org/docs/ssl/SSL_CTX_set_cert_verify_callback.html http://linux.die.net/man/3/x509_store_ctx_set_verify_cb /api/ssl.html#OpenSSL.SSL.Context.set_verify * no way to get the full cert chain including the root certificate. * no API to get the subject public key information (SPKI). The SPKI hash can be used to identify a certificate. For example it's used in Google's CRLSet. http://dev.chromium.org/Home/chromium-security/crlsets * the cert validation exception could use some additional information. There are probably some more things mising. An X509 object would help, too. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Le 31/08/2014 19:03, Paul Moore a écrit : On 31 August 2014 17:27, Christian Heimes christ...@python.org wrote: It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store. Stuff it into the trust store is the hard bit, though. I have honestly no idea how to do that. You certainly shouldn't do so. If an application has special needs that require trusting a self-signed certificate, then it should expose a configuration setting to let users specify the cert's location. Stuffing self-signed certs into the system trust store is really a measure of last resort. There's another case which isn't solved by this, though, which is when a cert is invalid. The common situation being that it has expired (renewing certs is a PITA and therefore expired certs are more common than it sounds they should be). In this case, there is no way to whitelist it: you have to disable certificate checking altogether. This can be exposed by the application as configuration option if necessary, as well. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 18:29, Antoine Pitrou anto...@python.org wrote: If an application has special needs that require trusting a self-signed certificate, then it should expose a configuration setting to let users specify the cert's location. I can't see how that would be something the application would know. For example, pip allows me to specify an alternate cert bundle but not a single additional cert. So IIUC, I can't use my local index that serves https using a self-signed cert. I'd find it hard to argue that it's pip's responsibility to think of that use case - pretty much any program that interacts with a web service *might* need to interact with a self-signed dummy version, if only under test conditions. Or did you mean that Python should provide such a setting that would cover all applications written in Python? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Le 31/08/2014 20:28, Paul Moore a écrit : I can't see how that would be something the application would know. For example, pip allows me to specify an alternate cert bundle but not a single additional cert. So IIUC, I can't use my local index that serves https using a self-signed cert. I'd find it hard to argue that it's pip's responsibility to think of that use case - pretty much any program that interacts with a web service *might* need to interact with a self-signed dummy version, if only under test conditions. Well, it's certainly pip's responsibility more than Python's. What would Python do? Provide a setting that would blindly add a cert for all uses of httplib? pip knows about the use cases here, Python doesn't. (perhaps you want to serve your local index using http, though) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 19:37, Antoine Pitrou anto...@python.org wrote: Well, it's certainly pip's responsibility more than Python's. What would Python do? Provide a setting that would blindly add a cert for all uses of httplib? That's more or less my point, pip doesn't have that much better idea than Python. I was talking about putting the cert in my local cert store, so that *I* can decide, and applications don't need to take special care to allow me to handle this case. You said that doing so was bad, but I don't see why. It seems to me that you're saying that I should raise a feature request for pip instead, which seems unreasonable. Am I missing something? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Le 31/08/2014 21:12, Paul Moore a écrit : On 31 August 2014 19:37, Antoine Pitrou anto...@python.org wrote: Well, it's certainly pip's responsibility more than Python's. What would Python do? Provide a setting that would blindly add a cert for all uses of httplib? That's more or less my point, pip doesn't have that much better idea than Python. I was talking about putting the cert in my local cert store, so that *I* can decide, and applications don't need to take special care to allow me to handle this case. What do you call your local cert store? If you mean the system cert store, then that will affect all users. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 19:29, Antoine Pitrou wrote: You certainly shouldn't do so. If an application has special needs that require trusting a self-signed certificate, then it should expose a configuration setting to let users specify the cert's location. Stuffing self-signed certs into the system trust store is really a measure of last resort. Correct! I merely wanted to state that OpenSSL can verify a self-signed certificate easily. The certificate 'just' have to be added to the SSLContext's store of trusted root certs. Somebody has to figure out how Python can accomplish the task. There's another case which isn't solved by this, though, which is when a cert is invalid. The common situation being that it has expired (renewing certs is a PITA and therefore expired certs are more common than it sounds they should be). In this case, there is no way to whitelist it: you have to disable certificate checking altogether. This can be exposed by the application as configuration option if necessary, as well. It's possible to ignore errors with a verify callback. OpenSSL's wiki has an example for the expired certs http://wiki.openssl.org/index.php/Manual:X509_STORE_CTX_set_verify_cb%283%29#EXAMPLES Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 21:15, Antoine Pitrou anto...@python.org wrote: What do you call your local cert store? I was referring to Christian's comment It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store. From his recent response, I guess he meant the system store, and he agrees that this is a bad option. OK, that's fair, but: a) Is there really no OS-level personal trust store? I'm thinking of Windows here for my own personal use, but the same question applies elsewhere. b) I doubt my confusion over Christian's response is atypical. Based on what he said, if we hadn't had the subsequent discussion, I would probably have found a way to add a cert to the store without understanding the implications. While it's not Python's job to educate users, it would be a shame if its default behaviour led people to make ill-informed decisions. Maybe an SSL HOWTO would be a useful addition to the docs, if anyone feels motivated to write one. Regardless, thanks for the education! Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 1 Sep 2014 06:32, Paul Moore p.f.mo...@gmail.com wrote: On 31 August 2014 21:15, Antoine Pitrou anto...@python.org wrote: What do you call your local cert store? I was referring to Christian's comment It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store. From his recent response, I guess he meant the system store, and he agrees that this is a bad option. OK, that's fair, but: a) Is there really no OS-level personal trust store? I'm thinking of Windows here for my own personal use, but the same question applies elsewhere. b) I doubt my confusion over Christian's response is atypical. Based on what he said, if we hadn't had the subsequent discussion, I would probably have found a way to add a cert to the store without understanding the implications. While it's not Python's job to educate users, it would be a shame if its default behaviour led people to make ill-informed decisions. Right, this is why I came to the conclusion we need to follow the browser vendors lead here and support a per-user Python specific supplementary certificate cache before we can start validating certs by default at the *Python* level. There are still too many failure modes for cert management on private networks for us to safely ignore the use case of needing to force connections to services with invalid certs. We don't need to *solve* that problem here today - we can push it back to Alex (and anyone else interested) as a building block to investigate providing as part of cryptography.io or certi.fi, with a view to making a standard library version of that (along with any SSL module updates) part of PEP 476. In the meantime, we can update the security considerations for the ssl module to make it clearer that the defaults are set up for trusted networks and that using it safely on the public internet may mean you're better off with a third party library like requests or Twisted. (I'll start another thread shortly that is highly relevant to that topic) Regards, Nick. Maybe an SSL HOWTO would be a useful addition to the docs, if anyone feels motivated to write one. Regardless, thanks for the education! Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 08:09, Nick Coghlan wrote: As Antoine says here, I'm also opposed to adding more Python specific configuration options. However, I think there may be something worthwhile we can do that's closer to the way browsers work, and has the significant benefit of being implementable as a PyPI module first (more on that in a separate reply). I'm on your and Antoine's side and strictly against any additional environment variables or command line arguments. That would make the whole validation process even more complex and harder to understand. There might be a better option to give people and companies the option to tune the SSL module to their needs. Python already have a customization hook for the site module called sitecustomize. How about another module named sslcustomize? Such a module could be used to tune the ssl module to the needs of users, e.g. configure a different default context, add certificates to a default context etc. Companies could install them in a system global directory on their servers. Users could put them in their own user site directory and even each virtual env can have one sslcustomize of its own. It's fully backward compatible, doesn't add any flags and developers have the full power of Python for configuration and customization. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Le 31/08/2014 23:41, Nick Coghlan a écrit : Right, this is why I came to the conclusion we need to follow the browser vendors lead here and support a per-user Python specific supplementary certificate cache before we can start validating certs by default at the *Python* level. There are still too many failure modes for cert management on private networks for us to safely ignore the use case of needing to force connections to services with invalid certs. We are not ignoring that use case. The proper solution is simply to disable cert validation in the application code (or, for more sophisticated needs, provide an application configuration setting for cert validation). In the meantime, we can update the security considerations for the ssl module to make it clearer that the defaults are set up for trusted networks and that using it safely on the public internet may mean you're better off with a third party library like requests or Twisted. No, you simply have to select the proper validation settings. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 22:30, Paul Moore wrote: On 31 August 2014 21:15, Antoine Pitrou anto...@python.org wrote: What do you call your local cert store? I was referring to Christian's comment It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store. I was referring to the the trust store of the SSLContext object and not to any kind of cert store of the operating system. Sorry for the confusion. a) Is there really no OS-level personal trust store? I'm thinking of Windows here for my own personal use, but the same question applies elsewhere. Windows and OSX have superior cert stores compared to Linux and BSD. They have means for user and system wide cert stores and trust settings Linux just have one central directory or file with all trusted certs. My KDE has some options to disable certs but I don't know how to make use of the configuration. Even worse: Linux distros doesn't make a different between purposes. On Windows a user can trust a certificate for S/MIME but not for server auth or client auth. Ubuntu just puts all certification in one directory but it's wrong. :( https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1207004 Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 1 Sep 2014 07:43, Christian Heimes christ...@python.org wrote: On 31.08.2014 08:09, Nick Coghlan wrote: As Antoine says here, I'm also opposed to adding more Python specific configuration options. However, I think there may be something worthwhile we can do that's closer to the way browsers work, and has the significant benefit of being implementable as a PyPI module first (more on that in a separate reply). I'm on your and Antoine's side and strictly against any additional environment variables or command line arguments. That would make the whole validation process even more complex and harder to understand. There might be a better option to give people and companies the option to tune the SSL module to their needs. Python already have a customization hook for the site module called sitecustomize. How about another module named sslcustomize? Such a module could be used to tune the ssl module to the needs of users, e.g. configure a different default context, add certificates to a default context etc. Companies could install them in a system global directory on their servers. Users could put them in their own user site directory and even each virtual env can have one sslcustomize of its own. It's fully backward compatible, doesn't add any flags and developers have the full power of Python for configuration and customization. And means a user specific store (if one became available) could be configured there. Yes, I think this would address my concerns, especially if combined with a clear recipe in the documentation on how to optionally disable cert validation at the application layer. Assuming sslcustomize was in site-packages rather than the standard library directories, you would also be able to use virtual environments with an appropriate sslcustomize module to disable cert checking even if the application you were running didn't support direct configuration. Cheers, Nick. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Aug 31, 2014, at 5:43 PM, Christian Heimes christ...@python.org wrote: On 31.08.2014 08:09, Nick Coghlan wrote: As Antoine says here, I'm also opposed to adding more Python specific configuration options. However, I think there may be something worthwhile we can do that's closer to the way browsers work, and has the significant benefit of being implementable as a PyPI module first (more on that in a separate reply). I'm on your and Antoine's side and strictly against any additional environment variables or command line arguments. That would make the whole validation process even more complex and harder to understand. There might be a better option to give people and companies the option to tune the SSL module to their needs. Python already have a customization hook for the site module called sitecustomize. How about another module named sslcustomize? Such a module could be used to tune the ssl module to the needs of users, e.g. configure a different default context, add certificates to a default context etc. Companies could install them in a system global directory on their servers. Users could put them in their own user site directory and even each virtual env can have one sslcustomize of its own. It's fully backward compatible, doesn't add any flags and developers have the full power of Python for configuration and customization. This may be a dumb question, but why can’t sitecustomize do this already? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 1 Sep 2014 08:15, Donald Stufft don...@stufft.io wrote: On Aug 31, 2014, at 5:43 PM, Christian Heimes christ...@python.org wrote: Companies could install them in a system global directory on their servers. Users could put them in their own user site directory and even each virtual env can have one sslcustomize of its own. It's fully backward compatible, doesn't add any flags and developers have the full power of Python for configuration and customization. This may be a dumb question, but why can’t sitecustomize do this already? It can. The advantage of a separate file is that it won't conflict with existing sitecustomize modules, so (for example) redistributors can add a default sslcustomize, and you can add one to your virtual environments that are integrated with the system Python environment without needing to worry about whether or not there's a global sitecustomize (you'd only have trouble if there was a global sslcustomize). Cheers, Nick. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Mon, 01 Sep 2014 08:10:58 +1000, Nick Coghlan ncogh...@gmail.com wrote: On 1 Sep 2014 07:43, Christian Heimes christ...@python.org wrote: On 31.08.2014 08:09, Nick Coghlan wrote: As Antoine says here, I'm also opposed to adding more Python specific configuration options. However, I think there may be something worthwhile we can do that's closer to the way browsers work, and has the significant benefit of being implementable as a PyPI module first (more on that in a separate reply). I'm on your and Antoine's side and strictly against any additional environment variables or command line arguments. That would make the whole validation process even more complex and harder to understand. There might be a better option to give people and companies the option to tune the SSL module to their needs. Python already have a customization hook for the site module called sitecustomize. How about another module named sslcustomize? Such a module could be used to tune the ssl module to the needs of users, e.g. configure a different default context, add certificates to a default context etc. Companies could install them in a system global directory on their servers. Users could put them in their own user site directory and even each virtual env can have one sslcustomize of its own. It's fully backward compatible, doesn't add any flags and developers have the full power of Python for configuration and customization. And means a user specific store (if one became available) could be configured there. Yes, I think this would address my concerns, especially if combined with a clear recipe in the documentation on how to optionally disable cert validation at the application layer. Assuming sslcustomize was in site-packages rather than the standard library directories, you would also be able to use virtual environments with an appropriate sslcustomize module to disable cert checking even if the application you were running didn't support direct configuration. It sounds like this would address my concerns as well (I don't really care *how* it is implemented as long as I don't have to touch the code of a third party application when I upgrade my python version to 3.5...remember, the context here is backward compatibility concerns). Does it address the issue of accepting an invalid cert, though? --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 30.08.2014 04:44, Alex Gaynor wrote: Thanks for the rapid feedback everyone! I want to summarize the action items and discussion points that have come up so far: To add to the PEP: * Emit a warning in 3.4.next for cases that would raise a Exception in 3.5 * Clearly state that the existing OpenSSL environment variables will be respected for setting the trust root I'd also suggest to compile Python with OPENSSL_LOAD_CONF, since that causes OpenSSL to read the global openssl.cnf file for additional configuration. Discussion points: * Disabling verification entirely externally to the program, through a CLI flag or environment variable. I'm pretty down on this idea, the problem you hit is that it's a pretty blunt instrument to swing, and it's almost impossible to imagine it not hitting things it shouldn't; it's far too likely to be used in applications that make two sets of outbound connections: 1) to some internal service which you want to disable verification on, and 2) some external service which needs strong validation. A global flag causes the latter to fail silently when subjected to a MITM attack, and that's exactly what we're trying to avoid. It also makes things much harder for library authors: I write an API client for some API, and make TLS connections to it. I want those to be verified by default. I can't even rely on the httplib defaults, because someone might disable them from the outside. The reasoning here is the same as for hash randomization. There are cases where you want to test your application using self-signed certificates which don't validate against the system CA root list. In those cases, you do know what you're doing. The test would fail otherwise and the reason is not a bug in your code, it's just the fact that the environment you're running it in is a test environment. Ideally, all applications should give you this choice, but this is unlikely to happen, so it's good to be able to change the Python default, since with the proposed change, most applications will probably continue to use the Python defaults as they do now. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 30 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ...20 days to go 2014-09-27: PyDDF Sprint 2014 ... 28 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sat, 30 Aug 2014 12:19:11 +0200 M.-A. Lemburg m...@egenix.com wrote: To add to the PEP: * Emit a warning in 3.4.next for cases that would raise a Exception in 3.5 * Clearly state that the existing OpenSSL environment variables will be respected for setting the trust root I'd also suggest to compile Python with OPENSSL_LOAD_CONF, since that causes OpenSSL to read the global openssl.cnf file for additional configuration. Python links against OpenSSL as a shared library, not statically. It's unlikely that setting a compile constant inside Python would affect OpenSSL at all. Discussion points: * Disabling verification entirely externally to the program, through a CLI flag or environment variable. I'm pretty down on this idea, the problem you hit is that it's a pretty blunt instrument to swing, and it's almost impossible to imagine it not hitting things it shouldn't; it's far too likely to be used in applications that make two sets of outbound connections: 1) to some internal service which you want to disable verification on, and 2) some external service which needs strong validation. A global flag causes the latter to fail silently when subjected to a MITM attack, and that's exactly what we're trying to avoid. It also makes things much harder for library authors: I write an API client for some API, and make TLS connections to it. I want those to be verified by default. I can't even rely on the httplib defaults, because someone might disable them from the outside. The reasoning here is the same as for hash randomization. There are cases where you want to test your application using self-signed certificates which don't validate against the system CA root list. That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE env vars (or, better, by specific settings *inside* the application). I'm against multiplying environment variables, as it makes it more difficult to assess the actual security of a setting. The danger of an ill-secure setting is much more severe than with hash randomization. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 30.08.2014 12:40, Antoine Pitrou wrote: On Sat, 30 Aug 2014 12:19:11 +0200 M.-A. Lemburg m...@egenix.com wrote: To add to the PEP: * Emit a warning in 3.4.next for cases that would raise a Exception in 3.5 * Clearly state that the existing OpenSSL environment variables will be respected for setting the trust root I'd also suggest to compile Python with OPENSSL_LOAD_CONF, since that causes OpenSSL to read the global openssl.cnf file for additional configuration. Python links against OpenSSL as a shared library, not statically. It's unlikely that setting a compile constant inside Python would affect OpenSSL at all. The change is to the OpenSSL API, not the OpenSSL lib. By setting the variable you enable a few special calls to the config loader functions in OpenSSL when calling the initializer it: https://www.openssl.org/docs/crypto/OPENSSL_config.html Discussion points: * Disabling verification entirely externally to the program, through a CLI flag or environment variable. I'm pretty down on this idea, the problem you hit is that it's a pretty blunt instrument to swing, and it's almost impossible to imagine it not hitting things it shouldn't; it's far too likely to be used in applications that make two sets of outbound connections: 1) to some internal service which you want to disable verification on, and 2) some external service which needs strong validation. A global flag causes the latter to fail silently when subjected to a MITM attack, and that's exactly what we're trying to avoid. It also makes things much harder for library authors: I write an API client for some API, and make TLS connections to it. I want those to be verified by default. I can't even rely on the httplib defaults, because someone might disable them from the outside. The reasoning here is the same as for hash randomization. There are cases where you want to test your application using self-signed certificates which don't validate against the system CA root list. That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE env vars (or, better, by specific settings *inside* the application). I'm against multiplying environment variables, as it makes it more difficult to assess the actual security of a setting. The danger of an ill-secure setting is much more severe than with hash randomization. You have a point there. So how about just a python run-time switch and no env var ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 30 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ...20 days to go 2014-09-27: PyDDF Sprint 2014 ... 28 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
30 August 2014 03:44, Alex Gaynor alex.gay...@gmail.com wrote: Discussion points: * Disabling verification entirely externally to the program, through a CLI flag or environment variable. I'm pretty down on this idea, the problem you hit is that it's a pretty blunt instrument to swing, and it's almost impossible to imagine it not hitting things it shouldn't As a data point, I use --no-check-certificates extensively, in wget, curl and some Python programs which have it, like youtube-dl. The reason I do so is typically because the programs do not use the Windows cerificate store, and configuring a second certificate store on a per-program basis is too much of a pain to be worth it (per-program because the hacks such programs use to get round the fact that Windows has no central location like /etc are inconsistent). The key question for me is therefore, does Python's ssl support use the Windows store directly these days? I checked the docs and couldn't find anything explicitly stating this (but all the terminology is foreign to me, so I may have missed it). If it does, programs like youtube-dl will start to just work and I won't have the need for a switch off everything flag. If a new Python 3.5 installation on a Windows machine will enforce https cert checking and yet will not check the system store (or, I guess, come with an embedded store, but aren't there maintenance issues with doing that?) then I believe a global don't check flag will be needed, as not all programs offer a don't check certificates mode. And naive users like me may not even know how to code the behaviour for such an option - and the tone of the debate here leads me to believe that it'll be hard for developers to get unbiased advice on how to switch off checking, so it'll end up being patchily implemented. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sat, 30 Aug 2014 12:46:47 +0200 M.-A. Lemburg m...@egenix.com wrote: The change is to the OpenSSL API, not the OpenSSL lib. By setting the variable you enable a few special calls to the config loader functions in OpenSSL when calling the initializer it: https://www.openssl.org/docs/crypto/OPENSSL_config.html Ah, ok. Do you have experience with openssl.cnf? Apparently, it is meant for offline tools such as certificate generation, I am not sure how it could impact certification validation. That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE env vars (or, better, by specific settings *inside* the application). I'm against multiplying environment variables, as it makes it more difficult to assess the actual security of a setting. The danger of an ill-secure setting is much more severe than with hash randomization. You have a point there. So how about just a python run-time switch and no env var ? Well, why not, but does it have a value over letting the code properly configure their SSLContext? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 30.08.2014 12:55, Antoine Pitrou wrote: On Sat, 30 Aug 2014 12:46:47 +0200 M.-A. Lemburg m...@egenix.com wrote: The change is to the OpenSSL API, not the OpenSSL lib. By setting the variable you enable a few special calls to the config loader functions in OpenSSL when calling the initializer it: https://www.openssl.org/docs/crypto/OPENSSL_config.html Ah, ok. Do you have experience with openssl.cnf? Apparently, it is meant for offline tools such as certificate generation, I am not sure how it could impact certification validation. I'm still exploring this: the OpenSSL documentation is, well, less than complete on these things, so searching mailing lists and reading source code appears to be the only reasonable way to figure out what is possible and what not. The openssl.cnf config file is indeed mostly used by the various openssl subcommands (e.g. req and ca), but it can also be used to configuring engines and my hope is that configuration of e.g. default certificate stores also becomes possible. One of the engines can tap into the Windows certificate store, for example. That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE env vars (or, better, by specific settings *inside* the application). I'm against multiplying environment variables, as it makes it more difficult to assess the actual security of a setting. The danger of an ill-secure setting is much more severe than with hash randomization. You have a point there. So how about just a python run-time switch and no env var ? Well, why not, but does it have a value over letting the code properly configure their SSLContext? Yes, because when Python changes the default to be validating and more secure, application developers will do the same as they do now: simply use the defaults ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 30 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ...20 days to go 2014-09-27: PyDDF Sprint 2014 ... 28 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sat, 30 Aug 2014 14:03:57 +0200, M.-A. Lemburg m...@egenix.com wrote: On 30.08.2014 12:55, Antoine Pitrou wrote: On Sat, 30 Aug 2014 12:46:47 +0200 M.-A. Lemburg m...@egenix.com wrote: That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE env vars (or, better, by specific settings *inside* the application). I'm against multiplying environment variables, as it makes it more difficult to assess the actual security of a setting. The danger of an ill-secure setting is much more severe than with hash randomization. You have a point there. So how about just a python run-time switch and no env var ? Well, why not, but does it have a value over letting the code properly configure their SSLContext? Yes, because when Python changes the default to be validating and more secure, application developers will do the same as they do now: simply use the defaults ;-) But neither of those addresses the articulated use case: someone *using* a program implemented in python that does not itself provide a way to disable the new default security (because it is *new*). Only an environment variable will do that. Since the environment variable is opt-in, I think the consenting adults argument applies to Alex's demure about multiple connections. It could still emit the warnings. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
