Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 24 January 2014 03:06, Stephen J. Turnbull step...@xemacs.org wrote: Are you kidding? These *aren't* the apps that I care about breaking, and I know that the PHBs won't pay attention to what I say about fixing their sites and cert chains (believe me, I've tried, and the answer is as Paul Moore says: the users can punch the go ahead anyway button, what's the big deal here?), they'll just deprecate Python. Surely the solution here isn't to say well then, let's be insecure by default, it's to provide a go ahead anyway button. That at least lets us push the choice to be insecure by default onto someone else. The idea that an enterprise will deprecate Python instead of adding a single environment variable or one line at the top of their scripts seems hugely unlikely. As an example, Requests provides a stop verifying certs button, and that works fine for us. (I know that Requests is outside the stdlib and so it's not a perfect analogy, but it's serviceable.) I suspect most people who want this change don't care if users have an easy way to circumvent it, we just want to have the user/enterprise make that choice for themselves. Put another way, we want the average user to fall into a pit of success. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 23 January 2014 08:37, Stephen J. Turnbull step...@xemacs.org wrote: I don't know what the right answer is, but this needs careful discussion and amelioration, not just you're broken, so take the consequences! Absolutely. =) With that said, having a great big button to turn the change off (e.g. environment variable, global setting) feels like an acceptable mitigation to me, but then I'm probably still more aggressive than the core developers! Either way, it looks like the deprecation dance is going to be the consensus decision here. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
Cory Benfield writes: I'm overwhelmingly, dramatically +1 on this. There's no good architectural reason to not use the built-in certificate chains by default. I'd like to be in favour of backporting this change to earlier Python versions as well, but it feels just a bit too aggressive. -1 This is just a bit too aggressive, too. I'll guarantee this breaks applications all over Japan, especially in universities because the Ministry of Education uses certificates rooted somewhere nobody's ever heard of, and typically don't bother to ensure the domain name matches the cert being presented. I've even run into such domain-match issues with banks (not banks I deal with any more, of course!) This is quite different from web browsers and other interactive applications. It has the potential to break secure mail and news and other automatic data transfers. Breaking people's software that should run silently in the background just because they upgrade Python shouldn't happen, and people here will blame Python, not their broken websites and network apps. I don't know what the right answer is, but this needs careful discussion and amelioration, not just you're broken, so take the consequences! ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
Donald Stufft writes: As an additional side note, anecdotal evidence and what not, but *every* time I bring this up somewhere I get at least one reply that looks similar to https://twitter.com/ojiidotch/status/425986619879866368 Hey, wait a cotton-picking minute! Are you telling me that Perl, PHP, and Ruby *do* verify certs by default in their batteries included stdlibs, and developers using those languages have been turning that feature off in their code for, like, you know, well, for-EVER man!? (They surely don't leave it on, or my employer would have fixed their broken cert chain and hostnames by now.) If so, that's evidence for the practicality of the proposal, and maybe even for fast-tracking it to catch up. My employer and the Ministry of Education, Culture, Science, and Technology be damned (and they will be). But if it's only the already security-conscious developers and managers who go WTF?, and other environments don't do this by default, I'd consider that a dangerous curve, slow down sign. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
Am 23.01.14 07:45, schrieb Scott Dial: Anecdotally, I already know of a system at work that is using HTTPS purely for encryption, because the authentication is done in-band. So, a self-signed cert was wholly sufficient. The management tools use a RESTful interface over HTTPS for control, but you are telling me this will be broken by default now. What do I tell our developers (who often adopt the latest and greatest versions of things to play with)? If they play with the newest version before actually using it in production, all is well. You can then tell them that they have four options: - not upgrade to the newest Python release (at least not until they are willing to pursue any of the other alternatives) - update the code to disable cert validation, or explicitly add the self-signed cert as a trusted one programmatically. - update the client system configuration, to add the self-signed certificate as trusted (system-wide or per user). - update the server, to use a cert signed by one of the trusted CAs. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 23 January 2014 22:41, Martin v. Löwis mar...@v.loewis.de wrote: Am 23.01.14 07:45, schrieb Scott Dial: Anecdotally, I already know of a system at work that is using HTTPS purely for encryption, because the authentication is done in-band. So, a self-signed cert was wholly sufficient. The management tools use a RESTful interface over HTTPS for control, but you are telling me this will be broken by default now. What do I tell our developers (who often adopt the latest and greatest versions of things to play with)? If they play with the newest version before actually using it in production, all is well. You can then tell them that they have four options: - not upgrade to the newest Python release (at least not until they are willing to pursue any of the other alternatives) - update the code to disable cert validation, or explicitly add the self-signed cert as a trusted one programmatically. - update the client system configuration, to add the self-signed certificate as trusted (system-wide or per user). - update the server, to use a cert signed by one of the trusted CAs. Or, depending on the exact transition plan, potentially set: PYTHONSSLDEFAULT=NOVERIFY (akin to the no, really, don't randomise the hashes option). That's the kind of question a PEP would be needed to thrash out, though. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Thu, 23 Jan 2014 01:45:15 -0500 Scott Dial scott+python-...@scottdial.com wrote: Anecdotally, I already know of a system at work that is using HTTPS purely for encryption, because the authentication is done in-band. So, a self-signed cert was wholly sufficient. The management tools use a RESTful interface over HTTPS for control, but you are telling me this will be broken by default now. What do I tell our developers (who often adopt the latest and greatest versions of things to play with)? That the system may be vulnerable to MITM attacks? (depending on how the authentication is done) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Thu, 23 Jan 2014 06:02:18 + Kristján Valur Jónsson krist...@ccpgames.com wrote: If not already possible, I suggest that we allow the use of a certificate validation callback (it isn't possible for 2.7, I just hacked in one yesterday to allow me to ignore out-date-failure for certificates.) Using this, it would be possible to e.g. emit warnings when certificiate failures occur, rather than deny connection outright. That's a possible transition measure indeed. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 2014-01-22 9:33 AM, Donald Stufft wrote:
For everything but pip, you’d add it to your OS cert store. Pip doesn’t
use that so you’d have to use the —cert config.
What if I don't want that self-signed cert to be trusted by all users on
the system?
Specify a client cert and an appropriate CA bundle.
What if I don't have administrative rights?
Specify an appropriate CA bundle for which there are appropriate permissions.
How do I do it then? Is this common knowledge for average users? Are we
trading one big
red box in the documentation for another?
That's really not the case.
Current web browsers are not susceptible to this particular issue.
Anecdotally, I already know of a system at work that is using HTTPS
purely for encryption, because the authentication is done in-band. So, a
self-signed cert was wholly sufficient. The management tools use a
RESTful interface over HTTPS for control, but you are telling me this
will be broken by default now. What do I tell our developers (who often
adopt the latest and greatest versions of things to play with)?
There are layers. OSI layers, if you prefer. It sounds like the
relevant layers here are:
* HTTP
* SSL/TLS
* TCP
A MITM compromise of the channel (e.g. by a rogue security tester
responding with a different SSL certificate that is not signed by a
CA, with a different hostname than requested) renders most 'in-band'
authentication mechanisms (such as HTTP Basic, Digest, and
Cookie-based Sessions) invalid.
'Higher layers' generally operate by sharing tokens as plaintext. With
SSL compromised through a MITM (as allowed by not validating hostnames
by default), said security tester could trivially intercept and modify
any of the requests and responses in the channel.
CWE-300: Channel Accessible by Non-Endpoint ('Man-in-the-Middle')
http://cwe.mitre.org/data/definitions/300.html
To use a bad metaphor: it's like the carrier piegeon stops on the way
home and there's no seal.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
But if it's only the already security-conscious developers and managers who go WTF?, and other environments don't do this by default, I'd consider that a dangerous curve, slow down sign. Mitigations: **Packaging** * Upgrade setuptools (distribute, zc.buildout) * Avoid easy_install, python setup.py install, and python setup.py develop (until it can be verified that the installed version of setuptools contains VerifyingHTTPSHandler [1]) https://bitbucket.org/pypa/setuptools/history-node/tip/setuptools/ssl_support.py * +1 for Pip install -e vcs+ssh://v...@example.org/username/pkgname@semver@egg=pkgname * +1 for Conda * +1 for OS packages **Implementation** * Python 3.4 : https://pypi.python.org/pypi/backports.ssl_match_hostname **Awareness** * Big red warning boxes: (.. warning:: in RST): Documentation * This must not be easy to test. * http://www.cvedetails.com/vulnerability-list/vendor_id-10210/product_id-18230/Python-Python.html -- Wes Turner On Thu, Jan 23, 2014 at 3:05 AM, Stephen J. Turnbull step...@xemacs.org wrote: Donald Stufft writes: As an additional side note, anecdotal evidence and what not, but *every* time I bring this up somewhere I get at least one reply that looks similar to https://twitter.com/ojiidotch/status/425986619879866368 Hey, wait a cotton-picking minute! Are you telling me that Perl, PHP, and Ruby *do* verify certs by default in their batteries included stdlibs, and developers using those languages have been turning that feature off in their code for, like, you know, well, for-EVER man!? (They surely don't leave it on, or my employer would have fixed their broken cert chain and hostnames by now.) If so, that's evidence for the practicality of the proposal, and maybe even for fast-tracking it to catch up. My employer and the Ministry of Education, Culture, Science, and Technology be damned (and they will be). But if it's only the already security-conscious developers and managers who go WTF?, and other environments don't do this by default, I'd consider that a dangerous curve, slow down sign. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 01/22/2014 04:15 AM, Donald Stufft wrote: As I’ve said multiple times, I think it’s fine to send it through the deprecation process which is still pretty long and gives people a good chunk of time to update. Agreed. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 01/22/2014 05:16 AM, M.-A. Lemburg wrote: On 22.01.2014 13:43, Jesse Noller wrote: Donald is perfectly right: today, it's trivial to MITM an application that relies off of the current behavior; this is bad news bears for users and developers as it means they need domain knowledge to secure their applications by default they may not have. I don't think you need much domain knowledge to insert a single line of code into applications to enable the checks. I find myself on the dumb user side of this argument, and I think it is much like the str/unicode transition of 3.0 -- which is it say, there are many who didn't understand unicode until forced to by 3.0, and likewise there will be many who don't understand security until forced to by enabling this new feature. One big difference is it's possible to opt-out of this security feature (which is a good thing, considering all the ill-configured systems out there). -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
Wes Turner writes: But if it's only the already security-conscious developers and managers who go WTF?, and other environments don't do this by default, I'd consider that a dangerous curve, slow down sign. Mitigations: **Packaging** * Upgrade setuptools (distribute, zc.buildout) * Avoid easy_install, python setup.py install, and python setup.py develop (until it can be verified that the installed version of setuptools contains VerifyingHTTPSHandler [1]) Are you kidding? These *aren't* the apps that I care about breaking, and I know that the PHBs won't pay attention to what I say about fixing their sites and cert chains (believe me, I've tried, and the answer is as Paul Moore says: the users can punch the go ahead anyway button, what's the big deal here?), they'll just deprecate Python. My question remains: Are you telling me that Perl, PHP, and Ruby *do* verify certs by default in their batteries included stdlibs, and developers using those languages have been turning that feature off in their code for, like, you know, well, for-EVER man!? I find that hard to believe, given that the security of the network remains broken yet there aren't warnings out to avoid these platforms. (BTW, my employer prides itself on being Matz's alma mater ... they actually might do something if Ruby was breaking things!) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 23, 2014, at 10:06 PM, Stephen J. Turnbull step...@xemacs.org wrote: Wes Turner writes: But if it's only the already security-conscious developers and managers who go WTF?, and other environments don't do this by default, I'd consider that a dangerous curve, slow down sign. Mitigations: **Packaging** * Upgrade setuptools (distribute, zc.buildout) * Avoid easy_install, python setup.py install, and python setup.py develop (until it can be verified that the installed version of setuptools contains VerifyingHTTPSHandler [1]) Are you kidding? These *aren't* the apps that I care about breaking, and I know that the PHBs won't pay attention to what I say about fixing their sites and cert chains (believe me, I've tried, and the answer is as Paul Moore says: the users can punch the go ahead anyway button, what's the big deal here?), they'll just deprecate Python. My question remains: Are you telling me that Perl, PHP, and Ruby *do* verify certs by default in their batteries included stdlibs, and developers using those languages have been turning that feature off in their code for, like, you know, well, for-EVER man!? I find that hard to believe, given that the security of the network remains broken yet there aren't warnings out to avoid these platforms. (BTW, my employer prides itself on being Matz's alma mater ... they actually might do something if Ruby was breaking things!) Ruby has verified the peer by default since Ruby 1.9 Go also verifies by default, I’m not aware if PHP or Perl do. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 23, 2014, at 10:09 PM, Donald Stufft don...@stufft.io wrote: On Jan 23, 2014, at 10:06 PM, Stephen J. Turnbull step...@xemacs.org wrote: Wes Turner writes: But if it's only the already security-conscious developers and managers who go WTF?, and other environments don't do this by default, I'd consider that a dangerous curve, slow down sign. Mitigations: **Packaging** * Upgrade setuptools (distribute, zc.buildout) * Avoid easy_install, python setup.py install, and python setup.py develop (until it can be verified that the installed version of setuptools contains VerifyingHTTPSHandler [1]) Are you kidding? These *aren't* the apps that I care about breaking, and I know that the PHBs won't pay attention to what I say about fixing their sites and cert chains (believe me, I've tried, and the answer is as Paul Moore says: the users can punch the go ahead anyway button, what's the big deal here?), they'll just deprecate Python. My question remains: Are you telling me that Perl, PHP, and Ruby *do* verify certs by default in their batteries included stdlibs, and developers using those languages have been turning that feature off in their code for, like, you know, well, for-EVER man!? I find that hard to believe, given that the security of the network remains broken yet there aren't warnings out to avoid these platforms. (BTW, my employer prides itself on being Matz's alma mater ... they actually might do something if Ruby was breaking things!) Ruby has verified the peer by default since Ruby 1.9 Go also verifies by default, I’m not aware if PHP or Perl do. Oh, Node.js also verifies by default, PHP apparently does not. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Enable Hostname and Certificate Chain Validation
I would like to propose that a backwards incompatible change be made to Python to make verification of hostname and certificate chain the default instead of requiring it to be opt in. Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc and validate by default and in the typical style of security related issues it will appear to work just fine however be grossly insecure. In the real world “opt in security” typically translates to just plain old insecure for the bulk of applications/libraries. I believe that Python has a responsibility to do the right thing by default here and it is in the best position to do so. The alternative requires every Python developer who wants to access a secure resource to be educated on the fact that they need to flip some switch to do what most of them would expect. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22.01.2014 11:30, Donald Stufft wrote: I would like to propose that a backwards incompatible change be made to Python to make verification of hostname and certificate chain the default instead of requiring it to be opt in. Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc and validate by default and in the typical style of security related issues it will appear to work just fine however be grossly insecure. In the real world “opt in security” typically translates to just plain old insecure for the bulk of applications/libraries. I believe that Python has a responsibility to do the right thing by default here and it is in the best position to do so. The alternative requires every Python developer who wants to access a secure resource to be educated on the fact that they need to flip some switch to do what most of them would expect. Such a change would introduce considerable breakage. This would either have to be done using our usual pending deprecation, deprecation, removal dance (over three releases) or be postponed until Python 4. Note that several python.org services use CAcerts which would no longer be accessible per default following such a change. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 22 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 5:51 AM, M.-A. Lemburg m...@egenix.com wrote: On 22.01.2014 11:30, Donald Stufft wrote: I would like to propose that a backwards incompatible change be made to Python to make verification of hostname and certificate chain the default instead of requiring it to be opt in. Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc and validate by default and in the typical style of security related issues it will appear to work just fine however be grossly insecure. In the real world “opt in security” typically translates to just plain old insecure for the bulk of applications/libraries. I believe that Python has a responsibility to do the right thing by default here and it is in the best position to do so. The alternative requires every Python developer who wants to access a secure resource to be educated on the fact that they need to flip some switch to do what most of them would expect. Such a change would introduce considerable breakage. This would either have to be done using our usual pending deprecation, deprecation, removal dance (over three releases) or be postponed until Python 4. I can understand the need for doing the typical deprecation dance, although I believe such policies are often overlooked or accelerated for security sensitive changes. I do believe that waiting until Python 4 would be doing a great disservice to the users of Python though. Note that several python.org services use CAcerts which would no longer be accessible per default following such a change. Not that it has much to do with this proposal, but those services should be switched to use certificates that are well trusted. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 22 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 6:21 AM, Paul Moore p.f.mo...@gmail.com wrote: On 22 January 2014 10:30, Donald Stufft don...@stufft.io wrote: Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc and validate by default and in the typical style of security related issues it will appear to work just fine however be grossly insecure. Two things: 1. To be like the browser we'd need to use the OS certificate store, which isn't the case on Windows at the moment (managing those certificate bundle files is most definitely *not* like the browser - I'd have no idea how to add a self-certificate to the bundle file embedded in pip, for example). Python 3.4 added the ability to use the OS cert store on Windows, see http://bugs.python.org/issue17134. 2. Your proposal is that because some application authors have not opted in yet, we should penalise the end users of those applications by stopping them being able to use unverified https? And don't forget, applications that haven't opted in will have no switch to allow unverified use. That seems to be punishing the wrong people. Some applications will need to be updated yes to provide such a switch but the alternative is that every user of this API needs to configure it to verify certificates. The difference is that with my proposal the error condition is very obvious, the SSL certificate will fail to validate, a bug can be filed and it can be fixed. With the current behavior they only way you’d know is if you expected it to fail and didn’t, or you went specifically looking. It’s a dangerous by default API that punishes people for not knowing that they need to turn it on, and punishes people who use those applications. However it won’t punish them directly, instead it’ll just make it possible to MITM their connection, possibly leaking sensitive material. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22.01.2014 11:56, Donald Stufft wrote: On Jan 22, 2014, at 5:51 AM, M.-A. Lemburg m...@egenix.com wrote: On 22.01.2014 11:30, Donald Stufft wrote: I would like to propose that a backwards incompatible change be made to Python to make verification of hostname and certificate chain the default instead of requiring it to be opt in. Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc and validate by default and in the typical style of security related issues it will appear to work just fine however be grossly insecure. In the real world “opt in security” typically translates to just plain old insecure for the bulk of applications/libraries. I believe that Python has a responsibility to do the right thing by default here and it is in the best position to do so. The alternative requires every Python developer who wants to access a secure resource to be educated on the fact that they need to flip some switch to do what most of them would expect. Such a change would introduce considerable breakage. This would either have to be done using our usual pending deprecation, deprecation, removal dance (over three releases) or be postponed until Python 4. I can understand the need for doing the typical deprecation dance, although I believe such policies are often overlooked or accelerated for security sensitive changes. I do believe that waiting until Python 4 would be doing a great disservice to the users of Python though. Well, it's not really a security issue, since the security features are present in Python 3.4. It's just that the user has to enable them. Note that several python.org services use CAcerts which would no longer be accessible per default following such a change. Not that it has much to do with this proposal, but those services should be switched to use certificates that are well trusted. Oh, it does have to do with the proposal, since it's a prominent example of services that would no longer work. While we can potentially change the situation for python.org servers, you have the same issues with other services which we don't have any influence on. The change would also disable all services using self-signed certificates which are very common in internal networks and for ad-hoc setups. Many routers and other devices use self-signed certificates when offering HTTPS services. I think overall, it's good to have default security, but locking out all certificates which do not have their root CA certs installed in default installations of systems per default would likely lead to people seeking other more insecure ways of getting things to work, rather than asking their admins to add their CA certs to the certificate chain configuration. So I'm not sure whether raising errors is the best way to achieve better default security. Perhaps just using warnings would be better. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 22 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 6:30 AM, M.-A. Lemburg m...@egenix.com wrote: On 22.01.2014 11:56, Donald Stufft wrote: On Jan 22, 2014, at 5:51 AM, M.-A. Lemburg m...@egenix.com wrote: On 22.01.2014 11:30, Donald Stufft wrote: I would like to propose that a backwards incompatible change be made to Python to make verification of hostname and certificate chain the default instead of requiring it to be opt in. Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc and validate by default and in the typical style of security related issues it will appear to work just fine however be grossly insecure. In the real world “opt in security” typically translates to just plain old insecure for the bulk of applications/libraries. I believe that Python has a responsibility to do the right thing by default here and it is in the best position to do so. The alternative requires every Python developer who wants to access a secure resource to be educated on the fact that they need to flip some switch to do what most of them would expect. Such a change would introduce considerable breakage. This would either have to be done using our usual pending deprecation, deprecation, removal dance (over three releases) or be postponed until Python 4. I can understand the need for doing the typical deprecation dance, although I believe such policies are often overlooked or accelerated for security sensitive changes. I do believe that waiting until Python 4 would be doing a great disservice to the users of Python though. Well, it's not really a security issue, since the security features are present in Python 3.4. It's just that the user has to enable them. I explicitly didn’t call it a security issue, but rather a security sensitive change, which it is. This would effectively be making Python more secure by default and removing a security related footgun. Note that several python.org services use CAcerts which would no longer be accessible per default following such a change. Not that it has much to do with this proposal, but those services should be switched to use certificates that are well trusted. Oh, it does have to do with the proposal, since it's a prominent example of services that would no longer work. While we can potentially change the situation for python.org servers, you have the same issues with other services which we don't have any influence on. I meant the statement that they should be switched to a well trusted certificate didn’t have much to do with this proposal. The change would also disable all services using self-signed certificates which are very common in internal networks and for ad-hoc setups. Many routers and other devices use self-signed certificates when offering HTTPS services. It will just disable them by default, they can still easily be accessed you’d just need to pass the “do not verify” flag. This clearly indicates that you’re opting out of the S in HTTPS. I think overall, it's good to have default security, but locking out all certificates which do not have their root CA certs installed in default installations of systems per default would likely lead to people seeking other more insecure ways of getting things to work, rather than asking their admins to add their CA certs to the certificate chain configuration. So I'm not sure whether raising errors is the best way to achieve better default security. Perhaps just using warnings would be better. Again it’s not “locking out”, it’s simply “requires explicitly saying I want to not validate”. It’s hard to be more insecure than not verifying. Just about the only other way is to use plaintext but the only real difference there is passive vs attack attacks. However typically if you’re in the situation to do a passive attack you can also do an active attack. So “more insecure” is minorly more insecure while an error is drastically more secure. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 22 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 6:21 AM, Paul Moore p.f.mo...@gmail.com wrote: 2. Your proposal is that because some application authors have not opted in yet, we should penalise the end users of those applications by stopping them being able to use unverified https? And don't forget, applications that haven't opted in will have no switch to allow unverified use. That seems to be punishing the wrong people. Another thought, if this is seriously a blocker something simple like an environment variable could be added that switches the default. Which would act as a global sort of —insecure flag for applications that don’t provide one. I really don’t like the idea of doing that, but it would be better than not validating by default. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22 January 2014 21:21, Paul Moore p.f.mo...@gmail.com wrote: On 22 January 2014 10:30, Donald Stufft don...@stufft.io wrote: Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc and validate by default and in the typical style of security related issues it will appear to work just fine however be grossly insecure. Two things: 1. To be like the browser we'd need to use the OS certificate store, which isn't the case on Windows at the moment (managing those certificate bundle files is most definitely *not* like the browser - I'd have no idea how to add a self-certificate to the bundle file embedded in pip, for example). 2. Your proposal is that because some application authors have not opted in yet, we should penalise the end users of those applications by stopping them being able to use unverified https? And don't forget, applications that haven't opted in will have no switch to allow unverified use. That seems to be punishing the wrong people. Right, the browsers have a whole system of click through security to make the web (and corporate intranets!) still usable even when they only accept CA signed certs by default. With a programming language, there's no such interactivity, so applications just break and users don't know why. It's notable that even Linux distros haven't made this change in their system Python builds, and commercial Linux distros have raised paranoia to an art form (since that's a respectable chunk of what their users are paying for). We also have to account for the fact that an awful lot of Python applications are corporate ones relying on perimeter defence for security, or private CAs, or just self-signed certificates that their users have already accepted. There are limits to the amount of backwards incompatible change users will tolerate, and at this point in time we're still trying to get people to accept proper Unicode support. Even the package distribution tools are struggling with the consequences of trying to lock things down by default - every new release of pip currently brings someone else out of the woodwork pointing out they were relying on whichever insecure default we have changed on them this time. Securing the web is a boil the ocean type task - Python 3.4 takes us a step closer by making it possible for people to easily use the system certs via ssl.create_default_context() (http://docs.python.org/dev/library/ssl.html#ssl.create_default_context), but move fast and break things isn't going to work on this one any more than it does for proper Unicode support or the IPv4 to IPv6 transition. Security concerns are too abstract for most people for them to accept it as an excuse when you tell them you broke their software for their own good. The only before-Python-4 approach I can potentially see working is this: 1. Provide ssl.create_default_context() in 3.4 (done) 2. Deprecate the implicit SSL context in 3.5 3. Remove the implicit SSL context in 3.6 4. Make the default context the implicit context in 3.7 It would be slow, but we *could* get there. You *might* even be able to make the case for collapsing steps 3 and 4 together so that the more secure default is brought in as part of 3.6 in 2017. I'd suggest a PEP along those lines, but I think you already have plenty to keep you busy helping to ride herd on the packaging ecosystem :) In the meantime, making it so that it *is* just a single additional function call to get proper TLS security settings in 3.4 is a major step forward (thanks Christian!). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22.01.2014 12:36, Donald Stufft wrote: On Jan 22, 2014, at 6:30 AM, M.-A. Lemburg m...@egenix.com wrote: The change would also disable all services using self-signed certificates which are very common in internal networks and for ad-hoc setups. Many routers and other devices use self-signed certificates when offering HTTPS services. It will just disable them by default, they can still easily be accessed you’d just need to pass the “do not verify” flag. This clearly indicates that you’re opting out of the S in HTTPS. I think overall, it's good to have default security, but locking out all certificates which do not have their root CA certs installed in default installations of systems per default would likely lead to people seeking other more insecure ways of getting things to work, rather than asking their admins to add their CA certs to the certificate chain configuration. So I'm not sure whether raising errors is the best way to achieve better default security. Perhaps just using warnings would be better. Again it’s not “locking out”, it’s simply “requires explicitly saying I want to not validate”. This would have to configurable without changing application code, e.g. using an environment settings. Otherwise, you do lock out existing scripts and applications from using Python 3.6 by requiring supporting custom configurations from all of them. Simply saying: oh, just change your code to never validate is not a good solution either. It’s hard to be more insecure than not verifying. Just about the only other way is to use plaintext but the only real difference there is passive vs attack attacks. However typically if you’re in the situation to do a passive attack you can also do an active attack. So “more insecure” is minorly more insecure while an error is drastically more secure. I disagree with that statement. Using HTTPS without verification is still far more secure than using plain text. I know that verification is a lot better, but please remember that practicality beats purity. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 22 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22 January 2014 21:36, Donald Stufft don...@stufft.io wrote: On Jan 22, 2014, at 6:30 AM, M.-A. Lemburg m...@egenix.com wrote: The change would also disable all services using self-signed certificates which are very common in internal networks and for ad-hoc setups. Many routers and other devices use self-signed certificates when offering HTTPS services. It will just disable them by default, they can still easily be accessed you’d just need to pass the “do not verify” flag. This clearly indicates that you’re opting out of the S in HTTPS. You need to remember that *Python is fundamentally not an application*. We don't control the interaction with the user, application developers do, and every decision we make has to take that into account. The kinds of decisions that an application like a web browser or a package installer can make aren't necessarily available to a runtime. We had to be cautious even with the initial hash randomisation change to avoid breaking currently working applications. Look at the anger that people express about us making Python 3 more sensitive to environment misconfiguration on POSIX systems: people don't blame the misconfigured environment that Python 2 tolerated with an increased risk of data corruption, they blame *us* for breaking something that used to work by default. The change you're proposing would mean that *every* Python application would either need to be updated to explicitly opt in to insecurity (the path most of them will take, because humans) or else to provide a set this option to make your computer work again insecurity flag (which is a bad idea anyway, again because humans). There are currently still too many valid reasons for not using verified SSL for us to realistically make it the default without a seriously long transition period (not quite IPv6 or even Python 3 long, but certainly not as short as the time period involved in introducing hash randomisation). Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 6:45 AM, Nick Coghlan ncogh...@gmail.com wrote: On 22 January 2014 21:21, Paul Moore p.f.mo...@gmail.com wrote: On 22 January 2014 10:30, Donald Stufft don...@stufft.io wrote: Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc and validate by default and in the typical style of security related issues it will appear to work just fine however be grossly insecure. Two things: 1. To be like the browser we'd need to use the OS certificate store, which isn't the case on Windows at the moment (managing those certificate bundle files is most definitely *not* like the browser - I'd have no idea how to add a self-certificate to the bundle file embedded in pip, for example). 2. Your proposal is that because some application authors have not opted in yet, we should penalise the end users of those applications by stopping them being able to use unverified https? And don't forget, applications that haven't opted in will have no switch to allow unverified use. That seems to be punishing the wrong people. Right, the browsers have a whole system of click through security to make the web (and corporate intranets!) still usable even when they only accept CA signed certs by default. With a programming language, there's no such interactivity, so applications just break and users don't know why. It's notable that even Linux distros haven't made this change in their system Python builds, and commercial Linux distros have raised paranoia to an art form (since that's a respectable chunk of what their users are paying for). I was actually talking to a Debian maintainer about the likelihood of making this change there earlier today :) If I fail at making this change in upstream I’ll be lobbying downstream and then we’ll just have different behaviors based on where you get your Python from which I think stinks. We also have to account for the fact that an awful lot of Python applications are corporate ones relying on perimeter defence for security, or private CAs, or just self-signed certificates that their users have already accepted. There are limits to the amount of backwards incompatible change users will tolerate, and at this point in time we're still trying to get people to accept proper Unicode support. Most of those add their private CAs to the system cert stores which would still work fine. I don’t think this change is one that users would be very upset about. We received very positive feedback in doing similar for Pip and we did break things for a few people. Even the package distribution tools are struggling with the consequences of trying to lock things down by default - every new release of pip currently brings someone else out of the woodwork pointing out they were relying on whichever insecure default we have changed on them this time. Again speaking to the packaging tools, the switch to verifying SSL was very well tolerated, it was other changes that are unrelated this this proposal that have caused the bulk of the issues. Securing the web is a boil the ocean type task - Python 3.4 takes us a step closer by making it possible for people to easily use the system certs via ssl.create_default_context() (http://docs.python.org/dev/library/ssl.html#ssl.create_default_context), but move fast and break things isn't going to work on this one any more than it does for proper Unicode support or the IPv4 to IPv6 transition. Security concerns are too abstract for most people for them to accept it as an excuse when you tell them you broke their software for their own good. Calling this a “boil the ocean” type task is facetious. Yes securing the web as a whole would be like boiling the ocean, but enabling TLS validation by default is like basic 101 level. The only before-Python-4 approach I can potentially see working is this: 1. Provide ssl.create_default_context() in 3.4 (done) 2. Deprecate the implicit SSL context in 3.5 3. Remove the implicit SSL context in 3.6 4. Make the default context the implicit context in 3.7 It would be slow, but we *could* get there. You *might* even be able to make the case for collapsing steps 3 and 4 together so that the more secure default is brought in as part of 3.6 in 2017. This would be OK, obviously i’d prefer it to happen on as quickly as timeline as possible but like I said earlier I can totally see the need to have the deprecation process. I'd suggest a PEP along those lines, but I think you already have plenty to keep you busy helping to ride herd on the packaging ecosystem :) If a PEP is required for this I’ll write one. I have no idea what changes need a PEP or not and I default to not writing one because it’s easier :) In the meantime, making it so that it *is*
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 7:03 AM, Paul Moore p.f.mo...@gmail.com wrote: On 22 January 2014 11:29, Donald Stufft don...@stufft.io wrote: 1. To be like the browser we'd need to use the OS certificate store, which isn't the case on Windows at the moment (managing those certificate bundle files is most definitely *not* like the browser - I'd have no idea how to add a self-certificate to the bundle file embedded in pip, for example). Python 3.4 added the ability to use the OS cert store on Windows, see http://bugs.python.org/issue17134. Brilliant. I didn't know that. Will pip when run on Python 3.4 use the OS cert store? I guess the answer is probably no (but i'd love to be pleasantly surprised). Paul The answer is (I believe) no, mostly for consistency’s sake. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22 January 2014 11:29, Donald Stufft don...@stufft.io wrote: 1. To be like the browser we'd need to use the OS certificate store, which isn't the case on Windows at the moment (managing those certificate bundle files is most definitely *not* like the browser - I'd have no idea how to add a self-certificate to the bundle file embedded in pip, for example). Python 3.4 added the ability to use the OS cert store on Windows, see http://bugs.python.org/issue17134. Brilliant. I didn't know that. Will pip when run on Python 3.4 use the OS cert store? I guess the answer is probably no (but i'd love to be pleasantly surprised). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 6:58 AM, Nick Coghlan ncogh...@gmail.com wrote: On 22 January 2014 21:36, Donald Stufft don...@stufft.io wrote: On Jan 22, 2014, at 6:30 AM, M.-A. Lemburg m...@egenix.com wrote: The change would also disable all services using self-signed certificates which are very common in internal networks and for ad-hoc setups. Many routers and other devices use self-signed certificates when offering HTTPS services. It will just disable them by default, they can still easily be accessed you’d just need to pass the “do not verify” flag. This clearly indicates that you’re opting out of the S in HTTPS. You need to remember that *Python is fundamentally not an application*. We don't control the interaction with the user, application developers do, and every decision we make has to take that into account. The kinds of decisions that an application like a web browser or a package installer can make aren't necessarily available to a runtime. We had to be cautious even with the initial hash randomisation change to avoid breaking currently working applications. The same could be said for requests, it’s fundamentally not an application and can’t control the interaction with the user and yet it validates TLS by default just fine. Look at the anger that people express about us making Python 3 more sensitive to environment misconfiguration on POSIX systems: people don't blame the misconfigured environment that Python 2 tolerated with an increased risk of data corruption, they blame *us* for breaking something that used to work by default. They blame Python for breaking something that used to work by default for something they feel has little or no benefit, and which is also difficult or impossible to adapt to without often significant code changes (for the people who I’ve seen blaming Python dev for). Do you really think those people would be making the same complaints if they could restore the previous behavior with a simple boolean flag delivered either via environment variable or in their own code? The change you're proposing would mean that *every* Python application would either need to be updated to explicitly opt in to insecurity (the path most of them will take, because humans) or else to provide a set this option to make your computer work again insecurity flag (which is a bad idea anyway, again because humans). Every Python application *that depends on an invalid certificate* and is written for Python3. It’s not hardly every Python application since a good majority of them will simply continue working, and given that most Python applications are still written for Python2 it’s not even something that is going to affect the bulk of *Python* applications. There are currently still too many valid reasons for not using verified SSL for us to realistically make it the default without a seriously long transition period (not quite IPv6 or even Python 3 long, but certainly not as short as the time period involved in introducing hash randomisation). As I’ve said multiple times, I think it’s fine to send it through the deprecation process which is still pretty long and gives people a good chunk of time to update. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22 January 2014 12:02, Donald Stufft don...@stufft.io wrote: We also have to account for the fact that an awful lot of Python applications are corporate ones relying on perimeter defence for security, or private CAs, or just self-signed certificates that their users have already accepted. There are limits to the amount of backwards incompatible change users will tolerate, and at this point in time we're still trying to get people to accept proper Unicode support. Most of those add their private CAs to the system cert stores which would still work fine. I don’t think this change is one that users would be very upset about. We received very positive feedback in doing similar for Pip and we did break things for a few people. Speaking as someone whose day job consists entirely of working in a corporate behind the firewall environment, in my experience this is simply wrong. Most companies do *not* add private or self certificates to the system stores. Rather, they expect their end users to click on Yes, Allow in the browser *every* *time* they access the webpage. In many cases even the local PC store and exception list is locked down, so the user has no way of even avoiding this on a local basis. Python and applications built on Python are often used unofficially in such organisations for productivity-enhancing applications. Because it's unofficial, it's often latest versions. Because it's to improve productivity, grabbing existing apps and libraries and having them work rather than writing your own is crucial. Seriously - the security viewpoints I'm seeing here are so far from corporate life that it's ridiculous. (But to be fair to corporate environments, the firewalls involved mean that the systems involved often have so little internet access that you can essentially ignore anything other than internal threats). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 5:30 AM, M.-A. Lemburg m...@egenix.com wrote: On 22.01.2014 11:56, Donald Stufft wrote: On Jan 22, 2014, at 5:51 AM, M.-A. Lemburg m...@egenix.com wrote: On 22.01.2014 11:30, Donald Stufft wrote: I would like to propose that a backwards incompatible change be made to Python to make verification of hostname and certificate chain the default instead of requiring it to be opt in. Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc and validate by default and in the typical style of security related issues it will appear to work just fine however be grossly insecure. In the real world “opt in security” typically translates to just plain old insecure for the bulk of applications/libraries. I believe that Python has a responsibility to do the right thing by default here and it is in the best position to do so. The alternative requires every Python developer who wants to access a secure resource to be educated on the fact that they need to flip some switch to do what most of them would expect. Such a change would introduce considerable breakage. This would either have to be done using our usual pending deprecation, deprecation, removal dance (over three releases) or be postponed until Python 4. I can understand the need for doing the typical deprecation dance, although I believe such policies are often overlooked or accelerated for security sensitive changes. I do believe that waiting until Python 4 would be doing a great disservice to the users of Python though. Well, it's not really a security issue, since the security features are present in Python 3.4. It's just that the user has to enable them. I have to concur with Donald here - in the case of security, especially language security which directly impacts the implicit security of downstream applications, I should not have to opt in to the most secure defaults. Yes; this potentially breaks applications relying on insecure / loose defaults. However it changes the model to you are by default, explicitly secure then relying on the domain knowledge of an application developer to harden their application. When, if this changes, an application breaks, it will be in a plainly obvious way which can quickly be resolved. Donald is perfectly right: today, it's trivial to MITM an application that relies off of the current behavior; this is bad news bears for users and developers as it means they need domain knowledge to secure their applications by default they may not have. Note that several python.org services use CAcerts which would no longer be accessible per default following such a change. Not that it has much to do with this proposal, but those services should be switched to use certificates that are well trusted. Oh, it does have to do with the proposal, since it's a prominent example of services that would no longer work. While we can potentially change the situation for python.org servers, you have the same issues with other services which we don't have any influence on. The change would also disable all services using self-signed certificates which are very common in internal networks and for ad-hoc setups. Many routers and other devices use self-signed certificates when offering HTTPS services. I think overall, it's good to have default security, but locking out all certificates which do not have their root CA certs installed in default installations of systems per default would likely lead to people seeking other more insecure ways of getting things to work, rather than asking their admins to add their CA certs to the certificate chain configuration. So I'm not sure whether raising errors is the best way to achieve better default security. Perhaps just using warnings would be better. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 22 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, Jan 22, 2014 at 11:15 PM, Donald Stufft don...@stufft.io wrote: Do you really think those people would be making the same complaints if they could restore the previous behavior with a simple boolean flag delivered either via environment variable or in their own code? You assume that it's easy to tweak the code. From personal experience just today I can say that this isn't always the case. I was asked a question about an internal program that had been in use since the late 1990s, and which had originally been written to work with Netscape Navigator and had been updated to work with Firefox, but not Chrome. The original author is still around, but it's too much hassle to get that code dug into, so it's far easier just to accept a small issue with Chrome (since the program's not used very often anyway). But if Chrome had completely broken that program, the solution would simply be keep using Firefox, not fix the program - it's not considered a bug. Now, maybe it wouldn't be a problem if the fix is an environment variable, but imagine a thousand-computer deployment and you have to tweak the environment on all of them. Feel like doing that just because the newest Python needs it? Not so much. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22 January 2014 22:15, Donald Stufft don...@stufft.io wrote: On Jan 22, 2014, at 6:58 AM, Nick Coghlan ncogh...@gmail.com wrote: The kinds of decisions that an application like a web browser or a package installer can make aren't necessarily available to a runtime. We had to be cautious even with the initial hash randomisation change to avoid breaking currently working applications. The same could be said for requests, it’s fundamentally not an application and can’t control the interaction with the user and yet it validates TLS by default just fine. The requests library is used by a relatively small fraction of the Python community, and they're mostly web specialists that already understand the need for TLS-by-default and hence don't need it carefully explained to them. New users adopting it just treat that as the way that requests works and not something the language developers are forcing on them. Look at the anger that people express about us making Python 3 more sensitive to environment misconfiguration on POSIX systems: people don't blame the misconfigured environment that Python 2 tolerated with an increased risk of data corruption, they blame *us* for breaking something that used to work by default. They blame Python for breaking something that used to work by default for something they feel has little or no benefit, and which is also difficult or impossible to adapt to without often significant code changes (for the people who I’ve seen blaming Python dev for). Which is exactly the way most non-web-specialists working inside the comfort of corporate and academic firewalls will react to a change that breaks their access to internal applications, where self-signed certs and improperly configured internal CAs are endemic (of course, that's assuming they're using HTTPS at all, which I admit is an optimistic assumption). There are currently still too many valid reasons for not using verified SSL for us to realistically make it the default without a seriously long transition period (not quite IPv6 or even Python 3 long, but certainly not as short as the time period involved in introducing hash randomisation). As I’ve said multiple times, I think it’s fine to send it through the deprecation process which is still pretty long and gives people a good chunk of time to update. Then I don't believe we actually have an argument here. Create the PEP with the rationale (written in terms a non-web-specialist can understand) and the deprecation strategy, and we can consider adding the initial deprecation warning in 3.5 next year. So, no rush on getting the PEP together :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, 22 Jan 2014 05:30:40 -0500 Donald Stufft don...@stufft.io wrote: I would like to propose that a backwards incompatible change be made to Python to make verification of hostname and certificate chain the default instead of requiring it to be opt in. Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc Python is not a Web client. Are you talking specifically about urllib? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22 January 2014 23:19, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 22 Jan 2014 05:30:40 -0500 Donald Stufft don...@stufft.io wrote: I would like to propose that a backwards incompatible change be made to Python to make verification of hostname and certificate chain the default instead of requiring it to be opt in. Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc Python is not a Web client. Are you talking specifically about urllib? And all the other client modules that can make secure network connections (but don't validate that the certificate matches the hostname by default). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22.01.2014 12:45, Nick Coghlan wrote: We also have to account for the fact that an awful lot of Python applications are corporate ones relying on perimeter defence for security, or private CAs, or just self-signed certificates that their users have already accepted. There are limits to the amount of backwards incompatible change users will tolerate, and at this point in time we're still trying to get people to accept proper Unicode support. Side note: Users can simple add self-signed certs to OpenSSL's cert store and get validation for free. It's possible to do that with an environment variable, too. But I recommend against the environment variable because you may overwrite to operating store. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 8:29 AM, Christian Heimes christ...@python.org wrote: On 22.01.2014 12:45, Nick Coghlan wrote: We also have to account for the fact that an awful lot of Python applications are corporate ones relying on perimeter defence for security, or private CAs, or just self-signed certificates that their users have already accepted. There are limits to the amount of backwards incompatible change users will tolerate, and at this point in time we're still trying to get people to accept proper Unicode support. Side note: Users can simple add self-signed certs to OpenSSL's cert store and get validation for free. It's possible to do that with an environment variable, too. But I recommend against the environment variable because you may overwrite to operating store. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io As an additional side note, anecdotal evidence and what not, but *every* time I bring this up somewhere I get at least one reply that looks similar to https://twitter.com/ojiidotch/status/425986619879866368 - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22.01.2014 14:55, Donald Stufft wrote: As an additional side note, anecdotal evidence and what not, but *every* time I bring this up somewhere I get at least one reply that looks similar to https://twitter.com/ojiidotch/status/425986619879866368 Yeah :( The ssl module documentation http://docs.python.org/3/library/ssl.html features a big red warning box for a good reason. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 6:58 AM, Chris Angelico ros...@gmail.com wrote: On Wed, Jan 22, 2014 at 11:15 PM, Donald Stufft don...@stufft.io wrote: Do you really think those people would be making the same complaints if they could restore the previous behavior with a simple boolean flag delivered either via environment variable or in their own code? You assume that it's easy to tweak the code. From personal experience just today I can say that this isn't always the case. I was asked a question about an internal program that had been in use since the late 1990s, and which had originally been written to work with Netscape Navigator and had been updated to work with Firefox, but not Chrome. The original author is still around, but it's too much hassle to get that code dug into, so it's far easier just to accept a small issue with Chrome (since the program's not used very often anyway). But if Chrome had completely broken that program, the solution would simply be keep using Firefox, not fix the program - it's not considered a bug. Now, maybe it wouldn't be a problem if the fix is an environment variable, but imagine a thousand-computer deployment and you have to tweak the environment on all of them. Feel like doing that just because the newest Python needs it? Not so much. What's the bet that that application will be ported to python 3.4/3.5 if this is the case? I'd say approaching 0, which is ok. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22.01.2014 14:24, Nick Coghlan wrote: On 22 January 2014 23:19, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 22 Jan 2014 05:30:40 -0500 Donald Stufft don...@stufft.io wrote: I would like to propose that a backwards incompatible change be made to Python to make verification of hostname and certificate chain the default instead of requiring it to be opt in. Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc Python is not a Web client. Are you talking specifically about urllib? And all the other client modules that can make secure network connections (but don't validate that the certificate matches the hostname by default). With Python 3.4 all stdlib modules can verify the hostname and in fact do with ssl.create_default_context(). Several modules like ftplib didn't support SNI and hostname verification. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 8:03 AM, Christian Heimes christ...@python.org wrote: On 22.01.2014 14:55, Donald Stufft wrote: As an additional side note, anecdotal evidence and what not, but *every* time I bring this up somewhere I get at least one reply that looks similar to https://twitter.com/ojiidotch/status/425986619879866368 Yeah :( The ssl module documentation http://docs.python.org/3/library/ssl.html features a big red warning box for a good reason. And no one reads it. I can't count the number of times I've gotten called into a managers office when they find out python doesn't do cert validation by default (and in 2, it's not been trivial) and gotten told to fix it, or we move off of python. Donald is perfectly right: every time you point out to users that this is the default behavior the response is almost universally you can't be serious, is this a joke? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22.01.2014 13:43, Jesse Noller wrote: I have to concur with Donald here - in the case of security, especially language security which directly impacts the implicit security of downstream applications, I should not have to opt in to the most secure defaults. Yes; this potentially breaks applications relying on insecure / loose defaults. However it changes the model to you are by default, explicitly secure then relying on the domain knowledge of an application developer to harden their application. When, if this changes, an application breaks, it will be in a plainly obvious way which can quickly be resolved. Donald is perfectly right: today, it's trivial to MITM an application that relies off of the current behavior; this is bad news bears for users and developers as it means they need domain knowledge to secure their applications by default they may not have. For 3.5 I'd like to work on a policy framework for the ssl where application can define policies like SSL/TLS version, cert store, verification modes etc. etc. I'll discuss my ideas with Donald, Alex and the other crypto guys as soon as I have settled in with my new job and town. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Thu, Jan 23, 2014 at 1:08 AM, Jesse Noller jnol...@gmail.com wrote: Now, maybe it wouldn't be a problem if the fix is an environment variable, but imagine a thousand-computer deployment and you have to tweak the environment on all of them. Feel like doing that just because the newest Python needs it? Not so much. What's the bet that that application will be ported to python 3.4/3.5 if this is the case? I'd say approaching 0, which is ok. Define ported to. (This particular application isn't Python, so the specifics don't apply, but in general.) Usually that means simply run on. Something that was written for Python 3.2 will probably run on 3.3, and on 3.4, and on 3.5 as well. You certainly wouldn't expect one small corner of it to suddenly start doing different stuff, and if you do, you'll blame Python... which would mean that you're right, that program wouldn't be run on 3.4. Is that a good thing? I don't know, but I think not. In a big company with lots of seats, every option is looking like a sysadmin's nightmare. That said, though, I agree *in principle* that secure-by-default is the way to go. It's just the backward-incompatibility of *changing* it. I like how requests is going. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, 22 Jan 2014 15:13:00 +0100 Christian Heimes christ...@python.org wrote: On 22.01.2014 13:43, Jesse Noller wrote: I have to concur with Donald here - in the case of security, especially language security which directly impacts the implicit security of downstream applications, I should not have to opt in to the most secure defaults. Yes; this potentially breaks applications relying on insecure / loose defaults. However it changes the model to you are by default, explicitly secure then relying on the domain knowledge of an application developer to harden their application. When, if this changes, an application breaks, it will be in a plainly obvious way which can quickly be resolved. Donald is perfectly right: today, it's trivial to MITM an application that relies off of the current behavior; this is bad news bears for users and developers as it means they need domain knowledge to secure their applications by default they may not have. For 3.5 I'd like to work on a policy framework for the ssl where application can define policies like SSL/TLS version, cert store, verification modes etc. etc. Isn't that called a SSLContext? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22 January 2014 13:55, Donald Stufft don...@stufft.io wrote: As an additional side note, anecdotal evidence and what not, but *every* time I bring this up somewhere I get at least one reply that looks similar to https://twitter.com/ojiidotch/status/425986619879866368 Surprise that Python doesn't verify certs is one thing. I would also like to live in a world where Python has always verified certs, and all the issues have already been resolved. Imposing breakage on end users because we haven't managed to persuade application developers to do the right thing yet (even though it appears we've made it one-line-of-code easy to do so) is another thing entirely. But the deprecation cycle gives application developers time (and a deadline) so I'm happy with that. Although from MAL's original comment: Note that several python.org services use CAcerts which would no longer be accessible per default following such a change. ,The PSF needs to get that sorted before making cert validation the default in Python, IMO. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 9:19 AM, Paul Moore p.f.mo...@gmail.com wrote: On 22 January 2014 13:55, Donald Stufft don...@stufft.io wrote: As an additional side note, anecdotal evidence and what not, but *every* time I bring this up somewhere I get at least one reply that looks similar to https://twitter.com/ojiidotch/status/425986619879866368 Surprise that Python doesn't verify certs is one thing. I would also like to live in a world where Python has always verified certs, and all the issues have already been resolved. Imposing breakage on end users because we haven't managed to persuade application developers to do the right thing yet (even though it appears we've made it one-line-of-code easy to do so) is another thing entirely. Note: That it requires users to even be aware they *need* to do that one line of code, which many are not. But the deprecation cycle gives application developers time (and a deadline) so I'm happy with that. Awesome, It looks like I’ll be writing a PEP to handle this, I wasn’t sure if it needed one or not. Although from MAL's original comment: Note that several python.org services use CAcerts which would no longer be accessible per default following such a change. ,The PSF needs to get that sorted before making cert validation the default in Python, IMO. I’m not aware of which services those are, if MAL (or anyone else) can point them out I’ll see what I can do to make that happen. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22 January 2014 13:29, Christian Heimes christ...@python.org wrote: Side note: Users can simple add self-signed certs to OpenSSL's cert store and get validation for free. It's possible to do that with an environment variable, too. But I recommend against the environment variable because you may overwrite to operating store. I'm pretty sure what I'm about to ask isn't what you mean, but take it as an example of how people may misunderstand and/or misinterpret comments in this area ;-) So if I set up a PyPI mirror running under https, with a self-signed certificate, can you explain how I get it to work? For work, assume I mean pip will use it, I can browse to it with my web browser, and my various Python scripts (now running under Python 3.5 with SSL verification on by default) that query the index all work without needing extra flags, code changes, or interactive prompts. I'm on Windows, by the way, just for added fun. (This is a one of the real-world reasons I've never set up a local https index - not a big one, laziness trumps it by miles :-) as does the effectiveness of simpler solutions - but it's there. I did think about it at one stage. If I *were* to set up an index, it's definitely why I'd use http rather than bothering with https.) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 9:28 AM, Paul Moore p.f.mo...@gmail.com wrote: On 22 January 2014 13:29, Christian Heimes christ...@python.org wrote: Side note: Users can simple add self-signed certs to OpenSSL's cert store and get validation for free. It's possible to do that with an environment variable, too. But I recommend against the environment variable because you may overwrite to operating store. I'm pretty sure what I'm about to ask isn't what you mean, but take it as an example of how people may misunderstand and/or misinterpret comments in this area ;-) So if I set up a PyPI mirror running under https, with a self-signed certificate, can you explain how I get it to work? For work, assume I mean pip will use it, I can browse to it with my web browser, and my various Python scripts (now running under Python 3.5 with SSL verification on by default) that query the index all work without needing extra flags, code changes, or interactive prompts. I'm on Windows, by the way, just for added fun. For everything but pip, you’d add it to your OS cert store. Pip doesn’t use that so you’d have to use the —cert config. (This is a one of the real-world reasons I've never set up a local https index - not a big one, laziness trumps it by miles :-) as does the effectiveness of simpler solutions - but it's there. I did think about it at one stage. If I *were* to set up an index, it's definitely why I'd use http rather than bothering with https.) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22.01.2014 15:12, Jesse Noller wrote: And no one reads it. I can't count the number of times I've gotten called into a managers office when they find out python doesn't do cert validation by default (and in 2, it's not been trivial) and gotten told to fix it, or we move off of python. Donald is perfectly right: every time you point out to users that this is the default behavior the response is almost universally you can't be serious, is this a joke? Yes, you are right. :( About two months ago (maybe three) I proposed to deprecated implicit SSL context, unverified certs and unverified hostnames all together. But I was voted down. Donald made a similar attempt half an year ago, too. Can't we just mark these things as pending deprecated in Python 3.4 so people start fixing their code *now*? Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 9:33 AM, Christian Heimes christ...@python.org wrote: On 22.01.2014 15:12, Jesse Noller wrote: And no one reads it. I can't count the number of times I've gotten called into a managers office when they find out python doesn't do cert validation by default (and in 2, it's not been trivial) and gotten told to fix it, or we move off of python. Donald is perfectly right: every time you point out to users that this is the default behavior the response is almost universally you can't be serious, is this a joke? Yes, you are right. :( About two months ago (maybe three) I proposed to deprecated implicit SSL context, unverified certs and unverified hostnames all together. But I was voted down. Donald made a similar attempt half an year ago, too. Last time I tried the reasoning was that Python couldn’t ship root certs and we couldn’t get to the OS certs everywhere. Thanks to you this is fixed now, so “once more unto the breach”. Can't we just mark these things as pending deprecated in Python 3.4 so people start fixing their code *now*? +1 - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, Jan 22, 2014, at 04:15 AM, Donald Stufft wrote: On Jan 22, 2014, at 6:58 AM, Nick Coghlan ncogh...@gmail.com wrote: On 22 January 2014 21:36, Donald Stufft don...@stufft.io wrote: On Jan 22, 2014, at 6:30 AM, M.-A. Lemburg m...@egenix.com wrote: The change would also disable all services using self-signed certificates which are very common in internal networks and for ad-hoc setups. Many routers and other devices use self-signed certificates when offering HTTPS services. It will just disable them by default, they can still easily be accessed you’d just need to pass the “do not verify” flag. This clearly indicates that you’re opting out of the S in HTTPS. You need to remember that *Python is fundamentally not an application*. We don't control the interaction with the user, application developers do, and every decision we make has to take that into account. The kinds of decisions that an application like a web browser or a package installer can make aren't necessarily available to a runtime. We had to be cautious even with the initial hash randomisation change to avoid breaking currently working applications. The same could be said for requests, it’s fundamentally not an application and can’t control the interaction with the user and yet it validates TLS by default just fine. Speaking of requests, I think another way to address this issue would be import a requests-like APIs into the stdlib (something which should happen anyway) and make that verify certificates by default. This would address the casual urllib-type usecase of fetching files over http/ftp etc. (I expect most people using their own protocols over raw TLS already know to force certificate verification.) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22.01.2014 15:36, Donald Stufft wrote: Last time I tried the reasoning was that Python couldn’t ship root certs and we couldn’t get to the OS certs everywhere. Thanks to you this is fixed now, so “once more unto the breach”. The Windows situation is still not perfect, though. I'd love to use Chrome's approach and directly hook Windows' crypt32 API into OpenSSL verify function. That would trigger automatic retrieval of unknown root certs and CRL checks. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, 22 Jan 2014 15:33:21 +0100 Christian Heimes christ...@python.org wrote: About two months ago (maybe three) I proposed to deprecated implicit SSL context, unverified certs and unverified hostnames all together. But I was voted down. Donald made a similar attempt half an year ago, too. So why are you trying a third time? Do you have any new arguments compared to last time? Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 10:05 AM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 22 Jan 2014 15:33:21 +0100 Christian Heimes christ...@python.org wrote: About two months ago (maybe three) I proposed to deprecated implicit SSL context, unverified certs and unverified hostnames all together. But I was voted down. Donald made a similar attempt half an year ago, too. So why are you trying a third time? Do you have any new arguments compared to last time? See my other email, Last time I tried I was told the reason was there wasn’t a reliable enough default certificate store that worked on platforms such as Windows and Python was unwilling to ship it’s own certificate bundle. Christian has improved this situation so that it appears that this issue has been largely resolved. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014 6:17 AM, M.-A. Lemburg m...@egenix.com wrote: Using an environment switch the extra checks could even be enabled without any code changes. When Donald brought this up it sounded good. It still does. This is similar to what we did for hash randomization. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, Jan 22, 2014, at 04:02 AM, Donald Stufft wrote: On Jan 22, 2014, at 6:45 AM, Nick Coghlan ncogh...@gmail.com wrote: On 22 January 2014 21:21, Paul Moore p.f.mo...@gmail.com wrote: On 22 January 2014 10:30, Donald Stufft don...@stufft.io wrote: Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc and validate by default and in the typical style of security related issues it will appear to work just fine however be grossly insecure. Two things: 1. To be like the browser we'd need to use the OS certificate store, which isn't the case on Windows at the moment (managing those certificate bundle files is most definitely *not* like the browser - I'd have no idea how to add a self-certificate to the bundle file embedded in pip, for example). 2. Your proposal is that because some application authors have not opted in yet, we should penalise the end users of those applications by stopping them being able to use unverified https? And don't forget, applications that haven't opted in will have no switch to allow unverified use. That seems to be punishing the wrong people. Right, the browsers have a whole system of click through security to make the web (and corporate intranets!) still usable even when they only accept CA signed certs by default. With a programming language, there's no such interactivity, so applications just break and users don't know why. It's notable that even Linux distros haven't made this change in their system Python builds, and commercial Linux distros have raised paranoia to an art form (since that's a respectable chunk of what their users are paying for). I was actually talking to a Debian maintainer about the likelihood of making this change there earlier today :) If I fail at making this change in upstream I’ll be lobbying downstream and then we’ll just have different behaviors based on where you get your Python from which I think stinks. I suppose if Debian wants to serve as a test ground to determine whether everyone is happy about having their scripts broken, that's fine, too. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
Donald Stufft donald at stufft.io writes: I would like to propose that a backwards incompatible change be made to Python to make verification of hostname and certificate chain the default instead of requiring it to be opt in. I'm overwhelmingly, dramatically +1 on this. There's no good architectural reason to not use the built-in certificate chains by default. I'd like to be in favour of backporting this change to earlier Python versions as well, but it feels too aggressive, even to me. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22 January 2014 10:30, Donald Stufft don...@stufft.io wrote: I would like to propose that a backwards incompatible change be made to Python to make verification of hostname and certificate chain the default instead of requiring it to be opt in. I'm overwhelmingly, dramatically +1 on this. There's no good architectural reason to not use the built-in certificate chains by default. I'd like to be in favour of backporting this change to earlier Python versions as well, but it feels just a bit too aggressive. Cory ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, 22 Jan 2014 08:12:06 -0700 Eric Snow ericsnowcurren...@gmail.com wrote: On Jan 22, 2014 6:17 AM, M.-A. Lemburg m...@egenix.com wrote: Using an environment switch the extra checks could even be enabled without any code changes. When Donald brought this up it sounded good. It still does. This is similar to what we did for hash randomization. The comparison is baseless. Hash randomization is a language feature that can only be enabled at interpreter startup, and is at best a per-application decision. SSL settings, on the other hand, have to be decided per-client endpoint, not per-process, and they will depend on the external service you connect to rather than the way your code is written. I'm -1 on adding env vars because we can't agree on SSL configuration options. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 1/22/14 8:16 AM, Nick Coghlan wrote: Which is exactly the way most non-web-specialists working inside the comfort of corporate and academic firewalls will react to a change that breaks their access to internal applications, where self-signed certs and improperly configured internal CAs are endemic (of course, that's assuming they're using HTTPS at all, which I admit is an optimistic assumption). The number of people who are using 3.4+ in these environments is probably very very low to be honest. I don't have a number to prove, but in that environment people are more likely to still be using 2.6+. I think a deprecation in 2.7+ would be nice, but forward we should just enable it by default. When requests changed property calls (e.g. requests.json) to callable instead of an attribute(from requests.json to requests.json()), I was shocked. I had to figure out by Googling it. I found out from github issue I think a hard fail is somehow necessary. Also, a lot of people overlook at deprecation warnings. They either don't care or don't see it. I see a lot of deprecation warnings in the older applications I write, but I can careless until it breaks. So as we moving forward, we can break it. For those stuck behind, deprecation is the right approach. John ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, Jan 22, 2014 at 12:10 PM, John Yeuk Hon Wong gokoproj...@gmail.com wrote: On 1/22/14 8:16 AM, Nick Coghlan wrote: Which is exactly the way most non-web-specialists working inside the comfort of corporate and academic firewalls will react to a change that breaks their access to internal applications, where self-signed certs and improperly configured internal CAs are endemic (of course, that's assuming they're using HTTPS at all, which I admit is an optimistic assumption). The number of people who are using 3.4+ in these environments is probably very very low to be honest. I don't have a number to prove, but in that environment people are more likely to still be using 2.6+. I think a deprecation in 2.7+ would be nice, but forward we should just enable it by default. When requests changed property calls (e.g. requests.json) to callable instead of an attribute(from requests.json to requests.json()), I was shocked. I had to figure out by Googling it. I found out from github issue I think a hard fail is somehow necessary. Also, a lot of people overlook at deprecation warnings. They either don't care or don't see it. I see a lot of deprecation warnings in the older applications I write, but I can careless until it breaks. So as we moving forward, we can break it. For those stuck behind, deprecation is the right approach. They're disabled by default, so a lot of people simply don't know they exist because they also don't read the documentation. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 1:46 PM, Brian Curtin br...@python.org wrote: On Wed, Jan 22, 2014 at 12:10 PM, John Yeuk Hon Wong gokoproj...@gmail.com wrote: On 1/22/14 8:16 AM, Nick Coghlan wrote: Which is exactly the way most non-web-specialists working inside the comfort of corporate and academic firewalls will react to a change that breaks their access to internal applications, where self-signed certs and improperly configured internal CAs are endemic (of course, that's assuming they're using HTTPS at all, which I admit is an optimistic assumption). The number of people who are using 3.4+ in these environments is probably very very low to be honest. I don't have a number to prove, but in that environment people are more likely to still be using 2.6+. I think a deprecation in 2.7+ would be nice, but forward we should just enable it by default. When requests changed property calls (e.g. requests.json) to callable instead of an attribute(from requests.json to requests.json()), I was shocked. I had to figure out by Googling it. I found out from github issue I think a hard fail is somehow necessary. Also, a lot of people overlook at deprecation warnings. They either don't care or don't see it. I see a lot of deprecation warnings in the older applications I write, but I can careless until it breaks. So as we moving forward, we can break it. For those stuck behind, deprecation is the right approach. They're disabled by default, so a lot of people simply don't know they exist because they also don't read the documentation. Ironically this is the exact reason why validation should happen by default :] - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, 22 Jan 2014 13:50:37 -0500 Donald Stufft don...@stufft.io wrote: Ironically this is the exact reason why validation should happen by default :] I think most of us would agree that a new client API should do validation by default (with an easy way to opt out). So let's concentrate on the question of whether and how to migrate existing APIs while remaining user-friendly. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
If I could summarize this discussion (please correct me if I am wrong): 1. Status Quo All existing versions of Python are unsecure by default because by not doing SSL hostname verification, libraries which wrap sockets with SSLContext allow SSL MITM (man-in-the-middle) with no warning. Christian Heimes has introduced ss.create_default_context for 3.4 Donald Stuft is proposing that hostname verification should be the default behavior in future versions of Python; such as Python 3.4, ideally as soon as possible. python-dev has been resistant to preventing SSL MITM by default because humans, pypi:keyring, and pypi:certifi 2. Generalizations about end-users: * corporate users aren't susceptible to MITM because perimeter * fallacious on all accounts * mismatched metaphors abound * nobody knows how to add self-signed CA certs to their chains * Agreed. * most people haven't / aren't going to read the docs: http://docs.python.org/3/library/ssl.html#verifying-certificates A **warning::** would probably be appropriate (and a big red warning box has been added to the 3.3 docs) It could also say PYTHON DOES NOT VALIDATE SSL CERTIFICATES BY DEFAULT * most people don't realize / have modification rights / have the ability to submit patches upstream; either to * set CERT_REQUIRED for every connection * set (a forthcoming) CERT_NOVALIDATE * people who complain about breaking security defaults which allow MITM are most relevant * it doesn't work on my SOHO router * people would need to understand how Python (and many other languages' SSL implementations) is less secure than current browsers * people don't read What's New; distributions test and upgrade their interpreter for them * some people are aware that third-party libraries requests and urllib3 do SSL hostname validation by default, now, with Python 2 3. Compatibility There could be a PYTHONSSLNOVALIDATE environment variable: * +1 : This would allow workarounds for applications which cannot be updated * -1 : This is not preferable because PYTHONHASHSEED 4. Chain management * It should be possible to update the cert chain * It could be easier to specify a different cert chain (?) * Python 3.4 now supports using the Windows cert chain * Pip does not yet support using the Windows cert chain * pypi:certifi adds the Mozilla Firefox keychain for Python 2 and 3 (like requests and pip) 5. Deprecation * Should deprecate slowly (2017) because people would complain about having been secured against MITM (in their upgrade to Python 3.4) * Should not deprecate slowly because the status quo (insecure by default) is risky * Should add pending deprecation now (see CERT_NOVALIDATE NOP below) 6. Testing Responses * It's probably good to start testing downstream patches to Python 3.4 packages * The error message may be primed My two cents: * CERT_NOVALIDATE could/should be present now (if even as a no-op) * 2to3 could/should add CERT_NOVALIDATE Please feel free to add any, all, or none of this to the forthcoming PEP. Thank you for addressing this issue. (This is buried here because the mailman archive gzip hasn't yet updated with the latest Message-Id to specify for In-Reply-To) -- Wes Turner On Wed, Jan 22, 2014 at 12:46 PM, Brian Curtin br...@python.org wrote: On Wed, Jan 22, 2014 at 12:10 PM, John Yeuk Hon Wong gokoproj...@gmail.com wrote: On 1/22/14 8:16 AM, Nick Coghlan wrote: Which is exactly the way most non-web-specialists working inside the comfort of corporate and academic firewalls will react to a change that breaks their access to internal applications, where self-signed certs and improperly configured internal CAs are endemic (of course, that's assuming they're using HTTPS at all, which I admit is an optimistic assumption). The number of people who are using 3.4+ in these environments is probably very very low to be honest. I don't have a number to prove, but in that environment people are more likely to still be using 2.6+. I think a deprecation in 2.7+ would be nice, but forward we should just enable it by default. When requests changed property calls (e.g. requests.json) to callable instead of an attribute(from requests.json to requests.json()), I was shocked. I had to figure out by Googling it. I found out from github issue I think a hard fail is somehow necessary. Also, a lot of people overlook at deprecation warnings. They either don't care or don't see it. I see a lot of deprecation warnings in the older applications I write, but I can careless until it breaks. So as we moving forward, we can break it. For those stuck behind, deprecation is the right approach. They're disabled by default, so a lot of people simply don't know they exist because they also don't
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 1/22/2014 9:25 AM, Donald Stufft wrote: Awesome, It looks like I’ll be writing a PEP to handle this, I wasn’t sure if it needed one or not. Definitely. I think the transition from insecure by default to secure by default is somewhat comparable to the transition from ascii by default to unicode by default. I suspect more than one PEP will be needed for various aspects. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 23 Jan 2014 00:39, Benjamin Peterson benja...@python.org wrote: On Wed, Jan 22, 2014, at 04:15 AM, Donald Stufft wrote: On Jan 22, 2014, at 6:58 AM, Nick Coghlan ncogh...@gmail.com wrote: On 22 January 2014 21:36, Donald Stufft don...@stufft.io wrote: On Jan 22, 2014, at 6:30 AM, M.-A. Lemburg m...@egenix.com wrote: The change would also disable all services using self-signed certificates which are very common in internal networks and for ad-hoc setups. Many routers and other devices use self-signed certificates when offering HTTPS services. It will just disable them by default, they can still easily be accessed you’d just need to pass the “do not verify” flag. This clearly indicates that you’re opting out of the S in HTTPS. You need to remember that *Python is fundamentally not an application*. We don't control the interaction with the user, application developers do, and every decision we make has to take that into account. The kinds of decisions that an application like a web browser or a package installer can make aren't necessarily available to a runtime. We had to be cautious even with the initial hash randomisation change to avoid breaking currently working applications. The same could be said for requests, it’s fundamentally not an application and can’t control the interaction with the user and yet it validates TLS by default just fine. Speaking of requests, I think another way to address this issue would be import a requests-like APIs into the stdlib (something which should happen anyway) and make that verify certificates by default. This would address the casual urllib-type usecase of fetching files over http/ftp etc. (I expect most people using their own protocols over raw TLS already know to force certificate verification.) Guido gave in principle approval for an asyncio backed requests clone as the preferred HTTP client API last year, but that's going to take someone to write and publish it if we're going to be able to include it in 3.5. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, Jan 22, 2014, at 12:25 PM, Nick Coghlan wrote: On 23 Jan 2014 00:39, Benjamin Peterson benja...@python.org wrote: Speaking of requests, I think another way to address this issue would be import a requests-like APIs into the stdlib (something which should happen anyway) and make that verify certificates by default. This would address the casual urllib-type usecase of fetching files over http/ftp etc. (I expect most people using their own protocols over raw TLS already know to force certificate verification.) Guido gave in principle approval for an asyncio backed requests clone as the preferred HTTP client API last year, but that's going to take someone to write and publish it if we're going to be able to include it in 3.5. But requests is synchronous, so I'm not sure how much you can use of asyncio. I was thinking of something bolted onto urllib. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, Jan 22, 2014 at 3:56 PM, Benjamin Peterson benja...@python.orgwrote: On Wed, Jan 22, 2014, at 12:25 PM, Nick Coghlan wrote: On 23 Jan 2014 00:39, Benjamin Peterson benja...@python.org wrote: Speaking of requests, I think another way to address this issue would be import a requests-like APIs into the stdlib (something which should happen anyway) and make that verify certificates by default. This would address the casual urllib-type usecase of fetching files over http/ftp etc. (I expect most people using their own protocols over raw TLS already know to force certificate verification.) Guido gave in principle approval for an asyncio backed requests clone as the preferred HTTP client API last year, but that's going to take someone to write and publish it if we're going to be able to include it in 3.5. But requests is synchronous, so I'm not sure how much you can use of asyncio. I was thinking of something bolted onto urllib. Sure, but the key point is that a new async API can be made synchronous as well by blocking as necessary. The eventual synchronous API can resemble or take inspiration from requests. Point is, though, is it was admitted a new module is probably called for thanks to asyncio (and to give us a chance to fix mistakes in urllib). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
Never mind. If someone else cares they can propose it. I withdraw. On Jan 22, 2014, at 4:29 PM, Brett Cannon br...@python.org wrote: On Wed, Jan 22, 2014 at 3:56 PM, Benjamin Peterson benja...@python.org wrote: On Wed, Jan 22, 2014, at 12:25 PM, Nick Coghlan wrote: On 23 Jan 2014 00:39, Benjamin Peterson benja...@python.org wrote: Speaking of requests, I think another way to address this issue would be import a requests-like APIs into the stdlib (something which should happen anyway) and make that verify certificates by default. This would address the casual urllib-type usecase of fetching files over http/ftp etc. (I expect most people using their own protocols over raw TLS already know to force certificate verification.) Guido gave in principle approval for an asyncio backed requests clone as the preferred HTTP client API last year, but that's going to take someone to write and publish it if we're going to be able to include it in 3.5. But requests is synchronous, so I'm not sure how much you can use of asyncio. I was thinking of something bolted onto urllib. Sure, but the key point is that a new async API can be made synchronous as well by blocking as necessary. The eventual synchronous API can resemble or take inspiration from requests. Point is, though, is it was admitted a new module is probably called for thanks to asyncio (and to give us a chance to fix mistakes in urllib). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, Jan 22, 2014 at 3:48 PM, Donald Stufft don...@stufft.io wrote: Never mind. If someone else cares they can propose it. I withdraw. I'll throw writing a PEP for 3.5 to do this following the deprecation policy on my todo list so 3.4 fixing can move on. I needed to brush up on my ReST anyway. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, Jan 22, 2014, at 01:48 PM, Donald Stufft wrote: Never mind. If someone else cares they can propose it. I withdraw. I'm sorry to see this thread went down hill so quickly. I think we can all agree than not validating certs by default is bad and that it should change soon. It's only a question of when. And since we're python-dev, we like to take things in excruciating slow and careful pace. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 23 Jan 2014 07:48, Donald Stufft don...@stufft.io wrote: Never mind. If someone else cares they can propose it. I withdraw. That's unfortunate, but understandable - we already have a lot to deal with just trying to get even the software distribution infrastructure to a secure by default status. However, now we have access to the system cert stores on all major platforms, I *do* think it's a good idea to eventually change the default settings to include host verification. It's just any such concrete proposal, like any other major backwards incompatible change, needs to be written up as a PEP, including a transition plan for insecure environments where users will blame *Python* if an upgrade breaks things, rather than the insecurity of their configuration. We know all too well from the Python 3 transition how unhappy it makes users when a new version complains about environmental issues that previous versions blithely ignored. While the normal deprecation process should suffice, that still means Python 3.6 (2017-ish) is the earliest feasible target for new default settings. Such a proposal will also need to address the implications for source compatible Python 2/3 code across *all* secure network protocols, not just HTTPS (the latter can be handled relatively easily using the requests module). A new, preferred, secure-by-default HTTPS client API for the standard library (based on the requests API) is an orthogonal proposal, and one that already has in-principle approval from Guido, preferably as a synchronous front end to the asyncio networking backend. Regards, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 22.01.2014 23:20, Nick Coghlan wrote: However, now we have access to the system cert stores on all major platforms, I *do* think it's a good idea to eventually change the default settings to include host verification. Somebody has revise the situation on OSX for Python 3.5 and possible create new bindings to the keychain API. OSX has only 0.9.8. Apple has deprecated OpenSSL and I'd like to drop 0.9.8 support in 3.5. Such a proposal will also need to address the implications for source compatible Python 2/3 code across *all* secure network protocols, not just HTTPS (the latter can be handled relatively easily using the requests module). Please count me in! I see two options to handle Python 3.4: backport the ssl module or hope that the cryptography library is ready. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
While the normal deprecation process should suffice, that still means Python 3.6 (2017-ish) is the earliest feasible target for new default settings. Could a CERT_NOVERIFY no-op be added now? (no-op because it would just be more explicit about the default behavior anyway) [1] Such a proposal will also need to address the implications for source compatible Python 2/3 code across *all* secure network protocols, not just HTTPS (the latter can be handled relatively easily using the requests module). Could/should this be a feature of 2to3? [1] (and/or automated AST code scanners for Python source)? [1] https://mail.python.org/pipermail/python-dev/2014-January/131974.html A new, preferred, secure-by-default HTTPS client API for the standard library (based on the requests API) is an orthogonal proposal, and one that already has in-principle approval from Guido, preferably as a synchronous front end to the asyncio networking backend. I'm aware of aiohttp (BSD), but haven't yet had much chance to review the source. It looks like the tests currently require nose and gunicorn, because it provides an asyncio gunicorn worker. http client/server for asyncio https://github.com/fafhrd91/aiohttp -- Wes Turner On Wed, Jan 22, 2014 at 4:20 PM, Nick Coghlan ncogh...@gmail.com wrote: On 23 Jan 2014 07:48, Donald Stufft don...@stufft.io wrote: Never mind. If someone else cares they can propose it. I withdraw. That's unfortunate, but understandable - we already have a lot to deal with just trying to get even the software distribution infrastructure to a secure by default status. However, now we have access to the system cert stores on all major platforms, I *do* think it's a good idea to eventually change the default settings to include host verification. It's just any such concrete proposal, like any other major backwards incompatible change, needs to be written up as a PEP, including a transition plan for insecure environments where users will blame *Python* if an upgrade breaks things, rather than the insecurity of their configuration. We know all too well from the Python 3 transition how unhappy it makes users when a new version complains about environmental issues that previous versions blithely ignored. While the normal deprecation process should suffice, that still means Python 3.6 (2017-ish) is the earliest feasible target for new default settings. Such a proposal will also need to address the implications for source compatible Python 2/3 code across *all* secure network protocols, not just HTTPS (the latter can be handled relatively easily using the requests module). A new, preferred, secure-by-default HTTPS client API for the standard library (based on the requests API) is an orthogonal proposal, and one that already has in-principle approval from Guido, preferably as a synchronous front end to the asyncio networking backend. Regards, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
-Original Message- From: Python-Dev [mailto:python-dev- bounces+kristjan=ccpgames@python.org] On Behalf Of Nick Coghlan Sent: Wednesday, January 22, 2014 19:45 To: Paul Moore Cc: Python-Dev Subject: Re: [Python-Dev] Enable Hostname and Certificate Chain Validation Right, the browsers have a whole system of click through security to make the web (and corporate intranets!) still usable even when they only accept CA signed certs by default. With a programming language, there's no such interactivity, so applications just break and users don't know why. If not already possible, I suggest that we allow the use of a certificate validation callback (it isn't possible for 2.7, I just hacked in one yesterday to allow me to ignore out-date-failure for certificates.) Using this, it would be possible to e.g. emit warnings when certificiate failures occur, rather than deny connection outright. K ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 2014-01-22 9:33 AM, Donald Stufft wrote: For everything but pip, you’d add it to your OS cert store. Pip doesn’t use that so you’d have to use the —cert config. What if I don't want that self-signed cert to be trusted by all users on the system? What if I don't have administrative rights? How do I do it then? Is this common knowledge for average users? Are we trading one big red box in the documentation for another? Anecdotally, I already know of a system at work that is using HTTPS purely for encryption, because the authentication is done in-band. So, a self-signed cert was wholly sufficient. The management tools use a RESTful interface over HTTPS for control, but you are telling me this will be broken by default now. What do I tell our developers (who often adopt the latest and greatest versions of things to play with)? -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
