Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Tue, Jun 4, 2013 at 9:15 AM, Donald Stufft don...@stufft.io wrote: Generally any of these will be completely valid options, even disabling the checks. The idea behind my proposal is that people generally only use TLS for a reason and that reason is they want to protect against the kinds of attacks that TLS protects against. You really only get the bulk of those protections if you validate the certificates. So for the vast bulk of people validation is the option they want. It also happens that validation on by default is secure by default which aims a fairly large foot gun away from peoples feet. Yep. I absolutely agree, and if Python had been like this ever since 3.0, there would be no issue whatsoever. The only problem is that it's going to trip up people who have otherwise-working code that gets broken by a change of Python version. The problem isn't even in their code, it's in the remote end. I support making the change, but all of the solutions you propose require being aware of the problem. Somehow people will need to be told, preferably before it breaks something, that the default has changed. ChrisA ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On 3 Jun, 2013, at 7:58, Benjamin Peterson benja...@python.org wrote: 2013/6/2 Donald Stufft don...@stufft.io: As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. +1 Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. My preference would be actually be for the included certificates file to be used by default. This would provide a consistent experience across platforms. We could provide options to look for system cert repositories if desired. I'd prefer to use the system CA list when that's available. I've had to hunt down the CA list for a number of application when a custom CA for internal use and that's not fun, using the system list is much friendlier to users. Ronald ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
2013/6/2 Donald Stufft don...@stufft.io: As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. +1 Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. My preference would be actually be for the included certificates file to be used by default. This would provide a consistent experience across platforms. We could provide options to look for system cert repositories if desired. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 1:58 AM, Benjamin Peterson benja...@python.org wrote: 2013/6/2 Donald Stufft don...@stufft.io: As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. +1 Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. My preference would be actually be for the included certificates file to be used by default. This would provide a consistent experience across platforms. We could provide options to look for system cert repositories if desired. That's fine with me too. My only reason for wanting to use the system certs first is so if someone has modified their system certs (say to include a corporate cert) that it would ideally take affect for Python as well. But honestly the Linux distros will probably modify things to use system certs anyways and non Linux (esp Windows) probably doesn't have a way to get those system certs into OpenSSL. -- Regards, Benjamin - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
2013/6/3 Donald Stufft don...@stufft.io: On Jun 3, 2013, at 1:58 AM, Benjamin Peterson benja...@python.org wrote: 2013/6/2 Donald Stufft don...@stufft.io: As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. +1 Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. My preference would be actually be for the included certificates file to be used by default. This would provide a consistent experience across platforms. We could provide options to look for system cert repositories if desired. That's fine with me too. My only reason for wanting to use the system certs first is so if someone has modified their system certs (say to include a corporate cert) that it would ideally take affect for Python as well. I don't think users should be able to modify stdlib behaviors (in this case could be unintentionally) without application consent. But honestly the Linux distros will probably modify things to use system certs anyways and non Linux (esp Windows) probably doesn't have a way to get those system certs into OpenSSL. Yes, I'm happy to let them figure it out. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
Love this idea. Some third-party HTTP libraries turn this on by default in any case (eg: requests, and I think others), so this would mean Python would get their safe-by-default behaviour in its stdlib. Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. My preference would be actually be for the included certificates file to be used by default. This would provide a consistent experience across platforms. We could provide options to look for system cert repositories if desired. Very much agreed. When the Windows version of the mimetypes module tried to use Windows' system mimetype mappings by default, chaos and bugs ensued (for example, http://bugs.python.org/issue15207 and http://bugs.python.org/issue10551). -Ben ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Mon, 3 Jun 2013 01:20:42 -0400 Donald Stufft don...@stufft.io wrote: As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. Also for backwards compatibility. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Sun, 2 Jun 2013 22:57:15 -0700 Chris Rebert c...@rebertia.com wrote: On Jun 2, 2013 10:22 PM, Donald Stufft don...@stufft.io wrote: As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. Relevant: http://bugs.python.org/issue13647 So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x does). Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend solely on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. There's an existing request for this: http://bugs.python.org/issue13655 See also http://bugs.python.org/issue17134 Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Mon, 03 Jun 2013 19:38:45 +1200, Ben Hoyt benh...@gmail.com wrote: Love this idea. Some third-party HTTP libraries turn this on by default in any case (eg: requests, and I think others), so this would mean Python would get their safe-by-default behaviour in its stdlib. Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. My preference would be actually be for the included certificates file to be used by default. This would provide a consistent experience across platforms. We could provide options to look for system cert repositories if desired. Very much agreed. When the Windows version of the mimetypes module tried to use Windows' system mimetype mappings by default, chaos and bugs ensued (for example, http://bugs.python.org/issue15207 and http://bugs.python.org/issue10551). On the other hand, the fact that it uses /etc/mimetypes on unix does *not* cause chaos, quite the opposite: it means Python recognizes new mime types provided by the distro automatically, which provides for a more consistent experience for the Python application user. The situation for certs is probably fairly parallel: on unix, it would probably be an advantage as Python would automatically follow distro decisions about cert chains, while on windows trying to use a system cert store would probably be a disaster. An application can choose to explicitly ignore the system mimetypes file, by the way. That said, I don't feel strongly either way about the cert store case. --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
The situation for certs is probably fairly parallel: on unix, it would probably be an advantage as Python would automatically follow distro decisions about cert chains, while on windows trying to use a system cert store would probably be a disaster. Yeah, fair enough. If it's stable and just works on Linux, this would be an advantage. An application can choose to explicitly ignore the system mimetypes file, by the way. My main concern is that it could be broken out of the box on Windows (mimetypes currently is), and callers have to go out of their way to find this workaround. I'm not familiar with Unix/Linux, but on Windows, if it's anything like mimetypes it'll be really hard to get consistent behaviour across different boxes/versions from the registry, or wherever certs might be stored on Windows. I'd much rather have a slightly outdated but consistent experience by default. -Ben ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
Am 03.06.2013 07:20, schrieb Donald Stufft: Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. On Windows it's rather easy to access the cert storage with a couple of calls to crypt32.dll. I have created a smallish ctypes interface: https://bitbucket.org/tiran/wincertstore/overview . setuptools 0.7 is using it to for its ssl support. I'm going to write a patch for Python 3.4, too. See http://bugs.python.org/issue17134 I welcome a cert bundle in CPython. But how are we suppose to handle updates and invalidation of CAs? A couple of months ago there was an intense discussion about a copy of the Olson time zone database in Python's stdlib. People argued against it as they feared outdated tz information. IMHO outdated CA certs are much more severe than tz ... I'd like to see OCSP support, too. Unfortunately OpenSSL's OCSP API is far from trivial. I had a look at it once. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Mon, 3 Jun 2013 21:37:10 +1200 Ben Hoyt benh...@gmail.com wrote: I'm not familiar with Unix/Linux, but on Windows, if it's anything like mimetypes it'll be really hard to get consistent behaviour across different boxes/versions from the registry, or wherever certs might be stored on Windows. I'd much rather have a slightly outdated but consistent experience by default. The problem with a slightly outdated CA store is that it can be a security risk. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
I'm not familiar with Unix/Linux, but on Windows, if it's anything like mimetypes it'll be really hard to get consistent behaviour across different boxes/versions from the registry, or wherever certs might be stored on Windows. I'd much rather have a slightly outdated but consistent experience by default. The problem with a slightly outdated CA store is that it can be a security risk. True. This is different from mimetypes in that respect. Also, with Christian's post about Windows cert store, it does look like Windows has a stable/documented way to get this from the system. I wonder, are the crypt32.dll Cert* functions what IE uses? -Ben ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Mon, Jun 3, 2013 at 1:20 AM, Donald Stufft don...@stufft.io wrote: As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x does). Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. +1. I bundle a copy of the Mozilla CA list with Tornado, but would love to access the system's CA roots and/or use a Python-provided copy. I'd prefer to use the certificates from the operating system when possible, as that list is most likely to receive timely security updates (or be updated with a local corporate CA, for example). It's better to aim for consistency with the user's browser than consistency of Python applications across different installations. The data is analogous to the time zone database (PEP 431) in that it may need to be updated independently of Python's own release schedule, so we may want to use similar techniques to manage both. Also see certifi ( https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list in a pip-installable form. -Ben - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On 06/02/2013 10:20 PM, Donald Stufft wrote: So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. +1 Also, +1 on using system certs when available. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 5:51 AM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 3 Jun 2013 21:37:10 +1200 Ben Hoyt benh...@gmail.com wrote: I'm not familiar with Unix/Linux, but on Windows, if it's anything like mimetypes it'll be really hard to get consistent behaviour across different boxes/versions from the registry, or wherever certs might be stored on Windows. I'd much rather have a slightly outdated but consistent experience by default. The problem with a slightly outdated CA store is that it can be a security risk. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just need to watch them for blacklisting certs and roll that into a security update. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Mon, 3 Jun 2013 12:43:32 -0400 Donald Stufft don...@stufft.io wrote: On Jun 3, 2013, at 5:51 AM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 3 Jun 2013 21:37:10 +1200 Ben Hoyt benh...@gmail.com wrote: I'm not familiar with Unix/Linux, but on Windows, if it's anything like mimetypes it'll be really hard to get consistent behaviour across different boxes/versions from the registry, or wherever certs might be stored on Windows. I'd much rather have a slightly outdated but consistent experience by default. The problem with a slightly outdated CA store is that it can be a security risk. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just need to watch them for blacklisting certs and roll that into a security update. Let's see if our security release managers want to do that job. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On 06/03/2013 09:43 AM, Donald Stufft wrote: On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote: The problem with a slightly outdated CA store is that it can be a security risk. Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just need to watch them for blacklisting certs and roll that into a security update. Personally, I'm not interested in waiting six months for an update. And why can't I have the new additions? Seems to me a better solution is to have routines that can query and update at will (meaning the app has to call them), as well as having the bundle (black lists as well as new additions) in the regular updates. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 12:52 PM, Ethan Furman et...@stoneleaf.us wrote: On 06/03/2013 09:43 AM, Donald Stufft wrote: On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote: The problem with a slightly outdated CA store is that it can be a security risk. Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just need to watch them for blacklisting certs and roll that into a security update. Personally, I'm not interested in waiting six months for an update. And why can't I have the new additions? Seems to me a better solution is to have routines that can query and update at will (meaning the app has to call them), as well as having the bundle (black lists as well as new additions) in the regular updates. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io Personally I view that as a nice to have but if an app is willing to call hem they can easily depend on certifi and just replace the default certificates. The goal here is to get secure by default without any work on the part of the developers. If a developer knows to update the certs they can know to use certifi. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 03, 2013, at 01:20 AM, Donald Stufft wrote: So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x does). For the verify HTTPS by default, do you mean specifically changing the cadefault argument to urllib.request.urlopen() to True? Note that I recently closed a bug related to this: http://bugs.python.org/issue17977 +1 for changing the default to True. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 03, 2013, at 09:05 AM, Ben Darnell wrote: The data is analogous to the time zone database (PEP 431) in that it may need to be updated independently of Python's own release schedule, so we may want to use similar techniques to manage both. Also see certifi ( https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list in a pip-installable form. Right, this is very much analogous, except with the additional twist that out-of-date certificates can pose a significant security risk. I'm fairly certain that Debian and Ubuntu would explicitly not use any certificates shipped with Python, for two main reasons: 1) our security teams already manage the certificate store distro-wide and we want to make sure that one update fixes everything; 2) we don't want to duplicate code in multiple packages[1]. So *if* Python decides to do this (and I'm -0, but from a decidedly Linux-distro bias), it must be easily disabled. I generally like the way PEP 431 handles the tzdata, so I think we should do the same here. -Barry [1] This gives us headaches in upstreams like coverage caused by bundling externally available JavaScript libraries, or like urllib3 bundling chardet and urllib3, not to mention their own certificates yet again. :( ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 12:36 PM, Barry Warsaw ba...@python.org wrote: On Jun 03, 2013, at 01:20 AM, Donald Stufft wrote: So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x does). For the verify HTTPS by default, do you mean specifically changing the cadefault argument to urllib.request.urlopen() to True? Note that I recently closed a bug related to this: http://bugs.python.org/issue17977 +1 for changing the default to True. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io If that's all it takes to make sure that people have to opt out of an invalid certificate throwing an error than yes :) My goal here is to make it that when someone accesses a TLS secured network resource (I said HTTP, but if there are other things, xmlrpclib, or what not where people can reasonably expect valid TLS certificates those too) they are protected by a MITM attack by default. I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are safe. As with most things security the outcome of failing to do it right when the default is insecure is well nothing until someone attacks you. So Hopefully we make things secure by default for folks :) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 03, 2013, at 03:12 AM, Donald Stufft wrote: That's fine with me too. My only reason for wanting to use the system certs first is so if someone has modified their system certs (say to include a corporate cert) that it would ideally take affect for Python as well. This reminds me of one other thing. We have to make sure that the APIs (e.g urlopen()) continue to allow us to use self-signed certificates, if for no other reason than for testing purposes. OTOH, taking this away would be a backward incompatible change in API so probably wouldn't happen anyway. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 12:52 PM, Barry Warsaw ba...@python.org wrote: On Jun 03, 2013, at 03:12 AM, Donald Stufft wrote: That's fine with me too. My only reason for wanting to use the system certs first is so if someone has modified their system certs (say to include a corporate cert) that it would ideally take affect for Python as well. This reminds me of one other thing. We have to make sure that the APIs (e.g urlopen()) continue to allow us to use self-signed certificates, if for no other reason than for testing purposes. OTOH, taking this away would be a backward incompatible change in API so probably wouldn't happen anyway. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io The other additional comment I'd like to throw in here is that if we don't bundle SSL certs I think we should still verify by default (which means HTTPS urls will throw an error by default if we can't locate a certificate store) because I think the risk to people unknowingly thinking that their HTTPS urls are protected are significant enough that this error shouldn't be silent by default. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 03, 2013, at 02:21 PM, Donald Stufft wrote: The other additional comment I'd like to throw in here is that if we don't bundle SSL certs I think we should still verify by default (which means HTTPS urls will throw an error by default if we can't locate a certificate store) because I think the risk to people unknowingly thinking that their HTTPS urls are protected are significant enough that this error shouldn't be silent by default. +1, especially if we ensure that the APIs are available to not verify, as is currently the case with urlopen(). I don't think people will want to do that in production, but it will be useful for testing (e.g. guess how I found issues 17977 :). -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 12:48 PM, Barry Warsaw ba...@python.org wrote: On Jun 03, 2013, at 09:05 AM, Ben Darnell wrote: The data is analogous to the time zone database (PEP 431) in that it may need to be updated independently of Python's own release schedule, so we may want to use similar techniques to manage both. Also see certifi ( https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list in a pip-installable form. Right, this is very much analogous, except with the additional twist that out-of-date certificates can pose a significant security risk. I'm fairly certain that Debian and Ubuntu would explicitly not use any certificates shipped with Python, for two main reasons: 1) our security teams already manage the certificate store distro-wide and we want to make sure that one update fixes everything; 2) we don't want to duplicate code in multiple packages[1]. So *if* Python decides to do this (and I'm -0, but from a decidedly Linux-distro bias), it must be easily disabled. I generally like the way PEP 431 handles the tzdata, so I think we should do the same here. I'd actually prefer for Linux to not use the bundled certs when installed from a package manager because it should use the system certs, but people can't depend on certs being there if they are only there on linux. Adding them into Python means people _can_ depend on them being there, and Windows and other systems without system integrators to modify it to use the system store will still get certs and Ubuntu can make it just work(™). -Barry [1] This gives us headaches in upstreams like coverage caused by bundling externally available JavaScript libraries, or like urllib3 bundling chardet and urllib3, not to mention their own certificates yet again. :( This would probably (eventually) make the bundling of certificates better too. Meaning that once it's been in long enough people are willing to depend on it, they won't need to bundle their own certs and ubuntu/debian can just modify the one location instead of needing to modify it for every package that does it. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 03, 2013, at 02:17 PM, Donald Stufft wrote: I'd actually prefer for Linux to not use the bundled certs when installed from a package manager because it should use the system certs, but people can't depend on certs being there if they are only there on linux. I think we agree on that. Adding them into Python means people _can_ depend on them being there, and Windows and other systems without system integrators to modify it to use the system store will still get certs and Ubuntu can make it just work(™). Again, I think PEP 431 provides a pretty good model for how this should be done. Maybe it's worth factoring out this specific part of PEP 431 into an informational PEP? This would probably (eventually) make the bundling of certificates better too. Meaning that once it's been in long enough people are willing to depend on it, they won't need to bundle their own certs and ubuntu/debian can just modify the one location instead of needing to modify it for every package that does it. Can we do the same for the JavaScript libraries? :) -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On 06/03/2013 10:55 AM, Donald Stufft wrote: On Jun 3, 2013, at 12:52 PM, Ethan Furman wrote: On 06/03/2013 09:43 AM, Donald Stufft wrote: On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote: The problem with a slightly outdated CA store is that it can be a security risk. Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just need to watch them for blacklisting certs and roll that into a security update. Personally, I'm not interested in waiting six months for an update. And why can't I have the new additions? Seems to me a better solution is to have routines that can query and update at will (meaning the app has to call them), as well as having the bundle (black lists as well as new additions) in the regular updates. Personally I view that as a nice to have but if an app is willing to call hem they can easily depend on certifi and just replace the default certificates. The goal here is to get secure by default without any work on the part of the developers. If a developer knows to update the certs they can know to use certifi. Let's pretend I want to do something with https. I'm pretty much a complete novice in that area, but I do know about these certificate things and that I should use them. (Which will probably happen sooner rather than later. ;) From my POV, having the bundled certs in the stdlib is great, but also having something in the stdlib to query for and download updates would also be great. I just looked at certifi and the only thing it has on there is `where` which tells me where the certs are on the system. This on only half the problem. If we're going to solve it, why not solve the whole problem? -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Mon, 3 Jun 2013 14:12:34 -0400 Donald Stufft don...@stufft.io wrote: I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are safe. Yet there's quite a visible warning in the docs: http://docs.python.org/dev/library/urllib.request.html Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 1:07 PM, Barry Warsaw ba...@python.org wrote: On Jun 03, 2013, at 02:17 PM, Donald Stufft wrote: I'd actually prefer for Linux to not use the bundled certs when installed from a package manager because it should use the system certs, but people can't depend on certs being there if they are only there on linux. I think we agree on that. Adding them into Python means people _can_ depend on them being there, and Windows and other systems without system integrators to modify it to use the system store will still get certs and Ubuntu can make it just work(™). Again, I think PEP 431 provides a pretty good model for how this should be done. Maybe it's worth factoring out this specific part of PEP 431 into an informational PEP? Looks fine to me minus the not updating in security releases (but that's just a difference in the type of data). This would probably (eventually) make the bundling of certificates better too. Meaning that once it's been in long enough people are willing to depend on it, they won't need to bundle their own certs and ubuntu/debian can just modify the one location instead of needing to modify it for every package that does it. Can we do the same for the JavaScript libraries? :) -Barry - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Mon, Jun 3, 2013 at 1:04 PM, Barry Warsaw ba...@python.org wrote: On Jun 03, 2013, at 02:21 PM, Donald Stufft wrote: The other additional comment I'd like to throw in here is that if we don't bundle SSL certs I think we should still verify by default (which means HTTPS urls will throw an error by default if we can't locate a certificate store) because I think the risk to people unknowingly thinking that their HTTPS urls are protected are significant enough that this error shouldn't be silent by default. +1, especially if we ensure that the APIs are available to not verify, as is currently the case with urlopen(). I don't think people will want to do that in production, but it will be useful for testing (e.g. guess how I found issues 17977 :). +1 from me as well. Whether we bundle or simply provide a command to download the certs I think making this default is the bare-minimum, especially if setting nothing more than cadefault=True is all that is needed to get this behaviour since that's backwards-compatible to Python 3.3. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On 06/03/2013 11:34 AM, Antoine Pitrou wrote: On Mon, 3 Jun 2013 14:12:34 -0400 Donald Stufft wrote: I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are safe. Yet there's quite a visible warning in the docs: http://docs.python.org/dev/library/urllib.request.html As has been been mentioned elsewhere, at the very least we should change cadefault to True for secure-type connections. The dangerous/unexpected behavior should not be the default. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Mon, 03 Jun 2013 11:56:45 -0700 Ethan Furman et...@stoneleaf.us wrote: On 06/03/2013 11:34 AM, Antoine Pitrou wrote: On Mon, 3 Jun 2013 14:12:34 -0400 Donald Stufft wrote: I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are safe. Yet there's quite a visible warning in the docs: http://docs.python.org/dev/library/urllib.request.html As has been been mentioned elsewhere, at the very least we should change cadefault to True for secure-type connections. The dangerous/unexpected behavior should not be the default. cadefault=True will probably be fail if the system certs are not properly configured in OpenSSL, e.g. under Windows or with a hand-made OpenSSL build. And, because of the way the OpenSSL API works, there's no way of knowing if it is the case or not: http://docs.python.org/3.4/library/ssl.html#ssl.SSLContext.set_default_verify_paths While I'm not saying we shouldn't try to verify certs by default, doing so is a little more delicate than setting cadefault to True, unfortunately. Bundling our own CA cert store is an option, assuming Donald or someone else wants to take responsibility for it. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Tue, Jun 4, 2013 at 4:56 AM, Ethan Furman et...@stoneleaf.us wrote: On 06/03/2013 11:34 AM, Antoine Pitrou wrote: On Mon, 3 Jun 2013 14:12:34 -0400 Donald Stufft wrote: I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are safe. Yet there's quite a visible warning in the docs: http://docs.python.org/dev/library/urllib.request.html As has been been mentioned elsewhere, at the very least we should change cadefault to True for secure-type connections. The dangerous/unexpected behavior should not be the default. +1 for having the default be safe, but this will have to be very loudly announced (when migrating from 3.3 to 3.4, TLS connections will cease to work if blah blah). Some legit sites with proper certificates still manage to muck something up administratively (developer.quicksales.com.au has a cert from RapidSSL but doesn't bundle the intermediates, and I've told their devs about it, but all I can do is disable cert checking). This will break code in ways that will surprise people greatly. But I'd still rather the default be True. ChrisA ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Mon, 2013-06-03 at 12:48 -0400, Barry Warsaw wrote: On Jun 03, 2013, at 09:05 AM, Ben Darnell wrote: The data is analogous to the time zone database (PEP 431) in that it may need to be updated independently of Python's own release schedule, so we may want to use similar techniques to manage both. Also see certifi ( https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list in a pip-installable form. Right, this is very much analogous, except with the additional twist that out-of-date certificates can pose a significant security risk. I'm fairly certain that Debian and Ubuntu would explicitly not use any certificates shipped with Python, for two main reasons: 1) our security teams already manage the certificate store distro-wide and we want to make sure that one update fixes everything; 2) we don't want to duplicate code in multiple packages[1]. Fedora/RHEL are in a similar position; I expect we'd rip out the bundled certs in our builds shortly after unzipping the tarball, and use a system-wide cert store (I rm -rf bundled libraries in our builds, to make sure we're not using them). [...snip...] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
Am 03.06.2013 21:52, schrieb Antoine Pitrou: cadefault=True will probably be fail if the system certs are not properly configured in OpenSSL, e.g. under Windows or with a hand-made OpenSSL build. And, because of the way the OpenSSL API works, there's no way of knowing if it is the case or not: http://docs.python.org/3.4/library/ssl.html#ssl.SSLContext.set_default_verify_paths I only see an issue for uncommon Linux distributions and exotic Unices. For Windows an interface to crypt32 API solves the CA issue as shown in my wincertstore module. It gives the user the same SSL experience as Internet Explorer. Most Linux and BSD-ish operating systems have SSL certs at some standard location. https://bitbucket.org/pypa/setuptools/src/6de3186fdfd9f5b543380e9aca2d48976cfc38cd/setuptools/ssl_support.py?at=default#cl-15 lists a couple of standard locations. Under which conditions do we need to ship a CA cert file? Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 4:19 PM, Christian Heimes christ...@python.org wrote: Am 03.06.2013 21:52, schrieb Antoine Pitrou: cadefault=True will probably be fail if the system certs are not properly configured in OpenSSL, e.g. under Windows or with a hand-made OpenSSL build. And, because of the way the OpenSSL API works, there's no way of knowing if it is the case or not: http://docs.python.org/3.4/library/ssl.html#ssl.SSLContext.set_default_verify_paths I only see an issue for uncommon Linux distributions and exotic Unices. For Windows an interface to crypt32 API solves the CA issue as shown in my wincertstore module. It gives the user the same SSL experience as Internet Explorer. Most Linux and BSD-ish operating systems have SSL certs at some standard location. https://bitbucket.org/pypa/setuptools/src/6de3186fdfd9f5b543380e9aca2d48976cfc38cd/setuptools/ssl_support.py?at=default#cl-15 lists a couple of standard locations. Under which conditions do we need to ship a CA cert file? Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io What about OSX? - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
In article 699d9ab5-26e0-4cce-89c5-8e92c3305...@stufft.io, Donald Stufft don...@stufft.io wrote: On Jun 3, 2013, at 4:19 PM, Christian Heimes christ...@python.org wrote: Most Linux and BSD-ish operating systems have SSL certs at some standard location. https://bitbucket.org/pypa/setuptools/src/6de3186fdfd9f5b543380e9aca2d48976c fc38cd/setuptools/ssl_support.py?at=default#cl-15 lists a couple of standard locations. Under which conditions do we need to ship a CA cert file? What about OSX? OS X has an actively managed set of CA certs, by Apple and with provision for users to tailor their security policies (add/modify/delete certs and their policies via either a gui or cli). For python.org OS X installers, we currently depend on the system-provided versions of libssl and libcrypto which use the system/user cert infrastructure. But Apple has deprecated the OpenSSL-compatible APIs and we have an open issue (Issue17128) to move to supplying our own copy of OpenSSL libs. One hangup has been what to do about the certs. One solution would be to use a Python distribution of them as discussed here. Another, probably better for users, solution is to continue to use the existing system/user cert infrastructure; recently, we've seen a new approach to that which we need to look into: http://bugs.python.org/issue17128#msg189244. Third-party distributors of Python on OS X may use other solutions; for example, MacPorts already ships and links with its own version of OpenSSL. -- Ned Deily, n...@acm.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On 3 June 2013 21:05, Chris Angelico ros...@gmail.com wrote: +1 for having the default be safe, but this will have to be very loudly announced (when migrating from 3.3 to 3.4, TLS connections will cease to work if blah blah). +1 on the default being safe, certainly. But with the proviso that the same code should work in 3.3 and 3.4, with no user impact (other than that the connection is safer, but that's not user-visible unless there's an attack :-)) In other words, that will cease to work clause should not exist - but see below... Some legit sites with proper certificates still manage to muck something up administratively (developer.quicksales.com.au has a cert from RapidSSL but doesn't bundle the intermediates, and I've told their devs about it, but all I can do is disable cert checking). This will break code in ways that will surprise people greatly. But I'd still rather the default be True. I'm happy if the will cease to work clause only says some sites with broken security configurations may stop working with a clear explanation that it is *their* fault, not Python's. I'd also expect that the same sites would fail in browsers - if not, we should also be able to make them work (or face cries of well, Internet Explorer/Firefox doesn't have a problem with my site, why does Python?). Also, we should consider the issue for application users. Suppose I'm using a Python application that downloads something from the web. I upgrade to 3.4, and the app stops working because of a will cease to work case. As an end user, how can I get the app working again? Having to patch the sources isn't an option, and reverting to 3.3 provokes the reaction Python broke my app. Summary - I'm +1 as long as either the will cease to work list is empty, or we have a *very* good story for (legitimate) sites which do cease to work. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Mon, 3 Jun 2013 22:31:40 +0100 Paul Moore p.f.mo...@gmail.com wrote: Some legit sites with proper certificates still manage to muck something up administratively (developer.quicksales.com.au has a cert from RapidSSL but doesn't bundle the intermediates, and I've told their devs about it, but all I can do is disable cert checking). This will break code in ways that will surprise people greatly. But I'd still rather the default be True. I'm happy if the will cease to work clause only says some sites with broken security configurations may stop working with a clear explanation that it is *their* fault, not Python's. I'd also expect that the same sites would fail in browsers - if not, we should also be able to make them work (or face cries of well, Internet Explorer/Firefox doesn't have a problem with my site, why does Python?). Keep in mind that not every HTTPS service is a Web site that is meant to be readable with a browser. Some are Web services, possibly internal, possibly without a domain name (and, therefore, probably a non-matching certificate subject name). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Tue, Jun 4, 2013 at 7:31 AM, Paul Moore p.f.mo...@gmail.com wrote: On 3 June 2013 21:05, Chris Angelico ros...@gmail.com wrote: Some legit sites with proper certificates still manage to muck something up administratively (developer.quicksales.com.au has a cert from RapidSSL but doesn't bundle the intermediates, and I've told their devs about it, but all I can do is disable cert checking). This will break code in ways that will surprise people greatly. But I'd still rather the default be True. I'm happy if the will cease to work clause only says some sites with broken security configurations may stop working with a clear explanation that it is *their* fault, not Python's. I'd also expect that the same sites would fail in browsers - if not, we should also be able to make them work (or face cries of well, Internet Explorer/Firefox doesn't have a problem with my site, why does Python?). In this specific case, I get no warning from Google Chrome on either Windows or Linux, but I do get a warning from Iceweasel on the same Linux (it's Debian Wheezy, fwiw). So there probably will be cries of well, Chrome doesn't etc etc. Also, we should consider the issue for application users. Suppose I'm using a Python application that downloads something from the web. I upgrade to 3.4, and the app stops working because of a will cease to work case. As an end user, how can I get the app working again? Having to patch the sources isn't an option, and reverting to 3.3 provokes the reaction Python broke my app. Right. The instance I'm citing comes from what I'm doing at work; I had to disable certificate checking in order to get a non-interactive application to work. If I'd never thought to make sure cert checking was happening, the app could easily have been unexpectedly broken by this change. But you're absolutely right. It's a failing at the remote end, or... On Tue, Jun 4, 2013 at 7:41 AM, Antoine Pitrou solip...@pitrou.net wrote: Keep in mind that not every HTTPS service is a Web site that is meant to be readable with a browser. Some are Web services, possibly internal, possibly without a domain name (and, therefore, probably a non-matching certificate subject name). ... this. This isn't a problem, as checking can easily be disabled, but I'd be inclined to recommend that this one be solved without changing code by explicitly importing that certificate. That would resolve everything, but would require administrative action. ChrisA ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 5:41 PM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 3 Jun 2013 22:31:40 +0100 Paul Moore p.f.mo...@gmail.com wrote: Some legit sites with proper certificates still manage to muck something up administratively (developer.quicksales.com.au has a cert from RapidSSL but doesn't bundle the intermediates, and I've told their devs about it, but all I can do is disable cert checking). This will break code in ways that will surprise people greatly. But I'd still rather the default be True. I'm happy if the will cease to work clause only says some sites with broken security configurations may stop working with a clear explanation that it is *their* fault, not Python's. I'd also expect that the same sites would fail in browsers - if not, we should also be able to make them work (or face cries of well, Internet Explorer/Firefox doesn't have a problem with my site, why does Python?). Keep in mind that not every HTTPS service is a Web site that is meant to be readable with a browser. Some are Web services, possibly internal, possibly without a domain name (and, therefore, probably a non-matching certificate subject name). They should need to explicitly opt in to disabling the checks that allow that to work. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 5:31 PM, Paul Moore p.f.mo...@gmail.com wrote: On 3 June 2013 21:05, Chris Angelico ros...@gmail.com wrote: +1 for having the default be safe, but this will have to be very loudly announced (when migrating from 3.3 to 3.4, TLS connections will cease to work if blah blah). +1 on the default being safe, certainly. But with the proviso that the same code should work in 3.3 and 3.4, with no user impact (other than that the connection is safer, but that's not user-visible unless there's an attack :-)) If we bundle certs that will be the case sans connections where it doesn't validate. In other words, that will cease to work clause should not exist - but see below... Some legit sites with proper certificates still manage to muck something up administratively (developer.quicksales.com.au has a cert from RapidSSL but doesn't bundle the intermediates, and I've told their devs about it, but all I can do is disable cert checking). This will break code in ways that will surprise people greatly. But I'd still rather the default be True. I'm happy if the will cease to work clause only says some sites with broken security configurations may stop working with a clear explanation that it is *their* fault, not Python's. I'd also expect that the same sites would fail in browsers - if not, we should also be able to make them work (or face cries of well, Internet Explorer/Firefox doesn't have a problem with my site, why does Python?). Browsers tend to download intermediate certs while I don't think Python does. Also, we should consider the issue for application users. Suppose I'm using a Python application that downloads something from the web. I upgrade to 3.4, and the app stops working because of a will cease to work case. As an end user, how can I get the app working again? Having to patch the sources isn't an option, and reverting to 3.3 provokes the reaction Python broke my app. Supply a SSL vert using the environment variable? Summary - I'm +1 as long as either the will cease to work list is empty, or we have a *very* good story for (legitimate) sites which do cease to work. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Mon, 3 Jun 2013 17:47:31 -0400 Donald Stufft don...@stufft.io wrote: On Jun 3, 2013, at 5:41 PM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 3 Jun 2013 22:31:40 +0100 Paul Moore p.f.mo...@gmail.com wrote: Some legit sites with proper certificates still manage to muck something up administratively (developer.quicksales.com.au has a cert from RapidSSL but doesn't bundle the intermediates, and I've told their devs about it, but all I can do is disable cert checking). This will break code in ways that will surprise people greatly. But I'd still rather the default be True. I'm happy if the will cease to work clause only says some sites with broken security configurations may stop working with a clear explanation that it is *their* fault, not Python's. I'd also expect that the same sites would fail in browsers - if not, we should also be able to make them work (or face cries of well, Internet Explorer/Firefox doesn't have a problem with my site, why does Python?). Keep in mind that not every HTTPS service is a Web site that is meant to be readable with a browser. Some are Web services, possibly internal, possibly without a domain name (and, therefore, probably a non-matching certificate subject name). They should need to explicitly opt in to disabling the checks that allow that to work. Obviously, which means compatibility is broken with existing code. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 5:51 PM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 3 Jun 2013 17:47:31 -0400 Donald Stufft don...@stufft.io wrote: On Jun 3, 2013, at 5:41 PM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 3 Jun 2013 22:31:40 +0100 Paul Moore p.f.mo...@gmail.com wrote: Some legit sites with proper certificates still manage to muck something up administratively (developer.quicksales.com.au has a cert from RapidSSL but doesn't bundle the intermediates, and I've told their devs about it, but all I can do is disable cert checking). This will break code in ways that will surprise people greatly. But I'd still rather the default be True. I'm happy if the will cease to work clause only says some sites with broken security configurations may stop working with a clear explanation that it is *their* fault, not Python's. I'd also expect that the same sites would fail in browsers - if not, we should also be able to make them work (or face cries of well, Internet Explorer/Firefox doesn't have a problem with my site, why does Python?). Keep in mind that not every HTTPS service is a Web site that is meant to be readable with a browser. Some are Web services, possibly internal, possibly without a domain name (and, therefore, probably a non-matching certificate subject name). They should need to explicitly opt in to disabling the checks that allow that to work. Obviously, which means compatibility is broken with existing code. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io Yes in that case compat will be broken and they'll need to either specify a cert that can be used to validate the connection or disable the protection. I think it's very surprising for people that they need to *enable* secure mode when most tools have that on by default. It's handing users a security foot gun, and like most things security related broken is silent until it's too late. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On 3 June 2013 22:46, Donald Stufft don...@stufft.io wrote: Also, we should consider the issue for application users. Suppose I'm using a Python application that downloads something from the web. I upgrade to 3.4, and the app stops working because of a will cease to work case. As an end user, how can I get the app working again? Having to patch the sources isn't an option, and reverting to 3.3 provokes the reaction Python broke my app. Supply a SSL vert using the environment variable? Hmm, that would be acceptable, I guess, for many users (although Windows users are somewhat more environment-variable-averse than Unix users). But you say that as if it's obvious how to do that (or where to get a cert). It's certainly not obvious to me, and if it works in Internet Explorer, I'd have no idea where to get a cert from that I could use in an environment variable. Just to repeat - I agree with the principle, but in many environments, users are pretty much clueless about security and actively object to being educated for their own safety. These users will disable all security quite happily if it stops the internal app failing, and will blame Python for making things harder and breaking backward compatibility. On the other hand, I suspect we're talking about an extremely low percentage of cases, so let's not blow the issue out of proportion :-) Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 6:01 PM, Paul Moore p.f.mo...@gmail.com wrote: On 3 June 2013 22:46, Donald Stufft don...@stufft.io wrote: Also, we should consider the issue for application users. Suppose I'm using a Python application that downloads something from the web. I upgrade to 3.4, and the app stops working because of a will cease to work case. As an end user, how can I get the app working again? Having to patch the sources isn't an option, and reverting to 3.3 provokes the reaction Python broke my app. Supply a SSL vert using the environment variable? Hmm, that would be acceptable, I guess, for many users (although Windows users are somewhat more environment-variable-averse than Unix users). But you say that as if it's obvious how to do that (or where to get a cert). It's certainly not obvious to me, and if it works in Internet Explorer, I'd have no idea where to get a cert from that I could use in an environment variable. Just to repeat - I agree with the principle, but in many environments, users are pretty much clueless about security and actively object to being educated for their own safety. These users will disable all security quite happily if it stops the internal app failing, and will blame Python for making things harder and breaking backward compatibility. On the other hand, I suspect we're talking about an extremely low percentage of cases, so let's not blow the issue out of proportion :-) Paul Let me make myself a bit more clear because maybe I haven't been. If someone is relying on the insecure by default behavior and Python 3.4 gets this change, their code will break and their options will be: Client side: - Disable Verification, essentially reverting back to the old behavior. - Setup their Python install to be able to validate that certificate - This may be importing it into their OS certificate store - This may be using an env var to specify - This may be passing it directly somewhere Server side: - Make the certificate valid - Purchase a Valid by default certificate - Serve immedaries etc. Generally any of these will be completely valid options, even disabling the checks. The idea behind my proposal is that people generally only use TLS for a reason and that reason is they want to protect against the kinds of attacks that TLS protects against. You really only get the bulk of those protections if you validate the certificates. So for the vast bulk of people validation is the option they want. It also happens that validation on by default is secure by default which aims a fairly large foot gun away from peoples feet. To expand on something else, I'm completely willing to do whatever I can to hep make this happen. I don't really know what the process is. If I need to write a PEP I'll do that. If I need to volunteer to manage the certificates, I'll do that. If we need docs or tooling to help the people who this change might break I'll do that too. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x does). Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 2, 2013 10:22 PM, Donald Stufft don...@stufft.io wrote: As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. Relevant: http://bugs.python.org/issue13647 So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x does). Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend solely on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. There's an existing request for this: http://bugs.python.org/issue13655 Cheers, Chris ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
