Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 12.05.2015 05:03, Nick Coghlan wrote: On 12 May 2015 at 04:49, M.-A. Lemburg m...@egenix.com wrote: On 11.05.2015 12:15, Nick Coghlan wrote: By contrast, the configuration file shouldn't provide a new attack vector (or simplify any existing attack vector), as if you have the permissions needed to modify the config file, you likely also have the permissions needed to modify the system certificate store, and the latter is a *far* more interesting attack vector than a downgrade attack solely against Python. Thus the environment variable based off switch is neither necessary (as an administrator controlled configuration file can do the job), nor sufficient (it can't handle the -E switch), *and* it represents an increase in the attack surface area relative to a Python implementation without the capability. Whether or not -E will have an effect on the env var depends on the implementation. At the moment, -E only has an effect on the C runtime, while the stdlib happily reads from os.environ without taking the flag into account. I had an off-list discussion with Christian Heimes about that in relation to the OpenSSL flags, and he pointed out the reason -E specifically needs to be a command line switch is because that is the only way to affect how environment variables are processed during interpreter startup. Once an application is up and running, further environment variable sanitisation can be handled at an application level by whitelisting entries in os.environ and deleting everything else. As proposed, the PYTHONHTTPSVERIFY would only affect the ssl module and only be checked when loading this module, i.e. not at Python startup time. Right, the same is true for the configuration file proposal. It would be nice to hear from ActiveState, Enthought Continuum Analytics as well, but if they don't chime in here, I don't see any particular need to go chasing them explicitly. I think the approach to only consider a subset of redistributors as viable targets for such a switch is a bit too narrow. You are leaving out all the parties which use custom Python installations to run their applications, e.g. the Plone and Zope community, the ZenOSS community, the many Windows applications built on Python, etc. No, they already have a solution: monkeypatch (or just plain patch) the SSL module. That's an upstream supported technique, which is why it's documented in the PEP. Again, this is not a proper solution for your friendly sys admin. They won't go in and patch Python, but rather consider it broken, if it doesn't provide a configuration option and simply stay with 2.7.8. We've had that discussion already, so I won't go into details again. The problem we (as in Red Hat) ran into was that that technique *doesn't work* for the case of backporting PEP 476 to Python 2.7.5 as an opt-in feature. And neither does it work for people deploying Windows apps built on Python (where the code is usually hidden away in a ZIP archive or even compiled into a DLL), or people deploying Plone or ZenOSS. It's not only Red Hat's customers that are affected :-) to fix this regression in 2.7.9. We made the decision when PEP 476 was accepted that this change turned a silent security failure into a noisy one, rather than being a regression in its own right. PEP 493 isn't about disagreeing with that decision, it's about providing a smoother upgrade path in contexts where letting the security failure remain silent is deemed to be preferred in the near term. The change wasn't regression. The missing downgrade path is a regression. It's a shame we don't have -X options in Python 2, as that would be a nice hard-to-attack option (although it wouldn't play well with subprocesses) Some other comments on PEP 493: * I don't think we really want to add the overhead of having to parse an INI file every time Python starts up. Please remember that we just parsing of the sysconfig data not long ago because we wanted to avoid this startup time. Compared to the overhead of reading from the system cert database, reading a config file at ssl module import time should be trivial. The cert database is only read when importing the ssl module, not with each Python startup, so that makes a big difference. The INI file would have to parsed at ssl module load time as well to work around this. * I don't see why the attack surface of using an INI file somewhere in the system should be smaller than e.g. using sitecustomize.py You can put sitecustomize.py in a user directory, and if there's no system wide sitecustomize, Python will read it automatically (unless user site directories are turned off). In a user based installation (which most applications shipping their own Python installation are), you can always do this provided you can gain the application user permissions. For a system installed Python, you'd always have to run Python with -E -S -s to avoid
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
- Original Message -
From: Donald Stufft don...@stufft.io
To: Nick Coghlan ncogh...@gmail.com
Cc: python-dev python-dev@python.org, M.-A. Lemburg m...@egenix.com
Sent: Monday, May 11, 2015 1:16:58 PM
Subject: Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On May 11, 2015, at 6:47 AM, Nick Coghlan ncogh...@gmail.com wrote:
On 11 May 2015 at 20:23, Donald Stufft don...@stufft.io wrote:
On May 11, 2015, at 6:15 AM, Nick Coghlan ncogh...@gmail.com wrote:
We made the decision when PEP 476 was accepted that this change turned
a silent security failure into a noisy one, rather than being a
regression in its own right. PEP 493 isn't about disagreeing with that
decision, it's about providing a smoother upgrade path in contexts
where letting the security failure remain silent is deemed to be
preferred in the near term.
I don't really agree that the decision to disable TLS is an environment
one,
it's really a per application decision. This is why I was against having
some
sort of global off switch for all of Python because just because one
application needs it turned off doesn't mean you want it turned off for
another
Python application.
The scenario I'm interested in is the one where it *was* off globally
(i.e. you were already running Python 2.7.8 or earlier) and you want
to manage a global rollout of a new Python version that supports being
configured to verify HTTPS certificates by default, while making the
decision on whether or not to enable HTTPS certificate verification on
a server-by-server basis, rather than having that decision be coupled
directly to the rollout of the updated version of Python.
I agree that the desired end state is where Python 3 is, and where
upstream Python 2.7.9+ is, this is solely about how to facilitate
folks getting from point A to point B without an intervening window of
I broke the world and now my boss is yelling at me about it :)
Oh, another issue that I forgot to mention--
A fair number of people had no idea that Python wasn't validating TLS before
2.7.9/3.4.3 however as part of the processing of changing that in 2.7.9 a lot
of people became aware that Python's before 2.7.9 didn't validate but that
Python 2.7.9+ does. I worry that if Redhat (or anyone) ships a Python 2.7.9
that doesn't verify by default then they are going to be shipping something
which defies the expectations of those users who were relying on the fact
that
Python 2.7.9+ was supposed to be secure by default now. You're
(understandibly)
focusing on I already have my thing running on Python 2.7.8 and I want to
yum update and get 2.7.9 and have things not visibly break, however there is
the other use case of I'm setting up a new environment, and I installed RHEL
and got 2.7.9, I remembered reading in LWN that 2.7.9 verifies now so I must
be safe. If you *do* provide such a switch, defaulting it to verify and
having
We (Red Hat) will not update python to 2.7.9, we ship 2.7.5 and backport
bugfixes/features based on users demand.
people where that breaks go in and turn it off is probably a safer mechanism
since the cases where 2.7.9 verification breaks things for people is a
visible
change where the case that someone expects 2.7.9 to verify and it doesn't
isn't
a visible change and is easily missed unless they go out of their way to try
and test it against a server with an invalid certificate.
Either way, if there is some sort of global off switch, having that off
switch
set to off should raise some kind of warning (like urllib3 does if you use
the unverified HTTPS methods). To be clear, I don't mean that using the built
in ssl module APIs to disable verification should raise a warning, I mean the
hypothetical make my Python insecurely access HTTPS configuration file (or
environment variable) that is being proposed.
---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/rkuska%40redhat.com
Regards
Robert Kuska
{rkuska}
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 05/12/2015 04:19 AM, Nick Coghlan wrote: It occurs to me that the subtitle of PEP 493 could be All software is terrible, but it's often a system administrator's job to make it run anyway :) +1 QoTW -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
If you control the app you don't need to do that. All relevant api accept the context parameter. The shims are only useful when you don't control the app. So an app shipping their own python doesn't fall under that. On May 12, 2015, at 6:56 AM, M.-A. Lemburg m...@egenix.com wrote: On 12.05.2015 12:04, Donald Stufft wrote: On May 12, 2015, at 3:57 AM, M.-A. Lemburg m...@egenix.com wrote: In a user based installation (which most applications shipping their own Python installation are), you can always do this provided you can gain the application user permissions. Of course, if the application is shipping it’s own Python then it has to actually do something to update to 2.7.9 and it can add it’s own option to disable TLS verification. I personally think that the application providing that option is the *right* way and all these other things are, at best, just temporary shims until the applications do that. I still believe that requiring to monkeypatch Python is a very poor approach in terms of quality software design. We can do better and we should. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On May 12, 2015, at 7:17 AM, Nick Coghlan ncogh...@gmail.com wrote: On 12 May 2015 at 21:09, Donald Stufft don...@stufft.io wrote: If you control the app you don't need to do that. All relevant api accept the context parameter. The shims are only useful when you don't control the app. So an app shipping their own python doesn't fall under that. I think the bundled Python scenario MAL is interested in is this one: 1. An application with a bundled CPython runtime is using the verification defaults 2. Upgraded the bundled Python to 2.7.9 3. Didn't provide new configuration settings to disable certificate verification 4. Is being upgraded in an environment where verifying certificates makes the app unusable for environmental reasons related to certificate management The PyRun single-file Python interpreter has a similar need, where some apps than ran fine under 2.7.8 will need a way to disable cert verification in 2.7.9+ on a per-application basis, *without* modifying the applications. Both of those make sense to me as cases where the environment variable based security downgrade approach is the least bad answer available, which is why I eventually agreed it should be one of the recommendations in the PEP. Why is without modifying the app a reasonable goal? If Python is bundled with the app then you have direct control over when that upgrade happens, so you can delay the upgrade to 2.7.9 until your application which is bundling Python has the relevant switches. This is distinctly different from a situation like downstream distributors where the version of Python being provided is being provided by a group different than the person providing the application. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 12.05.2015 13:21, Donald Stufft wrote: On May 12, 2015, at 7:17 AM, Nick Coghlan ncogh...@gmail.com wrote: On 12 May 2015 at 21:09, Donald Stufft don...@stufft.io wrote: If you control the app you don't need to do that. All relevant api accept the context parameter. The shims are only useful when you don't control the app. So an app shipping their own python doesn't fall under that. I think the bundled Python scenario MAL is interested in is this one: 1. An application with a bundled CPython runtime is using the verification defaults 2. Upgraded the bundled Python to 2.7.9 3. Didn't provide new configuration settings to disable certificate verification 4. Is being upgraded in an environment where verifying certificates makes the app unusable for environmental reasons related to certificate management The PyRun single-file Python interpreter has a similar need, where some apps than ran fine under 2.7.8 will need a way to disable cert verification in 2.7.9+ on a per-application basis, *without* modifying the applications. Both of those make sense to me as cases where the environment variable based security downgrade approach is the least bad answer available, which is why I eventually agreed it should be one of the recommendations in the PEP. Why is without modifying the app a reasonable goal? If Python is bundled with the app then you have direct control over when that upgrade happens, so you can delay the upgrade to 2.7.9 until your application which is bundling Python has the relevant switches. This is distinctly different from a situation like downstream distributors where the version of Python being provided is being provided by a group different than the person providing the application. Take a Plone Intranet as example: The unified installer downloads and installs Python 2.7 for you. As of Plone 4.3.3 the version is Python 2.7.6. Now say you are a sys admin and your Intranet users are affected by some bug in 2.7.6 which is fixed in 2.7.9. The natural approach would be to upgrade the bundled Python to 2.7.9. Because it's an Intranet and Plone is used to aggregate information from other systems which use self-signed certificates, you don't want to risk breaking your Plone installation and need a way to disable the cert checks. The best way to do this is by configuring the bundled Python to disable the checks, since you would not want to mess with the Plone application itself. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 12 May 2015 at 17:57, M.-A. Lemburg m...@egenix.com wrote: The point here is that sys admins should not have to patch Python to make things work again, in case an application is not prepared for the certificate verification - which is rather likely, since the pre-Python 2.7.9 doesn't even provide the necessary APIs to pass certificate locations to urllib or urllib2. You've persuaded me that we should describe *both* configuration mechanisms in the recommendations PEP (one for a cross-platform environment variable based approach that also works for embedded Python instances and user level distributions, one for a configuration file based approach that only covers *nix system Python installations), and leave the decision to redistributors as to which approach makes the most sense for their particular audience. Neither goes into upstream CPython 2.7, and neither goes into any version of Python 3. Yes, getting an administrative application to the point where -S can be used means getting it to a point where it has *no* Python dependencies outside the standard library. It can certainly be done, but often won't be worth the hassle. As a result, using -s to turn off the user site directory and -E to turn off PYTHONPATH processing are the more common sys.path related hardening techniques in Python 2.7. Overall, I find the config file approach a too big a hammer to solve this simple problem. The problem of managing this security issue on behalf of customers without risking breaking their world is in no way simple - hence why most redistributors lobbed it into the too hard basket instead, and why we've been discussing possible UX improvements with upstream on-and-off for a couple of months now :) If there were more use cases for a Python config file, the added overhead could pay off, but then we should put more thought into optimizing the config file load time and probably end up using a Python module as config file (which provides these optimizations for free). This is why the proposal is for an SSL specific configuration file, loaded only when the SSL module is imported. In the end, we'd be introducing another sitecustomize.py, usercustomize.py and perhaps localcustomize.py - only with fixed locations rather than importing them via sys.path. It won't come to that, as Linux system package managers don't support any of these - that's what containers are for. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 12 May 2015 at 21:09, Donald Stufft don...@stufft.io wrote: If you control the app you don't need to do that. All relevant api accept the context parameter. The shims are only useful when you don't control the app. So an app shipping their own python doesn't fall under that. I think the bundled Python scenario MAL is interested in is this one: 1. An application with a bundled CPython runtime is using the verification defaults 2. Upgraded the bundled Python to 2.7.9 3. Didn't provide new configuration settings to disable certificate verification 4. Is being upgraded in an environment where verifying certificates makes the app unusable for environmental reasons related to certificate management The PyRun single-file Python interpreter has a similar need, where some apps than ran fine under 2.7.8 will need a way to disable cert verification in 2.7.9+ on a per-application basis, *without* modifying the applications. Both of those make sense to me as cases where the environment variable based security downgrade approach is the least bad answer available, which is why I eventually agreed it should be one of the recommendations in the PEP. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 12 May 2015 at 21:21, Donald Stufft don...@stufft.io wrote: On May 12, 2015, at 7:17 AM, Nick Coghlan ncogh...@gmail.com wrote: On 12 May 2015 at 21:09, Donald Stufft don...@stufft.io wrote: If you control the app you don't need to do that. All relevant api accept the context parameter. The shims are only useful when you don't control the app. So an app shipping their own python doesn't fall under that. I think the bundled Python scenario MAL is interested in is this one: 1. An application with a bundled CPython runtime is using the verification defaults 2. Upgraded the bundled Python to 2.7.9 3. Didn't provide new configuration settings to disable certificate verification 4. Is being upgraded in an environment where verifying certificates makes the app unusable for environmental reasons related to certificate management The PyRun single-file Python interpreter has a similar need, where some apps than ran fine under 2.7.8 will need a way to disable cert verification in 2.7.9+ on a per-application basis, *without* modifying the applications. Both of those make sense to me as cases where the environment variable based security downgrade approach is the least bad answer available, which is why I eventually agreed it should be one of the recommendations in the PEP. Why is without modifying the app a reasonable goal? If Python is bundled with the app then you have direct control over when that upgrade happens, so you can delay the upgrade to 2.7.9 until your application which is bundling Python has the relevant switches. This is distinctly different from a situation like downstream distributors where the version of Python being provided is being provided by a group different than the person providing the application. Because of the way redistribution works. MAL was right that I was thinking specifically in terms of the Linux distributor case, where we're the OS vendor, so we need a way to offer off by default, opt-in on a per-server basis. Once I got past that perspective, I was able to figure out where he was coming from as someone that offers explicit support for the redistribution for bundling use case. When apps bundle Python, what's usually happening is that they'll just bundle whatever version is used on the build server that does the bundling. If the app developer's testing all uses valid HTTPS certificates (or simply doesn't test HTTPS at all), they won't see any problems with the 2.7.9 upgrade, and hence will ship that along to their customers, where it may break if that customer's environment turns out to be relying on the lack of certificate verification in 2.7.8 and earlier. If that scenario happens with unmodified upstream 2.7.9, the redistributor has no workaround they can pass along to app developers to in turn pass on to customers - the app developer simply has to tell their customers to downgrade back to the previous release, and then issue an updated version with a configuration setting to disable HTTPS verification as fast as they can. Customers tend to get rather grouchy about things like that :) By contrast, if the redistributor for the bundled version of Python has injected PYTHONHTTPSVERIFY support, then the app developers can at least pass along set PYTHONHTTPSVERIFY=0 in the environment to their customers as an interim workaround until they get a release out the door with a proper configuration setting to control whether or not the app verifies certificates (assuming they don't decide the environment variable is a good enough workaround). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 12.05.2015 12:04, Donald Stufft wrote: On May 12, 2015, at 3:57 AM, M.-A. Lemburg m...@egenix.com wrote: In a user based installation (which most applications shipping their own Python installation are), you can always do this provided you can gain the application user permissions. Of course, if the application is shipping it’s own Python then it has to actually do something to update to 2.7.9 and it can add it’s own option to disable TLS verification. I personally think that the application providing that option is the *right* way and all these other things are, at best, just temporary shims until the applications do that. I still believe that requiring to monkeypatch Python is a very poor approach in terms of quality software design. We can do better and we should. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 12 May 2015 at 21:17, Nick Coghlan ncogh...@gmail.com wrote: Both of those make sense to me as cases where the environment variable based security downgrade approach is the least bad answer available, which is why I eventually agreed it should be one of the recommendations in the PEP. It occurs to me that the subtitle of PEP 493 could be All software is terrible, but it's often a system administrator's job to make it run anyway :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 12 May 2015 at 21:45, Donald Stufft don...@stufft.io wrote: On May 12, 2015, at 7:40 AM, Nick Coghlan ncogh...@gmail.com wrote: On 12 May 2015 at 21:21, Donald Stufft don...@stufft.io wrote: On May 12, 2015, at 7:17 AM, Nick Coghlan ncogh...@gmail.com wrote: On 12 May 2015 at 21:09, Donald Stufft don...@stufft.io wrote: If you control the app you don't need to do that. All relevant api accept the context parameter. The shims are only useful when you don't control the app. So an app shipping their own python doesn't fall under that. I think the bundled Python scenario MAL is interested in is this one: 1. An application with a bundled CPython runtime is using the verification defaults 2. Upgraded the bundled Python to 2.7.9 3. Didn't provide new configuration settings to disable certificate verification 4. Is being upgraded in an environment where verifying certificates makes the app unusable for environmental reasons related to certificate management The PyRun single-file Python interpreter has a similar need, where some apps than ran fine under 2.7.8 will need a way to disable cert verification in 2.7.9+ on a per-application basis, *without* modifying the applications. Both of those make sense to me as cases where the environment variable based security downgrade approach is the least bad answer available, which is why I eventually agreed it should be one of the recommendations in the PEP. Why is without modifying the app a reasonable goal? If Python is bundled with the app then you have direct control over when that upgrade happens, so you can delay the upgrade to 2.7.9 until your application which is bundling Python has the relevant switches. This is distinctly different from a situation like downstream distributors where the version of Python being provided is being provided by a group different than the person providing the application. Because of the way redistribution works. MAL was right that I was thinking specifically in terms of the Linux distributor case, where we're the OS vendor, so we need a way to offer off by default, opt-in on a per-server basis. Once I got past that perspective, I was able to figure out where he was coming from as someone that offers explicit support for the redistribution for bundling use case. When apps bundle Python, what's usually happening is that they'll just bundle whatever version is used on the build server that does the bundling. If the app developer's testing all uses valid HTTPS certificates (or simply doesn't test HTTPS at all), they won't see any problems with the 2.7.9 upgrade, and hence will ship that along to their customers, where it may break if that customer's environment turns out to be relying on the lack of certificate verification in 2.7.8 and earlier. If that scenario happens with unmodified upstream 2.7.9, the redistributor has no workaround they can pass along to app developers to in turn pass on to customers - the app developer simply has to tell their customers to downgrade back to the previous release, and then issue an updated version with a configuration setting to disable HTTPS verification as fast as they can. Customers tend to get rather grouchy about things like that :) By contrast, if the redistributor for the bundled version of Python has injected PYTHONHTTPSVERIFY support, then the app developers can at least pass along set PYTHONHTTPSVERIFY=0 in the environment to their customers as an interim workaround until they get a release out the door with a proper configuration setting to control whether or not the app verifies certificates (assuming they don't decide the environment variable is a good enough workaround). Honestly, this reads like If the person bundling 2.7.9 with their app doesn't bother to pay attention to what 2.7.9 means then things might break, but that's hardly special to TLS, there are lots of things that change in a release that may end up breaking in certain cases. Cert verification is a special case, as cert management in intranets tends to rely on two things: * browser users just click through SSL security warnings * automated internal tools don't check certs at all Organisations with a specific interest in network security may have their systems in a happier state, but I wouldn't bet on them being a substantial minority of organisations, let alone a signficant majority. It's hard to overstate how big a mindset shift getting intranet network security right isn't optional represents in our industry, and it's going to take years for that attitude change to filter out through the later parts of the technology adoption curve. Looking at Marc-Andre's latest email though, it appears we're using bundling in a different context? I'm thinking of things like PyInstaller or such where you're distributing Python + your own App, but this appears to just be some third party tool is installing Python and an App? I'm personally
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On May 12, 2015, at 7:40 AM, Nick Coghlan ncogh...@gmail.com wrote: On 12 May 2015 at 21:21, Donald Stufft don...@stufft.io wrote: On May 12, 2015, at 7:17 AM, Nick Coghlan ncogh...@gmail.com wrote: On 12 May 2015 at 21:09, Donald Stufft don...@stufft.io wrote: If you control the app you don't need to do that. All relevant api accept the context parameter. The shims are only useful when you don't control the app. So an app shipping their own python doesn't fall under that. I think the bundled Python scenario MAL is interested in is this one: 1. An application with a bundled CPython runtime is using the verification defaults 2. Upgraded the bundled Python to 2.7.9 3. Didn't provide new configuration settings to disable certificate verification 4. Is being upgraded in an environment where verifying certificates makes the app unusable for environmental reasons related to certificate management The PyRun single-file Python interpreter has a similar need, where some apps than ran fine under 2.7.8 will need a way to disable cert verification in 2.7.9+ on a per-application basis, *without* modifying the applications. Both of those make sense to me as cases where the environment variable based security downgrade approach is the least bad answer available, which is why I eventually agreed it should be one of the recommendations in the PEP. Why is without modifying the app a reasonable goal? If Python is bundled with the app then you have direct control over when that upgrade happens, so you can delay the upgrade to 2.7.9 until your application which is bundling Python has the relevant switches. This is distinctly different from a situation like downstream distributors where the version of Python being provided is being provided by a group different than the person providing the application. Because of the way redistribution works. MAL was right that I was thinking specifically in terms of the Linux distributor case, where we're the OS vendor, so we need a way to offer off by default, opt-in on a per-server basis. Once I got past that perspective, I was able to figure out where he was coming from as someone that offers explicit support for the redistribution for bundling use case. When apps bundle Python, what's usually happening is that they'll just bundle whatever version is used on the build server that does the bundling. If the app developer's testing all uses valid HTTPS certificates (or simply doesn't test HTTPS at all), they won't see any problems with the 2.7.9 upgrade, and hence will ship that along to their customers, where it may break if that customer's environment turns out to be relying on the lack of certificate verification in 2.7.8 and earlier. If that scenario happens with unmodified upstream 2.7.9, the redistributor has no workaround they can pass along to app developers to in turn pass on to customers - the app developer simply has to tell their customers to downgrade back to the previous release, and then issue an updated version with a configuration setting to disable HTTPS verification as fast as they can. Customers tend to get rather grouchy about things like that :) By contrast, if the redistributor for the bundled version of Python has injected PYTHONHTTPSVERIFY support, then the app developers can at least pass along set PYTHONHTTPSVERIFY=0 in the environment to their customers as an interim workaround until they get a release out the door with a proper configuration setting to control whether or not the app verifies certificates (assuming they don't decide the environment variable is a good enough workaround). Honestly, this reads like If the person bundling 2.7.9 with their app doesn't bother to pay attention to what 2.7.9 means then things might break, but that's hardly special to TLS, there are lots of things that change in a release that may end up breaking in certain cases. Looking at Marc-Andre's latest email though, it appears we're using bundling in a different context? I'm thinking of things like PyInstaller or such where you're distributing Python + your own App, but this appears to just be some third party tool is installing Python and an App? Ultimately, as long as it doesn't end up in upstream CPython the PEP can recommend any approach and I'm OK with it in the sense that it won't affect me. Though the PEP should be clear it's for 2.7 only and not 3.x or 2.8 if that ever gets reversed and we end up with a 2.8. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 12 May 2015 at 20:56, M.-A. Lemburg m...@egenix.com wrote: On 12.05.2015 12:04, Donald Stufft wrote: On May 12, 2015, at 3:57 AM, M.-A. Lemburg m...@egenix.com wrote: In a user based installation (which most applications shipping their own Python installation are), you can always do this provided you can gain the application user permissions. Of course, if the application is shipping it’s own Python then it has to actually do something to update to 2.7.9 and it can add it’s own option to disable TLS verification. I personally think that the application providing that option is the *right* way and all these other things are, at best, just temporary shims until the applications do that. I still believe that requiring to monkeypatch Python is a very poor approach in terms of quality software design. We can do better and we should. It's a deliberate design choice to actively discourage people from doing it - your Ewww reaction to monkeypatching is exactly the one we want. There's no technical reason for people to be bothered by it, since it's a documented and supported technique covered by the relevant PEP - it just so happens that the configuration being done is to switch a function alias between two different functions. Both of the recommended options I'm putting in the PEP (essentially the Red Hat design and the eGenix design, since we cover two different use cases) still adopt that same basic implementation model, they just provide ways for redistributors to move the configuration inside the SSL module itself if they decide it is in their users' interests for them to do so. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 12.05.2015 13:19, Nick Coghlan wrote: On 12 May 2015 at 21:17, Nick Coghlan ncogh...@gmail.com wrote: Both of those make sense to me as cases where the environment variable based security downgrade approach is the least bad answer available, which is why I eventually agreed it should be one of the recommendations in the PEP. It occurs to me that the subtitle of PEP 493 could be All software is terrible, but it's often a system administrator's job to make it run anyway :) +1 :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On May 12, 2015, at 3:57 AM, M.-A. Lemburg m...@egenix.com wrote: In a user based installation (which most applications shipping their own Python installation are), you can always do this provided you can gain the application user permissions. Of course, if the application is shipping it’s own Python then it has to actually do something to update to 2.7.9 and it can add it’s own option to disable TLS verification. I personally think that the application providing that option is the *right* way and all these other things are, at best, just temporary shims until the applications do that. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 11 May 2015 at 18:04, M.-A. Lemburg m...@egenix.com wrote: On 10.05.2015 05:04, Robert Collins wrote: On 10 May 2015 at 11:44, Chris Angelico ros...@gmail.com wrote: On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg m...@egenix.com wrote: By providing a way to intentionally switch off the new default, we do make people aware of the risks and that's good enough, while still maintaining the contract people rightly expect of patch level releases of Python. Just as long as it's the sysadmin, and NOT some random attacker over the internet, who has the power to downgrade security. Environment variables can be attacked in various ways. They can, and the bash fun was very good evidence of that. OTOH if someones environment is at risk, PATH and PYTHONPATH are already very effective attack vectors. If an attacker has access to the process environment, you're doomed anyway, so that's not really an argument for or against using environment variables :-) The core issue lies in managing the user vs administrator permissions split. Even for self-administered systems, it's recommended practice to run *without* administrative permissions normally, and only elevate when you need them. One of the things you're watching out for in such cases is that it shouldn't be possible for an attacker to make a change to the user environment, and have that propagate to have an effect on a process running with administrative access. One of the recommended hardening measures against that kind of attack vector is to *turn off* Python's environment variable processing when launching Python processes with administrative access. We didn't care about that in the hash randomisation case, as the compatibility concern there applied on a per application basis, and caring about hash order was technically a bug in its own right. By contrast, in the situation we're worried about for certificate verification compatibility, the issue is environmental: certificate management in many private intranets isn't yet to the same standard as that on the public internet, so administrators may have a valid reason for defaulting Python back to the old behaviour, and redistributors may feel obliged to provide an opt-in period prior to switching the default behaviour to opt-out. Having the new setting be ignored in Python processes run under a hardened configuration means that an environment variable based solution won't have the desired effect in providing that smoother migration path to the more hardened configuration. I've now written a draft recommendations to redistributors PEP for Robert's configuration file based design: https://www.python.org/dev/peps/pep-0493/ (exact file names config setting names TBD) I wouldn't be opposed to seeing that as an upstream Python 2.7.x feature, but agreement amongst redistributors on using a file-based approach is the main outcome I'm looking for. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On May 11, 2015, at 6:15 AM, Nick Coghlan ncogh...@gmail.com wrote: On 11 May 2015 at 19:22, M.-A. Lemburg m...@egenix.com wrote: On 11.05.2015 11:13, Nick Coghlan wrote: I wouldn't be opposed to seeing that as an upstream Python 2.7.x feature, but agreement amongst redistributors on using a file-based approach is the main outcome I'm looking for. Can't we have both ? I'd strongly advise against it, as we're deliberately increasing the attack surface area here by providing a potential path to carry out a downgrade attack on the HTTPS certificate verification by forcing it back to the old behaviour. The main existing environment variable based attack vector would be to manage to run a process with administrative privileges and SSL_CERT_DIR and/or SSL_CERT_FILE pointing to a certificate written to a user or world-writable directory by the attacker. Providing a don't verify HTTPS flag at the interpreter level would let an attacker skip the first step of writing the certificate file to disk somewhere, making a system compromise harder to detect. (An especially paranoid SSL implementation would disallow reading certs from locations that allow write access to non-administrative users, but I don't believe OpenSSL is that paranoid) By contrast, the configuration file shouldn't provide a new attack vector (or simplify any existing attack vector), as if you have the permissions needed to modify the config file, you likely also have the permissions needed to modify the system certificate store, and the latter is a *far* more interesting attack vector than a downgrade attack solely against Python. Thus the environment variable based off switch is neither necessary (as an administrator controlled configuration file can do the job), nor sufficient (it can't handle the -E switch), *and* it represents an increase in the attack surface area relative to a Python implementation without the capability. I don't think that we can wait for a whole PEP process to run through Matrix multiplication made it through the PEP process in 8 days. If we do this as a redistributor recommendation rather than attempting to get it into upstream Python 2.7, we could potentially propose you take on the role of BDFL-Delegate and mark it as Accepted as soon as the two of us agree on a common approach. The reason I think that's a reasonable way forward is because we already know there are folks opposed to making the change upstream. If the PEP just provides recommendations for redistributors that *do* decide to provide a global off switch to revert to the old behaviour, then the perspective of the folks opposed to the feature is respected by the fact that this is a feature some redistributors *may* choose to add to provide a smoother migration path to more secure default HTTPS handling, rather than something upstream provides by default. I assume the Debian, Ubuntu and Suse folks won't care, as they have all already decided against backporting the change to their long term support releases where the compatibility break would pose a problem (and I can certainly sympathise with that perspective given the dependency on backporting the PEP 466 SSL changes first, and the work involved in seeking consensus on a smoother migration path from the old default to the new one). It would be nice to hear from ActiveState, Enthought Continuum Analytics as well, but if they don't chime in here, I don't see any particular need to go chasing them explicitly. to fix this regression in 2.7.9. We made the decision when PEP 476 was accepted that this change turned a silent security failure into a noisy one, rather than being a regression in its own right. PEP 493 isn't about disagreeing with that decision, it's about providing a smoother upgrade path in contexts where letting the security failure remain silent is deemed to be preferred in the near term. I don't really agree that the decision to disable TLS is an environment one, it's really a per application decision. This is why I was against having some sort of global off switch for all of Python because just because one application needs it turned off doesn't mean you want it turned off for another Python application. You might have some script that is interacting with a custom internal server which doesn’t have a valid TLS certificate but then you also have pip* installed which is reaching out to PyPI and downloading code from the internet. You might want to disable TLS verification for the first but you almost certainly don't want it to disable TLS verification for the second one. In this regard I think that environment variables are somewhat better because they are far easier to set per application instead of in a way that affects every python program. Per application is the *right* scope for this setting, especially in a system where people may or may not realize what is written in Python and what isn't. I think
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 11 May 2015 at 20:23, Donald Stufft don...@stufft.io wrote: On May 11, 2015, at 6:15 AM, Nick Coghlan ncogh...@gmail.com wrote: We made the decision when PEP 476 was accepted that this change turned a silent security failure into a noisy one, rather than being a regression in its own right. PEP 493 isn't about disagreeing with that decision, it's about providing a smoother upgrade path in contexts where letting the security failure remain silent is deemed to be preferred in the near term. I don't really agree that the decision to disable TLS is an environment one, it's really a per application decision. This is why I was against having some sort of global off switch for all of Python because just because one application needs it turned off doesn't mean you want it turned off for another Python application. The scenario I'm interested in is the one where it *was* off globally (i.e. you were already running Python 2.7.8 or earlier) and you want to manage a global rollout of a new Python version that supports being configured to verify HTTPS certificates by default, while making the decision on whether or not to enable HTTPS certificate verification on a server-by-server basis, rather than having that decision be coupled directly to the rollout of the updated version of Python. I agree that the desired end state is where Python 3 is, and where upstream Python 2.7.9+ is, this is solely about how to facilitate folks getting from point A to point B without an intervening window of I broke the world and now my boss is yelling at me about it :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
I'm in entire agreement with Donald below. Regards Antoine. On Mon, 11 May 2015 06:23:11 -0400 Donald Stufft don...@stufft.io wrote: I don't really agree that the decision to disable TLS is an environment one, it's really a per application decision. This is why I was against having some sort of global off switch for all of Python because just because one application needs it turned off doesn't mean you want it turned off for another Python application. You might have some script that is interacting with a custom internal server which doesn’t have a valid TLS certificate but then you also have pip* installed which is reaching out to PyPI and downloading code from the internet. You might want to disable TLS verification for the first but you almost certainly don't want it to disable TLS verification for the second one. In this regard I think that environment variables are somewhat better because they are far easier to set per application instead of in a way that affects every python program. Per application is the *right* scope for this setting, especially in a system where people may or may not realize what is written in Python and what isn't. I think it's absolutely wrong to give people a footgun in the terms of a switch that turns off all of Python's TLS verification when for many applications the fact they use Python is simply an implementation detail. That being said, since it's not being included in Python core and it's only some patch that some downstream's are going to apply I also don't really care that much because it's not going to effect me and if it turns out to be a bad idea and a footgun like I think it is, then the blame can rest on those downstreams and not us :) I'm also not a fan of the environment variable either really for a lot of the reasons you've outlined here. * Ignoring the fact that pip has (via requests/urllib3) worked around this deficiency in Python and isn't going to be affected by this configuration switch at all. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 11.05.2015 12:47, Nick Coghlan wrote: On 11 May 2015 at 20:23, Donald Stufft don...@stufft.io wrote: On May 11, 2015, at 6:15 AM, Nick Coghlan ncogh...@gmail.com wrote: We made the decision when PEP 476 was accepted that this change turned a silent security failure into a noisy one, rather than being a regression in its own right. PEP 493 isn't about disagreeing with that decision, it's about providing a smoother upgrade path in contexts where letting the security failure remain silent is deemed to be preferred in the near term. I don't really agree that the decision to disable TLS is an environment one, it's really a per application decision. This is why I was against having some sort of global off switch for all of Python because just because one application needs it turned off doesn't mean you want it turned off for another Python application. The scenario I'm interested in is the one where it *was* off globally (i.e. you were already running Python 2.7.8 or earlier) and you want to manage a global rollout of a new Python version that supports being configured to verify HTTPS certificates by default, while making the decision on whether or not to enable HTTPS certificate verification on a server-by-server basis, rather than having that decision be coupled directly to the rollout of the updated version of Python. I guess we're approaching this from different perspectives :-) I'm mostly interested in having a switch that can be set on a per application basis, not globally. I think the global default is fine and I'm just looking for a way to have admins disable it on a case-by-case basis for those applications which have problems with the new default. Hence the env var approach - the admin would simply edit the application's startup shell script, add the env var and that's it. For pip et al. which don't use the ssl module, the admins can simply continue using older versions for those applications - ones which don't implement the extra verification. In many cases, this is not necessary, since production environments typically don't use PyPI at all: they use a local directory with the needed distribution files, which is both more secure and reliable. I agree that the desired end state is where Python 3 is, and where upstream Python 2.7.9+ is, this is solely about how to facilitate folks getting from point A to point B without an intervening window of I broke the world and now my boss is yelling at me about it :) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 11 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 10.05.2015 05:04, Robert Collins wrote: On 10 May 2015 at 11:44, Chris Angelico ros...@gmail.com wrote: On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg m...@egenix.com wrote: By providing a way to intentionally switch off the new default, we do make people aware of the risks and that's good enough, while still maintaining the contract people rightly expect of patch level releases of Python. Just as long as it's the sysadmin, and NOT some random attacker over the internet, who has the power to downgrade security. Environment variables can be attacked in various ways. They can, and the bash fun was very good evidence of that. OTOH if someones environment is at risk, PATH and PYTHONPATH are already very effective attack vectors. If an attacker has access to the process environment, you're doomed anyway, so that's not really an argument for or against using environment variables :-) You'd just need to create a file os.py and point PYTHONPATH at it. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 11 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 11.05.2015 11:13, Nick Coghlan wrote: On 11 May 2015 at 18:04, M.-A. Lemburg m...@egenix.com wrote: On 10.05.2015 05:04, Robert Collins wrote: On 10 May 2015 at 11:44, Chris Angelico ros...@gmail.com wrote: On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg m...@egenix.com wrote: By providing a way to intentionally switch off the new default, we do make people aware of the risks and that's good enough, while still maintaining the contract people rightly expect of patch level releases of Python. Just as long as it's the sysadmin, and NOT some random attacker over the internet, who has the power to downgrade security. Environment variables can be attacked in various ways. They can, and the bash fun was very good evidence of that. OTOH if someones environment is at risk, PATH and PYTHONPATH are already very effective attack vectors. If an attacker has access to the process environment, you're doomed anyway, so that's not really an argument for or against using environment variables :-) The core issue lies in managing the user vs administrator permissions split. Even for self-administered systems, it's recommended practice to run *without* administrative permissions normally, and only elevate when you need them. One of the things you're watching out for in such cases is that it shouldn't be possible for an attacker to make a change to the user environment, and have that propagate to have an effect on a process running with administrative access. One of the recommended hardening measures against that kind of attack vector is to *turn off* Python's environment variable processing when launching Python processes with administrative access. The env var would not be read at Python startup time, only when loading the ssl module, so the -E switch would not have the effect of disabling it - unlike the hash seed logic, which is run (and has to be run) at Python startup time. We didn't care about that in the hash randomisation case, as the compatibility concern there applied on a per application basis, and caring about hash order was technically a bug in its own right. By contrast, in the situation we're worried about for certificate verification compatibility, the issue is environmental: certificate management in many private intranets isn't yet to the same standard as that on the public internet, so administrators may have a valid reason for defaulting Python back to the old behaviour, and redistributors may feel obliged to provide an opt-in period prior to switching the default behaviour to opt-out. Having the new setting be ignored in Python processes run under a hardened configuration means that an environment variable based solution won't have the desired effect in providing that smoother migration path to the more hardened configuration. I've now written a draft recommendations to redistributors PEP for Robert's configuration file based design: https://www.python.org/dev/peps/pep-0493/ (exact file names config setting names TBD) The Fastly cache seems to be having problems again. I only get: 503 Backend is unhealthy - Details: cache-fra1225-FRA 1431335851 2631441948 I wouldn't be opposed to seeing that as an upstream Python 2.7.x feature, but agreement amongst redistributors on using a file-based approach is the main outcome I'm looking for. Can't we have both ? I don't think that we can wait for a whole PEP process to run through to fix this regression in 2.7.9. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 11 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 11 May 2015 at 19:22, M.-A. Lemburg m...@egenix.com wrote: On 11.05.2015 11:13, Nick Coghlan wrote: I wouldn't be opposed to seeing that as an upstream Python 2.7.x feature, but agreement amongst redistributors on using a file-based approach is the main outcome I'm looking for. Can't we have both ? I'd strongly advise against it, as we're deliberately increasing the attack surface area here by providing a potential path to carry out a downgrade attack on the HTTPS certificate verification by forcing it back to the old behaviour. The main existing environment variable based attack vector would be to manage to run a process with administrative privileges and SSL_CERT_DIR and/or SSL_CERT_FILE pointing to a certificate written to a user or world-writable directory by the attacker. Providing a don't verify HTTPS flag at the interpreter level would let an attacker skip the first step of writing the certificate file to disk somewhere, making a system compromise harder to detect. (An especially paranoid SSL implementation would disallow reading certs from locations that allow write access to non-administrative users, but I don't believe OpenSSL is that paranoid) By contrast, the configuration file shouldn't provide a new attack vector (or simplify any existing attack vector), as if you have the permissions needed to modify the config file, you likely also have the permissions needed to modify the system certificate store, and the latter is a *far* more interesting attack vector than a downgrade attack solely against Python. Thus the environment variable based off switch is neither necessary (as an administrator controlled configuration file can do the job), nor sufficient (it can't handle the -E switch), *and* it represents an increase in the attack surface area relative to a Python implementation without the capability. I don't think that we can wait for a whole PEP process to run through Matrix multiplication made it through the PEP process in 8 days. If we do this as a redistributor recommendation rather than attempting to get it into upstream Python 2.7, we could potentially propose you take on the role of BDFL-Delegate and mark it as Accepted as soon as the two of us agree on a common approach. The reason I think that's a reasonable way forward is because we already know there are folks opposed to making the change upstream. If the PEP just provides recommendations for redistributors that *do* decide to provide a global off switch to revert to the old behaviour, then the perspective of the folks opposed to the feature is respected by the fact that this is a feature some redistributors *may* choose to add to provide a smoother migration path to more secure default HTTPS handling, rather than something upstream provides by default. I assume the Debian, Ubuntu and Suse folks won't care, as they have all already decided against backporting the change to their long term support releases where the compatibility break would pose a problem (and I can certainly sympathise with that perspective given the dependency on backporting the PEP 466 SSL changes first, and the work involved in seeking consensus on a smoother migration path from the old default to the new one). It would be nice to hear from ActiveState, Enthought Continuum Analytics as well, but if they don't chime in here, I don't see any particular need to go chasing them explicitly. to fix this regression in 2.7.9. We made the decision when PEP 476 was accepted that this change turned a silent security failure into a noisy one, rather than being a regression in its own right. PEP 493 isn't about disagreeing with that decision, it's about providing a smoother upgrade path in contexts where letting the security failure remain silent is deemed to be preferred in the near term. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On May 11, 2015, at 6:47 AM, Nick Coghlan ncogh...@gmail.com wrote: On 11 May 2015 at 20:23, Donald Stufft don...@stufft.io wrote: On May 11, 2015, at 6:15 AM, Nick Coghlan ncogh...@gmail.com wrote: We made the decision when PEP 476 was accepted that this change turned a silent security failure into a noisy one, rather than being a regression in its own right. PEP 493 isn't about disagreeing with that decision, it's about providing a smoother upgrade path in contexts where letting the security failure remain silent is deemed to be preferred in the near term. I don't really agree that the decision to disable TLS is an environment one, it's really a per application decision. This is why I was against having some sort of global off switch for all of Python because just because one application needs it turned off doesn't mean you want it turned off for another Python application. The scenario I'm interested in is the one where it *was* off globally (i.e. you were already running Python 2.7.8 or earlier) and you want to manage a global rollout of a new Python version that supports being configured to verify HTTPS certificates by default, while making the decision on whether or not to enable HTTPS certificate verification on a server-by-server basis, rather than having that decision be coupled directly to the rollout of the updated version of Python. I agree that the desired end state is where Python 3 is, and where upstream Python 2.7.9+ is, this is solely about how to facilitate folks getting from point A to point B without an intervening window of I broke the world and now my boss is yelling at me about it :) Oh, another issue that I forgot to mention-- A fair number of people had no idea that Python wasn't validating TLS before 2.7.9/3.4.3 however as part of the processing of changing that in 2.7.9 a lot of people became aware that Python's before 2.7.9 didn't validate but that Python 2.7.9+ does. I worry that if Redhat (or anyone) ships a Python 2.7.9 that doesn't verify by default then they are going to be shipping something which defies the expectations of those users who were relying on the fact that Python 2.7.9+ was supposed to be secure by default now. You're (understandibly) focusing on I already have my thing running on Python 2.7.8 and I want to yum update and get 2.7.9 and have things not visibly break, however there is the other use case of I'm setting up a new environment, and I installed RHEL and got 2.7.9, I remembered reading in LWN that 2.7.9 verifies now so I must be safe. If you *do* provide such a switch, defaulting it to verify and having people where that breaks go in and turn it off is probably a safer mechanism since the cases where 2.7.9 verification breaks things for people is a visible change where the case that someone expects 2.7.9 to verify and it doesn't isn't a visible change and is easily missed unless they go out of their way to try and test it against a server with an invalid certificate. Either way, if there is some sort of global off switch, having that off switch set to off should raise some kind of warning (like urllib3 does if you use the unverified HTTPS methods). To be clear, I don't mean that using the built in ssl module APIs to disable verification should raise a warning, I mean the hypothetical make my Python insecurely access HTTPS configuration file (or environment variable) that is being proposed. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 11 May 2015 10:16 pm, Robert Kuska rku...@redhat.com wrote: Oh, another issue that I forgot to mention-- A fair number of people had no idea that Python wasn't validating TLS before 2.7.9/3.4.3 however as part of the processing of changing that in 2.7.9 a lot of people became aware that Python's before 2.7.9 didn't validate but that Python 2.7.9+ does. I worry that if Redhat (or anyone) ships a Python 2.7.9 that doesn't verify by default then they are going to be shipping something which defies the expectations of those users who were relying on the fact that Python 2.7.9+ was supposed to be secure by default now. You're (understandibly) focusing on I already have my thing running on Python 2.7.8 and I want to yum update and get 2.7.9 and have things not visibly break, As Robert noted, it would be a matter of updating to a 2.7.5 with more patches backported, rather than rebasing to a newer upstream version. I can make the do not change the default behaviour relative to the corresponding upstream version guidance explicit in the PEP, though. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 11.05.2015 12:15, Nick Coghlan wrote: On 11 May 2015 at 19:22, M.-A. Lemburg m...@egenix.com wrote: On 11.05.2015 11:13, Nick Coghlan wrote: I wouldn't be opposed to seeing that as an upstream Python 2.7.x feature, but agreement amongst redistributors on using a file-based approach is the main outcome I'm looking for. Can't we have both ? I'd strongly advise against it, as we're deliberately increasing the attack surface area here by providing a potential path to carry out a downgrade attack on the HTTPS certificate verification by forcing it back to the old behaviour. The main existing environment variable based attack vector would be to manage to run a process with administrative privileges and SSL_CERT_DIR and/or SSL_CERT_FILE pointing to a certificate written to a user or world-writable directory by the attacker. Providing a don't verify HTTPS flag at the interpreter level would let an attacker skip the first step of writing the certificate file to disk somewhere, making a system compromise harder to detect. (An especially paranoid SSL implementation would disallow reading certs from locations that allow write access to non-administrative users, but I don't believe OpenSSL is that paranoid) Correct. OpenSSL will happily read the cert files from anywhere you point it to. By contrast, the configuration file shouldn't provide a new attack vector (or simplify any existing attack vector), as if you have the permissions needed to modify the config file, you likely also have the permissions needed to modify the system certificate store, and the latter is a *far* more interesting attack vector than a downgrade attack solely against Python. Thus the environment variable based off switch is neither necessary (as an administrator controlled configuration file can do the job), nor sufficient (it can't handle the -E switch), *and* it represents an increase in the attack surface area relative to a Python implementation without the capability. Whether or not -E will have an effect on the env var depends on the implementation. At the moment, -E only has an effect on the C runtime, while the stdlib happily reads from os.environ without taking the flag into account. As proposed, the PYTHONHTTPSVERIFY would only affect the ssl module and only be checked when loading this module, i.e. not at Python startup time. I don't think that we can wait for a whole PEP process to run through Matrix multiplication made it through the PEP process in 8 days. If we do this as a redistributor recommendation rather than attempting to get it into upstream Python 2.7, we could potentially propose you take on the role of BDFL-Delegate and mark it as Accepted as soon as the two of us agree on a common approach. The reason I think that's a reasonable way forward is because we already know there are folks opposed to making the change upstream. If the PEP just provides recommendations for redistributors that *do* decide to provide a global off switch to revert to the old behaviour, then the perspective of the folks opposed to the feature is respected by the fact that this is a feature some redistributors *may* choose to add to provide a smoother migration path to more secure default HTTPS handling, rather than something upstream provides by default. I assume the Debian, Ubuntu and Suse folks won't care, as they have all already decided against backporting the change to their long term support releases where the compatibility break would pose a problem (and I can certainly sympathise with that perspective given the dependency on backporting the PEP 466 SSL changes first, and the work involved in seeking consensus on a smoother migration path from the old default to the new one). It would be nice to hear from ActiveState, Enthought Continuum Analytics as well, but if they don't chime in here, I don't see any particular need to go chasing them explicitly. I think the approach to only consider a subset of redistributors as viable targets for such a switch is a bit too narrow. You are leaving out all the parties which use custom Python installations to run their applications, e.g. the Plone and Zope community, the ZenOSS community, the many Windows applications built on Python, etc. to fix this regression in 2.7.9. We made the decision when PEP 476 was accepted that this change turned a silent security failure into a noisy one, rather than being a regression in its own right. PEP 493 isn't about disagreeing with that decision, it's about providing a smoother upgrade path in contexts where letting the security failure remain silent is deemed to be preferred in the near term. The change wasn't regression. The missing downgrade path is a regression. Some other comments on PEP 493: * I don't think we really want to add the overhead of having to parse an INI file every time Python starts up. Please remember that we just parsing of the sysconfig data not
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 12 May 2015 at 04:49, M.-A. Lemburg m...@egenix.com wrote: On 11.05.2015 12:15, Nick Coghlan wrote: By contrast, the configuration file shouldn't provide a new attack vector (or simplify any existing attack vector), as if you have the permissions needed to modify the config file, you likely also have the permissions needed to modify the system certificate store, and the latter is a *far* more interesting attack vector than a downgrade attack solely against Python. Thus the environment variable based off switch is neither necessary (as an administrator controlled configuration file can do the job), nor sufficient (it can't handle the -E switch), *and* it represents an increase in the attack surface area relative to a Python implementation without the capability. Whether or not -E will have an effect on the env var depends on the implementation. At the moment, -E only has an effect on the C runtime, while the stdlib happily reads from os.environ without taking the flag into account. I had an off-list discussion with Christian Heimes about that in relation to the OpenSSL flags, and he pointed out the reason -E specifically needs to be a command line switch is because that is the only way to affect how environment variables are processed during interpreter startup. Once an application is up and running, further environment variable sanitisation can be handled at an application level by whitelisting entries in os.environ and deleting everything else. As proposed, the PYTHONHTTPSVERIFY would only affect the ssl module and only be checked when loading this module, i.e. not at Python startup time. Right, the same is true for the configuration file proposal. It would be nice to hear from ActiveState, Enthought Continuum Analytics as well, but if they don't chime in here, I don't see any particular need to go chasing them explicitly. I think the approach to only consider a subset of redistributors as viable targets for such a switch is a bit too narrow. You are leaving out all the parties which use custom Python installations to run their applications, e.g. the Plone and Zope community, the ZenOSS community, the many Windows applications built on Python, etc. No, they already have a solution: monkeypatch (or just plain patch) the SSL module. That's an upstream supported technique, which is why it's documented in the PEP. The problem we (as in Red Hat) ran into was that that technique *doesn't work* for the case of backporting PEP 476 to Python 2.7.5 as an opt-in feature. to fix this regression in 2.7.9. We made the decision when PEP 476 was accepted that this change turned a silent security failure into a noisy one, rather than being a regression in its own right. PEP 493 isn't about disagreeing with that decision, it's about providing a smoother upgrade path in contexts where letting the security failure remain silent is deemed to be preferred in the near term. The change wasn't regression. The missing downgrade path is a regression. It's a shame we don't have -X options in Python 2, as that would be a nice hard-to-attack option (although it wouldn't play well with subprocesses) Some other comments on PEP 493: * I don't think we really want to add the overhead of having to parse an INI file every time Python starts up. Please remember that we just parsing of the sysconfig data not long ago because we wanted to avoid this startup time. Compared to the overhead of reading from the system cert database, reading a config file at ssl module import time should be trivial. * I don't see why the attack surface of using an INI file somewhere in the system should be smaller than e.g. using sitecustomize.py You can put sitecustomize.py in a user directory, and if there's no system wide sitecustomize, Python will read it automatically (unless user site directories are turned off). * If done right, we'd also need a switch to ignore this global config file and recommend using it to reduce the attack surface (for the same reason you explain in the PEP) No, the recommendation there would be to upgrade to a newer version of Python that doesn't offer this downgrade capability. It's a proposal for a transition smoothing technique, not a permanent capability (I did suggest the latter at one point, but the discussion on the issue tracker persuaded me that was a bad idea, with the increased attack surface being a key part of that change of heart). * I don't think a global switch is the right way forward. Many applications on properly configured systems will work fine with the new default. The downgrade option is only needed for those cases, where they don't and you don't have a good way to fix the application. And techniques like chroots and containers let you do that selectively. The key thing I'm after is an agreed technique for backporting to earlier 2.7.x releases that allows PEP 476 to be provided as an opt-in capability,
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg m...@egenix.com wrote: By providing a way to intentionally switch off the new default, we do make people aware of the risks and that's good enough, while still maintaining the contract people rightly expect of patch level releases of Python. Just as long as it's the sysadmin, and NOT some random attacker over the internet, who has the power to downgrade security. Environment variables can be attacked in various ways. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 10 May 2015 at 13:04, Robert Collins robe...@robertcollins.net wrote: On 10 May 2015 at 11:44, Chris Angelico ros...@gmail.com wrote: On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg m...@egenix.com wrote: By providing a way to intentionally switch off the new default, we do make people aware of the risks and that's good enough, while still maintaining the contract people rightly expect of patch level releases of Python. Just as long as it's the sysadmin, and NOT some random attacker over the internet, who has the power to downgrade security. Environment variables can be attacked in various ways. They can, and the bash fun was very good evidence of that. OTOH if someones environment is at risk, PATH and PYTHONPATH are already very effective attack vectors. Right, which is why -E is an important existing hardening technique for protecting privileged services against local attackers. Isolated mode in Python 3.4+ is easier to use, but you can get a functional equivalent in Python 2 using: * running from a directory under /usr (Program Files on Windows) rather than your home directory (to protect against sys.path[0] based attacks) * running with -E (to protect against PYTHON* environment variable attacks) * running with -S (to protect against site.py and sitecustomize.py based attacks) * running with -s (to protect against hostile packages in the user site directory) That's how I came to the conclusion that adding a new environment variable to turn off a network security hardening feature isn't a good idea: * it significantly increases the attack surface area if you're *not* using -E when running a privileged service * it doesn't work at all if you *are* using -E when running a privileged service That was OK when we were dealing with the hash randomisation problem, mostly because the consequence of that vulnerability was denial of service, and the question of whether or not hash randomisation caused problems came up on an application-by-application basis, rather than being related to the way an entire network environment was managed. The question becomes very different when the failure mode we're talking about is transparent interception of nominally confidential communication. Instead, we want a configuration file stored in a protected directory, such that for an attacker to modify it they *already* need to have achieved a local privilege escalation, in which case, they can just attack the system certificate store directly, rather than messing about downgrading Python's default HTTPS verification settings. In my case, I don't actually need the *feature itself* in upstream CPython, but I *would* like to have upstream CPython's blessing of the design as a recommendation to redistributors that need a capability like this to meet the needs of their end users. I've been talking about someone putting together a PEP to that effect, so given this discussion, I'll go ahead and do that myself, with Robert Kuska listed as co-author (since he came up with the general design I'm advocating for). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 10 May 2015 at 11:44, Chris Angelico ros...@gmail.com wrote: On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg m...@egenix.com wrote: By providing a way to intentionally switch off the new default, we do make people aware of the risks and that's good enough, while still maintaining the contract people rightly expect of patch level releases of Python. Just as long as it's the sysadmin, and NOT some random attacker over the internet, who has the power to downgrade security. Environment variables can be attacked in various ways. They can, and the bash fun was very good evidence of that. OTOH if someones environment is at risk, PATH and PYTHONPATH are already very effective attack vectors. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 09.05.2015 02:29, Nick Coghlan wrote: On 8 May 2015 8:14 pm, M.-A. Lemburg m...@egenix.com wrote: On 08.05.2015 11:36, Nick Coghlan wrote: On 8 May 2015 6:52 pm, M.-A. Lemburg m...@egenix.com wrote: On 07.05.2015 04:30, Nick Coghlan wrote: Can we please make the monkeypatch a regular part of Python's site.py which can enabled via an environment variable, say export PYTHONHTTPSVERIFY=0. See http://bugs.python.org/issue23857 for the discussion. ... I actually do think it would be good to have such a feature as a native part of Python 2.7 in order to provide a nicer revert to the pre-PEP-476 behaviour experience for Python 2.7 users (leaving the there's no easy way to turn HTTPS certificate verification off globally state of affairs to Python 3), but I don't currently have the time available to push for that against the end users can't be trusted not to turn certificate verification off when they should be fixing their certificate management instead perspective. We're currently working on a new release of eGenix PyRun and this will include Python 2.7.9. We do want to add such an env switch to disable the cert verification, so would like to know whether we can use PYTHONHTTPSVERIFY for this or not. That's a slightly misleading quotation of my post, as I'm opposed to the use of an environment variable for this, due to the fact that using the -E switch will then revert to the upstream default behaviour of verifying certificates, rather defeating the point of introducing the legacy infrastructure compatibility feature in the first place. Oh, sorry. I read your email implying that you are fine with the env var approach. I don't really see the issue with -E, though. It's well possible to internally set PYTHONHTTPSVERIFY=0 when -E is used to regain backwards compatibility per default for Python 2.7. Regarding the config file approach and letting distributions set their own defaults: I don't think it's a good idea to have one distribution default to verifying HTTPS certs via a global config file and another distribution do the opposite. Python itself should define the defaults to be used, not the distributions. As a result of the discussions around both PEP 466 and 476, I'm now firmly of the view that it's correct for the upstream Python community to assume the public internet as its standard operating environment when it comes to network security settings, and for those of us working for commercial redistributors to subsequently bear the cost of adapting from that upstream assumption to the different assumptions that may apply in the context of organisational intranets. That's also why I've come around to the view that an informational PEP with recommendations for redistributors, rather than an actual change to Python 2.7, is the right answer (at least initially) when it comes to providing a smoother transition plan for PEP 476 - the folks saying it's a bad idea to make this easy to turn off are *right* from the perspective of operating over the public internet, or with well designed internal SSL/TLS certificate management, it's just also a *necessary* idea (in my view) to help CIOs and other infrastructure leads responsibly and effectively manage the wide range of risks associated with internal infrastructure upgrades. I don't agree. We've broken the contract that people had with Python 2.7 by introducing a major breakage in a patch level release very far down the line in 2.7.9, without providing an easy and official way to opt-out that does not involve hacking your installation. IMO, we should not fall for the somewhat arrogant view that we know better than all the Python users out there when it comes to running secure systems. By providing a way to intentionally switch off the new default, we do make people aware of the risks and that's good enough, while still maintaining the contract people rightly expect of patch level releases of Python. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 09 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var (was: Clarification of PEP 476 opting out section)
On 07.05.2015 04:30, Nick Coghlan wrote: Can we please make the monkeypatch a regular part of Python's site.py which can enabled via an environment variable, say export PYTHONHTTPSVERIFY=0. See http://bugs.python.org/issue23857 for the discussion. ... I actually do think it would be good to have such a feature as a native part of Python 2.7 in order to provide a nicer revert to the pre-PEP-476 behaviour experience for Python 2.7 users (leaving the there's no easy way to turn HTTPS certificate verification off globally state of affairs to Python 3), but I don't currently have the time available to push for that against the end users can't be trusted not to turn certificate verification off when they should be fixing their certificate management instead perspective. We're currently working on a new release of eGenix PyRun and this will include Python 2.7.9. We do want to add such an env switch to disable the cert verification, so would like to know whether we can use PYTHONHTTPSVERIFY for this or not. We mainly need this to reenable simple use of self-signed certificates which 2.7.9 disables. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 08.05.2015 11:36, Nick Coghlan wrote: On 8 May 2015 6:52 pm, M.-A. Lemburg m...@egenix.com wrote: On 07.05.2015 04:30, Nick Coghlan wrote: Can we please make the monkeypatch a regular part of Python's site.py which can enabled via an environment variable, say export PYTHONHTTPSVERIFY=0. See http://bugs.python.org/issue23857 for the discussion. ... I actually do think it would be good to have such a feature as a native part of Python 2.7 in order to provide a nicer revert to the pre-PEP-476 behaviour experience for Python 2.7 users (leaving the there's no easy way to turn HTTPS certificate verification off globally state of affairs to Python 3), but I don't currently have the time available to push for that against the end users can't be trusted not to turn certificate verification off when they should be fixing their certificate management instead perspective. We're currently working on a new release of eGenix PyRun and this will include Python 2.7.9. We do want to add such an env switch to disable the cert verification, so would like to know whether we can use PYTHONHTTPSVERIFY for this or not. That's a slightly misleading quotation of my post, as I'm opposed to the use of an environment variable for this, due to the fact that using the -E switch will then revert to the upstream default behaviour of verifying certificates, rather defeating the point of introducing the legacy infrastructure compatibility feature in the first place. Oh, sorry. I read your email implying that you are fine with the env var approach. I don't really see the issue with -E, though. It's well possible to internally set PYTHONHTTPSVERIFY=0 when -E is used to regain backwards compatibility per default for Python 2.7. Regarding the config file approach and letting distributions set their own defaults: I don't think it's a good idea to have one distribution default to verifying HTTPS certs via a global config file and another distribution do the opposite. Python itself should define the defaults to be used, not the distributions. The Python Linux distribution is too complex already due to the many different ways Python is installed on the systems. Not having to deal with this complexity was the main motivation for us to create eGenix PyRun, since it works the same on all Linux platforms and doesn't use any of the system wide installed Python interpreters, settings or packages (unless you tell it to). A new informational PEP akin to PEP 394 that defines a config file location contents for downstream redistributors that need a smoother transition plan for PEP 476 will let us handle this in a consistent way across redistributors that's also compatible with runtime use of the -E switch. Regardless of whether a global config file is a good idea or not, I don't think we can wait with 2.7.10 until a whole new PEP process has run through. Cheers, Nick. We mainly need this to reenable simple use of self-signed certificates which 2.7.9 disables. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var (was: Clarification of PEP 476 opting out section)
On 8 May 2015 6:52 pm, M.-A. Lemburg m...@egenix.com wrote: On 07.05.2015 04:30, Nick Coghlan wrote: Can we please make the monkeypatch a regular part of Python's site.py which can enabled via an environment variable, say export PYTHONHTTPSVERIFY=0. See http://bugs.python.org/issue23857 for the discussion. ... I actually do think it would be good to have such a feature as a native part of Python 2.7 in order to provide a nicer revert to the pre-PEP-476 behaviour experience for Python 2.7 users (leaving the there's no easy way to turn HTTPS certificate verification off globally state of affairs to Python 3), but I don't currently have the time available to push for that against the end users can't be trusted not to turn certificate verification off when they should be fixing their certificate management instead perspective. We're currently working on a new release of eGenix PyRun and this will include Python 2.7.9. We do want to add such an env switch to disable the cert verification, so would like to know whether we can use PYTHONHTTPSVERIFY for this or not. That's a slightly misleading quotation of my post, as I'm opposed to the use of an environment variable for this, due to the fact that using the -E switch will then revert to the upstream default behaviour of verifying certificates, rather defeating the point of introducing the legacy infrastructure compatibility feature in the first place. A new informational PEP akin to PEP 394 that defines a config file location contents for downstream redistributors that need a smoother transition plan for PEP 476 will let us handle this in a consistent way across redistributors that's also compatible with runtime use of the -E switch. Cheers, Nick. We mainly need this to reenable simple use of self-signed certificates which 2.7.9 disables. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
Using an environment variable to disable a security feature sounds like an extremely bad idea. Environment variables are hidden state. Generally you don't know up front which values they will have when running an executable, and people don't think about inspecting them. This opens the door to mistakes (or even attacks) where certificate validation is disabled without the user knowing. Regards Antoine. On Fri, 08 May 2015 12:13:52 +0200 M.-A. Lemburg m...@egenix.com wrote: On 08.05.2015 11:36, Nick Coghlan wrote: On 8 May 2015 6:52 pm, M.-A. Lemburg m...@egenix.com wrote: On 07.05.2015 04:30, Nick Coghlan wrote: Can we please make the monkeypatch a regular part of Python's site.py which can enabled via an environment variable, say export PYTHONHTTPSVERIFY=0. See http://bugs.python.org/issue23857 for the discussion. ... I actually do think it would be good to have such a feature as a native part of Python 2.7 in order to provide a nicer revert to the pre-PEP-476 behaviour experience for Python 2.7 users (leaving the there's no easy way to turn HTTPS certificate verification off globally state of affairs to Python 3), but I don't currently have the time available to push for that against the end users can't be trusted not to turn certificate verification off when they should be fixing their certificate management instead perspective. We're currently working on a new release of eGenix PyRun and this will include Python 2.7.9. We do want to add such an env switch to disable the cert verification, so would like to know whether we can use PYTHONHTTPSVERIFY for this or not. That's a slightly misleading quotation of my post, as I'm opposed to the use of an environment variable for this, due to the fact that using the -E switch will then revert to the upstream default behaviour of verifying certificates, rather defeating the point of introducing the legacy infrastructure compatibility feature in the first place. Oh, sorry. I read your email implying that you are fine with the env var approach. I don't really see the issue with -E, though. It's well possible to internally set PYTHONHTTPSVERIFY=0 when -E is used to regain backwards compatibility per default for Python 2.7. Regarding the config file approach and letting distributions set their own defaults: I don't think it's a good idea to have one distribution default to verifying HTTPS certs via a global config file and another distribution do the opposite. Python itself should define the defaults to be used, not the distributions. The Python Linux distribution is too complex already due to the many different ways Python is installed on the systems. Not having to deal with this complexity was the main motivation for us to create eGenix PyRun, since it works the same on all Linux platforms and doesn't use any of the system wide installed Python interpreters, settings or packages (unless you tell it to). A new informational PEP akin to PEP 394 that defines a config file location contents for downstream redistributors that need a smoother transition plan for PEP 476 will let us handle this in a consistent way across redistributors that's also compatible with runtime use of the -E switch. Regardless of whether a global config file is a good idea or not, I don't think we can wait with 2.7.10 until a whole new PEP process has run through. Cheers, Nick. We mainly need this to reenable simple use of self-signed certificates which 2.7.9 disables. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 8 May 2015 9:52 pm, Antoine Pitrou solip...@pitrou.net wrote: Using an environment variable to disable a security feature sounds like an extremely bad idea. Environment variables are hidden state. Generally you don't know up front which values they will have when running an executable, and people don't think about inspecting them. This opens the door to mistakes (or even attacks) where certificate validation is disabled without the user knowing. Yes, that's also a consideration. A config file lets us bring the full battery of Linux security tools (most obviously file system permissions, but also other systems like SELinux AppArmor) to bear on controlling who (and what) has permission to change the default security settings. Cheers, Nick. Regards Antoine. On Fri, 08 May 2015 12:13:52 +0200 M.-A. Lemburg m...@egenix.com wrote: On 08.05.2015 11:36, Nick Coghlan wrote: On 8 May 2015 6:52 pm, M.-A. Lemburg m...@egenix.com wrote: On 07.05.2015 04:30, Nick Coghlan wrote: Can we please make the monkeypatch a regular part of Python's site.py which can enabled via an environment variable, say export PYTHONHTTPSVERIFY=0. See http://bugs.python.org/issue23857 for the discussion. ... I actually do think it would be good to have such a feature as a native part of Python 2.7 in order to provide a nicer revert to the pre-PEP-476 behaviour experience for Python 2.7 users (leaving the there's no easy way to turn HTTPS certificate verification off globally state of affairs to Python 3), but I don't currently have the time available to push for that against the end users can't be trusted not to turn certificate verification off when they should be fixing their certificate management instead perspective. We're currently working on a new release of eGenix PyRun and this will include Python 2.7.9. We do want to add such an env switch to disable the cert verification, so would like to know whether we can use PYTHONHTTPSVERIFY for this or not. That's a slightly misleading quotation of my post, as I'm opposed to the use of an environment variable for this, due to the fact that using the -E switch will then revert to the upstream default behaviour of verifying certificates, rather defeating the point of introducing the legacy infrastructure compatibility feature in the first place. Oh, sorry. I read your email implying that you are fine with the env var approach. I don't really see the issue with -E, though. It's well possible to internally set PYTHONHTTPSVERIFY=0 when -E is used to regain backwards compatibility per default for Python 2.7. Regarding the config file approach and letting distributions set their own defaults: I don't think it's a good idea to have one distribution default to verifying HTTPS certs via a global config file and another distribution do the opposite. Python itself should define the defaults to be used, not the distributions. The Python Linux distribution is too complex already due to the many different ways Python is installed on the systems. Not having to deal with this complexity was the main motivation for us to create eGenix PyRun, since it works the same on all Linux platforms and doesn't use any of the system wide installed Python interpreters, settings or packages (unless you tell it to). A new informational PEP akin to PEP 394 that defines a config file location contents for downstream redistributors that need a smoother transition plan for PEP 476 will let us handle this in a consistent way across redistributors that's also compatible with runtime use of the -E switch. Regardless of whether a global config file is a good idea or not, I don't think we can wait with 2.7.10 until a whole new PEP process has run through. Cheers, Nick. We mainly need this to reenable simple use of self-signed certificates which 2.7.9 disables. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 8 May 2015 8:14 pm, M.-A. Lemburg m...@egenix.com wrote: On 08.05.2015 11:36, Nick Coghlan wrote: On 8 May 2015 6:52 pm, M.-A. Lemburg m...@egenix.com wrote: On 07.05.2015 04:30, Nick Coghlan wrote: Can we please make the monkeypatch a regular part of Python's site.py which can enabled via an environment variable, say export PYTHONHTTPSVERIFY=0. See http://bugs.python.org/issue23857 for the discussion. ... I actually do think it would be good to have such a feature as a native part of Python 2.7 in order to provide a nicer revert to the pre-PEP-476 behaviour experience for Python 2.7 users (leaving the there's no easy way to turn HTTPS certificate verification off globally state of affairs to Python 3), but I don't currently have the time available to push for that against the end users can't be trusted not to turn certificate verification off when they should be fixing their certificate management instead perspective. We're currently working on a new release of eGenix PyRun and this will include Python 2.7.9. We do want to add such an env switch to disable the cert verification, so would like to know whether we can use PYTHONHTTPSVERIFY for this or not. That's a slightly misleading quotation of my post, as I'm opposed to the use of an environment variable for this, due to the fact that using the -E switch will then revert to the upstream default behaviour of verifying certificates, rather defeating the point of introducing the legacy infrastructure compatibility feature in the first place. Oh, sorry. I read your email implying that you are fine with the env var approach. I don't really see the issue with -E, though. It's well possible to internally set PYTHONHTTPSVERIFY=0 when -E is used to regain backwards compatibility per default for Python 2.7. Regarding the config file approach and letting distributions set their own defaults: I don't think it's a good idea to have one distribution default to verifying HTTPS certs via a global config file and another distribution do the opposite. Python itself should define the defaults to be used, not the distributions. As a result of the discussions around both PEP 466 and 476, I'm now firmly of the view that it's correct for the upstream Python community to assume the public internet as its standard operating environment when it comes to network security settings, and for those of us working for commercial redistributors to subsequently bear the cost of adapting from that upstream assumption to the different assumptions that may apply in the context of organisational intranets. That's also why I've come around to the view that an informational PEP with recommendations for redistributors, rather than an actual change to Python 2.7, is the right answer (at least initially) when it comes to providing a smoother transition plan for PEP 476 - the folks saying it's a bad idea to make this easy to turn off are *right* from the perspective of operating over the public internet, or with well designed internal SSL/TLS certificate management, it's just also a *necessary* idea (in my view) to help CIOs and other infrastructure leads responsibly and effectively manage the wide range of risks associated with internal infrastructure upgrades. Regards, Nick. The Python Linux distribution is too complex already due to the many different ways Python is installed on the systems. Not having to deal with this complexity was the main motivation for us to create eGenix PyRun, since it works the same on all Linux platforms and doesn't use any of the system wide installed Python interpreters, settings or packages (unless you tell it to). A new informational PEP akin to PEP 394 that defines a config file location contents for downstream redistributors that need a smoother transition plan for PEP 476 will let us handle this in a consistent way across redistributors that's also compatible with runtime use of the -E switch. Regardless of whether a global config file is a good idea or not, I don't think we can wait with 2.7.10 until a whole new PEP process has run through. Cheers, Nick. We mainly need this to reenable simple use of self-signed certificates which 2.7.9 disables. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
