Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 09/21/2014 11:30 PM, Ian Cordasco wrote: Hi Thomas, Several people, many of whom are core contributors to other projects, have asked that this discussion not be continued in this venue. Discussion of the decisions of the core-developers of requests are not appropriate for this list. All three of us have email addresses that you can retrieve from anywhere you please. There’s a mailing list for request, albeit very lightly trafficked, and there’s twitter. Cheers, Ian Ian, I don't use non-free software as a service platform such as twitter, even less for discussing about software development. I anyway tried twitter and didn't like it. I first didn't reply to any discussion about vendoring, but then I saw *a lot* of this discussion happening in this list. Because of that, I thought I couldn't left it unanswered. Now that I did answer to so many points of your argumentation, you're telling me to go away from this list and do it somewhere else. In some cases, its looking like you're just closing discussions and tell everyone to go away from your own channel of communication. It shows here: https://github.com/kennethreitz/requests/pull/1812 Will the discussion stay open if I'm joining your list? Will you guys be open minded with someone with a different view? If so, I may try to make a new attempt. Please do open a topic on your list with my last reply as a start, and just CC me (I don't really want to register to yet-another-new-mailing-list...). Further, I’m disappointed that you felt it appropriate or necessary to result to personal attacks on this list. At the very least you could have contained those to Twitter like others in this thread have done. I expected a more civil response on the openstack-dev mailing list. I have re-read myself multiple times to make sure that there was no such thing as personal attacks. I tried to use a nice tone, and have a solid argumentation. It's looking like I failed. :( If there's some words which you consider as a personal attack, please feel free to quote them in a private mail, and let me know (away from this list) where I was not nice enough, so that I get to know which part you didn't like. Though remember something: it's very common to read someone from a list, and believe that there's an aggressive, when in fact, the intention is only to be convincing. Please assume good faith. Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
Hi Ian and Donald, I've read the full thread, and couldn't help to reply to it still, even though I previously thought I shouldn't, as what I care is OpenStack, not really requests, and more largely, the topic of the wrong reasons why upstream are embedding foreign library code copies. I completely agree with someone else who wrote that this thread is nearly uninteresting for OpenStack itself. However, it is IMO my role, as a package maintainer, to let you know about my view on your argumentation. If you ignore my argumentation, then at least I'll have tried! :) On 09/18/2014 03:42 AM, Ian Cordasco wrote: Circling back to the issue of vendoring though: it’s a conscious decision to do this, and in the last two years there have been 2 CVEs reported for requests. There have been none for urllib3 and none for chardet. (Frankly I don’t think either urllib3 or chardet have had any CVEs reported against them, but let’s ignore that for now.) While security is typically the chief concern with vendoring, none of the libraries we use have had security issues rendering it a moot point in my opinion. The benefits of vendoring for us as a team have been numerous and we will likely continue to do it until it stops benefiting us and our users. Could you please list the benefits *for end users*? I'm really saying users, as in, not developers. Because I don't see any benefit at all for the end users. I don't think any of them would like to see many version of the same thing on their system. Also, the issue is not only security. Let me give you an example. Simply do this in a Debian sid machine: apt-file search six.py | grep -v python3 | grep -v pyshared | wc -l We have in Debian, about 50 versions of six.py around, embedded in packages. And this doesn't even counts those where only bits of six are just embedded in a file which isn't called six.py. Of course, we (in Debian) would like them to be removed. Why? Because it's a useless complexity, with so many different version, some with embedded bugs which have been fixed upstream, and the like. That's not even about security at this point (I hardly would see how six.py would have security issue). There is also a waste of server resources (install time, size of packages in the Debian archive, increased download time, RAM footprint of everything, etc.). We don't need to install (and compile as .pyc at install time) 50 versions of six.py, a single one is enough. We're trying to address this as much as we can, and you'll see lots of packages were we did, but it's not always easy for various reasons, like upstream code not up-to-date with latest version, or lack of time from the Debian package maintainer. Also, consider the fact that six is small: a quite small single file. It's still unacceptable from a Unix distribution stand point, but this makes the vendorizing less of an absurdity. Now, for urllib3, it's a WAY bigger. There's about 25 Python files. So multiply the resulting waste and issues... This was a simple example for six. Now just generalize to all. There's numerous upstream authors who also think that it's ok, and they can be one of the few exception. But really, every upstream who does this think that he's special. That's not the case. Requests isn't more special than any other Python module. On 09/18/2014 04:31 AM, Ian Cordasco wrote: Isn’t the whole point of distributing a library to let people use it as they see fit? The point of a library, is that it is shared among multiple consumers. Oh, maybe not if you're using Windows, but that's maybe out of the scope of this debate. Maintaining a coherent distribution with a single version of every library, is what distributions do as much as possible. It is unfortunately not always possible, but we do it as much as we can. On 09/18/2014 04:31 AM, Ian Cordasco wrote: Project X pins a version of requests. Alice doesn’t know anything about requests and does pip install X. Until Alice takes a more active role in the development of Project X and looks into requests, she will never know she’s installed software that has exposures in it. In all likelihood, any person who just uses something that pins requests will never check for it. If they just use pip and Project X never updates, it’s not our responsibility for anything that happens to the user. This is exactly why we should, at all costs, avoid using pinning. This is very dangerous, and leads to all sorts of issues. We should make sure that we stay current with absolutely all libraries, and when possible, support both the old and the new version of some incompatible API when possible. On 09/18/2014 04:31 AM, Ian Cordasco wrote: I think more applications bundle it than you realize. You’re likely using one daily that does it. It's not because stupidity is wide spread that it becomes smartness. (nothing personal, just making a general statement...) On 09/18/2014 04:31 AM, Ian Cordasco wrote: The reality is that by vendoring its
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 09/18/2014 05:22 AM, Dean Troyer wrote: On Wed, Sep 17, 2014 at 3:53 PM, Robert Collins robe...@robertcollins.net mailto:robe...@robertcollins.net wrote: On 18 September 2014 08:01, Dean Troyer dtro...@gmail.com mailto:dtro...@gmail.com wrote: Interestingly enough, the distros are doing exactly what they don't want us to do, ie, rebuilding things to use 'their' tested version of dependencies rather than the included one... We don't use our tested version, we use upstream's, and a single version of it. Indeed - but the distros are solving for two specific issues: No argument, just observing the recursive nature of this... Also, if we pin to a version, is the downstream consequence different? IIRC Thomas has had to do this with Django (1.7?) and Horizon, probably with others too. Correct. And there's still some open issues with it, though mostly it has been dealt with. There was also SQLAlchemy 0.8 then 0.9 a year ago as well. Though since Mike Bayer works on OpenStack support now, I'm sure I wont have to deal with any SQLA issue again. It's a common mistake to think that we just need to pin. Pinning (the upper bound) doesn't solve any issue, apart from having the tests pass the gate. This is sometimes urgent to fix the gate, so I understand why this is done. The reality is that packages in a distribution share common dependency, and OpenStack isn't alone in the distro. Lucky, (almost?) everyone in the OpenStack community understand this, and so far, I've received *a lot* of help from everyone. You have no idea how much this is important to me. Kudos to everyone who do help or who is at least gives moral support! :) As a provider of an app package directly to users, dealing with the front-line consequences of changing dependencies falls on me. And its one reason with this hat on I want static linking, or a Python equivalent of it (bundling/vendoring) at the app level. Since a few days, the Debian policy manual explicitly forbids static linking. I fully agree with the decision to make it explicit. Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
Hi Thomas, Several people, many of whom are core contributors to other projects, have asked that this discussion not be continued in this venue. Discussion of the decisions of the core-developers of requests are not appropriate for this list. All three of us have email addresses that you can retrieve from anywhere you please. There’s a mailing list for request, albeit very lightly trafficked, and there’s twitter. Further, I’m disappointed that you felt it appropriate or necessary to result to personal attacks on this list. At the very least you could have contained those to Twitter like others in this thread have done. I expected a more civil response on the openstack-dev mailing list. Cheers, Ian On 9/21/14, 7:21 AM, Thomas Goirand z...@debian.org wrote: Hi Ian and Donald, I've read the full thread, and couldn't help to reply to it still, even though I previously thought I shouldn't, as what I care is OpenStack, not really requests, and more largely, the topic of the wrong reasons why upstream are embedding foreign library code copies. I completely agree with someone else who wrote that this thread is nearly uninteresting for OpenStack itself. However, it is IMO my role, as a package maintainer, to let you know about my view on your argumentation. If you ignore my argumentation, then at least I'll have tried! :) On 09/18/2014 03:42 AM, Ian Cordasco wrote: Circling back to the issue of vendoring though: it’s a conscious decision to do this, and in the last two years there have been 2 CVEs reported for requests. There have been none for urllib3 and none for chardet. (Frankly I don’t think either urllib3 or chardet have had any CVEs reported against them, but let’s ignore that for now.) While security is typically the chief concern with vendoring, none of the libraries we use have had security issues rendering it a moot point in my opinion. The benefits of vendoring for us as a team have been numerous and we will likely continue to do it until it stops benefiting us and our users. Could you please list the benefits *for end users*? I'm really saying users, as in, not developers. Because I don't see any benefit at all for the end users. I don't think any of them would like to see many version of the same thing on their system. Also, the issue is not only security. Let me give you an example. Simply do this in a Debian sid machine: apt-file search six.py | grep -v python3 | grep -v pyshared | wc -l We have in Debian, about 50 versions of six.py around, embedded in packages. And this doesn't even counts those where only bits of six are just embedded in a file which isn't called six.py. Of course, we (in Debian) would like them to be removed. Why? Because it's a useless complexity, with so many different version, some with embedded bugs which have been fixed upstream, and the like. That's not even about security at this point (I hardly would see how six.py would have security issue). There is also a waste of server resources (install time, size of packages in the Debian archive, increased download time, RAM footprint of everything, etc.). We don't need to install (and compile as .pyc at install time) 50 versions of six.py, a single one is enough. We're trying to address this as much as we can, and you'll see lots of packages were we did, but it's not always easy for various reasons, like upstream code not up-to-date with latest version, or lack of time from the Debian package maintainer. Also, consider the fact that six is small: a quite small single file. It's still unacceptable from a Unix distribution stand point, but this makes the vendorizing less of an absurdity. Now, for urllib3, it's a WAY bigger. There's about 25 Python files. So multiply the resulting waste and issues... This was a simple example for six. Now just generalize to all. There's numerous upstream authors who also think that it's ok, and they can be one of the few exception. But really, every upstream who does this think that he's special. That's not the case. Requests isn't more special than any other Python module. On 09/18/2014 04:31 AM, Ian Cordasco wrote: Isn’t the whole point of distributing a library to let people use it as they see fit? The point of a library, is that it is shared among multiple consumers. Oh, maybe not if you're using Windows, but that's maybe out of the scope of this debate. Maintaining a coherent distribution with a single version of every library, is what distributions do as much as possible. It is unfortunately not always possible, but we do it as much as we can. On 09/18/2014 04:31 AM, Ian Cordasco wrote: Project X pins a version of requests. Alice doesn’t know anything about requests and does pip install X. Until Alice takes a more active role in the development of Project X and looks into requests, she will never know she’s installed software that has exposures in it. In all likelihood, any person who just uses something that pins requests will never check for it. If they
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 09/21/2014 11:30 AM, Ian Cordasco wrote: Hi Thomas, Several people, many of whom are core contributors to other projects, have asked that this discussion not be continued in this venue. Discussion of the decisions of the core-developers of requests are not appropriate for this list. All three of us have email addresses that you can retrieve from anywhere you please. There’s a mailing list for request, albeit very lightly trafficked, and there’s twitter. Further, I’m disappointed that you felt it appropriate or necessary to result to personal attacks on this list. At the very least you could have contained those to Twitter like others in this thread have done. I expected a more civil response on the openstack-dev mailing list. For those of us interested in this conversation, I ask that whatever decisions (if any) that come out of those discussions, that somebody please do post to the openstack-dev ML a summary of those decisions or actions. Thanks much in advance, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 2014-09-18 14:35:10 + (+), Ian Cordasco wrote: Except that even OpenStack doesn’t pin requests because of how extraordinarily stable our API is. [...] FWIW, I nearly capped it a few weeks ago with https://review.openstack.org/117848 but since the affected projects were able to rush in changes to their use of the library to work around the ways it was breaking I ended up abandoning that. Also for some months we capped requests in our global requirements because of https://launchpad.net/bugs/1182271 but that was finally lifted about a year ago with https://review.openstack.org/37461 (so I don't think it's entirely fair to assert that OpenStack doesn’t pin requests because...extraordinarily stable). -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 9/19/14, 9:01 AM, Jeremy Stanley fu...@yuggoth.org wrote: On 2014-09-18 14:35:10 + (+), Ian Cordasco wrote: Except that even OpenStack doesn’t pin requests because of how extraordinarily stable our API is. [...] FWIW, I nearly capped it a few weeks ago with https://review.openstack.org/117848 but since the affected projects were able to rush in changes to their use of the library to work around the ways it was breaking I ended up abandoning that. Also for some months we capped requests in our global requirements because of https://launchpad.net/bugs/1182271 but that was finally lifted about a year ago with https://review.openstack.org/37461 (so I don't think it's entirely fair to assert that OpenStack doesn’t pin requests because...extraordinarily stable). -- Jeremy Stanley A) Not the thread for this discussion. B) I didn’t say that OpenStack *never* pinned it, I said it didn’t currently C) Did you read the whole thread? I mentioned 2.4.0 as an exception because of ProtocolErrors and the redirect_cache members of OpenStack lobbied for. D) Before someone else replies, I assumed the transition from 0.x - 1.0 was also the other obvious (and not worth mentioning) break in stability given that since then we’ve had no API changes (with the exception of 2.4.0 not re-wrapping a single urllib3 exception). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
I don't think anyone would be complaining if glanceclient didn't have the need to reach into and monkeypatch requests's connection pool manager[1]. Is there a way to tell requests to build the https connections differently without monkeypatching urllib3.poolmanager? glanceclient's monkeypatching of the global variable here is dangerous since it will mess with the application and every other library if the application or another library uses glanceclient. [1] http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/common/https.py#n75 - Brant On Fri, Sep 19, 2014 at 10:33 AM, Ian Cordasco ian.corda...@rackspace.com wrote: On 9/19/14, 9:01 AM, Jeremy Stanley fu...@yuggoth.org wrote: On 2014-09-18 14:35:10 + (+), Ian Cordasco wrote: Except that even OpenStack doesn’t pin requests because of how extraordinarily stable our API is. [...] FWIW, I nearly capped it a few weeks ago with https://review.openstack.org/117848 but since the affected projects were able to rush in changes to their use of the library to work around the ways it was breaking I ended up abandoning that. Also for some months we capped requests in our global requirements because of https://launchpad.net/bugs/1182271 but that was finally lifted about a year ago with https://review.openstack.org/37461 (so I don't think it's entirely fair to assert that OpenStack doesn’t pin requests because...extraordinarily stable). -- Jeremy Stanley A) Not the thread for this discussion. B) I didn’t say that OpenStack *never* pinned it, I said it didn’t currently C) Did you read the whole thread? I mentioned 2.4.0 as an exception because of ProtocolErrors and the redirect_cache members of OpenStack lobbied for. D) Before someone else replies, I assumed the transition from 0.x - 1.0 was also the other obvious (and not worth mentioning) break in stability given that since then we’ve had no API changes (with the exception of 2.4.0 not re-wrapping a single urllib3 exception). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Sep 19, 2014, at 11:54 AM, Brant Knudson b...@acm.org wrote: I don't think anyone would be complaining if glanceclient didn't have the need to reach into and monkeypatch requests's connection pool manager[1]. Is there a way to tell requests to build the https connections differently without monkeypatching urllib3.poolmanager? glanceclient's monkeypatching of the global variable here is dangerous since it will mess with the application and every other library if the application or another library uses glanceclient. [1] http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/common/https.py#n75 http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/common/https.py#n75 Why does it need to use it’s own VerifiedHTTPSConnection class? Ironically reimplementing that is probably more dangerous for security than requests bundling urllib3 ;) --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Fri, Sep 19, 2014 at 8:59 AM, Donald Stufft don...@stufft.io wrote: On Sep 19, 2014, at 11:54 AM, Brant Knudson b...@acm.org wrote: I don't think anyone would be complaining if glanceclient didn't have the need to reach into and monkeypatch requests's connection pool manager[1]. Is there a way to tell requests to build the https connections differently without monkeypatching urllib3.poolmanager? glanceclient's monkeypatching of the global variable here is dangerous since it will mess with the application and every other library if the application or another library uses glanceclient. [1] http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/common/https.py#n75 Why does it need to use it’s own VerifiedHTTPSConnection class? Ironically reimplementing that is probably more dangerous for security than requests bundling urllib3 ;) We supported the option to skip SSL compression since before adopting requests (see 556082cd6632dbce52ccb67ace57410d61057d66), useful when uploading already compressed images. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Sep 19, 2014, at 12:42 PM, Mark Washenberger mark.washenber...@markwash.net wrote: On Fri, Sep 19, 2014 at 8:59 AM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: On Sep 19, 2014, at 11:54 AM, Brant Knudson b...@acm.org mailto:b...@acm.org wrote: I don't think anyone would be complaining if glanceclient didn't have the need to reach into and monkeypatch requests's connection pool manager[1]. Is there a way to tell requests to build the https connections differently without monkeypatching urllib3.poolmanager? glanceclient's monkeypatching of the global variable here is dangerous since it will mess with the application and every other library if the application or another library uses glanceclient. [1] http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/common/https.py#n75 http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/common/https.py#n75 Why does it need to use it’s own VerifiedHTTPSConnection class? Ironically reimplementing that is probably more dangerous for security than requests bundling urllib3 ;) We supported the option to skip SSL compression since before adopting requests (see 556082cd6632dbce52ccb67ace57410d61057d66), useful when uploading already compressed images. Is that all it’s used for? Probably it’s sane to just delete it then. On Python 3.2+, 2.7.9+ Python provides the APIs to do it in the stdlib and urllib3 (and thus requests) will remove TLS Compression by default. Python 2.6, and 2.7.0-2.7.8 do not provide the APIs to do so, however on Python 2.x if you install pyOpenSSL, ndg-httpsclient, and pyasn1 then it’ll also disable TLS compression (automatically if you use requests, you have to do an import + function call with raw urllib3). So you can remove all that code and just let requests/urllib3 handle it on 3.2+, 2.7.9+ and for anything less than that either use conditional dependencies to have glance client depend on pyOpenSSL, ndg-httpsclient, and pyasn1 on Python 2.x, or let them be optional and if people want to disable TLS compression in those versions they can install those versions themselves. By the way, everything above holds true for SNI as well. This seems like the best of both worlds, glance client isn’t importing stuff from the vendored requests.packages.*, people get TLS Compression disabled (by default or optional depending on the choice the project makes), and it no longer has to maintain it’s own copy of security sensitive code. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Fri, Sep 19, 2014 at 6:58 PM, Donald Stufft don...@stufft.io wrote: So you can remove all that code and just let requests/urllib3 handle it on 3.2+, 2.7.9+ and for anything less than that either use conditional dependencies to have glance client depend on pyOpenSSL, ndg-httpsclient, and pyasn1 on Python 2.x, or let them be optional and if people want to disable TLS compression in those versions they can install those versions themselves. we have that issue as well for swiftclient, see the great write-up from stuart here : https://answers.launchpad.net/swift/+question/196920 just removing it this and let hope that users uses bleeding edge python (which they don't) is not going to work for us. and the pyOpenSSL way is very unfriendly to the end-user as well. Chmouel ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Sep 19, 2014, at 2:26 PM, Chmouel Boudjnah chmo...@enovance.com wrote: On Fri, Sep 19, 2014 at 6:58 PM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: So you can remove all that code and just let requests/urllib3 handle it on 3.2+, 2.7.9+ and for anything less than that either use conditional dependencies to have glance client depend on pyOpenSSL, ndg-httpsclient, and pyasn1 on Python 2.x, or let them be optional and if people want to disable TLS compression in those versions they can install those versions themselves. we have that issue as well for swiftclient, see the great write-up from stuart here : https://answers.launchpad.net/swift/+question/196920 https://answers.launchpad.net/swift/+question/196920 just removing it this and let hope that users uses bleeding edge python (which they don't) is not going to work for us. and the pyOpenSSL way is very unfriendly to the end-user as well. Unfortunately those are the only options besides using a different TLS implementation besides pyOpenSSL all together. Python 2.x standard library did not include the requisite nobs for configuring this, it wasn’t until Python 3.2+ that the ssl module in the standard library gained the ability to have these kinds of things applied to it. Python 2.7.9 contains a back port of the 3.x ssl module to Python 2.7, so that’s the first time in the 2.x line that the standard library has the knobs to change these things. The alternative to 3.2+ or 2.7.9+ is using an alternative TLS implementation, of which pyOpenSSL is by far the most popular (and it’s what glanceclient is using right now). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
Excerpts from Ian Cordasco's message of 2014-09-18 10:33:04 -0700: On 9/18/14, 11:29 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700: On Sep 18, 2014, at 10:18 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700: On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote: Linux distributions are not the end be all of distribution models and they don’t get to dictate to upstream. Well, distributions is where the final user is, and where software gets consumed. Our priority should be the end users. Distributions are not the only place that people get their software from, unless you think that the ~3 million downloads requests has received on PyPI in the last 30 days are distributions downloading requests to package in their OSs. Do pypi users not also need to be able to detect and fix any versions of libraries they might have? If one has some virtualenvs with various libraries and apps installed and no --system-site-packages, one would probably still want to run 'pip freeze' in all of them and find out what libraries are there and need to be fixed. Anyway, generally security updates require a comprehensive strategy. One common comprehensive strategy is version assertion. Vendoring complicates that immensely. It doesn’t really matter. PyPI doesn’t dictate to projects who host there what that project is allowed to do except in some very broad circumstances. Whether or not requests *should* do this doesn't really have any bearing on what Openstack should do to cope with it. The facts are that requests does it, and that people pulling things from PyPI is an actual platform that needs thought about. This leaves Openstack with a few reasonable/sane options: 1) Decide that vendoring in requests is unacceptable to what Openstack as a project is willing to support, and cease the use of requests. 2) Decide that what requests offers is good enough that it outweighs the fact that it vendors urllib3 and continue using it. There's also 3) fork requests, which is the democratic way to vote out an upstream that isn't supporting the needs of the masses. Given requests’ download count, I have to doubt that OpenStack users constitute the masses in this case. This wasn't the masses from the requests stand point, but from the OpenStack standpoint. Consider the case of a small island territory of a much larger nation. At some point most of them have claimed their independence from the larger nation unless the larger nation is willing to step up and make them a full member with a real vote. This allows them to act in their best interest. So even if it means a much more difficult road, it is the road most advantageous to them. Also upon reflection, it's a bit interesting that forking requests is being dismissed so quickly, when in essence, requests maintains a fork of urllib3 in tree (albeit, one that is just a fork from the _releases_, not from the whole project). I don't think we're anywhere near there, but I wanted to make it clear there _is_ a more extreme option. If the 2nd option is chosen, then doing anything but supporting the fact that requests vendors urllib3 within the code that openstack writes is hurting the users who fetch these projects from PyPI because you don't agree with one of the choices that requests makes. By all means do conditional imports to lessen the impact that the choice requests has made (and the one that Openstack has made to use requests) on downstream distributors, but unconditionally importing from the top level urllib3 for use within requests is flat out wrong. Obviously neither of these options excludes the choice to lean on requests to reverse this decision as well. However that is best done elsewhere as the person making that decision isn't a member of these mailing lists as far as I am aware. To be clear, I think we should keep using requests. But we should lend our influence upstream and explain that our users are required to deal with this in a way that perhaps hasn't been considered or given the appropriate priority. It’s been considered several times. There have been multiple issues. There’s more than just the one you linked to. The decision is highly unlikely to change whether it’s coming from a group of people in OpenStack or another distribution package maintainer. Indeed, hence my thinking that forking requests might be in order. Even if that fork is just a long lived fork that stays mostly in sync, but without urllib3 vendored. I think that has actually already happened in the distros... so I wonder how painful it would be to do the same thing on pypi, and let the distros just consume that. Anyway, I'm not going to take that challenge on,
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Fri, Sep 19, 2014 at 11:26 AM, Chmouel Boudjnah chmo...@enovance.com wrote: On Fri, Sep 19, 2014 at 6:58 PM, Donald Stufft don...@stufft.io wrote: So you can remove all that code and just let requests/urllib3 handle it on 3.2+, 2.7.9+ and for anything less than that either use conditional dependencies to have glance client depend on pyOpenSSL, ndg-httpsclient, and pyasn1 on Python 2.x, or let them be optional and if people want to disable TLS compression in those versions they can install those versions themselves. we have that issue as well for swiftclient, see the great write-up from stuart here : https://answers.launchpad.net/swift/+question/196920 just removing it this and let hope that users uses bleeding edge python (which they don't) is not going to work for us. and the pyOpenSSL way is very unfriendly to the end-user as well. Chmouel ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I'm very sympathetic with Chmouel's assessment, but it seems like adding pyasn1 and ndg-httpsclient dependencies is relatively straightforward and does not incur a substantial additional overhead on the install process. We already depend on pyOpenSSL, which seems to be the main contributor to glanceclient's list of unsavory dependencies. We would need to add ndg-httpsclient to openstack/requirements, as well, but I assume that is doable. I'm a bit disappointed that even with the fix, the requests/urllib3 stack is still trying to completely make this transport level decision for me. Its fine to have defaults, but they should be able to be overridden. For this release cycle, the best answer IMO is still just to switch to a conditional import of requests.packages.urllib3 in our test module, which was the original complaint. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
Ian Cordasco ian.corda...@rackspace.com writes: urllib3 do that automatically. I haven’t started to investigate exactly why they do this. Likewise, glance client has custom certificate verification in glanceclient.common.https. Why? I’m not exactly certain this probably come from pre-requests port uses when it was using httplib which didn't have SSL verification. There is a old wiki page about it here https://wiki.openstack.org/wiki/SecureClientConnections Slightly off-topic, speaking about requests and OpenStack client, it would be nice to have Expect/100-Continue feature tackled for glanceclient and swiftclient : https://github.com/kennethreitz/requests/issues/713 Chmouel ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 09/18/2014 04:43 AM, Donald Stufft wrote: On Sep 17, 2014, at 10:24 PM, Thomas Goirand z...@debian.org mailto:z...@debian.org wrote: On 09/18/2014 08:22 AM, Morgan Fainberg wrote: I think that all of the conversation to this point has been valuable, the general consensus is vendoring a library is not as desirable as using it strictly as a dependency. It would be nice in a perfect world if vendoring wasn’t and issue, but in this case I think the root of the matter is that Debian un-vendors urllib3 and we have referenced the vendored urllib3 instead of installing and utilizing urllib3 directly. This poses at least one problem for us: we are not able to guarantee we’re using the same urllib3 library as requests is. I am unsure how big of a deal this ends up being, but it is a concern and has brought up a question of how to handle this in the most appropriate and consistent way across all of the distributions we as OpenStack support. Does this make requests a bad library we should toss aside for something else? Instead of being concerned with the reasons for vendoring urllib3 (or un-vendoring it) we should shift the conversation towards two questions: 1. Is it a real issue if the version of urllib3 is mismatched between our client libraries and requests? 2. If it is a real issue how are we solving it? The main issue is that urllib3 in requests, as other pointed out, is not up-to-date, and will not be updated. In fact, that's the main reason why the upstream authors of requests are vendorizing: it's because they don't want to carry the burden of staying up-to-date. I don’t think this is remotely true, often times requests updates itself to versions of urllib3 which aren’t even released yet. Sometimes urllib3 might make commits and do a release that happens between requests versions though. I mean technically they might be not up to date until their next version release though. And then, there's incompatibilities and divergences that appear, leading to all sorts of unexpected issues, like one thing working with pip, but not with the packages. This kind of issues are very hard to understand and debug. Distributions may report the issue upstream, then upstream will say but it's working for me, and then we may loose a lot of time. This happened already, and may happen again if we don't care enough. I think this is bound to happen anytime you have downstream modifying things. It happens in pip (pip vendors things too) and yea it’s quite annoying but part of PEP 440 is providing ways for downstream to signal they’ve modified things so that instead of “foo 1.0” you have “foo 1.0+ubuntu1” or whatever. Obviously we can work with the requests team to figure out the best approach. There's only a single approach that works: have the requests upstream authors to stop embedding foreign code, and use the dependency instead. There are legitimate reasons for a project to vendor things. Linux distributions are not the end be all of distribution models and they don’t get to dictate to upstream. Generally I agree that requests should not vendor urllib3, but it’s not a clear cut thing where there is one right way to do it. We should focus on the solution here rather than continuing down the path of whether requests should/shouldn’t be vendoring it’s dependencies since it is clear that the team has their reasons and does not want to switch to the dependency model again. I'm sure they have tons of wrong reasons. If they don't want to change anything, then we can only try to work around the issue, and never use the embedded version. Generally you either work with the embedded versions or you don’t use requests. You’re going to get very strange incompatibility problems if you try to mis requests.packages.urllib3 and urllib3 in one codebase and if you’re using requests at all it’s going to be expecting to use the embedded copy of urllib3. After having gone through the whole thread and read all the concerns, problems and reasonings, I think we should stick to requests as-is for now and deal with this particular issue. Regardless of the vendorized urllib3 package, I believe requests is a good library with an easy-to-consume API and it has solved several problems throughout OpenStack. Not to mention it's also helpped with making OpenStack more consistent. We've put a lot of effort to get to this point and I don't think we should revert all that because of the vendorized `urllib3` package. Cheers, Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
2014-09-17 17:15 GMT+02:00 Thomas Goirand z...@debian.org: File bla/tests/test_ssl.py, line 19, in module from requests.packages.urllib3 import poolmanager ImportError: No module named packages.urllib3 This is in tests only, in runtime code there is conditional import of vendorized urllib3 falling back to system library. So what about https://review.openstack.org/122379 Cheers, Alan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 09/18/2014 04:01 PM, Flavio Percoco wrote: After having gone through the whole thread and read all the concerns, problems and reasonings, I think we should stick to requests as-is for now and deal with this particular issue. Regardless of the vendorized urllib3 package, I believe requests is a good library with an easy-to-consume API and it has solved several problems throughout OpenStack. Not to mention it's also helpped with making OpenStack more consistent. We've put a lot of effort to get to this point and I don't think we should revert all that because of the vendorized `urllib3` package. Cheers, Flavio I, at least, haven't suggested that we stop using requests. Just that we don't internally use any of the requests.packages.* stuff. The rest of the debate about the good or bad things with vendorizing, even if it is my view that it's a really bad thing, is IMO not interesting for the OpenStack project. Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Sep 18, 2014, at 7:43 AM, Thomas Goirand z...@debian.org wrote: On 09/18/2014 04:01 PM, Flavio Percoco wrote: After having gone through the whole thread and read all the concerns, problems and reasonings, I think we should stick to requests as-is for now and deal with this particular issue. Regardless of the vendorized urllib3 package, I believe requests is a good library with an easy-to-consume API and it has solved several problems throughout OpenStack. Not to mention it's also helpped with making OpenStack more consistent. We've put a lot of effort to get to this point and I don't think we should revert all that because of the vendorized `urllib3` package. Cheers, Flavio I, at least, haven't suggested that we stop using requests. Just that we don't internally use any of the requests.packages.* stuff. The rest of the debate about the good or bad things with vendorizing, even if it is my view that it's a really bad thing, is IMO not interesting for the OpenStack project. I don’t believe that’s a good idea. If you’re wanting to use urllib3 in order to interact with requests than you *should* be using requests.packages.urllib3, to do anything else risks having two different versions of urllib3 primitives at play in one subsystem. It’s not even completely possible in the case that prompted this thread originally since the reason requests.packages.urllib3 was being imported from was so that there could be an is instance() check against one of the classes. If that wasn’t imported from requests.packages.urllib3 but instead from just urllib3 than the isinstance check would always fail. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 09/18/2014 10:43 AM, Donald Stufft wrote: Obviously we can work with the requests team to figure out the best approach. There's only a single approach that works: have the requests upstream authors to stop embedding foreign code, and use the dependency instead. There are legitimate reasons for a project to vendor things. Yes, there's lot of reasons. But so fare, I haven't read about any valid one. Linux distributions are not the end be all of distribution models and they don’t get to dictate to upstream. Well, distributions is where the final user is, and where software gets consumed. Our priority should be the end users. Generally I agree that requests should not vendor urllib3 Unfortunately, it doesn't seem requests upstream agree, so we can only deal with the issue. This means not using requests.packages.*. You’re going to get very strange incompatibility problems if you try to mis requests.packages.urllib3 and urllib3 in one codebase and if you’re using requests at all it’s going to be expecting to use the embedded copy of urllib3. I'm well aware of this. As I wrote, I already had to deal with issues like that, and I'm expecting even more in the future. Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote: Linux distributions are not the end be all of distribution models and they don’t get to dictate to upstream. Well, distributions is where the final user is, and where software gets consumed. Our priority should be the end users. Distributions are not the only place that people get their software from, unless you think that the ~3 million downloads requests has received on PyPI in the last 30 days are distributions downloading requests to package in their OSs. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Thu, Sep 18, 2014 at 1:58 PM, Donald Stufft don...@stufft.io wrote: Distributions are not the only place that people get their software from, unless you think that the ~3 million downloads requests has received on PyPI in the last 30 days are distributions downloading requests to package in their OSs. I think Thomas was speaking in the context of how OpenStack is used by the end user and that probably the point of debate here, requests ships libraries inside to make it easy for users that doen't use Linux distro packages, when in OpenStack (or at least in prod) packagers are something we generally very much care about. Chmouel ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Sep 18, 2014, at 9:00 AM, Chmouel Boudjnah chmo...@enovance.com wrote: On Thu, Sep 18, 2014 at 1:58 PM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: Distributions are not the only place that people get their software from, unless you think that the ~3 million downloads requests has received on PyPI in the last 30 days are distributions downloading requests to package in their OSs. I think Thomas was speaking in the context of how OpenStack is used by the end user and that probably the point of debate here, requests ships libraries inside to make it easy for users that doen't use Linux distro packages, when in OpenStack (or at least in prod) packagers are something we generally very much care about. Even then, my statement holds true, just with different numbers. Every distribution modifies upstream in different ways, I think it's insane to do contortions which will break things for people *not* getting things through those channels. If distributions are going to modify one upstream project they should expect to need to modify things that depend on that project in ways that are sensitive to what they've modified. The only real sane thing IMO is for openstack to consider requests as it is on PyPI. If openstack wants to make it easier for downstream to de-vendor urllib3 from requests then when openstack wants to import from requests.packages.* it can instead do: try: from requests.packages import urllib3 except ImportError: import urllib3 This will cause it to work correctly when requests is installed in a pristine state, and will fallback to handling the modifications that some downstream redistributors make. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700: On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote: Linux distributions are not the end be all of distribution models and they don’t get to dictate to upstream. Well, distributions is where the final user is, and where software gets consumed. Our priority should be the end users. Distributions are not the only place that people get their software from, unless you think that the ~3 million downloads requests has received on PyPI in the last 30 days are distributions downloading requests to package in their OSs. Do pypi users not also need to be able to detect and fix any versions of libraries they might have? If one has some virtualenvs with various libraries and apps installed and no --system-site-packages, one would probably still want to run 'pip freeze' in all of them and find out what libraries are there and need to be fixed. Anyway, generally security updates require a comprehensive strategy. One common comprehensive strategy is version assertion. Vendoring complicates that immensely. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 9/17/14, 9:24 PM, Thomas Goirand z...@debian.org wrote:
On 09/18/2014 08:22 AM, Morgan Fainberg wrote:
I think that all of the conversation to this point has been valuable,
the general consensus is vendoring a library is not as desirable as
using it strictly as a dependency. It would be nice in a perfect
world if vendoring wasn’t and issue, but in this case I think the
root of the matter is that Debian un-vendors urllib3 and we have
referenced the vendored urllib3 instead of installing and utilizing
urllib3 directly.
This poses at least one problem for us: we are not able to guarantee
we’re using the same urllib3 library as requests is. I am unsure how
big of a deal this ends up being, but it is a concern and has brought
up a question of how to handle this in the most appropriate and
consistent way across all of the distributions we as OpenStack support.
Does this make requests a bad library we should toss aside for
something else? Instead of being concerned with the reasons for
vendoring urllib3 (or un-vendoring it) we should shift the conversation
towards two questions:
1. Is it a real issue if the version of urllib3 is mismatched between
our client libraries and requests?
2. If it is a real issue how are we solving it?
The main issue is that urllib3 in requests, as other pointed out, is not
up-to-date, and will not be updated. In fact, that's the main reason why
the upstream authors of requests are vendorizing: it's because they
don't want to carry the burden of staying up-to-date.
How involved are you with requests’ development process? You must not be
very involved because this is the exact opposite reason of why we vendor.
More often that not we pull in urllib3 to get unreleased features that we
build upon for our newest release. If what you said was true, 2.4.{0,1}
would not have the ability to pass socket level options that nova client
decides it wants to set. If we were pinning to an old version of urllib3,
this feature would never be possible. We vendor, because in order to
provide these features we don’t want to have to support the ancient
versions of urllib3 that used to cause us headaches on platforms like
Debian.
And then, there's incompatibilities and divergences that appear, leading
to all sorts of unexpected issues, like one thing working with pip, but
not with the packages. This kind of issues are very hard to understand
and debug. Distributions may report the issue upstream, then upstream
will say but it's working for me, and then we may loose a lot of time.
This happened already, and may happen again if we don't care enough.
Obviously we can work with the requests team to figure out the best
approach.
There's only a single approach that works: have the requests upstream
authors to stop embedding foreign code, and use the dependency instead.
We should focus on the solution here rather than continuing
down the path of whether requests should/shouldn’t be vendoring it’s
dependencies since it is clear that the team has their reasons and
does not want to switch to the dependency model again.
I'm sure they have tons of wrong reasons. If they don't want to change
anything, then we can only try to work around the issue, and never use
the embedded version.
The fact is, anyone who doesn’t run their tests in Devstack runs them in a
virtual environment. The line you’re complaining about is actually correct
in that context (the context where pip installs requests). That said, the
test should instead use a conditional import and fallback to the vendored
copy.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 9/18/14, 2:27 AM, Chmouel Boudjnah chmo...@enovance.com wrote: Ian Cordasco ian.corda...@rackspace.com writes: urllib3 do that automatically. I haven’t started to investigate exactly why they do this. Likewise, glance client has custom certificate verification in glanceclient.common.https. Why? I’m not exactly certain this probably come from pre-requests port uses when it was using httplib which didn't have SSL verification. There is a old wiki page about it here https://wiki.openstack.org/wiki/SecureClientConnections Slightly off-topic, speaking about requests and OpenStack client, it would be nice to have Expect/100-Continue feature tackled for glanceclient and swiftclient : https://github.com/kennethreitz/requests/issues/713 I’m glad you linked to that issue Chmouel because it has all of the information you need to realize that it’s an entirely impossible feature while still relying on httplib. You can still send that header, but requests has no meaningful way of special-casing it. The standard library does not return the header we need to know that we can continue with the upload and the RFC is wonderfully obscure enough to make requests’ current behavior correct. It says a client needs to wait for the server to respond or some amount of time before starting the upload. Granted we don’t call “sleep” and start the upload immediately, but we have no way of determining if the server has responded thanks to httplib. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Sep 18, 2014, at 10:18 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700: On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote: Linux distributions are not the end be all of distribution models and they don’t get to dictate to upstream. Well, distributions is where the final user is, and where software gets consumed. Our priority should be the end users. Distributions are not the only place that people get their software from, unless you think that the ~3 million downloads requests has received on PyPI in the last 30 days are distributions downloading requests to package in their OSs. Do pypi users not also need to be able to detect and fix any versions of libraries they might have? If one has some virtualenvs with various libraries and apps installed and no --system-site-packages, one would probably still want to run 'pip freeze' in all of them and find out what libraries are there and need to be fixed. Anyway, generally security updates require a comprehensive strategy. One common comprehensive strategy is version assertion. Vendoring complicates that immensely. It doesn’t really matter. PyPI doesn’t dictate to projects who host there what that project is allowed to do except in some very broad circumstances. Whether or not requests *should* do this doesn't really have any bearing on what Openstack should do to cope with it. The facts are that requests does it, and that people pulling things from PyPI is an actual platform that needs thought about. This leaves Openstack with a few reasonable/sane options: 1) Decide that vendoring in requests is unacceptable to what Openstack as a project is willing to support, and cease the use of requests. 2) Decide that what requests offers is good enough that it outweighs the fact that it vendors urllib3 and continue using it. If the 2nd option is chosen, then doing anything but supporting the fact that requests vendors urllib3 within the code that openstack writes is hurting the users who fetch these projects from PyPI because you don't agree with one of the choices that requests makes. By all means do conditional imports to lessen the impact that the choice requests has made (and the one that Openstack has made to use requests) on downstream distributors, but unconditionally importing from the top level urllib3 for use within requests is flat out wrong. Obviously neither of these options excludes the choice to lean on requests to reverse this decision as well. However that is best done elsewhere as the person making that decision isn't a member of these mailing lists as far as I am aware. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 9/18/14, 9:18 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700: On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote: Linux distributions are not the end be all of distribution models and they don’t get to dictate to upstream. Well, distributions is where the final user is, and where software gets consumed. Our priority should be the end users. Distributions are not the only place that people get their software from, unless you think that the ~3 million downloads requests has received on PyPI in the last 30 days are distributions downloading requests to package in their OSs. Do pypi users not also need to be able to detect and fix any versions of libraries they might have? If one has some virtualenvs with various libraries and apps installed and no --system-site-packages, one would probably still want to run 'pip freeze' in all of them and find out what libraries are there and need to be fixed. Anyway, generally security updates require a comprehensive strategy. One common comprehensive strategy is version assertion. Vendoring complicates that immensely. Except that even OpenStack doesn’t pin requests because of how extraordinarily stable our API is. While you can argue that Kenneth has non-standard opinions about his library, Cory and I take backwards compatibility and stability very seriously. This means anyone can upgrade to a newer version of requests without worrying that it will be backwards incompatible. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700: On Sep 18, 2014, at 10:18 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700: On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote: Linux distributions are not the end be all of distribution models and they don’t get to dictate to upstream. Well, distributions is where the final user is, and where software gets consumed. Our priority should be the end users. Distributions are not the only place that people get their software from, unless you think that the ~3 million downloads requests has received on PyPI in the last 30 days are distributions downloading requests to package in their OSs. Do pypi users not also need to be able to detect and fix any versions of libraries they might have? If one has some virtualenvs with various libraries and apps installed and no --system-site-packages, one would probably still want to run 'pip freeze' in all of them and find out what libraries are there and need to be fixed. Anyway, generally security updates require a comprehensive strategy. One common comprehensive strategy is version assertion. Vendoring complicates that immensely. It doesn’t really matter. PyPI doesn’t dictate to projects who host there what that project is allowed to do except in some very broad circumstances. Whether or not requests *should* do this doesn't really have any bearing on what Openstack should do to cope with it. The facts are that requests does it, and that people pulling things from PyPI is an actual platform that needs thought about. This leaves Openstack with a few reasonable/sane options: 1) Decide that vendoring in requests is unacceptable to what Openstack as a project is willing to support, and cease the use of requests. 2) Decide that what requests offers is good enough that it outweighs the fact that it vendors urllib3 and continue using it. There's also 3) fork requests, which is the democratic way to vote out an upstream that isn't supporting the needs of the masses. I don't think we're anywhere near there, but I wanted to make it clear there _is_ a more extreme option. If the 2nd option is chosen, then doing anything but supporting the fact that requests vendors urllib3 within the code that openstack writes is hurting the users who fetch these projects from PyPI because you don't agree with one of the choices that requests makes. By all means do conditional imports to lessen the impact that the choice requests has made (and the one that Openstack has made to use requests) on downstream distributors, but unconditionally importing from the top level urllib3 for use within requests is flat out wrong. Obviously neither of these options excludes the choice to lean on requests to reverse this decision as well. However that is best done elsewhere as the person making that decision isn't a member of these mailing lists as far as I am aware. To be clear, I think we should keep using requests. But we should lend our influence upstream and explain that our users are required to deal with this in a way that perhaps hasn't been considered or given the appropriate priority. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Sep 18, 2014, at 12:29 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700: On Sep 18, 2014, at 10:18 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700: On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote: Linux distributions are not the end be all of distribution models and they don’t get to dictate to upstream. Well, distributions is where the final user is, and where software gets consumed. Our priority should be the end users. Distributions are not the only place that people get their software from, unless you think that the ~3 million downloads requests has received on PyPI in the last 30 days are distributions downloading requests to package in their OSs. Do pypi users not also need to be able to detect and fix any versions of libraries they might have? If one has some virtualenvs with various libraries and apps installed and no --system-site-packages, one would probably still want to run 'pip freeze' in all of them and find out what libraries are there and need to be fixed. Anyway, generally security updates require a comprehensive strategy. One common comprehensive strategy is version assertion. Vendoring complicates that immensely. It doesn’t really matter. PyPI doesn’t dictate to projects who host there what that project is allowed to do except in some very broad circumstances. Whether or not requests *should* do this doesn't really have any bearing on what Openstack should do to cope with it. The facts are that requests does it, and that people pulling things from PyPI is an actual platform that needs thought about. This leaves Openstack with a few reasonable/sane options: 1) Decide that vendoring in requests is unacceptable to what Openstack as a project is willing to support, and cease the use of requests. 2) Decide that what requests offers is good enough that it outweighs the fact that it vendors urllib3 and continue using it. There's also 3) fork requests, which is the democratic way to vote out an upstream that isn't supporting the needs of the masses. I don't think we're anywhere near there, but I wanted to make it clear there _is_ a more extreme option. Technically that’s just a specific case of option 1) ;) But yes that’s a thing Openstack can do. If the 2nd option is chosen, then doing anything but supporting the fact that requests vendors urllib3 within the code that openstack writes is hurting the users who fetch these projects from PyPI because you don't agree with one of the choices that requests makes. By all means do conditional imports to lessen the impact that the choice requests has made (and the one that Openstack has made to use requests) on downstream distributors, but unconditionally importing from the top level urllib3 for use within requests is flat out wrong. Obviously neither of these options excludes the choice to lean on requests to reverse this decision as well. However that is best done elsewhere as the person making that decision isn't a member of these mailing lists as far as I am aware. To be clear, I think we should keep using requests. But we should lend our influence upstream and explain that our users are required to deal with this in a way that perhaps hasn't been considered or given the appropriate priority. I think that’s completely reasonable. I don’t think there’s going to be much movement, I’ve had this argument with Kenneth on more than one occasion and he’s very happy with his decision to vendor urllib3, but hey maybe Openstack would have better luck. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
Excerpts from Ian Cordasco's message of 2014-09-18 07:35:10 -0700: On 9/18/14, 9:18 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700: On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote: Linux distributions are not the end be all of distribution models and they don’t get to dictate to upstream. Well, distributions is where the final user is, and where software gets consumed. Our priority should be the end users. Distributions are not the only place that people get their software from, unless you think that the ~3 million downloads requests has received on PyPI in the last 30 days are distributions downloading requests to package in their OSs. Do pypi users not also need to be able to detect and fix any versions of libraries they might have? If one has some virtualenvs with various libraries and apps installed and no --system-site-packages, one would probably still want to run 'pip freeze' in all of them and find out what libraries are there and need to be fixed. Anyway, generally security updates require a comprehensive strategy. One common comprehensive strategy is version assertion. Vendoring complicates that immensely. Except that even OpenStack doesn’t pin requests because of how extraordinarily stable our API is. While you can argue that Kenneth has non-standard opinions about his library, Cory and I take backwards compatibility and stability very seriously. This means anyone can upgrade to a newer version of requests without worrying that it will be backwards incompatible. All of your hard work is very much appreciated. I don't understand what your assertion means though. We don't pin things. However, our users end up pinning when they install via pip, and our distros end up pinning when they deliver a version. Without any indication that urllib3 is in the system, they will fail at any cursory version audit that looks for it. I'm not saying either way is right or wrong either.. I'm suggesting that this is a valid, proven method for large scale risk assessment, and it is complicated quite a bit by vendored libraries. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 9/18/14, 11:29 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700: On Sep 18, 2014, at 10:18 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700: On Sep 18, 2014, at 7:54 AM, Thomas Goirand z...@debian.org wrote: Linux distributions are not the end be all of distribution models and they don’t get to dictate to upstream. Well, distributions is where the final user is, and where software gets consumed. Our priority should be the end users. Distributions are not the only place that people get their software from, unless you think that the ~3 million downloads requests has received on PyPI in the last 30 days are distributions downloading requests to package in their OSs. Do pypi users not also need to be able to detect and fix any versions of libraries they might have? If one has some virtualenvs with various libraries and apps installed and no --system-site-packages, one would probably still want to run 'pip freeze' in all of them and find out what libraries are there and need to be fixed. Anyway, generally security updates require a comprehensive strategy. One common comprehensive strategy is version assertion. Vendoring complicates that immensely. It doesn’t really matter. PyPI doesn’t dictate to projects who host there what that project is allowed to do except in some very broad circumstances. Whether or not requests *should* do this doesn't really have any bearing on what Openstack should do to cope with it. The facts are that requests does it, and that people pulling things from PyPI is an actual platform that needs thought about. This leaves Openstack with a few reasonable/sane options: 1) Decide that vendoring in requests is unacceptable to what Openstack as a project is willing to support, and cease the use of requests. 2) Decide that what requests offers is good enough that it outweighs the fact that it vendors urllib3 and continue using it. There's also 3) fork requests, which is the democratic way to vote out an upstream that isn't supporting the needs of the masses. Given requests’ download count, I have to doubt that OpenStack users constitute the masses in this case. I don't think we're anywhere near there, but I wanted to make it clear there _is_ a more extreme option. If the 2nd option is chosen, then doing anything but supporting the fact that requests vendors urllib3 within the code that openstack writes is hurting the users who fetch these projects from PyPI because you don't agree with one of the choices that requests makes. By all means do conditional imports to lessen the impact that the choice requests has made (and the one that Openstack has made to use requests) on downstream distributors, but unconditionally importing from the top level urllib3 for use within requests is flat out wrong. Obviously neither of these options excludes the choice to lean on requests to reverse this decision as well. However that is best done elsewhere as the person making that decision isn't a member of these mailing lists as far as I am aware. To be clear, I think we should keep using requests. But we should lend our influence upstream and explain that our users are required to deal with this in a way that perhaps hasn't been considered or given the appropriate priority. It’s been considered several times. There have been multiple issues. There’s more than just the one you linked to. The decision is highly unlikely to change whether it’s coming from a group of people in OpenStack or another distribution package maintainer. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
I don't know the specific situation but it's appropriate to do this if you're using requests and wish to interact with the urllib3 that requests is using. On Sep 17, 2014, at 11:15 AM, Thomas Goirand z...@debian.org wrote: Hi, I'm horrified by what I just found. I have just found out this in glanceclient: File bla/tests/test_ssl.py, line 19, in module from requests.packages.urllib3 import poolmanager ImportError: No module named packages.urllib3 Please *DO NOT* do this. Instead, please use urllib3 from ... urllib3. Not from requests. The fact that requests is embedding its own version of urllib3 is an heresy. In Debian, the embedded version of urllib3 is removed from requests. In Debian, we spend a lot of time to un-vendorize stuff, because that's a security nightmare. I don't want to have to patch all of OpenStack to do it there as well. And no, there's no good excuse here... Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
Looking at the code on my phone it looks completely correct to use the vendored copy here and it wouldn't actually work otherwise. On Sep 17, 2014, at 11:17 AM, Donald Stufft don...@stufft.io wrote: I don't know the specific situation but it's appropriate to do this if you're using requests and wish to interact with the urllib3 that requests is using. On Sep 17, 2014, at 11:15 AM, Thomas Goirand z...@debian.org wrote: Hi, I'm horrified by what I just found. I have just found out this in glanceclient: File bla/tests/test_ssl.py, line 19, in module from requests.packages.urllib3 import poolmanager ImportError: No module named packages.urllib3 Please *DO NOT* do this. Instead, please use urllib3 from ... urllib3. Not from requests. The fact that requests is embedding its own version of urllib3 is an heresy. In Debian, the embedded version of urllib3 is removed from requests. In Debian, we spend a lot of time to un-vendorize stuff, because that's a security nightmare. I don't want to have to patch all of OpenStack to do it there as well. And no, there's no good excuse here... Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
This is where Debian's one urllib3 to rule them all model fails in a modern fast paced world. Debian is arguably doing the right thing by pushing everyone to use one API, and one library, so that when that one library is found to be vulnerable to security problems, one update covers everyone. Also, this is an HTTP/HTTPS library.. so nobody can make the argument that security isn't paramount in this context. But we all know that the app store model has started to bleed down into backend applications, and now you just ship the virtualenv or docker container that has your app as you tested it, and if that means you're 20 versions behind on urllib3, that's your problem, not the OS vendor's. I think it is _completely_ irresponsible of requests, a library, to embed another library. But I don't know if we can avoid making use of it if we are going to be exposed to objects that are attached to it. Anyway, Thomas, if you're going to send the mob with pitchforks and torches somewhere, I'd say send them to wherever requests makes its home. OpenStack is just buying their mutated product. Excerpts from Donald Stufft's message of 2014-09-17 08:22:48 -0700: Looking at the code on my phone it looks completely correct to use the vendored copy here and it wouldn't actually work otherwise. On Sep 17, 2014, at 11:17 AM, Donald Stufft don...@stufft.io wrote: I don't know the specific situation but it's appropriate to do this if you're using requests and wish to interact with the urllib3 that requests is using. On Sep 17, 2014, at 11:15 AM, Thomas Goirand z...@debian.org wrote: Hi, I'm horrified by what I just found. I have just found out this in glanceclient: File bla/tests/test_ssl.py, line 19, in module from requests.packages.urllib3 import poolmanager ImportError: No module named packages.urllib3 Please *DO NOT* do this. Instead, please use urllib3 from ... urllib3. Not from requests. The fact that requests is embedding its own version of urllib3 is an heresy. In Debian, the embedded version of urllib3 is removed from requests. In Debian, we spend a lot of time to un-vendorize stuff, because that's a security nightmare. I don't want to have to patch all of OpenStack to do it there as well. And no, there's no good excuse here... Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
I was trying request-ifying oslo.vmware and ran into this as well: https://review.openstack.org/#/c/121956/ And we don't seem to have urllib3 in global-requirements either. Should we do that first? -- dims On Wed, Sep 17, 2014 at 1:05 PM, Clint Byrum cl...@fewbar.com wrote: This is where Debian's one urllib3 to rule them all model fails in a modern fast paced world. Debian is arguably doing the right thing by pushing everyone to use one API, and one library, so that when that one library is found to be vulnerable to security problems, one update covers everyone. Also, this is an HTTP/HTTPS library.. so nobody can make the argument that security isn't paramount in this context. But we all know that the app store model has started to bleed down into backend applications, and now you just ship the virtualenv or docker container that has your app as you tested it, and if that means you're 20 versions behind on urllib3, that's your problem, not the OS vendor's. I think it is _completely_ irresponsible of requests, a library, to embed another library. But I don't know if we can avoid making use of it if we are going to be exposed to objects that are attached to it. Anyway, Thomas, if you're going to send the mob with pitchforks and torches somewhere, I'd say send them to wherever requests makes its home. OpenStack is just buying their mutated product. Excerpts from Donald Stufft's message of 2014-09-17 08:22:48 -0700: Looking at the code on my phone it looks completely correct to use the vendored copy here and it wouldn't actually work otherwise. On Sep 17, 2014, at 11:17 AM, Donald Stufft don...@stufft.io wrote: I don't know the specific situation but it's appropriate to do this if you're using requests and wish to interact with the urllib3 that requests is using. On Sep 17, 2014, at 11:15 AM, Thomas Goirand z...@debian.org wrote: Hi, I'm horrified by what I just found. I have just found out this in glanceclient: File bla/tests/test_ssl.py, line 19, in module from requests.packages.urllib3 import poolmanager ImportError: No module named packages.urllib3 Please *DO NOT* do this. Instead, please use urllib3 from ... urllib3. Not from requests. The fact that requests is embedding its own version of urllib3 is an heresy. In Debian, the embedded version of urllib3 is removed from requests. In Debian, we spend a lot of time to un-vendorize stuff, because that's a security nightmare. I don't want to have to patch all of OpenStack to do it there as well. And no, there's no good excuse here... Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: https://twitter.com/dims ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
Excerpts from Davanum Srinivas's message of 2014-09-17 10:15:29 -0700: I was trying request-ifying oslo.vmware and ran into this as well: https://review.openstack.org/#/c/121956/ And we don't seem to have urllib3 in global-requirements either. Should we do that first? Honestly, after reading this: https://github.com/kennethreitz/requests/pull/1812 I think we might want to consider requests a poor option. Its author clearly doesn't understand the role a _library_ plays in software development and considers requests an application, not a library. For instance, why is requests exposing internal implementation details at all? It should be wrapping any exceptions or objects to avoid forcing users to make this choice at all. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Sep 17, 2014, at 2:46 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Davanum Srinivas's message of 2014-09-17 10:15:29 -0700: I was trying request-ifying oslo.vmware and ran into this as well: https://review.openstack.org/#/c/121956/ And we don't seem to have urllib3 in global-requirements either. Should we do that first? Honestly, after reading this: https://github.com/kennethreitz/requests/pull/1812 I think we might want to consider requests a poor option. Its author clearly doesn't understand the role a _library_ plays in software development and considers requests an application, not a library. For instance, why is requests exposing internal implementation details at all? It should be wrapping any exceptions or objects to avoid forcing users to make this choice at all. that link is horrifying. I’m really surprised Requests does this, and that nobody has complained very loudly about it. It’s wrong on every level not the least of which is the huge security implications. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 9/17/14, 1:46 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Davanum Srinivas's message of 2014-09-17 10:15:29 -0700: I was trying request-ifying oslo.vmware and ran into this as well: https://review.openstack.org/#/c/121956/ And we don't seem to have urllib3 in global-requirements either. Should we do that first? Honestly, after reading this: https://github.com/kennethreitz/requests/pull/1812 I think we might want to consider requests a poor option. Its author clearly doesn't understand the role a _library_ plays in software development and considers requests an application, not a library. Yes that is Kenneth’s opinion. That is not the opinion of the core developers though. We see it as a library but this is something we aren’t going to currently change any time soon. For instance, why is requests exposing internal implementation details at all? Where exactly are we exposing internal implementation details? A normal user (even advanced users) can use requests without ever digging into requests.packages. What implementation details are we exposing and where? It should be wrapping any exceptions or objects to avoid forcing users to make this choice at all. We do. Occasionally (like in 2.4.0) urllib3 adds an exception that we missed notice of and it slips through. We released 2.4.1 a couple days later with the fix for that. Pretty much every error we’ve seen or know about is caught and rewrapped as a requests exception. I’m not sure what you’re arguing here, unless of course you have not used requests. That aside, I’ve been mulling over how effectively the clients use requests. I haven’t investigated all of them, but many seem to reach into implementation details on their own. If I remember nova client has something it has commented as “connection pooling” while requests and urllib3 do that automatically. I haven’t started to investigate exactly why they do this. Likewise, glance client has custom certificate verification in glanceclient.common.https. Why? I’m not exactly certain yet. It seems for the most part from what little I’ve seen that requests is too high-level a library for OpenStack’s needs at best, and actively obscures details OpenStack developers need (or don’t realize requests provides in most cases). Circling back to the issue of vendoring though: it’s a conscious decision to do this, and in the last two years there have been 2 CVEs reported for requests. There have been none for urllib3 and none for chardet. (Frankly I don’t think either urllib3 or chardet have had any CVEs reported against them, but let’s ignore that for now.) While security is typically the chief concern with vendoring, none of the libraries we use have had security issues rendering it a moot point in my opinion. The benefits of vendoring for us as a team have been numerous and we will likely continue to do it until it stops benefiting us and our users. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
Interestingly enough, the distros are doing exactly what they don't want us to do, ie, rebuilding things to use 'their' tested version of dependencies rather than the included one... On Wed, Sep 17, 2014 at 2:42 PM, Ian Cordasco ian.corda...@rackspace.com wrote: That aside, I’ve been mulling over how effectively the clients use requests. I haven’t investigated all of them, but many seem to reach into implementation details on their own. If I remember nova client has something it has commented as “connection pooling” while requests and urllib3 do that automatically. I haven’t started to investigate exactly why they do this. Likewise, glance client has custom certificate verification in glanceclient.common.https. Why? I’m not exactly certain yet. It seems for the most part from what little I’ve seen that requests is too high-level a library for OpenStack’s needs at best, and actively obscures details OpenStack developers need (or don’t realize requests provides in most cases). Part of that is my doing, the initial conversion from httplib2 to requests was intended to be as simple as possible in order to get the benefits of proper certificate verification. glanceclient never got this (maybe until recently?) because it uses OpenSSL. The come-back-and-clean-things-up work was intended to be Alessio's apiclient stuff that I think is still in oslo-incubator. That was never finished for a variety of reasons. Since that time you're seeing the results of other fixes (connection-pooling being one) that look at the existing code and not at the proper re-factor to push that stuff into requests. The real fix for the clients is to start over and re-build them on top of (in this case) requests to utilize all that it brings. This is already happening... FWIW I totally understand the desire for vendoring...I want to do the same thing with OSC because of the number of times we've been broken by requests, prettytable and others changing out from under us. It is easy enough for me to fix my box but a cloud user that just want to get his VMs running isn't going to be happy, especially on Windows. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Sep 17, 2014, at 3:42 PM, Ian Cordasco ian.corda...@rackspace.com wrote: Circling back to the issue of vendoring though: it’s a conscious decision to do this, and in the last two years there have been 2 CVEs reported for requests. There have been none for urllib3 and none for chardet. (Frankly I don’t think either urllib3 or chardet have had any CVEs reported against them, but let’s ignore that for now.) While security is typically the chief concern with vendoring, none of the libraries we use have had security issues rendering it a moot point in my opinion. That’s just amazing. Requests actually deals with security features *directly*, certificates, TLS connections, everything; to take the attitude that “well there’ve been hardly any security issues in a *whole two years*, so I’m not so concerned” is really not one that is acceptable by serious development teams. Wouldn’t it be a problem for *you* if Requests itself were vendored? You fix a major security hole, but your consuming projects don’t respond, their developers are on vacation, sorry, so that hole just keeps right on going. People make sure to upgrade their Requests libraries locally, but for all those poor saps who have *no idea* they have widely used apps that are bundling it silently, they remain totally open to vulnerabilities and the black hats have disneyland at their disposal. The blame keeps going right to you as well. Is that really how things should be done? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 9/17/14, 3:11 PM, Mike Bayer mba...@redhat.com wrote: On Sep 17, 2014, at 3:42 PM, Ian Cordasco ian.corda...@rackspace.com wrote: Circling back to the issue of vendoring though: it’s a conscious decision to do this, and in the last two years there have been 2 CVEs reported for requests. There have been none for urllib3 and none for chardet. (Frankly I don’t think either urllib3 or chardet have had any CVEs reported against them, but let’s ignore that for now.) While security is typically the chief concern with vendoring, none of the libraries we use have had security issues rendering it a moot point in my opinion. That’s just amazing. Requests actually deals with security features *directly*, certificates, TLS connections, everything; to take the attitude that “well there’ve been hardly any security issues in a *whole two years*, so I’m not so concerned” is really not one that is acceptable by serious development teams. I said 2 years, because I wasn’t involved much before that, but looking at the histories of the involved projects there aren’t mentions of CVEs before then either. Wouldn’t it be a problem for *you* if Requests itself were vendored? You fix a major security hole, but your consuming projects don’t respond, their developers are on vacation, sorry, so that hole just keeps right on going. Isn’t the whole point of distributing a library to let people use it as they see fit? If requests fixes it immediately and releases it, it’s not our responsibility to search out every piece of software to fix it for them. We took all of the appropriate measures to document the two CVEs that were reported earlier this year. Software that vendored requests including still vulnerable to those two exposures are responsible for their own updates. Further, let’s consider this potential situation: Project X pins a version of requests. Alice doesn’t know anything about requests and does pip install X. Until Alice takes a more active role in the development of Project X and looks into requests, she will never know she’s installed software that has exposures in it. In all likelihood, any person who just uses something that pins requests will never check for it. If they just use pip and Project X never updates, it’s not our responsibility for anything that happens to the user. People make sure to upgrade their Requests libraries locally, but for all those poor saps who have *no idea* they have widely used apps that are bundling it silently, they remain totally open to vulnerabilities and the black hats have disneyland at their disposal. I think more applications bundle it than you realize. You’re likely using one daily that does it. The blame keeps going right to you as well. Is that really how things should be done? And yeah, we’ll continue to take the blame for the mistake that was made for those two exposures. As for “Is that how things should be done?” that’s not for me to answer. More than enough projects do it and do it out of necessity. The reality is that by vendoring its dependencies, requests allows its users more flexibility than other projects. Even if we didn’t, users would still likely find ways to vendor requests and its dependencies for their own use and in doing so would have to modify requests to rewrite the import statements to point at those vendored dependencies. The fact is that vendoring is a real solution and it’s deployed more often than you likely realize. It benefits our project and it benefits our users. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 18 September 2014 08:01, Dean Troyer dtro...@gmail.com wrote: Interestingly enough, the distros are doing exactly what they don't want us to do, ie, rebuilding things to use 'their' tested version of dependencies rather than the included one... Indeed - but the distros are solving for two specific issues: 1) Effort by the distro team 1a) 'how to minimise effort delivering up-to-date packages when the package count is 20k+'. This is a pure numbers game: update one binary on a users system, or 10, or 20 etc. Things deep down a dependency tree can turn up in huge numbers of places, if vendoring is commonplace. 1b) 'how to security fix packages where the upstream has stopped being responsive' - updating vendored trees is often harder than just unpacking a new release, since they may have deltas in addition to being vendored - and vendoring may also require patches (depending on the language). Just waiting for a new vendor release can be a long process sometimes :) And both of these are in the context of 2) how to fix things promptly for users 2a) binary packages are often quite substantial - particularly for some c++ programs - a non-binary delta based approach (and thats what all the distros started with) will consume a tonne of bandwidth if you have to multiply out the uses of a package. 2b) distros were privileged in our modern responsible disclosure world (via the vendor-sec list - I'm not sure what the current state of play is) - but at one point they found out about security issues *before* small consumers of packages do, and would fix it and then the upstream release is made. You can see, I think, how vendoring plays havoc with the amount of effort a small team has to exert to keep a large set of packages patched ahead of upstream releases of the vendored libraries. Its not an intrinsic problem - its a problem we've constructed by centralising and limiting notifications of CVEs: unless requests authors are part of the urllib3 security response team, they can never respond to CVE's in as timely a manner *while vendoring is in use*. ... FWIW I totally understand the desire for vendoring...I want to do the same thing with OSC because of the number of times we've been broken by requests, prettytable and others changing out from under us. It is easy enough for me to fix my box but a cloud user that just want to get his VMs running isn't going to be happy, especially on Windows. dt OOI were thouse changes API breaks or were we depending on nonpublic aspects? -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Wed, Sep 17, 2014 at 3:53 PM, Robert Collins robe...@robertcollins.net wrote: On 18 September 2014 08:01, Dean Troyer dtro...@gmail.com wrote: Interestingly enough, the distros are doing exactly what they don't want us to do, ie, rebuilding things to use 'their' tested version of dependencies rather than the included one... Indeed - but the distros are solving for two specific issues: No argument, just observing the recursive nature of this... Also, if we pin to a version, is the downstream consequence different? IIRC Thomas has had to do this with Django (1.7?) and Horizon, probably with others too. As a provider of an app package directly to users, dealing with the front-line consequences of changing dependencies falls on me. And its one reason with this hat on I want static linking, or a Python equivalent of it (bundling/vendoring) at the app level. As an upstream to a distro, I'm happy to let them deal with all of that. Isn't it fun being in the middle? OOI were thouse changes API breaks or were we depending on nonpublic aspects? prettytable was packaging once and I don't recall the other. requests, aside from the recent 2.4.0 release, was the 1.0.0 release when we weren't expecting it and nothing was pinned 1.0.0. I think that was an API change that bit us. The 1.0.0 version was clear, but not having the control over the timing of the change is what makes me understand Kenneth's position on urllib3 and why those who bundle requests do that too... Is my Go-ness showing yet? dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Sep 17, 2014, at 4:31 PM, Ian Cordasco ian.corda...@rackspace.com wrote: Project X pins a version of requests. Alice doesn’t know anything about requests and does pip install X. Until Alice takes a more active role in the development of Project X and looks into requests, she will never know she’s installed software that has exposures in it. If a vulnerability is reported in urllib3 1.9.1, Alice, as well as me and everyone else who is not a novice, will at least know we need to run: $ pip show urllib3 --- Name: urllib3 Version: 1.9.1 Location: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages Requires: and we know right there we have to upgrade. We upgrade, and we’re done.If we see that some library is pinning it, we will know. We will complain loudly to that library’s author and/or replace that library. The tools are there to give us what we need to be aware and to escalate the problem. When a library silently bundles the source code and bypasses any normal means of us knowing it’s present unless we read the source code or scour the documentation, we have no way to know we’re affected.Some applications, particularly pip, have to do this, however, it should only be for technical reasons. It should not be because you don’t want novice users to have to learn something, or because you’re angling to have lots of downloads on pypi. People make sure to upgrade their Requests libraries locally, but for all those poor saps who have *no idea* they have widely used apps that are bundling it silently, they remain totally open to vulnerabilities and the black hats have disneyland at their disposal. I think more applications bundle it than you realize. You’re likely using one daily that does it. SQLAlchemy itself vendorizes Queue and some fragments of six, but that is of a much smaller scale, and is for technical reasons, rather than appeasing-newbie reasons. But HTTP has a lot of security-critical surface area. If I were to just bundle my own fork of an HMAC library with a few of my own special enhancements, that would be seen as a problem. And yeah, we’ll continue to take the blame for the mistake that was made for those two exposures. As for “Is that how things should be done?” that’s not for me to answer. More than enough projects do it and do it out of necessity. The reality is that by vendoring its dependencies, requests allows its users more flexibility than other projects. I haven’t seen the technical reason for Requests doing this, I’ve only seen this reason: I want my users to be free to not use packaging if they don't won't to. They can just grab the tarball and go.”. If that’s really the only reason, then I fail to see how that reason has anything to do with flexibility, other than the flexibility to remain lazy and ignorant of basic computer programming skills - and Requests is a library *for programmers*. It doesn’t do anything without typing code. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
-Original Message- From: Ian Cordasco ian.corda...@rackspace.com Reply: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: September 17, 2014 at 16:28:57 To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3) On 9/17/14, 5:39 PM, Mike Bayer wrote: On Sep 17, 2014, at 4:31 PM, Ian Cordasco wrote: Project X pins a version of requests. Alice doesn’t know anything about requests and does pip install X. Until Alice takes a more active role in the development of Project X and looks into requests, she will never know she’s installed software that has exposures in it. If a vulnerability is reported in urllib3 1.9.1, Alice, as well as me and everyone else who is not a novice, will at least know we need to run: $ pip show urllib3 --- Name: urllib3 Version: 1.9.1 Location: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packa ges Requires: and we know right there we have to upgrade. We upgrade, and we’re done. If we see that some library is pinning it, we will know. We will complain loudly to that library’s author and/or replace that library. The tools are there to give us what we need to be aware and to escalate the problem. And when the library is unmaintained, you’ll be yelling into the echo chamber. If it is popular enough, it’ll continue to be used regardless because every recommendation that will be highest ranked by :search_engine will tell people looking for related libraries to do so. In all likelihood, most users who stumble across these projects will also be ignorant of the fact that :some_dependency has a CVE and needs to be upgraded. When a library silently bundles the source code and bypasses any normal means of us knowing it’s present unless we read the source code or scour the documentation, we have no way to know we’re affected. Some applications, particularly pip, have to do this, however, it should only be for technical reasons. It should not be because you don’t want novice users to have to learn something, or because you’re angling to have lots of downloads on pypi. It is for technical reasons, but this is not the appropriate place to discuss them. There are at least 3 other closed issues on requests’ issue tracker that discuss most (if not all) of them. This is the list to discuss the development of OpenStack. Finally, for the last time, the fact that we vendor these libraries exists in multiple places (beyond the source). People make sure to upgrade their Requests libraries locally, but for all those poor saps who have *no idea* they have widely used apps that are bundling it silently, they remain totally open to vulnerabilities and the black hats have disneyland at their disposal. I think more applications bundle it than you realize. You’re likely using one daily that does it. SQLAlchemy itself vendorizes Queue and some fragments of six, but that is of a much smaller scale, and is for technical reasons, rather than appeasing-newbie reasons. But HTTP has a lot of security-critical surface area. If I were to just bundle my own fork of an HMAC library with a few of my own special enhancements, that would be seen as a problem. It would be seen as a problem. Except we don’t do anything even remotely as security related. The majority of what we do is certificate verification. We don’t bundle ssl. We don’t bundle pyOpenSSL. We don’t bundle back ports.ssl. We don’t have custom TLS handlers that we wrote from scratch. You’re analogy is way out of proportion. And for what it’s worth, pyCrypto (if I remember correctly) has never been audited and yet it is used: https://github.com/openstack/requirements/blob/master/global-requirements.t xt#L86. That seems like a bigger issue than whether requests vendors an implementation detail. And yeah, we’ll continue to take the blame for the mistake that was made for those two exposures. As for “Is that how things should be done?” that’s not for me to answer. More than enough projects do it and do it out of necessity. The reality is that by vendoring its dependencies, requests allows its users more flexibility than other projects. I haven’t seen the technical reason for Requests doing this, I’ve only seen this reason: I want my users to be free to not use packaging if they don't won't to. They can just grab the tarball and go.”. If that’s really the only reason, then I fail to see how that reason has anything to do with flexibility, other than the flexibility to remain lazy and ignorant of basic computer programming skills - and Requests is a library *for programmers*. It doesn’t do anything without typing code. Perhaps I wasn’t clear enough. If I wasn’t, I
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On 09/18/2014 08:22 AM, Morgan Fainberg wrote: I think that all of the conversation to this point has been valuable, the general consensus is vendoring a library is not as desirable as using it strictly as a dependency. It would be nice in a perfect world if vendoring wasn’t and issue, but in this case I think the root of the matter is that Debian un-vendors urllib3 and we have referenced the vendored urllib3 instead of installing and utilizing urllib3 directly. This poses at least one problem for us: we are not able to guarantee we’re using the same urllib3 library as requests is. I am unsure how big of a deal this ends up being, but it is a concern and has brought up a question of how to handle this in the most appropriate and consistent way across all of the distributions we as OpenStack support. Does this make requests a bad library we should toss aside for something else? Instead of being concerned with the reasons for vendoring urllib3 (or un-vendoring it) we should shift the conversation towards two questions: 1. Is it a real issue if the version of urllib3 is mismatched between our client libraries and requests? 2. If it is a real issue how are we solving it? The main issue is that urllib3 in requests, as other pointed out, is not up-to-date, and will not be updated. In fact, that's the main reason why the upstream authors of requests are vendorizing: it's because they don't want to carry the burden of staying up-to-date. And then, there's incompatibilities and divergences that appear, leading to all sorts of unexpected issues, like one thing working with pip, but not with the packages. This kind of issues are very hard to understand and debug. Distributions may report the issue upstream, then upstream will say but it's working for me, and then we may loose a lot of time. This happened already, and may happen again if we don't care enough. Obviously we can work with the requests team to figure out the best approach. There's only a single approach that works: have the requests upstream authors to stop embedding foreign code, and use the dependency instead. We should focus on the solution here rather than continuing down the path of whether requests should/shouldn’t be vendoring it’s dependencies since it is clear that the team has their reasons and does not want to switch to the dependency model again. I'm sure they have tons of wrong reasons. If they don't want to change anything, then we can only try to work around the issue, and never use the embedded version. Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)
On Sep 17, 2014, at 10:24 PM, Thomas Goirand z...@debian.org wrote: On 09/18/2014 08:22 AM, Morgan Fainberg wrote: I think that all of the conversation to this point has been valuable, the general consensus is vendoring a library is not as desirable as using it strictly as a dependency. It would be nice in a perfect world if vendoring wasn’t and issue, but in this case I think the root of the matter is that Debian un-vendors urllib3 and we have referenced the vendored urllib3 instead of installing and utilizing urllib3 directly. This poses at least one problem for us: we are not able to guarantee we’re using the same urllib3 library as requests is. I am unsure how big of a deal this ends up being, but it is a concern and has brought up a question of how to handle this in the most appropriate and consistent way across all of the distributions we as OpenStack support. Does this make requests a bad library we should toss aside for something else? Instead of being concerned with the reasons for vendoring urllib3 (or un-vendoring it) we should shift the conversation towards two questions: 1. Is it a real issue if the version of urllib3 is mismatched between our client libraries and requests? 2. If it is a real issue how are we solving it? The main issue is that urllib3 in requests, as other pointed out, is not up-to-date, and will not be updated. In fact, that's the main reason why the upstream authors of requests are vendorizing: it's because they don't want to carry the burden of staying up-to-date. I don’t think this is remotely true, often times requests updates itself to versions of urllib3 which aren’t even released yet. Sometimes urllib3 might make commits and do a release that happens between requests versions though. I mean technically they might be not up to date until their next version release though. And then, there's incompatibilities and divergences that appear, leading to all sorts of unexpected issues, like one thing working with pip, but not with the packages. This kind of issues are very hard to understand and debug. Distributions may report the issue upstream, then upstream will say but it's working for me, and then we may loose a lot of time. This happened already, and may happen again if we don't care enough. I think this is bound to happen anytime you have downstream modifying things. It happens in pip (pip vendors things too) and yea it’s quite annoying but part of PEP 440 is providing ways for downstream to signal they’ve modified things so that instead of “foo 1.0” you have “foo 1.0+ubuntu1” or whatever. Obviously we can work with the requests team to figure out the best approach. There's only a single approach that works: have the requests upstream authors to stop embedding foreign code, and use the dependency instead. There are legitimate reasons for a project to vendor things. Linux distributions are not the end be all of distribution models and they don’t get to dictate to upstream. Generally I agree that requests should not vendor urllib3, but it’s not a clear cut thing where there is one right way to do it. We should focus on the solution here rather than continuing down the path of whether requests should/shouldn’t be vendoring it’s dependencies since it is clear that the team has their reasons and does not want to switch to the dependency model again. I'm sure they have tons of wrong reasons. If they don't want to change anything, then we can only try to work around the issue, and never use the embedded version. Generally you either work with the embedded versions or you don’t use requests. You’re going to get very strange incompatibility problems if you try to mis requests.packages.urllib3 and urllib3 in one codebase and if you’re using requests at all it’s going to be expecting to use the embedded copy of urllib3. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
