Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Thomas Goirand]

On 09/21/2014 11:30 PM, Ian Cordasco wrote:
> Hi Thomas,
> 
> Several people, many of whom are core contributors to other projects, have
> asked that this discussion not be continued in this venue. Discussion of
> the decisions of the core-developers of requests are not appropriate for
> this list. All three of us have email addresses that you can retrieve from
> anywhere you please. There’s a mailing list for request, albeit very
> lightly trafficked, and there’s twitter.
> 
> Cheers,
> Ian

Ian,

I don't use non-free software as a service platform such as twitter,
even less for discussing about software development. I anyway tried
twitter and didn't like it.

I first didn't reply to any discussion about vendoring, but then I saw
*a lot* of this discussion happening in this list. Because of that, I
thought I couldn't left it unanswered. Now that I did answer to so many
points of your argumentation, you're telling me to go away from this
list and do it somewhere else.

In some cases, its looking like you're just closing discussions and tell
everyone to "go away" from your own channel of communication. It shows here:
https://github.com/kennethreitz/requests/pull/1812

Will the discussion stay open if I'm joining your list? Will you guys be
open minded with someone with a different view? If so, I may try to make
a new attempt. Please do open a topic on your list with my last reply as
a start, and just CC me (I don't really want to register to
yet-another-new-mailing-list...).

> Further, I’m disappointed that you felt it appropriate or necessary
> to result to personal attacks on this list. At the very least you
> could have contained those to Twitter like others in this thread have
> done. I expected a more civil response on the openstack-dev mailing
> list.

I have re-read myself multiple times to make sure that there was no such
thing as personal attacks. I tried to use a nice tone, and have a solid
argumentation. It's looking like I failed. :(

If there's some words which you consider as a personal attack, please
feel free to quote them in a private mail, and let me know (away from
this list) where I was not nice enough, so that I get to know which part
you didn't like.

Though remember something: it's very common to read someone from a list,
and believe that there's an aggressive, when in fact, the intention is
only to be convincing. Please assume good faith.

Cheers,

Thomas Goirand (zigo)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Jay Pipes]

On 09/21/2014 11:30 AM, Ian Cordasco wrote:

Hi Thomas,

Several people, many of whom are core contributors to other projects, have
asked that this discussion not be continued in this venue. Discussion of
the decisions of the core-developers of requests are not appropriate for
this list. All three of us have email addresses that you can retrieve from
anywhere you please. There’s a mailing list for request, albeit very
lightly trafficked, and there’s twitter. Further, I’m disappointed that
you felt it appropriate or necessary to result to personal attacks on this
list. At the very least you could have contained those to Twitter like
others in this thread have done. I expected a more civil response on the
openstack-dev mailing list.


For those of us interested in this conversation, I ask that whatever 
decisions (if any) that come out of those discussions, that somebody 
please do post to the openstack-dev ML a summary of those decisions or 
actions.


Thanks much in advance,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

Hi Thomas,

Several people, many of whom are core contributors to other projects, have
asked that this discussion not be continued in this venue. Discussion of
the decisions of the core-developers of requests are not appropriate for
this list. All three of us have email addresses that you can retrieve from
anywhere you please. There’s a mailing list for request, albeit very
lightly trafficked, and there’s twitter. Further, I’m disappointed that
you felt it appropriate or necessary to result to personal attacks on this
list. At the very least you could have contained those to Twitter like
others in this thread have done. I expected a more civil response on the
openstack-dev mailing list.

Cheers,
Ian

On 9/21/14, 7:21 AM, "Thomas Goirand"  wrote:

>Hi Ian and Donald,
>
>I've read the full thread, and couldn't help to reply to it still, even
>though I previously thought I shouldn't, as what I care is OpenStack,
>not really requests, and more largely, the topic of the wrong reasons
>why upstream are embedding foreign library code copies. I completely
>agree with someone else who wrote that this thread is nearly
>uninteresting for OpenStack itself. However, it is IMO my role, as a
>package maintainer, to let you know about my view on your argumentation.
>
>If you ignore my argumentation, then at least I'll have tried! :)
>
>On 09/18/2014 03:42 AM, Ian Cordasco wrote:
>> Circling back to the issue of vendoring though: it’s a conscious
>>decision
>> to do this, and in the last two years there have been 2 CVEs reported
>>for
>> requests. There have been none for urllib3 and none for chardet.
>>(Frankly
>> I don’t think either urllib3 or chardet have had any CVEs reported
>>against
>> them, but let’s ignore that for now.) While security is typically the
>> chief concern with vendoring, none of the libraries we use have had
>> security issues rendering it a moot point in my opinion. The benefits of
>> vendoring for us as a team have been numerous and we will likely
>>continue
>> to do it until it stops benefiting us and our users.
>
>Could you please list the benefits *for end users*? I'm really saying
>users, as in, not developers. Because I don't see any benefit at all for
>the end users. I don't think any of them would like to see many version
>of the same thing on their system.
>
>Also, the issue is not only security. Let me give you an example. Simply
>do this in a Debian sid machine:
>
>apt-file search six.py | grep -v python3 | grep -v pyshared | wc -l
>
>We have in Debian, about 50 versions of six.py around, embedded in
>packages. And this doesn't even counts those where only bits of six are
>just embedded in a file which isn't called six.py.
>
>Of course, we (in Debian) would like them to be removed. Why? Because
>it's a useless complexity, with so many different version, some with
>embedded bugs which have been fixed upstream, and the like. That's not
>even about security at this point (I hardly would see how six.py would
>have security issue).
>
>There is also a waste of server resources (install time, size of
>packages in the Debian archive, increased download time, RAM footprint
>of everything, etc.). We don't need to install (and compile as .pyc at
>install time) 50 versions of six.py, a single one is enough. We're
>trying to address this as much as we can, and you'll see lots of
>packages were we did, but it's not always easy for various reasons, like
>upstream code not up-to-date with latest version, or lack of time from
>the Debian package maintainer.
>
>Also, consider the fact that six is small: a quite small single file.
>It's still unacceptable from a Unix distribution stand point, but this
>makes the "vendorizing" less of an absurdity. Now, for urllib3, it's a
>WAY bigger. There's about 25 Python files. So multiply the resulting
>waste and issues...
>
>This was a simple example for six. Now just generalize to all. There's
>numerous upstream authors who also think that it's ok, and they can be
>one of the few exception. But really, every upstream who does this think
>that he's "special". That's not the case. Requests isn't more special
>than any other Python module.
>
>On 09/18/2014 04:31 AM, Ian Cordasco wrote:
>> Isn’t the whole point of distributing a library to let people use it
>> as they see fit?
>
>The point of a library, is that it is shared among multiple consumers.
>Oh, maybe not if you're using Windows, but that's maybe out of the scope
>of this debate. Maintaining a coherent distribution with a single
>version of every library, is what distributions do as much as possible.
>It is unfortunately not always possible, but we do it as much as we can.
>
>On 09/18/2014 04:31 AM, Ian Cordasco wrote:
>> Project X pins a version of requests. Alice doesn’t know anything
>> about requests and does pip install X. Until Alice takes a more
>> active role in the development of Project X and looks into requests,
>> she will never know she’s installed software that has exposures in
>> it. In all likel

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Thomas Goirand]

On 09/18/2014 05:22 AM, Dean Troyer wrote:
> On Wed, Sep 17, 2014 at 3:53 PM, Robert Collins
> mailto:robe...@robertcollins.net>> wrote:
> 
> On 18 September 2014 08:01, Dean Troyer  > wrote:
> > Interestingly enough, the distros are doing exactly what they don't 
> want us
> > to do, ie, rebuilding things to use 'their' tested version of 
> dependencies
> > rather than the included one...

We don't use "our" tested version, we use upstream's, and a single
version of it.

> Indeed - but the distros are solving for two specific issues:
> 
> 
> No argument, just observing the recursive nature of this...
> 
> Also, if we pin to a version, is the downstream consequence different?
>  IIRC Thomas has had to do this with Django (1.7?) and Horizon, probably
> with others too.

Correct. And there's still some open issues with it, though mostly it
has been dealt with. There was also SQLAlchemy 0.8 then 0.9 a year ago
as well. Though since Mike Bayer works on OpenStack support now, I'm
sure I wont have to deal with any SQLA issue again.

It's a common mistake to think that "we just need to pin". Pinning (the
upper bound) doesn't solve any issue, apart from having the tests pass
the gate. This is sometimes urgent to "fix" the gate, so I understand
why this is done. The reality is that packages in a distribution share
common dependency, and OpenStack isn't alone in the distro.

Lucky, (almost?) everyone in the OpenStack community understand this,
and so far, I've received *a lot* of help from everyone. You have no
idea how much this is important to me. Kudos to everyone who do help or
who is at least gives moral support! :)

> As a provider of an app package directly to users, dealing with the
> front-line consequences of changing dependencies falls on me.  And its
> one reason with this hat on I want static linking, or a Python
> equivalent of it (bundling/vendoring) at the app level.

Since a few days, the Debian policy manual explicitly forbids static
linking. I fully agree with the decision to make it explicit.

Cheers,

Thomas Goirand (zigo)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Thomas Goirand]

Hi Ian and Donald,

I've read the full thread, and couldn't help to reply to it still, even
though I previously thought I shouldn't, as what I care is OpenStack,
not really requests, and more largely, the topic of the wrong reasons
why upstream are embedding foreign library code copies. I completely
agree with someone else who wrote that this thread is nearly
uninteresting for OpenStack itself. However, it is IMO my role, as a
package maintainer, to let you know about my view on your argumentation.

If you ignore my argumentation, then at least I'll have tried! :)

On 09/18/2014 03:42 AM, Ian Cordasco wrote:
> Circling back to the issue of vendoring though: it’s a conscious decision
> to do this, and in the last two years there have been 2 CVEs reported for
> requests. There have been none for urllib3 and none for chardet. (Frankly
> I don’t think either urllib3 or chardet have had any CVEs reported against
> them, but let’s ignore that for now.) While security is typically the
> chief concern with vendoring, none of the libraries we use have had
> security issues rendering it a moot point in my opinion. The benefits of
> vendoring for us as a team have been numerous and we will likely continue
> to do it until it stops benefiting us and our users.

Could you please list the benefits *for end users*? I'm really saying
users, as in, not developers. Because I don't see any benefit at all for
the end users. I don't think any of them would like to see many version
of the same thing on their system.

Also, the issue is not only security. Let me give you an example. Simply
do this in a Debian sid machine:

apt-file search six.py | grep -v python3 | grep -v pyshared | wc -l

We have in Debian, about 50 versions of six.py around, embedded in
packages. And this doesn't even counts those where only bits of six are
just embedded in a file which isn't called six.py.

Of course, we (in Debian) would like them to be removed. Why? Because
it's a useless complexity, with so many different version, some with
embedded bugs which have been fixed upstream, and the like. That's not
even about security at this point (I hardly would see how six.py would
have security issue).

There is also a waste of server resources (install time, size of
packages in the Debian archive, increased download time, RAM footprint
of everything, etc.). We don't need to install (and compile as .pyc at
install time) 50 versions of six.py, a single one is enough. We're
trying to address this as much as we can, and you'll see lots of
packages were we did, but it's not always easy for various reasons, like
upstream code not up-to-date with latest version, or lack of time from
the Debian package maintainer.

Also, consider the fact that six is small: a quite small single file.
It's still unacceptable from a Unix distribution stand point, but this
makes the "vendorizing" less of an absurdity. Now, for urllib3, it's a
WAY bigger. There's about 25 Python files. So multiply the resulting
waste and issues...

This was a simple example for six. Now just generalize to all. There's
numerous upstream authors who also think that it's ok, and they can be
one of the few exception. But really, every upstream who does this think
that he's "special". That's not the case. Requests isn't more special
than any other Python module.

On 09/18/2014 04:31 AM, Ian Cordasco wrote:
> Isn’t the whole point of distributing a library to let people use it
> as they see fit?

The point of a library, is that it is shared among multiple consumers.
Oh, maybe not if you're using Windows, but that's maybe out of the scope
of this debate. Maintaining a coherent distribution with a single
version of every library, is what distributions do as much as possible.
It is unfortunately not always possible, but we do it as much as we can.

On 09/18/2014 04:31 AM, Ian Cordasco wrote:
> Project X pins a version of requests. Alice doesn’t know anything
> about requests and does pip install X. Until Alice takes a more
> active role in the development of Project X and looks into requests,
> she will never know she’s installed software that has exposures in
> it. In all likelihood, any person who just uses something that pins
> requests will never check for it. If they just use pip and Project X
> never updates, it’s not our responsibility for anything that happens
> to the user.

This is exactly why we should, at all costs, avoid using pinning. This
is very dangerous, and leads to all sorts of issues. We should make sure
that we stay current with absolutely all libraries, and when possible,
support both the old and the new version of some incompatible API when
possible.

On 09/18/2014 04:31 AM, Ian Cordasco wrote:
> I think more applications bundle it than you realize. You’re likely
> using one daily that does it.

It's not because stupidity is wide spread that it becomes smartness.
(nothing personal, just making a general statement...)

On 09/18/2014 04:31 AM, Ian Cordasco wrote:
> The reality is that by

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Mark Washenberger]

On Fri, Sep 19, 2014 at 11:26 AM, Chmouel Boudjnah 
wrote:

>
> On Fri, Sep 19, 2014 at 6:58 PM, Donald Stufft  wrote:
>
>> So you can remove all that code and just let requests/urllib3 handle it
>> on 3.2+, 2.7.9+ and for anything less than that either use conditional
>> dependencies to have glance client depend on pyOpenSSL, ndg-httpsclient,
>> and pyasn1 on Python 2.x, or let them be optional and if people want to
>> disable TLS compression in those versions they can install those versions
>> themselves.
>
>
>
> we have that issue as well for swiftclient, see the great write-up from
> stuart here :
>
> https://answers.launchpad.net/swift/+question/196920
>
> just removing it this and let hope that users uses bleeding edge python
> (which they don't) is not going to work for us. and the pyOpenSSL way is
> very unfriendly to the end-user as well.
>
> Chmouel
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
I'm very sympathetic with Chmouel's assessment, but it seems like adding
pyasn1 and ndg-httpsclient dependencies is relatively straightforward and
does not incur a substantial additional overhead on the install process. We
already depend on pyOpenSSL, which seems to be the main contributor to
glanceclient's list of unsavory dependencies. We would need to add
ndg-httpsclient to openstack/requirements, as well, but I assume that is
doable.

I'm a bit disappointed that even with the "fix", the requests/urllib3 stack
is still trying to completely make this transport level decision for me.
Its fine to have defaults, but they should be able to be overridden.

For this release cycle, the best answer IMO is still just to switch to a
conditional import of requests.packages.urllib3 in our test module, which
was the original complaint.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Clint Byrum]

Excerpts from Ian Cordasco's message of 2014-09-18 10:33:04 -0700:
> 
> On 9/18/14, 11:29 AM, "Clint Byrum"  wrote:
> 
> >Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700:
> >> 
> >> > On Sep 18, 2014, at 10:18 AM, Clint Byrum  wrote:
> >> > 
> >> > Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
> >> >> 
> >> >>> On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
> >> >>> 
> >>  
> >>  Linux distributions are not the end be all of distribution models
> >>and
> >>  they don’t get to dictate to upstream.
> >> >>> 
> >> >>> Well, distributions is where the final user is, and where software
> >>gets
> >> >>> consumed. Our priority should be the end users.
> >> >> 
> >> >> 
> >> >> Distributions are not the only place that people get their software
> >>from,
> >> >> unless you think that the ~3 million downloads requests has received
> >> >> on PyPI in the last 30 days are distributions downloading requests to
> >> >> package in their OSs.
> >> >> 
> >> > 
> >> > Do pypi users not also need to be able to detect and fix any versions
> >> > of libraries they might have? If one has some virtualenvs with various
> >> > libraries and apps installed and no --system-site-packages, one would
> >> > probably still want to run 'pip freeze' in all of them and find out
> >>what
> >> > libraries are there and need to be fixed.
> >> > 
> >> > Anyway, generally security updates require a comprehensive strategy.
> >> > One common comprehensive strategy is version assertion.
> >> > 
> >> > Vendoring complicates that immensely.
> >> 
> >> It doesn’t really matter. PyPI doesn’t dictate to projects who host
> >>there what
> >> that project is allowed to do except in some very broad circumstances.
> >>Whether
> >> or not requests *should* do this doesn't really have any bearing on what
> >> Openstack should do to cope with it. The facts are that requests does
> >>it, and
> >> that people pulling things from PyPI is an actual platform that needs
> >>thought
> >> about.
> >> 
> >> This leaves Openstack with a few reasonable/sane options:
> >> 
> >> 1) Decide that vendoring in requests is unacceptable to what Openstack
> >>as a
> >>project is willing to support, and cease the use of requests.
> >> 2) Decide that what requests offers is good enough that it outweighs
> >>the fact
> >>that it vendors urllib3 and continue using it.
> >> 
> >
> >There's also 3) fork requests, which is the democratic way to vote out
> >an upstream that isn't supporting the needs of the masses.
> 
> Given requests’ download count, I have to doubt that OpenStack users
> constitute the masses in this case.
> 

This wasn't "the masses" from the requests stand point, but from the
OpenStack standpoint. Consider the case of a small island territory
of a much larger nation. At some point most of them have claimed their
independence from the larger nation unless the larger nation is willing
to step up and make them a full member with a real vote. This allows
them to act in their best interest. So even if it means a much more
difficult road, it is the road most advantageous to them.

Also upon reflection, it's a bit interesting that forking requests is
being dismissed so quickly, when in essence, requests maintains a fork
of urllib3 in tree (albeit, one that is just a fork from the _releases_,
not from the whole project).

> >I don't think we're anywhere near there, but I wanted to make it clear
> >there _is_ a more extreme option.
> >
> >> If the 2nd option is chosen, then doing anything but supporting the
> >>fact that
> >> requests vendors urllib3 within the code that openstack writes is
> >>hurting the
> >> users who fetch these projects from PyPI because you don't agree with
> >>one of
> >> the choices that requests makes. By all means do conditional imports to
> >>lessen
> >> the impact that the choice requests has made (and the one that
> >>Openstack has
> >> made to use requests) on downstream distributors, but unconditionally
> >>importing
> >> from the top level urllib3 for use within requests is flat out wrong.
> >> 
> >> Obviously neither of these options excludes the choice to lean on
> >>requests to
> >> reverse this decision as well. However that is best done elsewhere as
> >>the
> >> person making that decision isn't a member of these mailing lists as
> >>far as
> >> I am aware.
> >> 
> >
> >To be clear, I think we should keep using requests. But we should lend
> >our influence upstream and explain that our users are required to deal
> >with this in a way that perhaps hasn't been considered or given the
> >appropriate priority.
> 
> It’s been considered several times. There have been multiple issues.
> There’s more than just the one you linked to. The decision is highly
> unlikely to change whether it’s coming from a group of people in OpenStack
> or another distribution package maintainer.
> 

Indeed, hence my thinking that forking requests might be in order. Even
if that fork is just a long

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 19, 2014, at 2:26 PM, Chmouel Boudjnah  wrote:
> 
> 
> On Fri, Sep 19, 2014 at 6:58 PM, Donald Stufft  > wrote:
> So you can remove all that code and just let requests/urllib3 handle it on 
> 3.2+, 2.7.9+ and for anything less than that either use conditional 
> dependencies to have glance client depend on pyOpenSSL, ndg-httpsclient, and 
> pyasn1 on Python 2.x, or let them be optional and if people want to disable 
> TLS compression in those versions they can install those versions themselves.
> 
> 
> we have that issue as well for swiftclient, see the great write-up from 
> stuart here :
> 
> https://answers.launchpad.net/swift/+question/196920 
> 
> 
> just removing it this and let hope that users uses bleeding edge python 
> (which they don't) is not going to work for us. and the pyOpenSSL way is very 
> unfriendly to the end-user as well.
> 
> 

Unfortunately those are the only options besides using a different TLS 
implementation besides pyOpenSSL all together.

Python 2.x standard library did not include the requisite nobs for configuring 
this, it wasn’t until Python 3.2+ that the ssl module in the standard library 
gained the ability to have these kinds of things applied to it. Python 2.7.9 
contains a back port of the 3.x ssl module to Python 2.7, so that’s the first 
time in the 2.x line that the standard library has the knobs to change these 
things.

The alternative to 3.2+ or 2.7.9+ is using an alternative TLS implementation, 
of which pyOpenSSL is by far the most popular (and it’s what glanceclient is 
using right now).

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Chmouel Boudjnah]

On Fri, Sep 19, 2014 at 6:58 PM, Donald Stufft  wrote:

> So you can remove all that code and just let requests/urllib3 handle it on
> 3.2+, 2.7.9+ and for anything less than that either use conditional
> dependencies to have glance client depend on pyOpenSSL, ndg-httpsclient,
> and pyasn1 on Python 2.x, or let them be optional and if people want to
> disable TLS compression in those versions they can install those versions
> themselves.



we have that issue as well for swiftclient, see the great write-up from
stuart here :

https://answers.launchpad.net/swift/+question/196920

just removing it this and let hope that users uses bleeding edge python
(which they don't) is not going to work for us. and the pyOpenSSL way is
very unfriendly to the end-user as well.

Chmouel
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 19, 2014, at 12:42 PM, Mark Washenberger 
>  wrote:
> 
> 
> 
> On Fri, Sep 19, 2014 at 8:59 AM, Donald Stufft  > wrote:
> 
>> On Sep 19, 2014, at 11:54 AM, Brant Knudson > > wrote:
>> 
>> 
>> I don't think anyone would be complaining if glanceclient didn't have the 
>> need to reach into and monkeypatch requests's connection pool manager[1]. Is 
>> there a way to tell requests to build the https connections differently 
>> without monkeypatching urllib3.poolmanager?
>> 
>> glanceclient's monkeypatching of the global variable here is dangerous since 
>> it will mess with the application and every other library if the application 
>> or another library uses glanceclient.
>> 
>> [1] 
>> http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/common/https.py#n75
>>  
>> 
>> 
> 
> Why does it need to use it’s own VerifiedHTTPSConnection class? Ironically
> reimplementing that is probably more dangerous for security than requests
> bundling urllib3 ;)
> 
> We supported the option to skip SSL compression since before adopting 
> requests (see 556082cd6632dbce52ccb67ace57410d61057d66), useful when 
> uploading already compressed images.
> 

Is that all it’s used for? Probably it’s sane to just delete it then.

On Python 3.2+, 2.7.9+ Python provides the APIs to do it in the stdlib and 
urllib3 (and thus requests) will remove TLS Compression by default.

Python 2.6, and 2.7.0-2.7.8 do not provide the APIs to do so, however on Python 
2.x if you install pyOpenSSL, ndg-httpsclient, and pyasn1 then it’ll also 
disable TLS compression (automatically if you use requests, you have to do an 
import + function call with raw urllib3).

So you can remove all that code and just let requests/urllib3 handle it on 
3.2+, 2.7.9+ and for anything less than that either use conditional 
dependencies to have glance client depend on pyOpenSSL, ndg-httpsclient, and 
pyasn1 on Python 2.x, or let them be optional and if people want to disable TLS 
compression in those versions they can install those versions themselves.

By the way, everything above holds true for SNI as well.

This seems like the best of both worlds, glance client isn’t importing stuff 
from the vendored requests.packages.*, people get TLS Compression disabled (by 
default or optional depending on the choice the project makes), and it no 
longer has to maintain it’s own copy of security sensitive code.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Mark Washenberger]

On Fri, Sep 19, 2014 at 8:59 AM, Donald Stufft  wrote:

>
> On Sep 19, 2014, at 11:54 AM, Brant Knudson  wrote:
>
>
> I don't think anyone would be complaining if glanceclient didn't have the
> need to reach into and monkeypatch requests's connection pool manager[1].
> Is there a way to tell requests to build the https connections differently
> without monkeypatching urllib3.poolmanager?
>
> glanceclient's monkeypatching of the global variable here is dangerous
> since it will mess with the application and every other library if the
> application or another library uses glanceclient.
>
> [1]
> http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/common/https.py#n75
>
>
> Why does it need to use it’s own VerifiedHTTPSConnection class? Ironically
> reimplementing that is probably more dangerous for security than requests
> bundling urllib3 ;)
>

We supported the option to skip SSL compression since before adopting
requests (see 556082cd6632dbce52ccb67ace57410d61057d66), useful when
uploading already compressed images.



>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 19, 2014, at 11:54 AM, Brant Knudson  wrote:
> 
> 
> I don't think anyone would be complaining if glanceclient didn't have the 
> need to reach into and monkeypatch requests's connection pool manager[1]. Is 
> there a way to tell requests to build the https connections differently 
> without monkeypatching urllib3.poolmanager?
> 
> glanceclient's monkeypatching of the global variable here is dangerous since 
> it will mess with the application and every other library if the application 
> or another library uses glanceclient.
> 
> [1] 
> http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/common/https.py#n75
>  
> 
> 

Why does it need to use it’s own VerifiedHTTPSConnection class? Ironically
reimplementing that is probably more dangerous for security than requests
bundling urllib3 ;)

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Brant Knudson]

I don't think anyone would be complaining if glanceclient didn't have the
need to reach into and monkeypatch requests's connection pool manager[1].
Is there a way to tell requests to build the https connections differently
without monkeypatching urllib3.poolmanager?

glanceclient's monkeypatching of the global variable here is dangerous
since it will mess with the application and every other library if the
application or another library uses glanceclient.

[1]
http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/common/https.py#n75

- Brant


On Fri, Sep 19, 2014 at 10:33 AM, Ian Cordasco 
wrote:

>
>
> On 9/19/14, 9:01 AM, "Jeremy Stanley"  wrote:
>
> >On 2014-09-18 14:35:10 + (+), Ian Cordasco wrote:
> >> Except that even OpenStack doesn’t pin requests because of how
> >> extraordinarily stable our API is.
> >[...]
> >
> >FWIW, I nearly capped it a few weeks ago with
> >https://review.openstack.org/117848 but since the affected projects
> >were able to rush in changes to their use of the library to work
> >around the ways it was breaking I ended up abandoning that. Also for
> >some months we capped requests in our global requirements because of
> >https://launchpad.net/bugs/1182271 but that was finally lifted about
> >a year ago with https://review.openstack.org/37461 (so I don't think
> >it's entirely fair to assert that "OpenStack doesn’t pin requests
> >because...extraordinarily stable").
> >--
> >Jeremy Stanley
>
> A) Not the thread for this discussion.
> B) I didn’t say that OpenStack *never* pinned it, I said it didn’t
> currently
> C) Did you read the whole thread? I mentioned 2.4.0 as an exception
> because of ProtocolErrors and the redirect_cache members of OpenStack
> lobbied for.
> D) Before someone else replies, I assumed the transition from 0.x -> 1.0
> was also the other obvious (and not worth mentioning) break in stability
> given that since then we’ve had no API changes (with the exception of
> 2.4.0 not re-wrapping a single urllib3 exception).
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

On 9/19/14, 9:01 AM, "Jeremy Stanley"  wrote:

>On 2014-09-18 14:35:10 + (+), Ian Cordasco wrote:
>> Except that even OpenStack doesn’t pin requests because of how
>> extraordinarily stable our API is.
>[...]
>
>FWIW, I nearly capped it a few weeks ago with
>https://review.openstack.org/117848 but since the affected projects
>were able to rush in changes to their use of the library to work
>around the ways it was breaking I ended up abandoning that. Also for
>some months we capped requests in our global requirements because of
>https://launchpad.net/bugs/1182271 but that was finally lifted about
>a year ago with https://review.openstack.org/37461 (so I don't think
>it's entirely fair to assert that "OpenStack doesn’t pin requests
>because...extraordinarily stable").
>-- 
>Jeremy Stanley

A) Not the thread for this discussion.
B) I didn’t say that OpenStack *never* pinned it, I said it didn’t
currently
C) Did you read the whole thread? I mentioned 2.4.0 as an exception
because of ProtocolErrors and the redirect_cache members of OpenStack
lobbied for.
D) Before someone else replies, I assumed the transition from 0.x -> 1.0
was also the other obvious (and not worth mentioning) break in stability
given that since then we’ve had no API changes (with the exception of
2.4.0 not re-wrapping a single urllib3 exception).

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Jeremy Stanley]

On 2014-09-18 14:35:10 + (+), Ian Cordasco wrote:
> Except that even OpenStack doesn’t pin requests because of how
> extraordinarily stable our API is.
[...]

FWIW, I nearly capped it a few weeks ago with
https://review.openstack.org/117848 but since the affected projects
were able to rush in changes to their use of the library to work
around the ways it was breaking I ended up abandoning that. Also for
some months we capped requests in our global requirements because of
https://launchpad.net/bugs/1182271 but that was finally lifted about
a year ago with https://review.openstack.org/37461 (so I don't think
it's entirely fair to assert that "OpenStack doesn’t pin requests
because...extraordinarily stable").
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

On 9/18/14, 11:29 AM, "Clint Byrum"  wrote:

>Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700:
>> 
>> > On Sep 18, 2014, at 10:18 AM, Clint Byrum  wrote:
>> > 
>> > Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
>> >> 
>> >>> On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
>> >>> 
>>  
>>  Linux distributions are not the end be all of distribution models
>>and
>>  they don’t get to dictate to upstream.
>> >>> 
>> >>> Well, distributions is where the final user is, and where software
>>gets
>> >>> consumed. Our priority should be the end users.
>> >> 
>> >> 
>> >> Distributions are not the only place that people get their software
>>from,
>> >> unless you think that the ~3 million downloads requests has received
>> >> on PyPI in the last 30 days are distributions downloading requests to
>> >> package in their OSs.
>> >> 
>> > 
>> > Do pypi users not also need to be able to detect and fix any versions
>> > of libraries they might have? If one has some virtualenvs with various
>> > libraries and apps installed and no --system-site-packages, one would
>> > probably still want to run 'pip freeze' in all of them and find out
>>what
>> > libraries are there and need to be fixed.
>> > 
>> > Anyway, generally security updates require a comprehensive strategy.
>> > One common comprehensive strategy is version assertion.
>> > 
>> > Vendoring complicates that immensely.
>> 
>> It doesn’t really matter. PyPI doesn’t dictate to projects who host
>>there what
>> that project is allowed to do except in some very broad circumstances.
>>Whether
>> or not requests *should* do this doesn't really have any bearing on what
>> Openstack should do to cope with it. The facts are that requests does
>>it, and
>> that people pulling things from PyPI is an actual platform that needs
>>thought
>> about.
>> 
>> This leaves Openstack with a few reasonable/sane options:
>> 
>> 1) Decide that vendoring in requests is unacceptable to what Openstack
>>as a
>>project is willing to support, and cease the use of requests.
>> 2) Decide that what requests offers is good enough that it outweighs
>>the fact
>>that it vendors urllib3 and continue using it.
>> 
>
>There's also 3) fork requests, which is the democratic way to vote out
>an upstream that isn't supporting the needs of the masses.

Given requests’ download count, I have to doubt that OpenStack users
constitute the masses in this case.

>I don't think we're anywhere near there, but I wanted to make it clear
>there _is_ a more extreme option.
>
>> If the 2nd option is chosen, then doing anything but supporting the
>>fact that
>> requests vendors urllib3 within the code that openstack writes is
>>hurting the
>> users who fetch these projects from PyPI because you don't agree with
>>one of
>> the choices that requests makes. By all means do conditional imports to
>>lessen
>> the impact that the choice requests has made (and the one that
>>Openstack has
>> made to use requests) on downstream distributors, but unconditionally
>>importing
>> from the top level urllib3 for use within requests is flat out wrong.
>> 
>> Obviously neither of these options excludes the choice to lean on
>>requests to
>> reverse this decision as well. However that is best done elsewhere as
>>the
>> person making that decision isn't a member of these mailing lists as
>>far as
>> I am aware.
>> 
>
>To be clear, I think we should keep using requests. But we should lend
>our influence upstream and explain that our users are required to deal
>with this in a way that perhaps hasn't been considered or given the
>appropriate priority.

It’s been considered several times. There have been multiple issues.
There’s more than just the one you linked to. The decision is highly
unlikely to change whether it’s coming from a group of people in OpenStack
or another distribution package maintainer.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Clint Byrum]

Excerpts from Ian Cordasco's message of 2014-09-18 07:35:10 -0700:
> On 9/18/14, 9:18 AM, "Clint Byrum"  wrote:
> 
> >Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
> >> 
> >> > On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
> >> > 
> >> >> 
> >> >> Linux distributions are not the end be all of distribution models and
> >> >> they don’t get to dictate to upstream.
> >> > 
> >> > Well, distributions is where the final user is, and where software
> >>gets
> >> > consumed. Our priority should be the end users.
> >> 
> >> 
> >> Distributions are not the only place that people get their software
> >>from,
> >> unless you think that the ~3 million downloads requests has received
> >> on PyPI in the last 30 days are distributions downloading requests to
> >> package in their OSs.
> >> 
> >
> >Do pypi users not also need to be able to detect and fix any versions
> >of libraries they might have? If one has some virtualenvs with various
> >libraries and apps installed and no --system-site-packages, one would
> >probably still want to run 'pip freeze' in all of them and find out what
> >libraries are there and need to be fixed.
> >
> >Anyway, generally security updates require a comprehensive strategy.
> >One common comprehensive strategy is version assertion.
> >
> >Vendoring complicates that immensely.
> 
> Except that even OpenStack doesn’t pin requests because of how
> extraordinarily stable our API is. While you can argue that Kenneth has
> non-standard opinions about his library, Cory and I take backwards
> compatibility and stability very seriously. This means anyone can upgrade
> to a newer version of requests without worrying that it will be backwards
> incompatible. 
> 

All of your hard work is very much appreciated. I don't understand what
your assertion means though. We don't pin things. However, our users end
up "pinning" when they install via pip, and our distros end up "pinning"
when they deliver a version. Without any indication that urllib3 is in
the system, they will fail at any cursory version audit that looks for it.

I'm not saying either way is right or wrong either.. I'm suggesting
that this is a valid, proven method for large scale risk assessment,
and it is complicated quite a bit by vendored libraries.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 18, 2014, at 12:29 PM, Clint Byrum  wrote:
> 
> Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700:
>> 
>>> On Sep 18, 2014, at 10:18 AM, Clint Byrum  wrote:
>>> 
>>> Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
 
> On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
> 
>> 
>> Linux distributions are not the end be all of distribution models and
>> they don’t get to dictate to upstream.
> 
> Well, distributions is where the final user is, and where software gets
> consumed. Our priority should be the end users.
 
 
 Distributions are not the only place that people get their software from,
 unless you think that the ~3 million downloads requests has received
 on PyPI in the last 30 days are distributions downloading requests to
 package in their OSs.
 
>>> 
>>> Do pypi users not also need to be able to detect and fix any versions
>>> of libraries they might have? If one has some virtualenvs with various
>>> libraries and apps installed and no --system-site-packages, one would
>>> probably still want to run 'pip freeze' in all of them and find out what
>>> libraries are there and need to be fixed.
>>> 
>>> Anyway, generally security updates require a comprehensive strategy.
>>> One common comprehensive strategy is version assertion.
>>> 
>>> Vendoring complicates that immensely.
>> 
>> It doesn’t really matter. PyPI doesn’t dictate to projects who host there 
>> what
>> that project is allowed to do except in some very broad circumstances. 
>> Whether
>> or not requests *should* do this doesn't really have any bearing on what
>> Openstack should do to cope with it. The facts are that requests does it, and
>> that people pulling things from PyPI is an actual platform that needs thought
>> about.
>> 
>> This leaves Openstack with a few reasonable/sane options:
>> 
>> 1) Decide that vendoring in requests is unacceptable to what Openstack as a
>>   project is willing to support, and cease the use of requests.
>> 2) Decide that what requests offers is good enough that it outweighs the fact
>>   that it vendors urllib3 and continue using it.
>> 
> 
> There's also 3) fork requests, which is the democratic way to vote out
> an upstream that isn't supporting the needs of the masses.
> 
> I don't think we're anywhere near there, but I wanted to make it clear
> there _is_ a more extreme option.

Technically that’s just a specific case of option 1) ;)

But yes that’s a thing Openstack can do.

> 
>> If the 2nd option is chosen, then doing anything but supporting the fact that
>> requests vendors urllib3 within the code that openstack writes is hurting the
>> users who fetch these projects from PyPI because you don't agree with one of
>> the choices that requests makes. By all means do conditional imports to 
>> lessen
>> the impact that the choice requests has made (and the one that Openstack has
>> made to use requests) on downstream distributors, but unconditionally 
>> importing
>> from the top level urllib3 for use within requests is flat out wrong.
>> 
>> Obviously neither of these options excludes the choice to lean on requests to
>> reverse this decision as well. However that is best done elsewhere as the
>> person making that decision isn't a member of these mailing lists as far as
>> I am aware.
>> 
> 
> To be clear, I think we should keep using requests. But we should lend
> our influence upstream and explain that our users are required to deal
> with this in a way that perhaps hasn't been considered or given the
> appropriate priority.

I think that’s completely reasonable. I don’t think there’s going to be much 
movement,
I’ve had this argument with Kenneth on more than one occasion and he’s very 
happy
with his decision to vendor urllib3, but hey maybe Openstack would have better 
luck.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Clint Byrum]

Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700:
> 
> > On Sep 18, 2014, at 10:18 AM, Clint Byrum  wrote:
> > 
> > Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
> >> 
> >>> On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
> >>> 
>  
>  Linux distributions are not the end be all of distribution models and
>  they don’t get to dictate to upstream.
> >>> 
> >>> Well, distributions is where the final user is, and where software gets
> >>> consumed. Our priority should be the end users.
> >> 
> >> 
> >> Distributions are not the only place that people get their software from,
> >> unless you think that the ~3 million downloads requests has received
> >> on PyPI in the last 30 days are distributions downloading requests to
> >> package in their OSs.
> >> 
> > 
> > Do pypi users not also need to be able to detect and fix any versions
> > of libraries they might have? If one has some virtualenvs with various
> > libraries and apps installed and no --system-site-packages, one would
> > probably still want to run 'pip freeze' in all of them and find out what
> > libraries are there and need to be fixed.
> > 
> > Anyway, generally security updates require a comprehensive strategy.
> > One common comprehensive strategy is version assertion.
> > 
> > Vendoring complicates that immensely.
> 
> It doesn’t really matter. PyPI doesn’t dictate to projects who host there what
> that project is allowed to do except in some very broad circumstances. Whether
> or not requests *should* do this doesn't really have any bearing on what
> Openstack should do to cope with it. The facts are that requests does it, and
> that people pulling things from PyPI is an actual platform that needs thought
> about.
> 
> This leaves Openstack with a few reasonable/sane options:
> 
> 1) Decide that vendoring in requests is unacceptable to what Openstack as a
>project is willing to support, and cease the use of requests.
> 2) Decide that what requests offers is good enough that it outweighs the fact
>that it vendors urllib3 and continue using it.
> 

There's also 3) fork requests, which is the democratic way to vote out
an upstream that isn't supporting the needs of the masses.

I don't think we're anywhere near there, but I wanted to make it clear
there _is_ a more extreme option.

> If the 2nd option is chosen, then doing anything but supporting the fact that
> requests vendors urllib3 within the code that openstack writes is hurting the
> users who fetch these projects from PyPI because you don't agree with one of
> the choices that requests makes. By all means do conditional imports to lessen
> the impact that the choice requests has made (and the one that Openstack has
> made to use requests) on downstream distributors, but unconditionally 
> importing
> from the top level urllib3 for use within requests is flat out wrong.
> 
> Obviously neither of these options excludes the choice to lean on requests to
> reverse this decision as well. However that is best done elsewhere as the
> person making that decision isn't a member of these mailing lists as far as
> I am aware.
> 

To be clear, I think we should keep using requests. But we should lend
our influence upstream and explain that our users are required to deal
with this in a way that perhaps hasn't been considered or given the
appropriate priority.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

On 9/18/14, 9:18 AM, "Clint Byrum"  wrote:

>Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
>> 
>> > On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
>> > 
>> >> 
>> >> Linux distributions are not the end be all of distribution models and
>> >> they don’t get to dictate to upstream.
>> > 
>> > Well, distributions is where the final user is, and where software
>>gets
>> > consumed. Our priority should be the end users.
>> 
>> 
>> Distributions are not the only place that people get their software
>>from,
>> unless you think that the ~3 million downloads requests has received
>> on PyPI in the last 30 days are distributions downloading requests to
>> package in their OSs.
>> 
>
>Do pypi users not also need to be able to detect and fix any versions
>of libraries they might have? If one has some virtualenvs with various
>libraries and apps installed and no --system-site-packages, one would
>probably still want to run 'pip freeze' in all of them and find out what
>libraries are there and need to be fixed.
>
>Anyway, generally security updates require a comprehensive strategy.
>One common comprehensive strategy is version assertion.
>
>Vendoring complicates that immensely.

Except that even OpenStack doesn’t pin requests because of how
extraordinarily stable our API is. While you can argue that Kenneth has
non-standard opinions about his library, Cory and I take backwards
compatibility and stability very seriously. This means anyone can upgrade
to a newer version of requests without worrying that it will be backwards
incompatible. 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 18, 2014, at 10:18 AM, Clint Byrum  wrote:
> 
> Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
>> 
>>> On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
>>> 
 
 Linux distributions are not the end be all of distribution models and
 they don’t get to dictate to upstream.
>>> 
>>> Well, distributions is where the final user is, and where software gets
>>> consumed. Our priority should be the end users.
>> 
>> 
>> Distributions are not the only place that people get their software from,
>> unless you think that the ~3 million downloads requests has received
>> on PyPI in the last 30 days are distributions downloading requests to
>> package in their OSs.
>> 
> 
> Do pypi users not also need to be able to detect and fix any versions
> of libraries they might have? If one has some virtualenvs with various
> libraries and apps installed and no --system-site-packages, one would
> probably still want to run 'pip freeze' in all of them and find out what
> libraries are there and need to be fixed.
> 
> Anyway, generally security updates require a comprehensive strategy.
> One common comprehensive strategy is version assertion.
> 
> Vendoring complicates that immensely.

It doesn’t really matter. PyPI doesn’t dictate to projects who host there what
that project is allowed to do except in some very broad circumstances. Whether
or not requests *should* do this doesn't really have any bearing on what
Openstack should do to cope with it. The facts are that requests does it, and
that people pulling things from PyPI is an actual platform that needs thought
about.

This leaves Openstack with a few reasonable/sane options:

1) Decide that vendoring in requests is unacceptable to what Openstack as a
   project is willing to support, and cease the use of requests.
2) Decide that what requests offers is good enough that it outweighs the fact
   that it vendors urllib3 and continue using it.

If the 2nd option is chosen, then doing anything but supporting the fact that
requests vendors urllib3 within the code that openstack writes is hurting the
users who fetch these projects from PyPI because you don't agree with one of
the choices that requests makes. By all means do conditional imports to lessen
the impact that the choice requests has made (and the one that Openstack has
made to use requests) on downstream distributors, but unconditionally importing
from the top level urllib3 for use within requests is flat out wrong.

Obviously neither of these options excludes the choice to lean on requests to
reverse this decision as well. However that is best done elsewhere as the
person making that decision isn't a member of these mailing lists as far as
I am aware.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

On 9/18/14, 2:27 AM, "Chmouel Boudjnah"  wrote:

>Ian Cordasco  writes:
>
>> urllib3 do that automatically. I haven’t started to investigate exactly
>> why they do this. Likewise, glance client has custom certificate
>> verification in glanceclient.common.https. Why? I’m not exactly certain
>
>this probably come from pre-requests port uses when it was using httplib
>which didn't have SSL verification. There is a old wiki page about it
>here https://wiki.openstack.org/wiki/SecureClientConnections
>
>Slightly off-topic, speaking about requests and OpenStack client, it
>would be nice to have Expect/100-Continue feature tackled for
>glanceclient and swiftclient :
>
>https://github.com/kennethreitz/requests/issues/713

I’m glad you linked to that issue Chmouel because it has all of the
information you need to realize that it’s an entirely impossible feature
while still relying on httplib. You can still send that header, but
requests has no meaningful way of special-casing it. The standard library
does not return the header we need to know that we can continue with the
upload and the RFC is wonderfully obscure enough to make requests’ current
behavior correct. It says a client needs to wait for the server to respond
or some amount of time before starting the upload. Granted we don’t call
“sleep” and start the upload immediately, but we have no way of
determining if the server has responded thanks to httplib.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

On 9/17/14, 9:24 PM, "Thomas Goirand"  wrote:

>On 09/18/2014 08:22 AM, Morgan Fainberg wrote:
>> I think that all of the conversation to this point has been valuable,
>> the general consensus is vendoring a library is not as desirable as
>> using it strictly as a dependency. It would be nice in a perfect
>> world if vendoring wasn’t and issue, but in this case I think the
>> root of the matter is that Debian un-vendors urllib3 and we have
>> referenced the vendored urllib3 instead of installing and utilizing
>> urllib3 directly.
>> 
>> This poses at least one problem for us: we are not able to guarantee
>> we’re using the same urllib3 library as requests is. I am unsure how
>> big of a deal this ends up being, but it is a concern and has brought
>> up a question of how to handle this in the most appropriate and
>> consistent way across all of the distributions we as OpenStack support.
>> 
>> Does this make requests a bad library we should toss aside for
>> something else? Instead of being concerned with the reasons for
>> vendoring urllib3 (or un-vendoring it) we should shift the conversation
>> towards two questions:
>> 
>> 1. Is it a real issue if the version of urllib3 is mismatched between
>> our client libraries and requests?
>> 2. If it is a real issue how are we solving it?
>
>The main issue is that urllib3 in requests, as other pointed out, is not
>up-to-date, and will not be updated. In fact, that's the main reason why
>the upstream authors of requests are vendorizing: it's because they
>don't want to carry the burden of staying up-to-date.

How involved are you with requests’ development process? You must not be
very involved because this is the exact opposite reason of why we vendor.
More often that not we pull in urllib3 to get unreleased features that we
build upon for our newest release. If what you said was true, 2.4.{0,1}
would not have the ability to pass socket level options that nova client
decides it wants to set. If we were pinning to an old version of urllib3,
this feature would never be possible. We vendor, because in order to
provide these features we don’t want to have to support the ancient
versions of urllib3 that used to cause us headaches on platforms like
Debian.

>And then, there's incompatibilities and divergences that appear, leading
>to all sorts of unexpected issues, like one thing working with pip, but
>not with the packages. This kind of issues are very hard to understand
>and debug. Distributions may report the issue upstream, then upstream
>will say "but it's working for me", and then we may loose a lot of time.
>This happened already, and may happen again if we don't care enough.
>
>> Obviously we can work with the requests team to figure out the best
>> approach.
>
>There's only a single approach that works: have the requests upstream
>authors to stop embedding foreign code, and use the dependency instead.
>
>> We should focus on the solution here rather than continuing
>> down the path of whether requests should/shouldn’t be vendoring it’s
>> dependencies since it is clear that the team has their reasons and
>> does not want to switch to the dependency model again.
>
>I'm sure they have tons of wrong reasons. If they don't want to change
>anything, then we can only try to work around the issue, and never use
>the embedded version.

The fact is, anyone who doesn’t run their tests in Devstack runs them in a
virtual environment. The line you’re complaining about is actually correct
in that context (the context where pip installs requests). That said, the
test should instead use a conditional import and fallback to the vendored
copy.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Clint Byrum]

Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
> 
> > On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
> > 
> >> 
> >> Linux distributions are not the end be all of distribution models and
> >> they don’t get to dictate to upstream.
> > 
> > Well, distributions is where the final user is, and where software gets
> > consumed. Our priority should be the end users.
> 
> 
> Distributions are not the only place that people get their software from,
> unless you think that the ~3 million downloads requests has received
> on PyPI in the last 30 days are distributions downloading requests to
> package in their OSs.
> 

Do pypi users not also need to be able to detect and fix any versions
of libraries they might have? If one has some virtualenvs with various
libraries and apps installed and no --system-site-packages, one would
probably still want to run 'pip freeze' in all of them and find out what
libraries are there and need to be fixed.

Anyway, generally security updates require a comprehensive strategy.
One common comprehensive strategy is version assertion.

Vendoring complicates that immensely.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 18, 2014, at 9:00 AM, Chmouel Boudjnah  wrote:
> 
> 
> On Thu, Sep 18, 2014 at 1:58 PM, Donald Stufft  > wrote:
> Distributions are not the only place that people get their software from,
> unless you think that the ~3 million downloads requests has received
> on PyPI in the last 30 days are distributions downloading requests to
> package in their OSs.
> 
> 
> I think Thomas was speaking in the context of how OpenStack is used by the 
> end user and that probably the point of debate here, requests ships libraries 
> inside to make it easy for users that doen't use Linux distro packages, when 
> in OpenStack (or at least in prod) packagers are something we generally very 
> much care about.

Even then, my statement holds true, just with different numbers.

Every distribution modifies upstream in different ways, I think it's insane to
do contortions which will break things for people *not* getting things through
those channels. If distributions are going to modify one upstream project they
should expect to need to modify things that depend on that project in ways that
are sensitive to what they've modified.

The only real sane thing IMO is for openstack to consider requests as it is on
PyPI. If openstack wants to make it easier for downstream to de-vendor urllib3
from requests then when openstack wants to import from requests.packages.* it
can instead do:

try:
from requests.packages import urllib3
except ImportError:
import urllib3

This will cause it to work correctly when requests is installed in a pristine
state, and will fallback to handling the modifications that some downstream
redistributors make.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Chmouel Boudjnah]

On Thu, Sep 18, 2014 at 1:58 PM, Donald Stufft  wrote:

> Distributions are not the only place that people get their software from,
> unless you think that the ~3 million downloads requests has received
> on PyPI in the last 30 days are distributions downloading requests to
> package in their OSs.
>


I think Thomas was speaking in the context of how OpenStack is used by the
end user and that probably the point of debate here, requests ships
libraries inside to make it easy for users that doen't use Linux distro
packages, when in OpenStack (or at least in prod) packagers are something
we generally very much care about.

Chmouel
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
> 
>> 
>> Linux distributions are not the end be all of distribution models and
>> they don’t get to dictate to upstream.
> 
> Well, distributions is where the final user is, and where software gets
> consumed. Our priority should be the end users.


Distributions are not the only place that people get their software from,
unless you think that the ~3 million downloads requests has received
on PyPI in the last 30 days are distributions downloading requests to
package in their OSs.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Thomas Goirand]

On 09/18/2014 10:43 AM, Donald Stufft wrote:
>>> Obviously we can work with the requests team to figure out the best
>>> approach.
>>
>> There's only a single approach that works: have the requests upstream
>> authors to stop embedding foreign code, and use the dependency instead.
> 
> There are legitimate reasons for a project to vendor things.

Yes, there's lot of reasons. But so fare, I haven't read about any valid
one.

> Linux distributions are not the end be all of distribution models and
> they don’t get to dictate to upstream.

Well, distributions is where the final user is, and where software gets
consumed. Our priority should be the end users.

> Generally I agree that requests should not vendor urllib3

Unfortunately, it doesn't seem requests upstream agree, so we can only
deal with the issue. This means not using requests.packages.*.

>   You’re going to get very strange incompatibility problems
> if you try to mis requests.packages.urllib3 and urllib3 in one codebase
> and if you’re using requests at all it’s going to be expecting to use
> the embedded copy of urllib3.

I'm well aware of this. As I wrote, I already had to deal with issues
like that, and I'm expecting even more in the future.

Thomas Goirand (zigo)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 18, 2014, at 7:43 AM, Thomas Goirand  wrote:
> 
> On 09/18/2014 04:01 PM, Flavio Percoco wrote:
>> After having gone through the whole thread and read all the concerns,
>> problems and reasonings, I think we should stick to requests as-is for
>> now and deal with this particular issue.
>> 
>> Regardless of the vendorized urllib3 package, I believe requests is a
>> good library with an easy-to-consume API and it has solved several
>> problems throughout OpenStack. Not to mention it's also helpped with
>> making OpenStack more consistent. We've put a lot of effort to get to
>> this point and I don't think we should revert all that because of the
>> vendorized `urllib3` package.
>> 
>> Cheers,
>> Flavio
> 
> I, at least, haven't suggested that we stop using requests. Just that we
> don't internally use any of the requests.packages.* stuff.
> 
> The rest of the debate about the good or bad things with vendorizing,
> even if it is my view that it's a really bad thing, is IMO not
> interesting for the OpenStack project.

I don’t believe that’s a good idea. If you’re wanting to use urllib3 in order
to interact with requests than you *should* be using requests.packages.urllib3,
to do anything else risks having two different versions of urllib3 primitives at
play in one subsystem.

It’s not even completely possible in the case that prompted this thread 
originally
since the reason requests.packages.urllib3 was being imported from was so that
there could be an is instance() check against one of the classes. If that wasn’t
imported from requests.packages.urllib3 but instead from just urllib3 than the
isinstance check would always fail.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Thomas Goirand]

On 09/18/2014 04:01 PM, Flavio Percoco wrote:
> After having gone through the whole thread and read all the concerns,
> problems and reasonings, I think we should stick to requests as-is for
> now and deal with this particular issue.
> 
> Regardless of the vendorized urllib3 package, I believe requests is a
> good library with an easy-to-consume API and it has solved several
> problems throughout OpenStack. Not to mention it's also helpped with
> making OpenStack more consistent. We've put a lot of effort to get to
> this point and I don't think we should revert all that because of the
> vendorized `urllib3` package.
> 
> Cheers,
> Flavio

I, at least, haven't suggested that we stop using requests. Just that we
don't internally use any of the requests.packages.* stuff.

The rest of the debate about the good or bad things with vendorizing,
even if it is my view that it's a really bad thing, is IMO not
interesting for the OpenStack project.

Thomas Goirand (zigo)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Alan Pevec]

2014-09-17 17:15 GMT+02:00 Thomas Goirand :
>   File "/tests/test_ssl.py", line 19, in 
> from requests.packages.urllib3 import poolmanager
> ImportError: No module named packages.urllib3

This is in tests only, in runtime code there is conditional import of
vendorized urllib3 falling back to system library.
So what about https://review.openstack.org/122379

Cheers,
Alan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Flavio Percoco]

On 09/18/2014 04:43 AM, Donald Stufft wrote:
> 
>> On Sep 17, 2014, at 10:24 PM, Thomas Goirand > > wrote:
>>
>> On 09/18/2014 08:22 AM, Morgan Fainberg wrote:
>>> I think that all of the conversation to this point has been valuable,
>>> the general consensus is vendoring a library is not as desirable as
>>> using it strictly as a dependency. It would be nice in a perfect
>>> world if vendoring wasn’t and issue, but in this case I think the
>>> root of the matter is that Debian un-vendors urllib3 and we have
>>> referenced the vendored urllib3 instead of installing and utilizing
>>> urllib3 directly.
>>>
>>> This poses at least one problem for us: we are not able to guarantee
>>> we’re using the same urllib3 library as requests is. I am unsure how
>>> big of a deal this ends up being, but it is a concern and has brought
>>> up a question of how to handle this in the most appropriate and
>>> consistent way across all of the distributions we as OpenStack support.
>>>
>>> Does this make requests a bad library we should toss aside for
>>> something else? Instead of being concerned with the reasons for
>>> vendoring urllib3 (or un-vendoring it) we should shift the conversation
>>> towards two questions:
>>>
>>> 1. Is it a real issue if the version of urllib3 is mismatched between
>>> our client libraries and requests?
>>> 2. If it is a real issue how are we solving it?
>>
>> The main issue is that urllib3 in requests, as other pointed out, is not
>> up-to-date, and will not be updated. In fact, that's the main reason why
>> the upstream authors of requests are vendorizing: it's because they
>> don't want to carry the burden of staying up-to-date.
> 
> I don’t think this is remotely true, often times requests updates itself
> to versions of urllib3 which aren’t even released yet. Sometimes urllib3
> might make commits and do a release that happens between requests
> versions though. I mean technically they might be not up to date until
> their next version release though.
> 
>>
>> And then, there's incompatibilities and divergences that appear, leading
>> to all sorts of unexpected issues, like one thing working with pip, but
>> not with the packages. This kind of issues are very hard to understand
>> and debug. Distributions may report the issue upstream, then upstream
>> will say "but it's working for me", and then we may loose a lot of time.
>> This happened already, and may happen again if we don't care enough.
> 
> I think this is bound to happen anytime you have downstream modifying
> things. It happens in pip (pip vendors things too) and yea it’s quite
> annoying
> but part of PEP 440 is providing ways for downstream to signal they’ve
> modified things so that instead of “foo 1.0” you have “foo 1.0+ubuntu1” or
> whatever.
> 
>>
>>> Obviously we can work with the requests team to figure out the best
>>> approach.
>>
>> There's only a single approach that works: have the requests upstream
>> authors to stop embedding foreign code, and use the dependency instead.
> 
> There are legitimate reasons for a project to vendor things. Linux
> distributions
> are not the end be all of distribution models and they don’t get to
> dictate to
> upstream.
> 
> Generally I agree that requests should not vendor urllib3, but it’s not
> a clear
> cut thing where there is one right way to do it.
> 
>>
>>> We should focus on the solution here rather than continuing
>>> down the path of whether requests should/shouldn’t be vendoring it’s
>>> dependencies since it is clear that the team has their reasons and
>>> does not want to switch to the dependency model again.
>>
>> I'm sure they have tons of wrong reasons. If they don't want to change
>> anything, then we can only try to work around the issue, and never use
>> the embedded version.
> 
> Generally you either work with the embedded versions or you don’t
> use requests. You’re going to get very strange incompatibility problems
> if you try to mis requests.packages.urllib3 and urllib3 in one codebase
> and if you’re using requests at all it’s going to be expecting to use
> the embedded copy of urllib3.


After having gone through the whole thread and read all the concerns,
problems and reasonings, I think we should stick to requests as-is for
now and deal with this particular issue.

Regardless of the vendorized urllib3 package, I believe requests is a
good library with an easy-to-consume API and it has solved several
problems throughout OpenStack. Not to mention it's also helpped with
making OpenStack more consistent. We've put a lot of effort to get to
this point and I don't think we should revert all that because of the
vendorized `urllib3` package.

Cheers,
Flavio


-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Chmouel Boudjnah]

Ian Cordasco  writes:

> urllib3 do that automatically. I haven’t started to investigate exactly
> why they do this. Likewise, glance client has custom certificate
> verification in glanceclient.common.https. Why? I’m not exactly certain

this probably come from pre-requests port uses when it was using httplib
which didn't have SSL verification. There is a old wiki page about it
here https://wiki.openstack.org/wiki/SecureClientConnections

Slightly off-topic, speaking about requests and OpenStack client, it
would be nice to have Expect/100-Continue feature tackled for
glanceclient and swiftclient :

https://github.com/kennethreitz/requests/issues/713

Chmouel

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 17, 2014, at 10:24 PM, Thomas Goirand  wrote:
> 
> On 09/18/2014 08:22 AM, Morgan Fainberg wrote:
>> I think that all of the conversation to this point has been valuable,
>> the general consensus is vendoring a library is not as desirable as
>> using it strictly as a dependency. It would be nice in a perfect
>> world if vendoring wasn’t and issue, but in this case I think the
>> root of the matter is that Debian un-vendors urllib3 and we have
>> referenced the vendored urllib3 instead of installing and utilizing
>> urllib3 directly.
>> 
>> This poses at least one problem for us: we are not able to guarantee
>> we’re using the same urllib3 library as requests is. I am unsure how
>> big of a deal this ends up being, but it is a concern and has brought
>> up a question of how to handle this in the most appropriate and
>> consistent way across all of the distributions we as OpenStack support. 
>> 
>> Does this make requests a bad library we should toss aside for
>> something else? Instead of being concerned with the reasons for
>> vendoring urllib3 (or un-vendoring it) we should shift the conversation
>> towards two questions:
>> 
>> 1. Is it a real issue if the version of urllib3 is mismatched between
>> our client libraries and requests? 
>> 2. If it is a real issue how are we solving it?
> 
> The main issue is that urllib3 in requests, as other pointed out, is not
> up-to-date, and will not be updated. In fact, that's the main reason why
> the upstream authors of requests are vendorizing: it's because they
> don't want to carry the burden of staying up-to-date.

I don’t think this is remotely true, often times requests updates itself
to versions of urllib3 which aren’t even released yet. Sometimes urllib3
might make commits and do a release that happens between requests
versions though. I mean technically they might be not up to date until
their next version release though.

> 
> And then, there's incompatibilities and divergences that appear, leading
> to all sorts of unexpected issues, like one thing working with pip, but
> not with the packages. This kind of issues are very hard to understand
> and debug. Distributions may report the issue upstream, then upstream
> will say "but it's working for me", and then we may loose a lot of time.
> This happened already, and may happen again if we don't care enough.

I think this is bound to happen anytime you have downstream modifying
things. It happens in pip (pip vendors things too) and yea it’s quite annoying
but part of PEP 440 is providing ways for downstream to signal they’ve
modified things so that instead of “foo 1.0” you have “foo 1.0+ubuntu1” or
whatever.

> 
>> Obviously we can work with the requests team to figure out the best
>> approach.
> 
> There's only a single approach that works: have the requests upstream
> authors to stop embedding foreign code, and use the dependency instead.

There are legitimate reasons for a project to vendor things. Linux distributions
are not the end be all of distribution models and they don’t get to dictate to
upstream.

Generally I agree that requests should not vendor urllib3, but it’s not a clear
cut thing where there is one right way to do it.

> 
>> We should focus on the solution here rather than continuing
>> down the path of whether requests should/shouldn’t be vendoring it’s
>> dependencies since it is clear that the team has their reasons and
>> does not want to switch to the dependency model again.
> 
> I'm sure they have tons of wrong reasons. If they don't want to change
> anything, then we can only try to work around the issue, and never use
> the embedded version.

Generally you either work with the embedded versions or you don’t
use requests. You’re going to get very strange incompatibility problems
if you try to mis requests.packages.urllib3 and urllib3 in one codebase
and if you’re using requests at all it’s going to be expecting to use
the embedded copy of urllib3.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Thomas Goirand]

On 09/18/2014 08:22 AM, Morgan Fainberg wrote:
> I think that all of the conversation to this point has been valuable,
> the general consensus is vendoring a library is not as desirable as
> using it strictly as a dependency. It would be nice in a perfect
> world if vendoring wasn’t and issue, but in this case I think the
> root of the matter is that Debian un-vendors urllib3 and we have
> referenced the vendored urllib3 instead of installing and utilizing
> urllib3 directly.
> 
> This poses at least one problem for us: we are not able to guarantee
> we’re using the same urllib3 library as requests is. I am unsure how
> big of a deal this ends up being, but it is a concern and has brought
> up a question of how to handle this in the most appropriate and
> consistent way across all of the distributions we as OpenStack support. 
> 
> Does this make requests a bad library we should toss aside for
> something else? Instead of being concerned with the reasons for
> vendoring urllib3 (or un-vendoring it) we should shift the conversation
> towards two questions:
> 
> 1. Is it a real issue if the version of urllib3 is mismatched between
> our client libraries and requests? 
> 2. If it is a real issue how are we solving it?

The main issue is that urllib3 in requests, as other pointed out, is not
up-to-date, and will not be updated. In fact, that's the main reason why
the upstream authors of requests are vendorizing: it's because they
don't want to carry the burden of staying up-to-date.

And then, there's incompatibilities and divergences that appear, leading
to all sorts of unexpected issues, like one thing working with pip, but
not with the packages. This kind of issues are very hard to understand
and debug. Distributions may report the issue upstream, then upstream
will say "but it's working for me", and then we may loose a lot of time.
This happened already, and may happen again if we don't care enough.

> Obviously we can work with the requests team to figure out the best
> approach.

There's only a single approach that works: have the requests upstream
authors to stop embedding foreign code, and use the dependency instead.

> We should focus on the solution here rather than continuing
> down the path of whether requests should/shouldn’t be vendoring it’s
> dependencies since it is clear that the team has their reasons and
> does not want to switch to the dependency model again.

I'm sure they have tons of wrong reasons. If they don't want to change
anything, then we can only try to work around the issue, and never use
the embedded version.

Thomas Goirand (zigo)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Morgan Fainberg]

-Original Message-
From: Ian Cordasco 
Reply: OpenStack Development Mailing List (not for usage questions) 
>
Date: September 17, 2014 at 16:28:57
To: OpenStack Development Mailing List (not for usage questions) 
>
Subject:  Re: [openstack-dev] Please do *NOT* use "vendorized" versions of 
anything (here: glanceclient using requests.packages.urllib3)

> On 9/17/14, 5:39 PM, "Mike Bayer" wrote:
>  
> >
> >On Sep 17, 2014, at 4:31 PM, Ian Cordasco  
> >wrote:
> >
> >> Project X pins a version of requests. Alice doesn’t know anything about
> >> requests and does pip install X. Until Alice takes a more active role in
> >> the development of Project X and looks into requests, she will never
> >>know
> >> she’s installed software that has exposures in it.
> >
> >If a vulnerability is reported in urllib3 1.9.1, Alice, as well as me and
> >everyone else who is not a novice, will at least know we need to run:
> >
> >$ pip show urllib3
> >---
> >Name: urllib3
> >Version: 1.9.1
> >Location:
> >/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packa  
> >ges
> >Requires:
> >
> >
> >and we know right there we have to upgrade. We upgrade, and we’re done.
> > If we see that some library is pinning it, we will know. We will
> >complain loudly to that library’s author and/or replace that library.
> >The tools are there to give us what we need to be aware and to escalate
> >the problem.
>  
> And when the library is unmaintained, you’ll be yelling into the echo
> chamber. If it is popular enough, it’ll continue to be used regardless
> because every recommendation that will be highest ranked by :search_engine
> will tell people looking for related libraries to do so. In all
> likelihood, most users who stumble across these projects will also be
> ignorant of the fact that :some_dependency has a CVE and needs to be
> upgraded.
>  
> >When a library silently bundles the source code and bypasses any normal
> >means of us knowing it’s present unless we read the source code or scour
> >the documentation, we have no way to know we’re affected. Some
> >applications, particularly pip, have to do this, however, it should only
> >be for technical reasons. It should not be because you don’t want novice
> >users to have to learn something, or because you’re angling to have lots
> >of downloads on pypi.
>  
> It is for technical reasons, but this is not the appropriate place to
> discuss them. There are at least 3 other closed issues on requests’ issue
> tracker that discuss most (if not all) of them. This is the list to
> discuss the development of OpenStack. Finally, for the last time, the fact
> that we vendor these libraries exists in multiple places (beyond the
> source).
>  
> >>> People make sure to upgrade their Requests libraries locally, but for
> >>>all
> >>> those poor saps who have *no idea* they have widely used apps that are
> >>> bundling it silently, they remain totally open to vulnerabilities and
> >>>the
> >>> black hats have disneyland at their disposal.
> >>
> >> I think more applications bundle it than you realize. You’re likely
> >>using
> >> one daily that does it.
> >
> >
> >SQLAlchemy itself vendorizes Queue and some fragments of six, but that is
> >of a much smaller scale, and is for technical reasons, rather than
> >appeasing-newbie reasons. But HTTP has a lot of security-critical
> >surface area. If I were to just bundle my own fork of an HMAC library
> >with a few of my own special enhancements, that would be seen as a
> >problem.
>  
> It would be seen as a problem. Except we don’t do anything even remotely
> as security related. The majority of what we do is certificate
> verification. We don’t bundle ssl. We don’t bundle pyOpenSSL. We don’t
> bundle back ports.ssl. We don’t have custom TLS handlers that we wrote
> from scratch. You’re analogy is way out of proportion. And for what it’s
> worth, pyCrypto (if I remember correctly) has never been audited and yet
> it is used:
> https://github.com/openstack/requirements/blob/master/global-requirements.t  
> xt#L86. That seems like a bigger issue than whether requests vendors an
> implementation detail.
>  
> >
> >> And yeah, we’ll continue to take the blame for the mistake that was made
> >> for those two exposures. As for “Is that how things should be done?”
> >> that’s not for me to answer. More than enough projects do it and do it
> >>out
> >> of necessity. The reality is that by 

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

On 9/17/14, 5:39 PM, "Mike Bayer"  wrote:

>
>On Sep 17, 2014, at 4:31 PM, Ian Cordasco 
>wrote:
>
>> Project X pins a version of requests. Alice doesn’t know anything about
>> requests and does pip install X. Until Alice takes a more active role in
>> the development of Project X and looks into requests, she will never
>>know
>> she’s installed software that has exposures in it.
>
>If a vulnerability is reported in urllib3 1.9.1, Alice, as well as me and
>everyone else who is not a novice, will at least know we need to run:
>
>$ pip show urllib3
>---
>Name: urllib3
>Version: 1.9.1
>Location: 
>/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packa
>ges
>Requires: 
>
>
>and we know right there we have to upgrade.  We upgrade, and we’re done.
>  If we see that some library is pinning it, we will know.  We will
>complain loudly to that library’s author and/or replace that library.
>The tools are there to give us what we need to be aware and to escalate
>the problem.

And when the library is unmaintained, you’ll be yelling into the echo
chamber. If it is popular enough, it’ll continue to be used regardless
because every recommendation that will be highest ranked by :search_engine
will tell people looking for related libraries to do so. In all
likelihood, most users who stumble across these projects will also be
ignorant of the fact that :some_dependency has a CVE and needs to be
upgraded.

>When a library silently bundles the source code and bypasses any normal
>means of us knowing it’s present unless we read the source code or scour
>the documentation, we have no way to know we’re affected.Some
>applications, particularly pip, have to do this, however, it should only
>be for technical reasons.  It should not be because you don’t want novice
>users to have to learn something, or because you’re angling to have lots
>of downloads on pypi.

It is for technical reasons, but this is not the appropriate place to
discuss them. There are at least 3 other closed issues on requests’ issue
tracker that discuss most (if not all) of them. This is the list to
discuss the development of OpenStack. Finally, for the last time, the fact
that we vendor these libraries exists in multiple places (beyond the
source). 

>>> People make sure to upgrade their Requests libraries locally, but for
>>>all
>>> those poor saps who have *no idea* they have widely used apps that are
>>> bundling it silently, they remain totally open to vulnerabilities and
>>>the
>>> black hats have disneyland at their disposal.
>> 
>> I think more applications bundle it than you realize. You’re likely
>>using
>> one daily that does it.
>
>
>SQLAlchemy itself vendorizes Queue and some fragments of six, but that is
>of a much smaller scale, and is for technical reasons, rather than
>appeasing-newbie reasons.   But HTTP has a lot of security-critical
>surface area.   If I were to just bundle my own fork of an HMAC library
>with a few of my own special enhancements, that would be seen as a
>problem.

It would be seen as a problem. Except we don’t do anything even remotely
as security related. The majority of what we do is certificate
verification. We don’t bundle ssl. We don’t bundle pyOpenSSL. We don’t
bundle back ports.ssl. We don’t have custom TLS handlers that we wrote
from scratch. You’re analogy is way out of proportion. And for what it’s
worth, pyCrypto (if I remember correctly) has never been audited and yet
it is used: 
https://github.com/openstack/requirements/blob/master/global-requirements.t
xt#L86. That seems like a bigger issue than whether requests vendors an
implementation detail.

>
>> And yeah, we’ll continue to take the blame for the mistake that was made
>> for those two exposures. As for “Is that how things should be done?”
>> that’s not for me to answer. More than enough projects do it and do it
>>out
>> of necessity. The reality is that by vendoring its dependencies,
>>requests
>> allows its users more flexibility than other projects.
>
>I haven’t seen the technical reason for Requests doing this, I’ve only
>seen this reason: "I want my users to be free to not use packaging if
>they don't won't to. They can just grab the tarball and go.”.   If that’s
>really the only reason, then I fail to see how that reason has anything
>to do with flexibility, other than the flexibility to remain lazy and
>ignorant of basic computer programming skills - and Requests is a library
>*for programmers*.  It doesn’t do anything without typing code.

Perhaps I wasn’t clear enough. If I wasn’t, I apologize. What I meant to
say above is that requests gives the users the ability to vendor requests
into their libraries or applications as well. It does not advocate it. It
does not require it (although I’m starting to wonder if we should change
our license to make it a requirement :P). It gives the user the ability to
use requests as a vendored dependency without having to edit a single line
of the source. No imports need to be mangled and no

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Mike Bayer]

On Sep 17, 2014, at 4:31 PM, Ian Cordasco  wrote:

> Project X pins a version of requests. Alice doesn’t know anything about
> requests and does pip install X. Until Alice takes a more active role in
> the development of Project X and looks into requests, she will never know
> she’s installed software that has exposures in it.

If a vulnerability is reported in urllib3 1.9.1, Alice, as well as me and 
everyone else who is not a novice, will at least know we need to run:

$ pip show urllib3 
---
Name: urllib3
Version: 1.9.1
Location: 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
Requires: 


and we know right there we have to upgrade.  We upgrade, and we’re done.If 
we see that some library is pinning it, we will know.  We will complain loudly 
to that library’s author and/or replace that library.   The tools are there to 
give us what we need to be aware and to escalate the problem.

When a library silently bundles the source code and bypasses any normal means 
of us knowing it’s present unless we read the source code or scour the 
documentation, we have no way to know we’re affected.Some applications, 
particularly pip, have to do this, however, it should only be for technical 
reasons.  It should not be because you don’t want novice users to have to learn 
something, or because you’re angling to have lots of downloads on pypi.


>> People make sure to upgrade their Requests libraries locally, but for all
>> those poor saps who have *no idea* they have widely used apps that are
>> bundling it silently, they remain totally open to vulnerabilities and the
>> black hats have disneyland at their disposal.
> 
> I think more applications bundle it than you realize. You’re likely using
> one daily that does it.


SQLAlchemy itself vendorizes Queue and some fragments of six, but that is of a 
much smaller scale, and is for technical reasons, rather than appeasing-newbie 
reasons.   But HTTP has a lot of security-critical surface area.   If I were to 
just bundle my own fork of an HMAC library with a few of my own special 
enhancements, that would be seen as a problem.


> And yeah, we’ll continue to take the blame for the mistake that was made
> for those two exposures. As for “Is that how things should be done?”
> that’s not for me to answer. More than enough projects do it and do it out
> of necessity. The reality is that by vendoring its dependencies, requests
> allows its users more flexibility than other projects.

I haven’t seen the technical reason for Requests doing this, I’ve only seen 
this reason: "I want my users to be free to not use packaging if they don't 
won't to. They can just grab the tarball and go.”.   If that’s really the only 
reason, then I fail to see how that reason has anything to do with flexibility, 
other than the flexibility to remain lazy and ignorant of basic computer 
programming skills - and Requests is a library *for programmers*.  It doesn’t 
do anything without typing code.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Clint Byrum]

Excerpts from Ian Cordasco's message of 2014-09-17 12:42:57 -0700:
> On 9/17/14, 1:46 PM, "Clint Byrum"  wrote:
> 
> >Excerpts from Davanum Srinivas's message of 2014-09-17 10:15:29 -0700:
> >> I was trying request-ifying oslo.vmware and ran into this as well:
> >> https://review.openstack.org/#/c/121956/
> >> 
> >> And we don't seem to have urllib3 in global-requirements either.
> >> Should we do that first?
> >
> >Honestly, after reading this:
> >
> >https://github.com/kennethreitz/requests/pull/1812
> >
> >I think we might want to consider requests a poor option. Its author
> >clearly doesn't understand the role a _library_ plays in software
> >development and considers requests an application, not a library.
> 
> Yes that is Kenneth’s opinion. That is not the opinion of the core
> developers though. We see it as a library but this is something we aren’t
> going to currently change any time soon.
> 

Good to know but troubling to hear that there's no change in sight.

> >For instance, why is requests exposing internal implementation details
> >at all?
> 
> Where exactly are we exposing internal implementation details? A normal
> user (even advanced users) can use requests without ever digging into
> requests.packages. What implementation details are we exposing and where?
> 
> >It should be wrapping any exceptions or objects to avoid
> >forcing users to make this choice at all.
> 
> We do. Occasionally (like in 2.4.0) urllib3 adds an exception that we
> missed notice of and it slips through. We released 2.4.1 a couple days
> later with the fix for that. Pretty much every error we’ve seen or know
> about is caught and rewrapped as a requests exception. I’m not sure what
> you’re arguing here, unless of course you have not used requests.
> 

I had seen handling of urllib3 exceptions in some requests code and was
seeing a few issues about that. I understand now that it is not
intentional, but a side effect of misunderstanding.

> That aside, I’ve been mulling over how effectively the clients use
> requests. I haven’t investigated all of them, but many seem to reach into
> implementation details on their own. If I remember nova client has
> something it has commented as “connection pooling” while requests and
> urllib3 do that automatically. I haven’t started to investigate exactly
> why they do this. Likewise, glance client has custom certificate
> verification in glanceclient.common.https. Why? I’m not exactly certain
> yet. It seems for the most part from what little I’ve seen that requests
> is too high-level a library for OpenStack’s needs at best, and actively
> obscures details OpenStack developers need (or don’t realize requests
> provides in most cases).
> 

Indeed, it sounds like there has been some misuse of requests that led
to the situation.

> Circling back to the issue of vendoring though: it’s a conscious decision
> to do this, and in the last two years there have been 2 CVEs reported for
> requests. There have been none for urllib3 and none for chardet. (Frankly
> I don’t think either urllib3 or chardet have had any CVEs reported against
> them, but let’s ignore that for now.) While security is typically the
> chief concern with vendoring, none of the libraries we use have had
> security issues rendering it a moot point in my opinion. The benefits of
> vendoring for us as a team have been numerous and we will likely continue
> to do it until it stops benefiting us and our users.
> 

On a micro scale, you are absolutely correct. The point is not that
there is a tactical reason for urllib3 to be de-vendored, but rather
that there is a strategic reason to not use vendored libraries. When
updates are necessary, they must be complete, and rapid. Please refer
to the zlib travesty to see why everyone, not just distribution vendors,
should be extremely concerned about embedded libraries.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Dean Troyer]

On Wed, Sep 17, 2014 at 3:53 PM, Robert Collins 
wrote:

> On 18 September 2014 08:01, Dean Troyer  wrote:
> > Interestingly enough, the distros are doing exactly what they don't want
> us
> > to do, ie, rebuilding things to use 'their' tested version of
> dependencies
> > rather than the included one...
>
> Indeed - but the distros are solving for two specific issues:
>

No argument, just observing the recursive nature of this...

Also, if we pin to a version, is the downstream consequence different?
 IIRC Thomas has had to do this with Django (1.7?) and Horizon, probably
with others too.

As a provider of an app package directly to users, dealing with the
front-line consequences of changing dependencies falls on me.  And its one
reason with this hat on I want static linking, or a Python equivalent of it
(bundling/vendoring) at the app level.

As an upstream to a distro, I'm happy to let them deal with all of that.
 Isn't it fun being in the middle?

OOI were thouse changes API breaks or were we depending on nonpublic
> aspects?
>

prettytable was packaging once and I don't recall the other.  requests,
aside from the recent 2.4.0 release, was the 1.0.0 release when we weren't
expecting it and nothing was pinned <1.0.0.  I think that was an API change
that bit us.  The 1.0.0 version was clear, but not having the control over
the timing of the change is what makes me understand Kenneth's position on
urllib3 and why those who bundle requests do that too...

Is my Go-ness showing yet?

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Robert Collins]

On 18 September 2014 08:01, Dean Troyer  wrote:
> Interestingly enough, the distros are doing exactly what they don't want us
> to do, ie, rebuilding things to use 'their' tested version of dependencies
> rather than the included one...

Indeed - but the distros are solving for two specific issues:

1) Effort by the distro team

1a) 'how to minimise effort delivering up-to-date packages when the
package count is 20k+'.  This is a pure numbers game: update one
binary on a users system, or 10, or 20 etc. Things deep down a
dependency tree can turn up in huge numbers of places, if vendoring is
commonplace.

1b) 'how to security fix packages where the upstream has stopped being
responsive' - updating vendored trees is often harder than just
unpacking a new release, since they may have deltas in addition to
being vendored - and vendoring may also require patches (depending on
the language). Just waiting for a new vendor release can be a long
process sometimes :)

And both of these are in the context of

2) how to fix things promptly for users

2a) binary packages are often quite substantial - particularly for
some c++ programs - a non-binary delta based approach (and thats what
all the distros started with) will consume a tonne of bandwidth if you
have to multiply out the uses of a package.

2b) distros were privileged in our modern responsible disclosure world
(via the vendor-sec list - I'm not sure what the current state of play
is) - but at one point they found out about security issues *before*
small consumers of packages do, and would fix it and then the upstream
release is made.

You can see, I think, how vendoring plays havoc with the amount of
effort a small team has to exert to keep a large set of packages
patched ahead of upstream releases of the vendored libraries. Its not
an intrinsic problem - its a problem we've constructed by centralising
and limiting notifications of CVEs: unless requests authors are part
of the urllib3 security response team, they can never respond to CVE's
in as timely a manner *while vendoring is in use*.

...
> FWIW I totally understand the desire for vendoring...I want to do the same
> thing with OSC because of the number of times we've been broken by requests,
> prettytable and others changing out from under us.  It is easy enough for me
> to fix my box but a cloud user that just want to get his VMs running isn't
> going to be happy, especially on Windows.
>
> dt

OOI were thouse changes API breaks or were we depending on nonpublic aspects?

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

On 9/17/14, 3:11 PM, "Mike Bayer"  wrote:

>
>On Sep 17, 2014, at 3:42 PM, Ian Cordasco 
>wrote:
>
>> 
>> Circling back to the issue of vendoring though: it’s a conscious
>>decision
>> to do this, and in the last two years there have been 2 CVEs reported
>>for
>> requests. There have been none for urllib3 and none for chardet.
>>(Frankly
>> I don’t think either urllib3 or chardet have had any CVEs reported
>>against
>> them, but let’s ignore that for now.) While security is typically the
>> chief concern with vendoring, none of the libraries we use have had
>> security issues rendering it a moot point in my opinion.
>
>That’s just amazing.  Requests actually deals with security features
>*directly*, certificates, TLS connections, everything; to take the
>attitude that “well there’ve been hardly any security issues in a *whole
>two years*, so I’m not so concerned” is really not one that is acceptable
>by serious development teams.

I said 2 years, because I wasn’t involved much before that, but looking at
the histories of the involved projects there aren’t mentions of CVEs
before then either.

>Wouldn’t it be a problem for *you* if Requests itself were vendored?
>You fix a major security hole, but your consuming projects don’t respond,
>their developers are on vacation, sorry, so that hole just keeps right on
>going. 

Isn’t the whole point of distributing a library to let people use it as
they see fit? If requests fixes it immediately and releases it, it’s not
our responsibility to search out every piece of software to fix it for
them. We took all of the appropriate measures to document the two CVEs
that were reported earlier this year. Software that vendored requests
including still vulnerable to those two exposures are responsible for
their own updates. Further, let’s consider this potential situation:

Project X pins a version of requests. Alice doesn’t know anything about
requests and does pip install X. Until Alice takes a more active role in
the development of Project X and looks into requests, she will never know
she’s installed software that has exposures in it. In all likelihood, any
person who just uses something that pins requests will never check for it.
If they just use pip and Project X never updates, it’s not our
responsibility for anything that happens to the user.

>People make sure to upgrade their Requests libraries locally, but for all
>those poor saps who have *no idea* they have widely used apps that are
>bundling it silently, they remain totally open to vulnerabilities and the
>black hats have disneyland at their disposal.

I think more applications bundle it than you realize. You’re likely using
one daily that does it.

>The blame keeps going right to you as well.  Is that really how things
>should be done?

And yeah, we’ll continue to take the blame for the mistake that was made
for those two exposures. As for “Is that how things should be done?”
that’s not for me to answer. More than enough projects do it and do it out
of necessity. The reality is that by vendoring its dependencies, requests
allows its users more flexibility than other projects. Even if we didn’t,
users would still likely find ways to vendor requests and its dependencies
for their own use and in doing so would have to modify requests to rewrite
the import statements to point at those vendored dependencies. The fact is
that vendoring is a real solution and it’s deployed more often than you
likely realize. It benefits our project and it benefits our users.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Mike Bayer]

On Sep 17, 2014, at 3:42 PM, Ian Cordasco  wrote:

> 
> Circling back to the issue of vendoring though: it’s a conscious decision
> to do this, and in the last two years there have been 2 CVEs reported for
> requests. There have been none for urllib3 and none for chardet. (Frankly
> I don’t think either urllib3 or chardet have had any CVEs reported against
> them, but let’s ignore that for now.) While security is typically the
> chief concern with vendoring, none of the libraries we use have had
> security issues rendering it a moot point in my opinion.

That’s just amazing.  Requests actually deals with security features 
*directly*, certificates, TLS connections, everything; to take the attitude 
that “well there’ve been hardly any security issues in a *whole two years*, so 
I’m not so concerned” is really not one that is acceptable by serious 
development teams.

Wouldn’t it be a problem for *you* if Requests itself were vendored?   You fix 
a major security hole, but your consuming projects don’t respond, their 
developers are on vacation, sorry, so that hole just keeps right on going.   
People make sure to upgrade their Requests libraries locally, but for all those 
poor saps who have *no idea* they have widely used apps that are bundling it 
silently, they remain totally open to vulnerabilities and the black hats have 
disneyland at their disposal.   The blame keeps going right to you as well.  Is 
that really how things should be done?



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Dean Troyer]

Interestingly enough, the distros are doing exactly what they don't want us
to do, ie, rebuilding things to use 'their' tested version of dependencies
rather than the included one...

On Wed, Sep 17, 2014 at 2:42 PM, Ian Cordasco 
wrote:
>
> That aside, I’ve been mulling over how effectively the clients use
> requests. I haven’t investigated all of them, but many seem to reach into
> implementation details on their own. If I remember nova client has
> something it has commented as “connection pooling” while requests and
> urllib3 do that automatically. I haven’t started to investigate exactly
> why they do this. Likewise, glance client has custom certificate
> verification in glanceclient.common.https. Why? I’m not exactly certain
> yet. It seems for the most part from what little I’ve seen that requests
> is too high-level a library for OpenStack’s needs at best, and actively
> obscures details OpenStack developers need (or don’t realize requests
> provides in most cases).
>

Part of that is my doing, the initial conversion from httplib2 to requests
was intended to be as simple as possible in order to get the benefits of
proper certificate verification.  glanceclient never got this (maybe until
recently?) because it uses OpenSSL.  The come-back-and-clean-things-up work
was intended to be Alessio's apiclient stuff that I think is still in
oslo-incubator.  That was never finished for a variety of reasons.  Since
that time you're seeing the results of other fixes (connection-pooling
being one) that look at the existing code and not at the proper re-factor
to push that stuff into requests.

The real fix for the clients is to start over and re-build them on top of
(in this case) requests to utilize all that it brings.  This is already
happening...

FWIW I totally understand the desire for vendoring...I want to do the same
thing with OSC because of the number of times we've been broken by
requests, prettytable and others changing out from under us.  It is easy
enough for me to fix my box but a cloud user that just want to get his VMs
running isn't going to be happy, especially on Windows.

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

On 9/17/14, 1:46 PM, "Clint Byrum"  wrote:

>Excerpts from Davanum Srinivas's message of 2014-09-17 10:15:29 -0700:
>> I was trying request-ifying oslo.vmware and ran into this as well:
>> https://review.openstack.org/#/c/121956/
>> 
>> And we don't seem to have urllib3 in global-requirements either.
>> Should we do that first?
>
>Honestly, after reading this:
>
>https://github.com/kennethreitz/requests/pull/1812
>
>I think we might want to consider requests a poor option. Its author
>clearly doesn't understand the role a _library_ plays in software
>development and considers requests an application, not a library.

Yes that is Kenneth’s opinion. That is not the opinion of the core
developers though. We see it as a library but this is something we aren’t
going to currently change any time soon.

>For instance, why is requests exposing internal implementation details
>at all?

Where exactly are we exposing internal implementation details? A normal
user (even advanced users) can use requests without ever digging into
requests.packages. What implementation details are we exposing and where?

>It should be wrapping any exceptions or objects to avoid
>forcing users to make this choice at all.

We do. Occasionally (like in 2.4.0) urllib3 adds an exception that we
missed notice of and it slips through. We released 2.4.1 a couple days
later with the fix for that. Pretty much every error we’ve seen or know
about is caught and rewrapped as a requests exception. I’m not sure what
you’re arguing here, unless of course you have not used requests.

That aside, I’ve been mulling over how effectively the clients use
requests. I haven’t investigated all of them, but many seem to reach into
implementation details on their own. If I remember nova client has
something it has commented as “connection pooling” while requests and
urllib3 do that automatically. I haven’t started to investigate exactly
why they do this. Likewise, glance client has custom certificate
verification in glanceclient.common.https. Why? I’m not exactly certain
yet. It seems for the most part from what little I’ve seen that requests
is too high-level a library for OpenStack’s needs at best, and actively
obscures details OpenStack developers need (or don’t realize requests
provides in most cases).

Circling back to the issue of vendoring though: it’s a conscious decision
to do this, and in the last two years there have been 2 CVEs reported for
requests. There have been none for urllib3 and none for chardet. (Frankly
I don’t think either urllib3 or chardet have had any CVEs reported against
them, but let’s ignore that for now.) While security is typically the
chief concern with vendoring, none of the libraries we use have had
security issues rendering it a moot point in my opinion. The benefits of
vendoring for us as a team have been numerous and we will likely continue
to do it until it stops benefiting us and our users.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Mike Bayer]

On Sep 17, 2014, at 2:46 PM, Clint Byrum  wrote:

> Excerpts from Davanum Srinivas's message of 2014-09-17 10:15:29 -0700:
>> I was trying request-ifying oslo.vmware and ran into this as well:
>> https://review.openstack.org/#/c/121956/
>> 
>> And we don't seem to have urllib3 in global-requirements either.
>> Should we do that first?
> 
> Honestly, after reading this:
> 
> https://github.com/kennethreitz/requests/pull/1812
> 
> I think we might want to consider requests a poor option. Its author
> clearly doesn't understand the role a _library_ plays in software
> development and considers requests an application, not a library.
> 
> For instance, why is requests exposing internal implementation details
> at all?  It should be wrapping any exceptions or objects to avoid
> forcing users to make this choice at all.

that link is horrifying.   I’m really surprised Requests does this, and that 
nobody has complained very loudly about it.   It’s wrong on every level not the 
least of which is the huge security implications.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Clint Byrum]

Excerpts from Davanum Srinivas's message of 2014-09-17 10:15:29 -0700:
> I was trying request-ifying oslo.vmware and ran into this as well:
> https://review.openstack.org/#/c/121956/
> 
> And we don't seem to have urllib3 in global-requirements either.
> Should we do that first?

Honestly, after reading this:

https://github.com/kennethreitz/requests/pull/1812

I think we might want to consider requests a poor option. Its author
clearly doesn't understand the role a _library_ plays in software
development and considers requests an application, not a library.

For instance, why is requests exposing internal implementation details
at all?  It should be wrapping any exceptions or objects to avoid
forcing users to make this choice at all.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Davanum Srinivas]

I was trying request-ifying oslo.vmware and ran into this as well:
https://review.openstack.org/#/c/121956/

And we don't seem to have urllib3 in global-requirements either.
Should we do that first?

-- dims

On Wed, Sep 17, 2014 at 1:05 PM, Clint Byrum  wrote:
> This is where Debian's "one urllib3 to rule them all" model fails in
> a modern fast paced world. Debian is arguably doing the right thing by
> pushing everyone to use one API, and one library, so that when that one
> library is found to be vulnerable to security problems, one update covers
> everyone. Also, this is an HTTP/HTTPS library.. so nobody can make the
> argument that security isn't paramount in this context.
>
> But we all know that the "app store" model has started to bleed down into
> backend applications, and now you just ship the virtualenv or docker
> container that has your app as you tested it, and if that means you're
> 20 versions behind on urllib3, that's your problem, not the OS vendor's.
>
> I think it is _completely_ irresponsible of requests, a library, to
> embed another library. But I don't know if we can avoid making use of
> it if we are going to be exposed to objects that are attached to it.
>
> Anyway, Thomas, if you're going to send the mob with pitchforks and
> torches somewhere, I'd say send them to wherever requests makes its
> home. OpenStack is just buying their mutated product.
>
> Excerpts from Donald Stufft's message of 2014-09-17 08:22:48 -0700:
>> Looking at the code on my phone it looks completely correct to use the 
>> vendored copy here and it wouldn't actually work otherwise.
>>
>> > On Sep 17, 2014, at 11:17 AM, Donald Stufft  wrote:
>> >
>> > I don't know the specific situation but it's appropriate to do this if 
>> > you're using requests and wish to interact with the urllib3 that requests 
>> > is using.
>> >
>> >> On Sep 17, 2014, at 11:15 AM, Thomas Goirand  wrote:
>> >>
>> >> Hi,
>> >>
>> >> I'm horrified by what I just found. I have just found out this in
>> >> glanceclient:
>> >>
>> >> File "/tests/test_ssl.py", line 19, in 
>> >>   from requests.packages.urllib3 import poolmanager
>> >> ImportError: No module named packages.urllib3
>> >>
>> >> Please *DO NOT* do this. Instead, please use urllib3 from ... urllib3.
>> >> Not from requests. The fact that requests is embedding its own version
>> >> of urllib3 is an heresy. In Debian, the embedded version of urllib3 is
>> >> removed from requests.
>> >>
>> >> In Debian, we spend a lot of time to "un-vendorize" stuff, because
>> >> that's a security nightmare. I don't want to have to patch all of
>> >> OpenStack to do it there as well.
>> >>
>> >> And no, there's no good excuse here...
>> >>
>> >> Thomas Goirand (zigo)
>> >>
>> >> ___
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev@lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> > ___
>> > OpenStack-dev mailing list
>> > OpenStack-dev@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Clint Byrum]

This is where Debian's "one urllib3 to rule them all" model fails in
a modern fast paced world. Debian is arguably doing the right thing by
pushing everyone to use one API, and one library, so that when that one
library is found to be vulnerable to security problems, one update covers
everyone. Also, this is an HTTP/HTTPS library.. so nobody can make the
argument that security isn't paramount in this context.

But we all know that the "app store" model has started to bleed down into
backend applications, and now you just ship the virtualenv or docker
container that has your app as you tested it, and if that means you're
20 versions behind on urllib3, that's your problem, not the OS vendor's.

I think it is _completely_ irresponsible of requests, a library, to
embed another library. But I don't know if we can avoid making use of
it if we are going to be exposed to objects that are attached to it.

Anyway, Thomas, if you're going to send the mob with pitchforks and
torches somewhere, I'd say send them to wherever requests makes its
home. OpenStack is just buying their mutated product.

Excerpts from Donald Stufft's message of 2014-09-17 08:22:48 -0700:
> Looking at the code on my phone it looks completely correct to use the 
> vendored copy here and it wouldn't actually work otherwise. 
> 
> > On Sep 17, 2014, at 11:17 AM, Donald Stufft  wrote:
> > 
> > I don't know the specific situation but it's appropriate to do this if 
> > you're using requests and wish to interact with the urllib3 that requests 
> > is using.
> > 
> >> On Sep 17, 2014, at 11:15 AM, Thomas Goirand  wrote:
> >> 
> >> Hi,
> >> 
> >> I'm horrified by what I just found. I have just found out this in
> >> glanceclient:
> >> 
> >> File "/tests/test_ssl.py", line 19, in 
> >>   from requests.packages.urllib3 import poolmanager
> >> ImportError: No module named packages.urllib3
> >> 
> >> Please *DO NOT* do this. Instead, please use urllib3 from ... urllib3.
> >> Not from requests. The fact that requests is embedding its own version
> >> of urllib3 is an heresy. In Debian, the embedded version of urllib3 is
> >> removed from requests.
> >> 
> >> In Debian, we spend a lot of time to "un-vendorize" stuff, because
> >> that's a security nightmare. I don't want to have to patch all of
> >> OpenStack to do it there as well.
> >> 
> >> And no, there's no good excuse here...
> >> 
> >> Thomas Goirand (zigo)
> >> 
> >> ___
> >> OpenStack-dev mailing list
> >> OpenStack-dev@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

Looking at the code on my phone it looks completely correct to use the vendored 
copy here and it wouldn't actually work otherwise. 

> On Sep 17, 2014, at 11:17 AM, Donald Stufft  wrote:
> 
> I don't know the specific situation but it's appropriate to do this if you're 
> using requests and wish to interact with the urllib3 that requests is using.
> 
>> On Sep 17, 2014, at 11:15 AM, Thomas Goirand  wrote:
>> 
>> Hi,
>> 
>> I'm horrified by what I just found. I have just found out this in
>> glanceclient:
>> 
>> File "/tests/test_ssl.py", line 19, in 
>>   from requests.packages.urllib3 import poolmanager
>> ImportError: No module named packages.urllib3
>> 
>> Please *DO NOT* do this. Instead, please use urllib3 from ... urllib3.
>> Not from requests. The fact that requests is embedding its own version
>> of urllib3 is an heresy. In Debian, the embedded version of urllib3 is
>> removed from requests.
>> 
>> In Debian, we spend a lot of time to "un-vendorize" stuff, because
>> that's a security nightmare. I don't want to have to patch all of
>> OpenStack to do it there as well.
>> 
>> And no, there's no good excuse here...
>> 
>> Thomas Goirand (zigo)
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

I don't know the specific situation but it's appropriate to do this if you're 
using requests and wish to interact with the urllib3 that requests is using.

> On Sep 17, 2014, at 11:15 AM, Thomas Goirand  wrote:
> 
> Hi,
> 
> I'm horrified by what I just found. I have just found out this in
> glanceclient:
> 
>  File "/tests/test_ssl.py", line 19, in 
>from requests.packages.urllib3 import poolmanager
> ImportError: No module named packages.urllib3
> 
> Please *DO NOT* do this. Instead, please use urllib3 from ... urllib3.
> Not from requests. The fact that requests is embedding its own version
> of urllib3 is an heresy. In Debian, the embedded version of urllib3 is
> removed from requests.
> 
> In Debian, we spend a lot of time to "un-vendorize" stuff, because
> that's a security nightmare. I don't want to have to patch all of
> OpenStack to do it there as well.
> 
> And no, there's no good excuse here...
> 
> Thomas Goirand (zigo)
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev