Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-15 Thread Cory Benfield 

> On 14 Oct 2015, at 23:23, Thomas Goirand  wrote:
> I do understand that you don't like being called this way, though this
> is still the reality. Vendorizing still inflicting some major pain to a
> lot of your users:
> - This thread one of the demonstration of it.
> - You having to contact downstream distros is another.
> - The unbundling work inflicted to downstream package maintainers is a
> 3rd another.
> 
> So like it or not, it is a fact that it is difficult to work with
> requests because of the way it is released upstream.

As I said earlier, I’m not getting drawn into a debate about vendorizing in 
this forum. The last one of these was sufficiently toxic that I’m simply 
unwilling to have the discussion here. If you really want to talk about this 
again, I’m happy to take it out of this mailing list to somewhere where fewer 
people are going to make the discussion personal.

Note however that point 2 is not accurate. The main reason we have 
relationships with our downstream repackagers is for security release purposes. 
Per our security policy, we have exchanged GPG keys with them, and will make 
sure we contact them ahead of time so we can perform a synchronised release of 
security updates. In this instance we chose to use our relationship with our 
repackagers to get this change made, but it is not the main reason we 
communicate with them. (Also, they’re nice people!)

>> has had a policy in place for six months
>> that ensures that you can have the same result with pip and
>> system packages. For six months we have only updated to versions
>> of urllib3 that are actually released, and therefore that are
>> definitely available from pip (and potentially available from
>> the distribution).
>> 
>> The reason this has not been working is because the distributions,
>> when they unbundle us, have not been populating their setup.py to
>> reflect the dependency: only their own metadata. We’ve been in
>> contact with them, and this change is being made in the
>> distributions we have relationships with.
> 
> Though you could have avoid all of this pain if you were not bundling.
> Isn't all of this make you re-think your vendorizing policy? Or still
> not? I'm asking because I still didn't read your answer about the
> important question: since you aren't using specially crafted versions of
> urllib3 anymore, and now only using official releases, what's the reason
> that keeps you vendorizing? Not trying to convince you here, just trying
> to understand.

Again, I’m not being drawn into this discussion here.

Let me make one point, though. There are three people involved in a 
decision-making role on the requests project, and this is an important issue to 
every member of the team. This policy has been part of the requests project for 
a very long time, and we aren’t going to change it in a short space of time: 
I’m certainly not going to unilaterally do so. All I can promise you is that we 
continue to talk about this internally, and if we *unanimously* feel 
comfortable changing our policy we will do so. Until then, I’m happy to do my 
best to accommodate as many people as possible (which in this case I believe we 
have done).

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-15 Thread Dmitry Tantsur 

On 10/15/2015 12:18 AM, Robert Collins wrote:

On 15 October 2015 at 11:11, Thomas Goirand  wrote:


One major pain point is unfortunately something ridiculously easy to
fix, but which nobody seems to care about: the long & short descriptions
format. These are usually buried into the setup.py black magic, which by
the way I feel is very unsafe (does PyPi actually execute "python
setup.py" to find out about description texts? I hope they are running
this in a sandbox...).

Since everyone uses the fact that PyPi accepts RST format for the long
description, there's nothing that can really easily fit the
debian/control. Probably a rst2txt tool would help, but still, the long
description would still be polluted with things like changelog, examples
and such (damned, why people think it's the correct place to put that...).

The only way I'd see to fix this situation, would be a PEP. This will
probably take a decade to have everyone switching to a new correct way
to write a long & short description...


Perhaps Debian (1 thing) should change, rather than trying to change
all the upstreams packaged in it (>20K) :)


+1. Both README and PyPI are for users, and I personally find detailed 
descriptions (especially a couple of simple examples) on the PyPI page 
to be of so much value.




-Rob





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-15 Thread Thomas Goirand 
On 10/15/2015 12:18 AM, Robert Collins wrote:
> On 15 October 2015 at 11:11, Thomas Goirand  wrote:
>>
>> One major pain point is unfortunately something ridiculously easy to
>> fix, but which nobody seems to care about: the long & short descriptions
>> format. These are usually buried into the setup.py black magic, which by
>> the way I feel is very unsafe (does PyPi actually execute "python
>> setup.py" to find out about description texts? I hope they are running
>> this in a sandbox...).
>>
>> Since everyone uses the fact that PyPi accepts RST format for the long
>> description, there's nothing that can really easily fit the
>> debian/control. Probably a rst2txt tool would help, but still, the long
>> description would still be polluted with things like changelog, examples
>> and such (damned, why people think it's the correct place to put that...).
>>
>> The only way I'd see to fix this situation, would be a PEP. This will
>> probably take a decade to have everyone switching to a new correct way
>> to write a long & short description...
> 
> Perhaps Debian (1 thing) should change, rather than trying to change
> all the upstreams packaged in it (>20K) :)
> 
> -Rob

Well, having the changlog (and other stuff) of packages merged into the
long description is not helpful, not for Debian, nor for upstream Python
packages.

Thomas


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-15 Thread Monty Taylor 

On 10/15/2015 02:53 AM, Thomas Goirand wrote:

On 10/15/2015 12:18 AM, Robert Collins wrote:

On 15 October 2015 at 11:11, Thomas Goirand  wrote:


One major pain point is unfortunately something ridiculously easy to
fix, but which nobody seems to care about: the long & short descriptions
format. These are usually buried into the setup.py black magic, which by
the way I feel is very unsafe (does PyPi actually execute "python
setup.py" to find out about description texts? I hope they are running
this in a sandbox...).


You should go look at $package.egg-info/PKG-INFO in the source tarball. 
The short and long description are in that and it's a parseable format


It's an RFC822 format information file - so the following python gets 
the short description inside of an unpacked shade tarball:


import rfc822
a=rfc822.Message(open('shade.egg-info/PKG-INFO', 'r'))
print a['summary']

I think this should solve your problem of getting the information. (you 
don't need python - rfc822 is pretty parseable by a human)



Since everyone uses the fact that PyPi accepts RST format for the long
description, there's nothing that can really easily fit the
debian/control. Probably a rst2txt tool would help, but still, the long
description would still be polluted with things like changelog, examples
and such (damned, why people think it's the correct place to put that...).


The reason they think that is because PyPI is a place where people go to 
download things, and often quick and easy examples of how the library 
works help people to decide if it's for them or not. The uses cases are 
different.



The only way I'd see to fix this situation, would be a PEP. This will
probably take a decade to have everyone switching to a new correct way
to write a long & short description...


Perhaps Debian (1 thing) should change, rather than trying to change
all the upstreams packaged in it (>20K) :)

-Rob


Well, having the changlog (and other stuff) of packages merged into the
long description is not helpful, not for Debian, nor for upstream Python
packages.


I actually have gotten multiple requests from people to get changelogs 
uploaded to PyPI (which involves appending to the long description) 
because it is helpful to them. . I think right now people find value in 
the information being there so I don't think this one is very winnable 
right now. Sorry ...


In

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-15 Thread Thomas Goirand 
On 10/15/2015 03:59 PM, Monty Taylor wrote:
> On 10/15/2015 02:53 AM, Thomas Goirand wrote:
>> Well, having the changlog (and other stuff) of packages merged into the
>> long description is not helpful, not for Debian, nor for upstream Python
>> packages.
> 
> I actually have gotten multiple requests from people to get changelogs
> uploaded to PyPI (which involves appending to the long description)
> because it is helpful to them. . I think right now people find value in
> the information being there so I don't think this one is very winnable
> right now. Sorry ...

It could be a win if PiPy was supporting other (more relevant) places
where to put the information published on the site.

Thomas


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-15 Thread Thomas Goirand 
On 10/15/2015 12:08 PM, Cory Benfield wrote:
>> On 14 Oct 2015, at 23:23, Thomas Goirand  wrote:
>> Though you could have avoid all of this pain if you were not bundling.
>> Isn't all of this make you re-think your vendorizing policy? Or still
>> not? I'm asking because I still didn't read your answer about the
>> important question: since you aren't using specially crafted versions of
>> urllib3 anymore, and now only using official releases, what's the reason
>> that keeps you vendorizing? Not trying to convince you here, just trying
>> to understand.
> 
> Again, I’m not being drawn into this discussion here.
> 
> Let me make one point, though. There are three people involved in a
> decision-making role on the requests project, and this is an important
> issue to every member of the team. This policy has been part of the
> requests project for a very long time, and we aren’t going to change
> it in a short space of time: I’m certainly not going to unilaterally
> do so. All I can promise you is that we continue to talk about this
> internally, and if we *unanimously* feel comfortable changing our policy
> we will do so. Until then, I’m happy to do my best to accommodate as
> many people as possible (which in this case I believe we have done).
> 
> Cory

Hi Cory,

Thanks for the above. However, it is still frustrating to not understand
your motivations, which was the only thing that pushed me to write these
lines (ie: still not trying to convince you...).

Thomas


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-15 Thread Thomas Goirand 
On 10/15/2015 11:20 AM, Dmitry Tantsur wrote:
> On 10/15/2015 12:18 AM, Robert Collins wrote:
>> On 15 October 2015 at 11:11, Thomas Goirand  wrote:
>>>
>>> One major pain point is unfortunately something ridiculously easy to
>>> fix, but which nobody seems to care about: the long & short descriptions
>>> format. These are usually buried into the setup.py black magic, which by
>>> the way I feel is very unsafe (does PyPi actually execute "python
>>> setup.py" to find out about description texts? I hope they are running
>>> this in a sandbox...).
>>>
>>> Since everyone uses the fact that PyPi accepts RST format for the long
>>> description, there's nothing that can really easily fit the
>>> debian/control. Probably a rst2txt tool would help, but still, the long
>>> description would still be polluted with things like changelog, examples
>>> and such (damned, why people think it's the correct place to put
>>> that...).
>>>
>>> The only way I'd see to fix this situation, would be a PEP. This will
>>> probably take a decade to have everyone switching to a new correct way
>>> to write a long & short description...
>>
>> Perhaps Debian (1 thing) should change, rather than trying to change
>> all the upstreams packaged in it (>20K) :)
> 
> +1. Both README and PyPI are for users, and I personally find detailed
> descriptions (especially a couple of simple examples) on the PyPI page
> to be of so much value.

I do agree it is useful to have such thing in the PyPi pages. But it
doesn't change my opinion that it'd be nice if this was not mixed with
the long description in Python modules.

Thomas


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-14 Thread Thomas Goirand 
On 10/13/2015 09:41 AM, Cory Benfield wrote:
> 
>> On 13 Oct 2015, at 07:42, Thomas Goirand  wrote:
>> In this particular case (ie: a difficult upstream which makes it
>> impossible to have the same result with pip and system packages)
> 
> I don’t know how carefully you’ve followed this email trail

I did read carefully.

> but the “difficult upstream”

I do understand that you don't like being called this way, though this
is still the reality. Vendorizing still inflicting some major pain to a
lot of your users:
- This thread one of the demonstration of it.
- You having to contact downstream distros is another.
- The unbundling work inflicted to downstream package maintainers is a
3rd another.

So like it or not, it is a fact that it is difficult to work with
requests because of the way it is released upstream.

> has had a policy in place for six months
> that ensures that you can have the same result with pip and
> system packages. For six months we have only updated to versions
> of urllib3 that are actually released, and therefore that are
> definitely available from pip (and potentially available from
> the distribution).
> 
> The reason this has not been working is because the distributions,
> when they unbundle us, have not been populating their setup.py to
> reflect the dependency: only their own metadata. We’ve been in
> contact with them, and this change is being made in the
> distributions we have relationships with.

Though you could have avoid all of this pain if you were not bundling.
Isn't all of this make you re-think your vendorizing policy? Or still
not? I'm asking because I still didn't read your answer about the
important question: since you aren't using specially crafted versions of
urllib3 anymore, and now only using official releases, what's the reason
that keeps you vendorizing? Not trying to convince you here, just trying
to understand.

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-14 Thread Thomas Goirand 
On 10/13/2015 06:04 PM, Joshua Harlow wrote:
> Thomas Goirand wrote:
>> On 10/13/2015 12:44 AM, Joshua Harlow wrote:
>>> Anvil gets somewhat far on this, although its not supporting DEBs it
>>> does build its best attempt at RPMs building them automatically and
>>> turning git repos of projects into RPMs.
>>>
>>> http://anvil.readthedocs.org/en/latest/topics/summary.html (hopefully
>>> the existence of this is not news to folks).
>>>
>>> A log of this in action (very verbose) is at:
>>>
>>> http://logs.openstack.org/40/225240/4/check/gate-anvil-rpms-dsvm-devstack-centos7/0eea2a9/console.html
>>>
>>
>> Automation can only bring you so far. I also have automation which we
>> could use for debs (see the pkgos-debpypi script from the
>> openstack-pkg-tools package), however, there's always the need for
>> manual reviews. I don't believe it ever will be possible to do full
>> automation, as each Python package has specificities. Note that this is
>> mainly an issue with Python modules, if it was PHP pear packages, it
>> could be fully automated. So probably there's some PEP that we could
>> start to ease this. If only everyone was using testr, pbr, defining
>> copyright correctly and providing a parseable long and short
>> description, it wouldn't be such an issue.
> 
> Agreed, there will always be that damn 1% (ok its probably around 10%)
> of weird pypi packages that will require hand-tuning, the hope (and I
> think the reality) is that most actually don't require hand-tuning.

One major pain point is unfortunately something ridiculously easy to
fix, but which nobody seems to care about: the long & short descriptions
format. These are usually buried into the setup.py black magic, which by
the way I feel is very unsafe (does PyPi actually execute "python
setup.py" to find out about description texts? I hope they are running
this in a sandbox...).

Since everyone uses the fact that PyPi accepts RST format for the long
description, there's nothing that can really easily fit the
debian/control. Probably a rst2txt tool would help, but still, the long
description would still be polluted with things like changelog, examples
and such (damned, why people think it's the correct place to put that...).

The only way I'd see to fix this situation, would be a PEP. This will
probably take a decade to have everyone switching to a new correct way
to write a long & short description...

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-14 Thread Robert Collins 
On 15 October 2015 at 11:11, Thomas Goirand  wrote:
>
> One major pain point is unfortunately something ridiculously easy to
> fix, but which nobody seems to care about: the long & short descriptions
> format. These are usually buried into the setup.py black magic, which by
> the way I feel is very unsafe (does PyPi actually execute "python
> setup.py" to find out about description texts? I hope they are running
> this in a sandbox...).
>
> Since everyone uses the fact that PyPi accepts RST format for the long
> description, there's nothing that can really easily fit the
> debian/control. Probably a rst2txt tool would help, but still, the long
> description would still be polluted with things like changelog, examples
> and such (damned, why people think it's the correct place to put that...).
>
> The only way I'd see to fix this situation, would be a PEP. This will
> probably take a decade to have everyone switching to a new correct way
> to write a long & short description...

Perhaps Debian (1 thing) should change, rather than trying to change
all the upstreams packaged in it (>20K) :)

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-13 Thread Thomas Goirand 
On 10/12/2015 03:58 PM, Jeremy Stanley wrote:
> On 2015-10-12 15:40:48 +0200 (+0200), Thomas Goirand wrote:
> [...]
>> Has the infra team ever thought about doing that for (at least) all of
>> the 3rd party libs we use? I'd love to work closer with the infra team
>> to provide them with missing packages they would need, and I'm sure my
>> RPM buddy Haikel would too. This also would help getting our
>> openstack/{deb,rpm}- projects up to speed as well.
> [...]
> 
> Long ago there was an idea that we might somehow be able to
> near-instantly package anything on PyPI and serve RPMs and DEBs of
> it up to CI jobs, but doing that would be a pretty massive (and in
> my opinion very error-prone) undertaking.
> 
> Right now we can take advantage of the fact that the Python
> ecosystem uploads new releases to PyPI as their primary distribution
> channel. By using pip to install dependencies from PyPI in our CI
> system, we can pretty instantly test compatibility of our software
> with new releases of dependencies (much faster that they can get
> properly packaged in distros), and easily test different versions by
> proposing changes to the openstack/requirements repository.

In this particular case (ie: a difficult upstream which makes it
impossible to have the same result with pip and system packages), and
for a package which has a rather stable API, we could make an exception.

For the more general case, I'm not sure we are in such a hurry that we
would need to have updates by the minute. I would accept to commit
myself for a reasonable delay such as 48 hours max until a new
dependency is packaged and available for the infra team to use NEW
packages. For just updates, I'm convince it could be semi-automated
(like for example a proposal bot).

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-13 Thread Cory Benfield 

> On 13 Oct 2015, at 07:42, Thomas Goirand  wrote:
> In this particular case (ie: a difficult upstream which makes it
> impossible to have the same result with pip and system packages)

I don’t know how carefully you’ve followed this email trail, but the “difficult 
upstream” has had a policy in place for six months that ensures that you can 
have the same result with pip and system packages. For six months we have only 
updated to versions of urllib3 that are actually released, and therefore that 
are definitely available from pip (and potentially available from the 
distribution).

The reason this has not been working is because the distributions, when they 
unbundle us, have not been populating their setup.py to reflect the dependency: 
only their own metadata. We’ve been in contact with them, and this change is 
being made in the distributions we have relationships with.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-13 Thread Thomas Goirand 
On 10/13/2015 12:44 AM, Joshua Harlow wrote:
> Anvil gets somewhat far on this, although its not supporting DEBs it
> does build its best attempt at RPMs building them automatically and
> turning git repos of projects into RPMs.
> 
> http://anvil.readthedocs.org/en/latest/topics/summary.html (hopefully
> the existence of this is not news to folks).
> 
> A log of this in action (very verbose) is at:
> 
> http://logs.openstack.org/40/225240/4/check/gate-anvil-rpms-dsvm-devstack-centos7/0eea2a9/console.html

Automation can only bring you so far. I also have automation which we
could use for debs (see the pkgos-debpypi script from the
openstack-pkg-tools package), however, there's always the need for
manual reviews. I don't believe it ever will be possible to do full
automation, as each Python package has specificities. Note that this is
mainly an issue with Python modules, if it was PHP pear packages, it
could be fully automated. So probably there's some PEP that we could
start to ease this. If only everyone was using testr, pbr, defining
copyright correctly and providing a parseable long and short
description, it wouldn't be such an issue.

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-13 Thread Matthew Thode 
On 10/08/2015 07:39 PM, Robert Collins wrote:
> This is a bugbear that keeps cropping up and biting us. I'm hoping we
> can figure out a permanent fix.
> 
> The problem that occurs is the result of a few interacting things:
>  - requests has very very specific versions of urllib3 it works with.
> So specific they aren't always released yet.
> 
>  - Linux vendors often unbundle urllib3 from requests and then apply
> what patches were needed to their urllib3; while not updating their
> requests package dependencies to reflect this.
> 
>  - we use urllib3 in some places and requests in others (but we don't
> mix them up)
> 
>  - if for any reason we have a distro-altered requests + a
> pip-installed urllib3, requests will [usually] break... see the 'not
> always released yet' key thing above.
> 
> Now, there are lots of places this last thing can happen; they all
> depend on us having a dependency on requests that is compatible with
> the version installed by the distro, but a urllib3 dependency that
> triggers an upgrade of just urllib3. When constraints are in use, the
> requests version has to match the distro requests version exactly, but
> that will happen from time to time.
> 
> e.g.
> 
>  - dvsm test jobs where the base image already has python-requests
> installed in it
> 
>  - virtualenvs where system-site-packages are enabled
> 
> 
> There are a few strategies that have been proposed to fix this. AIUI they are:
>  - make sure none of our testing environments include distro requests 
> packages.
> 
>  - make our requirements be tightly matched to what requests needs to
> deal with the unbundling
> 
>  - teach pip how to identify and avoid this situation by always
> upgrading requests (even if thats just a re-install of the version
> from PyPI) when the installed requests is a distro installed version
> **and** urllib3 is being modified.
> 
>  - get the distros to stop un-vendoring urllib3
> 
> 
> The first one addresses the situation for the CI gate but doesn't
> avoid developers getting bitten on their local machines. And
> installing any distro thing that uses requests would re-instate the
> potential for breakage. So while its not harmful, I don't think its
> sufficient to make this go away.
> 
> The second is trivially insufficient - anytime requests vendored
> urllib3 is not precisely identical to a released urllib3, it becomes
> impossible to satisfy that via dependency version pinning - the only
> way to satisfy it is with the urllib3 in the distro that has whatever
> change was needed included.
> 
> The third approach will require some negotiation I suspect - because
> its aesthetically wrong: from an upstream perspective urllib3 is
> independent of requests, and vice-versa, but from a distro perspective
> they are tightly coupled, no variation permitted.
> 
> The fourth approach meets the stone wall of 'but security' and 'no
> redundancy permitted' - I don't have the energy to try and get through
> the near-religious mindset I've encountered there before, though hey -
> if Fedora and Debian and Ubuntu folk are all interested in figuring
> out a sustainable way forward, that would be great: please don't feel
> cut out, I'm just not expecting anything.
> 
> If there are other approaches, great - please throw them up here.
> 
> -Rob
> 

At least for my packaging for Gentoo I haven't run into any issues here.
We have multiple versions of both requests and urllib3 available and I
can add a missing version if needed.

-- 
Matthew Thode (prometheanfire)



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-13 Thread Joshua Harlow 

Thomas Goirand wrote:

On 10/13/2015 12:44 AM, Joshua Harlow wrote:

Anvil gets somewhat far on this, although its not supporting DEBs it
does build its best attempt at RPMs building them automatically and
turning git repos of projects into RPMs.

http://anvil.readthedocs.org/en/latest/topics/summary.html (hopefully
the existence of this is not news to folks).

A log of this in action (very verbose) is at:

http://logs.openstack.org/40/225240/4/check/gate-anvil-rpms-dsvm-devstack-centos7/0eea2a9/console.html


Automation can only bring you so far. I also have automation which we
could use for debs (see the pkgos-debpypi script from the
openstack-pkg-tools package), however, there's always the need for
manual reviews. I don't believe it ever will be possible to do full
automation, as each Python package has specificities. Note that this is
mainly an issue with Python modules, if it was PHP pear packages, it
could be fully automated. So probably there's some PEP that we could
start to ease this. If only everyone was using testr, pbr, defining
copyright correctly and providing a parseable long and short
description, it wouldn't be such an issue.


Agreed, there will always be that damn 1% (ok its probably around 10%) 
of weird pypi packages that will require hand-tuning, the hope (and I 
think the reality) is that most actually don't require hand-tuning.


Maybe someday it will be down to 0% (one can hope).



Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-12 Thread Cory Benfield 

> On 12 Oct 2015, at 01:16, Robert Collins  wrote:
> To sum up the thread, it sounds to me like a viable way-forward is:
> 
> - get distros to fixup their requests Python dependencies (and
> hopefully they can update that in stable releases).
> - fix the existing known bugs in pip where such accurate dependencies
> are violated by some operations.

Agreed.

And we’re taking this pretty seriously at requests: we got in touch with our 
downstream unbundlers at Debian and Fedora asking them to make sure they 
populate setup.py correctly. Fedora has created updates for F21, F22, and F23:

- https://bodhi.fedoraproject.org/updates/FEDORA-2015-20de3774f4
- https://bodhi.fedoraproject.org/updates/FEDORA-2015-1f580ccfa4
- https://bodhi.fedoraproject.org/updates/FEDORA-2015-d7c710a812

We’ve also heard positive noises from our Debian packager, though I don’t yet 
have a link to any change there.

If Ubuntu has a separate downstream packager from Debian, we don’t have a 
relationship with them, so it’s harder for us to affect change there.

This should mean that we’re only missing the pip fix.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-12 Thread Thomas Goirand 
On 10/12/2015 02:16 AM, Robert Collins wrote:
> On 10 October 2015 at 02:58, Cory Benfield  wrote:
>>
>>> On 9 Oct 2015, at 14:40, William M Edmonds  wrote:
>>>
>>> Cory Benfield  writes:
> The problem that occurs is the result of a few interacting things:
>  - requests has very very specific versions of urllib3 it works with.
> So specific they aren't always released yet.

 This should no longer be true. Our downstream redistributors pointedout to 
 us
 that this  was making their lives harder than they needed to be, so it's 
 now
 our policy to only  update to actual release versions of urllib3.
>>>
>>> That's great... except that I'm confused as to why requests would continue 
>>> to repackage urllib3 if that's the case. Why not just prereq the version of 
>>> urllib3 that it needs? I thought the one and only answer to that question 
>>> had been so that requests could package non-standard versions.
>>>
>>
>> That is not and was never the only reason for vendoring urllib3. However, 
>> and I cannot stress this enough, the decision to vendor urllib3 is *not 
>> going to be changed on this thread*. If and when it changes, it will be by 
>> consensus decision from the requests maintenance team, which we do not have 
>> at this time.
>>
>> Further, as I pointed out to Donald Stufft on IRC, if requests unbundled 
>> urllib3 *today* that would not fix the problem. The reason is that we’d 
>> specify our urllib3 dependency as: urllib3>=1.12,<1.13. This dependency note 
>> would still cause exactly the problem observed in this thread.
> 
> Actually, that would fix the problem (in conjunction with a fix to
> https://github.com/pypa/pip/issues/2687 - which is conceptually
> trivial once 988 is fixed).
> 
>> As you correctly identify in your subsequent email, William, the core 
>> problem is mixing of packages from distributions and PyPI. This happens with 
>> any tool with external dependencies: if you subsequently install a different 
>> version of a dependency using a packaging tool that is not aware of some of 
>> the dependency tree, it is entirely plausible that an incompatible version 
>> will be installed. It’s not hard to trigger this kind of thing on Ubuntu. 
>> IMO, what OpenStack needs is a decision about where it’s getting its 
>> packages from, and then to refuse to mix the two.
> 
> We can't do that for all our downstreams. Further, Ubuntu preserve
> dependency information - I think a key underlying issue is that they
> don't fix up the dependency data for requests when they alter it. I've
> filed https://bugs.launchpad.net/ubuntu/+source/python-requests/+bug/1505039
> to complement the one filed on Fedora earlier in this thread.

Well, since Ubuntu uses the Debian package simply by syncing from
Debian, I would suggest to file a bug against the Debian package instead
[1].

> Obviously a trivial workaround is to always use virtualenvs and not
> system-site-packages.

or opposite way... always use system-site-packages! :)

Has the infra team ever thought about doing that for (at least) all of
the 3rd party libs we use? I'd love to work closer with the infra team
to provide them with missing packages they would need, and I'm sure my
RPM buddy Haikel would too. This also would help getting our
openstack/{deb,rpm}- projects up to speed as well.

> To sum up the thread, it sounds to me like a viable way-forward is:
> 
>  - get distros to fixup their requests Python dependencies (and
> hopefully they can update that in stable releases).

The maintainer for both urllib3 and requests is a single person, so I'm
quite sure he is aware of the issue, and make sure that both packages
are compatible. Otherwise, opening bugs in Debian [1] to have him fix it
would obviously work. I don't believe that adding version upper-bounds
in the deb package would solve anything, we just need to not make
incompatible versions co-exist at a given moment in the distro.

Cheers,

Thomas Goirand (zigo)

[1] https://www.debian.org/Bugs/Reporting (note: I know Robert knows how
to file a bug in Debian, so this is obviously not for him that I'm
giving this URL... :) )


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-12 Thread Thomas Goirand 
Note: it's not my intention to restart a flame war about vendorizing of
urllib3 in requests, however, I can't let you write wrong things, and
the point of this message is only that, not discussing if requests
should stop vendorizing (I've given up a long time ago any attempt to
convince upstream about this...).

On 10/09/2015 09:21 AM, Cory Benfield wrote:
> Additionally, getting *all* of
> Fedora/Debian/Ubuntu on board with not unbundling requests is about as likely
> as hell freezing over.

Debian + Ubuntu is only a single maintainer (Daniele Tricoli, plus the
Debian Python Module Team), and he will (for very valid reasons) not
want to do it. I support his decision.

The issue here isn't downstream distros though, but an upstream who
don't want to care about downstream. We're only trying to deal with
this, let's not spread the propaganda that the issue is in downstream
distros: it is *not*.

On 10/09/2015 03:58 PM, Cory Benfield wrote:
> As you correctly identify in your subsequent email, William, the
> core problem is mixing of packages from distributions and PyPI.

It's not. The core problem is vendorizing. It would work perfectly to do
such mix-up otherwise (see below).

On 10/09/2015 03:58 PM, Cory Benfield wrote:
> This happens with any tool with external dependencies: if you
> subsequently install a different version of a dependency using a
> packaging tool that is not aware of some of the dependency tree, it
> is entirely plausible that an incompatible version will be installed.
> It’s not hard to trigger this kind of thing on Ubuntu. IMO, what
> OpenStack needs is a decision about where it’s getting its packages
> from, and then to refuse to mix the two.

I regret to say it in a so direct way, but this is simply false. Pip
install'ed packages very easily co-exist with system install'ed
packages, because there's a central registry composed the a collection
of egg-info files. Just, the virtualenv will have precedence over the
system version. But if no package does vendorizing, mixing system
packages and virtualenv with pip install works perfectly. There's only
one issue with this: Python 3 and namespace. This is by the way one of
the reason we removed the namespace from the Oslo libs, it didn't work
well for Python 3 and namespace to run tests for distributions.

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-12 Thread Thomas Goirand 
On 10/09/2015 02:39 AM, Robert Collins wrote:
> - get the distros to stop un-vendoring urllib3

I'm not the package maintainer of requests, but I know that Barry
Warsaw, the actual maintainer, will not want to do that.

The other solutions which you didn't mention:

1- Stop using a vendorized version of requests, and fork that project to
make it use dependencies at it should be from the start.

2- Convince upstream to stop vendorizing urllib3 and work more with
upstream of urllib3 to have them release when they need it.

3- Always use the distro version of requests, never the one from venv

While 1- is not realistic (unless someone volunteers), and we already
tried 2- without luck, 3- can happen easily.

BTW, the same applies for tablib which is in a even more horrible state
that makes it impossible to package with Py3 support. But tablib could
be removed from our (build-)dependency list, if someone cares about
re-writing cliff-tablib, which IMO wouldn't be that much work. Doug, how
many beers shall I offer you for that work? :)

Just my 2 cents hopping to provide some contrib in this discussion,
Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-12 Thread Thomas Goirand 
On 10/09/2015 03:39 PM, William M Edmonds wrote:
> When you're using a distro, you're always going to have to worry about
> someone pip installing something that conflicts with the rpm, no?

The point of this thread is: no you don't. You do only if some (bad)
upstream decide to vendorize.

> Unless the distros have a way to put in protection against
> this, preventing pip install of something that is already installed by RPM?

Well, the point of pip is so that you can use it to install a package
which may already be installed in the system, but you want another
version, and then they both co-exist without an issue.

I don't think we want to remove this feature (or at least, we'd need to
have a kind of global switch for that if we were to implement it in pip).

>>  - make sure none of our testing environments include distro
>> requests packages.
> 
> It's not like requests is an unusual package for someone to have
> installed from their distro in a base OS image. So when they take that
> base OS and go to setup OpenStack, they'll be hitting this case, whether
> we tested it or not. So while not testing this case seems nice from a
> development perspective, it doesn't seem to fit real-world usage. I
> don't think it would make operators very happy.

This thread has nothing to do with operators. Operators typically
install from distro packages only (unless they do like Helion does,
which is pretty rare...) and wouldn't be affected by this problem.

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-12 Thread Jeremy Stanley 
On 2015-10-12 15:40:48 +0200 (+0200), Thomas Goirand wrote:
[...]
> Has the infra team ever thought about doing that for (at least) all of
> the 3rd party libs we use? I'd love to work closer with the infra team
> to provide them with missing packages they would need, and I'm sure my
> RPM buddy Haikel would too. This also would help getting our
> openstack/{deb,rpm}- projects up to speed as well.
[...]

Long ago there was an idea that we might somehow be able to
near-instantly package anything on PyPI and serve RPMs and DEBs of
it up to CI jobs, but doing that would be a pretty massive (and in
my opinion very error-prone) undertaking.

Right now we can take advantage of the fact that the Python
ecosystem uploads new releases to PyPI as their primary distribution
channel. By using pip to install dependencies from PyPI in our CI
system, we can pretty instantly test compatibility of our software
with new releases of dependencies (much faster that they can get
properly packaged in distros), and easily test different versions by
proposing changes to the openstack/requirements repository.

The only way I see this changing is if authors of Python libraries
switch to packaging their own software for major distributions and
have a pass to get them included by those distributions
near-instantly. Also, distros would have to cease caring about
reducing the number of concurrent versions of libraries they provide
(and I posit that as soon as debian/sid ships a DEB for every
version ever released for packages like python-requests and
python-urllib3, apt-get will begin to exhibit similar dependency
resolution challenges).
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-12 Thread Steve Baker 

On 13/10/15 02:05, Thomas Goirand wrote:


BTW, the same applies for tablib which is in a even more horrible state
that makes it impossible to package with Py3 support. But tablib could
be removed from our (build-)dependency list, if someone cares about
re-writing cliff-tablib, which IMO wouldn't be that much work. Doug, how
many beers shall I offer you for that work? :)

Regarding tablib, cliff has had its own table formatter for some time, 
and now has its own json and yaml formatters. I believe the only tablib 
formatter left is the HTML one, which likely wouldn't be missed if it 
was just dropped (or it could be simply reimplemented inside cliff).


If the cliff deb depends on cliff-tablib I would recommend removing that 
dependency and just stop packaging cliff-tablib.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-12 Thread Joshua Harlow 

Jeremy Stanley wrote:

On 2015-10-12 15:40:48 +0200 (+0200), Thomas Goirand wrote:
[...]

Has the infra team ever thought about doing that for (at least) all of
the 3rd party libs we use? I'd love to work closer with the infra team
to provide them with missing packages they would need, and I'm sure my
RPM buddy Haikel would too. This also would help getting our
openstack/{deb,rpm}- projects up to speed as well.

[...]

Long ago there was an idea that we might somehow be able to
near-instantly package anything on PyPI and serve RPMs and DEBs of
it up to CI jobs, but doing that would be a pretty massive (and in
my opinion very error-prone) undertaking.


Anvil gets somewhat far on this, although its not supporting DEBs it 
does build its best attempt at RPMs building them automatically and 
turning git repos of projects into RPMs.


http://anvil.readthedocs.org/en/latest/topics/summary.html (hopefully 
the existence of this is not news to folks).


A log of this in action (very verbose) is at:

http://logs.openstack.org/40/225240/4/check/gate-anvil-rpms-dsvm-devstack-centos7/0eea2a9/console.html

Feel free to ask questions as to how this is being done on 
#openstack-anvil or find me on other channels... I believe it is also 
one of the goals of the rpm and packaging teams to do something similar, 
although that might be a ways off?




Right now we can take advantage of the fact that the Python
ecosystem uploads new releases to PyPI as their primary distribution
channel. By using pip to install dependencies from PyPI in our CI
system, we can pretty instantly test compatibility of our software
with new releases of dependencies (much faster that they can get
properly packaged in distros), and easily test different versions by
proposing changes to the openstack/requirements repository.

The only way I see this changing is if authors of Python libraries
switch to packaging their own software for major distributions and
have a pass to get them included by those distributions
near-instantly. Also, distros would have to cease caring about
reducing the number of concurrent versions of libraries they provide
(and I posit that as soon as debian/sid ships a DEB for every
version ever released for packages like python-requests and
python-urllib3, apt-get will begin to exhibit similar dependency
resolution challenges).


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-11 Thread Robert Collins 
On 10 October 2015 at 02:58, Cory Benfield  wrote:
>
>> On 9 Oct 2015, at 14:40, William M Edmonds  wrote:
>>
>> Cory Benfield  writes:
>> > > The problem that occurs is the result of a few interacting things:
>> > >  - requests has very very specific versions of urllib3 it works with.
>> > > So specific they aren't always released yet.
>> >
>> > This should no longer be true. Our downstream redistributors pointedout to 
>> > us
>> > that this  was making their lives harder than they needed to be, so it's 
>> > now
>> > our policy to only  update to actual release versions of urllib3.
>>
>> That's great... except that I'm confused as to why requests would continue 
>> to repackage urllib3 if that's the case. Why not just prereq the version of 
>> urllib3 that it needs? I thought the one and only answer to that question 
>> had been so that requests could package non-standard versions.
>>
>
> That is not and was never the only reason for vendoring urllib3. However, and 
> I cannot stress this enough, the decision to vendor urllib3 is *not going to 
> be changed on this thread*. If and when it changes, it will be by consensus 
> decision from the requests maintenance team, which we do not have at this 
> time.
>
> Further, as I pointed out to Donald Stufft on IRC, if requests unbundled 
> urllib3 *today* that would not fix the problem. The reason is that we’d 
> specify our urllib3 dependency as: urllib3>=1.12,<1.13. This dependency note 
> would still cause exactly the problem observed in this thread.

Actually, that would fix the problem (in conjunction with a fix to
https://github.com/pypa/pip/issues/2687 - which is conceptually
trivial once 988 is fixed).

> As you correctly identify in your subsequent email, William, the core problem 
> is mixing of packages from distributions and PyPI. This happens with any tool 
> with external dependencies: if you subsequently install a different version 
> of a dependency using a packaging tool that is not aware of some of the 
> dependency tree, it is entirely plausible that an incompatible version will 
> be installed. It’s not hard to trigger this kind of thing on Ubuntu. IMO, 
> what OpenStack needs is a decision about where it’s getting its packages 
> from, and then to refuse to mix the two.

We can't do that for all our downstreams. Further, Ubuntu preserve
dependency information - I think a key underlying issue is that they
don't fix up the dependency data for requests when they alter it. I've
filed https://bugs.launchpad.net/ubuntu/+source/python-requests/+bug/1505039
to complement the one filed on Fedora earlier in this thread.

*We* have the privilege of working directly with folk like libvirt
that have been problematic in the past and getting those things
addressed, so that we can run in a virtualenv happily. But we can't
insist that user X who want to use some openstack library that uses
requests but also some other thing (maybe some SWIG binding or
something). So framing this as being driven by the mix is false: the
thing that drives this is a combination - e.g. coming in part from
defects in pip, and the existence of things that can't be installed in
virtualenvs.

Obviously a trivial workaround is to always use virtualenvs and not
system-site-packages.

To sum up the thread, it sounds to me like a viable way-forward is:

 - get distros to fixup their requests Python dependencies (and
hopefully they can update that in stable releases).
 - fix the existing known bugs in pip where such accurate dependencies
are violated by some operations.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-09 Thread Cory Benfield 
Robert Collins  writes:
> The problem that occurs is the result of a few interacting things:
>  - requests has very very specific versions of urllib3 it works with.
> So specific they aren't always released yet.

This should no longer be true. Our downstream redistributors pointed out to us
that this  was making their lives harder than they needed to be, so it's now
our policy to only  update to actual release versions of urllib3.
 
> The second is trivially insufficient - anytime requests vendored
> urllib3 is not precisely identical to a released urllib3, it becomes
> impossible to satisfy that via dependency version pinning - the only
> way to satisfy it is with the urllib3 in the distro that has whatever
> change was needed included.

Per my note above, if we restrict ourselves to relatively recent versions of
requests  (2.7.3+ IIRC) we should be fine. Of course, that doesn't mean we can
actually do that...

> The fourth approach meets the stone wall of 'but security' and 'no
> redundancy permitted' - I don't have the energy to try and get through
> the near-religious mindset I've encountered there before, though hey -
> if Fedora and Debian and Ubuntu folk are all interested in figuring
> out a sustainable way forward, that would be great: please don't feel
> cut out, I'm just not expecting anything.

It should be assumed that approach number four is a non-starter. This list has
had that  conversation before, which was a stunningly unpleasant experience for
me and not one I  want to repeat. Additionally, getting *all* of
Fedora/Debian/Ubuntu on board with not unbundling requests is about as likely
as hell freezing over.

Cory


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-09 Thread Jeremy Stanley 
On 2015-10-09 14:58:36 +0100 (+0100), Cory Benfield wrote:
[...]
> IMO, what OpenStack needs is a decision about where it’s getting
> its packages from, and then to refuse to mix the two.

I have yet to find a Python-based operating system installable in
whole via pip. There will always be _at_least_some_ packages you
install from your operating system's package management. What you
seem to be missing is that Linux distros are now shipping base
images which include their python-requests and python-urllib3
packages already pre-installed as dependencies of Python-based tools
they deem important to their users.

To work around this in our test infrastructure we're effectively
abandoning all hope of using distro-provided server images, and
building our own from scratch to avoid the possibility that they may
bring with them their own versions of any Python libraries
whatsoever. We're at the point where we're basically maintaining our
own derivative Linux distributions. The web of dependencies in
OpenStack has reached a level of complexity where it's guaranteed to
overlap with just about any pre-installed python-.* packages in a
distro-supplied image.

We're only now reaching the point where our Python dependencies
actually all function within the context of a virtualenv without
needing system-site-packages contamination, so the next logical step
is probably to see if virtualenv isolation is possible for
frameworks like DevStack (the QA team may already be trying to
figure that out, I'm not sure).
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-09 Thread Cory Benfield 

> On 9 Oct 2015, at 14:40, William M Edmonds  wrote:
> 
> Cory Benfield  writes:
> > > The problem that occurs is the result of a few interacting things:
> > >  - requests has very very specific versions of urllib3 it works with.
> > > So specific they aren't always released yet.
> >
> > This should no longer be true. Our downstream redistributors pointedout to 
> > us
> > that this  was making their lives harder than they needed to be, so it's now
> > our policy to only  update to actual release versions of urllib3.
> 
> That's great... except that I'm confused as to why requests would continue to 
> repackage urllib3 if that's the case. Why not just prereq the version of 
> urllib3 that it needs? I thought the one and only answer to that question had 
> been so that requests could package non-standard versions.
> 

That is not and was never the only reason for vendoring urllib3. However, and I 
cannot stress this enough, the decision to vendor urllib3 is *not going to be 
changed on this thread*. If and when it changes, it will be by consensus 
decision from the requests maintenance team, which we do not have at this time.

Further, as I pointed out to Donald Stufft on IRC, if requests unbundled 
urllib3 *today* that would not fix the problem. The reason is that we’d specify 
our urllib3 dependency as: urllib3>=1.12,<1.13. This dependency note would 
still cause exactly the problem observed in this thread.

As you correctly identify in your subsequent email, William, the core problem 
is mixing of packages from distributions and PyPI. This happens with any tool 
with external dependencies: if you subsequently install a different version of 
a dependency using a packaging tool that is not aware of some of the dependency 
tree, it is entirely plausible that an incompatible version will be installed. 
It’s not hard to trigger this kind of thing on Ubuntu. IMO, what OpenStack 
needs is a decision about where it’s getting its packages from, and then to 
refuse to mix the two.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-09 Thread William M Edmonds 

Cory Benfield  writes:
> > The problem that occurs is the result of a few interacting things:
> >  - requests has very very specific versions of urllib3 it works with.
> > So specific they aren't always released yet.
>
> This should no longer be true. Our downstream redistributors pointedout
to us
> that this  was making their lives harder than they needed to be, so it's
now
> our policy to only  update to actual release versions of urllib3.

That's great... except that I'm confused as to why requests would continue
to repackage urllib3 if that's the case. Why not just prereq the version of
urllib3 that it needs? I thought the one and only answer to that question
had been so that requests could package non-standard versions.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-09 Thread William M Edmonds 

Robert Collins  writes:
>  - Linux vendors often unbundle urllib3 from requests and then apply
> what patches were needed to their urllib3; while not updating their
> requests package dependencies to reflect this.

I opened a bug on Fedora for them to update their requests package
dependencies. See https://bugzilla.redhat.com/show_bug.cgi?id=1253823. Of
course that may continue to be an issue on older versions and other
distros.

>  - if for any reason we have a distro-altered requests + a
> pip-installed urllib3, requests will [usually] break... see the 'not
> always released yet' key thing above.
>
> Now, there are lots of places this last thing can happen; they all
> depend on us having a dependency on requests that is compatible with
> the version installed by the distro, but a urllib3 dependency that
> triggers an upgrade of just urllib3. When constraints are in use, the
> requests version has to match the distro requests version exactly, but
> that will happen from time to time.

When you're using a distro, you're always going to have to worry about
someone pip installing something that conflicts with the rpm, no? That
could be for any reason, could be completely unrelated to OpenStack
dependencies. Unless the distros have a way to put in protection against
this, preventing pip install of something that is already installed by RPM?

>  - make sure none of our testing environments include distro
> requests packages.

It's not like requests is an unusual package for someone to have installed
from their distro in a base OS image. So when they take that base OS and go
to setup OpenStack, they'll be hitting this case, whether we tested it or
not. So while not testing this case seems nice from a development
perspective, it doesn't seem to fit real-world usage. I don't think it
would make operators very happy.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-09 Thread Robert Collins 
On 10 October 2015 at 03:57, Cory Benfield  wrote:
>
>> On 9 Oct 2015, at 15:18, Jeremy Stanley  wrote:
>>
>> On 2015-10-09 14:58:36 +0100 (+0100), Cory Benfield wrote:
>> [...]
>>> IMO, what OpenStack needs is a decision about where it’s getting
>>> its packages from, and then to refuse to mix the two.
>>
>> I have yet to find a Python-based operating system installable in
>> whole via pip. There will always be _at_least_some_ packages you
>> install from your operating system's package management. What you
>> seem to be missing is that Linux distros are now shipping base
>> images which include their python-requests and python-urllib3
>> packages already pre-installed as dependencies of Python-based tools
>> they deem important to their users.
>>
>
> Yeah, this has been an ongoing problem.
>
> For my part, Donald Stufft has informed me that if the distribution-provided 
> requests package has the appropriate install_requires field in its setup.py, 
> pip will respect that dependency.

It should but it won't :).

https://github.com/pypa/pip/issues/2687
and
https://github.com/pypa/pip/issues/988

The first one means that if someone does 'pip install -U urllib3' and
an unbundled requests with appropriate pin on urllib3 is already
installed, that pip will happily upgrade urllib3, breaking requests,
without complaining. It is fixable (with correct metadata of course).

The second one means that if anything - another package, or the user
via direct mention or requirements/constraints files - specifies a
urllib3 dependency (of any sort) then the requests dependency will be
silently ignored.

Both of these will be solved in the medium future - we're now at the
point of having POC branches, and once we've finished with the
constraints rollout and PEP-426 marker polish will be moving onto the
resolver work.

> Given that requests has recently switched to not providing mid-cycle urllib3 
> versions, it should be entirely possible for downstream redistributors in 
> Debian/Fedora to put that metadata into their packages when they unbundle 
> requests. I’m chasing up with our downstream redistributors right now to ask 
> them to start doing that.
>
> This should resolve the problem for systems where requests 2.7.0 or higher 
> are being used. In other systems, this problem still exists and cannot be 
> fixed by requests directly.

Well, if we get to a future where it is in-principle fixed, I'll be happy.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-09 Thread Joshua Harlow 
For those who are interested in more of the historical aspect around this,

https://github.com/kennethreitz/requests/issues/1811

https://github.com/kennethreitz/requests/pull/1812

My own thoughts are varied here, I get the angle of vendoring, but I don't get 
the resistance to unvendoring it (which it seems like quite a few people have 
asked for); if many people want it unvendored then this just ends up creating a 
bad taste in the mouth of many people (this is a bad thing to have happen in 
opensource and is how forks and such get created...).

But as was stated,

The decision to stop vendoring it likely won't be made here anyway ;)

From: c...@lukasa.co.uk
Date: Fri, 9 Oct 2015 14:58:36 +0100
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] Requests + urllib3 + distro packages

 
> On 9 Oct 2015, at 14:40, William M Edmonds  wrote:
> 
> Cory Benfield  writes:
>>> The problem that occurs is the result of a few interacting things:
>>>  - requests has very very specific versions of urllib3 it works with.
>>> So specific they aren't always released yet.
>>
>> This should no longer be true. Our downstream redistributors pointedout to us
>> that this  was making their lives harder than they needed to be, so it's now
>> our policy to only  update to actual release versions of urllib3.
> 
> That's great... except that I'm confused as to why requests would continue to 
> repackage urllib3 if that's the case. Why not just prereq the version of 
> urllib3 that it needs? I thought the one and only answer to that question had 
> been so that requests could package non-standard versions.
> 
 
That is not and was never the only reason for vendoring urllib3. However, and I 
cannot stress this enough, the decision to vendor urllib3 is *not going to be 
changed on this thread*. If and when it changes, it will be by consensus 
decision from the requests maintenance team, which we do not have at this time.
 
Further, as I pointed out to Donald Stufft on IRC, if requests unbundled 
urllib3 *today* that would not fix the problem. The reason is that we’d specify 
our urllib3 dependency as: urllib3>=1.12,   
 __
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-09 Thread Cory Benfield 

> On 9 Oct 2015, at 15:18, Jeremy Stanley  wrote:
> 
> On 2015-10-09 14:58:36 +0100 (+0100), Cory Benfield wrote:
> [...]
>> IMO, what OpenStack needs is a decision about where it’s getting
>> its packages from, and then to refuse to mix the two.
> 
> I have yet to find a Python-based operating system installable in
> whole via pip. There will always be _at_least_some_ packages you
> install from your operating system's package management. What you
> seem to be missing is that Linux distros are now shipping base
> images which include their python-requests and python-urllib3
> packages already pre-installed as dependencies of Python-based tools
> they deem important to their users.
> 

Yeah, this has been an ongoing problem.

For my part, Donald Stufft has informed me that if the distribution-provided 
requests package has the appropriate install_requires field in its setup.py, 
pip will respect that dependency. Given that requests has recently switched to 
not providing mid-cycle urllib3 versions, it should be entirely possible for 
downstream redistributors in Debian/Fedora to put that metadata into their 
packages when they unbundle requests. I’m chasing up with our downstream 
redistributors right now to ask them to start doing that.

This should resolve the problem for systems where requests 2.7.0 or higher are 
being used. In other systems, this problem still exists and cannot be fixed by 
requests directly.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-08 Thread Monty Taylor 

On 10/08/2015 08:39 PM, Robert Collins wrote:

This is a bugbear that keeps cropping up and biting us. I'm hoping we
can figure out a permanent fix.

The problem that occurs is the result of a few interacting things:
  - requests has very very specific versions of urllib3 it works with.
So specific they aren't always released yet.

  - Linux vendors often unbundle urllib3 from requests and then apply
what patches were needed to their urllib3; while not updating their
requests package dependencies to reflect this.

  - we use urllib3 in some places and requests in others (but we don't
mix them up)

  - if for any reason we have a distro-altered requests + a
pip-installed urllib3, requests will [usually] break... see the 'not
always released yet' key thing above.

Now, there are lots of places this last thing can happen; they all
depend on us having a dependency on requests that is compatible with
the version installed by the distro, but a urllib3 dependency that
triggers an upgrade of just urllib3. When constraints are in use, the
requests version has to match the distro requests version exactly, but
that will happen from time to time.

e.g.

  - dvsm test jobs where the base image already has python-requests
installed in it


We're working hard to get to the point where this one goes away, fwiw.


  - virtualenvs where system-site-packages are enabled


These make the easter bunny have sad.


There are a few strategies that have been proposed to fix this. AIUI they are:
  - make sure none of our testing environments include distro requests packages.


yes!


  - make our requirements be tightly matched to what requests needs to
deal with the unbundling

  - teach pip how to identify and avoid this situation by always
upgrading requests (even if thats just a re-install of the version
from PyPI) when the installed requests is a distro installed version
**and** urllib3 is being modified.

  - get the distros to stop un-vendoring urllib3


The first one addresses the situation for the CI gate but doesn't
avoid developers getting bitten on their local machines. And
installing any distro thing that uses requests would re-instate the
potential for breakage. So while its not harmful, I don't think its
sufficient to make this go away.

The second is trivially insufficient - anytime requests vendored
urllib3 is not precisely identical to a released urllib3, it becomes
impossible to satisfy that via dependency version pinning - the only
way to satisfy it is with the urllib3 in the distro that has whatever
change was needed included.

The third approach will require some negotiation I suspect - because
its aesthetically wrong: from an upstream perspective urllib3 is
independent of requests, and vice-versa, but from a distro perspective
they are tightly coupled, no variation permitted.

The fourth approach meets the stone wall of 'but security' and 'no
redundancy permitted' - I don't have the energy to try and get through
the near-religious mindset I've encountered there before, though hey -
if Fedora and Debian and Ubuntu folk are all interested in figuring
out a sustainable way forward, that would be great: please don't feel
cut out, I'm just not expecting anything.

If there are other approaches, great - please throw them up here.


I've got nothing. I'll continue hacking on #1 just because GATE. But I 
agree, it's necessary but not sufficient.


Thanks for the writeup.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

2015-10-08 Thread Matt Riedemann 



On 10/8/2015 7:57 PM, Monty Taylor wrote:

On 10/08/2015 08:39 PM, Robert Collins wrote:

This is a bugbear that keeps cropping up and biting us. I'm hoping we
can figure out a permanent fix.

The problem that occurs is the result of a few interacting things:
  - requests has very very specific versions of urllib3 it works with.
So specific they aren't always released yet.

  - Linux vendors often unbundle urllib3 from requests and then apply
what patches were needed to their urllib3; while not updating their
requests package dependencies to reflect this.

  - we use urllib3 in some places and requests in others (but we don't
mix them up)

  - if for any reason we have a distro-altered requests + a
pip-installed urllib3, requests will [usually] break... see the 'not
always released yet' key thing above.

Now, there are lots of places this last thing can happen; they all
depend on us having a dependency on requests that is compatible with
the version installed by the distro, but a urllib3 dependency that
triggers an upgrade of just urllib3. When constraints are in use, the
requests version has to match the distro requests version exactly, but
that will happen from time to time.

e.g.

  - dvsm test jobs where the base image already has python-requests
installed in it


We're working hard to get to the point where this one goes away, fwiw.


  - virtualenvs where system-site-packages are enabled


These make the easter bunny have sad.


There are a few strategies that have been proposed to fix this. AIUI
they are:
  - make sure none of our testing environments include distro requests
packages.


yes!


  - make our requirements be tightly matched to what requests needs to
deal with the unbundling

  - teach pip how to identify and avoid this situation by always
upgrading requests (even if thats just a re-install of the version
from PyPI) when the installed requests is a distro installed version
**and** urllib3 is being modified.

  - get the distros to stop un-vendoring urllib3


The first one addresses the situation for the CI gate but doesn't
avoid developers getting bitten on their local machines. And
installing any distro thing that uses requests would re-instate the
potential for breakage. So while its not harmful, I don't think its
sufficient to make this go away.

The second is trivially insufficient - anytime requests vendored
urllib3 is not precisely identical to a released urllib3, it becomes
impossible to satisfy that via dependency version pinning - the only
way to satisfy it is with the urllib3 in the distro that has whatever
change was needed included.

The third approach will require some negotiation I suspect - because
its aesthetically wrong: from an upstream perspective urllib3 is
independent of requests, and vice-versa, but from a distro perspective
they are tightly coupled, no variation permitted.

The fourth approach meets the stone wall of 'but security' and 'no
redundancy permitted' - I don't have the energy to try and get through
the near-religious mindset I've encountered there before, though hey -
if Fedora and Debian and Ubuntu folk are all interested in figuring
out a sustainable way forward, that would be great: please don't feel
cut out, I'm just not expecting anything.

If there are other approaches, great - please throw them up here.


I've got nothing. I'll continue hacking on #1 just because GATE. But I
agree, it's necessary but not sufficient.

Thanks for the writeup.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



FYI, related change that is triggering the conversation:

https://review.openstack.org/#/c/213310/

And there are related bugs in there with more details on other ways this 
fails for people outside the gate system.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Requests + urllib3 + distro packages

2015-10-08 Thread Robert Collins 
This is a bugbear that keeps cropping up and biting us. I'm hoping we
can figure out a permanent fix.

The problem that occurs is the result of a few interacting things:
 - requests has very very specific versions of urllib3 it works with.
So specific they aren't always released yet.

 - Linux vendors often unbundle urllib3 from requests and then apply
what patches were needed to their urllib3; while not updating their
requests package dependencies to reflect this.

 - we use urllib3 in some places and requests in others (but we don't
mix them up)

 - if for any reason we have a distro-altered requests + a
pip-installed urllib3, requests will [usually] break... see the 'not
always released yet' key thing above.

Now, there are lots of places this last thing can happen; they all
depend on us having a dependency on requests that is compatible with
the version installed by the distro, but a urllib3 dependency that
triggers an upgrade of just urllib3. When constraints are in use, the
requests version has to match the distro requests version exactly, but
that will happen from time to time.

e.g.

 - dvsm test jobs where the base image already has python-requests
installed in it

 - virtualenvs where system-site-packages are enabled


There are a few strategies that have been proposed to fix this. AIUI they are:
 - make sure none of our testing environments include distro requests packages.

 - make our requirements be tightly matched to what requests needs to
deal with the unbundling

 - teach pip how to identify and avoid this situation by always
upgrading requests (even if thats just a re-install of the version
from PyPI) when the installed requests is a distro installed version
**and** urllib3 is being modified.

 - get the distros to stop un-vendoring urllib3


The first one addresses the situation for the CI gate but doesn't
avoid developers getting bitten on their local machines. And
installing any distro thing that uses requests would re-instate the
potential for breakage. So while its not harmful, I don't think its
sufficient to make this go away.

The second is trivially insufficient - anytime requests vendored
urllib3 is not precisely identical to a released urllib3, it becomes
impossible to satisfy that via dependency version pinning - the only
way to satisfy it is with the urllib3 in the distro that has whatever
change was needed included.

The third approach will require some negotiation I suspect - because
its aesthetically wrong: from an upstream perspective urllib3 is
independent of requests, and vice-versa, but from a distro perspective
they are tightly coupled, no variation permitted.

The fourth approach meets the stone wall of 'but security' and 'no
redundancy permitted' - I don't have the energy to try and get through
the near-religious mindset I've encountered there before, though hey -
if Fedora and Debian and Ubuntu folk are all interested in figuring
out a sustainable way forward, that would be great: please don't feel
cut out, I'm just not expecting anything.

If there are other approaches, great - please throw them up here.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev