Re: [Distutils] Metadata 2.0: Is there a formal spec for a requirement?
On 18 September 2014 10:08, Donald Stufft don...@stufft.io wrote: On Sep 17, 2014, at 1:05 AM, Nick Coghlan ncogh...@gmail.com wrote: Perhaps we should make that official policy? Anything in PEP 426 and PEP 459 (and other packaging metadata and installation database related PEPs) needs to be trialled in distlib/distil before the PEPs can be accepted? distlib could operate permanently under a PEP 411 style provisional API guideline, and if folks aren't comfortable with this may break without warning, then they can stick to the stable packaging/pip layer. I’m OK with calling out this relationship though I don’t think it should be a mandatory thing. I think we’re all adults and able to figure out when it makes sense to trial it in distil/distlib or not. Works for me. I do suspect we're going to want to trial PEP 426 and the PEP 459 metadata extensions :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The Simple API - What URLs are supported
On Wed, Sep 17, 2014 at 20:59 -0400, Donald Stufft wrote:
Right now pip (and originally setuptools, which does it as well) will do this
sort of dance when looking for things on the PyPI simple index. This isn't the
actual code though:
thing_to_install = foo==1.0
page = None
if thing_to_install.contains(==): # First look at a versioned url if ==
page = request_url(
https://pypi.python.org/simple/; + thing_to_install.name
+ / + thing_to_install.version
)
if not page: # If we don't have something, look for unversioned
page = request_url(
https://pypi.python.org/simple/; + thing_to_install.name
)
if not page: # Finally, look at the /simple/ index itself
page = request_url(https://pypi.python.org/simple/;)
# From here, look at the page to discover things.
As far as I can tell a lot of this is largely historical.
The /simple/{name}/{verison}/ pages come from a time when there wasn't a
simple
index I think and sometimes packages would need to go to /pypi/foo/version/
in order to actually get a list of things. However we now do have the simple
API and AFAICT the /simple/ API does not nor has ever had a reponse for
/simple/{name}/{version}/. This always 404's and falls back to the
/simple/{name}/. I would like to consider this URL unsupported in pip and
remove
checking for it. It will reduce the number of needless HTTP requests by one
per pinned version.
Does anyone know anything this will break?
Sounds fine to me to remove the /{version}/ check. FWIW devpi also doesn't
support it.
The other thing that happens is if the /simple/{name}/ thing 404's it'll
fallback to /simple/. This is done so that if someone mistypes a name in a way
that is still considered equivilant after normalization is applied, instead
of a 404 they get the /simple/ page and the tooling can discover the name from
there.
If you remember back a little while ago I changed PyPI so that it considered
the normalized form of of the name the cannonical name for the simple index,
this means that tooling will be able to know ahead of time if a project called
say Django should be requested with /simple/Django/ or /simple/django/.
What I would like to do now is remove the fallback to /simple/. If we fall
back
to that it is a 2.1MB download that occurs which is a fairly big deal and can
slow down a ``pip install`` quite signifcantly. I have a PR against
bandersnatch which will make bandersnatch generate a /simple/{name}/ URL where
name is the normalized form, and another PR against pip which will cause it
to always request the normalized form. When both of these land it would mean
that the only time pip will fallback to /simple/ is:
1. If someone is using a *new* pip (with the PR merged) but with a mirror that
doesn't support the normalized form of the URLs (old bandersnatch,
pep381client, maybe others?)
I think devpi-server would also still redirect on normalized form to
the real name but we plan to change that to behave as pypi does now
(serve everything under the normalized name). How/do you plan to detect
if the normalized form is supported from a server?
2. If someone typed ``pip install foo`` where foo is a thing that doesn't
actually exist.
If we had a meta header pypi-simple-protocol-version set to to 1 by
pypi.python.org and compliant servers we could avoid all falling back to
simple/ safely. You do a single lookup for each project and are done,
safely, no falling back to simple/ at all.
FWIW devpi-server on simple/name already returns a 200 with an empty
link set for projects that don't exist. I've never had complaints from
users about this hack. Note that pip/easy_install don't behave much
differently today if they get an empty link set on simple/name or a
404 and then no match on the full simple/ page.
Does anyone have any complaints if pip stopped falling back to /simple/ once
the bandersnatch PR is merged and released?
I am fine, preferably in lieu with introducing indication from pypi.python.org
about the simple-api version.
Further more does anyone have any problems with narrowing the supported URLS
of the simple API to /simple/ and /simple/normalized-name/ and make
fetching /simple/ optional?
Thanks for cleaning up the protocols.
holger
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The Simple API - What URLs are supported
What about an approach where pip first tries the canonical name, and if that fails, tries the exact given name? Seems to me like that should handle legacy mirrors without the big download. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The Simple API - What URLs are supported
On 18 Sep 2014 17:48, Nick Coghlan ncogh...@gmail.com wrote: What about an approach where pip first tries the canonical name, and if that fails, tries the exact given name? And by canonical I mean normalised. Seems to me like that should handle legacy mirrors without the big download. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The Simple API - What URLs are supported
On Sep 18, 2014, at 3:48 AM, Nick Coghlan ncogh...@gmail.com wrote:
What about an approach where pip first tries the canonical name, and if that
fails, tries the exact given name?
Seems to me like that should handle legacy mirrors without the big download.
Cheers,
Nick.
The exact implementation I had in mind has the /simple/{name}/{version}/ url
being removed immediately since it’s a cost that's *always* paid if you use ==
and as far as I know it's not supported/used anywhere. However the fallback to
/simple/ would go through the 2 release deprecation cycle since it's both
actually useful for use with older mirrors and it's only paid if it needs to
be used right now.
I think that the deprecation cycle is probably fine to handle that, because
even if we do as you suggested we'll still need to fall back to /simple/
because of cases like ``pip install django`` where the real name is Django
and we have no way of knowing that.
---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The Simple API - What URLs are supported
On Thu, Sep 18, 2014 at 06:17 -0400, Donald Stufft wrote:
On Sep 18, 2014, at 3:48 AM, Nick Coghlan ncogh...@gmail.com wrote:
What about an approach where pip first tries the canonical name, and if
that fails, tries the exact given name?
Seems to me like that should handle legacy mirrors without the big download.
Cheers,
Nick.
The exact implementation I had in mind has the /simple/{name}/{version}/ url
being removed immediately since it’s a cost that's *always* paid if you use ==
and as far as I know it's not supported/used anywhere.
I agree and think that's just fine.
However the fallback to
/simple/ would go through the 2 release deprecation cycle since it's both
actually useful for use with older mirrors and it's only paid if it needs to
be used right now.
I think that the deprecation cycle is probably fine to handle that, because
even if we do as you suggested we'll still need to fall back to /simple/
because of cases like ``pip install django`` where the real name is Django
and we have no way of knowing that.
Not sure i follow. Do you mean pip install django needs to fall back
to /simple/ when it works against a static file server?
What significance does the real name have in pip install context?
holger
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The Simple API - What URLs are supported
On Sep 18, 2014, at 3:27 AM, holger krekel hol...@merlinux.eu wrote:
On Wed, Sep 17, 2014 at 20:59 -0400, Donald Stufft wrote:
Right now pip (and originally setuptools, which does it as well) will do this
sort of dance when looking for things on the PyPI simple index. This isn't
the
actual code though:
thing_to_install = foo==1.0
page = None
if thing_to_install.contains(==): # First look at a versioned url if ==
page = request_url(
https://pypi.python.org/simple/; + thing_to_install.name
+ / + thing_to_install.version
)
if not page: # If we don't have something, look for unversioned
page = request_url(
https://pypi.python.org/simple/; + thing_to_install.name
)
if not page: # Finally, look at the /simple/ index itself
page = request_url(https://pypi.python.org/simple/;)
# From here, look at the page to discover things.
As far as I can tell a lot of this is largely historical.
The /simple/{name}/{verison}/ pages come from a time when there wasn't a
simple
index I think and sometimes packages would need to go to /pypi/foo/version/
in order to actually get a list of things. However we now do have the simple
API and AFAICT the /simple/ API does not nor has ever had a reponse for
/simple/{name}/{version}/. This always 404's and falls back to the
/simple/{name}/. I would like to consider this URL unsupported in pip and
remove
checking for it. It will reduce the number of needless HTTP requests by one
per pinned version.
Does anyone know anything this will break?
Sounds fine to me to remove the /{version}/ check. FWIW devpi also doesn't
support it.
The other thing that happens is if the /simple/{name}/ thing 404's it'll
fallback to /simple/. This is done so that if someone mistypes a name in a
way
that is still considered equivilant after normalization is applied, instead
of a 404 they get the /simple/ page and the tooling can discover the name
from
there.
If you remember back a little while ago I changed PyPI so that it considered
the normalized form of of the name the cannonical name for the simple
index,
this means that tooling will be able to know ahead of time if a project
called
say Django should be requested with /simple/Django/ or /simple/django/.
What I would like to do now is remove the fallback to /simple/. If we fall
back
to that it is a 2.1MB download that occurs which is a fairly big deal and can
slow down a ``pip install`` quite signifcantly. I have a PR against
bandersnatch which will make bandersnatch generate a /simple/{name}/ URL
where
name is the normalized form, and another PR against pip which will cause it
to always request the normalized form. When both of these land it would mean
that the only time pip will fallback to /simple/ is:
1. If someone is using a *new* pip (with the PR merged) but with a mirror
that
doesn't support the normalized form of the URLs (old bandersnatch,
pep381client, maybe others?)
I think devpi-server would also still redirect on normalized form to
the real name but we plan to change that to behave as pypi does now
(serve everything under the normalized name). How/do you plan to detect
if the normalized form is supported from a server?
Yea I forgot to explicitly mention devpi-server, but I knew it did the redirect
so
it would be fine there.
I don’t plan on detecting it, I just plan on sending it through the normal
deprecation cycle in pip. Since this doesn’t reflect any changes on the PyPI
side beyond what's already been done, and the changes to bandersnatch/mirrors
can be made in an completely backwards compatible fashion the only problem
would be when you use a *new* pip with an older mirroring client or index
server. I think a 2 release deprecation cycle (so pip 6.0 is our next release,
it would be deprecated there and ultimately removed in pip 8.0) is ample time
for folks to see the warning and adjust their own internal mirrors.
2. If someone typed ``pip install foo`` where foo is a thing that doesn't
actually exist.
If we had a meta header pypi-simple-protocol-version set to to 1 by
pypi.python.org and compliant servers we could avoid all falling back to
simple/ safely. You do a single lookup for each project and are done,
safely, no falling back to simple/ at all.
FWIW devpi-server on simple/name already returns a 200 with an empty
link set for projects that don't exist. I've never had complaints from
users about this hack. Note that pip/easy_install don't behave much
differently today if they get an empty link set on simple/name or a
404 and then no match on the full simple/ page.
Correct, if there’s no match on the /simple/ page, then it’s functionally
equivalent
to getting a blank /simple/{name}/ except there’s one less HTTP request made.
I don’t think we need to introduce a header, other than the change to serve
normalized
Re: [Distutils] The Simple API - What URLs are supported
On Sep 18, 2014, at 6:22 AM, holger krekel hol...@merlinux.eu wrote:
On Thu, Sep 18, 2014 at 06:17 -0400, Donald Stufft wrote:
On Sep 18, 2014, at 3:48 AM, Nick Coghlan ncogh...@gmail.com wrote:
What about an approach where pip first tries the canonical name, and if
that fails, tries the exact given name?
Seems to me like that should handle legacy mirrors without the big download.
Cheers,
Nick.
The exact implementation I had in mind has the /simple/{name}/{version}/ url
being removed immediately since it’s a cost that's *always* paid if you use
==
and as far as I know it's not supported/used anywhere.
I agree and think that's just fine.
However the fallback to
/simple/ would go through the 2 release deprecation cycle since it's both
actually useful for use with older mirrors and it's only paid if it needs to
be used right now.
I think that the deprecation cycle is probably fine to handle that, because
even if we do as you suggested we'll still need to fall back to /simple/
because of cases like ``pip install django`` where the real name is Django
and we have no way of knowing that.
Not sure i follow. Do you mean pip install django needs to fall back
to /simple/ when it works against a static file server?
What significance does the real name have in pip install context?
As of right now, bandersnatch uses the real name of a project as the
directory name, so in the case of Django, which captilizes it's real name on
PyPI, it has URLs like /simple/Django/. Historically this matched what PyPI
did. Since bandersnatch is a static mirror it generally couldn't do a redirect
from /simple/django/ to /simple/Django/ so if someone typed
``pip install django`` pip would request /simple/django/, get a 404 and then
request /simple/, find that the real name of django is Django, and then
it will request /simple/Django/.
The benefit of the change I made to PyPI to prefer the normalized name instead
of the real name is that tools like pip can know ahead of time what
/simple/{name}/ URL to request regardless of what the real name is on PyPI.
---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Buildout setuid
On Wed, Sep 17, 2014 at 1:47 PM, Lennart Regebro rege...@gmail.com wrote: While writing a blog post about software configuration management I looked into buildout, and using it as an SCM tool. And it has one big restriction: You can't run certain parts as root. I think adding that would actually not be too hard. Are there any principal arguments against it? I looked at making an extension, but I would need a hook that is run before and after each step in that case. I was thinking that you could define which parts should run as root in one of two ways: 1. A parameter in the part config 2. Having a global configuration with a list of parts. This for the case when the parts recipe itself has a parameter that clashes with the parameter in 1. I'm leaning towards having a setuid parameter, so you can set to other id's than 0. Technically it would be done by setuid to root for the configured parts, and then back after it has run. You would have to run buildout as a whole with sudo for this to work. It would use the login name as the normal setuid, unless configured explicitly with a global setuid parameter. Thoughts? We deploy to production with buildout and have never needed this. Our approach is to have separate buildouts for building software (RPMs currently) and for deploying to production machines. The deployment buildouts are run as root (typically from another process that runs from root, https://bitbucket.org/zc/zkdeployment). These 2 buildouts are run at very different times and situations. A better approach IMO is to deploy with Docker. With Docker, all of your deployment is done when you build an image, still as root. Unfortunately, our current scheme is working well enough and we have enough other priorities that I fear I won't find time to dockerfy our processes soon. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
