On Wed, Sep 17, 2014 at 20:59 -0400, Donald Stufft wrote:
> Right now pip (and originally setuptools, which does it as well) will do this
> sort of dance when looking for things on the PyPI simple index. This isn't the
> actual code though:
>
> thing_to_install = "foo==1.0"
> page = None
>
> if thing_to_install.contains("=="): # First look at a versioned url if ==
> page = request_url(
> "https://pypi.python.org/simple/"; + thing_to_install.name
> + "/" + thing_to_install.version
> )
>
> if not page: # If we don't have something, look for unversioned
> page = request_url(
> "https://pypi.python.org/simple/"; + thing_to_install.name
> )
>
> if not page: # Finally, look at the /simple/ index itself
> page = request_url("https://pypi.python.org/simple/";)
>
> # From here, look at the page to discover things.
>
>
> As far as I can tell a lot of this is largely historical.
>
> The /simple/{name}/{verison}/ pages come from a time when there wasn't a
> simple
> index I think and sometimes packages would need to go to /pypi/foo/version/
> in order to actually get a list of things. However we now do have the simple
> API and AFAICT the /simple/ API does not nor has ever had a reponse for
> /simple/{name}/{version}/. This always 404's and falls back to the
> /simple/{name}/. I would like to consider this URL unsupported in pip and
> remove
> checking for it. It will reduce the number of needless HTTP requests by one
> per pinned version.
>
> Does anyone know anything this will break?
Sounds fine to me to remove the /{version}/ check. FWIW devpi also doesn't
support it.
> The other thing that happens is if the /simple/{name}/ thing 404's it'll
> fallback to /simple/. This is done so that if someone mistypes a name in a way
> that is still considered equivilant after normalization is applied, instead
> of a 404 they get the /simple/ page and the tooling can discover the name from
> there.
>
> If you remember back a little while ago I changed PyPI so that it considered
> the normalized form of of the name the "cannonical" name for the simple index,
> this means that tooling will be able to know ahead of time if a project called
> say "Django" should be requested with /simple/Django/ or /simple/django/.
>
> What I would like to do now is remove the fallback to /simple/. If we fall
> back
> to that it is a 2.1MB download that occurs which is a fairly big deal and can
> slow down a ``pip install`` quite signifcantly. I have a PR against
> bandersnatch which will make bandersnatch generate a /simple/{name}/ URL where
> name is the normalized form, and another PR against pip which will cause it
> to always request the normalized form. When both of these land it would mean
> that the only time pip will fallback to /simple/ is:
>
> 1. If someone is using a *new* pip (with the PR merged) but with a mirror that
> doesn't support the normalized form of the URLs (old bandersnatch,
> pep381client, maybe others?)
I think devpi-server would also still redirect on normalized form to
the "real name" but we plan to change that to behave as pypi does now
(serve everything under the normalized name). How/do you plan to detect
if the "normalized form" is supported from a server?
> 2. If someone typed ``pip install <foo>`` where <foo> is a thing that doesn't
> actually exist.
If we had a meta header "pypi-simple-protocol-version" set to to "1" by
pypi.python.org and compliant servers we could avoid all falling back to
simple/ safely. You do a single lookup for each project and are done,
safely, no falling back to simple/ at all.
FWIW devpi-server on simple/<name> already returns a "200" with an empty
link set for projects that don't exist. I've never had complaints from
users about this hack. Note that pip/easy_install don't behave much
differently today if they get an empty link set on simple/<name> or a
404 and then no match on the full simple/ page.
> Does anyone have any complaints if pip stopped falling back to /simple/ once
> the bandersnatch PR is merged and released?
I am fine, preferably in lieu with introducing indication from pypi.python.org
about the simple-api version.
> Further more does anyone have any problems with narrowing the "supported URLS"
> of the "simple API" to /simple/ and /simple/<normalized-name>/ and make
> fetching /simple/ optional?
Thanks for cleaning up the protocols.
holger
_______________________________________________
Distutils-SIG maillist - [email protected]
https://mail.python.org/mailman/listinfo/distutils-sig
