Hi,
looks like I'm late to the party to figure out that I'm going to be hurt again.
I'd like to suggest explicitly considering what is going to break due
to this and how much work you are forcefully inflicting on others. My
whole experience around the packaging (distribute/setuptools) and
mirroring/CDN in this year estimates cost for my company somewhere
between 10k-20k EUR just for keeping up with the breakage those changes
incure. It might be that we're wonderfully stupid (..enough to
contribute) and all of this causes no headaches for anybody else ….
Overall, guessing that the packaging infrastructure is used by probably
multiple thousands of companies then I'd expect that at least 100 of
them might be experiencing problems like us. Juggling arbritrary
numbers I can see that we're inflicting around a million EURs of cost
that nobody asked for.
More specific statements below.
On 2013-08-04 22:25:01 +0000, Donald Stufft said:
Here's my PEP for Deprecating and Removing the Official Public Mirrors
It's source is at:
https://github.com/dstufft/peps/blob/master/mirror-removal.rst
Abstract
=======
This PEP provides a path to deprecate and ultimately remove the official
public mirroring infrastructure for `PyPI`_. It does not propose the removal
of mirroring support in general.
-1 - maybe I don't have the right to speak up on CDN usage, but
personally I feel it's a bad idea to delegate overall PyPI availability
exclusively to a commercial third party. It's OK for me that we're
using them to improve PyPI availability, but completely putting our
faith in their hands, doesn't sound right to me.
Rationale
========
The PyPI mirroring infrastructure (defined in `PEP381`_) provides a means to
mirror the content of PyPI used by the automatic installers. It also provides
a method for autodiscovery of mirrors and a consistent naming scheme.
There are a number of problems with the official public mirrors:
* They give control over a \*.python.org domain name to a third party,
allowing that third party to set or read cookies on the pypi.python.org and
python.org domain name.
Agreed, that's a problem.
* The use of a sub domain of pypi.python.org means that the mirror operators
will never be able to get a certificate of their own, and giving them
one for a python.org domain name is unlikely to happen.
Agreed.
* They are often out of date, most often by several hours to a few days, but
regularly several days and even months.
That's something that the mirroring infrastructure should have been
constructed for. I completely agree that the way the mirroring was
established was way sub-optimal. I think we can do better.
* With the introduction of the CDN on PyPI the public mirroring infrastructure
is not as important as it once was as the CDN is also a globally distributed
network of servers which will function even if PyPI is down.
Well, now we have one breakage point more which keeps annoying me. This
argument is not completely true. They may be getting better over time
but we have invested heavily to accomodate the breakage - that needs to
be balanced with some benefit in the near future.
* Although there is provisions in place for it, there is currently no known
installer which uses the authenticity checks discussed in `PEP381`_ which
means that any download from a mirror is subject to attack by a malicious
mirror operator, but further more due to the lack of TLS it also means that
any download from a mirror is also subject to a MITM attack.
Again, I think that was a mistake during the introduction of the
mirroring infrastructure: too few people, too confusing PEP.
* They have only ever been implemented by one installer (pip), and its
implementation, besides being insecure, has serious issues with performance
and is slated for removal with it's next release (1.5).
Only if you consider the mirror auto-discovery protocol. I'm not sure
whether using DNS was such a smart move. A simple HTTP request to find
mirrors would have been nice. I think we can still do that.
Also, not everyone wants or needs auto-detection the way that the
protocol describes it. I personally just hand-pick a mirror (my own,
hah) and keep using that.
We are also thinking about providing system-level default configuration
to hint tools like PIP and setuptools to a different default index that
is closer from a network perspective. From a customer perspective this
should be "PyPI".
I'd like to avoid breakage. Again, if you don't let me choose where to
spend my time, I'd rather invest the time I need for cleaning up the
breakage into something constructive.
The indices are in active use. f.pypi.python.org is seeing between
150-300GB of traffic per month, the patterns widely ranging over the
last month. This is traffic that is not used internally from gocept.
Due to the number of issues, some of them very serious, and the CDN which more
or less provides much of the same benefits this PEP proposes to first
deprecate and then remove the public mirroring infrastructure. The ability to
mirror and the method of mirroring will not be affected and the existing
public mirrors are encouraged to acquire their own domains to host their
mirrors on if they wish to continue hosting them.
The biggest benefit of the mirroring infrastructure is that it is
intended to be de-centralized.
As a community member I can step up and take over responsibility of
availability, performance, and security of a mirror.
As a community member I have to completely submit to whatever the CDN
does and contacting another community member who hopefully will be with
us for a long time and stay in good contact with the CDN for us. That's
centralization and I don't like that a bit.
Plan for Deprecation & Removal
=============================
Immediately upon acceptance of this PEP documentation on PyPI will be updated
to reflect the deprecated nature of the official public mirrors and will
direct users to external resources like http://www.pypi-mirrors.org/ to
discover unofficial public mirrors if they wish to use one.
On October 1st, 2013, roughly 2 months from the date of this PEP, the DNS names
of the public mirrors ([a-g].pypi.python.org) will be changed to point back to
PyPI which will be modified to accept requests from those domains. At this
point in time the public mirrors will be considered deprecated.
Then, roughly 2 months after the release of the first version of pip to have
mirroring support removed (currently slated for pip 1.5) the DNS entries for
[a-g].pypi.python.org and last.pypi.python.org will be removed and PyPI will
no longer accept requests at those domains.
Oh great. That means in about 4 months I have to go through *any
installation that my company maintains* and sift through whether we're
still referencing f.pypi.python.org anywhere.
Can I write a check?
Unofficial Public or Private Mirrors
===================================
The mirroring protocol will continue to exist as defined in `PEP381`_ and
people are encouraged to utilize to host unofficial public and private mirrors
if they so desire. For operators of unofficial public or private mirrors the
recommended mirroring client is `Bandersnatch`_.
Thanks for the recommendation.
Instead of this dance breaking many things yet again, I'd love if we
could find a way forward keeping the infrastructure.
Some ideas:
- Take control of *.pypi.python.org back
- Record other public names of the mirrors
- Use 301 redirects to send old installations over to the new mirror names.
- Make it easier for community members to help maintain the list of mirrors.
- Make a better (faster) removal policy of mirrors if the owners are
not responsive.
- Make it easier for other community members to set up and maintain
mirrors. I'm happy to improve bandersnatch where needed.
Lastly, again, and I might be getting on everyones nerves.
Why does it seem that other communities have figured this out much
simpler, with less hassle, and with no significant changes for years
and we need to keep changing stuff over and over and over and break
things over and over and over.
It's really hard for me to write this mail without cussing - the
situation is very frustrating: the community dynamics seem to "want to
move forward" where they from my perspective "wander left and right and
break stuff like a drunken elephant driving a tank throught the Louvre".
Christian
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
