Issues created:
* Ignore dup uploads instead of error:
https://github.com/pypa/warehouse/issues/2284
* Rename "/legacy" API URL: https://github.com/pypa/warehouse/issues/2285
* Propose "packaging announce" mailing list:
https://github.com/pypa/packaging-problems/issues/98
o Previous issue noting challenges in communicating on packaging,
many of the same concerns still seem relevant:
https://github.com/pypa/packaging-problems/issues/21
* Explain what "pre-production" means on Warehouse:
https://github.com/pypa/warehouse/issues/2287
* Deprecation notice added to: https://wiki.python.org/moin/TestPyPI
o Issue created to enhance docs for TestPyPI:
https://github.com/pypa/python-packaging-user-guide/issues/362
* Better error message for PEP 527 accepted file types:
https://github.com/pypa/packaging-problems/issues/99
I hope I got all the right places for those issues.
Thanks Donald for the clear reply from which it was easy to create these
separate issues. I didn't realize you had so little paid time to work
on packaging issues. I'm surprised with all the large companies that
use Python that the PSF can't secure more donations to invest in fixing
these issues. Or that someone like Redhat can't get a couple engineers
to devote time for a couple years like they do to other OSS projects.
(No criticism intended, just musing).
*Randy Syring*
Husband | Father | Redeemed Sinner
/"For what does it profit a man to gain the whole world
and forfeit his soul?" (Mark 8:36 ESV)/
On 08/04/2017 05:51 PM, Donald Stufft wrote:
On Aug 4, 2017, at 4:43 PM, Lucas Boppre Niehues
<lucasbop...@gmail.com <mailto:lucasbop...@gmail.com>> wrote:
I just had a dreadful day dealing with PyPI and the migrations that
are happening. I'm posting this rant here in hopes of reaching the
people who can fix it (apologies if I'm in the wrong list), or at
least offer an explanation of the events that happened.
Disclaimers:
- Any and all feedback is welcome, but please be charitable.
- I'm angry *at* things, I hope not to offend anybody.
- I'm posting this rant here and not reddit or hn to ensure the
discussion stays between the interested parties and not the whole
internet.
For formatting reasons the text is here:
https://gist.github.com/boppreh/ac7522b3a4ac46b4f6010eecddc57f21
Thank you for your attention
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
<mailto:Distutils-SIG@python.org>
https://mail.python.org/mailman/listinfo/distutils-sig
Going through your text, there’s a mixture of things that sort of
combined here, I’m kind of just going to stream of conscious reply to
them here, because I can’t reply inline.
> error: No dist file created in earlier command
Yes, ``setup.py upload`` *requires* you to create the files it is
uploading within the same command during the upload. This is one of
the reasons that twine as a tool exists and is generally recommended.
> It's the exact same file. You could add a check for it and ignore,
but no, it had to be an error.
This seems like a reasonable feature request, please file an issue
with Warehouse asking for it so I don’t forget it.
> It seems that long_description in setup.py was missed entirely.
I’m not sure what happened here. If you share the package I can
probably figure it out though. I’m not currently aware of any bugs
with how long_description is handled.
> By the way, why was this feature included in the register command?
This was not an explicit feature of the register command, and more of
an accident in the way things got implemented in PyPI’s decade+ history.
> [Stuff about editing the description]
You located the issue on Warehouse. The fundamental issue here is
trying to find the right balance between constraining authors so that
end users can have a consistent behavior between packages and giving
authors power to best manage their own projects. Discussion about
editing the description is probably best handled on the warehouse
issue though (Which I forgot to update with my thoughts!).
> My setup.py has been communicating with
https://upload.pypi.org/legacy/ by default. Note the legacy part.
This is confusing, and it is essentially due to the fact there are two
things here which are considered “legacy”. One is the legacy
codebase/deployment that currently powers pypi.python.org
<http://pypi.python.org> which are are slowly replacing and migrating
things over to the new, modern code base that powers pypi.org
<http://pypi.org>. The other thing is the actual upload API itself,
which has stayed the same currently, but which I plan to replace at
some point in the future. This API is also considered “legacy” (it
just doesn’t have a replacement yet). So the legacy in
https://upload.pypi.org/legacy/ has to do with the API rather than the
location/deployment.
I probably should have named it something other than /legacy/, my goal
was mostly that changing URLs is hard (it requires a bunch of
documentation/updates in different packages and N years for that to
percolate out) so since we were forcing people to change the URLs with
the move from pypi.python.org <http://pypi.python.org> to pypi.org
<http://pypi.org>, might as well get it all done at once. It might be
reasonable to name it something else now, and just keep the /legacy/
around as an alias. I’m not sure if that adds or subtracts from the
confusion, but if you think that would have helped you, please open a
new issue on Warehouse.
> The "new APIs" seem to be missing at last one important feature
Some of this is purposeful as we attempt to rein in some of the more
“random” features that PyPI has grown over time. Some of it is
accidental as there’s not really any documentation about what PyPI
provides, and people have, over the decade of it’s existence, depended
on all sorts of random bits and pieces of PyPI which makes it hard to
tell what people are even using.
> I must have missed the deprecation period that preceded the complete
removal of the old APIs. A friendly email would have been nice if you
expect me to migrate to a new solution.
This is a hard balance to strike. When I have generated emails in the
past I tend to get a slew of people berating me for sending them
emails about things. Particularly for changes like this where there
belief is that for most people, the impact should be minimal, if there
is any at all. I do try to at least tweet and post to distutils-sig
about possibly disruptive changes like this. For this specific change,
that thread was at
https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html.
> The new website is "pre-production”.
This is mostly just a warning at this point that the UI might return
funky results, some things might not be great yet, and to not start
throwing the full force of people’s production traffic at it. The
current hosting arrangement can’t handle the full amount of traffic we
get from ``pip install …`` (but uploads are fine!) and we’re working
on building on the new infrastructure with our future hosting
arrangement that can handle the full production load.
> HTTPError: 400 Client Error: Invalid file extension. for url:
https://upload.pypi.org/legacy/
The ability to upload anything besides sdists, wheels, and eggs was
deprecated and removed. You can read the PEP that removed them at
https://www.python.org/dev/peps/pep-0527/.
> Maybe the tutorial is outdated, and TestPyPI supports
auto-registration now?
I don’t think anyone has kept the tutorial on wiki.python.org
<http://wiki.python.org> up to date. To be frank, I don’t even know
how to update wiki.python.org <http://wiki.python.org>. Generally
we’ve been trying to consolidate packaging documentation onto
packaging.python.org <http://packaging.python.org> — although I don’t
think we have one for TestPyPI yet.
TestPyPI itself is also severely under maintained. I don’t use it so I
rarely even think about it and my time is stretched thin to an extreme
degree. I plan on getting rid of TestPyPI at some point and replacing
it with something better, You can see discussion about that at
https://github.com/pypa/warehouse/issues/726.
> I try python setup.py bdist upload
As I mentioned above, we deprecated/removed everything but sdist,
wheels, and eggs as per PEP 527. We could probably do better error
messages here though.
> I can't even specify the description of a package, not even during
registration or upload.
Tell me your package name privately or publicly and I’ll figure out
what went wrong.
> I cannot even report the issues. [ ... ] because there's no clear
location to report them.
This is a problem, and we don’t really have a good solution for them.
I think generally what happens is people just open them on whichever
tool they think is the best fit, and the authors of these tools all
know each other, and if it ends up getting filed in the incorrect
place, we just redirect people tot he correct place.
> I am trying to donate my precious free time to the Python community
by offering free software, receiving absolutely no compensation
whatsoever, and every step of the experience has been shitty.
I/we understand that, and we’re generally trying to do the best we can
with what limited resources we have. The flip side of this is that
we’re *also* (for the most part) donating our precious free time to
the Python community to run PyPI (and the other PSF services too!).
For the record, in July PyPI used something just under 1 Petabyte of
bandwidth and served 38 billion HTTP requests.
People working on that include myself with 16h/week of paid time
(split between *all* of the packaging related tools I work on) plus
something like 20-30h/week of volunteer time (on top of the additional
24h/week of time I spent working on my employers projects). On the ops
side there is another individual (who I won’t name here to prevent him
from getting drug into this [1]) who donates his spare time as well
(though I don’t know offhand how much that is).
As far as keeping PyPI running as well as getting the new code base
developed and deployed… that’s about it [2]. This is a service used by
~everyone in the Python community without even a single full time
person on it.
We’re trying to slowly migrate to a new code base which is (A) easier
to maintain/scale taking up fewer hours (B) easier to actually work on
to develop new features, which will hopefully help attract and retain
new contributors and (C) has removed decades of cruft that has built
up and either explicitly made something supported or explicit decided
it is no longer supported, instead of a quasi supported “well it’s
what PyPI does” scenario we have now.
This probably won’t be the last less than perfect phase of the
migration to the new code base. My best recommendation if you don’t
want to discover these things JIT (even though we *do* try to minimize
them) is to subscribe to and follow distutils-sig. Alternatively if
you want to get more involved, one thing we’re severely lacking is a
better mechanism for communicating these things to users, and if you
wanted to propose/get sign off/implement/etc something for that, then
that is another possibility. Finally if you wanted you could also get
involved in actually shaping the direction of these tools, either by
discussing on distutils-sig and in their respective issue trackers, or
by becoming an active contributor. That’s the best way to ensure that
your relevant use cases are considered during the design or execution
of things.
[1] Folks can yell at me if they want, don’t go yelling at other
contributors. Generally the buck stops with me anyways.
[2] There is a designer working on the new code base, and there have
been some other contributors, but since this is about the APIs and
maintenance I left them out.
—
Donald Stufft
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
