Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[tritium-list]

As a heavy windows user, I think I can say for the vast majority of windows 
users on python (that aren’t brand spanking new at python...)

BURN BDIST_WININST!
BURN BDIST_MSI!

> -Original Message-
> From: Nick Coghlan [mailto:ncogh...@gmail.com]
> Sent: Tuesday, August 23, 2016 7:25 AM
> To: Paul Moore <p.f.mo...@gmail.com>
> Cc: Alexander Walters <tritium-l...@sdamon.com>; distutils-sig  s...@python.org>
> Subject: Re: [Distutils] Deprecating little used file types/extensions on 
> PyPI?
> 
> On 23 August 2016 at 19:36, Paul Moore <p.f.mo...@gmail.com> wrote:
> > So I don't think that in the medium term there's going to be much
> > practical change in the state of things on Windows:
> >
> > - Users install Python from the published python.org installers
> > - Users install packages using pip and wheels from PyPI
> > - Plus some exceptions, where people need to use sdists, or
> > independently published wheels, or worse still, wininst/msi installers
> > because that's all available
> >
> > Whether that process is manual, or hidden behind some form of scripted
> > process, won't alter the underlying infrastructure.
> >
> > I don't see any sign of *anyone* working on a curated distribution for
> > Windows along the lines of Linux distros or Homebrew. (Unless you
> > count cross-platform stacks like conda, which IMO are a different
> > scenario than "system" Python installs).
> 
> OK, cool - that gives us all the more reason to retain bdist_wininst
> and bdist_msi hosting support. However, I do think it makes sense for
> us to say up front that we'll reconsider that decision if something
> akin to homebrew gains traction amongst developers running Windows the
> way homebrew has amongst open source users running Mac OS X.
> 
> Cheers,
> Nick.
> 
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 23, 2016, at 4:40 PM, Steve Dower  wrote:
> 
> On 23Aug2016 0937, Brett Cannon wrote:
>> I should also mention I have never come across anyone at Microsoft use
>> the bdist_msi or bdist_winst installers (I've added Steve to this email
>> in case my experience is wrong).
> 
> In large part this is because I've gotten to them first :)
> 
> Personally I don't like bdist_msi or bdist_winst as neither of them correctly 
> sets dependencies on the underlying Python install, so removing Python does 
> not remove the package. But I can see why people may prefer to use them.

I mean, we have data that suggests that people do *not* prefer to use them 
since 90% of the downloads for those files are mirrors just mirroring them and 
7% is just setuptools tearing the .exe apart to treat it similarly to an .egg 
(for wininst, for msi it’s like 97% mirroring and only like 400 of them 
uploaded ever anyways).

> 
> I see no harm in not installing them via pip though. They should be 
> downloadable "associated files" at minimum, and I don't have any opinion 
> about making them available via index data (except that installers should be 
> free to ignore them).
> 
> Cheers,
> Steve


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Steve Dower]

On 23Aug2016 0937, Brett Cannon wrote:

I should also mention I have never come across anyone at Microsoft use
the bdist_msi or bdist_winst installers (I've added Steve to this email
in case my experience is wrong).


In large part this is because I've gotten to them first :)

Personally I don't like bdist_msi or bdist_winst as neither of them 
correctly sets dependencies on the underlying Python install, so 
removing Python does not remove the package. But I can see why people 
may prefer to use them.


I see no harm in not installing them via pip though. They should be 
downloadable "associated files" at minimum, and I don't have any opinion 
about making them available via index data (except that installers 
should be free to ignore them).


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Brett Cannon]

On Tue, 23 Aug 2016 at 07:33 Donald Stufft  wrote:

>
> > On Aug 23, 2016, at 7:25 AM, Nick Coghlan  wrote:
> >
> > OK, cool - that gives us all the more reason to retain bdist_wininst
> > and bdist_msi hosting support. However, I do think it makes sense for
> > us to say up front that we'll reconsider that decision if something
> > akin to homebrew gains traction amongst developers running Windows the
> > way homebrew has amongst open source users running Mac OS X.
>
>
> I still don’t think there’s a whole lot of benefit to retaining them even
> now. In the last 30 days, 90% of the downloads of bdist_wininst were
> generated by things that I know for a fact to be mirroring clients (almost
> all entirely bandersnatch). The next highest source of downloads was coming
> from setuptools, at 7%. Over 75% of the downloads from setuptools are for
> coverage.py, which tells me that it’s likely being triggered by
> test_requires
> and would be covered by teaching setuptools how to wheel instead.
>
> For bdist_msi, 96% of all downloads come from things we know to be
> mirroring
> clients.
>
> For bdist_dmg, 97% of all downloads come from things we know to be
> mirroring
> clients.
>
> For bdist_egg, 80% of all downloads come from things we know to be
> mirroring
> clients.
>
> For reference:
>
> For sdist, 30% of all downloads come from things we know to be mirroring
> clients.
>
> For bdist_wheel, 6% of all downloads come from things we know to be
> mirroring
> clients.
>
> It’s hard to get per project numbers for these (or at least, it takes a
> more
> complex query than I can manage with my head here). However, I think it’s
> pretty telling that when you start looking at other formats, not only is
> the
> primary consumer tools that just indiscriminately download everything from
> PyPI,
> but almost *all* of the consumers of those files are tools that just
> indiscriminately download everything. Unless there are users of those
> mirrors who
> follow vastly different usage patterns than what we see on PyPI itself,
> the primary
> purpose of bdist_wininst, bdist_msi, bdist_dmg, etc on PyPI is to consume
> disk space
> and bandwidth via the mirroring infrastructure.
>
> I’d also like to note, that the numbers above are conservative on what they
> consider to be a “mirroring client”. For instance, devpi used to use the
> default
> requests user-agent, and we see downloads via the requests user agent, but
> did not
> count them as mirroring clients because it could be some other script
> doing the
> downloading.
>

I should also mention I have never come across anyone at Microsoft use the
bdist_msi or bdist_winst installers (I've added Steve to this email in case
my experience is wrong). Everyone I have encountered either uses conda or
pip+wheels (hence why I keep poking the sdist and build API ideas as I want
to give Christophe Gohlke something else to do with his time than provide
wheels on Windows).
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

See https://mail.python.org/pipermail/distutils-sig/2016-August/029542.html 
 for a 
PEP!

> On Aug 23, 2016, at 12:37 PM, Brett Cannon  wrote:
> 
> 
> 
> On Tue, 23 Aug 2016 at 07:33 Donald Stufft  > wrote:
> 
> > On Aug 23, 2016, at 7:25 AM, Nick Coghlan  > > wrote:
> >
> > OK, cool - that gives us all the more reason to retain bdist_wininst
> > and bdist_msi hosting support. However, I do think it makes sense for
> > us to say up front that we'll reconsider that decision if something
> > akin to homebrew gains traction amongst developers running Windows the
> > way homebrew has amongst open source users running Mac OS X.
> 
> 
> I still don’t think there’s a whole lot of benefit to retaining them even
> now. In the last 30 days, 90% of the downloads of bdist_wininst were
> generated by things that I know for a fact to be mirroring clients (almost
> all entirely bandersnatch). The next highest source of downloads was coming
> from setuptools, at 7%. Over 75% of the downloads from setuptools are for
> coverage.py, which tells me that it’s likely being triggered by test_requires
> and would be covered by teaching setuptools how to wheel instead.
> 
> For bdist_msi, 96% of all downloads come from things we know to be mirroring
> clients.
> 
> For bdist_dmg, 97% of all downloads come from things we know to be mirroring
> clients.
> 
> For bdist_egg, 80% of all downloads come from things we know to be mirroring
> clients.
> 
> For reference:
> 
> For sdist, 30% of all downloads come from things we know to be mirroring
> clients.
> 
> For bdist_wheel, 6% of all downloads come from things we know to be mirroring
> clients.
> 
> It’s hard to get per project numbers for these (or at least, it takes a more
> complex query than I can manage with my head here). However, I think it’s
> pretty telling that when you start looking at other formats, not only is the
> primary consumer tools that just indiscriminately download everything from 
> PyPI,
> but almost *all* of the consumers of those files are tools that just
> indiscriminately download everything. Unless there are users of those mirrors 
> who
> follow vastly different usage patterns than what we see on PyPI itself, the 
> primary
> purpose of bdist_wininst, bdist_msi, bdist_dmg, etc on PyPI is to consume 
> disk space
> and bandwidth via the mirroring infrastructure.
> 
> I’d also like to note, that the numbers above are conservative on what they
> consider to be a “mirroring client”. For instance, devpi used to use the 
> default
> requests user-agent, and we see downloads via the requests user agent, but 
> did not
> count them as mirroring clients because it could be some other script doing 
> the
> downloading.
> 
> I should also mention I have never come across anyone at Microsoft use the 
> bdist_msi or bdist_winst installers (I've added Steve to this email in case 
> my experience is wrong). Everyone I have encountered either uses conda or 
> pip+wheels (hence why I keep poking the sdist and build API ideas as I want 
> to give Christophe Gohlke something else to do with his time than provide 
> wheels on Windows).


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Paul Moore]

On 23 August 2016 at 15:12, Antoine Pitrou  wrote:
> On Tue, 23 Aug 2016 10:36:35 +0100
> Paul Moore  wrote:
>>
>> I don't see any sign of *anyone* working on a curated distribution for
>> Windows along the lines of Linux distros or Homebrew. (Unless you
>> count cross-platform stacks like conda, which IMO are a different
>> scenario than "system" Python installs).
>
> Under Windows, there isn't much moral difference between a conda install
> (really a Miniconda or Anaconda install) and a python.org Python
> install.  You can even install Anaconda or Miniconda system-wide.
>
> (Miniconda is a minimal install of Python + conda, while Anaconda is a
> pretty large selection of packages in addition - though not the entire
> official repo contents, and not counting community packages)

Agreed - sorry, I was responding more to Nick's implication around
"system" management of Python, conda already has a well-established
management process, but AIUI, it's largely independent of PyPI/pip
(although it interoperates with those). So in the context of Windows
package managers, conda is no more relevant there than it is in
relation to (say) rpm or deb.

Realistically, though, I'd expect people wanting a Python stack on
Windows to gravitate more and more towards distributions like Anaconda
(maybe less so outside its core area of scientific use) with the
remaining users sticking to the python.org builds and pip.

However, I'm not that keen on continuing support for bdist_wininst and
bdist_msi. I'd rather put the effort into making pip and wheels a
compelling solution for such users. IMO, it already is for command
line use (the more so as more hard to build packages start providing
wheels). The place where pip falls down is for people who want to
stick to a GUI. For those people, I believe Idle is likely to be
getting an interface to pip, and I'd like to see other tools like PTVS
and PyCharm get similar capabilities (if they don't have them
already). That may mean exposing an interface to pip's install
mechanisms that's better than "use subprocess to call pip", but we'd
need some concrete use cases to work out the best form of such
interfaces.

On the matter of Chocolatey, it should be pretty trivial to create
recipes for installing Python packages via pip. So I'd see pip+wheel
as remaining the standard interface pretty much indefinitely.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 23, 2016, at 7:25 AM, Nick Coghlan  wrote:
> 
> OK, cool - that gives us all the more reason to retain bdist_wininst
> and bdist_msi hosting support. However, I do think it makes sense for
> us to say up front that we'll reconsider that decision if something
> akin to homebrew gains traction amongst developers running Windows the
> way homebrew has amongst open source users running Mac OS X.


I still don’t think there’s a whole lot of benefit to retaining them even
now. In the last 30 days, 90% of the downloads of bdist_wininst were
generated by things that I know for a fact to be mirroring clients (almost
all entirely bandersnatch). The next highest source of downloads was coming
from setuptools, at 7%. Over 75% of the downloads from setuptools are for
coverage.py, which tells me that it’s likely being triggered by test_requires
and would be covered by teaching setuptools how to wheel instead.

For bdist_msi, 96% of all downloads come from things we know to be mirroring
clients.

For bdist_dmg, 97% of all downloads come from things we know to be mirroring
clients.

For bdist_egg, 80% of all downloads come from things we know to be mirroring
clients.

For reference:

For sdist, 30% of all downloads come from things we know to be mirroring
clients.

For bdist_wheel, 6% of all downloads come from things we know to be mirroring
clients.

It’s hard to get per project numbers for these (or at least, it takes a more
complex query than I can manage with my head here). However, I think it’s
pretty telling that when you start looking at other formats, not only is the
primary consumer tools that just indiscriminately download everything from PyPI,
but almost *all* of the consumers of those files are tools that just
indiscriminately download everything. Unless there are users of those mirrors 
who
follow vastly different usage patterns than what we see on PyPI itself, the 
primary
purpose of bdist_wininst, bdist_msi, bdist_dmg, etc on PyPI is to consume disk 
space
and bandwidth via the mirroring infrastructure.

I’d also like to note, that the numbers above are conservative on what they
consider to be a “mirroring client”. For instance, devpi used to use the default
requests user-agent, and we see downloads via the requests user agent, but did 
not
count them as mirroring clients because it could be some other script doing the
downloading.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Antoine Pitrou]

On Tue, 23 Aug 2016 10:36:35 +0100
Paul Moore  wrote:
> 
> I don't see any sign of *anyone* working on a curated distribution for
> Windows along the lines of Linux distros or Homebrew. (Unless you
> count cross-platform stacks like conda, which IMO are a different
> scenario than "system" Python installs).

Under Windows, there isn't much moral difference between a conda install
(really a Miniconda or Anaconda install) and a python.org Python
install.  You can even install Anaconda or Miniconda system-wide.

(Miniconda is a minimal install of Python + conda, while Anaconda is a
pretty large selection of packages in addition - though not the entire
official repo contents, and not counting community packages)

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 23 August 2016 at 19:36, Paul Moore  wrote:
> So I don't think that in the medium term there's going to be much
> practical change in the state of things on Windows:
>
> - Users install Python from the published python.org installers
> - Users install packages using pip and wheels from PyPI
> - Plus some exceptions, where people need to use sdists, or
> independently published wheels, or worse still, wininst/msi installers
> because that's all available
>
> Whether that process is manual, or hidden behind some form of scripted
> process, won't alter the underlying infrastructure.
>
> I don't see any sign of *anyone* working on a curated distribution for
> Windows along the lines of Linux distros or Homebrew. (Unless you
> count cross-platform stacks like conda, which IMO are a different
> scenario than "system" Python installs).

OK, cool - that gives us all the more reason to retain bdist_wininst
and bdist_msi hosting support. However, I do think it makes sense for
us to say up front that we'll reconsider that decision if something
akin to homebrew gains traction amongst developers running Windows the
way homebrew has amongst open source users running Mac OS X.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Paul Moore]

On 23 August 2016 at 04:36, Nick Coghlan  wrote:
> I meant choco (community archive) and PackageManagement (system
> integration, formerly known as OneGet):
> https://blogs.technet.microsoft.com/packagemanagement/2015/04/28/introducing-packagemanagement-in-windows-10/

Neither Chocolatey nor OneGet do any form of repackaging or vendor
management, in the way that Linux distros do, though.

OneGet isn't a package manager - it's a "package manager manager" in
that it simply provides a mechanism to unify *other* package managers
(such as NuGet or Chocolatey) under a single command set.

Chocolatey *is* a package manager, but for projects like Python it
simply provides a script that says "Locate the Python MSI from this
URL, download and install it" - so nothing more than automated
instructions for what users are doing right now. And for Python
packages, it similarly just packages up a script for silent install.
There's very few such bundles for Python packages that I could find at
the moment, but if & when they do get contributed, I'd pretty much
hope that all they did was "pip install foo==1.2.3" (with a bit of
dependency and error checking).

So I don't think that in the medium term there's going to be much
practical change in the state of things on Windows:

- Users install Python from the published python.org installers
- Users install packages using pip and wheels from PyPI
- Plus some exceptions, where people need to use sdists, or
independently published wheels, or worse still, wininst/msi installers
because that's all available

Whether that process is manual, or hidden behind some form of scripted
process, won't alter the underlying infrastructure.

I don't see any sign of *anyone* working on a curated distribution for
Windows along the lines of Linux distros or Homebrew. (Unless you
count cross-platform stacks like conda, which IMO are a different
scenario than "system" Python installs).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 16 August 2016 at 05:09, Donald Stufft  wrote:
> Hello!
>
> I'd like to restrict what folks can upload to PyPI in an effort to help narrow
> the scope down and to enable more a more consistent experience for everyone.
>
> First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I 
> think
> that we should try to get rid of support for most of these. Just for reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.
>
> First of all, when I say "remove", I mean disallow new uploads, but do not
> delete the existing files.

[snip]

> Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these 
> together
> because they're all OS specific installers for OSs that don't already have 
> some
> sort of repository. This lack of a repository format for them means that 
> random
> downloads are already the norm for people using these systems. For these, I
> think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
> are not very important to continue to support, but it would be weird to
> eliminate them without also elminating bdist_wininst. The wininst format has
> the most usage out of all of the seldom used formats, however when we look at
> the downloads for the last 30 days only 0.42% of the downloads were for 
> wininst
> files, so I think that it's pretty safe to remove them.

Based on the discussion about Windows package management, I realised
that "percentage of overall downloads" is probably the wrong way to
measure the importance of the platform specific downloads, since those
percentages will be suppressed by the overwhelming dominance of sdist
and wheel downloads for sdist-only and sdist+wheel projects.

Instead, I think the numbers we need to make good data driven
decisions about publisher and end user impact here are:

- for projects publishing bdist_egg, what percentage of downloads are bdist_egg?
- for projects publishing bdist_wininst, what percentage of downloads
are bdist_wininst?
- ditto for bdist_msi and bdist_dmg

For bdist_rpm, the Fedora/CentOS/RHEL ecosystem has the Fedora COPR
service at http://copr.fedoraproject.org/ now, and openSuSE has long
offered http://openbuildservice.org/ to build and publish RPMs for a
range of RPM based distributions, so I'm already happy for us to end
bdist_rpm support on PyPI and direct people that want to publish RPMs
to those services instead.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 23 August 2016 at 06:53,   wrote:
>> The rationale for why the Windows formats get to stay when the other
>> platform specific formats are being dropped is implied by that last
>> line: we're expecting users on other platforms to be more comfortable
>> with using platform specific tooling to manage platform specific
>> formats (e.g. the system package manager on Linux, homebrew on Mac OS
>> X).
>>
>> Cheers,
>> Nick.
>
> I'm a heavy Windows user.  Are you aware of a system package manager that I 
> am not?  There's nuget (vs), choco (third party) and the Windows Store ...

I meant choco (community archive) and PackageManagement (system
integration, formerly known as OneGet):
https://blogs.technet.microsoft.com/packagemanagement/2015/04/28/introducing-packagemanagement-in-windows-10/

There's also the option of using conda and conda-forge as an option a
bit closer to what you get using homebrew on Mac OS X.

The reason I think we need to keep bdist_wininst and bdist_msi support
on PyPI for the time being is simply the fact that the idea of
software repositories and automated software management (rather than
downloading EXEs and MSIs through your web browser and running them)
is still a relatively novel concept on Windows, so it's going to take
time for it to be as accepted amongst Windows-focused Pythonistas as
comparable tools are amongst *nix focused developers.

However, now that Microsoft is actually providing support for
automated software management, I also think we should be encouraging
people to move in that direction, and if we're successful in that,
then eventually we'll be able to drop bdist_wininst and bdist_msi as
well, and nobody will miss them (since they're getting them from the
Chocalatey repo instead).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[tritium-list]

> The rationale for why the Windows formats get to stay when the other
> platform specific formats are being dropped is implied by that last
> line: we're expecting users on other platforms to be more comfortable
> with using platform specific tooling to manage platform specific
> formats (e.g. the system package manager on Linux, homebrew on Mac OS
> X).
> 
> Cheers,
> Nick.

I'm a heavy Windows user.  Are you aware of a system package manager that I am 
not?  There's nuget (vs), choco (third party) and the Windows Store ...

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Paul Moore]

On 22 August 2016 at 16:49, Nick Coghlan  wrote:
> - encouraging use of Windows Package Management over manual installer 
> execution
>
> The rationale for why the Windows formats get to stay when the other
> platform specific formats are being dropped is implied by that last
> line: we're expecting users on other platforms to be more comfortable
> with using platform specific tooling to manage platform specific
> formats (e.g. the system package manager on Linux, homebrew on Mac OS
> X).

Could you clarify what you mean by this? Current usage on Windows (for
people who aren't already using wheels) is to run a bdist_msi or
bsidt_wininst installer.

You seen to be suggesting "encouraging use of Windows Package
Management" over this approach, but I'm not sure what precisely that
means?
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 23 August 2016 at 00:46, Donald Stufft  wrote:
>
>> On Aug 22, 2016, at 7:14 AM, Nick Coghlan  wrote:
>>
>> Thanks, that's good info that shows I was clearly being unduly
>> pessimistic about toolchain compatibility. It means the only salient
>> technical difference we're aware of between the two formats is that
>> the distutils and setuptools default settings on .tar.gz currently
>> result in smaller archives than the default settings for .zip.
>
> I’m less worried about the Linux toolchains and I’m more worried about
> the adhoc toolchains created by all the various publishers on PyPI. It’s
> not a wide stretch to imagine release scripts that hard code in an
> assumption on either .tar.gz _or_ .zip and picking one or the other will
> inevitably break these people (albeit with a fairly simple fix in the
> typical case). Picking the lesser used format just increases the number
> of people who might end up having their release scripts or other misc
> scripts broken because of it.

Yeah, that's fair. To summarise where we're at right now:

- the change to always use .tar.gz by default has been made in
distutils for 3.6, so if we want to do something else (like continue
allowing both .tar.gz and .zip and keep the Windows default as .zip),
we need to make that clear before the final beta in November
(*reverting* a behaviour change like that is permitted within the beta
period - otherwise the beta period wouldn't be particularly useful,
since we couldn't respond to community feedback)

- folks seem to be broadly in favour of standardising on ".zip"
whenever "formally define a new iteration of the sdist format" makes
it to the top of the collective todo list

- however, if we were to consolidate on a single sdist format *right
now* (regardless of whether that's .tar.gz or .zip), the concern is
that we risk breaking ad hoc automation and other tooling without a
compelling sales pitch to the folks that need to update their
toolchains about how the change will make *their* lives better

Furthermore, at a language ecosystem level, we don't particularly want
to inflict more disruption on either *nix users *or* Windows users
that are sufficiently engaged with the Python community to be
publishing their own software on PyPI - we still have that everpresent
"change overload" concern, especially for folks that have been
affected by both Python 3 *and* the packaging tooling changes in
recent years.

So I think the next step would be to summarise the changes to release
file hosting support and the permitted extensions in a PEP, with
details along the following lines:

Permitted formats:

- sdist (*.zip, *.tar.gz)
- bdist_wheel (*.whl)
- bdist_egg (*.egg)
- bdist_wininst (*.exe)
- bdist_msi (*.msi)

Dropped formats:

- bdist_dumb
- bdist_rpm (*.rpm)
- bdist_dmg (*.dmg)

Possible future design changes:

- defining sdist 2.0 (as an explicitly defined zip-based format)
- defining a successor to the egg format for the non-wheel use cases
- encouraging use of Windows Package Management over manual installer execution

The rationale for why the Windows formats get to stay when the other
platform specific formats are being dropped is implied by that last
line: we're expecting users on other platforms to be more comfortable
with using platform specific tooling to manage platform specific
formats (e.g. the system package manager on Linux, homebrew on Mac OS
X).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 22, 2016, at 7:14 AM, Nick Coghlan  wrote:
> 
> Thanks, that's good info that shows I was clearly being unduly
> pessimistic about toolchain compatibility. It means the only salient
> technical difference we're aware of between the two formats is that
> the distutils and setuptools default settings on .tar.gz currently
> result in smaller archives than the default settings for .zip.


I’m less worried about the Linux toolchains and I’m more worried about
the adhoc toolchains created by all the various publishers on PyPI. It’s
not a wide stretch to imagine release scripts that hard code in an
assumption on either .tar.gz _or_ .zip and picking one or the other will
inevitably break these people (albeit with a fairly simple fix in the
typical case). Picking the lesser used format just increases the number
of people who might end up having their release scripts or other misc
scripts broken because of it.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Barry Warsaw]

On Aug 22, 2016, at 09:14 PM, Nick Coghlan wrote:

>In that case, perhaps the way to go for sdists (at least for now)
>would be to continue allowing both .tar.gz and .zip, but disallow
>uploading both of them for any given release?

I'd prefer allowing uploading of both formats for now, with mild encouragement
to at least include zips.

I'm fairly certain the Debian Python ecosystem in general can consume zip
sdists without problem, although there may be some individual packages that
need some adjustment.  I don't have any sense of large an impact it would be
to switch sdists from tar.gz to zip, but allowing for some overlap with
"gentle nudges" toward the preferred format would allow us to do the analysis
and give us some time to plan a transition.

Personally, I don't much care if the sdist format is tar.gz or zip, and it
does make sense to promote one archive format for both source and binary
distributions.  I mildly prefer tar.gz but for pretty shallow reasons
(i.e. not good enough :): one tool to rule them all (tar(1) both packs and
unpacks, while zip needs two commands), and I've been using tar for way longer
than zip so I'm a little more comfortable with it.

What I'd want most of all is a clearly documented transition plan, i.e. a PEP
that I can point other people to and say "we need to make sure all our tooling
is in place by date X because that's when .tar.gz is going away".

Cheers,
-Barry


pgpM_wi_53rBa.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 22 August 2016 at 09:03, Nathaniel Smith  wrote:
> On Aug 21, 2016 9:18 AM, "Nick Coghlan"  wrote:
>>
> [...]
>> By contrast, I *know* Linux distros are in the habit of pulling
>> release tarballs from PyPI and feeding them directly into their
>> release pipelines, so the potential for unanticipated breakage seems
>> much higher there, and more likely to fall on community distros rather
>> than on commercial Python redistributors (simply because there are
>> more volunteers working on Linux based tooling than there are on
>> WIndows developer tools).
>
> But those release pipelines today have no problem handling the ~10% of
> sdists that are zips. For my personal projects actually I've always released
> .zip sdists exclusively (just because it seemed like a more universally
> convenient format than .tar.gz), and those packages have shown up in distros
> and no one has ever complained. Are any distros really hardcoding an
> assumption that .tar.gz is that only possible sdist format?

Thanks, that's good info that shows I was clearly being unduly
pessimistic about toolchain compatibility. It means the only salient
technical difference we're aware of between the two formats is that
the distutils and setuptools default settings on .tar.gz currently
result in smaller archives than the default settings for .zip.

In that case, perhaps the way to go for sdists (at least for now)
would be to continue allowing both .tar.gz and .zip, but disallow
uploading both of them for any given release?

The only remaining thing holding me back from being gung-ho about "OK,
let's just standardise on .zip, then" is the filesize consideration,
since that means implicitly asking Fastly to donate more sponsored
bandwidth, and likewise for anyone else running private PyPI mirrors,
etc.

Then sdist 2.0 (whenever that ends up bugging someone enough for them
to set out to define it) can be defined as a zip format, but with
compression applied to match or better the file size of the current
.tar.gz files.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nathaniel Smith]

On Aug 21, 2016 9:18 AM, "Nick Coghlan"  wrote:
>
[...]
> By contrast, I *know* Linux distros are in the habit of pulling
> release tarballs from PyPI and feeding them directly into their
> release pipelines, so the potential for unanticipated breakage seems
> much higher there, and more likely to fall on community distros rather
> than on commercial Python redistributors (simply because there are
> more volunteers working on Linux based tooling than there are on
> WIndows developer tools).

But those release pipelines today have no problem handling the ~10% of
sdists that are zips. For my personal projects actually I've always
released .zip sdists exclusively (just because it seemed like a more
universally convenient format than .tar.gz), and those packages have shown
up in distros and no one has ever complained. Are any distros really
hardcoding an assumption that .tar.gz is that only possible sdist format?

(I'm agnostic on the overall question of whether to prefer .zip or .tar.gz
as the standard -- both seem reasonable with their own tradeoffs.)

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Robert Collins]

On 22 Aug 2016 4:18 AM, "Nick Coghlan"  wrote:
>
>
> Once we have a universal default, unilaterally changing that default
> gets easier, rather than harder - the main precedent being set here is
> that we *don't want* the default format to be platform dependent, we
> want there to be just one default.

Sorry for the separate reply, phone typing.

Monocultures are less flexible, so I'm assuming once we have Just One
format that bit rot in tooling will creep in and it will be harder and
harder to change, not easier.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Robert Collins]

On 22 Aug 2016 4:18 AM, "Nick Coghlan"  wrote:
>
> On 21 August 2016 at 18:21, Robert Collins 
wrote:
> > tl;dr: I think standardising on .tar.gz would be a rather shortsighted
> > thing to do, given how many Windows users Python has and how much of a
> > different supporting .zip makes for workflow on that platform - with
> > no negative impacts on any other platform.
>
> Once we have a universal default, unilaterally changing that default
> gets easier, rather than harder - the main precedent being set here is
> that we *don't want* the default format to be platform dependent, we
> want there to be just one default.
>
> My core assumption in recommending starting with tar.gz as that
> universal default is that Windows users will mostly be interacting
> with sdists through tooling that is closely linked to or at least
> originated in the Python ecosystem (easy_install, pip, PyPM, conda,
> Enthought Canopy, buildout, Christoph Gohlke's Windows installers),
> and that double-clicking a Python sdist in an Explorer window is not
> something most folks are in the habit of doing.

It is certainly wrong for me, when I'm using Windows, and when I've
shoulder surfed others also. E.g. at pycon sprints.

> By contrast, I *know* Linux distros are in the habit of pulling
> release tarballs from PyPI and feeding them directly into their
> release pipelines, so the potential for unanticipated breakage seems
> much higher there, and more likely to fall on community distros rather
> than on commercial Python redistributors (simply because there are
> more volunteers working on Linux based tooling than there are on
> WIndows developer tools).

Those pipelines are mature, poly format capable and centralized, containing
the damage.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

Targz is about 3/4 the size of zip for a bag of Python dists I tested. Zip
inside a second zip would provide the same compression. No word on the
Weissman score.

Tar isn't exactly a single format. For Unicode try POSIX-1.2001 aka pax
format tar. Python defaults to gnutar. Zip has good Unicode support.

Exactly how much simpler is exactly one file format?

On Sun, Aug 21, 2016, 06:57 Paul Moore  wrote:

> On 21 August 2016 at 09:21, Robert Collins 
> wrote:
> >
> > tl;dr: I think standardising on .tar.gz would be a rather shortsighted
> > thing to do, given how many Windows users Python has and how much of a
> > different supporting .zip makes for workflow on that platform - with
> > no negative impacts on any other platform.
>
> One thing that has (IIRC) come up in a pip bug in the past - how do
> tar and zip format fare in terms of Unicode support? IIRC, older
> versions of (I think) tar format don't include an encoding, nor do
> they mandate UTF-8, so they have the potential to break when used
> cross-platform. Sorry, I can't recall exact details.
>
> I think it's important that whatever format we mandate works with full
> Unicode filenames, and the available user tools support that.
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Paul Moore]

On 21 August 2016 at 09:21, Robert Collins  wrote:
>
> tl;dr: I think standardising on .tar.gz would be a rather shortsighted
> thing to do, given how many Windows users Python has and how much of a
> different supporting .zip makes for workflow on that platform - with
> no negative impacts on any other platform.

One thing that has (IIRC) come up in a pip bug in the past - how do
tar and zip format fare in terms of Unicode support? IIRC, older
versions of (I think) tar format don't include an encoding, nor do
they mandate UTF-8, so they have the potential to break when used
cross-platform. Sorry, I can't recall exact details.

I think it's important that whatever format we mandate works with full
Unicode filenames, and the available user tools support that.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Robert Collins]

On 21 August 2016 at 15:22, Nick Coghlan  wrote:
> On 21 August 2016 at 08:40, Chris Barker - NOAA Federal
>  wrote:
>>>
>>> If we're going through all this trouble, isn't it better just to jump to 
>>> .zip files like every other distribution format in existence?
>>
>> Yes. :-)
>
> zip is popular for *user facing* distribution formats, where the file
> format is exposed directly to end users, and end user applications
> (e.g. zipimport, ODF).
>
> That's not where tarballs shine, so it's not where they show up: tar
> is a backend infrastructure format, used for tool-to-tool
> communication, rather than user-to-user.
>
> The trajectory of distutils-sig for the past few years has been
> towards a world where sdists are primarily a tool-to-tool format for
> data interchange between publishing tools (e.g. twine), installation
> tools (e.g. pip) and redistribution tools (e.g. conda, RPM, deb,
> buildout), with wheels (and potentially eggs, or an equally zipimport
> friendly future derivative) being the preferred formats for working
> directly with Python software in a destination environment (whether
> that's a system integrator's build farm, a production software
> deployment, or a Python user's local system).

Jar's are zip's (~= wheels). *source* jars are also zips. (~= sdists).
I'm struggling to think of any two ways that .tar.gz is better than
.zip. The compression is ~=. .tar.lzma and friends can be radically
smaller whereas .zip has per-element compression, so thats one way,
but .tar.gz has no other benefits (it has other differences, but none
are relevant in the context of the sdist use cases AFAICT).

As far as the switching consts Donald raises, here's my take:
 - we're going to break a lot of peoples workflow no matter what: 10%
of a lot is a lot.
 - all setuptools in the while /can/ produce .zip's AIUI, so the
issues w.r.t. the long tail of Linux distros is entirely
uninteresting: the folk depending on the default behaviour will need
to be explict - but thats going to be the case for *everyone* [because
setuptools on windows is inconsistent with setuptools on Linux]: we
are basically telling everyone they have to hardcode to .zip until the
long tail of existing installs *everywhere* has gone away.
 - unzip and friends are as available on Linux distros as tar is.

tl;dr: I think standardising on .tar.gz would be a rather shortsighted
thing to do, given how many Windows users Python has and how much of a
different supporting .zip makes for workflow on that platform - with
no negative impacts on any other platform.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 21 August 2016 at 08:40, Chris Barker - NOAA Federal
 wrote:
>>
>> If we're going through all this trouble, isn't it better just to jump to 
>> .zip files like every other distribution format in existence?
>
> Yes. :-)

zip is popular for *user facing* distribution formats, where the file
format is exposed directly to end users, and end user applications
(e.g. zipimport, ODF).

That's not where tarballs shine, so it's not where they show up: tar
is a backend infrastructure format, used for tool-to-tool
communication, rather than user-to-user.

The trajectory of distutils-sig for the past few years has been
towards a world where sdists are primarily a tool-to-tool format for
data interchange between publishing tools (e.g. twine), installation
tools (e.g. pip) and redistribution tools (e.g. conda, RPM, deb,
buildout), with wheels (and potentially eggs, or an equally zipimport
friendly future derivative) being the preferred formats for working
directly with Python software in a destination environment (whether
that's a system integrator's build farm, a production software
deployment, or a Python user's local system).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

I don't think we are defining a new file type today.

On Sat, Aug 20, 2016 at 8:03 PM Wes Turner  wrote:

>
>
> On Saturday, August 20, 2016, Brett Cannon  wrote:
>
>> On Fri, 19 Aug 2016 at 12:53 Donald Stufft  wrote:
>>
>
>>>
>>> Oh, and TIL that anyone who has Python 3.4+ installed has a command line
>>> tool for extracting ``.tar.gz`` files [2]
>>>
>>
>> So I think you're both right, but at different time scales. :) I think
>> Donald is right that the short-term time scale of "now" by suggesting we
>> just go with tar.gz since it has the numbers. But I think Leonardo's point
>> of general alignment with zip for packaging overall is good for the
>> "formally define sdist" time scale and we potentially introduce an .sdist
>> file extension.
>>
>
> How about, as a convention,
> .sdist.zip
> .sdist.tar.gz
>
> So that file type associations with archive programs still work?
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Wes Turner]

On Saturday, August 20, 2016, Brett Cannon  wrote:

>
>
> On Fri, 19 Aug 2016 at 12:53 Donald Stufft  > wrote:
>
>>
>>
>> Oh, and TIL that anyone who has Python 3.4+ installed has a command line
>> tool for extracting ``.tar.gz`` files [2]
>>
>
> So I think you're both right, but at different time scales. :) I think
> Donald is right that the short-term time scale of "now" by suggesting we
> just go with tar.gz since it has the numbers. But I think Leonardo's point
> of general alignment with zip for packaging overall is good for the
> "formally define sdist" time scale and we potentially introduce an .sdist
> file extension.
>

How about, as a convention,
.sdist.zip
.sdist.tar.gz

So that file type associations with archive programs still work?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Chris Barker - NOAA Federal]

>
> If we're going through all this trouble, isn't it better just to jump to .zip 
> files like every other distribution format in existence?

Yes. :-)

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 21 August 2016 at 02:33, Brett Cannon  wrote:
> On Fri, 19 Aug 2016 at 12:53 Donald Stufft  wrote:
> So I think you're both right, but at different time scales. :) I think
> Donald is right that the short-term time scale of "now" by suggesting we
> just go with tar.gz since it has the numbers. But I think Leonardo's point
> of general alignment with zip for packaging overall is good for the
> "formally define sdist" time scale and we potentially introduce an .sdist
> file extension.

I think tarballs are a better long term release archiving format at
any time scale - they just assume you're going to untar them before
trying to do anything useful with them, which is a reasonable workflow
requirement to impose for sdists. While relatively few people actually
need to work directly with tarballs in that context, that's just
because they've faded into the background of various build systems
that produce some other format (e.g. RPMs, deb archives, Docker
container layers).

By contrast, for Python's built formats like wheel and egg, it's
useful to be able to easily access their contents *without* unpacking
them first, so it makes sense to go with the format that's friendlier
to that model (i.e. zip).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Jim Fulton]

On Tue, Aug 16, 2016 at 11:15 AM, Leonardo Rochael Almeida <
leoroch...@gmail.com> wrote:

...


> A wheel2egg might be nice, but unless it's integrated into setuptools
> proper, it would mean adding another dependency to buildout and the
> developers would rather depend on a single library for talking to PyPI
>

I'm not sure this is as important as it once was.  Depending on setuptools
and it's bootstrapping mechanism easier, but setuptools' bootstrapping
functionality is going away. We've discussed giving up on allowing buildout
to bootstrap itself without modifying the Python install, although I still
rather like that feature.

Jim

-- 
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Brett Cannon]

On Fri, 19 Aug 2016 at 12:53 Donald Stufft  wrote:

>
> On Aug 19, 2016, at 2:00 PM, Leonardo Rochael Almeida <
> leoroch...@gmail.com> wrote:
>
> ure, more people will be affected this way than just the folks releasing
> on Windows, but given the shortcuts for setting the sdist format per
> project or per home directory that Donald mentioned, I think the collective
> effort in the migration will be smaller than the continuous effort of
> explaining to newcomers that the reason we use a .tar.gz based format for
> sdists versus a .zip based format for wheels is some historical accident,
> specially if we plan to change sdists back to .zip format in the future...
>
>
>
> I don’t think I’ve ever had anyone ask me why people (generally) use
> .tar.gz for sdists and why wheels use zip, though I don’t do much beginner
> stuff. Do you have some sort of evidence or data to suggest that this is a
> problem that people are experiencing or are you theorizing that folks
> *might* get confused by this?
>
> I think the effort changing non-Windows platform is going to be a lot more
> effort than changing Windows platform for a few reasons:
>
> * There are less people releasing on Windows than on non-Windows, the more
> people you need to migrate to a new thing, the longer you can expect it to
> take.
>
> * Windows does not come with Python, thus people are generally free to
> upgrade their Python or setuptools installation at all [1] meanwhile Python
> and setuptools tends to be a core part of non-Windows OSs where upgrading
> one or the other can “void the warranty” and cause breakage to the entire
> OS, combine this with things like CentOS, RHEL, Ubuntu LTS and we lengthen
> the time that people are using these older tools by years, maybe even a
> decade.
>
> * More people are using .tar.gz than .zip, which means changing from
> .tar.gz is more likely to cause issues (under the assumption that any
> change can cause issue, and the more people who have some sort of change
> occur to them, the more likely an issue is to occur).
>
> Oh, and TIL that anyone who has Python 3.4+ installed has a command line
> tool for extracting ``.tar.gz`` files [2]
>

So I think you're both right, but at different time scales. :) I think
Donald is right that the short-term time scale of "now" by suggesting we
just go with tar.gz since it has the numbers. But I think Leonardo's point
of general alignment with zip for packaging overall is good for the
"formally define sdist" time scale and we potentially introduce an .sdist
file extension.

IOW I think we're all starting to talk in circles and since no one has
screamed "the sky is falling" against the proposal, it's probably time to
start formalizing a plan.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 19, 2016, at 2:00 PM, Leonardo Rochael Almeida  
> wrote:
> 
> ure, more people will be affected this way than just the folks releasing on 
> Windows, but given the shortcuts for setting the sdist format per project or 
> per home directory that Donald mentioned, I think the collective effort in 
> the migration will be smaller than the continuous effort of explaining to 
> newcomers that the reason we use a .tar.gz based format for sdists versus a 
> .zip based format for wheels is some historical accident, specially if we 
> plan to change sdists back to .zip format in the future...


I don’t think I’ve ever had anyone ask me why people (generally) use .tar.gz 
for sdists and why wheels use zip, though I don’t do much beginner stuff. Do 
you have some sort of evidence or data to suggest that this is a problem that 
people are experiencing or are you theorizing that folks *might* get confused 
by this?

I think the effort changing non-Windows platform is going to be a lot more 
effort than changing Windows platform for a few reasons:

* There are less people releasing on Windows than on non-Windows, the more 
people you need to migrate to a new thing, the longer you can expect it to take.

* Windows does not come with Python, thus people are generally free to upgrade 
their Python or setuptools installation at all [1] meanwhile Python and 
setuptools tends to be a core part of non-Windows OSs where upgrading one or 
the other can “void the warranty” and cause breakage to the entire OS, combine 
this with things like CentOS, RHEL, Ubuntu LTS and we lengthen the time that 
people are using these older tools by years, maybe even a decade.

* More people are using .tar.gz than .zip, which means changing from .tar.gz is 
more likely to cause issues (under the assumption that any change can cause 
issue, and the more people who have some sort of change occur to them, the more 
likely an issue is to occur).

Oh, and TIL that anyone who has Python 3.4+ installed has a command line tool 
for extracting ``.tar.gz`` files [2]


[1] Yes, some people *might* not have administrator access on their box and be 
unable to do that, but whoever *is* responsible for those boxes can.

[2] https://docs.python.org/3.4/library/tarfile.html#command-line-interface 


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Leonardo Rochael Almeida]

On 19 August 2016 at 14:14, Paul Moore  wrote:

> [...]
>
> So, the plan becomes:
>
> 1. Change Python 3.6 to default to .tar.gz on Windows
> 2. Change setuptoos to default to .tar.gz, to catch users of older versions
> 3. Document how to create a source distribution as "python setup.py
> sdist (add the --formats=gztar flag if you're on Windows and using a
> Python older than 3.6 and not using setuptools)" in the PUG
> 4. Update https://docs.python.org/3/distutils/sourcedist.html for 3.6
> to note the change in default.
>
> For (3) soften the laboured conditional comment as much as you like,
> but I think we need something to help people who don't understand
> what's going on. For (1) and (4), maybe we don't target 3.6 (given
> setuptools, it's not urgent) but I think we should change the core, if
> only to avoid surprises for people who don't include setuptools in
> their setup.py (maybe relying on pip injecting it for things like
> wheel builds), and so that distutils isn't (in effect) deliberately
> wrong going forward.
>
> > While I think it’s OK to allow .zip in the interim, I do think we should
> be
> > trying to move towards only allowing a single format.
>
> Your arguments are compelling, I have no problem with the conclusion -
> we just need a little more thought on the transition.
>

If we're going through all this trouble, isn't it better just to jump to
.zip files like every other distribution format in existence?

Sure, more people will be affected this way than just the folks releasing
on Windows, but given the shortcuts for setting the sdist format per
project or per home directory that Donald mentioned, I think the collective
effort in the migration will be smaller than the continuous effort of
explaining to newcomers that the reason we use a .tar.gz based format for
sdists versus a .zip based format for wheels is some historical accident,
specially if we plan to change sdists back to .zip format in the future...

Regards,

Leo
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Paul Moore]

On 19 August 2016 at 18:21, Donald Stufft  wrote:
> FWIW, they can also drop a setup.cfg in their project too that looks
> like: https://bpaste.net/show/90dd1280eba6 and then forget about it, which
> makes it somewhat less painful, since they won’t have to remember to do it
> on a per project basis past the first time. I believe that can also be put
> in ~/.pydistutils.cfg (or w/e that file is again…) to change the default on
> a per user basis.

Good point, that's a much better idea.

> Sure! I have some other things to do today, but I’ll try and figure out
> something a bit more fleshed out for the transition given this new (to
> me) information.

Cool. As I say, it was new to me as well, which I'm sad about as I'm
supposed to be the "remember Windows" voice around here ;-)

FWIW, even apart from the process of simplifying things for PyPI, I
think this is a good move just to eliminate the annoying
platform-dependent default behaviour.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 19, 2016, at 1:14 PM, Paul Moore  wrote:
> 
> On 19 August 2016 at 16:14, Donald Stufft  wrote:
>> Not sure if we’ll be able to back port it to 2.7 itself, but it’s a trivial
>> change to make inside of setuptools as well. Here’s a PR to setuptools that
>> will adjust the default: https://github.com/pypa/setuptools/pull/748.
> 
> Hmm. So the question becomes, do we care about supporting users
> building sdists without setuptools? It's not like this is a scenario
> controlled by pip, which auto-injects setuptools. So I guess we'd need
> to also publicise the --formats=gztar flag (at
> https://packaging.python.org/distributing/#source-distributions if
> nowhere else).
> 
> So, the plan becomes:
> 
> 1. Change Python 3.6 to default to .tar.gz on Windows
> 2. Change setuptoos to default to .tar.gz, to catch users of older versions
> 3. Document how to create a source distribution as "python setup.py
> sdist (add the --formats=gztar flag if you're on Windows and using a
> Python older than 3.6 and not using setuptools)" in the PUG
> 4. Update https://docs.python.org/3/distutils/sourcedist.html for 3.6
> to note the change in default.
> 
> For (3) soften the laboured conditional comment as much as you like,
> but I think we need something to help people who don't understand
> what's going on. For (1) and (4), maybe we don't target 3.6 (given
> setuptools, it's not urgent) but I think we should change the core, if
> only to avoid surprises for people who don't include setuptools in
> their setup.py (maybe relying on pip injecting it for things like
> wheel builds), and so that distutils isn't (in effect) deliberately
> wrong going forward.


FWIW, they can also drop a setup.cfg in their project too that looks
like: https://bpaste.net/show/90dd1280eba6 and then forget about it, which
makes it somewhat less painful, since they won’t have to remember to do it
on a per project basis past the first time. I believe that can also be put
in ~/.pydistutils.cfg (or w/e that file is again…) to change the default on
a per user basis.


>> While I think it’s OK to allow .zip in the interim, I do think we should be
>> trying to move towards only allowing a single format.
> 
> Your arguments are compelling, I have no problem with the conclusion -
> we just need a little more thought on the transition.
> 

Sure! I have some other things to do today, but I’ll try and figure out 
something
a bit more fleshed out for the transition given this new (to me) information.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Paul Moore]

On 19 August 2016 at 16:14, Donald Stufft  wrote:
> Not sure if we’ll be able to back port it to 2.7 itself, but it’s a trivial
> change to make inside of setuptools as well. Here’s a PR to setuptools that
> will adjust the default: https://github.com/pypa/setuptools/pull/748.

Hmm. So the question becomes, do we care about supporting users
building sdists without setuptools? It's not like this is a scenario
controlled by pip, which auto-injects setuptools. So I guess we'd need
to also publicise the --formats=gztar flag (at
https://packaging.python.org/distributing/#source-distributions if
nowhere else).

So, the plan becomes:

1. Change Python 3.6 to default to .tar.gz on Windows
2. Change setuptoos to default to .tar.gz, to catch users of older versions
3. Document how to create a source distribution as "python setup.py
sdist (add the --formats=gztar flag if you're on Windows and using a
Python older than 3.6 and not using setuptools)" in the PUG
4. Update https://docs.python.org/3/distutils/sourcedist.html for 3.6
to note the change in default.

For (3) soften the laboured conditional comment as much as you like,
but I think we need something to help people who don't understand
what's going on. For (1) and (4), maybe we don't target 3.6 (given
setuptools, it's not urgent) but I think we should change the core, if
only to avoid surprises for people who don't include setuptools in
their setup.py (maybe relying on pip injecting it for things like
wheel builds), and so that distutils isn't (in effect) deliberately
wrong going forward.

> While I think it’s OK to allow .zip in the interim, I do think we should be
> trying to move towards only allowing a single format.

Your arguments are compelling, I have no problem with the conclusion -
we just need a little more thought on the transition.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 19, 2016, at 10:34 AM, Paul Moore  wrote:
> 
> On 19 August 2016 at 15:27, Donald Stufft  wrote:
>> This is an important consideration that nobody mentioned thus far— I have
>> not used Windows to release a project to PyPI.. ever. So We should
>> absolutely start by switching that to defaulting to .tar.gz.
> 
> I had forgotten this fact, but yes, zip is the standard for sdists on Windows.
> 
> I would suggest that we need to allow both .zip and .tar.gz for
> sdists. Specifically, how would we handle people building sdists for
> upload using Python 2.7 on Windows? Would we backport this change?

Not sure if we’ll be able to back port it to 2.7 itself, but it’s a trivial
change to make inside of setuptools as well. Here’s a PR to setuptools that
will adjust the default: https://github.com/pypa/setuptools/pull/748.

While I think it’s OK to allow .zip in the interim, I do think we should be
trying to move towards only allowing a single format. I honestly don’t care
what that format is, but I think it’ll be easier to move Windows users onto
.tar.gz than it will be to move everyone else to .tar.gz (smaller user base,
Python and setuptools are not “ingrained” into the System like is the case
on RHEL/CentOS so easier upgrades, etc).

My motivation on this is trying to normalize things across platforms and to
fix persistent sort of weirdness things that I see coming up over and over.
One such one is people getting different results (for whatever reason) when
a project has two sdists available. So I’ve been working on trying to make
that the case, and it would greatly simplify things if there were only one
filename acceptable for a single (Project, Version)’s sdist. I also think it
helps make other tooling around PyPI easier to write, and it helps ensure
that the dependencies to install something are minimal (though not particularly
relevant for zip/gztar since they both use zlib).

> 
> (It doesn't matter whether we say ".tar.gz is the standard, what do we
> do about Windows users?" or ".zip is the standard, what do we do about
> Unix users?" the problems are the same).
> 
> Paul


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Paul Moore]

On 19 August 2016 at 15:27, Donald Stufft  wrote:
> This is an important consideration that nobody mentioned thus far— I have
> not used Windows to release a project to PyPI.. ever. So We should
> absolutely start by switching that to defaulting to .tar.gz.

I had forgotten this fact, but yes, zip is the standard for sdists on Windows.

I would suggest that we need to allow both .zip and .tar.gz for
sdists. Specifically, how would we handle people building sdists for
upload using Python 2.7 on Windows? Would we backport this change?

(It doesn't matter whether we say ".tar.gz is the standard, what do we
do about Windows users?" or ".zip is the standard, what do we do about
Unix users?" the problems are the same).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 19, 2016, at 10:23 AM, Daniel Holth  wrote:
> 
> Consider that "setup.py sdist" produces .zip by default on Windows.


This is an important consideration that nobody mentioned thus far— I have not 
used Windows to release a project to PyPI.. ever. So We should absolutely start 
by switching that to defaulting to .tar.gz.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

The only issue I have with the proposal is the removal of .zip sdists
because I would personally be inconvenienced by the removal of .zip.
Consider that "setup.py sdist" produces .zip by default on Windows.

On Fri, Aug 19, 2016 at 9:36 AM James Bennett  wrote:

> Every new configuration option added to a piece of software represents two
> opportunities:
>
> 1. An opportunity to introduce exciting new bugs.
>
> 2. An opportunity to introduce exciting new ways for users to click the
> thing that does the exact opposite of what they wanted to do.
>
> The proliferation of different package formats is not a sign of
> innovation; it's a sign of the desperation induced by how bad Python
> packaging was, and for how long. Now that things are in a much better
> state, it's time to remove the combinatorial explosion of opportunities for
> bugs and end-user mistakes, and be able to say clearly "here's how you
> package and distribute Python code".
>
> Comparing that to to "hostile architecture" is petulant and inappropriate
> and you know it.
>
>
> On Fri, Aug 19, 2016 at 5:46 AM, Daniel Holth  wrote:
>
>> Check out this Camden Bench. https://en.wikipedia.org/wiki/Camden_bench
>>  . The creators of the bench enumerated about 23 specific undesirable
>> behaviors that they feel normal benches allow (like sleeping and
>> skateboarding) and came up with a lump of concrete that you can sit on.
>>
>> On Fri, Aug 19, 2016 at 8:14 AM Donald Stufft  wrote:
>>
>>>
>>> > On Aug 19, 2016, at 2:53 AM, Nick Coghlan  wrote:
>>> >
>>> > Unilaterally turning the feature off would be extraordinarily hostile
>>> > to current users - grace periods and sunset clauses are standard
>>> > features of change management processes for a reason, even when they
>>> > come at the cost of additional implementation complexity.
>>>
>>>
>>> This wouldn’t be a new way of handling this, when we implemented PEP 470
>>> new projects immediately got set to the “only files on PyPI mode” and had
>>> no ability to change to a different mode (no reason to present an option
>>> that was going away). Emails were sent to maintains on all projects that
>>> had an existing external link and told that in X months all their
>>> external
>>> links would be removed (which was implemented just as switching them to
>>> the
>>> “only files on PyPI mode”.
>>>
>>> Therefore, given what’s been discussed thus far, my proposal would be:
>>>
>>> Add a hidden flag, “legacy file support” on a per project basis. All new
>>> projects
>>> have this flag switched off, any existing project that has not ever
>>> uploaded a
>>> file that would use this flag has it switched off, everyone else has it
>>> switched
>>> on. Emails get sent out to maintainers of projects where it is still
>>> switched on
>>> and they’re told “In 3 months you’ll no longer be able to upload legacy
>>> file types,
>>> but all existing files uploaded will continue to exist”.
>>>
>>> Files that would fall under legacy file type:
>>>
>>> * All types except sdist, bdist_wheel, and bdist_egg [1].
>>> * All sdist extensions besides .tar.gz.
>>>
>>>
>>> That prevents new (and existing) projects from getting to a point where
>>> they
>>> depend on something that is going away when they previously didn’t and
>>> gives
>>> existing projects a chance to update their automation and what not to
>>> handle
>>> this scenario.
>>>
>>>
>>> [1] We can tackle egg at a later point, when setuptools either has
>>> support for Wheels
>>> or is less needed.
>>>
>>> —
>>> Donald Stufft
>>>
>>>
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 19, 2016, at 8:46 AM, Daniel Holth  wrote:
> 
> Check out this Camden Bench. https://en.wikipedia.org/wiki/Camden_bench 
>  . The creators of the bench 
> enumerated about 23 specific undesirable behaviors that they feel normal 
> benches allow (like sleeping and skateboarding) and came up with a lump of 
> concrete that you can sit on.


Ok?

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

Check out this Camden Bench. https://en.wikipedia.org/wiki/Camden_bench . The
creators of the bench enumerated about 23 specific undesirable behaviors
that they feel normal benches allow (like sleeping and skateboarding) and
came up with a lump of concrete that you can sit on.

On Fri, Aug 19, 2016 at 8:14 AM Donald Stufft  wrote:

>
> > On Aug 19, 2016, at 2:53 AM, Nick Coghlan  wrote:
> >
> > Unilaterally turning the feature off would be extraordinarily hostile
> > to current users - grace periods and sunset clauses are standard
> > features of change management processes for a reason, even when they
> > come at the cost of additional implementation complexity.
>
>
> This wouldn’t be a new way of handling this, when we implemented PEP 470
> new projects immediately got set to the “only files on PyPI mode” and had
> no ability to change to a different mode (no reason to present an option
> that was going away). Emails were sent to maintains on all projects that
> had an existing external link and told that in X months all their external
> links would be removed (which was implemented just as switching them to the
> “only files on PyPI mode”.
>
> Therefore, given what’s been discussed thus far, my proposal would be:
>
> Add a hidden flag, “legacy file support” on a per project basis. All new
> projects
> have this flag switched off, any existing project that has not ever
> uploaded a
> file that would use this flag has it switched off, everyone else has it
> switched
> on. Emails get sent out to maintainers of projects where it is still
> switched on
> and they’re told “In 3 months you’ll no longer be able to upload legacy
> file types,
> but all existing files uploaded will continue to exist”.
>
> Files that would fall under legacy file type:
>
> * All types except sdist, bdist_wheel, and bdist_egg [1].
> * All sdist extensions besides .tar.gz.
>
>
> That prevents new (and existing) projects from getting to a point where
> they
> depend on something that is going away when they previously didn’t and
> gives
> existing projects a chance to update their automation and what not to
> handle
> this scenario.
>
>
> [1] We can tackle egg at a later point, when setuptools either has support
> for Wheels
> or is less needed.
>
> —
> Donald Stufft
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 19, 2016, at 2:53 AM, Nick Coghlan  wrote:
> 
> Unilaterally turning the feature off would be extraordinarily hostile
> to current users - grace periods and sunset clauses are standard
> features of change management processes for a reason, even when they
> come at the cost of additional implementation complexity.


This wouldn’t be a new way of handling this, when we implemented PEP 470
new projects immediately got set to the “only files on PyPI mode” and had
no ability to change to a different mode (no reason to present an option
that was going away). Emails were sent to maintains on all projects that
had an existing external link and told that in X months all their external
links would be removed (which was implemented just as switching them to the
“only files on PyPI mode”.

Therefore, given what’s been discussed thus far, my proposal would be:

Add a hidden flag, “legacy file support” on a per project basis. All new 
projects
have this flag switched off, any existing project that has not ever uploaded a
file that would use this flag has it switched off, everyone else has it switched
on. Emails get sent out to maintainers of projects where it is still switched on
and they’re told “In 3 months you’ll no longer be able to upload legacy file 
types,
but all existing files uploaded will continue to exist”. 

Files that would fall under legacy file type:

* All types except sdist, bdist_wheel, and bdist_egg [1].
* All sdist extensions besides .tar.gz.


That prevents new (and existing) projects from getting to a point where they
depend on something that is going away when they previously didn’t and gives
existing projects a chance to update their automation and what not to handle
this scenario.


[1] We can tackle egg at a later point, when setuptools either has support for 
Wheels
or is less needed.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 19 August 2016 at 15:32,   wrote:
> From: Distutils-SIG [mailto:distutils-sig-bounces+tritium-
>> Alternatively, we could simply not worry about a user level flag, and
>> just have a project level flag that's set to "No legacy formats" by
>> default for new projects - new users won't have any incentive to
>> change it, while existing users can change it (at least for the time
>> being) if that suits their workflow.
>
> Well... what if I am a new user, opening a new project to work with legacy
> systems?

*If* we went down the sunsetting path I suggested (which is by no
means a given, as even though I think it's an option worth discussing,
it still adds complexity and controversy for debatable benefit), then
the expectation would be that anyone in this situation will either be
explicitly tasked with modernising their infrastructure, or else
working with an existing maintainer of the legacy infrastructure that
can configure these newly created legacy projects appropriately.

Folks that *don't* have a previous maintainer to help out and *also*
don't have permission to modernise their infrastructure to align with
current community expectations would then have to go to battle with
their Finance department (since an active unwillingness to modernise
toolchains almost always indicates the presence of a Finance
department) to argue for investment in tooling modernisation, rather
than expecting the open source community to help them workaround their
organisational dysfunction indefinitely.

> How is it fair to me that I didn't get to the game in time to have
> my files accepted and the old-timers get to have theirs accepted?

The UX concern with leaving it enabled as a config option for everyone
is that we potentially miss out on the main goals of the change:
simplification of the new user experience, and simplification of
future tooling development.

Instead, we may accidentally make the new user experience *more
complicated*, as there's now another project config option for them to
learn about. Configuration options should always be considered bad by
default - they complicate test matrices, they complicate maintenance
(of both the service itself and of other tools that interoperate with
it), they complicate documentation, and they complicate the learning
process. Sometimes their benefits outweigh those inherent
disadvantages, but "This behaviour will not be configurable" remains
the default position (otherwise we'd never get anything done in
software at all)

Now, sometimes providing a config option to re-enable deprecated
legacy behaviour is a useful transition tool, but there still needs to
be clear guidance given to authors of documentation and developers of
related tools that there's no need for them to cover or support that
legacy behaviour if they don't want to.

> If we
> shut it off for everyone, its fair.  If we let anyone turn it back on, its
> fair.  I think this is exemplary of a trend on this sig - there is a
> contingent that wants to assume things about the intent of project
> maintainers, and I think that's the wrong thing to do.

I don't - as the design authority for PyPI, we're responsible for a
core part of the user experience of newcomers to the Python ecosystem,
and we need to take that responsibility seriously.

The key implication of that responsibility is an obligation to keep a
close eye on the overall cognitive complexity of our toolchain, which
is still mindbendingly complicated, despite the significant
improvements over the past few years.

In particular, we need to pay attention to the cases where low feature
usage metrics from PyPI indicate the possible presence of legacy
features that are making the new user experience and the toolchain
maintenance process more complicated without providing a sufficient
pay-off in increased power and flexibility for software publishers to
justify those complications.

> If we want to trim the acceptable formats for distribution to make pypi et.
> al. easier to maintain, then that's fine.  Do it.  Or don't do it.  Don't
> selectively do it.

Unilaterally turning the feature off would be extraordinarily hostile
to current users - grace periods and sunset clauses are standard
features of change management processes for a reason, even when they
come at the cost of additional implementation complexity.

That said, allowing new users to opt-in to the feature (for as long as
it sticks around) would be simpler than preventing them, so it's
likely a better option from a pure maintainability perspective,
without even considering whether or not it's fair to new users to
prevent them from opting in to practices we're in the process of
removing from the toolchain.

In my view, the purpose of PyPI is to provide Pythonistas with a
straighforward mechanism to publish open source software if they
choose to do so. Fairness in an abstract sense doesn't really enter
into it for me - the question I ask myself is "Will this change have a
long 

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 19 August 2016 at 08:12, Nathaniel Smith  wrote:
> On Mon, Aug 15, 2016 at 5:02 PM, Nick Coghlan  wrote:
>> *  new project by existing maintainer: here, I'd be inclined to flag
>> maintainer accounts at the start of the deprecation period based on whether
>> or not they're currently using the legacy formats on any of their projects -
>> that is, the migration period would be applied at the *user* level, in
>> addition to the project level. If someone has never used the legacy formats,
>> they can be treated like a new user immediately, and will presumably never
>> even notice the change.
>
> Like any invisible state, there's some potential for confusion here --
> "how come I can upload these files but my co-maintainer can't?" Not
> necessarily a fatal flaw, but something to be aware of.

I wasn't suggesting the state would be invisible, more something akin
to the PEP 438 settings:

- existing projects with no legacy formats uploaded get "No legacy
formats" flagged
- existing projects with legacy formats uploaded get "Legacy formats" flagged
- existing maintainers with projects in the second category get a "Can
enable legacy formats" flag

By default, new projects would have the "No legacy formats" flag
(regardless of who created them), but existing maintainers with the
"Can enable legacy formats" marker would be able to change that. Once
a maintainer flipped the setting for a given project, then anyone
maintaining that project could upload the legacy formats, but wouldn't
automatically gain the ability to change the setting on other
projects.

This approach would mean we could be fairly aggressive in pushing
*new* users towards the newer formats (since there'd be a sharp
historical line beyond which new accounts simply can't enable the
legacy formats for their projects), *without* breaking the workflows
of existing projects and maintainers (at least, not yet).

Alternatively, we could simply not worry about a user level flag, and
just have a project level flag that's set to "No legacy formats" by
default for new projects - new users won't have any incentive to
change it, while existing users can change it (at least for the time
being) if that suits their workflow.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 19 August 2016 at 07:58, Donald Stufft  wrote:
>> On Aug 18, 2016, at 5:53 PM, Barry Warsaw  wrote:
>> I'm very sure I don't understand something, because I thought wheels were 
>> just
>> fine for the import-from-sys.path use case.  I mean, pip does this and in
>> Debian, we have a program (dirtbike) that turns installed package's file
>> system layouts back into wheels so they can be put on sys.path and imported
>> from.
>
> It “Works”, sometimes, if the thing in the wheel doesn’t do something that
> might cause that to break. Thus it will work sometimes, but often times it
> won’t. It’s not an officially supported feature, but more of a thing where
> some decisions were made not to preclude it from working entirely.
>
> As an example, your Debian dirtbike’d pip only actually fully works because
> you’ve patched requests. If you didn’t have the patches you’ve applied to
> requests, it wouldn’t work. Additionally, pip only works at all because we’ve
> been careful to *only* depends on pure Python items that generally work when
> zipped (besides the requests thing, which we end up working around in 
> get-pip.py
> and just flat out avoiding in ensurepip).
>
> Basically, we don’t go out of our way to prevent it, but if you do it and it
> breaks, you get to keep both pieces.

+1

There's actually a somewhat analogous capability in the standard
library's test suite: test.support.import_fresh_module

This is a way of doing imports that effectively bypasses the normal
sys.modules cache, which lets us test both pure Python implementations
and C accelerator modules in the same run of the test suite.

It's also a marvelous foot gun that can mess up your Python process
state in a spectacularly hard to debug way if you don't know *exactly*
what you're doing.

However, it's just Python code, so anyone that wants to can go look at
how we do it, and do the same thing themselves, they're just on their
own when it comes to figuring out the precise boundaries of what does
and doesn't work, and which modules are written in such a way that
they will tolerate that kind of handling :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nathaniel Smith]

On Mon, Aug 15, 2016 at 5:02 PM, Nick Coghlan  wrote:
> *  new project by existing maintainer: here, I'd be inclined to flag
> maintainer accounts at the start of the deprecation period based on whether
> or not they're currently using the legacy formats on any of their projects -
> that is, the migration period would be applied at the *user* level, in
> addition to the project level. If someone has never used the legacy formats,
> they can be treated like a new user immediately, and will presumably never
> even notice the change.

Like any invisible state, there's some potential for confusion here --
"how come I can upload these files but my co-maintainer can't?" Not
necessarily a fatal flaw, but something to be aware of.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nathaniel Smith]

On Tue, Aug 16, 2016 at 10:21 PM, Nick Coghlan  wrote:
> On 17 August 2016 at 01:15, Leonardo Rochael Almeida
>  wrote:
>> PS: In the buildout community, we never really understood the impetus for
>> replacing egg as a format, which is really not all that complex and not all
>> the different from wheel as it stands, instead of just formalizing it in a
>> PEP. We got the feeling that, in the movement of getting rid of setuptools
>> as an "installation" dependency, we ended up throwing out the baby with the
>> bathwater...
>
> The majority of the problems with eggs are due to the fact that they
> were designed to operate *by default* as the plugin management system
> for Chandler. buildout uses them in a similar way, so they're a good
> fit there.
>
> However, they're *not* a good fit for other cases, hence
> "--single-version-externally-managed" and all of the other ways in
> which pip switches the defaults to be more developer centric, and less
> deployment focused. Most of the problems people ran into with the
> defaults were actually related to the way easy_install and
> pkg_resources worked rather than due to the egg format itself, but the
> lack of clear documentation and specifications meant that the broader
> adoption process for programmatic dependency management in general
> needed to be based on a more modular reimplementation of similar ideas
> with more explicit interoperability interfaces, rather than adopting
> the setuptools/easy_install/pkg_resources implementation of those
> ideas wholesale.
>
> This means that the situation we've managed to get to now is that we
> have wheels as a satisfactory replacement for the "binary
> distribution" use case, but we haven't even *tried* to replace eggs
> for the "standalone path entry" use case.
>
> That gives us a few options for possible ways forward:
>
> - define a new egg-inspired "*.pypath" format for directories and zip
> files that are designed to be placed directly on sys.path, rather than
> installed
> - just bless eggs as the recommended standalone importable format, and
> offer a wheel2egg conversion utility and perhaps an "--as-egg" install
> option in pip
> - as with the previous, but bake the wheel2egg conversion into
> setuptools/easy_install rather than into pip

I'm not at all familiar with buildout's internals, but from the
description upthread it sounds like it already has to have fairly
intimate knowledge of and control over the layout of the directory
it's managing (so that it can do things like generate sys.path
manipulation code inside console scripts, etc).

Would it make sense to declare that buildout has its own directory
layout and conventions that it uses (which happen to be suspiciously
similar to the legacy .egg format), and that buildout is responsible
for figuring out how to consume standardized formats like .whl and put
them into the layout it wants? I don't think there's much appetite for
trying to standardize .egg, and the advantage is that it moves control
of the format to the people who actually use it -- distutils-sig is
not really the best place to be trying to do anything with eggs at
this point AFAICT. It might even make sense to split out some bits of
setuptools into buildout.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Barry Warsaw]

On Aug 17, 2016, at 03:21 PM, Nick Coghlan wrote:

>This means that the situation we've managed to get to now is that we
>have wheels as a satisfactory replacement for the "binary
>distribution" use case, but we haven't even *tried* to replace eggs
>for the "standalone path entry" use case.

I'm very sure I don't understand something, because I thought wheels were just
fine for the import-from-sys.path use case.  I mean, pip does this and in
Debian, we have a program (dirtbike) that turns installed package's file
system layouts back into wheels so they can be put on sys.path and imported
from.

Cheers,
-Barry


pgpYxVUnFBslR.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 18, 2016, at 5:53 PM, Barry Warsaw  wrote:
> 
> On Aug 17, 2016, at 03:21 PM, Nick Coghlan wrote:
> 
>> This means that the situation we've managed to get to now is that we
>> have wheels as a satisfactory replacement for the "binary
>> distribution" use case, but we haven't even *tried* to replace eggs
>> for the "standalone path entry" use case.
> 
> I'm very sure I don't understand something, because I thought wheels were just
> fine for the import-from-sys.path use case.  I mean, pip does this and in
> Debian, we have a program (dirtbike) that turns installed package's file
> system layouts back into wheels so they can be put on sys.path and imported
> from.


It “Works”, sometimes, if the thing in the wheel doesn’t do something that
might cause that to break. Thus it will work sometimes, but often times it
won’t. It’s not an officially supported feature, but more of a thing where
some decisions were made not to preclude it from working entirely.

As an example, your Debian dirtbike’d pip only actually fully works because
you’ve patched requests. If you didn’t have the patches you’ve applied to
requests, it wouldn’t work. Additionally, pip only works at all because we’ve
been careful to *only* depends on pure Python items that generally work when
zipped (besides the requests thing, which we end up working around in get-pip.py
and just flat out avoiding in ensurepip).

Basically, we don’t go out of our way to prevent it, but if you do it and it
breaks, you get to keep both pieces.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

Now I remember that just the other week I had to switch a project to zip
sdists, because I was having trouble doing .tar.gz on Windows. Perhaps the
greater availability of tools to deal with zip is the reason why 10% of
sdists choose that format.

On Wed, Aug 17, 2016 at 6:08 PM Donald Stufft  wrote:

>
> On Aug 17, 2016, at 6:05 PM, Chris Barker  wrote:
>
> I do -- what advantage does tar.gz have over zip?
>
>
>
> The two formats are roughly isomorphic in terms of technical reasons to
> pick one over the other, but distutils defaults to .tar.gz and most uploads
> to PyPI are .tar.gz.
>
> —
>
> Donald Stufft
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 17, 2016, at 6:05 PM, Chris Barker  wrote:
> 
> I do -- what advantage does tar.gz have over zip?



The two formats are roughly isomorphic in terms of technical reasons to pick 
one over the other, but distutils defaults to .tar.gz and most uploads to PyPI 
are .tar.gz.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Chris Barker]

On Tue, Aug 16, 2016 at 9:10 AM, Donald Stufft  wrote:
>
> Ah, I knew I was forgetting something. I think we should hold off on
> preventing egg uploads until setuptools can download and install wheels.
>

Aren't we trying to get to the point where setuptools doesn't download and
install anything ?

Sigh.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Chris Barker]

On Tue, Aug 16, 2016 at 8:51 AM, Brett Cannon  wrote:

> One thing to remember is that Windows can't read tar files natively while
> it can for zip files. Now you can easily download tools on Windows to read
> tar files and thanks to Bash on Windows you even have it included once you
> turn that feature on.
>
> The other point is we have a zip importer in Python but not a .tar.gz one. I
> don't know how often anyone actually downloads a zip file directly from
> PyPI and then tack it on to their sys.path for importing, but that is
> currently possible.
>
> I doubt either of these points are important enough to continue to support
> zip files for sdists,
>

I do -- what advantage does tar.gz have over zip? Zip's always been
supported on any platform I've used...

But both are in python's stdlib, so I suppose everyone has it that has
Python.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Paul Moore]

On 17 August 2016 at 18:49, Wes Turner  wrote:
> PEX - Python Executable
>
> - It's a ZIP with an executable header and entry points.
>
> - PEX probably solves for the AWS Lambda case
> - PEX is unsupported by PyPI (e.g. sadly doesnt work with DevPi
> application-level package repository caching)

I'm not quite sure what point you're making here, but pex doesn't
support Windows, which is a major downside for anything intended as a
standard solution for Python (it is of course fine for the pex project
to not support Windows, but that's a different matter).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Thomas Kluyver]

On Mon, Aug 15, 2016, at 08:40 PM, Donald Stufft wrote:
> Or were you wondering for the last 30 days like I did for downloads? If
> so then:
> 
> * sdist: 15601 (66%)
> * bdist_wheel: 6398 (27%)
> * bdist_egg: 1076 (4.5%)
> * bdist_dumb: 195 (0.8%)
> * bdist_wininst: 167 (0.7%)
> * bdist_rpm: 38 (0.1%)
> * bdist_msi: 9 (0.03%)
> * Everything Else: 0 (0%)

Sorry for the lack of clarity, I was indeed looking for these numbers
for recent uploads.

This suggests there is a small but non-negligible group still uploading
eggs. Is it worth looking at what projects are doing this, and
potentially contacting some authors to understand why?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 17 August 2016 at 15:21, Nick Coghlan  wrote:
> While rebranding eggs (or a close derivative) as "pypath entries"
> could help make them more self-explanatory (especially when it comes
> to explaining how the zipped form differs from wheel files), doing so
> wouldn't actually give us any more architectural flexibility in
> practice than the last option.

That said, if folks *did* want to go down this path, then an important
use case to consider is "single archive applications" suitable for use
in services like AWS Lambda and Azure Cloud Functions, where you get a
language runtime to play with, but need to provide everything you want
to run in that environment as a single pre-bundled archive. Similar
problems also show up when building rich client applications for
Windows, Mac OS X, Android and iOS.

This means you want tooling that targets the direct dependency
bundling model, rather than the runtime environment configuration
model favoured by pip and virtualenv.

So this particular challenge *shouldn't* be framed as just a backwards
compatibility and migration concern for buildout et al - the "all
inclusive application bundle" approach is still an important use case,
even though it isn't the lowest common denominator cross-platform
dependency specification, retrieval and installation problem handled
by the default toolchain.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 17 August 2016 at 01:15, Leonardo Rochael Almeida
 wrote:
> PS: In the buildout community, we never really understood the impetus for
> replacing egg as a format, which is really not all that complex and not all
> the different from wheel as it stands, instead of just formalizing it in a
> PEP. We got the feeling that, in the movement of getting rid of setuptools
> as an "installation" dependency, we ended up throwing out the baby with the
> bathwater...

The majority of the problems with eggs are due to the fact that they
were designed to operate *by default* as the plugin management system
for Chandler. buildout uses them in a similar way, so they're a good
fit there.

However, they're *not* a good fit for other cases, hence
"--single-version-externally-managed" and all of the other ways in
which pip switches the defaults to be more developer centric, and less
deployment focused. Most of the problems people ran into with the
defaults were actually related to the way easy_install and
pkg_resources worked rather than due to the egg format itself, but the
lack of clear documentation and specifications meant that the broader
adoption process for programmatic dependency management in general
needed to be based on a more modular reimplementation of similar ideas
with more explicit interoperability interfaces, rather than adopting
the setuptools/easy_install/pkg_resources implementation of those
ideas wholesale.

This means that the situation we've managed to get to now is that we
have wheels as a satisfactory replacement for the "binary
distribution" use case, but we haven't even *tried* to replace eggs
for the "standalone path entry" use case.

That gives us a few options for possible ways forward:

- define a new egg-inspired "*.pypath" format for directories and zip
files that are designed to be placed directly on sys.path, rather than
installed
- just bless eggs as the recommended standalone importable format, and
offer a wheel2egg conversion utility and perhaps an "--as-egg" install
option in pip
- as with the previous, but bake the wheel2egg conversion into
setuptools/easy_install rather than into pip

At least for now, I think that last option probably makes the most
sense, as it moves easy_install, buildout, and other egg based tools
into a similar position to conda, PyPM and the Linux distros: a
downstream community with its own binary distribution format, but
associated tooling to also make it easy for participants in that
ecosystem to consume pre-built wheel files if they want to.

While rebranding eggs (or a close derivative) as "pypath entries"
could help make them more self-explanatory (especially when it comes
to explaining how the zipped form differs from wheel files), doing so
wouldn't actually give us any more architectural flexibility in
practice than the last option.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 16, 2016, at 12:58 PM, Daniel Holth  wrote:
> 
> On Tue, Aug 16, 2016 at 12:50 PM Ian Cordasco  > wrote:
> So perhaps I'm missing something, but why are we talking about trying
> to shoehorn a legacy design into Wheel? Why wouldn't we leave
> bdist_egg alone and start trying to find a better way to replace it?
> We could avoid over-complicating Wheel and we could spend time
> polishing whatever we come up with.
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org 
> 
> https://mail.python.org/mailman/listinfo/distutils-sig 
> 
> 
> It is because wheel is almost identical to egg, but the name 'newegg' was 
> already in use by a major online retailer. Possibly adding the one missing 
> egg feature to wheel would save someone some time. Maybe you would want to 
> give 'extracted wheel in its own directory' a different name, or no name, to 
> avoid confusion.

Installing a wheel to a single directory is fine. It’s no longer a wheel at 
that point, it is an installed distribution.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 16, 2016, at 12:46 PM, Daniel Holth  wrote:
> 
> On Tue, Aug 16, 2016 at 12:15 PM Donald Stufft  > wrote:
>> On Aug 16, 2016, at 11:51 AM, Brett Cannon > > wrote:
>> 
>> One thing to remember is that Windows can't read tar files natively while it 
>> can for zip files. Now you can easily download tools on Windows to read tar 
>> files and thanks to Bash on Windows you even have it included once you turn 
>> that feature on.
> 
> This is true, but I think that using .tar.gz by default still makes sense 
> because it’s still the vast bulk of what people actually release to PyPI. So 
> it represents the status quo and switching to zip as the default would break 
> a lot of things.
> 
>> 
>> The other point is we have a zip importer in Python but not a .tar.gz one. I 
>> don't know how often anyone actually downloads a zip file directly from PyPI 
>> and then tack it on to their sys.path for importing, but that is currently 
>> possible.
> 
> A sdist is not an acceptable format for adding to sys.path. While in many, 
> simple cases, it will “just work”, that’s more of an implementation detail 
> than anything else. There are many projects which simply do not run or error 
> out if you do this. I don’t think worrying about something that sort of 
> works, sometimes, is a big deal.
> 
>> 
>> I doubt either of these points are important enough to continue to support 
>> zip files for sdists, but I just wanted to point it out. At worst this is 
>> something to think about if we ever do formalize the sdist format and come 
>> up with a custom file extension.
> 
> If we make a sdist 2.0 with a new format, I do think it makes sense to make 
> it a zipfile like wheel already is (which reduces the internal formats down 
> from 2 to 1), not for the reasons above though, just for consistency with 
> wheel.
> 
> ZIP is a fantastically designed file format. JAR, IPA (iPhone), all are ZIP 
> files. The only thing I sometimes wonder about for wheel is whether it would 
> be worth the trouble to support greater compression with something like 
> .zip.xz (an uncompressed ZIP inside a wrapper, possibly another ZIP) since 
> ZIP compresses each file individually and does not compress its metadata. But 
> most packages are quite small.

I don’t think it would be worth it. Right now there’s only 1 optional c library 
required (zlib) to install from wheels. If we deprecate all of the other things 
then there’s only 1 optional c library required to install from sdists too 
(zlib). At best it would make sense to use gzip, but since we’re already use 
ZIP_DEFLATE I’m not sure the extra complexity is worth it.

> 
> ZIP's greatest weakness is also its greatest strength, as ZIP allows random 
> access to its members and very fast access to its metadata. This is why it 
> makes sense to have a ZIP importer but not a .tar.gz importer.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Paul Moore]

On 16 August 2016 at 17:49, Ian Cordasco  wrote:
> So perhaps I'm missing something, but why are we talking about trying
> to shoehorn a legacy design into Wheel? Why wouldn't we leave
> bdist_egg alone and start trying to find a better way to replace it?
> We could avoid over-complicating Wheel and we could spend time
> polishing whatever we come up with.

+1

Wheel should remain a distribution format (with running from wheel,
zipped or unzipped, an intentional but unsupported benefit - get-pip
and virtualenv use the feature, but it should be treated as "use at
your own risk").

Egg as distribution format is deprecated and we shouldn't look to
encourage it, but egg as runtime format is fine if people want/need it
(I don't see the value myself, but the work needed to modify something
like buildout to move away from it probably isn't worth it).
Downloading a wheel and converting it to a runtime egg format is
perfectly possible. Whether buildout's preference for not adding
dependencies is sufficient to push setuptools to add support for that
route, I can't say. But I'm perfectly OK with saying that in due
course, we'll stop supporting eggs for distribution (e.g., in terms of
hosting on PyPI) and then provide a period for the likes of
buildout/setuptools to decide how to migrate. If they choose not to,
then fine - we have to decide whether buildout is important enough to
unilaterally block the change. But I don't think we should rush on
this - let's just publish the statement of intent, and then give
projects like buildout the time they need to react.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

On Tue, Aug 16, 2016 at 12:50 PM Ian Cordasco 
wrote:

> So perhaps I'm missing something, but why are we talking about trying
> to shoehorn a legacy design into Wheel? Why wouldn't we leave
> bdist_egg alone and start trying to find a better way to replace it?
> We could avoid over-complicating Wheel and we could spend time
> polishing whatever we come up with.
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


It is because wheel is almost identical to egg, but the name 'newegg' was
already in use by a major online retailer. Possibly adding the one missing
egg feature to wheel would save someone some time. Maybe you would want to
give 'extracted wheel in its own directory' a different name, or no name,
to avoid confusion.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

On Tue, Aug 16, 2016 at 12:43 PM Alex Grönholm 
wrote:

> 16.08.2016, 19:37, Daniel Holth kirjoitti:
>
> On Tue, Aug 16, 2016 at 12:06 PM Donald Stufft  wrote:
>
>>
>> On Aug 16, 2016, at 8:50 AM, Daniel Holth  wrote:
>>
>> Wheel should be updated to support the egg use case before egg is
>> removed. IIUC this would mostly mean officially supporting 'unzipped wheel'
>> as a thing you can add to PYTHONPATH, possibly with some additional
>> restrictions for the specific wheel. We could go a little further and
>> officially support zipped wheels "if zip safe". We could implement
>> wheel2egg to complement egg2wheel?
>>
>>
>>
>> I don’t think Wheel should officially supported unzip wheels as a thing
>> you can add to PYTHONPATH nor do I think we should officially support
>> zipped wheels being added to PYTHONPATH. Neither of those things are going
>> to work universally and setuptools has gross heuristics to try and figure
>> out when they will and won’t work (which regularly break or report
>> inaccurately). Wheel is improved by remaining focused on being a format for
>> distributing and installed via an installer, not one that tries to do all
>> of the things like Egg did.
>>
>
> So this is how I envision "installing a wheel to its own directory".
>
> 1. Unzip the wheel into its own directory.
> 2. If there are any nested *.data/purelib, platlib, make sure they are
> also unzipped into the root of the unzipped wheel instead of their archive
> paths.
> 3. If there are scripts and you want them, install those too.
>
> If 2 and 3 don't apply to you, you are done almost before you've started.
> What's missing?
>
> How does one go about installing console_scripts this way?
>

No difference. Read entry_points.txt and generate script wrappers for all
of the listed console_scripts. In buildout's case it adds all of the
dependencies to sys.path in the generated console_script wrapper.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

On Tue, Aug 16, 2016 at 12:15 PM Donald Stufft  wrote:

> On Aug 16, 2016, at 11:51 AM, Brett Cannon  wrote:
>
> One thing to remember is that Windows can't read tar files natively while
> it can for zip files. Now you can easily download tools on Windows to read
> tar files and thanks to Bash on Windows you even have it included once you
> turn that feature on.
>
>
> This is true, but I think that using .tar.gz by default still makes sense
> because it’s still the vast bulk of what people actually release to PyPI.
> So it represents the status quo and switching to zip as the default would
> break a lot of things.
>
>
> The other point is we have a zip importer in Python but not a .tar.gz one. I
> don't know how often anyone actually downloads a zip file directly from
> PyPI and then tack it on to their sys.path for importing, but that is
> currently possible.
>
>
> A sdist is not an acceptable format for adding to sys.path. While in many,
> simple cases, it will “just work”, that’s more of an implementation detail
> than anything else. There are many projects which simply do not run or
> error out if you do this. I don’t think worrying about something that sort
> of works, sometimes, is a big deal.
>
>
> I doubt either of these points are important enough to continue to support
> zip files for sdists, but I just wanted to point it out. At worst this is
> something to think about if we ever do formalize the sdist format and come
> up with a custom file extension.
>
>
> If we make a sdist 2.0 with a new format, I do think it makes sense to
> make it a zipfile like wheel already is (which reduces the internal formats
> down from 2 to 1), not for the reasons above though, just for consistency
> with wheel.
>

ZIP is a fantastically designed file format. JAR, IPA (iPhone), all are ZIP
files. The only thing I sometimes wonder about for wheel is whether it
would be worth the trouble to support greater compression with something
like .zip.xz (an uncompressed ZIP inside a wrapper, possibly another ZIP)
since ZIP compresses each file individually and does not compress its
metadata. But most packages are quite small.

ZIP's greatest weakness is also its greatest strength, as ZIP allows random
access to its members and very fast access to its metadata. This is why it
makes sense to have a ZIP importer but not a .tar.gz importer.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

On Tue, Aug 16, 2016 at 12:06 PM Donald Stufft  wrote:

>
> On Aug 16, 2016, at 8:50 AM, Daniel Holth  wrote:
>
> Wheel should be updated to support the egg use case before egg is removed.
> IIUC this would mostly mean officially supporting 'unzipped wheel' as a
> thing you can add to PYTHONPATH, possibly with some additional restrictions
> for the specific wheel. We could go a little further and officially support
> zipped wheels "if zip safe". We could implement wheel2egg to complement
> egg2wheel?
>
>
>
> I don’t think Wheel should officially supported unzip wheels as a thing
> you can add to PYTHONPATH nor do I think we should officially support
> zipped wheels being added to PYTHONPATH. Neither of those things are going
> to work universally and setuptools has gross heuristics to try and figure
> out when they will and won’t work (which regularly break or report
> inaccurately). Wheel is improved by remaining focused on being a format for
> distributing and installed via an installer, not one that tries to do all
> of the things like Egg did.
>

So this is how I envision "installing a wheel to its own directory".

1. Unzip the wheel into its own directory.
2. If there are any nested *.data/purelib, platlib, make sure they are also
unzipped into the root of the unzipped wheel instead of their archive paths.
3. If there are scripts and you want them, install those too.

If 2 and 3 don't apply to you, you are done almost before you've started.
What's missing?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 16, 2016, at 11:51 AM, Brett Cannon  wrote:
> 
> One thing to remember is that Windows can't read tar files natively while it 
> can for zip files. Now you can easily download tools on Windows to read tar 
> files and thanks to Bash on Windows you even have it included once you turn 
> that feature on.

This is true, but I think that using .tar.gz by default still makes sense 
because it’s still the vast bulk of what people actually release to PyPI. So it 
represents the status quo and switching to zip as the default would break a lot 
of things.

> 
> The other point is we have a zip importer in Python but not a .tar.gz one. I 
> don't know how often anyone actually downloads a zip file directly from PyPI 
> and then tack it on to their sys.path for importing, but that is currently 
> possible.

A sdist is not an acceptable format for adding to sys.path. While in many, 
simple cases, it will “just work”, that’s more of an implementation detail than 
anything else. There are many projects which simply do not run or error out if 
you do this. I don’t think worrying about something that sort of works, 
sometimes, is a big deal.

> 
> I doubt either of these points are important enough to continue to support 
> zip files for sdists, but I just wanted to point it out. At worst this is 
> something to think about if we ever do formalize the sdist format and come up 
> with a custom file extension.

If we make a sdist 2.0 with a new format, I do think it makes sense to make it 
a zipfile like wheel already is (which reduces the internal formats down from 2 
to 1), not for the reasons above though, just for consistency with wheel.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 16, 2016, at 8:50 AM, Daniel Holth  wrote:
> 
> Wheel should be updated to support the egg use case before egg is removed. 
> IIUC this would mostly mean officially supporting 'unzipped wheel' as a thing 
> you can add to PYTHONPATH, possibly with some additional restrictions for the 
> specific wheel. We could go a little further and officially support zipped 
> wheels "if zip safe". We could implement wheel2egg to complement egg2wheel?


I don’t think Wheel should officially supported unzip wheels as a thing you can 
add to PYTHONPATH nor do I think we should officially support zipped wheels 
being added to PYTHONPATH. Neither of those things are going to work 
universally and setuptools has gross heuristics to try and figure out when they 
will and won’t work (which regularly break or report inaccurately). Wheel is 
improved by remaining focused on being a format for distributing and installed 
via an installer, not one that tries to do all of the things like Egg did.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Brett Cannon]

+1 on the whole idea. Trying to continue to nudge the community towards
more standardized approaches in packaging is always a good thing. :) I only
have one data point in relation to sdists file formats.

On Mon, 15 Aug 2016 at 12:09 Donald Stufft  wrote:

> [SNIP]
>
> Looking at numbers again, the current number of projects for each file ext
> are:
>
> * .tar.gz: 444,338
> * .zip: 58,774
> * .tar.bz2: 3,265
> * .tgz: 217
> * Everything Else: 0
>

One thing to remember is that Windows can't read tar files natively while
it can for zip files. Now you can easily download tools on Windows to read
tar files and thanks to Bash on Windows you even have it included once you
turn that feature on.

The other point is we have a zip importer in Python but not a .tar.gz one. I
don't know how often anyone actually downloads a zip file directly from
PyPI and then tack it on to their sys.path for importing, but that is
currently possible.

I doubt either of these points are important enough to continue to support
zip files for sdists, but I just wanted to point it out. At worst this is
something to think about if we ever do formalize the sdist format and come
up with a custom file extension.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Leonardo Rochael Almeida]

Thanks Daniel,

A few corrections and considerations below:

On 16 August 2016 at 09:50, Daniel Holth  wrote:

> Wheel should be updated to support the egg use case before egg is removed.
> IIUC this would mostly mean officially supporting 'unzipped wheel' as a
> thing you can add to PYTHONPATH, possibly with some additional restrictions
> for the specific wheel. We could go a little further and officially support
> zipped wheels "if zip safe". We could implement wheel2egg to complement
> egg2wheel?
>

Specifically, buildout right now has setuptools as its only dependence, and
buildout uses it to locate, download and build sdist dependencies, or
directly download and use .egg dependencies, which are important for
Windows platforms in the communities that use buildout and depend on
compiled extensions.

It also uses setuptools for reading entry points of packages for internal
use (buildout recipes and extensions), and for figuring out console scripts
to install in "bin"

(this last part is actually the job of the `zc.recipe.egg` recipe, which is
buildout's pip replacement that predates pip by a few years, but it's the
only recipe that is developed in the same repo as buildout itself, so we'll
treat them as one for this discussion).

This also means that so far buildout has been missing out on wheels. It
hasn't been much of a problem since almost all packages also have an sdist
that can be turned into an egg by setuptools, but if the trend of
sdist-less wheels increases, this could quickly become a problem.

It also means that, buildout is missing out on manylinux...

A wheel2egg might be nice, but unless it's integrated into setuptools
proper, it would mean adding another dependency to buildout and the
developers would rather depend on a single library for talking to PyPI and
making packages appear as "directories addable to `sys.path`.

The main reason wheel did not support these use cases from the beginning is
> that it was quicker not to. Also, it is a bit confusing for 'unzipped
> directory in the (egg/wheel) format', 'archive', and 'plugin you add to
> PYTHONPATH' to all have the same name 'eggs'.
>
> We would need to work with the buildout developers on this. Buildout uses
> a flat directory full of unzipped eggs
>

Not necessarily unzipped. Although buildout unzips all sdists and eggs it
installs, if it's in the "eggs" directory, has the right name (e.g.
suds_jurko-0.6-py2.7.egg), and can be added to sys.path, that's all
buildout cares about, but if it's not yet there, buildout will use
setuptools to install it as an "unzipped egg" (there used to be a switch to
install eggs as zipped, or rather, as unzipped, zipped being the default,
but nobody liked the eggs zipped so they threw that foot gun away).


> , not exactly the uploading to pypi case, and when installing
> console_scripts, makes sure the necessary eggs are added to sys.path in the
> script itself. The exact mechanism varies depending on the version of
> buildout used.
>

This hasn't changed for a while. The mechanism is, basically, to create a
script in "bin/" that appends all (recursive) dependencies of the script
from the "eggs" directory into `sys.path` then call the entry point
declared in the `console_scripts`.

The buildout configuration can specify both the name that the console
scripts will be installed, and also specify some code to be run immediately
before the entry point,  which is useful for pre-setting command line
arguments, for example. All of this is done using setuptools APIs.

This and the fact that buildout has a plethora of extensions and powerful
declarative configuration language that allows for dependency management
between different "parts" of the configuration (e.g. coordinating ports and
addresses between services like Varnish and Plone, or adding the paths of
the script and configuation file from Plone into a supervisord
configuration) means that, like many things coming out of the brilliant
mind of Jim Fulton, it's considered overly complex by many, but those who
use it would find it rather difficult to replace if it's no longer
functional.

zope.interface is an important package with up-to-date eggs.
> https://pypi.python.org/pypi/zope.interface
>

And it's used by Plone and Pyramid, to mention two mainstream communities
that would be affected by a lack of binary eggs on Windows, for example.

To sumarize, if we could convince setuptools to treat wheels on PyPI for
download and installation the same way it treats eggs, or if we could
replace setuptools with something that did, then buildout would be onboard
the deprecation of eggs.

Cheers,

PS: In the buildout community, we never really understood the impetus for
replacing egg as a format, which is really not all that complex and not all
the different from wheel as it stands, instead of just formalizing it in a
PEP. We got the feeling that, in the movement of getting rid of setuptools
as an "installation" dependency, we ended up throwing 

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

Wheel should be updated to support the egg use case before egg is removed.
IIUC this would mostly mean officially supporting 'unzipped wheel' as a
thing you can add to PYTHONPATH, possibly with some additional restrictions
for the specific wheel. We could go a little further and officially support
zipped wheels "if zip safe". We could implement wheel2egg to complement
egg2wheel?

The main reason wheel did not support these use cases from the beginning is
that it was quicker not to. Also, it is a bit confusing for 'unzipped
directory in the (egg/wheel) format', 'archive', and 'plugin you add to
PYTHONPATH' to all have the same name 'eggs'.

We would need to work with the buildout developers on this. Buildout uses a
flat directory full of unzipped eggs, not exactly the uploading to pypi
case, and when installing console_scripts, makes sure the necessary eggs
are added to sys.path in the script itself. The exact mechanism varies
depending on the version of buildout used.

zope.interface is an important package with up-to-date eggs.
https://pypi.python.org/pypi/zope.interface

On Tue, Aug 16, 2016 at 5:41 AM Paul Moore  wrote:

> On 15 August 2016 at 20:09, Donald Stufft  wrote:
> >
> > First off, we currently allow people to upload sdist, bdist_wheel,
> bdist_egg,
> > bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However
> I think
> > that we should try to get rid of support for most of these. Just for
> reference
> > currently the number of files uploaded for each type of file looks like:
> >
> > * sdist: 506,585
> > * bdist_wheel: 81,207
> > * bdist_egg: 48,282
> > * bdist_wininst: 14,002
> > * bdist_dumb: 5,502
> > * bdist_msi: 497
> > * bdist_rpm: 464
> > * bdist_dmg: 45
> >
> > Out of all of these, I think that we can easily remove bdist_dmg,
> bdist_rpm,
> > and bdist_dumb. I also believe that there is a strong case for removing
> > bdist_msi and bdist_wininst. I also think we should consider removing
> > bdist_egg.
>
> Overall +1 from me for this.
>
> Some thoughts:
>
> 1. It would be nice to provide some transition process for the
> bdist_wininst case, as these are automatically convertible (with some
> caveats) to wheel using "wheel convert". I'm not suggesting that we do
> an auto-convert, but a way of getting the message out to projects that
> use wininst that there is a migration path might be worth it (although
> it's equally possible that the projects in question are all
> unmaintained, in which case there's not much point in bothering).
> 2. Do we understand the remaining use cases for eggs? AIUI, buildout
> uses it, and I get the impression that there are other specialist
> groups that still use the egg format (the plone community?). While I
> don't think having two binary distribution formats is good for the
> ecosystem (it's confusing for users, and splits effort), I think we
> should be make sure we have those use cases covered (or at least have
> a plan on how we handle the situation) before deprecating the egg
> format.
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Paul Moore]

On 15 August 2016 at 20:09, Donald Stufft  wrote:
>
> First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I 
> think
> that we should try to get rid of support for most of these. Just for reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.

Overall +1 from me for this.

Some thoughts:

1. It would be nice to provide some transition process for the
bdist_wininst case, as these are automatically convertible (with some
caveats) to wheel using "wheel convert". I'm not suggesting that we do
an auto-convert, but a way of getting the message out to projects that
use wininst that there is a migration path might be worth it (although
it's equally possible that the projects in question are all
unmaintained, in which case there's not much point in bothering).
2. Do we understand the remaining use cases for eggs? AIUI, buildout
uses it, and I get the impression that there are other specialist
groups that still use the egg format (the plone community?). While I
don't think having two binary distribution formats is good for the
ecosystem (it's confusing for users, and splits effort), I think we
should be make sure we have those use cases covered (or at least have
a plan on how we handle the situation) before deprecating the egg
format.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 16 Aug 2016 05:09, "Donald Stufft"  wrote:
>
> Hello!
>
> I'd like to restrict what folks can upload to PyPI in an effort to help
narrow
> the scope down and to enable more a more consistent experience for
everyone.
>
> First off, we currently allow people to upload sdist, bdist_wheel,
bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I
think
> that we should try to get rid of support for most of these. Just for
reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg,
bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.
>
> First of all, when I say "remove", I mean disallow new uploads, but do not
> delete the existing files.

General +1 from me for data driven design simplification.

As far as ideas for *how* to go about it go, I think there a few user
categories to consider:

* new user making their first project: we can deprecate legacy formats
aggressively here (since there shouldn't be many, if any, backwards
compatibility considerations for either automated workflows or people's
habits)

* new release of existing project: here is where we want to actively warn
project maintainers using the old formats that PyPI's behaviour will be
changing *before* we force them to change their workflows and habits

*  new project by existing maintainer: here, I'd be inclined to flag
maintainer accounts at the start of the deprecation period based on whether
or not they're currently using the legacy formats on any of their projects
- that is, the migration period would be applied at the *user* level, in
addition to the project level. If someone has never used the legacy
formats, they can be treated like a new user immediately, and will
presumably never even notice the change.

Cheers,
Nick.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 15, 2016, at 7:12 PM, Glyph Lefkowitz  wrote:
> 
> 
>> On Aug 15, 2016, at 1:56 PM, Donald Stufft  wrote:
>> 
>> My main thought regarding this is that bdist_dmg != all dmg files (similarly 
>> for msi and wininst). These are specific files created by distutils without 
>> a standard or without the needed work to make them truly what users should 
>> be using. I also think they are a different class of upload, the general use 
>> case for PyPI's current file uploads are for automated installs (as 
>> evidenced by the simple API and mirroring).
> 
> I guess I'm just a little confused - are we talking about just hiding them 
> from some parts of the API or disallowing their upload entirely?
> 
> If we're talking about the literal output of bdist_dmg and bdist_rpm I 
> probably agree that they're almost useless.

Right now we’re basically talking about the literal output of bdist_dmg and 
bdist_rpm, because their outputs are the only thing we actually support 
uploading. Our current checks aren’t very stringent (or in some cases, exist at 
all) so it’s *possible* someone is constructing an actual user friendly .dmg 
and uploading it pretending to be a “bdist_dmg”, that’s not really a supported 
operation.

Ideally I’d like to disallow their upload completely and hide them from the 
API, but if the current use case of bdist_dmg, bdist_rpm, bdist_wininst, etc is 
useful enough to keep around, then I’d like to just hide them from the API.

> 
>> If we want to enable dmg, msi, etc uploads that are not the bdist_* variety 
>> for automated tooling, then we could do something like "related files" 
>> people can upload which don't get mirrored for pip and which don't show up 
>> in the repo API. Since they will be classified differently we could also do 
>> better work around the ux of discovering them and separate them from the 50 
>> wheels that some projects end up uploading and make them more obviously 
>> visible. I don't know if pypi as a distribution for _end user_ (vs 
>> developer/power user) software makes sense or not, but if it does we should 
>> support it better than accidentally via distutils. 
> 
> My concern here is that if someone has a hacky workaround working with the 
> current system, it might be better to add support for the new thing ("related 
> files") before killing the old thing.  If the plan is to do them both anyway, 
> wouldn't it be better to do it in that order?  As a community (and I mean the 
> broader open source community here, not distutils-sig; if anything distutils 
> is way better about this) we have an unfortunate habit of killing 
> potentially-useful-but-sub-optimal stuff, wandering off for half a decade, 
> and then only adding the better thing after the fact.

I don’t know if it makes sense to add the hypothetical new thing. Is PyPI the 
right place to distribute a .dmg that say, ships Python itself, some C 
libraries, some Python libraries, some bash scripts, and some static files? I 
don’t know. Currently I think the answer is “no”, PyPI is a repository of 
software _for_ Python, not a repository of software that just happens to be 
written in Python. If the ultimate end goal of a .dmg is that people no longer 
need to worry about what language the thing they are installing is written in, 
it seems weird for them to go to PyPI for a .dmg for one app, npmjs.org for a 
.dmg for another app, and so on.

Now, if it *does* make sense to support generalized uploads for applications 
written in Python with some sort of system level packaging (dmg, deb, conda, 
whatever) then we should figure out a way to actually support that, and support 
it well. As it is, I don’t have any evidence that the files that are currently 
being uploaded to PyPI are anything but “wheels, but in dmg format” (e.g. 
binary packages containing a single library). I don’t want to put a bunch of 
effort and work into making this well supported if there isn’t some evidence 
that folks will actually use it (I suspect a minimum we’d need some buy in from 
the authors of tooling to make these self contained packages).


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Glyph Lefkowitz]

> On Aug 15, 2016, at 1:56 PM, Donald Stufft  wrote:
> 
> My main thought regarding this is that bdist_dmg != all dmg files (similarly 
> for msi and wininst). These are specific files created by distutils without a 
> standard or without the needed work to make them truly what users should be 
> using. I also think they are a different class of upload, the general use 
> case for PyPI's current file uploads are for automated installs (as evidenced 
> by the simple API and mirroring).

I guess I'm just a little confused - are we talking about just hiding them from 
some parts of the API or disallowing their upload entirely?

If we're talking about the literal output of bdist_dmg and bdist_rpm I probably 
agree that they're almost useless.

> If we want to enable dmg, msi, etc uploads that are not the bdist_* variety 
> for automated tooling, then we could do something like "related files" people 
> can upload which don't get mirrored for pip and which don't show up in the 
> repo API. Since they will be classified differently we could also do better 
> work around the ux of discovering them and separate them from the 50 wheels 
> that some projects end up uploading and make them more obviously visible. I 
> don't know if pypi as a distribution for _end user_ (vs developer/power user) 
> software makes sense or not, but if it does we should support it better than 
> accidentally via distutils. 

My concern here is that if someone has a hacky workaround working with the 
current system, it might be better to add support for the new thing ("related 
files") before killing the old thing.  If the plan is to do them both anyway, 
wouldn't it be better to do it in that order?  As a community (and I mean the 
broader open source community here, not distutils-sig; if anything distutils is 
way better about this) we have an unfortunate habit of killing 
potentially-useful-but-sub-optimal stuff, wandering off for half a decade, and 
then only adding the better thing after the fact.

-glyph
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Barry Warsaw]

On Aug 15, 2016, at 05:39 PM, Donald Stufft wrote:

>so narrowing it down to only .tar.gz and .tgz is pretty safe, but practically
>nobody uses .tgz anyways so why bother?

I agree with all of your reasoning.  FWIW, I personally use .tgz all the time
when making backups of things locally, but rarely use it for publicly
consumable artifacts.  And since the sdists setupdistutilstools generates will
only be .tar.gz, I think it's fine to disallow .tgz.

Cheers,
-Barry


pgpdf_YU0aTD9.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 15, 2016, at 4:20 PM, Barry Warsaw  wrote:
> 
> On Aug 15, 2016, at 03:09 PM, Donald Stufft wrote:
> 
>> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
>> and bdist_dumb. I also believe that there is a strong case for removing
>> bdist_msi and bdist_wininst. I also think we should consider removing
>> bdist_egg.
> 
> +1 for all of these, or maybe with +0 for bdist_egg in the short term.  My
> only concern is that the deprecation period has been long enough, but I do
> agree that they should eventually be prohibited.


Yea, egg is sort of the stickiest thing here, but I also think it’s a good
idea, at least in the long term. We can play around with timescales as it
suits us of course.

>> On a related but different note, I'd also like to restrict the acceptable
>> extensions for sdists. Currently distutils allows people to generate .tar,
>> .tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly
>> others. This is a bit problematic because each of those types requires a
>> different set of optional libraries (which may or may not exist depending on
>> Python version) so you can't really use a lot of them (for example, while
>> .tar.xz might give you better compression, it's also not usable before Python
>> 3).
>> 
>> Looking at numbers again, the current number of projects for each file ext
>> are:
>> 
>> * .tar.gz: 444,338
>> * .zip: 58,774
>> * .tar.bz2: 3,265
>> * .tgz: 217
>> * Everything Else: 0
> 
> I'm a little less certain about this.  It's probably okay, although since the
> .tgz format is the same as .tar.gz, the tools argument doesn't hold for that
> extension.  I know compression nerds like to debate the merits of gz, bz2, and
> xz, but I doubt with the size of most of these files, it's going to make any
> difference.

Right, it’s a few reasons rolled up together:

* Single C library makes it easier to explain to people what they need 
capability
  wise to install a library (and the fact zlib is near universal doesn’t hurt).

* One extension (versus many) makes expectations easier, like your debian/watch
  file, but also things like ensuring a project has only *one* sdist per version
  instead of allowing 12.

* Variances breed bugs, particularly when you have disparate systems all working
  on this data. For example, your nose2 debian/watch file has a bug, it doesn’t
  support nose2-1.0.tar which is currently a supported by distutils/PyPI/pip. Of
  course the more variances you have the more likely you are to get bugs where
  different systems support different extensions, so narrowing it down to only
  .tar.gz and .tgz is pretty safe, but practically nobody uses .tgz anyways so
  why bother?

> 
> FWIW, restricting the extensions would simplify Debian's template for the
> PyPI debian/watch file (e.g. the pattern we use to download packages):
> 
> https://pypi.debian.net/nose2/nose2-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))
> 
> (Ignore the pypi.debian.net redirector.)
> 
> That would be a mild win.

Yea :D This is exactly the sort of thing that I think this makes better. There 
is
a lot of tooling built up around PyPI and the less flexible PyPI is the easier
those tools will have working with PyPI in terms of data. Of course, it is a
balancing act, but I think this is a pretty safe place to skew towards mandating
certain things.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Barry Warsaw]

On Aug 15, 2016, at 03:09 PM, Donald Stufft wrote:

>Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
>and bdist_dumb. I also believe that there is a strong case for removing
>bdist_msi and bdist_wininst. I also think we should consider removing
>bdist_egg.

+1 for all of these, or maybe with +0 for bdist_egg in the short term.  My
only concern is that the deprecation period has been long enough, but I do
agree that they should eventually be prohibited.

>On a related but different note, I'd also like to restrict the acceptable
>extensions for sdists. Currently distutils allows people to generate .tar,
>.tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly
>others. This is a bit problematic because each of those types requires a
>different set of optional libraries (which may or may not exist depending on
>Python version) so you can't really use a lot of them (for example, while
>.tar.xz might give you better compression, it's also not usable before Python
>3).
>
>Looking at numbers again, the current number of projects for each file ext
>are:
>
>* .tar.gz: 444,338
>* .zip: 58,774
>* .tar.bz2: 3,265
>* .tgz: 217
>* Everything Else: 0

I'm a little less certain about this.  It's probably okay, although since the
.tgz format is the same as .tar.gz, the tools argument doesn't hold for that
extension.  I know compression nerds like to debate the merits of gz, bz2, and
xz, but I doubt with the size of most of these files, it's going to make any
difference.

FWIW, restricting the extensions would simplify Debian's template for the
PyPI debian/watch file (e.g. the pattern we use to download packages):

https://pypi.debian.net/nose2/nose2-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

(Ignore the pypi.debian.net redirector.)

That would be a mild win.

Cheers,
-Barry


pgpW5bBvhyE9X.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Glyph Lefkowitz]

> On Aug 15, 2016, at 12:09 PM, Donald Stufft  wrote:
> 
> Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these 
> together
> because they're all OS specific installers for OSs that don't already have 
> some
> sort of repository. This lack of a repository format for them means that 
> random
> downloads are already the norm for people using these systems. For these, I
> think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
> are not very important to continue to support, but it would be weird to
> eliminate them without also elminating bdist_wininst. The wininst format has
> the most usage out of all of the seldom used formats, however when we look at
> the downloads for the last 30 days only 0.42% of the downloads were for 
> wininst
> files, so I think that it's pretty safe to remove them. I think in the past,
> these were more important due to the lack of real binary packages on Windows,
> but I think in 2016 we have wheel, and Wheel is a better solution. If however
> we want to keep them, then I think it's pretty safe to remove them from our
> /simple/ pages and any future repository pages and modify our mirroring 
> tooling
> to stop mirroring them. IOW, to treat them as some sort of "additional upload"
> rather than release uploads to PyPI.

I think you have a better handle on this than I do, but I did just want to 
provide a little input.  I think we should be cautious in the way these are 
disabled, because it's already hard enough to produce user-facing software with 
Python. We don't want to throw up yet another roadblock to creating a 
layperson-friendly download.  If they're not used now, it's not necessarily an 
indication of whether we _want_ them to be used in the future.

Also, since these formats aren't readily 'pip install'-able, and are really 
only suitable for applications anyway, perhaps the download numbers are skewed? 
 Automated systems doing 1000s of builds per day are likely to massively 
inflate download counts even if they're used by a far smaller number of users.

Anyway, like I said: not an expert here, just wanted to make sure the "python 
for desktop software (even if it's not used much right now)" angle is 
considered as well.

-glyph

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 15, 2016, at 3:27 PM, Thomas Kluyver  wrote:
> 
> On Mon, Aug 15, 2016, at 08:09 PM, Donald Stufft wrote:
>> Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
>> number of people still upload eggs, even though we have the wheel format.
>> However, I think that we should (and generally do) consider eggs to be
>> deprecated and while we don't want to break existing packages by removing
>> them,
>> we should block further uploads for them. Looking again at the download
>> numbers
>> eggs represented only 1.8% of total downloads in the last 30 days while
>> wheels
>> represented 41% and sdists represented 56%.
> 
> Is it possible to get the numbers for the proportion of uploads
> represented by different formats?

I’m not sure what you mean exactly, the numbers in my original email:

* sdist: 506,585
* bdist_wheel: 81,207
* bdist_egg: 48,282
* bdist_wininst: 14,002
* bdist_dumb: 5,502
* bdist_msi: 497
* bdist_rpm: 464
* bdist_dmg: 45

represent the total files uploaded for each file type. Do you just want
them converted to percentages? If so they’re:

* sdist: 77.1%
* bdist_wheel: 12.3%
* bdist_egg: 7.3%
* bdist_wininst: 2.1%
* bdist_dumb: 0.8%
* bdist_msi: 0.07%
* bdist_rpm: 0.07%
* bdist_dmg: 0.006%

Or were you wondering for the last 30 days like I did for downloads? If
so then:

* sdist: 15601 (66%)
* bdist_wheel: 6398 (27%)
* bdist_egg: 1076 (4.5%)
* bdist_dumb: 195 (0.8%)
* bdist_wininst: 167 (0.7%)
* bdist_rpm: 38 (0.1%)
* bdist_msi: 9 (0.03%)
* Everything Else: 0 (0%)

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Ian Cordasco]

On Mon, Aug 15, 2016 at 2:30 PM, Alex Grönholm  wrote:
> 15.08.2016, 22:28, Donald Stufft kirjoitti:
>>>
>>> On Aug 15, 2016, at 3:22 PM, Ian Cordasco 
>>> wrote:
>>>
>>> My only thought is how we convey this message to users. I wonder if it
>>> would be beneficial to have Twine cut a release that warns users when
>>> they are uploading something that will be unsupported, then have
>>> Warehouse/PyPI start returning a 415 (Unsupported media type)
>>> approximately a few weeks/month later.
>>
>> I wouldn’t be opposed to something like this, though I’m not entirely sure
>> it’s going to be super useful. I’m not sure if a 415 is the correct
>> response
>> code since these are being sent as binary blobs as far as HTTP is
>> concerned,
>> but the Content-Type of the HTTP request will still be correct, but just
>> the
>> data inside it will be wrong, so I think that’s solidly a 400 error? Not
>> that
>> it’s super important, that’s an implementation detail :)
>
> I think it'll be more important to let the users know exactly *why* their
> uploads failed (and what to do about it) when the change finally takes
> place.

Right. I'm not tied to 415 at all. I just want something firm that
people not using twine can understand. PyPI currently basically only
ever returns 400s and Warehouse has improved things. I'm just hoping
we find the right thing early on (and that may be a 400) but I just
want something obvious.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Alex Grönholm]

15.08.2016, 22:28, Donald Stufft kirjoitti:

On Aug 15, 2016, at 3:22 PM, Ian Cordasco  wrote:

My only thought is how we convey this message to users. I wonder if it
would be beneficial to have Twine cut a release that warns users when
they are uploading something that will be unsupported, then have
Warehouse/PyPI start returning a 415 (Unsupported media type)
approximately a few weeks/month later.

I wouldn’t be opposed to something like this, though I’m not entirely sure
it’s going to be super useful. I’m not sure if a 415 is the correct response
code since these are being sent as binary blobs as far as HTTP is concerned,
but the Content-Type of the HTTP request will still be correct, but just the
data inside it will be wrong, so I think that’s solidly a 400 error? Not that
it’s super important, that’s an implementation detail :)
I think it'll be more important to let the users know exactly *why* 
their uploads failed (and what to do about it) when the change finally 
takes place.



I'm +1 for restricting the kinds of things people can upload though.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 15, 2016, at 3:22 PM, Ian Cordasco  wrote:
> 
> My only thought is how we convey this message to users. I wonder if it
> would be beneficial to have Twine cut a release that warns users when
> they are uploading something that will be unsupported, then have
> Warehouse/PyPI start returning a 415 (Unsupported media type)
> approximately a few weeks/month later.

I wouldn’t be opposed to something like this, though I’m not entirely sure
it’s going to be super useful. I’m not sure if a 415 is the correct response
code since these are being sent as binary blobs as far as HTTP is concerned,
but the Content-Type of the HTTP request will still be correct, but just the
data inside it will be wrong, so I think that’s solidly a 400 error? Not that
it’s super important, that’s an implementation detail :)

> 
> I'm +1 for restricting the kinds of things people can upload though.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Thomas Kluyver]

On Mon, Aug 15, 2016, at 08:09 PM, Donald Stufft wrote:
> Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
> number of people still upload eggs, even though we have the wheel format.
> However, I think that we should (and generally do) consider eggs to be
> deprecated and while we don't want to break existing packages by removing
> them,
> we should block further uploads for them. Looking again at the download
> numbers
> eggs represented only 1.8% of total downloads in the last 30 days while
> wheels
> represented 41% and sdists represented 56%.

Is it possible to get the numbers for the proportion of uploads
represented by different formats?

I think this simplification is a good idea, but I expect there will be
some people frustrated when their egg uploads no longer work.

We still upload .zip sdists for some of our projects, in addition to
.tar.gz, but I think that's a habit left over from the days when it was
common to download and unpack sdists manually to run setup.py. I don't
think we lose anything if we drop the .zip downloads now.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Ian Cordasco]

On Mon, Aug 15, 2016 at 2:09 PM, Donald Stufft  wrote:
> Hello!
>
> I'd like to restrict what folks can upload to PyPI in an effort to help narrow
> the scope down and to enable more a more consistent experience for everyone.
>
> First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I 
> think
> that we should try to get rid of support for most of these. Just for reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.
>
> First of all, when I say "remove", I mean disallow new uploads, but do not
> delete the existing files.
>
> Looking at each file type:
>
> I think that bdist_dumb is a pretty easy one to remove. It's format is such
> that it's basically not a very useful format to begin with. It takes the full
> path to the files and stores them in the repository. So If I install something
> to /opt/mycoolproject/lib/python3.5/site-packages/froblib/ then it will have
> paths that look like opt/mycoolproject/lib/python3.5/site-packages/froblib/...
> I think this is obviously not very useful and not many people have uploaded 
> any
> bdist_dumb files at all. They are also problematic because they have the same
> file extension as sdists, so pip doesn't really have a great way to
> differentiate between bdist_dumbs and sdists except by trying to guess if it
> contains one of distutils's adhoc platform tags.
>
> Looking at bdist_rpm, practically nobody has ever used it with a total of 45
> files ever uploaded for it. It's not super useful to be able to upload rpms to
> PyPI since it's not an RPM repository so people have to manually download them
> and then install them manually. It's also a bit weird to have support for RPMs
> but not for all of the other package formats that people might want.
>
> Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these 
> together
> because they're all OS specific installers for OSs that don't already have 
> some
> sort of repository. This lack of a repository format for them means that 
> random
> downloads are already the norm for people using these systems. For these, I
> think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
> are not very important to continue to support, but it would be weird to
> eliminate them without also elminating bdist_wininst. The wininst format has
> the most usage out of all of the seldom used formats, however when we look at
> the downloads for the last 30 days only 0.42% of the downloads were for 
> wininst
> files, so I think that it's pretty safe to remove them. I think in the past,
> these were more important due to the lack of real binary packages on Windows,
> but I think in 2016 we have wheel, and Wheel is a better solution. If however
> we want to keep them, then I think it's pretty safe to remove them from our
> /simple/ pages and any future repository pages and modify our mirroring 
> tooling
> to stop mirroring them. IOW, to treat them as some sort of "additional upload"
> rather than release uploads to PyPI.
>
> Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
> number of people still upload eggs, even though we have the wheel format.
> However, I think that we should (and generally do) consider eggs to be
> deprecated and while we don't want to break existing packages by removing 
> them,
> we should block further uploads for them. Looking again at the download 
> numbers
> eggs represented only 1.8% of total downloads in the last 30 days while wheels
> represented 41% and sdists represented 56%. Further more, I think we should do
> this with the expectation that any new repository API will no longer include
> egg files in them, and will just be sdists and wheels.
>
> Doing all of this would leave us with:
>
> * The /simple/ repository only having sdists, wheels, and eggs and disallowing
>   new eggs to be uploaded.
> * The hypothetical repository 2.0 api only having sdists and wheels.
> * *MAYBE* Having "related files" that people could upload things like
>   Windows/OSX Installers.
>
> On a related but different note, I'd also like to restrict the acceptable
> extensions for sdists. Currently distutils allows people to generate .tar,
> .tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly others. This
> is a bit problematic because each of those types requires a different set of
> optional libraries (which may or may not exist depending on Python version) so
> you can't really use a lot of them (for 

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Alex Grönholm]

+1 for disallowing uploads of anything else than sdists and wheels.


15.08.2016, 22:09, Donald Stufft kirjoitti:

Hello!

I'd like to restrict what folks can upload to PyPI in an effort to help narrow
the scope down and to enable more a more consistent experience for everyone.

First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I think
that we should try to get rid of support for most of these. Just for reference
currently the number of files uploaded for each type of file looks like:

* sdist: 506,585
* bdist_wheel: 81,207
* bdist_egg: 48,282
* bdist_wininst: 14,002
* bdist_dumb: 5,502
* bdist_msi: 497
* bdist_rpm: 464
* bdist_dmg: 45

Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
and bdist_dumb. I also believe that there is a strong case for removing
bdist_msi and bdist_wininst. I also think we should consider removing
bdist_egg.

First of all, when I say "remove", I mean disallow new uploads, but do not
delete the existing files.

Looking at each file type:

I think that bdist_dumb is a pretty easy one to remove. It's format is such
that it's basically not a very useful format to begin with. It takes the full
path to the files and stores them in the repository. So If I install something
to /opt/mycoolproject/lib/python3.5/site-packages/froblib/ then it will have
paths that look like opt/mycoolproject/lib/python3.5/site-packages/froblib/...
I think this is obviously not very useful and not many people have uploaded any
bdist_dumb files at all. They are also problematic because they have the same
file extension as sdists, so pip doesn't really have a great way to
differentiate between bdist_dumbs and sdists except by trying to guess if it
contains one of distutils's adhoc platform tags.

Looking at bdist_rpm, practically nobody has ever used it with a total of 45
files ever uploaded for it. It's not super useful to be able to upload rpms to
PyPI since it's not an RPM repository so people have to manually download them
and then install them manually. It's also a bit weird to have support for RPMs
but not for all of the other package formats that people might want.

Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these together
because they're all OS specific installers for OSs that don't already have some
sort of repository. This lack of a repository format for them means that random
downloads are already the norm for people using these systems. For these, I
think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
are not very important to continue to support, but it would be weird to
eliminate them without also elminating bdist_wininst. The wininst format has
the most usage out of all of the seldom used formats, however when we look at
the downloads for the last 30 days only 0.42% of the downloads were for wininst
files, so I think that it's pretty safe to remove them. I think in the past,
these were more important due to the lack of real binary packages on Windows,
but I think in 2016 we have wheel, and Wheel is a better solution. If however
we want to keep them, then I think it's pretty safe to remove them from our
/simple/ pages and any future repository pages and modify our mirroring tooling
to stop mirroring them. IOW, to treat them as some sort of "additional upload"
rather than release uploads to PyPI.

Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
number of people still upload eggs, even though we have the wheel format.
However, I think that we should (and generally do) consider eggs to be
deprecated and while we don't want to break existing packages by removing them,
we should block further uploads for them. Looking again at the download numbers
eggs represented only 1.8% of total downloads in the last 30 days while wheels
represented 41% and sdists represented 56%. Further more, I think we should do
this with the expectation that any new repository API will no longer include
egg files in them, and will just be sdists and wheels.

Doing all of this would leave us with:

* The /simple/ repository only having sdists, wheels, and eggs and disallowing
   new eggs to be uploaded.
* The hypothetical repository 2.0 api only having sdists and wheels.
* *MAYBE* Having "related files" that people could upload things like
   Windows/OSX Installers.

On a related but different note, I'd also like to restrict the acceptable
extensions for sdists. Currently distutils allows people to generate .tar,
.tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly others. This
is a bit problematic because each of those types requires a different set of
optional libraries (which may or may not exist depending on Python version) so
you can't really use a lot of them (for example, while .tar.xz might give you
better compression, it's also not usable before Python 3).

Looking at numbers again, the current number of