Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 16 Aug 2016 05:09, "Donald Stufft"  wrote:
>
> Hello!
>
> I'd like to restrict what folks can upload to PyPI in an effort to help
narrow
> the scope down and to enable more a more consistent experience for
everyone.
>
> First off, we currently allow people to upload sdist, bdist_wheel,
bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I
think
> that we should try to get rid of support for most of these. Just for
reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg,
bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.
>
> First of all, when I say "remove", I mean disallow new uploads, but do not
> delete the existing files.

General +1 from me for data driven design simplification.

As far as ideas for *how* to go about it go, I think there a few user
categories to consider:

* new user making their first project: we can deprecate legacy formats
aggressively here (since there shouldn't be many, if any, backwards
compatibility considerations for either automated workflows or people's
habits)

* new release of existing project: here is where we want to actively warn
project maintainers using the old formats that PyPI's behaviour will be
changing *before* we force them to change their workflows and habits

*  new project by existing maintainer: here, I'd be inclined to flag
maintainer accounts at the start of the deprecation period based on whether
or not they're currently using the legacy formats on any of their projects
- that is, the migration period would be applied at the *user* level, in
addition to the project level. If someone has never used the legacy
formats, they can be treated like a new user immediately, and will
presumably never even notice the change.

Cheers,
Nick.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 15, 2016, at 7:12 PM, Glyph Lefkowitz  wrote:
> 
> 
>> On Aug 15, 2016, at 1:56 PM, Donald Stufft  wrote:
>> 
>> My main thought regarding this is that bdist_dmg != all dmg files (similarly 
>> for msi and wininst). These are specific files created by distutils without 
>> a standard or without the needed work to make them truly what users should 
>> be using. I also think they are a different class of upload, the general use 
>> case for PyPI's current file uploads are for automated installs (as 
>> evidenced by the simple API and mirroring).
> 
> I guess I'm just a little confused - are we talking about just hiding them 
> from some parts of the API or disallowing their upload entirely?
> 
> If we're talking about the literal output of bdist_dmg and bdist_rpm I 
> probably agree that they're almost useless.

Right now we’re basically talking about the literal output of bdist_dmg and 
bdist_rpm, because their outputs are the only thing we actually support 
uploading. Our current checks aren’t very stringent (or in some cases, exist at 
all) so it’s *possible* someone is constructing an actual user friendly .dmg 
and uploading it pretending to be a “bdist_dmg”, that’s not really a supported 
operation.

Ideally I’d like to disallow their upload completely and hide them from the 
API, but if the current use case of bdist_dmg, bdist_rpm, bdist_wininst, etc is 
useful enough to keep around, then I’d like to just hide them from the API.

> 
>> If we want to enable dmg, msi, etc uploads that are not the bdist_* variety 
>> for automated tooling, then we could do something like "related files" 
>> people can upload which don't get mirrored for pip and which don't show up 
>> in the repo API. Since they will be classified differently we could also do 
>> better work around the ux of discovering them and separate them from the 50 
>> wheels that some projects end up uploading and make them more obviously 
>> visible. I don't know if pypi as a distribution for _end user_ (vs 
>> developer/power user) software makes sense or not, but if it does we should 
>> support it better than accidentally via distutils. 
> 
> My concern here is that if someone has a hacky workaround working with the 
> current system, it might be better to add support for the new thing ("related 
> files") before killing the old thing.  If the plan is to do them both anyway, 
> wouldn't it be better to do it in that order?  As a community (and I mean the 
> broader open source community here, not distutils-sig; if anything distutils 
> is way better about this) we have an unfortunate habit of killing 
> potentially-useful-but-sub-optimal stuff, wandering off for half a decade, 
> and then only adding the better thing after the fact.

I don’t know if it makes sense to add the hypothetical new thing. Is PyPI the 
right place to distribute a .dmg that say, ships Python itself, some C 
libraries, some Python libraries, some bash scripts, and some static files? I 
don’t know. Currently I think the answer is “no”, PyPI is a repository of 
software _for_ Python, not a repository of software that just happens to be 
written in Python. If the ultimate end goal of a .dmg is that people no longer 
need to worry about what language the thing they are installing is written in, 
it seems weird for them to go to PyPI for a .dmg for one app, npmjs.org for a 
.dmg for another app, and so on.

Now, if it *does* make sense to support generalized uploads for applications 
written in Python with some sort of system level packaging (dmg, deb, conda, 
whatever) then we should figure out a way to actually support that, and support 
it well. As it is, I don’t have any evidence that the files that are currently 
being uploaded to PyPI are anything but “wheels, but in dmg format” (e.g. 
binary packages containing a single library). I don’t want to put a bunch of 
effort and work into making this well supported if there isn’t some evidence 
that folks will actually use it (I suspect a minimum we’d need some buy in from 
the authors of tooling to make these self contained packages).


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Glyph Lefkowitz]

> On Aug 15, 2016, at 1:56 PM, Donald Stufft  wrote:
> 
> My main thought regarding this is that bdist_dmg != all dmg files (similarly 
> for msi and wininst). These are specific files created by distutils without a 
> standard or without the needed work to make them truly what users should be 
> using. I also think they are a different class of upload, the general use 
> case for PyPI's current file uploads are for automated installs (as evidenced 
> by the simple API and mirroring).

I guess I'm just a little confused - are we talking about just hiding them from 
some parts of the API or disallowing their upload entirely?

If we're talking about the literal output of bdist_dmg and bdist_rpm I probably 
agree that they're almost useless.

> If we want to enable dmg, msi, etc uploads that are not the bdist_* variety 
> for automated tooling, then we could do something like "related files" people 
> can upload which don't get mirrored for pip and which don't show up in the 
> repo API. Since they will be classified differently we could also do better 
> work around the ux of discovering them and separate them from the 50 wheels 
> that some projects end up uploading and make them more obviously visible. I 
> don't know if pypi as a distribution for _end user_ (vs developer/power user) 
> software makes sense or not, but if it does we should support it better than 
> accidentally via distutils. 

My concern here is that if someone has a hacky workaround working with the 
current system, it might be better to add support for the new thing ("related 
files") before killing the old thing.  If the plan is to do them both anyway, 
wouldn't it be better to do it in that order?  As a community (and I mean the 
broader open source community here, not distutils-sig; if anything distutils is 
way better about this) we have an unfortunate habit of killing 
potentially-useful-but-sub-optimal stuff, wandering off for half a decade, and 
then only adding the better thing after the fact.

-glyph
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Barry Warsaw]

On Aug 15, 2016, at 05:39 PM, Donald Stufft wrote:

>so narrowing it down to only .tar.gz and .tgz is pretty safe, but practically
>nobody uses .tgz anyways so why bother?

I agree with all of your reasoning.  FWIW, I personally use .tgz all the time
when making backups of things locally, but rarely use it for publicly
consumable artifacts.  And since the sdists setupdistutilstools generates will
only be .tar.gz, I think it's fine to disallow .tgz.

Cheers,
-Barry


pgpdf_YU0aTD9.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 15, 2016, at 4:20 PM, Barry Warsaw  wrote:
> 
> On Aug 15, 2016, at 03:09 PM, Donald Stufft wrote:
> 
>> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
>> and bdist_dumb. I also believe that there is a strong case for removing
>> bdist_msi and bdist_wininst. I also think we should consider removing
>> bdist_egg.
> 
> +1 for all of these, or maybe with +0 for bdist_egg in the short term.  My
> only concern is that the deprecation period has been long enough, but I do
> agree that they should eventually be prohibited.


Yea, egg is sort of the stickiest thing here, but I also think it’s a good
idea, at least in the long term. We can play around with timescales as it
suits us of course.

>> On a related but different note, I'd also like to restrict the acceptable
>> extensions for sdists. Currently distutils allows people to generate .tar,
>> .tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly
>> others. This is a bit problematic because each of those types requires a
>> different set of optional libraries (which may or may not exist depending on
>> Python version) so you can't really use a lot of them (for example, while
>> .tar.xz might give you better compression, it's also not usable before Python
>> 3).
>> 
>> Looking at numbers again, the current number of projects for each file ext
>> are:
>> 
>> * .tar.gz: 444,338
>> * .zip: 58,774
>> * .tar.bz2: 3,265
>> * .tgz: 217
>> * Everything Else: 0
> 
> I'm a little less certain about this.  It's probably okay, although since the
> .tgz format is the same as .tar.gz, the tools argument doesn't hold for that
> extension.  I know compression nerds like to debate the merits of gz, bz2, and
> xz, but I doubt with the size of most of these files, it's going to make any
> difference.

Right, it’s a few reasons rolled up together:

* Single C library makes it easier to explain to people what they need 
capability
  wise to install a library (and the fact zlib is near universal doesn’t hurt).

* One extension (versus many) makes expectations easier, like your debian/watch
  file, but also things like ensuring a project has only *one* sdist per version
  instead of allowing 12.

* Variances breed bugs, particularly when you have disparate systems all working
  on this data. For example, your nose2 debian/watch file has a bug, it doesn’t
  support nose2-1.0.tar which is currently a supported by distutils/PyPI/pip. Of
  course the more variances you have the more likely you are to get bugs where
  different systems support different extensions, so narrowing it down to only
  .tar.gz and .tgz is pretty safe, but practically nobody uses .tgz anyways so
  why bother?

> 
> FWIW, restricting the extensions would simplify Debian's template for the
> PyPI debian/watch file (e.g. the pattern we use to download packages):
> 
> https://pypi.debian.net/nose2/nose2-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))
> 
> (Ignore the pypi.debian.net redirector.)
> 
> That would be a mild win.

Yea :D This is exactly the sort of thing that I think this makes better. There 
is
a lot of tooling built up around PyPI and the less flexible PyPI is the easier
those tools will have working with PyPI in terms of data. Of course, it is a
balancing act, but I think this is a pretty safe place to skew towards mandating
certain things.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Barry Warsaw]

On Aug 15, 2016, at 03:09 PM, Donald Stufft wrote:

>Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
>and bdist_dumb. I also believe that there is a strong case for removing
>bdist_msi and bdist_wininst. I also think we should consider removing
>bdist_egg.

+1 for all of these, or maybe with +0 for bdist_egg in the short term.  My
only concern is that the deprecation period has been long enough, but I do
agree that they should eventually be prohibited.

>On a related but different note, I'd also like to restrict the acceptable
>extensions for sdists. Currently distutils allows people to generate .tar,
>.tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly
>others. This is a bit problematic because each of those types requires a
>different set of optional libraries (which may or may not exist depending on
>Python version) so you can't really use a lot of them (for example, while
>.tar.xz might give you better compression, it's also not usable before Python
>3).
>
>Looking at numbers again, the current number of projects for each file ext
>are:
>
>* .tar.gz: 444,338
>* .zip: 58,774
>* .tar.bz2: 3,265
>* .tgz: 217
>* Everything Else: 0

I'm a little less certain about this.  It's probably okay, although since the
.tgz format is the same as .tar.gz, the tools argument doesn't hold for that
extension.  I know compression nerds like to debate the merits of gz, bz2, and
xz, but I doubt with the size of most of these files, it's going to make any
difference.

FWIW, restricting the extensions would simplify Debian's template for the
PyPI debian/watch file (e.g. the pattern we use to download packages):

https://pypi.debian.net/nose2/nose2-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

(Ignore the pypi.debian.net redirector.)

That would be a mild win.

Cheers,
-Barry


pgpW5bBvhyE9X.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Glyph Lefkowitz]

> On Aug 15, 2016, at 12:09 PM, Donald Stufft  wrote:
> 
> Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these 
> together
> because they're all OS specific installers for OSs that don't already have 
> some
> sort of repository. This lack of a repository format for them means that 
> random
> downloads are already the norm for people using these systems. For these, I
> think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
> are not very important to continue to support, but it would be weird to
> eliminate them without also elminating bdist_wininst. The wininst format has
> the most usage out of all of the seldom used formats, however when we look at
> the downloads for the last 30 days only 0.42% of the downloads were for 
> wininst
> files, so I think that it's pretty safe to remove them. I think in the past,
> these were more important due to the lack of real binary packages on Windows,
> but I think in 2016 we have wheel, and Wheel is a better solution. If however
> we want to keep them, then I think it's pretty safe to remove them from our
> /simple/ pages and any future repository pages and modify our mirroring 
> tooling
> to stop mirroring them. IOW, to treat them as some sort of "additional upload"
> rather than release uploads to PyPI.

I think you have a better handle on this than I do, but I did just want to 
provide a little input.  I think we should be cautious in the way these are 
disabled, because it's already hard enough to produce user-facing software with 
Python. We don't want to throw up yet another roadblock to creating a 
layperson-friendly download.  If they're not used now, it's not necessarily an 
indication of whether we _want_ them to be used in the future.

Also, since these formats aren't readily 'pip install'-able, and are really 
only suitable for applications anyway, perhaps the download numbers are skewed? 
 Automated systems doing 1000s of builds per day are likely to massively 
inflate download counts even if they're used by a far smaller number of users.

Anyway, like I said: not an expert here, just wanted to make sure the "python 
for desktop software (even if it's not used much right now)" angle is 
considered as well.

-glyph

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 15, 2016, at 3:27 PM, Thomas Kluyver  wrote:
> 
> On Mon, Aug 15, 2016, at 08:09 PM, Donald Stufft wrote:
>> Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
>> number of people still upload eggs, even though we have the wheel format.
>> However, I think that we should (and generally do) consider eggs to be
>> deprecated and while we don't want to break existing packages by removing
>> them,
>> we should block further uploads for them. Looking again at the download
>> numbers
>> eggs represented only 1.8% of total downloads in the last 30 days while
>> wheels
>> represented 41% and sdists represented 56%.
> 
> Is it possible to get the numbers for the proportion of uploads
> represented by different formats?

I’m not sure what you mean exactly, the numbers in my original email:

* sdist: 506,585
* bdist_wheel: 81,207
* bdist_egg: 48,282
* bdist_wininst: 14,002
* bdist_dumb: 5,502
* bdist_msi: 497
* bdist_rpm: 464
* bdist_dmg: 45

represent the total files uploaded for each file type. Do you just want
them converted to percentages? If so they’re:

* sdist: 77.1%
* bdist_wheel: 12.3%
* bdist_egg: 7.3%
* bdist_wininst: 2.1%
* bdist_dumb: 0.8%
* bdist_msi: 0.07%
* bdist_rpm: 0.07%
* bdist_dmg: 0.006%

Or were you wondering for the last 30 days like I did for downloads? If
so then:

* sdist: 15601 (66%)
* bdist_wheel: 6398 (27%)
* bdist_egg: 1076 (4.5%)
* bdist_dumb: 195 (0.8%)
* bdist_wininst: 167 (0.7%)
* bdist_rpm: 38 (0.1%)
* bdist_msi: 9 (0.03%)
* Everything Else: 0 (0%)

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Ian Cordasco]

On Mon, Aug 15, 2016 at 2:30 PM, Alex Grönholm  wrote:
> 15.08.2016, 22:28, Donald Stufft kirjoitti:
>>>
>>> On Aug 15, 2016, at 3:22 PM, Ian Cordasco 
>>> wrote:
>>>
>>> My only thought is how we convey this message to users. I wonder if it
>>> would be beneficial to have Twine cut a release that warns users when
>>> they are uploading something that will be unsupported, then have
>>> Warehouse/PyPI start returning a 415 (Unsupported media type)
>>> approximately a few weeks/month later.
>>
>> I wouldn’t be opposed to something like this, though I’m not entirely sure
>> it’s going to be super useful. I’m not sure if a 415 is the correct
>> response
>> code since these are being sent as binary blobs as far as HTTP is
>> concerned,
>> but the Content-Type of the HTTP request will still be correct, but just
>> the
>> data inside it will be wrong, so I think that’s solidly a 400 error? Not
>> that
>> it’s super important, that’s an implementation detail :)
>
> I think it'll be more important to let the users know exactly *why* their
> uploads failed (and what to do about it) when the change finally takes
> place.

Right. I'm not tied to 415 at all. I just want something firm that
people not using twine can understand. PyPI currently basically only
ever returns 400s and Warehouse has improved things. I'm just hoping
we find the right thing early on (and that may be a 400) but I just
want something obvious.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Alex Grönholm]

15.08.2016, 22:28, Donald Stufft kirjoitti:

On Aug 15, 2016, at 3:22 PM, Ian Cordasco  wrote:

My only thought is how we convey this message to users. I wonder if it
would be beneficial to have Twine cut a release that warns users when
they are uploading something that will be unsupported, then have
Warehouse/PyPI start returning a 415 (Unsupported media type)
approximately a few weeks/month later.

I wouldn’t be opposed to something like this, though I’m not entirely sure
it’s going to be super useful. I’m not sure if a 415 is the correct response
code since these are being sent as binary blobs as far as HTTP is concerned,
but the Content-Type of the HTTP request will still be correct, but just the
data inside it will be wrong, so I think that’s solidly a 400 error? Not that
it’s super important, that’s an implementation detail :)
I think it'll be more important to let the users know exactly *why* 
their uploads failed (and what to do about it) when the change finally 
takes place.



I'm +1 for restricting the kinds of things people can upload though.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 15, 2016, at 3:22 PM, Ian Cordasco  wrote:
> 
> My only thought is how we convey this message to users. I wonder if it
> would be beneficial to have Twine cut a release that warns users when
> they are uploading something that will be unsupported, then have
> Warehouse/PyPI start returning a 415 (Unsupported media type)
> approximately a few weeks/month later.

I wouldn’t be opposed to something like this, though I’m not entirely sure
it’s going to be super useful. I’m not sure if a 415 is the correct response
code since these are being sent as binary blobs as far as HTTP is concerned,
but the Content-Type of the HTTP request will still be correct, but just the
data inside it will be wrong, so I think that’s solidly a 400 error? Not that
it’s super important, that’s an implementation detail :)

> 
> I'm +1 for restricting the kinds of things people can upload though.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Thomas Kluyver]

On Mon, Aug 15, 2016, at 08:09 PM, Donald Stufft wrote:
> Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
> number of people still upload eggs, even though we have the wheel format.
> However, I think that we should (and generally do) consider eggs to be
> deprecated and while we don't want to break existing packages by removing
> them,
> we should block further uploads for them. Looking again at the download
> numbers
> eggs represented only 1.8% of total downloads in the last 30 days while
> wheels
> represented 41% and sdists represented 56%.

Is it possible to get the numbers for the proportion of uploads
represented by different formats?

I think this simplification is a good idea, but I expect there will be
some people frustrated when their egg uploads no longer work.

We still upload .zip sdists for some of our projects, in addition to
.tar.gz, but I think that's a habit left over from the days when it was
common to download and unpack sdists manually to run setup.py. I don't
think we lose anything if we drop the .zip downloads now.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Ian Cordasco]

On Mon, Aug 15, 2016 at 2:09 PM, Donald Stufft  wrote:
> Hello!
>
> I'd like to restrict what folks can upload to PyPI in an effort to help narrow
> the scope down and to enable more a more consistent experience for everyone.
>
> First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I 
> think
> that we should try to get rid of support for most of these. Just for reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.
>
> First of all, when I say "remove", I mean disallow new uploads, but do not
> delete the existing files.
>
> Looking at each file type:
>
> I think that bdist_dumb is a pretty easy one to remove. It's format is such
> that it's basically not a very useful format to begin with. It takes the full
> path to the files and stores them in the repository. So If I install something
> to /opt/mycoolproject/lib/python3.5/site-packages/froblib/ then it will have
> paths that look like opt/mycoolproject/lib/python3.5/site-packages/froblib/...
> I think this is obviously not very useful and not many people have uploaded 
> any
> bdist_dumb files at all. They are also problematic because they have the same
> file extension as sdists, so pip doesn't really have a great way to
> differentiate between bdist_dumbs and sdists except by trying to guess if it
> contains one of distutils's adhoc platform tags.
>
> Looking at bdist_rpm, practically nobody has ever used it with a total of 45
> files ever uploaded for it. It's not super useful to be able to upload rpms to
> PyPI since it's not an RPM repository so people have to manually download them
> and then install them manually. It's also a bit weird to have support for RPMs
> but not for all of the other package formats that people might want.
>
> Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these 
> together
> because they're all OS specific installers for OSs that don't already have 
> some
> sort of repository. This lack of a repository format for them means that 
> random
> downloads are already the norm for people using these systems. For these, I
> think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
> are not very important to continue to support, but it would be weird to
> eliminate them without also elminating bdist_wininst. The wininst format has
> the most usage out of all of the seldom used formats, however when we look at
> the downloads for the last 30 days only 0.42% of the downloads were for 
> wininst
> files, so I think that it's pretty safe to remove them. I think in the past,
> these were more important due to the lack of real binary packages on Windows,
> but I think in 2016 we have wheel, and Wheel is a better solution. If however
> we want to keep them, then I think it's pretty safe to remove them from our
> /simple/ pages and any future repository pages and modify our mirroring 
> tooling
> to stop mirroring them. IOW, to treat them as some sort of "additional upload"
> rather than release uploads to PyPI.
>
> Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
> number of people still upload eggs, even though we have the wheel format.
> However, I think that we should (and generally do) consider eggs to be
> deprecated and while we don't want to break existing packages by removing 
> them,
> we should block further uploads for them. Looking again at the download 
> numbers
> eggs represented only 1.8% of total downloads in the last 30 days while wheels
> represented 41% and sdists represented 56%. Further more, I think we should do
> this with the expectation that any new repository API will no longer include
> egg files in them, and will just be sdists and wheels.
>
> Doing all of this would leave us with:
>
> * The /simple/ repository only having sdists, wheels, and eggs and disallowing
>   new eggs to be uploaded.
> * The hypothetical repository 2.0 api only having sdists and wheels.
> * *MAYBE* Having "related files" that people could upload things like
>   Windows/OSX Installers.
>
> On a related but different note, I'd also like to restrict the acceptable
> extensions for sdists. Currently distutils allows people to generate .tar,
> .tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly others. This
> is a bit problematic because each of those types requires a different set of
> optional libraries (which may or may not exist depending on Python version) so
> you can't really use a lot of them (for 

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Alex Grönholm]

+1 for disallowing uploads of anything else than sdists and wheels.


15.08.2016, 22:09, Donald Stufft kirjoitti:

Hello!

I'd like to restrict what folks can upload to PyPI in an effort to help narrow
the scope down and to enable more a more consistent experience for everyone.

First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I think
that we should try to get rid of support for most of these. Just for reference
currently the number of files uploaded for each type of file looks like:

* sdist: 506,585
* bdist_wheel: 81,207
* bdist_egg: 48,282
* bdist_wininst: 14,002
* bdist_dumb: 5,502
* bdist_msi: 497
* bdist_rpm: 464
* bdist_dmg: 45

Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
and bdist_dumb. I also believe that there is a strong case for removing
bdist_msi and bdist_wininst. I also think we should consider removing
bdist_egg.

First of all, when I say "remove", I mean disallow new uploads, but do not
delete the existing files.

Looking at each file type:

I think that bdist_dumb is a pretty easy one to remove. It's format is such
that it's basically not a very useful format to begin with. It takes the full
path to the files and stores them in the repository. So If I install something
to /opt/mycoolproject/lib/python3.5/site-packages/froblib/ then it will have
paths that look like opt/mycoolproject/lib/python3.5/site-packages/froblib/...
I think this is obviously not very useful and not many people have uploaded any
bdist_dumb files at all. They are also problematic because they have the same
file extension as sdists, so pip doesn't really have a great way to
differentiate between bdist_dumbs and sdists except by trying to guess if it
contains one of distutils's adhoc platform tags.

Looking at bdist_rpm, practically nobody has ever used it with a total of 45
files ever uploaded for it. It's not super useful to be able to upload rpms to
PyPI since it's not an RPM repository so people have to manually download them
and then install them manually. It's also a bit weird to have support for RPMs
but not for all of the other package formats that people might want.

Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these together
because they're all OS specific installers for OSs that don't already have some
sort of repository. This lack of a repository format for them means that random
downloads are already the norm for people using these systems. For these, I
think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
are not very important to continue to support, but it would be weird to
eliminate them without also elminating bdist_wininst. The wininst format has
the most usage out of all of the seldom used formats, however when we look at
the downloads for the last 30 days only 0.42% of the downloads were for wininst
files, so I think that it's pretty safe to remove them. I think in the past,
these were more important due to the lack of real binary packages on Windows,
but I think in 2016 we have wheel, and Wheel is a better solution. If however
we want to keep them, then I think it's pretty safe to remove them from our
/simple/ pages and any future repository pages and modify our mirroring tooling
to stop mirroring them. IOW, to treat them as some sort of "additional upload"
rather than release uploads to PyPI.

Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
number of people still upload eggs, even though we have the wheel format.
However, I think that we should (and generally do) consider eggs to be
deprecated and while we don't want to break existing packages by removing them,
we should block further uploads for them. Looking again at the download numbers
eggs represented only 1.8% of total downloads in the last 30 days while wheels
represented 41% and sdists represented 56%. Further more, I think we should do
this with the expectation that any new repository API will no longer include
egg files in them, and will just be sdists and wheels.

Doing all of this would leave us with:

* The /simple/ repository only having sdists, wheels, and eggs and disallowing
   new eggs to be uploaded.
* The hypothetical repository 2.0 api only having sdists and wheels.
* *MAYBE* Having "related files" that people could upload things like
   Windows/OSX Installers.

On a related but different note, I'd also like to restrict the acceptable
extensions for sdists. Currently distutils allows people to generate .tar,
.tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly others. This
is a bit problematic because each of those types requires a different set of
optional libraries (which may or may not exist depending on Python version) so
you can't really use a lot of them (for example, while .tar.xz might give you
better compression, it's also not usable before Python 3).

Looking at numbers again, the current number of 

[Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

Hello!

I'd like to restrict what folks can upload to PyPI in an effort to help narrow
the scope down and to enable more a more consistent experience for everyone.

First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I think
that we should try to get rid of support for most of these. Just for reference
currently the number of files uploaded for each type of file looks like:

* sdist: 506,585
* bdist_wheel: 81,207
* bdist_egg: 48,282
* bdist_wininst: 14,002
* bdist_dumb: 5,502
* bdist_msi: 497
* bdist_rpm: 464
* bdist_dmg: 45

Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
and bdist_dumb. I also believe that there is a strong case for removing
bdist_msi and bdist_wininst. I also think we should consider removing
bdist_egg.

First of all, when I say "remove", I mean disallow new uploads, but do not
delete the existing files.

Looking at each file type:

I think that bdist_dumb is a pretty easy one to remove. It's format is such
that it's basically not a very useful format to begin with. It takes the full
path to the files and stores them in the repository. So If I install something
to /opt/mycoolproject/lib/python3.5/site-packages/froblib/ then it will have
paths that look like opt/mycoolproject/lib/python3.5/site-packages/froblib/...
I think this is obviously not very useful and not many people have uploaded any
bdist_dumb files at all. They are also problematic because they have the same
file extension as sdists, so pip doesn't really have a great way to
differentiate between bdist_dumbs and sdists except by trying to guess if it
contains one of distutils's adhoc platform tags.

Looking at bdist_rpm, practically nobody has ever used it with a total of 45
files ever uploaded for it. It's not super useful to be able to upload rpms to
PyPI since it's not an RPM repository so people have to manually download them
and then install them manually. It's also a bit weird to have support for RPMs
but not for all of the other package formats that people might want.

Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these together
because they're all OS specific installers for OSs that don't already have some
sort of repository. This lack of a repository format for them means that random
downloads are already the norm for people using these systems. For these, I
think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
are not very important to continue to support, but it would be weird to
eliminate them without also elminating bdist_wininst. The wininst format has
the most usage out of all of the seldom used formats, however when we look at
the downloads for the last 30 days only 0.42% of the downloads were for wininst
files, so I think that it's pretty safe to remove them. I think in the past,
these were more important due to the lack of real binary packages on Windows,
but I think in 2016 we have wheel, and Wheel is a better solution. If however
we want to keep them, then I think it's pretty safe to remove them from our
/simple/ pages and any future repository pages and modify our mirroring tooling
to stop mirroring them. IOW, to treat them as some sort of "additional upload"
rather than release uploads to PyPI.

Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
number of people still upload eggs, even though we have the wheel format.
However, I think that we should (and generally do) consider eggs to be
deprecated and while we don't want to break existing packages by removing them,
we should block further uploads for them. Looking again at the download numbers
eggs represented only 1.8% of total downloads in the last 30 days while wheels
represented 41% and sdists represented 56%. Further more, I think we should do
this with the expectation that any new repository API will no longer include
egg files in them, and will just be sdists and wheels.

Doing all of this would leave us with:

* The /simple/ repository only having sdists, wheels, and eggs and disallowing
  new eggs to be uploaded.
* The hypothetical repository 2.0 api only having sdists and wheels.
* *MAYBE* Having "related files" that people could upload things like
  Windows/OSX Installers.

On a related but different note, I'd also like to restrict the acceptable
extensions for sdists. Currently distutils allows people to generate .tar,
.tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly others. This
is a bit problematic because each of those types requires a different set of
optional libraries (which may or may not exist depending on Python version) so
you can't really use a lot of them (for example, while .tar.xz might give you
better compression, it's also not usable before Python 3).

Looking at numbers again, the current number of projects for each file ext are:

* .tar.gz: 444,338
* .zip: 58,774
* .tar.bz2: 3,265
* .tgz: 217
* Everything Else: 0

Re: [Distutils] Pip maintainers contact

[Robert Collins]

There is no email contact per se. The project tacker is here -
github.com/pypa/pip and there is a google group here -
https://groups.google.com/forum/#!forum/pypa-dev

On 13 August 2016 at 21:28, hyp3rlinx  wrote:
> Can somebody here please tell me the contact email for maintainers of pip.
>
> Thank you,
> John
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Pip maintainers contact

[hyp3rlinx]

Can somebody here please tell me the contact email for maintainers of pip.

Thank you,
John
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig