Re: [Distutils] GnuPG signatures on PyPI: why so few?

[Ian Cordasco]

On Mar 11, 2017 6:47 PM, "Ben Finney"  wrote:

Howdy all,

What prospects are there for PyPI to have GnuPG-signed packages by default?

Debian's UScan has the ability to find, download, and verify the GnuPG
signature for a package source release. Lintian will remind the
maintainer if a Debian source package is not taking advantage of this.

However, this only works if upstream releases are actually accompanied
by a valid GnuPG signature each time. The PyPI infrastructure supports
this; why isn't it more widely encouraged?

This thread from 2016 has a possible answer:

while you can use GPG as is to verify that yes, "Donald Stufft"
signed a particular package, you cannot use it to determine if
"Donald Stufft" is *allowed* to sign for that package, a valid
signature from me on the requests project should be just as invalid
as an invalid signature from anyone on the requests project. The
only namespacing provided by GPG itself is "trusted key" vs "not
trusted key".

[…] I am aware of a single tool anywhere that actively supports
verifying the signatures that people upload to PyPI, and that is
Debian's uscan program. […]

All in all, I think that there is not a whole lot of point to having
this feature in PyPI, it is predicated a bunch of invalid
assumptions (as detailed above) and I do not believe end users are
actually even using the keys that are being uploaded.

[…] Thus, I would like to remove this feature from PyPI […].



The thread has some discussion, and Barry Warsaw makes the case for
Debian's use for signed releases. The last (?) post in the thread has a
kind of interim conclusion:

My main concern when implementing this is how to communicate it to
users […]. [A move that gives the impression] "we're getting rid of
this thing that only kinda works now in favor of something amazing
that doesn't exist yet" is just not a popular move.



In response to polite requests for signed releases, some upstream


I've only ever seen condescending requests in the past but perhaps we have
different definitions of "polite" or perhaps things have genuinely changed.

maintainers are now pointing to that thread and closing bug reports as
“won't fix”.


You may have noticed in that thread that there are plans for better
mechanisms. Mechanisms that don't add significantly more burden to
maintainers of the software we know and love who do this for free and with
their spare time.


What prospect is there in the Python community to get signed upstream
releases become the obvious norm?


Not every package on PyPI is redistributed via Linux packagers. Why then
should someone publishing their tiny little first package have to go
through the hassle of creating a GPG key?

As a maintainer of Twine, I will never force someone to have learned how to
install GPG on their platform, create a key that package maintainers won't
belittle them for, and maintain the key's security in order to upload
something to PyPI.

Further GPG depends on trust. Do you mean to imply that Debian trusts PyPI
packages with a signature more than those without? Even if the key used to
sign it has never been signed by another person? What about keys signed by
people you've never met? Someone can manufacture their own web of trust if
they want to. Why is GPG seen as done kind of magic authenticity bullet?

If you can find a tool that is easy to install on Linux, Windows, and Mac,
which solves the problems above by virtue of having very good defaults, and
is accessible to anyone with less than a few hours to waste on it... Then
maybe I would collaborate to make it a requirement.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Can't upload sdist: "File already exists"

[Ian Cordasco]

On Thu, Dec 22, 2016 at 9:49 AM, Brett Cannon  wrote:
> Because you already uploaded a wheel for version 0.1.2 you can't upload any
> other files for that version, else people could accidentally upload e.g. an
> sdist with code different from what was already uploaded in the wheel. If
> you want an sdist then I would do another release as version 0.1.2post1 with
> the wheel and sdist (or whatever the proper post release version format is;
> on my phone so a pain to look up right now).

I'm pretty sure that's not correct. Twine is written to specifically
upload the wheel first because PyPI will extract metadata from that
and display it on the page. It won't do that if the sdist is uploaded
first.

I'm not able to reproduce the behaviour Nick is seeing. My only guess
is that something changed in Warehouse or the file existed, was
deleted, and is now being re-uploaded with the same version. That's
not something Warehouse or PyPI allows anymore (republishing with the
same version)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Travis-CI is not open source. Was: Current Python packaging status (from my point of view)

[Ian Cordasco]

On Wed, Nov 2, 2016 at 1:00 AM, Thomas Güttler
 wrote:
> I see an other hurdle. travis-ci is very widespread, but AFAIK it is not open 
> source:
>
>https://travis-ci.com/plans


It is open source: https://github.com/travis-ci

Sadly, the infrastructure it takes to operate it is not Free.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 527 - Removing Un(der)used file types/extensions on PyPI

[Ian Cordasco]

On Tue, Aug 23, 2016 at 2:03 PM, M.-A. Lemburg  wrote:
> On 23.08.2016 18:46, Donald Stufft wrote:
>> Since it seemed like there was enough here for a proper PEP I went ahead and
>> write one up, which is now PEP 527. The tl;dr of it is that:
>>
>> * Everything but sdist, bdist_wheel, and bdist_egg get deprecated.
>
> -1 on removing bdist_wininst and bdist_msi. If PyPI is supposed
> to retain the status of the main website to go search for Python
> package downloads, it needs to be able to provide ways of hosting
> all distribution types which are supported by distutils, including
> ones which target platform configuration management system such as
> the Windows one.
>
> The number of downloads is really irrelevant for this kind of
> argument. Since the PEP proposes to keep the existing uploads
> around, I also don't follow the argument of reduced maintenance.
> PyPI will still have to host and support downloading those file
> types.
>
> To me, all this sounds a lot like eventually turning PyPI into a
> pip package index, which no longer serves the original intent of
> a Python package index. I think that's taking a wrong turn in the
> development of such an index.
>
> IMO, we should aim to reunite separate indexes such as the
> one used for conda or the win32 index maintained by
> Christoph Golke back into PyPI, not create even more
> separation by removing platform specific formats.

I disagree. As it is, tools like Twine only introspect the more common
file formats to upload them to PyPI and has not had a single user
complain about it. Given that Twine is advertised as the preferred
method to upload to PyPI, you might expect that this would have been
requested already. Alas it has not. No one using modern tooling for
package management is uploading these file types.

Even if the statistic of downloads (which actually is valid) doesn't
sway you, maybe this will. Alternatively, maybe you will help maintain
support for these file types in Warehouse?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] license for setuptools

[Ian Cordasco]

On Mon, Aug 15, 2016 at 7:35 AM, Marinier, Claude
 wrote:
>> Date: Fri, 12 Aug 2016 02:45:20 +
>> From: Eric Dill 
>> Subject: Re: [Distutils] license for setuptools
>>
>> Hi Claude,
>>
>> There was a recent discussion of the lack of a license file in setuptools
>> here: https://github.com/pypa/setuptools/issues/612 and another important
>> discussion here: https://github.com/pypa/setuptools/issues/132.  This is
>> probably the most relevant quotable bit from those two issues, from Jason
>> Coombs (the primary developer of setuptools:
>>
>> "The [License :: OSI Approved :: MIT License] classifier isn't a suggestion 
>> but a
>> declaration and follows the distutils guide for declaring a license.
>> I consider inclusion of a license file redundant and error prone."
>>
>> Hopefully this resolves your issue about not having an explicit license file.
>
> Thank you Eric and the others. I will use this information for the request.

Claude,

Setuptools now ships a LICENSE file. I'm not sure if there's a release
available with it, but it's now in the SCM tree.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Ian Cordasco]

On Mon, Aug 15, 2016 at 2:30 PM, Alex Grönholm <alex.gronh...@nextday.fi> wrote:
> 15.08.2016, 22:28, Donald Stufft kirjoitti:
>>>
>>> On Aug 15, 2016, at 3:22 PM, Ian Cordasco <graffatcolmin...@gmail.com>
>>> wrote:
>>>
>>> My only thought is how we convey this message to users. I wonder if it
>>> would be beneficial to have Twine cut a release that warns users when
>>> they are uploading something that will be unsupported, then have
>>> Warehouse/PyPI start returning a 415 (Unsupported media type)
>>> approximately a few weeks/month later.
>>
>> I wouldn’t be opposed to something like this, though I’m not entirely sure
>> it’s going to be super useful. I’m not sure if a 415 is the correct
>> response
>> code since these are being sent as binary blobs as far as HTTP is
>> concerned,
>> but the Content-Type of the HTTP request will still be correct, but just
>> the
>> data inside it will be wrong, so I think that’s solidly a 400 error? Not
>> that
>> it’s super important, that’s an implementation detail :)
>
> I think it'll be more important to let the users know exactly *why* their
> uploads failed (and what to do about it) when the change finally takes
> place.

Right. I'm not tied to 415 at all. I just want something firm that
people not using twine can understand. PyPI currently basically only
ever returns 400s and Warehouse has improved things. I'm just hoping
we find the right thing early on (and that may be a 400) but I just
want something obvious.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Ian Cordasco]

On Mon, Aug 15, 2016 at 2:09 PM, Donald Stufft  wrote:
> Hello!
>
> I'd like to restrict what folks can upload to PyPI in an effort to help narrow
> the scope down and to enable more a more consistent experience for everyone.
>
> First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I 
> think
> that we should try to get rid of support for most of these. Just for reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.
>
> First of all, when I say "remove", I mean disallow new uploads, but do not
> delete the existing files.
>
> Looking at each file type:
>
> I think that bdist_dumb is a pretty easy one to remove. It's format is such
> that it's basically not a very useful format to begin with. It takes the full
> path to the files and stores them in the repository. So If I install something
> to /opt/mycoolproject/lib/python3.5/site-packages/froblib/ then it will have
> paths that look like opt/mycoolproject/lib/python3.5/site-packages/froblib/...
> I think this is obviously not very useful and not many people have uploaded 
> any
> bdist_dumb files at all. They are also problematic because they have the same
> file extension as sdists, so pip doesn't really have a great way to
> differentiate between bdist_dumbs and sdists except by trying to guess if it
> contains one of distutils's adhoc platform tags.
>
> Looking at bdist_rpm, practically nobody has ever used it with a total of 45
> files ever uploaded for it. It's not super useful to be able to upload rpms to
> PyPI since it's not an RPM repository so people have to manually download them
> and then install them manually. It's also a bit weird to have support for RPMs
> but not for all of the other package formats that people might want.
>
> Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these 
> together
> because they're all OS specific installers for OSs that don't already have 
> some
> sort of repository. This lack of a repository format for them means that 
> random
> downloads are already the norm for people using these systems. For these, I
> think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
> are not very important to continue to support, but it would be weird to
> eliminate them without also elminating bdist_wininst. The wininst format has
> the most usage out of all of the seldom used formats, however when we look at
> the downloads for the last 30 days only 0.42% of the downloads were for 
> wininst
> files, so I think that it's pretty safe to remove them. I think in the past,
> these were more important due to the lack of real binary packages on Windows,
> but I think in 2016 we have wheel, and Wheel is a better solution. If however
> we want to keep them, then I think it's pretty safe to remove them from our
> /simple/ pages and any future repository pages and modify our mirroring 
> tooling
> to stop mirroring them. IOW, to treat them as some sort of "additional upload"
> rather than release uploads to PyPI.
>
> Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
> number of people still upload eggs, even though we have the wheel format.
> However, I think that we should (and generally do) consider eggs to be
> deprecated and while we don't want to break existing packages by removing 
> them,
> we should block further uploads for them. Looking again at the download 
> numbers
> eggs represented only 1.8% of total downloads in the last 30 days while wheels
> represented 41% and sdists represented 56%. Further more, I think we should do
> this with the expectation that any new repository API will no longer include
> egg files in them, and will just be sdists and wheels.
>
> Doing all of this would leave us with:
>
> * The /simple/ repository only having sdists, wheels, and eggs and disallowing
>   new eggs to be uploaded.
> * The hypothetical repository 2.0 api only having sdists and wheels.
> * *MAYBE* Having "related files" that people could upload things like
>   Windows/OSX Installers.
>
> On a related but different note, I'd also like to restrict the acceptable
> extensions for sdists. Currently distutils allows people to generate .tar,
> .tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly others. This
> is a bit problematic because each of those types requires a different set of
> optional libraries (which may or may not exist depending on Python version) so
> you can't really use a lot of them (for 

Re: [Distutils] license for setuptools

[Ian Cordasco]

Thanks for that Geoffrey. There's a PR to add it as Jason decided to
accept it. I think we can all relax now. Okay?

On Fri, Aug 12, 2016 at 8:00 AM, Geoffrey Spear  wrote:
> It seems a bit silly to claim that a license that contains the sentence "The
> above copyright notice and this permission notice shall be included in all
> copies or substantial portions of the Software" shouldn't have a copy
> included with the software it applies to since it literally says you need
> it.
>
> (IANAL)
>
> On Thu, Aug 11, 2016 at 10:45 PM, Eric Dill  wrote:
>>
>> Hi Claude,
>>
>> There was a recent discussion of the lack of a license file in setuptools
>> here: https://github.com/pypa/setuptools/issues/612 and another important
>> discussion here: https://github.com/pypa/setuptools/issues/132.  This is
>> probably the most relevant quotable bit from those two issues, from Jason
>> Coombs (the primary developer of setuptools:
>>
>> "The [License :: OSI Approved :: MIT License] classifier isn't a
>> suggestion but a declaration and follows the distutils guide for declaring a
>> license. I consider inclusion of a license file redundant and error prone."
>>
>> Hopefully this resolves your issue about not having an explicit license
>> file.
>>
>> Best,
>>
>> Eric
>>
>> On Thu, Aug 11, 2016 at 10:20 PM Marinier, Claude
>>  wrote:
>>>
>>> Good afternoon (well it’s afternoon here in the EDT zone),
>>>
>>>
>>>
>>> I am in the process of requesting the installation of Python 3 with
>>> matplotlib. The company needs to approve licenses but I cannot find the
>>> license for setuptools. The description here says it uses an MIT license but
>>> I cannot confirm this. On github, the file setup.py says the same thing.
>>>
>>>
>>>
>>> License :: OSI Approved :: MIT License
>>>
>>>
>>>
>>> Could the maintainer please add an explicit license file.
>>>
>>>
>>>
>>> I would really like to use matplotlib but will not get approval unless we
>>> can confirm the license.
>>>
>>>
>>>
>>> Thank you.
>>>
>>>
>>>
>>> --
>>>
>>> Claude Marinier
>>>
>>>
>>>
>>>
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] license for setuptools

[Ian Cordasco]

It seems that in our attempt to help Claude, both Glyph and I have
found: https://github.com/pypa/setuptools/issues/612

On Thu, Aug 11, 2016 at 11:11 PM, Glyph  wrote:
> The OP specifically asked about setuptools, which is on-topic. In large 
> corporate environments it is standard to have a process that evaluates every 
> single dependency, so setuptools does end up in the bucket with matplotlib 
> since you need it to install.
>
>> On Aug 11, 2016, at 8:36 PM, Noah Kantrowitz  wrote:
>>
>> Hi there, this list is for the discussion of Python's core packaging tools 
>> like distutils. We have no control over the packages made or distributed 
>> with them. You would have to contact the matplotlib authors, not us.
>>
>> --Noah
>>
>>> On Aug 12, 2016, at 7:51 AM, Marinier, Claude  
>>> wrote:
>>>
>>> Good afternoon (well it’s afternoon here in the EDT zone),
>>>
>>> I am in the process of requesting the installation of Python 3 with 
>>> matplotlib. The company needs to approve licenses but I cannot find the 
>>> license for setuptools. The description here says it uses an MIT license 
>>> but I cannot confirm this. On github, the file setup.py says the same thing.
>>>
>>> License :: OSI Approved :: MIT License
>>>
>>> Could the maintainer please add an explicit license file.
>>>
>>> I would really like to use matplotlib but will not get approval unless we 
>>> can confirm the license.
>>>
>>> Thank you.
>>>
>>> --
>>> Claude Marinier
>>>
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Python package inquiry

[Ian Cordasco]

Can you tell us which instructions you followed and what the exact output
of that command is? Also, what version of Python are you using (python -V)?

On Aug 9, 2016 6:40 PM, "Meradj Aghdam"  wrote:

> To whom it may concern,
>
> Hi,
>
> I am new to Python and I am struggling to install Python packages. I have
> followed the instructions however it does not seem to be working. I am
> trying to install the package through the Python(Command Line), the package
> is economics, and when I put
>
> python -m pip install Economics
>
> It does not work and the error that it is a syntax error. I am using
> Python 2.7. Would you mind kindly let me know how I can install packages?
>
> Your help if very much appreciated,
>
> Kind regards,
>
> Meradj
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] can someone explain why I have started to get these pip failures

[Ian Cordasco]

Try:

pip install --force-reinstall setuptools -U

On Thu, Jul 14, 2016 at 8:37 AM, Robin Becker  wrote:
>> (myenv) rptlab@app0:~/myenv
>> $ python
>> Python 2.7.11+ (default, Apr 17 2016, 14:00:29)
>> [GCC 5.3.1 20160413] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>
>
>
>> (myenv) rptlab@app0:~/myenv
>>
>> $ pip --version
>> pip 8.1.2 from /home/rptlab/myenv/local/lib/python2.7/site-packages
>> (python 2.7)
>
>
>
>> (myenv) rptlab@app0:~/myenv
>>
>>  pip install -r requirements.txt
>> Collecting MySQL-python<1.3,>=1.2.5 (from -r requirements.txt (line 1))
>>   Downloading MySQL-python-1.2.5.zip (108kB)
>> 100% || 112kB 2.1MB/s
>> Could not import setuptools which is required to install from a source
>> distribution.
>> Traceback (most recent call last):
>>   File "/home/rptlab/> (myenv)
>> rptlab@app0:~/myenv/local/lib/python2.7/site-packages/pip/req/req_install.py",
>> line 372, in setup_py
>> import setuptools  # noqa
>>   File "/home/rptlab/> (myenv)
>> rptlab@app0:~/myenv/local/lib/python2.7/site-packages/setuptools/__init__.py",
>> line 11, in 
>> from setuptools.extern.six.moves import filterfalse, map
>>   File "/home/rptlab/> (myenv)
>> rptlab@app0:~/myenv/local/lib/python2.7/site-packages/setuptools/extern/__init__.py",
>> line 1, in 
>> from pkg_resources.extern import VendorImporter
>> ImportError: No module named pkg_resources.extern
>
>> (myenv) rptlab@app0:~/myenv
>>
>> $ pip install setuptools -U
>> Requirement already up-to-date: setuptools in
>> ./lib/python2.7/site-packages
>
>
>
> This is on a unbuntu 16.04lts system
>
>> $ uname -a
>> Linux app0 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016
>> x86_64 x86_64 x86_64 GNU/Linux
>
>
> --
> Robin Becker
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Request for comment: Proposal to change behaviour of pip install

[Ian Cordasco]

On Sat, Jun 25, 2016 at 5:25 AM, Pradyun Gedam  wrote:
> Hello List!
>
> There is currently a proposal to change the behaviour to pip install to
> upgrade a package that is passed even if it is already installed.
>
> This behaviour change is accompanied with a change in the upgrade strategy -
> pip would stop “eagerly” upgrading dependencies and would become more
> conservative, upgrading a dependency only when it doesn’t meet lower
> constraints of the newer version of a parent. Moreover, the behaviour of pip
> install --target would also be changed so that --upgrade no longer affects
> it.
>
> A PEP-style write-up
> (https://gist.github.com/pradyunsg/4c9db6a212239fee69b429c96cdc3d73)
> documents the reasoning behind this proposed change, what the change is and
> some alternate proposals that were rejected in the discussions.
>
> This is a request for comments on the pull-request implementing this
> proposal (https://github.com/pypa/pip/pull/3806) for your views, thoughts
> and possible concerns related to it.

Having `pip install` implicitly upgrade a package will break the way
tooling like ansible, puppet, salt, and chef all work. Most expect
that if you do `pip install requests` twice you won't get two
different versions. At the very least, there should be an option added
that implies that if the version of the specified package is already
installed and satisfactory, that it is not upgraded.

That said, this will also break those tools understanding of how `pip
install --upgrade` works if instead `-U/--upgrade` is kept. People who
are automating deployments  using pip (perhaps in virtualenvs or
however else, it really doesn't matter) will now be forced to
periodically increase their lower bounds or list out *every*
dependency to ensure they upgrade if that's what they want rather than
being able to rely on pip to do that. They'll now have to specify
their requirements list in more than one place, and this will not be
something that's an improvement to the tooling in the community. Will
it make some things more explicit? Maybe. Will it cause people trying
to deploy software to waste hours of their time? Definitely.

If you want to change the upgrade behaviour of pip, I suggest not
modifying the existing `pip install` and instead deprecating that in
favor of a new command. Silently pulling the rug out from under people
seems like an incredibly bad idea.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Switch PyPA from IRC to Gitter or similar

[Ian Cordasco]

On Fri, Jun 10, 2016 at 4:58 PM, Glyph  wrote:
>
> On Jun 10, 2016, at 1:42 PM, Robert Collins 
> wrote:
>
> FWIW, I won't be switching to gitter or slack - I find the push
> notifications and always-on experience exhausting. It gets in the way
> of thinking or doing anything useful. I often leave my phone in
> priority-only mode to shut facebook and twitter and such things up for
> hours - or days - at a time.
>
>
> If you read my blog, you'll know this already – I also find Slack tiring in
> exactly this way.  I acknowledge the usability benefits over IRC, but the
> effect of those benefits is mostly just prompting me to spend more time
> staring at chat.  Which is not what I want to be doing.

It's in the service's best interest to constantly grab your attention
because it provides a false sense of dependence. It's why Gitter
spammed me, it's why Slack emails me everytime someone says @channel,
and it's why I've uninstalled Twitter from my phone (so I don't get 2
or more notifications for someone trying to interact with me there).
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Switch PyPA from IRC to Gitter or similar

[Ian Cordasco]

On Fri, Jun 10, 2016 at 12:54 PM, Nathaniel Smith <n...@pobox.com> wrote:
> On Jun 10, 2016 07:25, "Ian Cordasco" <graffatcolmin...@gmail.com> wrote:
>>
> [...]
>> I've tried using Gitter several times in the past. Unless they've
>> fixed their bugs related to sending me emails every day about activity
>> in a channel I spoke in once and left, I think they should be
>> eliminated.
>
> As a point of information, there's definitely a toggle you can flip to turn
> this off.

At the time I did flip that toggle and the only thing that worked was
removing my account from Gitter. If that toggle actually does
something now, they at least eventually fix bugs.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Switch PyPA from IRC to Gitter or similar

[Ian Cordasco]

On Fri, Jun 10, 2016 at 8:22 AM, Jason R. Coombs  wrote:
> In #pypa-dev, I raised the possibility of moving our PyPA support channels 
> from IRC to another hosted solution that enables persistence. Although IRC 
> has served us well, there are systems now with clear feature advantages, 
> which are crucial to my continuous participation:

I'm choosing not to read this as a threat.

> - always-on experience; even if one’s device is suspended or otherwise 
> offline.
> - mobile support — the in-cloud experience is essential for low power and 
> intermittently connected devices.
> - push notifications allow a project leader to remain largely inactive in a 
> channel, but attention raised promptly when users make a relevant mention.
> - continuous, integrated logging for catching up on the conversation.

So here's a question: Why are these crucial to you? You've explained
potential benefits but not why they're crucial to you and should be
crucial to anyone else.

Why do you need an "always-on experience"? Why do you feel required to
always be on? Do other people tell you that you need to always be on
chat?

Push notifications allow for prompt attention to mentions, but are all
mentions push notification worthy? Do we all need to be herded to
platforms that will spam us because someone mentioned us by nick or
name? I personally see this as a net negative. I don't need an email
or push notification to my phone because someone said my name in a
channel. That's a distraction. It prevents me from working on things
because it creates a false sense of alarm.

Continuous logging is on for #pypa and #pypa-dev as I understand it.
Surely it's not "integrated" into your chat client, but it's not as if
the logging doesn't exist.

> Both Gitter and Slack offer the experience I’m after, with Gitter feeling 
> like a better fit for open-source projects (or groups of them).

I've tried using Gitter several times in the past. Unless they've
fixed their bugs related to sending me emails every day about activity
in a channel I spoke in once and left, I think they should be
eliminated.

Slack has also had several outages lately that should also disqualify
it (besides the fact that it's incredibly closed source and will be
expensive to maintain logs in).

> I’ve tried using IRCCloud, and it provides a similar, suitable experience on 
> the same IRC infrastructure, with one big difference. While Gitter and Slack 
> offer the above features for free, IRCCloud requires a $5/user/month 
> subscription (otherwise, connections are dropped after two hours). I did 
> reach out to them to see if they could offer some professional consideration 
> for contributors, but I haven’t heard from them. Furthermore, IRCCloud 
> requires an additional account on top of the account required for Freenode.
>
> In addition to the critical features above, Gitter and Slack offer other 
> advantages:
>
> - For Gitter, single-sign on using the same Github account for authentication 
> and authorization means no extra accounts. Slack requires one new account.

IRC requires one new account.

> - An elegant web-based interface as a first-class feature, a lower barrier of 
> entry for users.

webchat.freenode.net may not be elegant, but it is first-class.

> - Zero-install or config.

Slack pesters you to install their desktop client and if you don't
want constant channel notifications you do have to configure it.
webchat.freenode.net offers no config.

> - Integration with source code and other systems.

Do you mean things like GitHub? GitHub already integrates with IRC.
What special kind of integration are do you think Gitter and Slack
have that GitHub's IRC integration doesn't?

> It’s because of the limitations of these systems that I find myself rarely in 
> IRC, only joining when I have a specific issue, even though I’d like to be 
> permanently present.
>
> Donald has offered to run an IRC bouncer for me, but such a bouncer is only a 
> half-solution, not providing the push notifications, mobile apps (IRC apps 
> exist, but just get disconnected, and often fail to connect on mobile 
> provider networks), or integrated logging.
>
> I note that both Gitter and Slack offer IRC interfaces, so those users who 
> prefer their IRC workflow can continue to use that if they so choose.

They're very poor IRC interfaces, making people who want to use a
simple, free, standard second class citizens (which is par for the
course as far as open source projects go).

> I know there are other alternatives, like self-hosted solutions, but I’d like 
> to avoid adding the burden of administering such a system. If someone wanted 
> to take on that role, I’d be open to that alternative.
>
> I’d like to propose we move #pypa-dev to /pypa/dev and #pypa to /pypa/support 
> in gitter.
>
> Personally, the downsides to moving to Gitter (other than enacting the move 
> itself) seem negligible. What do you think? What downsides am I missing?

With IRC we can run 

Re: [Distutils] Deprecating PyPI as an OpenID Provider

[Ian Cordasco]

On Tue, Jun 7, 2016 at 2:59 PM, Donald Stufft  wrote:
> Longer term, Python should get a centralized Sign on across all it’s web 
> properties, which could re-implement this feature for folks who want it.

I agree. This is should be a goal for the PSF. This would help with
elections and a number of other things as well. I think there are a
few solutions like FreeIPA if anyone on this list is interested in
looking into helping the PSF implement this.

Cheers,
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi upload fails with TypeError

[Ian Cordasco]

Source distributions are .tar.gz files. I don't know what sdist option
you're using with twine.
On Jun 2, 2016 12:47 AM, "Luí­s de Sousa" 
wrote:

> Yes Chris, that is  correct. But note that I am using the sdist option
> both with setuptools and twine.
>
> Thank you,
>
> Luís
>
> *Sent from ProtonMail , encrypted email based in
> Switzerland.*
>
>
>  Original Message 
> Subject: Re: [Distutils] PyPi upload fails with TypeError
> Local Time: June 1, 2016 10:03 PM
> UTC Time: June 1, 2016 8:03 PM
> From: chris.jerdo...@gmail.com
> To: luis.de.so...@protonmail.ch
> CC: graffatcolmin...@gmail.com,Distutils-Sig@python.org
>
> It looks like you are telling it "dist/hex-utils-0.2.sdist" but the
> directory contains "hex-utils-0.2.tar.gz".
>
> --Chris
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi upload fails with TypeError

[Ian Cordasco]

There's a warning in your output. Is there anything in dist/ ?
On Jun 1, 2016 10:09 AM, "Luí­s de Sousa" 
wrote:

> Hi again Ian,
>
> Installing setuptools for python 3 dealt away with that error but the
> upload is still failing. I am not sure this is related to twine or not, but
> I'll leave here the log in any case, someone might suggest something.
>
> Thank you,
>
> Luís
>
> $ python3 setup.py sdist
> running sdist
> running egg_info
> creating hex_utils.egg-info
> writing hex_utils.egg-info/PKG-INFO
> writing top-level names to hex_utils.egg-info/top_level.txt
> writing dependency_links to hex_utils.egg-info/dependency_links.txt
> writing entry points to hex_utils.egg-info/entry_points.txt
> writing manifest file 'hex_utils.egg-info/SOURCES.txt'
> reading manifest file 'hex_utils.egg-info/SOURCES.txt'
> writing manifest file 'hex_utils.egg-info/SOURCES.txt'
> warning: sdist: standard file not found: should have one of README,
> README.rst, README.txt
>
> running check
> creating hex-utils-0.2
> creating hex-utils-0.2/hex_utils
> creating hex-utils-0.2/hex_utils.egg-info
> making hard links in hex-utils-0.2...
> hard linking setup.py -> hex-utils-0.2
> hard linking hex_utils/__init__.py -> hex-utils-0.2/hex_utils
> hard linking hex_utils/asc.py -> hex-utils-0.2/hex_utils
> hard linking hex_utils/asc2hasc.py -> hex-utils-0.2/hex_utils
> hard linking hex_utils/grid.py -> hex-utils-0.2/hex_utils
> hard linking hex_utils/hasc.py -> hex-utils-0.2/hex_utils
> hard linking hex_utils/hasc2gml.py -> hex-utils-0.2/hex_utils
> hard linking hex_utils/surfaceSimple.py -> hex-utils-0.2/hex_utils
> hard linking hex_utils.egg-info/PKG-INFO ->
> hex-utils-0.2/hex_utils.egg-info
> hard linking hex_utils.egg-info/SOURCES.txt ->
> hex-utils-0.2/hex_utils.egg-info
> hard linking hex_utils.egg-info/dependency_links.txt ->
> hex-utils-0.2/hex_utils.egg-info
> hard linking hex_utils.egg-info/entry_points.txt ->
> hex-utils-0.2/hex_utils.egg-info
> hard linking hex_utils.egg-info/top_level.txt ->
> hex-utils-0.2/hex_utils.egg-info
> Writing hex-utils-0.2/setup.cfg
> creating dist
> Creating tar archive
> removing 'hex-utils-0.2' (and everything under it)
>
> $ twine upload dist/hex-utils-0.2.sdist
> Uploading distributions to https://pypi.python.org/pypi
> ValueError: Cannot find file (or expand pattern):
> 'dist/hex-utils-0.2.sdist'
>
> $ ls -la dist
> total 12
> drwxrwxrwx 1 root root  176 Jun  1 19:04 .
> drwxrwxrwx 1 root root 4096 Jun  1 19:04 ..
> -rwxrwxrwx 1 root root 6091 Jun  1 19:04 hex-utils-0.2.tar.gz
>
>
>
> *Sent from ProtonMail , encrypted email based in
> Switzerland.*
>
>
>  Original Message 
> Subject: Re: [Distutils] PyPi upload fails with TypeError
> Local Time: 24 May 2016 8:16 PM
> UTC Time: 24 May 2016 18:16
> From: graffatcolmin...@gmail.com
> To: luis.de.so...@protonmail.ch
> CC: berker.pek...@gmail.com,Distutils-Sig@python.org
>
> Luis, it looks like you're running twine on Python 3 and setuptools is
> installed for Python 2. Try doing:
>
> python3 -m pip install setuptools
>
> or
>
> apt-get install -y python3-setuptools
>
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi upload fails with TypeError

[Ian Cordasco]

Luis, it looks like you're running twine on Python 3 and setuptools is
installed for Python 2. Try doing:

python3 -m pip install setuptools

or

apt-get install -y python3-setuptools

On Tue, May 24, 2016 at 1:13 PM, Luí­s de Sousa
 wrote:
> Hi there Ian. Twine is also failling, apparently it can not find setup
> tools, please check the log below.
>
> Cheers.
>
> $ twine upload dist/hex-utils-0.2.sdist
> Traceback (most recent call last):
>   File "/usr/bin/twine", line 9, in 
> load_entry_point('twine==1.5.0', 'console_scripts', 'twine')()
>   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 542,
> in load_entry_point
> return get_distribution(dist).load_entry_point(group, name)
>   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line
> 2569, in load_entry_point
> return ep.load()
>   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line
> 2229, in load
> return self.resolve()
>   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line
> 2235, in resolve
> module = __import__(self.module_name, fromlist=['__name__'], level=0)
>   File "/usr/lib/python3/dist-packages/twine/__main__.py", line 20, in
> 
> from twine.cli import dispatch
>   File "/usr/lib/python3/dist-packages/twine/cli.py", line 19, in 
> import setuptools
> ImportError: No module named 'setuptools'
>
> $ pip install setuptoolsRequirement already satisfied (use --upgrade to
> upgrade): setuptools in /usr/lib/python2.7/dist-packages
>
> $ dpkg -l | grep setuptools
> ii python-setuptools 20.7.0-1 all Python Distutils Enhancements
>
>
>
>  Original Message 
> Subject: Re: [Distutils] PyPi upload fails with TypeError
> Local Time: 20 May 2016 8:22 PM
> UTC Time: 20 May 2016 18:22
> From: graffatcolmin...@gmail.com
> To: berker.pek...@gmail.com
> CC: luis.de.so...@protonmail.ch,Distutils-Sig@python.org
>
> Until then, try using twine (https://github.com/pypa/twine). You'll
> have to make your sdist with `python setup.py sdist` and then upload
> it with `twine upload dist/mypackage.sdist`. But twine will prompt you
> when it can't find a credential for you instead of proceeding onward
> nobly.
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi upload fails with TypeError

[Ian Cordasco]

Until then, try using twine (https://github.com/pypa/twine). You'll
have to make your sdist with `python setup.py sdist` and then upload
it with `twine upload dist/mypackage.sdist`. But twine will prompt you
when it can't find a credential for you instead of proceeding onward
nobly.

On Fri, May 20, 2016 at 1:18 PM, Berker Peksağ  wrote:
> On Fri, May 20, 2016 at 9:12 PM, Berker Peksağ  
> wrote:
>> On Fri, May 20, 2016 at 9:00 PM, Luí­s de Sousa
>>  wrote:
>>>
>>> The TypeError is about the *last* line in the traceback: One of
>>> `self.username' or 'self.password' is set to 'None'.
>>>
>>>
>>> That being the case, how can I correct the bug? Must I upgrade setuptools?
>>> Or some other package?
>>
>> Is there a .pypirc file in your $HOME directory? If there is one, can
>> you compare its content with the example at
>> https://docs.python.org/3/distutils/packageindex.html#pypirc ?
>
> There is an open issue about this on bugs.python.org:
> http://bugs.python.org/issue18454
>
> I will try to fix it at PyCon US sprints.
>
> --Berker
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ian Cordasco]

The PEP has been accepted by Nick. Let's turn our thoughts to more
productive topics rather than talking about a moot point.

Cheers, Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Does pip Honour "Obsoletes"?

[Ian Cordasco]

On Thu, May 12, 2016 at 5:40 PM, Phil Thompson
 wrote:
> I may be doing something wrong, but...
>
> The METADATA in my wheel uses Obsoletes but pip does not remove the obsoleted 
> package on an install of my wheel.
>
> Is it supposed to?

No.

Unlike some other package managers, pip doesn't keep track of every
package that depends on another, so it will not remove the obsoleted
package unless you specifically do `pip uninstall obsoleted-package`.
This prevents silent/unintentional breakage.

Cheers,
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Ian Cordasco]

On Tue, May 10, 2016 at 8:47 PM, Donald Stufft <don...@stufft.io> wrote:
>
>> On May 10, 2016, at 9:43 PM, Ian Cordasco <graffatcolmin...@gmail.com> wrote:
>> I think "build-system" is more descriptive and the more descriptive we
>> can be, the better. (Think of choosing descriptive method and
>> attribute names as well as variables.)
>
> I’m in favor of “build”, mostly because I think
>
> [package.build-system]
> requires = [“setuptools”, “wheel”]
>
> is uglier than

Oh, absolutely it's ugly, but reading it aloud is what makes me prefer
it. "The package's build-system requires setuptools, and wheel"
Granted, Ruby is the perfect example of trying to make something read
perfectly like English being a terrible idea when taken to extremes,
but I don't think that will happen here.

Other than that, the PEP looks fine. I'm anti-TOML but for reasons
that don't deserve being brought up on the mailing list. It's the best
option of a bunch of bad options. The rest of the PEP looks great.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Ian Cordasco]

On Tue, May 10, 2016 at 8:24 PM, Nathaniel Smith  wrote:
> On Tue, May 10, 2016 at 5:39 PM, Brett Cannon  wrote:
>> Donald, Nathaniel, and I have finished our proposed PEP for specifying a
>> projects' build dependencies. The PEP is being kept at
>> https://github.com/brettcannon/build-deps-pep, so if you find spelling
>> mistakes and grammatical errors please feel free to send a PR to fix them.
>
> Thanks Brett!
>
>> The only open issue in the PEP at the moment is the bikeshedding topic of
>> what to name the sub-section containing the requirements: `[package.build]`
>> or `[package.build-system]` (we couldn't reach consensus among the three of
>> us on this).
>
> To maybe help nudge initial bikeshedding on this in useful directions,
> the main arguments (IIUC) were:
>
> In favor of "build-system": setup.py is used for more than just the
> strict "build" (source tree/sdist -> wheel) phase. For example,
> setup.py is also used to do VCS checkout -> sdist. And it seems likely
> that the new build system abstraction thing will grow similar
> capabilities at some point. So calling the section just "build" might
> be misleading.
>
> In favor of "build": it's just shorter and reads better.
>
> Maybe there's a third option that's even better -- [package.automation] ?
>
> Maybe it doesn't matter that much :-)

I think "build-system" is more descriptive and the more descriptive we
can be, the better. (Think of choosing descriptive method and
attribute names as well as variables.)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Ian Cordasco]

On Tue, May 3, 2016 at 12:10 PM, Paul Moore  wrote:
> On 3 May 2016 at 17:47, Donald Stufft  wrote:
>> It will likely get decided as part of the build system PEP, whenever that
>> gets picked up again.
>
> Yes, but on 15th March
> (https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html)
> Robert posted
>
>> Just to set expectations: this whole process seems stalled to me; I'm
>> going to context switch and focus on things that can move forward.
>> Someone please ping me when its relevant to put effort in again :).
>
> And I think that's right. The whole build system PEP issue appears
> stalled from a lack of someone willing (or with the time) to make a
> call on the approach we take.
>
> As far as I'm aware, the decision remains with Nick. With the possible
> exception of Donald's proposal (which AFAIK never got formally
> published as a PEP) everything that can be said on the other proposals
> has been said, and the remaining differences are ones of choice of
> approach rather than anything affecting capabilities. (Robert's
> message at 
> https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
> summarised the state of the 3 proposals at the time).
>
> I think this is something that should be resolved - we don't appear to
> be gaining anything by waiting, and until we have a decision on the
> approach that's being taken, we aren't going to get anyone writing
> code for their preferred option.
>
> Nick - do you have the time to pick this up? Or does it need someone
> to step up as BDFL-delegate? Robert, Nathaniel, do you have time to
> spend on a final round of discussion on this, on the assumption that
> the goal will be a final decision at the end of it? Donald, do you
> have the time and interest to complete and publish your proposal?
>
> Paul


I was following that PEP and going to implement it in Twine for the
PyPA. If it would help, I can help Nick with this process. I read both
PEPs a while ago and I think updates have been made so I'd need to
read them again, but I can probably make some time for this.

--
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Name arbitration on PyPI (was: The mypy package)

[Ian Cordasco]

On Mon, Apr 18, 2016 at 4:16 PM, Chris Barker  wrote:
> You've made a strong case, and this is probably not where PyPi should go --
> but it would hardly be a disaster:
>
>>
>> The idea of expiring out names has been brought up recently to resolve an
>> issue of two packages, one popular and large; another someone's weekend
>> project.
>
>
> The issue here is not that it's a weekend project, but that it may be an
> abandoned project. I don't think anyone suggest that we should have a
> popularity or quality test to see who gets to trump whom with name
> allocation -- I sure didn't.
>
> Which is quite relevant to below:
>
>>
>> 1.  PyYAML is a package that would be de-registered in such a scheme.  It
>> is a highly used, extremely popular, package that unserializes text into
>> arbitrary python objects.  It is a trusted package... and one that hasn't
>> been active in ages.
>
>
> and you don't think ANYONE would be willing to take on the miniscule amount
> of work to maintain the name? Plus there would be any number of other
> schemes for determining whether a project name is abandoned.

I have in fact offered but the author refuses to accept help from
anyone. They're also the author of the C library (libyaml) and they do
not maintain that either. It's actually quite frustrating as someone
who wants to fix some of the numerous bugs in the library + improve it
and add support for YAML 1.2 which is years old at this point.

>>
>> 2. the package tooling already assumes that names will always point to
>> one, and only one package.  ever.  until the heat death of the universe or
>> the death of the language whichever is first.
>
>
> IIUC, the current scheme allows for a name to be "taken over" by a new
> package if the original author so desires -- i.e. if the current owner of
> the mypy name was happy to relinquish it, then "pip install mypy" would get
> users something totally different 6 months from now. So no -- we don't
> currently guarantee anything about future use of names. Other that that the
> original author can do whatever they want with it.
>
>> 3. Who in the PSF really wants that bureaucratic nightmare of arbitrating
>> cases when this inevitably messes up, be this system manual or automatic?
>
>
> I think bureaucratic nightmare is pretty hyperbolic, but yes, there would be
> some overhead, for sure. And given that no one has the time and motivation
> to even maintain PyPi at this point -- this will probably kill the idea
> altogether.
>
>
>> To the specifics of the mypy-lang package that brought this up... It's
>> like naming your company "Yahoo", and getting upset that yahoo.com is
>> getting a bump in traffic because of your popularity.
>
>
> Again - this is not about minor weekend projects not be "important". It's
> about potential abandonware -- with domain registration, "Yahoo" can offer
> to buy the domain from the current holder, or, if the current owner has
> abandoned it, it will go into the public pool again when they stop paying to
> maintain the registration.
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip install -e foo (without Repo URL)

[Ian Cordasco]

On Apr 14, 2016 2:20 AM, "Thomas Güttler" 
wrote:
>
> I think it would be very cool if you could install a package editable
> without the repo-url.
>
> The default repo-url can be defined in the meta-data of the package.
>
> Background: I came across this becaus saltstack prefers the branch
"develop", but
> most other repos use the branch "master".
>
> Yes, this is no big problem. I figured out the right repo url soon.
>
> Next use case: you use software third-party-foo-lib in your project.
> Up to now you use it as package. You find a bug and want to fix it.
> Wouldn't it be great if you could just type "pip install -e
third-party-foo-lib"
> and now you are read to create pull requests?

I don't understand how this makes me ready to submit pull requests. Can you
explain a little more?

> What do you think?
>
> Regards,
>   Thomas Güttler
>
>
>
> --
> Thomas Guettler http://www.thomas-guettler.de/
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Python on AIX 7.1

[Ian Cordasco]

On Sun, Mar 20, 2016 at 7:35 PM, Gene Fontanilla
 wrote:
> Can anyone guide me in installing Python AIX? i have no experience in AIX.

This mailing list is about distributing python packages, not python
itself. You might have better luck asking on the general python-list.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Ian Cordasco]

On Fri, Jan 22, 2016 at 12:10 PM, Donald Stufft  wrote:
> PEP 376 added a file to the .dist-info directory called "INSTALLER" which was
> supposed to be:
>
> This option is the name of the tool used to invoke the installation.
>
> However, nothing has really ever implemented it and it's gone largely ignored
> until just recently pip 8.0 started writing the INSTALLER file into the
> metadata directories with a value of "pip".
>
> I'd like to propose adding a special cased value to add to the installer file
> that will tell projects like pip that this particular installed thing is being
> managed by someone else, and we should keep our hands off of it. According to
> PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I think
> since nobody has ever implemented it, we could expand that so that it so you
> can also have a special value, of "dpkg (system)" or maybe that's not worth it
> and we could just have "system" as a special value.

Why not use something like JSON to allow us to have a little more
information about the installer, e.g.,

{"name": "pip", "system": true, ...}

> The benefit of doing this, is that with a special value in that file that says
> "this file belongs to the OS", then pip could start looking for that file and
> require a --force flag before it modifies any files belonging to that project.
> Then distributors like Debian, Fedora, etc could simply write out the 
> INSTALLER
> file with the correct value, and pip would start to respect their files by
> default.
>
> Thoughts? Does this sound reasonable to everyone? Do we think it needs a new
> PEP or can we just amend PEP 376 to add this extra bit of information?

I think amending the PEP makes sense.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why github and bitbucket?

[Ian Cordasco]

On Thu, Nov 5, 2015 at 1:27 PM, Thomas Güttler
<guettl...@thomas-guettler.de> wrote:
> Am 04.11.2015 um 21:14 schrieb Ian Cordasco:
>> As I understand it, some people prefer Mercurial. Those projects tend
>> to live on bitbucket. Git projects can live in either place although I
>> suspect they tend to live on GitHub instead.
>
> Is there really a need for this?

Is there a need for letting project creators work as they please with
the VCS they prefer? Why not if it makes them more efficient
maintainers.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why github and bitbucket?

[Ian Cordasco]

On Thu, Nov 5, 2015 at 2:53 PM, Thomas Güttler
<guettl...@thomas-guettler.de> wrote:
> Am 05.11.2015 um 20:35 schrieb Ian Cordasco:
>> On Thu, Nov 5, 2015 at 1:27 PM, Thomas Güttler
>> <guettl...@thomas-guettler.de> wrote:
>>> Am 04.11.2015 um 21:14 schrieb Ian Cordasco:
>>>> As I understand it, some people prefer Mercurial. Those projects tend
>>>> to live on bitbucket. Git projects can live in either place although I
>>>> suspect they tend to live on GitHub instead.
>>>
>>> Is there really a need for this?
>>
>> Is there a need for letting project creators work as they please with
>> the VCS they prefer? Why not if it makes them more efficient
>> maintainers.
>
> If the projects are unrelated then every maintainer should use what
> he prefers. If the projects are related and maintained by one group (PyPa),
> then there should be **one** hosting platform. - my opinion -

That's super helpful. Thanks
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why github and bitbucket?

[Ian Cordasco]

As I understand it, some people prefer Mercurial. Those projects tend
to live on bitbucket. Git projects can live in either place although I
suspect they tend to live on GitHub instead.

On Wed, Nov 4, 2015 at 2:07 PM, Thomas Güttler
 wrote:
> From: http://python-packaging-user-guide.readthedocs.org/en/latest/glossary/
>
>> Python Packaging Authority (PyPA)
>> PyPA is a working group that maintains many of the relevant projects in 
>> Python packaging. They maintain a site at https://www.pypa.io, host projects 
>> on github and bitbucket, and discuss issues on the pypa-dev mailing list.
>
> Why are there pypa on github and bitbucket?
>
> Is one the master and the other the mirror?
>
> Or does one host part A and the other hosts part B?
>
> Regards,
>   Thomas Güttler
>
>
>
> --
> http://www.thomas-guettler.de/
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Please don't impose additional barriers to participation

[Ian Cordasco]

On Wed, Oct 28, 2015 at 3:31 AM, M.-A. Lemburg  wrote:
> On 28.10.2015 06:02, Ben Finney wrote:
>> Marcus Smith  writes:
>>
>>> 1) *Please*, *please*, *please* let's start doing PEP conversations as
>>> PRs to pypa/interoperability-peps : )
>>
>> Please keep the conversation on a mailing list where one can participate
>> without needing to sign up with any particular service provider.
>>
>> Your proposal would have the effect of excluding people from the
>> conversation if they don't agree to have a GitHub account. I think it's
>> valuable to avoid that barrier to entry, needing only an email account.
>
> I agree with Ben. Discussions on PEPs need to happen on mailing lists,
> not hidden away on some issue tracker or PR ticket.

Others may be willing to tolerate your FUD, but without concrete
reasons against GitHub (other than zomg it's a proprietary service) I
don't see a reason to not use the pull request flow on an open
repository that is free for people to clone, fork, contribute to, etc.

GitHub isn't my preferred hosting platform for git but it is the
defacto standard and it's workflow ubiquitous, documented, and far
more user-friendly than mailing list threads (especially when they
devolve into ideology wars).

Also nothing precludes mailing list discussions, so without details
about your objections, I don't see why this should be held up.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] README.rst vs DESCRIPTION.rst

[Ian Cordasco]

On Sun, Sep 27, 2015 at 1:18 PM, Ionel Cristian Mărieș
 wrote:
>
> On Sun, Sep 27, 2015 at 8:05 PM, Thomas Güttler
>  wrote:
>>
>>
>> OK, the sample project is not the definitive guide line.
>> Where can I find the definitive guide line?
>
>
> I don't think there can be a "definitive guide line". Unlike the core
> language the packaging part of Python is a messy soup of different and often
> competing ideas, styles and tools.
>
> So you cannot have an definitive or objective guide for something that's
> subjective in nature.
>
> About the README vs DESCRIPTION - ask yourself, what would you use README
> for then? I believe that's absolutely nothing. You only need one. :-)

I tend to disagree. Your project's long description doesn't need
detailed instructions on how to start contributing, reporting bugs,
etc. That should be in your README and documentation/project website.
I don't think that having two separate files is a problem. You could
even use the include directive from reStructuredText so that your
README includes your description without duplicating the content.

I haven't previously followed this guideline, but I think I'm going to
start. I quite like it.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] README.rst vs DESCRIPTION.rst

[Ian Cordasco]

On Sun, Sep 27, 2015 at 4:55 PM, Ionel Cristian Mărieș
<cont...@ionelmc.ro> wrote:
>
> On Mon, Sep 28, 2015 at 12:20 AM, Ian Cordasco <graffatcolmin...@gmail.com>
> wrote:
>>
>> Your project's long description doesn't need
>> detailed instructions on how to start contributing, reporting bugs,
>> etc.
>
>
> What would you put in README then?
>
> For contribution/bucktracker guide CONTRIBUTING.rst/md is a better place -
> GitHub loves it. But who cares, no one uses GitHub nowdays :)

Ah, the patented sarcasm that's always so helpful.

The README is typically (harkening back to pre-GitHub days) where one
lays out the relevant information for other developers and for users
in one easy to find place. Looking for the contributing guidelines? Go
read the CONTRIBUTING file. Looking for how to build the project on X
operating system? Go read X file. When well designed, this can still
be the front page of a project for the websites that display it as
such (GitHub, BitBucket, GitLab, etc.). Take for example:
https://gitlab.com/cryptsetup/cryptsetup/blob/master/README.md. It's
description lives there, sure, but there's a lot more information
there than that.

All of that said, as Paul said, it's a guideline. Just like PEP-0008.
Follow it, or don't. There's not much point in arguing this further.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Table of contents formatting in PyPI pages generated from long descriptions

[Ian Cordasco]

If I remember correctly the readme project is now used to render that
information. Does that project support rendering TOCs? If not support for
that may need to be added.
On Jul 28, 2015 8:01 AM, Jim Fulton j...@jimfulton.info wrote:

 I like to include tables of contents in my distribution long descriptions.
 Normally. ReST formats these with links, so that when someone clicks on
 and entry in the TOC, they jump to that position in the document.

 Recently (last month?) PyPI stopped displaying TOCs with links.
 Was this intentional?

 Jim

 --
 Jim Fulton
 http://jimfulton.info
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[Ian Cordasco]

On Mon, Jul 6, 2015 at 11:24 AM, Antoine Pitrou solip...@pitrou.net wrote:

 Hello,

 I'm having the following errors when trying to upload a wheel to PyPI.
 (yes, the version number is off - but that's besides the point here,
 I think):

 $ twine upload dist/llvmlite-0.0.0-py2.py3-none-linux_x86_64.whl
 /home/antoine/.local/lib/python3.4/site-packages/pkginfo/installed.py:53: 
 UserWarning: No PKG-INFO found for package: pkginfo
   warnings.warn('No PKG-INFO found for package: %s' % self.package_name)
 Uploading distributions to https://pypi.python.org/pypi
 Uploading llvmlite-0.0.0-py2.py3-none-linux_x86_64.whl
 HTTPError: 400 Client Error: Binary wheel for an unsupported platform

 $ twine upload dist/llvmlite-0.0.0-py3-none-linux_x86_64.whl
 /home/antoine/.local/lib/python3.4/site-packages/pkginfo/installed.py:53: 
 UserWarning: No PKG-INFO found for package: pkginfo
   warnings.warn('No PKG-INFO found for package: %s' % self.package_name)
 Uploading distributions to https://pypi.python.org/pypi
 Uploading llvmlite-0.0.0-py3-none-linux_x86_64.whl
 HTTPError: 400 Client Error: Binary wheel for an unsupported platform

Unrelated to your problem here (which I think is due in part to the
lack of classifiers for binaries for *nix systems (or something along
those lines that Nick or Donald will be more familiar with)), I filed
a twine bug for those warnings:
https://github.com/pypa/twine/issues/114

I've never seen those before, so I'm sorry for the noise in that output.

Cheers,
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI and Uploading Documentation

[Ian Cordasco]

On Fri, May 15, 2015 at 8:48 AM, Donald Stufft don...@stufft.io wrote:
 Hey!

 First, for anyone who isn't aware we recently migrated PyPI and TestPyPI so
 that instead of storing files and documentation locally (really in a glusterfs
 cluster) it will store them inside of S3. This will reduce maintenance 
 overhead
 of running PyPI by two servers since we'll no longer need to run our own
 glusterfs cluster as well as improve the reliaiblity and scalability of the
 PyPI service as a whole since we've had nothing but problems from glusterfs in
 this regard.

 One of the things that this brought to light was that the documentation
 upload ability in PyPI is something that is not often used* however it
 represents something which is one of our slowest routes. It's not a well
 supported feature and I feel that it's going outside of the core competancy 
 for
 PyPI itself and instead PyPI should be focused on the files themselves. In
 addition since the time this was added to PyPI a number of free services or
 cheap services have came about that allow people to sanely upload raw document
 without a reliance on any particular documentation system and we've also had
 the rise of ReadTheDocs for when someone is using Sphinx as their 
 documentation
 system.

 I think that it's time to retire this aspect of PyPI which has never been well
 supported and instead focus on just the things that are core to PyPI. I don't
 have a fully concrete proposal for doing this, but I wanted to reach out here
 and figure out if anyone had any ideas. The rough idea I have currently is to
 simply disable new documentation uploads and add two new small features. One
 will allow users to delete their existing documentation from PyPI and the 
 other
 would allow them to register a redirect which would take them from the current
 location to wherever they move their documentation too. In order to prevent
 breaking documentation for projects which are defunct or not actively
 maintained we would maintain the archived documentation (sans what anyone has
 deleted) indefinetely.

 Ideally I hope people start to use ReadTheDocs instead of PyPI itself. I think
 that ReadTheDocs is a great service with heavy ties to the Python community.
 They will do a better job at hosting documentation than PyPI ever could since
 that is their core goal. In addition there is a dialog between ReadTheDocs and
 PyPI where there is an opportunity to add integration between the two sites as
 well as features to ReadTheDocs that it currently lacks that people feel are a
 requirement before we move PyPI's documentation to read-only.

 Thoughts?

 * Out of ~60k projects only ~2.8k have ever uploaded documentation. It's not
   easy to tell if all of them are still using it as their primary source of
   documentation though or if it's old documentation that they just can't
   delete.


 ---
 Donald Stufft
 PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


I'm +1 on reducing the responsibilities of PyPI so it can act as an
index/repository in a much more efficient manner. I'm also +1 on
recommending people use ReadTheDocs. It supports more than just Sphinx
so it's a rather flexible option. It's also open source, which means
that anyone can contribute to it.

I'm curious to hear more about integrations between PyPI and
ReadTheDocs but I fully understand if they're not concrete enough to
be worthy of discussion.

--
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] it's happened - wheels without sdists (flit)

[Ian Cordasco]

On Mon, Mar 30, 2015 at 9:46 AM, Daniel Holth dho...@gmail.com wrote:

 The approach doesn't exclude the possibility of a source distribution,
 it's only a few weeks old. I would suggest that if you have to choose
 between having a setup.py, not having a setup.py, and not having the
 package on pypi at all because the packager can't figure out setup.py,
 prefer the second option.


Is figuring out setup.py still a thing? Between cookiecutter laying out
your project with a setup.py and the Packaging Guide, how many people still
have trouble setting up a setup.py for a package? It almost seems like this
is a solution wanting for a problem.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] it's happened - wheels without sdists (flit)

[Ian Cordasco]

On Mon, Mar 30, 2015 at 10:18 AM, Daniel Holth dho...@gmail.com wrote:

 setup.py as implemented with distutils/setuptools has a bit of a
 Goldilocks problem: it's just right for a medium-complexity project
 but when your project is very simple it's too hard, and when you get
 to the point where you are trying to extend distutils by writing a
 10,000 line extension, yikes. So it's fantastic to be able to just
 avoid distutils entirely if it isn't the right size for your project.
 This example, flit, does not invoke any code from distutils,
 setuptools or bdist_wheel to do its thing.

 A source release could just be an archive of the repository.


You still have not answered how reading flit's source code to get it
working is better than using cookiecutter to generate a project, and using
`python setup.py bdist_wheel sdist` (which is well-documented, and has tons
of answered questions on sites like StackOverflow to help in case of a
problem).
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] it's happened - wheels without sdists (flit)

[Ian Cordasco]

On Mon, Mar 30, 2015 at 10:23 AM, Ionel Cristian Mărieș cont...@ionelmc.ro
wrote:


 On Mon, Mar 30, 2015 at 6:05 PM, Ian Cordasco graffatcolmin...@gmail.com
 wrote:

 In other words, no one should read the docs because that's a waste of
 time? Because a lot of time has been poured into the packaging docs and if
 they're not sufficient, then instead of improving them, people should write
 undocumented tools that force people to read the source? I'm not sure how
 that's better than what we already have.


 ​A waste of time? No. But it sure is poor investment of time for newbie
 users. It's a futile struggle to document (or read about) all the features
 of distutils and setuptools, not because it can't be documented, but
 because there's too much ground to cover and PyPA's packaging.python.org
 approach is to ​just give an overview of what's available and avoid giving
 any real best practice recommendations as much as possible. There may be
 good reasons for that but that's not a sensible approach to giving users a
 pitfall free learning path.


So for new python programmers (or newbie users in general) reading the
entire source of another package to understand it is a better experience?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] it's happened - wheels without sdists (flit)

[Ian Cordasco]

On Mon, Mar 30, 2015 at 9:59 AM, Daniel Holth dho...@gmail.com wrote:

 Yes, setup.py should die. Flit is one example, and you can understand
 it not by copy/pasting, but by spending half an hour reading its
 complete source code.


In other words, no one should read the docs because that's a waste of time?
Because a lot of time has been poured into the packaging docs and if
they're not sufficient, then instead of improving them, people should write
undocumented tools that force people to read the source? I'm not sure how
that's better than what we already have.

Further, how does flit deal with C-extensions? How does flit deal with
C-extensions distributed on Linux? How does it generate the appropriate
wheels for that?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] it's happened - wheels without sdists (flit)

[Ian Cordasco]

On Mon, Mar 30, 2015 at 11:15 AM, Ionel Cristian Mărieș cont...@ionelmc.ro
wrote:



 On Mon, Mar 30, 2015 at 6:55 PM, Ian Cordasco graffatcolmin...@gmail.com
 wrote:

 Then, when the new user goes to publish it, there's tons of prior
 documentation on how to do it. If they run into problems using flit they
 have the skimpy documentation or the source.


 ​Now it might have skimpy docs and no users, but that's largely a product
 of time. I think `flit` should be judged on what it can be in the future,
 not all what it's right now. To put it in picture, the argument you're
 making is like comparing the amazon rainforest to a banana milkshake
 recipe.​


Well to make a better comparison, we're discussing a healthy diet to a
sugar laced treat. The healthy diet is sustainable because it is well
documented and has tons of users with experience with it, but has pitfalls
in that it can be expensive at times, while the sugar laced treat is good
for a quick blood sugar spike but will leave you thoroughly unsatisfied and
eventually wanting for the healthy diet.

New users, (like some people who prefer high sugar diets) may prefer the
initial simplicity, but at the cost of having to do a lot more work up
front. Those who follow a healthy diet (and ostensibly exercise regiment)
will have tooling to not have to worry about how to construct a healthy
diet. New users who find and work with those already on a healthy diet will
learn the tools that help them avoid the pitfalls of starting on a healthy
diet (e.g., tools that generate setup.py for you and maintain the 98% use
case, which is inevitably where new users fall*) and will be better off for
the long term.

* Note, new users still has yet to be defined by anyone advocating that
this is better for new users because of mystical reasons (like being able
to read the source code).
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] it's happened - wheels without sdists (flit)

[Ian Cordasco]

On Mon, Mar 30, 2015 at 11:14 AM, Daniel Holth dho...@gmail.com wrote:

 On Mon, Mar 30, 2015 at 11:55 AM, Ian Cordasco
 graffatcolmin...@gmail.com wrote:
 
 
  On Mon, Mar 30, 2015 at 10:47 AM, Ionel Cristian Mărieș 
 cont...@ionelmc.ro
  wrote:
 
  On Mon, Mar 30, 2015 at 6:39 PM, Xavier Fernandez
  xav.fernan...@gmail.com wrote:
 
  I think the point was not to say that documentation is useless (and
 there
  is some: http://flit.readthedocs.org/en/latest/ ) but that the
  code/implementation is much simpler than the combination of
  distutils/setuptools/bdist_wheel.
 
  On Mon, Mar 30, 2015 at 5:26 PM, Ian Cordasco
  graffatcolmin...@gmail.com wrote:
 
 
  So for new python programmers (or newbie users in general) reading the
  entire source of another package to understand it is a better
 experience?
 
  To put that in context, flit goes for less than 600 SLOC while
  distutils+setuptools+wheel amount to over 2 SLOC. At that ratio
  arguments for distutils+setuptools+wheel documentation seem
 unreasonable.
 
 
  To be clear, no one should ever be advocating to just read the source
 as a
  form of documentation. This is why the Packaging guide exists (because no
  one should ever be expected to read the distutils, setuptools, or wheel
  source to use it).
 
  Code is never as self-documenting as people like to believe. And since
 we're
  talking about new users (without defining what they're new to) reading
 the
  source should only be for educational purposes. cookiecutter will serve
 new
  users better than flit or anything else. cookiecutter will teach new
 users
  good package structure and take care of the (possibly hard parts) of a
  setup.py. Then, when the new user goes to publish it, there's tons of
  prior documentation on how to do it. If they run into problems using flit
  they have the skimpy documentation or the source.
 
  Yeah, it's easy to read 600 SLOC for you, but what about for some new
  user? Are they new to python? Why do they have to care about reading the
  source if something else will just work as documented for their
 simple
  use case?

 No one has advocated reading the source code instead of reading the
 documentation.


Thankfully this is a publicly archived list. Quoting yourself:

 Flit is one example, and you can understand it not by copy/pasting,
 but by spending half an hour reading its complete source code.

In which you advocate reading the source of a tool over using setup.py
which has countless resources written about it on the internet.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] it's happened - wheels without sdists (flit)

[Ian Cordasco]

On Mon, Mar 30, 2015 at 10:47 AM, Ionel Cristian Mărieș cont...@ionelmc.ro
wrote:

 On Mon, Mar 30, 2015 at 6:39 PM, Xavier Fernandez xav.fernan...@gmail.com
  wrote:

 I think the point was not to say that documentation is useless (and there
 is some: http://flit.readthedocs.org/en/latest/ ) but that the
 code/implementation is much simpler than the combination of
 distutils/setuptools/bdist_wheel.

 On Mon, Mar 30, 2015 at 5:26 PM, Ian Cordasco graffatcolmin...@gmail.com
  wrote:


 So for new python programmers (or newbie users in general) reading the
 entire source of another package to understand it is a better experience?

 ​
 ​To put that in context, flit goes for less than 600 SLOC while
 distutils+setuptools+wheel amount to​ over 2 SLOC. At that ratio
 arguments for distutils+setuptools+wheel documentation seem unreasonable.


To be clear, no one should ever be advocating to just read the source as
a form of documentation. This is why the Packaging guide exists (because no
one should ever be expected to read the distutils, setuptools, or wheel
source to use it).

Code is never as self-documenting as people like to believe. And since
we're talking about new users (without defining what they're new to)
reading the source should only be for educational purposes. cookiecutter
will serve new users better than flit or anything else. cookiecutter will
teach new users good package structure and take care of the (possibly hard
parts) of a setup.py. Then, when the new user goes to publish it, there's
tons of prior documentation on how to do it. If they run into problems
using flit they have the skimpy documentation or the source.

Yeah, it's easy to read 600 SLOC for you, but what about for some new
user? Are they new to python? Why do they have to care about reading the
source if something else will just work as documented for their simple
use case?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Granting permissions to Xavier Fernandez and Marc Abramowitz?

[Ian Cordasco]

On Fri, Mar 6, 2015 at 5:56 AM, Piotr Dobrogost
p...@2015.forums.dobrogost.net wrote:
 Hi

 As an external observer of pip project at github I see two men, namely
 Xavier Fernandez (https://github.com/xavfernandez) and Marc Abramowitz
 (https://github.com/msabramo) with many valuable contributions. I
 think it would be beneficial if they had been granted some more
 permissions to the project/repo.

 Regards,
 Piotr Dobrogost
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

It seems incredibly inappropriate for anyone except an existing core
to be proposing new cores.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Implementing large changes in small increments

[Ian Cordasco]

On Fri, Mar 6, 2015 at 2:17 AM, anatoly techtonik techto...@gmail.com wrote:
 Stealing some packaging code from Go and tweaking may not be that bad idea. =)
 At least they use code review system to let new people learn and old
 people share.

Yep and most of the people working on Go are paid to do so. Sharing
information through code review is necessary inside a corporation and
worth being fired over if you don't.

 On Fri, Mar 6, 2015 at 5:51 AM, Ian Cordasco graffatcolmin...@gmail.com 
 wrote:
 Wait, I have an idea. Let's rewrite pip in Rust! ;)

 On Thu, Mar 5, 2015 at 7:16 PM, Greg Ewing greg.ew...@canterbury.ac.nz 
 wrote:
 On 03/06/2015 11:06 AM, Ben Finney wrote:

 That's “small changes only, otherwise your change gets rejected”, of
 course.


 Yes, otherwise submitting a patch that replaces the entire source
 code of Python with Ruby would be a sure-fire way to get it
 accepted. :-)

 --
 Greg


 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig



 --
 anatoly t.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Implementing large changes in small increments

[Ian Cordasco]

Has PyPA considered contacting GitHub support? I'm happy to do the
same since I've wanted this for a while myself on other projects.

On Fri, Mar 6, 2015 at 5:15 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 6 March 2015 at 23:01, Donald Stufft don...@stufft.io wrote:
 Tooling wise, Github PRs work well for us. I don’t (and I don’t believe that
 any of the other core devs) have any major issues with them.

 Github issues on the other hand, they function “OK” but it would be nice to
 have something that we can allow anyone to modify the state of tickets to
 help with triage. However even this isn’t a super pressing concern because
 our ticket count is small enough that I don’t think there’s likely to be too
 many to be handled by people commenting on issues and a core team coming in
 to change things. However if someone has a proposal for a different issue
 tracker (and plans for how to migrate to it), personally I’d be willing to
 listen.

 I'm also fine with github. I don't have an issue with the issue
 tracker, although as Donald says it would be helpful if it had a
 concept of tracker privileges separate from core committer. But
 that's *not* a big enough concern to me that I'd want to go to a
 different tool.

 Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Implementing large changes in small increments

[Ian Cordasco]

On Fri, Mar 6, 2015 at 4:04 PM, Nick Coghlan ncogh...@gmail.com wrote:

 On 6 Mar 2015 22:10, Ben Finney ben+pyt...@benfinney.id.au wrote:

 Nick Coghlan ncogh...@gmail.com writes:

  CPython uses the Reitveld instance integrated with bugs.python.org,
  and has the same problem as pip: incremental changes are a pain to
  publish, review, and merge, so we review and accept monolithic patches
  instead (cf the problem statement in
  https://www.python.org/dev/peps/pep-0462/)

 Fair enough. I don't know of a good code review tool for Mercurial.

 I'd like to ensure Kallithea fits that bill, but the actual work on that
 seems to mostly be driven by the folks at Unity3D at the moment.

 In the meantime, Phabricator is a decent choice if you just want to use an
 existing GitHub independent tool that works with either git or Mercurial.
 pip adopting that workflow would also be a good proof of concept for
 Donald's proposal to also adopt that workflow for CPython (or at least its
 support repos).

  While the main UI is very busy, I've actually quite liked my own
  experience with Gerrit for http://gerrit.beaker-project.org/

 My understanding is that Gerrit makes it tedious to review a sequence of
 revisions, in proportion to the number of revisions in the sequence.

 When the goal is to break a change up into small, independently reviewable
 changes that's generally a feature rather than a defect :)

 If
 I understand correctly, such a sequence must have separate reviews for
 every revision, and an aggregate of all the changes is not available to
 the reviewer.

 Correct, but my understanding is that when using it in tandem with GitHub,
 there's nothing stopping you from also submitting a PR if a reviewer wants
 an all-inclusive view.

 I'm impressed by GitLab's code review tool UI; see an example at
 URL:https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/344/diffs.
 The merge request page has tabs for the discussion, the commit log, and
 the overall diff – and you choose from inline diff or side-by-side diff.

 GitLab is free software, including all its tools; anyone can set up a
 GitLab instance and the project data can move from one instance to
 another without loss. For the purposes of the past thread where some
 proposed migrating to the proprietary lock-in site GitHub, those
 objections don't exist with GitLab: a project can migrate to a different
 host and keep all the valuable data it accumulated.

 A move to GitLab would be unobjectionable, in my view. That it has good
 code review features would help the issues in this thread too.

 It doesn't have the integration with other services and the low barriers to
 contribution that are the main reasons a lot of projects prefer GitHub.

 Of course, when your problem is already we're receiving more contributions
 than we can process effectively, deciding to require a slightly higher
 level of engagement in order to submit a change for consideration isn't
 necessarily a bad thing :)

 If anyone knows of equivalent hosting for Mercurial with equivalent code
 review tools under free-software terms with no lock-in, that would be
 even better I think.

 That's what I'd like forge.python.org to eventually be for the core Python
 ecosystem, but we don't know yet whether that's going to be an entirely
 self-hosted Kallithea instance (my preference) or a Phabricator instance
 backed by GitHub (Donald's preference).

 Hence my suggestion that a forge.pypa.io Phabricator instance might be an
 interesting thing to set up and start using for pip. Donald's already done
 the research on that in the context of
 https://www.python.org/dev/peps/pep-0481/ and for pip that's a matter of
 just add Phabricator without having to migrate anything (except perhaps
 the issues if folks wanted to do that).

 Cheers,
 Nick.


 --
  \ “Don't be misled by the enormous flow of money into bad defacto |
   `\standards for unsophisticated buyers using poor adaptations of |
 _o__) incomplete ideas.” —Alan Kay |
 Ben Finney

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


I'm fairly concerned that what has turned into a how can we increase
the feedback received for people submitting pull requests has turned
into a bike shed moment for using F/LOSS tooling instead of GitHub
when the cores who actually work on the project have already expressed
a disinterest in moving and a satisfaction with GitHub.

GitLab's UI would do nothing to improve review management.

Phabricator, while nice, again adds yet another layer to the piece for
new contributors to involve themselves in. GitHub is one monolith and
closed source (and a company with culture problems) but that doesn't
change the fact 

Re: [Distutils] Getting more momentum for pip

[Ian Cordasco]

Anatoly,

We already ruled out AppVeyor
On Mar 6, 2015 2:11 AM, anatoly techtonik techto...@gmail.com wrote:

 On Thu, Mar 5, 2015 at 11:11 PM, Ian Cordasco
 graffatcolmin...@gmail.com wrote:
  And for CI, we need people who will help with the windows CI solution
  on more than one front clearly.

 https://ci.appveyor.com/ works for open source projects.


 --
 anatoly t.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting more momentum for pip

[Ian Cordasco]

On Fri, Mar 6, 2015 at 5:55 AM, Paul Moore p.f.mo...@gmail.com wrote:
 On 5 March 2015 at 16:38, Marc Abramowitz msabr...@gmail.com wrote:
 This makes me think that the folks who review the PRs are overburdened
 and/or the process needs a bit more structure (e.g.: each person thinks
 someone else is going to review the PR and so no one does it).

 One thing I, personally, find difficult when reviewing PRs
 (specifically feature requests) is the fact that I usually don't
 actually have a *need* for the functionality being proposed. It's very
 easy for me to say this doesn't help me personally, so I'll ignore
 it, but that is ducking a big part of the responsibility of being a
 core committer. But forming a view on something I've no experience of
 or direct interest in is *hard*, and takes a lot of time. Discussions
 tend to involve a lot of people with strong opinions (e.g. the PR
 author) who can't move the change forward, and a few people with
 weaker opinions (e.g. me :-)) who can. It's very easy to think just
 accept it because it helps someone. But that's a cop-out and
 long-term isn't a sustainable approach.

+1. This is a problem I have with Flake8. People keep asking for more
command-line arguments because It's just one more option. It won't
hurt anyone. But Flake8 is another project without a great set of
tests. It would be easy to say Yeah sure, just this one other option
that only one person has ever asked for but there's only ever one
person reviewing pull requests - me. It's also not sustainable to keep
adding poorly named command-line flags.

 It's not thinking someone else will review the PR, it's more making
 a conscious decision on how much energy and effort I'm willing to put
 into a PR that doesn't have any benefit for me. (And even just
 *discussing* a PR can be a lot of energy, it's not easy to politely
 explain to someone that you don't think their use case, that they went
 to a lot of trouble writing a PR for, isn't worth it).

 What would help a *lot* is some sort of agreement on what pip is, and
 what its core goals are. Something similar to what it means to be
 pythonic for the Python language itself. At the moment, I don't
 think this is very clearly understood even within the core dev group
 (so external contributors have no hope...) And for me, it'd help avoid
 the endless debates that often start with the phrase pip should...

+10

 For example, is the lack of a programmable API an issue for pip? I
 think it is, and having people able to write their own tools that use
 pip's finder, or its wheel installer, is a (long term) goal for pip,
 rather than, say, continually adding more pip subcommands. But I don't
 know if that's the consensus. And to my knowledge, no 3rd party PRs
 have *ever* been of the form Encapsulate pip's functionality X in a
 clean API so I can use it in my script...

If pip is ever refactored appropriately (which I acknowledge is not a
trivial condition to meet), maybe then pip could consider presenting a
public API, but I think there are currently too many people who
already reach into pip to ignore the need for such an interface.
Perhaps the answer is, as pip is refactored, to create libraries that
are then vendored into pip and that people can install independently
to do that one thing they need to do.

 Or is the pip search command a wart that should be removed because
 it isn't pip's job to do PyPI searches? There's some low-hanging fruit
 if a more focused tool is the goal...

There's also how many different replacements for pip search on PyPI?

 Or should pip give you tools to replicate your current environment
 (pip freeze, requirements files)? What about remove anything *not* in
 this requirement file? Personally, I only use requirements files to
 bundle up install this lot of stuff. I don't write the sort of thing
 where a pin every dependency philosophy is appropriate, so freeze
 isn't something I use. But lots of people do, so what's the workflow
 that pip freeze supports?

 The problem with discussing this sort of thing is that it's *very*
 wide-ranging, and tends to produce huge rambling mega-threads[1] when
 discussed in a public list. I'm not advocating any sort of private
 cabal deciding the fate of pip, but maybe somewhere where the core
 devs could agree their *own* opinions before having to face the public
 wouldn't be such a bad thing. That's more or less what I'd expected
 the pypa-dev list to be (as a parallel to the python-dev list) but it
 doesn't feel like it's turned out that way, maybe because it doesn't
 have a clear enough charter, or maybe because there's no obvious
 *other* place to direct people to for off-topic posts (like
 python-list is for python-dev).

So sometimes private cabals need to be made in order to get a basis of
what is reasonable. The WSGI working group tried to do that but that
failed after about a week as more people tried to join the cabal and
were allowed to do so.

 Or maybe grand designs are a 

Re: [Distutils] Getting more momentum for pip

[Ian Cordasco]

On Thu, Mar 5, 2015 at 1:55 PM, Marc Abramowitz msabr...@gmail.com wrote:
 Yeah my changes are quite trivial. And there is much more complex stuff that
 could be improved. IIRC, `pip/req/req_install.py` is the real behemoth. I
 remember feeling afraid to touch that thing. :)

 I do think this illustrates some of the problem though in that if those 3
 very simple PRs are not merged or closed, then I don't have a lot of faith
 that any more complex PR that I might submit would be worth the time
 investment.

 I feel like I'm somewhat conditioned to only submit small PRs to pip because
 they have the lowest risk (although also the lowest reward of course).

The lowest reward to whom? If it's a good change that fixes something
(regardless of the size that's a big reward to the project and its
users.

I'm also not going to go back in the thread to reply all of the
messages, but I want to be clear that I didn't mean every feature
should be rejected. Just that for a project as critical to an
ecosystem like Python's, rejecting features should be fairly easy to
do and very simple. If no current core developer wants to maintain it
or feels strongly about it, that should be closed. A nice, already
written, form explanation would probably antagonize contributors less
than a short reasoning that might come off curt. That said, you'll
always lose some contributors by closing their pull requests or by
trying to help them get it into a good state.

In my opinion, gigantic PRs that refactor things should be rejected
immediately. Those can almost certainly always be done in smaller,
easier to review, and easier to understand pull requests.
Mega-refactors that will take hours (or even days) to review that
modify all of the tests should be absolutely out of the question if
pip is going to come up with better guidelines. requests has received
a handful of them, and we've tried to coach those people (who are
often first-time contributors) into breaking it up but they always
leave, and the project has to realize that sometimes it's an
acceptable loss.

Having clear guidelines is great, they should be enforced and they
should be linked somewhere, like the top of the README.

Regarding empowering people to close, label, and triage issues is
going to be much harder. There are only two systems I can think of
that allow for this: Launchpad and Trac. Trac is Django's tracker (in
case people aren't aware) and Django has found a way to integrate it
with GitHub. We /could/ take that approach, but that leaves the
following questions:

- Who is going to maintain the server(s)?
- Who is going to provision them initially?
- What happens if those people (ideally it's more than just one
person) disappear? Will we have enough people to reduce the bus
factor?
- Will this ever become yet another thing that the core developers
have to spend time on instead of reviewing pull requests and fixing
bugs?

Regarding auto-closing, please don't do it. Especially for the ones
where someone didn't file a bug first, it may be the only way to
figure out what's going on?

And for CI, we need people who will help with the windows CI solution
on more than one front clearly. I think pyca/cryptography has OS X
builders on Travis, so we could probably add tests for that, but for
RHEL that's going to be much harder. I think this is where Marc's
reference to OpenStack fits in perfectly though. In OpenStack there's
a concept of third party CI. (There's a similar notion with the
buildbots for CPython.) The people who register those CI systems
maintain them but the project determines whether or not that system's
failing build should count against a change or not. GitHub allows for
multiple statuses to be set on a commit/PR from multiple services.
Thoughts?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Implementing large changes in small increments

[Ian Cordasco]

Wait, I have an idea. Let's rewrite pip in Rust! ;)

On Thu, Mar 5, 2015 at 7:16 PM, Greg Ewing greg.ew...@canterbury.ac.nz wrote:
 On 03/06/2015 11:06 AM, Ben Finney wrote:

 That's “small changes only, otherwise your change gets rejected”, of
 course.


 Yes, otherwise submitting a patch that replaces the entire source
 code of Python with Ruby would be a sure-fire way to get it
 accepted. :-)

 --
 Greg


 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting more momentum for pip

[Ian Cordasco]

On Thu, Mar 5, 2015 at 10:38 AM, Marc Abramowitz msabr...@gmail.com wrote:
 This post is meant to start a discussion on how to make sure that important
 PyPA projects like pip get enough eyes so that they can continually move
 forward. It is absolutely NOT meant to be critical of folks who volunteer
 their time to work on these projects, as I have the utmost respect for folks
 who do this often thankless work. And since the word thankless just popped
 into my head, let me say thank you right now to the folks who contribute
 to PyPA.

 I've noticed that when PRs are submitted to pip (by myself and by others),
 they often languish for a while and then sometimes become unmergeable
 because of conflicts.

 This makes me think that the folks who review the PRs are overburdened
 and/or the process needs a bit more structure (e.g.: each person thinks
 someone else is going to review the PR and so no one does it).

 So some ideas off the top of my head - please comment and/or add your own
 suggestions to the list:

 - Add more committers - this will ostensibly increase the number of folks
 reviewing so that the turnaround time is decreased. And takes some pressure
 off.

 - Obtain some kind of funding so that committers can be compensated for
 their work and don't feel as bad about spending time on it. These people
 have day jobs and families so maybe this would help; maybe not.

 - Introduce some bot or automation that periodically reminds about open PRs
 that are getting old or PRs that have become unmergeable, etc. Or maybe for
 each PR, it picks one or more persons who is responsible for reviewing that
 PR, so there is no ambiguity about who is responsible. The OpenStack folks
 have quite a bit of structure in their workflow (probably too much for PyPA
 projects which have less people working on them?), but perhaps there are
 some things that can be borrowed. In particular, changes need a certain
 amount of upvotes to be merged and reviewers usually request feedback from
 certain individuals.

 So I guess my suggestions boil down to:

 - Add more humans
 - Add more money to make humans more efficient
 - Add more computer automation

 #3 seems most appealing to me, but of course it requires humans to develop
 it in the first place, but at least it's an investment that could pay
 dividends.

 Thoughts?

Personally I value stability over a tendency to merge every PR. I know
you're not advocating that every PR be merged (or at least I hope
you're not), but the only way to add more core developers is to have
people who are already volunteering to review other people's pull
requests. Core developer is a title that is not so much a privilege
as it is a responsibility and it should only be given to those who are
already volunteering time to contribute to pip and other PyPA projects
through multiple efforts (e.g., reviewing other people's code - not
just sending their own, supporting the project either on
StackOverflow, irc, or somewhere else, etc.)

For the short time that I subscribed to pip notifications I noticed a
lot of PRs and a lot of pings but very little serious critical review.
Most PRs in that period of time seemed to be feature PRs and not bug
fixes. Personally, Bug Fixes are more important than Features and not
ever new feature deserves to be merged, in fact, I would say probably
more projects would do better by rejecting most new features.

OpenStack's workflow is incredibly different from most other projects
that aren't people's day jobs for a few reasons:

1. OpenStack is a globally distributed effort. Each project has
hundreds of contributors and most only have 5-10 core reviewers who
tend to be nominated only after a lot of sweat has been poured into a
project.
2. No one can commit directly to a repository.
3. There's a lot of automation around tests but there's also a lot of
ceremony in review
4. New features are strongly vetted through a blueprint and
specification process that more core reviewers contribute to but do
not drive (hence why people who are core on that process are called
drivers)
5. There's a much clearer review system in place where votes actually
matter and are aggregated
6. People (like myself) are literally paid to work on OpenStack. Few
people work on it in their free time except to keep their job
7. pip is (in my opinion) far more stable than OpenStack and I would
like to keep it that way. I know there are OpenStack Infrastructure
folks who monitor this list, but OpenStack continuously reaches into
private areas of projects and when those change, OpenStack tends to
get angry at upstream for not having backwards compatibility in
non-public APIs. pip on the other hand doesn't do that.

I'm sure Donald needs help. I don't think the right kind of help now
is adding more people who can merge things arbitrarily. I think the
right kind of help needs to be focused towards stability and we need a
better way of defining what new features belong in pip and what don't
to reduce the volume 

Re: [Distutils] bootstrap.pypa.io is down

[Ian Cordasco]

If bootstrap.pypa.io sits behind fastly, a 503 would indicate that it
couldn't reach the server. That means that it is either exactly what Nick
pointed out or a semi-normal error from fastly. There's a reason pip
retries package downloads when it sees a 503 response status code.
On Mar 3, 2015 5:30 AM, Nick Coghlan ncogh...@gmail.com wrote:

 On 3 March 2015 at 19:30, Reinout van Rees rein...@vanrees.org wrote:
  Hi,
 
  I'm getting a 503 service unavailable on https://bootstrap.pypa.io/
 
  This effectively kills all bootstrap.py scripts of our buildouts :-(
 
 
  Where is the best place to report this, normally? I looked at the pypi
  bug tracker but didn't see a mention of pypa.io there.

 We don't currently have a great ticket tracking system for the PSF
 infrastructure - we should probably work out something better, so
 folks don't have to play guess the project, and also so operational
 issues can be more clearly separated from the development projects
 that power the online services.

 Emailing infrastruct...@python.org is a decent last resort option,
 though, including for the PyPA parts.

 In this case, it looks like it's back now, so it may have just been
 affected by the Rackspace rolling restarts earlier
 (https://status.python.org/incidents/5w523lrn3587)

 Regards,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] bootstrap.pypa.io is down

[Ian Cordasco]

Heh. That wasn't apparent to me at all. But yeah, the 503 still
indicates that fastly had trouble reaching the server. It's almost
certainly Rackspace's restarts that caused this.

On Tue, Mar 3, 2015 at 8:09 AM, Reinout van Rees rein...@vanrees.org wrote:
 Ian Cordasco schreef op 03-03-15 om 14:52:

 If bootstrap.pypa.io sits behind fastly, a 503 would indicate that it
 couldn't reach the server. That means that it is either exactly what Nick
 pointed out or a semi-normal error from fastly. There's a reason pip retries
 package downloads when it sees a 503 response status code.

 Well, it wasn't a temporary one-second hickup. It was down for a couple of
 hours this morning. (morning = morning in Europe :-) ).


 Reinout

 Reinout

 --
 Reinout van Rees  http://reinout.vanrees.org/
 rein...@vanrees.org   http://www.nelen-schuurmans.nl/
 Learning history by destroying artifacts is a time-honored atrocity
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parsing requirements, pip has no API ...

[Ian Cordasco]

pip is a command-line tool. The fact that you can import it doesn't
make it a library. If it documents no public API then it has none and
you shouldn't be relying on things that you can import from pip. You
can import requests from pip but you shouldn't do that either.

There seems to be a need for a separate library for this use case but
there isn't at the moment. In general, requirements.txt seems to be an
anti-pattern. You either have to use likely to break tooling or you'll
have to reinvent that from scratch. You're better off putting it
directly in setup.py and using setup.py to install dependencies in a
virtualenv instead of requirements.txt

On Fri, Feb 13, 2015 at 3:27 PM, Thomas Güttler
guettl...@thomas-guettler.de wrote:
 I was told:

 {{{
 Pip does not have a public API and because of that there is no backwards 
 compatibility contract. It's impossible to fully parse every type of 
 requirements.txt without a session so either parse_requirements needs to 
 create one if it doesn't (which means if we forget to pass in a session 
 somewhere it'll use the wrong one) or it needs one passed in.
 }}}
 From https://github.com/pypa/pip/issues/2422#issuecomment-74271718


 Up to now we used parse_requirements() of pip, but in new versions you need 
 to pass in a
 session.

 If I see changes like this:

 setup.py
 - install_requires=[str(req.req) for req in 
 parse_requirements(requirements.txt)],
 + install_requires=[str(req.req) for req in 
 parse_requirements(requirements.txt, session=uuid.uuid1())],

  ... I think something is wrong.


 I am not an expert in python packaging details. I just want it to work.

 What is wrong here?

   - You should not use parse_requirements() in setup.py
   - pip should not change its API.
   - you should not use pip at all, you should use ...?

 Regards,
   Thomas


 --
 http://www.thomas-guettler.de/
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parsing requirements, pip has no API ...

[Ian Cordasco]

On Fri, Feb 13, 2015 at 3:59 PM, Marcus Smith qwc...@gmail.com wrote:

 In general, requirements.txt seems to be an
 anti-pattern. You either have to use likely to break tooling or you'll
 have to reinvent that from scratch. You're better off putting it
 directly in setup.py and using setup.py to install dependencies in a
 virtualenv instead of requirements.txt


 I don't know your context for calling it an anti-pattern, but there are
 valid use cases for requirements.txt vs install_requires.
 here's what the Python Packaging User Guide has on the distinction

 https://packaging.python.org/en/latest/requirements.html

 skipping to the distinctions, it lists four:

 Whereas install_requires defines the dependencies for a single project,
 Requirements Files are often used to define the requirements for a complete
 python environment.

 Whereas install_requires requirements are minimal, requirements files often
 contain an exhaustive listing of pinned versions for the purpose of
 achieving repeatable installations of a complete environment.

 Whereas install_requires requirements are “Abstract”, requirements files
 often contain pip options like --index-url or --find-links to make
 requirements “Concrete”. [1]

 Whereas install_requires metadata is automatically analyzed by pip during an
 install, requirements files are not, and only are used when a user
 specifically installs them using pip install -r.


In 90% of the cases I see, requirements.txt are used to define the
requirements for the project to function which typically are the exact
same requirements necessary when installing the project. People also
will then write a test-requirements.txt (or dev-requirements.txt) file
to have a complete development environment. In this case, Thomas seems
to be using requirements.txt to define the requirements necessary when
installing the software they're developing.

If requirements.txt were used solely for a development environment,
they would look more like

| .
| dev-requirement-1=0.1
| # etc.

Instead they seem to be used more to define the same requirements
someone would define in install_requires. This is the anti-pattern I'm
talking about.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 440 on PyPI

[Ian Cordasco]

On Sun, Jan 25, 2015 at 10:43 AM, M.-A. Lemburg m...@egenix.com wrote:
 On 25.01.2015 16:34, Donald Stufft wrote:

 On Jan 25, 2015, at 9:32 AM, M.-A. Lemburg m...@egenix.com wrote:

 I think you ought to make a more prominent announcement on
 c.l.p, c.l.p.a and perhaps a distutils blog (if there is one).

 What is c.l.p and c.l.p.a?

 That's short for comp.lang.python and comp.lang.python.announce,
 which are gateway'ed to mailing lists:

 https://mail.python.org/mailman/listinfo/python-list
 https://mail.python.org/mailman/listinfo/python-announce-list

 There is no distutils blog.

 Given how quickly things change, I think this would be a good
 way of getting the word out to a wider audience than the
 distutils mailing list.

I've only been on this list for about a year. This PEP (and others
like it) has been in motion for quite a while. I think the blog would
miss far more people than the mailing list would and I'm not sure I
agree with how quickly things change. There doesn't seem to be much
content for the blog and it seems like something that would just
become neglected. What topics do you really think would be better
suited for a blog post than a message to here and related announcement
lists?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 440 on PyPI

[Ian Cordasco]

On Sun, Jan 25, 2015 at 10:52 AM, M.-A. Lemburg m...@egenix.com wrote:
 On 25.01.2015 17:46, Ian Cordasco wrote:
 On Sun, Jan 25, 2015 at 10:43 AM, M.-A. Lemburg m...@egenix.com wrote:
 On 25.01.2015 16:34, Donald Stufft wrote:

 On Jan 25, 2015, at 9:32 AM, M.-A. Lemburg m...@egenix.com wrote:

 I think you ought to make a more prominent announcement on
 c.l.p, c.l.p.a and perhaps a distutils blog (if there is one).

 What is c.l.p and c.l.p.a?

 That's short for comp.lang.python and comp.lang.python.announce,
 which are gateway'ed to mailing lists:

 https://mail.python.org/mailman/listinfo/python-list
 https://mail.python.org/mailman/listinfo/python-announce-list

 There is no distutils blog.

 Given how quickly things change, I think this would be a good
 way of getting the word out to a wider audience than the
 distutils mailing list.

 I've only been on this list for about a year. This PEP (and others
 like it) has been in motion for quite a while. I think the blog would
 miss far more people than the mailing list would and I'm not sure I
 agree with how quickly things change. There doesn't seem to be much
 content for the blog and it seems like something that would just
 become neglected. What topics do you really think would be better
 suited for a blog post than a message to here and related announcement
 lists?

 The blog posts could automatically get copied over to
 Twitter and mailing lists using e.g. IFTTT, and it would
 be possible to subscribe using RSS/Atom feeds.

 The advantage of a blog is having news entries persist and be
 easy to find, while at the same time simplifying the whole
 publishing process.

 Anyway, just a suggestion.

I still don't understand the benefit. You could just as easily have
the mailing list mirrored to a blog if we felt it would help but
distutils-sig has public archives and (I think) is indexed by Google.
Donald, Paul, Nick, Richard, Jason, and everyone else whose name I'm
forgetting are already working a lot on PyPI, packaging, pip,
setuptools, etc. I don't see the benefit in maintaining the blog on
top of their other responsibilities. Maybe if someone else steps up to
be an editor and take mailing list posts and turn them into blog
posts, that'd be cool, but I don't see the value in making them
maintain yet another thing on top of what they do.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Closing the Delete File + Re-upload File Loophole.

[Ian Cordasco]

On Sat, Jan 24, 2015 at 11:38 AM, Donald Stufft don...@stufft.io wrote:

 On Jan 24, 2015, at 12:37 PM, John Anderson son...@gmail.com wrote:



 On Saturday, January 24, 2015, Donald Stufft don...@stufft.io wrote:

 I've pushed changes to PyPI where it is no longer possible to reuse a
 filename
 and attempting to do it will give an 400 error that says:

 This filename has previously been used, you should use a different
 version.

 This does NOT prevent authors from being allowed to delete files from
 PyPI,
 however if a file is deleted from PyPI it cannot be re-uploaded again.
 This
 means that if you upload say foobar-1.0.tar.gz, and your 1.0 has a mistake
 in
 it then you *must* issue a new release to correct it.

 ---
 Donald Stufft
 PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


 My only concern is that there is no reliable way to test that your README
 will be parsed correctly. Is there a timeline for switch it to use
 https://github.com/pypa/readme?

 I would say majority of the time I do a release of the same version it's
 because of the fragile rst parsing.

 If I have to run the risk of bumping versions just to fix a valid
 restructured text document to fit pypi parsing it'll make releasing a very
 stressful experience.


 You can re-run register as many times as you want which is all you need to
 adjust the README.

 ---
 Donald Stufft
 PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


.post{N} releases are also a good way of fixing this in the package
(assuming you want the most current and correct version of the README
to be what the user downloads). The .post{N} part of PEP440 is
semantically for build errors in a package where no other changes to
the package have been made. I think this qualifies as a use case.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: 1.7 vs =1.7

[Ian Cordasco]

I personally think that 1.7.1 matching 1.7 muddies some applications
of it being used with date-based versions with this pep also supports.
This (as best I can tell) means that now 2014.09.31 will match 
2014.09 which seems like a rather terrible idea. No one expects a date
to be padded with 0s. I'm also fully against special casing date-based
versions because the whole point of 440 was to make versioning
consistent and reliable and I wholeheartedly want that.

On Sun, Dec 28, 2014 at 1:43 PM, Chris Jerdonek
chris.jerdo...@gmail.com wrote:
 On Sun, Dec 28, 2014 at 1:20 PM, Marcus Smith qwc...@gmail.com wrote:

 
  * 1.7.1 matches 1.7 (previously it did not)

 This sounds like a straight up bug fix in the packaging module to me - the
 PEP 440 zero padding should apply to *all* checks, not just to equality
 checks, as you can't sensibly compare release segments with different
 numbers of elements.

 OK.  to be clear, I guess you really didn't follow the previous thread?
 I specifically raised the concern over 1.7.1 not matching 1.7 (in the
 current implementation), but most people were arguing it was a logical
 interpretation of PEP440.

 I think Nick's e-mail clarifies it for me.

 In my e-mail, I was reconciling the current behavior with the current
 wording of the PEP, which says, Exclusive ordered comparisons are
 similar to inclusive ordered comparisons, except that the comparison
 operators are  and  and the clause MUST be effectively interpreted
 as implying the prefix based version exclusion clause != V.*.

 I now see that the wording is a bit ambiguous (or at least that I was
 misinterpreting it).  I interpreted it to mean that prefix-based
 version exclusion should be used *instead* of zero-padding, whereas
 with Nick's e-mail, I see that the meaning is that prefix-based
 exclusion should be used *after* applying zero padding.

 The clarified interpretation also addresses an asymmetry of the
 previously mentioned (and now apparently incorrect) series
 interpretation, which I'm not sure was mentioned before.  Namely,
 1.7.2 satisfies =1.7 but does not satisfy =1.7.  With the series
 interpretation, the latter wouldn't be consistent (since 1.7.2 is part
 of the series under that interpretation).

 --Chris







 Marcus


 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: 1.7 vs =1.7

[Ian Cordasco]

On Sat, Dec 27, 2014 at 7:35 PM, Marcus Smith qwc...@gmail.com wrote:

 I'm starting a new thread to state cleanly what my current question/concern
 is...

 per PEP440 as I understand it:
 - for 1.7, 1.7 means roughly the 1.7 series or 1.7*

You mean 1.7.* right? Because 1.70 would satisfy 1.7

 - for =1.7, 1.7 means literally 1.7 (with zero-padding as needed)

 While I understand the motivation for the series conception of 1.7 to deal
 with  prereleases, the resulting inconsistency in meaning for 1.7 is what
 concerns me.  It's odd .  Can someone make it not seem odd for 1.7 to
 change meanings just because you switch between = and ?

 And for anyone who wants to say that 1.7 also means the 1.7 series for
=, note that 1.7 prereleases do not satisfy =1.7

 Would a solution be to make = also use the series concept?, i.e. make
 pre-releases satisfy =1.7

 Marcus





 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Ian Cordasco]

Yes but thanks to Marc it now provides a clear message as to why.

(Sent from my mobile)
On Dec 23, 2014 11:22 AM, James Bennett ubernost...@gmail.com wrote:

 So, if PyPI has foo-1.7 and foo-1.7.1, does 1.7 just fail to find
 anything installable?

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Ian Cordasco]

It also provides consistency with date-based versions. And versions aren't
decimals so thinking of them like that is not exactly useful.
On Dec 23, 2014 11:43 AM, Paul Moore p.f.mo...@gmail.com wrote:

 On 22 December 2014 at 20:44, Marcus Smith qwc...@gmail.com wrote:
  it would fail.  you'd need  1.7.0
 
  On Mon, Dec 22, 2014 at 12:36 PM, James Bennett ubernost...@gmail.com
  wrote:
 
  So, if PyPI has foo-1.7 and foo-1.7.1, does 1.7 just fail to find
  anything installable?

 I think the thing I'd missed, which makes this behaviour more
 understandable (for me) is that you wouldn't usually get that in
 reality. Projects tend to use a fixed number of digits in the version
 number, so it'd likely be 1.7.0 and 1.7.1, and you'd be writing
 1.7.0.

 Thinking of 1.7 as greater than the 1.7 series sort of helps me as
 well...

 Paul
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Ian Cordasco]

The other way in which this makes a whole lot more sense is when
considering projects that use dates as versions. Let's take for
example projects that have a released versions like

2013.08.10
2013.10.30
2013.12.03
2014.02.16
2014.09.23

If I say I want foo  2013, I expect anything in the 2014.*.* series.
I don't expect 2013.*.* The same works for  1.7

On Mon, Dec 22, 2014 at 3:12 PM, Marcus Smith qwc...@gmail.com wrote:

 * The new behavior maintains consistency between  and , so that
 specifiers that “look” the same act the same, maintaining consistency
 between them.
 * I think that having the  and  behavior vary is a *worse* confusion,
 and I believe that the behavior of  is far better than previous.


 I'm not following the old inconsistencies you're referring to. Maybe you can
 explain those, or reference this in the PEP.
 But in any case, the argument isn't over whether PEP440 is better than the
 old implementation, in total.


 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2

[Ian Cordasco]

On Mon, Dec 1, 2014 at 12:35 PM, Donald Stufft don...@stufft.io wrote:

 On Dec 1, 2014, at 4:25 AM, holger krekel hol...@merlinux.eu wrote:

 Hi Donald,

 On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
 On Nov 13, 2014, at 9:21 PM, Donald Stufft don...@stufft.io wrote:

 Starting a new thread with more explicit details at Richard’s request.
 Essentially the tl;dr here is that we'll switch to using sha2 (specifically
 sha256).

 Ping?

 Are we OK to make this change?

 sorry i didn't get back earlier.  Before the minor release of devpi-server
 last week i tried for two hours to change devpi-server to accomodate
 your planned pypi.python.org checksum changes.

 I found the change cannot easily be done without changes to the underlying
 database schema and thus needs a major new release of devpi-server because
 an export/import cycle is needed.  When doing that i also want to do
 some internal cleanup related to name normalization (and also relating
 to recent pypi.python.org changes) but i need a week or two i guess to
 do that.  However i now think that if you do the pypi.python.org checksum
 change it shouldn't directly break devpi-server but it would remove
 checksum checking.  I'd rather like to have a new major devpi-server
 release out when you do the change.  Is it ok for you to wait a bit still?

 best,
 holger

 Yes, we can wait a bit. I was just going over my TODO list and making sure
 things weren’t getting lost in the shuffle.

 ---
 Donald Stufft
 PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

Holger,

Is there anyway people on this list can help with the updates to devpi
so that we can get this out sooner?

Cheers,
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2

[Ian Cordasco]

On Mon, Dec 1, 2014 at 3:23 PM, holger krekel hol...@merlinux.eu wrote:
 On Mon, Dec 01, 2014 at 12:45 -0600, Ian Cordasco wrote:
 On Mon, Dec 1, 2014 at 12:35 PM, Donald Stufft don...@stufft.io wrote:
 
  On Dec 1, 2014, at 4:25 AM, holger krekel hol...@merlinux.eu wrote:
 
  Hi Donald,
 
  On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
  On Nov 13, 2014, at 9:21 PM, Donald Stufft don...@stufft.io wrote:
 
  Starting a new thread with more explicit details at Richard’s request.
  Essentially the tl;dr here is that we'll switch to using sha2 
  (specifically
  sha256).
 
  Ping?
 
  Are we OK to make this change?
 
  sorry i didn't get back earlier.  Before the minor release of devpi-server
  last week i tried for two hours to change devpi-server to accomodate
  your planned pypi.python.org checksum changes.
 
  I found the change cannot easily be done without changes to the underlying
  database schema and thus needs a major new release of devpi-server because
  an export/import cycle is needed.  When doing that i also want to do
  some internal cleanup related to name normalization (and also relating
  to recent pypi.python.org changes) but i need a week or two i guess to
  do that.  However i now think that if you do the pypi.python.org checksum
  change it shouldn't directly break devpi-server but it would remove
  checksum checking.  I'd rather like to have a new major devpi-server
  release out when you do the change.  Is it ok for you to wait a bit still?
 
  best,
  holger
 
  Yes, we can wait a bit. I was just going over my TODO list and making sure
  things weren’t getting lost in the shuffle.

 Holger,

 Is there anyway people on this list can help with the updates to devpi
 so that we can get this out sooner?

 Looking at devpi/server/devpi_server/extpypi.py and
 devpi/server/devpi_server/model.py mainly and changing most places
 where md5 is found in the source and adapting related tests.

 Is there a specific reason you are in a hurry if i may ask?

 best,
 holger

No real hurry. I just like helping out when there's an opening and
this thread has been around for a short while already. Given the topic
is related to the security of PyPI and its users, I'd like to help
move that forward if possible. That's all. (It's mostly me being
selfish.)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Package's declared latest version ignored by Warehouse

[Ian Cordasco]

On Dec 1, 2014 6:22 PM, Ben Finney ben+pyt...@benfinney.id.au wrote:

 Howdy all,

 The Warehouse is ignoring the feature of PyPI which sets particular
 versions of a package visible or not visible. It makes all versions
 visible regardless.

 This is a problem when, for example, a package has been uploaded but
 should not be shown by default.

 An example is the ‘python-daemon’ package. At PyPI, the latest visible
 version is 1.5.5, as requested by the package maintainer. That is
 reflected as the default version shown at
 URL:https://pypi.python.org/pypi/python-daemon/. Also, when viewing
 other versions, the “Latest version” link appears, and correctly shows
 that the latest version is 1.5.5.

 But at Warehouse, the settings for which versions should be hidden are
 ignored by the application, and a different version is shown by default
 URL:https://warehouse.python.org/project/python-daemon/. That page
 should show version 1.5.5, as selected by the package maintainer in the
 PyPI database.

 --
  \  “He who allows oppression, shares the crime.” —Erasmus Darwin, |
   `\ grandfather of Charles Darwin |
 _o__)  |
 Ben Finney

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

I haven't verified this but could you file a bug report at the warehouse
project with this information?

Thanks,
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Replacing md5 hashes for packages on PyPI

[Ian Cordasco]

+1 from me as well

On Wed, Nov 12, 2014 at 1:41 PM, Donald Stufft don...@stufft.io wrote:

 On Nov 12, 2014, at 2:34 PM, Alex Gaynor alex.gay...@gmail.com wrote:

 Hi everybody!

 Right now, PyPI provides MD5 hashes for packages, which is used by pip to
 check
 for corruption in transit. I'd like to propose we replace MD5 with SHA256 for
 PyPi, and move to deprecate MD5 support in pip and setuptools.

 Why should we do this? MD5 is broken. Collision resistance is totally 100%
 uselessly busted, and pre-image resistance is mathematically broken; 
 practical
 attacks aren't known publicly, but it's reasonable to assume private attacks
 are strong because (sing it with me): Attacks only get better.

 So MD5 doesn't provide the guarantees one might expect; SHA256 is not
 broken in
 these ways. But it's not just not providing value, it's actively causing
 problems: some machines, such as those with packages compiled to meet
 FIPS-140-2 do not have MD5 available at all, and so pip's verification raises
 an exception.

 While one might be inclined to find a way to silently support both machine
 configurations, I'd like to instead say we should abhor any additional
 configuration (whether user supplied or auto-detected) and instead simply
 upgrade the hashes offered by PyPI, and begin the deprecation process for
 MD5
 in pip.

 There are currently 60 packages on PyPI which are *not* hosted on PyPI, but
 do
 have MD5 hashes there. For these packages we could download the package,
 verify
 the MD5 hash, and then upgrade what PyPI stores to be SHA256.


 +1 from me.

 Security wise pip supported sha256 before it supported TLS so for pip anything
 that has the ability to securely fetch the sums from the /simple/ pages has
 the ability to use sha256. For setuptools there was a small window where
 setuptools implemented TLS verification (0.7) and implemented the support for
 things other than MD5 (0.9). However I don’t think this small window 
 represents
 a large (or any?) number of users.

 IOW the impact should be non-existent other than having better digests.

 ---
 Donald Stufft
 PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Hobby time (was: some questions about PEP470)

[Ian Cordasco]

On Oct 16, 2014 4:47 PM, Tres Seaver tsea...@palladion.com wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 On 10/14/2014 09:15 PM, Donald Stufft wrote:
  On Oct 14, 2014, at 8:50 PM, Stefan Krah stefank...@freenet.de
  wrote:
 
  Donald Stufft donald at stufft.io writes:
 
  If you're this upset over someone redistributing your work,
  then maybe Open Source Software is the wrong hobby for you.
 
  Usually one does not tell a core developer that his contributions
  are a hobby.  I have contributed 4+ lines of original, dense
  C code, backed by 100% code coverage and 3+ lines of ACL2
  proofs.
  Uhh, maybe you’re misunderstanding the word hobby, unless you’re
  getting paid for your OSS work you’re not doing it professionally. A
  hobby isn’t a negative thing, until last December my OSS work was
  entirely a hobby too, and it’s still a hobby in my spare time too.

 That use of hobby vs. professional is completely irrelevant to the
 discussion, and is effectively condescending and ad hominem.  In my
 experience, the quality and committment of a developer's open source work
 are often unrelated to whether she gets paid directly by an employer to
 do it.  Getting paid *does* make it possible to devote more time, rather
 than passion, to one's projects.



 Tres.
 - --
 ===
 Tres Seaver  +1 540-429-0999  tsea...@palladion.com
 Palladion Software   Excellence by Designhttp://palladion.com
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1.4.11 (GNU/Linux)

 iEYEARECAAYFAlRAPOIACgkQ+gerLs4ltQ5vWACfRDCmT5yjkqfeBB+4xGAiBnAv
 n7MAoKag+7GkicRdZ9eSpJdz+HKml6aG
 =5Uxr
 -END PGP SIGNATURE-


A hobby is by definition what you do in your free time. Donald is clearly
encouraging Stefan to spend his free time as he wishes. None of us who
participate in our free time are obligated to support anything beyond our
desire to do so, unlike someone who's paid to work on their project(s). It
almost seems like you're nitpicking the valid usage of a word to pile into
an argument in which you have nothing valid to say. I know it doesn't
usually happen on mailing lists, but that's what it seems like here.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] cdecimal licensing/hosting (was: some questions about PEP470)

[Ian Cordasco]

On Sun, Oct 12, 2014 at 1:44 PM, Alex Gaynor alex.gay...@gmail.com wrote:
 Stefan Krah stefankrah at freenet.de writes:



  (for example right now bytereef.org is down, so
  we’d not discover any files there).

 Indeed.  It was up reliably since 2005, down for maintenance on
 September 23rd (before ShellShock ...).  Then I discovered that
 someone had put up m3-cdecimal on PyPI (presumably abusing PyPI
 as their private repo --- there are several m3-* packages now).

 This triggered some reflection on whether I would make a significant
 effort in the future to keep things running smoothly for an open source
 community where authors are largely viewed as expendable.

 I don't know what it means for authors to be largely viewed as expendable,
 but half the point of hosting things on PyPI is that you *don't* need to do 
 any
 work at all as an author for reliable delivery of your package.


 Subsequently the downtime (again, the first one since 2005) was picked
 up for propagandistic purposes on Twitter and Reddit.

 Ok, but you seem to be doing the other side's propaganda. Every single person
 I've spoken to agrees that this just underscores the need to encourage 
 packages
 to be on PyPI.


 Last year I would have felt an obligation to minimize the downtime
 to an hour at most.  I no longer feel any such obligations and I'll
 do it when I have time.


 Ok. The PyPI administrators still feel an obligation to their users, so I'll
 prefer packages under their care.

 Stefan Krah


 Cheers,
 Alex

Perhaps Stefan's referring to my tweets about the inability to reach
bytereef but those weren't propaganda tweets. Those were tweets born
out of utter frustration. Further, I'm rather shocked that you've
decided to allow the site to remain unreachable because someone did
what your license allowed them to do (redistribute the software while
retaining the required information: copyright, license, etc). If you
think that makes you expendable, you're half right. Users can
redistribute your software, that's the nature of the license you chose
to use. You're wrong because you, the author, are still very valuable
to those very users who may encounter a bug in the future. I don't see
how intentionally keeping your site unreachable does anything but hurt
your users (unless of course you want them to redistribute it
themselves or switch to Python 3.4).

Does this mean that companies using devpi to keep an internal index
that also have copies of cdecimal are somehow violating your rights?
They're doing exactly what your license allows them to do. Or is it
just that some group has decided to redistribute it directly through
PyPI? I'm thoroughly confused here.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting

[Ian Cordasco]

On Fri, Oct 3, 2014 at 9:42 AM, Marius Gedminas mar...@pov.lt wrote:
 On Fri, Oct 03, 2014 at 02:05:50AM -0400, Donald Stufft wrote:
 Using this additional location within pip is also simple and can be included
 on a per invocation, per shell, or per user basis. The pip 6.0 will also
 include the ability to configure this on a per virtual environment or per
 machine basis as well. This can be as simple as:

 ::

 $ # As a CLI argument
 $ pip install --extra-index-url https://index.example.com/ myproject
 $ # As an environment variable
 $ PIP_EXTRA_INDEX_URL=https://pypi.example.com/ pip install myproject
 $ # With a configuration file
 $ echo [global]\nextra-index-url = https://pypi.example.com/;  
 ~/.pip/pip.conf
 $ pip install myproject

 This is where I get a question: what do I do if package X wants an
 extra repository FOO, and package Y wants an extra repository BAR, and
 my project relies on both X and Y?

 I assume the --extra-index-url=URL argument to pip install can be
 repeated multiple times.  It's less clear what to do about environment
 variables or config file settings.  Do I specify space-separated URLs?
 Newline separated?

 An example would be good.

 Marius Gedminas

I would assume something like

$ PIP_EXTRA_INDEX_URL=https://pypi1.example.com/,https://pypi2.example.com/
pip install myproject
$ pip install --extra-index-url=https://pypi1.example.com/
--extra-index-url=https://pypi2.example.com/ myproject
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Ian Cordasco]

On Mon, Sep 29, 2014 at 9:36 AM, Barry Warsaw ba...@python.org wrote:
 On Sep 28, 2014, at 07:31 PM, Donald Stufft wrote:

I'd like to discuss the idea of moving PyPI to having immutable files. This
would mean that once you publish a particular file you can never reupload
that file again with different contents. This would still allow deleting the
file or reuploading it if the checksums match what was there prior.

 Although I have abused this in the past, as others have pointed out, because
 once uploaded I realize there is a bug in the package.  There's a certain
 class of such bugs that prompt a quick re-upload rather than a version rev,
 such as some display problem on PyPI (because of package metadata), or some
 follow on packaging bug, such as a missing MANIFEST.in causing Debian package
 build to fail.  Yes, the latter is more easily checked before upload, but
 sometimes you feel optimistic. ;)

 This won't make your lives easier, but I'd like to propose some support for
 embargoed uploads.  These would be normal uploads except that they wouldn't
 be publicly available until a 'publish' button were pushed.  Such embargoed
 uploads wouldn't be subject to the checksum limitation, and we'd have to
 figure out exactly how such packages would be available (certainly to a logged
 in owner of the project via the web, but perhaps through an authenticated
 scriptable interface).

 Even if you decide against supporting something like this, I'd still be okay
 with the checksum restriction.  You never run out of version numbers.

 -Barry

That's essentially what I see as the chief use-case for
testpypi.python.org. I don't think pypi.python.org needs to support
this as well. Simple is better than complex after all :)

Cheers,
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Ian Cordasco]

On Mon, Sep 29, 2014 at 10:03 AM, Barry Warsaw ba...@python.org wrote:
 On Sep 29, 2014, at 09:40 AM, Ian Cordasco wrote:

That's essentially what I see as the chief use-case for
testpypi.python.org. I don't think pypi.python.org needs to support
this as well. Simple is better than complex after all :)

 Can we then make it easy to upload to testpypi via the cli?  Maybe it already
 is or I'm using the wrong tools.  (I'm just a dumb `setup.py upload` monkey.)

 Cheers,
 -Barry

There are some easy to follow instructions linked from testpypi's
homepage: https://wiki.python.org/moin/TestPyPI

Did you want more than that?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Ian Cordasco]

+1 I know I abused this a couple times a couple years ago, but it
bothered me that I could. It also worried me because if my account
were ever compromised, someone could release malware under files named
exactly the same as my real released software. This won't prevent them
from deleting those other versions and uploading something new, but it
will provide a small bit of extra assurance.

On Sun, Sep 28, 2014 at 7:23 PM, Marcus Smith qwc...@gmail.com wrote:


  It does happen that files need to be reuploaded because of a bug
  in the release process and how people manage their code is really
  *their* business, not that of PyPI.

 It's not just the business of the package authors, because as soon as it's
 uploaded it's visible to uesrs, and swapping it out from under their feet
 is a
 crummy thing to do.


 agreed,  +1 to the proposal.


 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Create formal process for claiming 'abandoned' packages

[Ian Cordasco]

On Sat, Sep 20, 2014 at 12:30 AM, John Wong gokoproj...@gmail.com wrote:
 Hi all.

 TL;DR version: I think

 * an option to enroll in automatic ownership transfer
 * an option to promote Request for Adoption
 * don't transfer unless there are no releases on the index

 will be reasonable to me.

 On Fri, Sep 19, 2014 at 9:26 PM, Richard Jones rich...@python.org wrote:


 In light of this specific case, I have an additional change that I think
 I'll implement to attempt to prevent it again: In the instances where the
 current owner is unresponsive to my attempts to contact them, *and* the
 project has releases in the index, I will not transfer ownership. In the
 cases where no releases have been made I will continue to transfer
 ownership.


 I believe this is the best solution, and frankly, people in the OSS world
 has been forking all these years
 should someone disagree with the upstream or just believe they are better
 off with the fork. I am not
 a lawyer, but one has to look at any legal issue with ownership transfer. I
 am not trying to scare
 anyone, but the way I see ownership transfer (or even modifying the index on
 behalf of me) is the same
 as asking Twitter or Github to grant me a username simply because the
 account has zero activity.

 Between transferring ownership automatically after N trials and the above, I
 choose the above.
 Note not everyone is on Github, twitter. Email, er, email send/receive can
 go wrong.

 As a somewhat extreme but not entirely rare example, Satoshi Nakamoto and
 Bitcoin would
 be an interesting argument. If Bitcoin was published as a package on PyPI,
 should someone
 just go and ask for transfer? We know this person shared his codes and the
 person disappeared.
 Is Bitcoin mission-critical? People downloaded the code, fork it and started
 building a community
 on their own. What I am illustrating here is that not everyone can be in
 touch. There are people
 who choose to remain anonymous, or away from popular social network.

 Toshio Kuratomi a.bad...@gmail.com wrote:

 But there are
 also security concerns with letting a package bitrot on pypi.


 Again, I think that people should simply fork. The best we can do is simply
 prevent
 the packages from being downloaded again. Basically, shield all the packages
 from public. We preserve what people did and had. We can post a notice
 so the public knows what is going on.

 Surely it sucks to have to use a fork when Django or Requests are forked and
 now everyone has to call it something different and rewrite their code.
 But that's the beginning of a new chapter. The community has to be reformed.
 It sucks but I think it is better in the long run. You don't have to argue
 with the
 original owner anymore in theory.

 Last, I think it is reasonable to add Request for Adoption to PyPI.
 Owners who feel obligated to step down can promote the intent over PyPI

 John

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


I, for one, am happy that this conversation is happening because I
wasn't aware of other communities that did this (but was aware that it
happened on PyPI). That said, I would really appreciate people's
suggestions be contained to improving the process, not towards
modifying PyPI. At this point, as I understand it, PyPI is incredibly
hard to modify safely for a number of reasons that others are likely
better to speak to. Warehouse has a clear definition, design,  and
goals and I don't know if adding these on after-the-fact in a
semi-haphazard way will improve anything. The more useful discussion
right now will be to talk about process and how we can improve it and
help Richard with it.

Cheers
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP draft on PyPI/pip package signing

[Ian Cordasco]

On Mon, Jul 28, 2014 at 8:12 PM, Giovanni Bajo ra...@develer.com wrote:
 Il giorno 29/lug/2014, alle ore 02:39, Nick Coghlan ncogh...@gmail.com ha
 scritto:

 On 29 Jul 2014 10:01, Giovanni Bajo ra...@develer.com wrote:

 Il giorno 29/lug/2014, alle ore 01:36, Nick Coghlan ncogh...@gmail.com
 ha scritto:

 On 29 Jul 2014 03:43, Giovanni Bajo ra...@develer.com wrote:
 
  Hello,
 
  on March 2013, on the now-closed catalog-sig mailing-list, I submitted
  a proposal for fixing several security problems in PyPI, pip and
  distutils[1]. Some of my proposals were obvious things like downloading
  packages through SSL, which was already in progress of being designed and
  implemented. Others, like GPG package signing, were discussed for several
  days/weeks, but ended up in discussion paralysis because of the upcoming 
  TUF
  framework.

 It stalled because end-to-end signing is a hard security problem and
 just add GPG! isn't an answer.

 I don’t find it very fair that you collapse a 700-line document to “just
 add GPG”. And even in March, I wrote and proposed an extensive document. I’m
 not jumping into discussions handwaving a couple of buzzwords.

 If your PEP defends against all the attacks TUF does, then it will be just
 as complicated as TUF. If it doesn't defend against all those attacks, then
 it offers even less justification for the complexity increase than TUF.

 1) TUF isn’t designed for PyPI and pip. It’s a generic system meant for many
 different scenarios, which is then adapted (with many different compromises)
 to our use case. So you can’t really postulate that you absolutely need
 something as complicated to get to the same level of defense.
 2) Security is never perfection. You might very well decide that the
 increased level of security is not worth the complexity increase.

 My solution is far far simpler than TUF. To me, it’s a reasonable compromise
 between complexity of implementation, time/costs required for all involved
 parties, and decreased security.

While there's a significant difference between the complexity of the
proposals to implement, there's also a significant difference in
complexity for the end users.

On the one hand, to implement PEP458, would mean a great deal of work
on those working on PyPI/Warehouse and pip, but it would have little
(if any) end user implications or complications.

Your PEP on the other hand, causes some instabilities (especially if
PGP/GPG isn't installed, or if someone has hijacked the PATH) and will
create only headaches for the users. They'll have to install GPG,
generate a Key, upload the key, secure the key, and make sure they
don't ever lose it. While there's less complexity for the
implementers, there's much more for end users. We don't want to make
packaging worse for users in exchange for a negligible (questionable?)
amount of increased security.

 If you add a threat model to the draft PEP, then we can have a useful
 discussion,

 There is a whole section called threat model”. If you would like to see
 the threat model extended to cover different attacks, I reckon there’s a
 less aggressive way to phrase your opinion.

 My apologies, it is customary to explain the threat model *before* proposing
 a solution to it, and I did indeed lose patience before reaching that part
 of the PEP. I have now read that section, and still find it lacking compared
 to the comprehensive threat analysis research backing TUF.

 If it’s only “lacking, then i’m happy, as that PEP is not my PhD
 dissertation :)

 1. People like Donald, Ernest, Richard Noah (i.e. PyPI and infrastructure
 admins) are part of the threat model for PEP 458. How does your PEP defend
 against full compromise of those accounts?


 It doesn’t, nor PEP 458 does. If an attacker owns a root shell on PyPI, it
 is game over with both PEPs; that is, unless you’re referring to the claimed
 role for which PEP 458 is severely lacking the most important and hard
 thing: how to populate it.

 PEP 458 includes both offline root keys and an N of M signing model
 specifically so compromise of a single administrator account can't break the
 whole system (aside from DoS attacks).

 It depends on your definition of “compromise of a single administrator”.
 Obviously if you compromise the root signing key, then yes, the N/M model
 helps, but that key doesn’t exist in my PEP in the first place (which means
 that my PEP is simpler, requires less work on administrators, less key
 management, etc.). If with “compromise of an administrator” you intend that
 an attacker can login into PyPI and become root, than I maintain that PEP458
 and my PEP are mostly equivalent, in that they offer close to no protection;
 the attacker can compromise all packages.

 PEP458 then has this “claimed” role that is signed with an offline key,
 which would protect from a PyPI compromise, and which Donald says it’s the
 only reason he cares about PEP458. But it totally punts on how to maintain
 that role from a practical 

Re: [Distutils] Need for respect

[Ian Cordasco]

On Thu, May 15, 2014 at 9:28 AM, Vladimir Diaz
vladimir.v.d...@gmail.com wrote:
 https://en.wikipedia.org/wiki/Python_Package_Index

 Description: While the PyPI website is maintained by the Python Software
 Foundation, its contents are uploaded by individual package maintainers.
 Python package managers such as pip default to downloading packages from
 PyPI.



 https://pypi.python.org/pypi

 Description: The Python Package Index is a repository of software for the
 Python programming language. There are currently 43698 packages here.



 https://pythonhosted.org/an_example_pypi_project/setuptools.html#registering-your-project

 The normal work flow seems to be:
 1. Register your project.
 2. Upload your project.


 I think a newcomer is likely to conclude that pip downloads packages from
 the official repository (PyPI) and these packages have been uploaded to PyPI
 by third-party developers.


 On Thu, May 15, 2014 at 6:53 AM, David Arnold dav...@pobox.com wrote:

 On 15/05/2014, at 8:32 AM, Mark Young wrote:

  I know I'm just an anecdote, but as just a regular user of pypi, pip,
  and friends (I've never even posted something to pypi), when I say pip
  install spam, I don't really care where it comes from and I have no
  expectation that pip has to get it from pypi.

 I guess just to prove Mark's not the only one (and perhaps because I'm
 feeling contrary), I should probably say that I don't expect that pip will
 necessarily download things from pypi either.  I'm used to packaging tools
 using CDNs, mirrors, and lists of sources to give me what I ask for in the
 way they think best at the time.  That's part of their value, really.


We're all anecdotes and none of us are alone or unique in our
opinions. That said, given my experience with Clojure (Clojars), Ruby
(Rubygems), Node (npm), R (CRAN), and Perl (CPAN), my expectations
match what Donald has been saying. Beyond that, I think the new PEP
that has been proposed will radically improve the user experience of
using pip and PyPI.

That said, while many users may have their own mental models, it is
apparent that several resources a new user might encounter point to
PyPI as a repository from which they download those packages.

Cheers,
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Need for respect

[Ian Cordasco]

On Thu, May 15, 2014 at 10:27 AM, Stefan Krah
stefan-use...@bytereef.org wrote:

 Vladimir Diaz vladimir.v.d...@gmail.com wrote:
 https://en.wikipedia.org/wiki/Python_Package_Index

 Description: While the PyPI website is maintained by the Python Software
 Foundation, its contents are uploaded by individual package maintainers. 
 Python
 package managers such as pip default to downloading packages from PyPI.

 Really? How about the previous revision:

 https://en.wikipedia.org/w/index.php?title=Python_Package_Indexoldid=575596902


 Or Jeremy Hylton's link:

 https://www.python.org/~jeremy/weblog/030924.html


 If anything, this edit is part of the ongoing reeducation and indoctrination.

There is no conspiracy. Python.org's wiki refers to PyPI as a
distribution mechanism:

 PyPI, the Python Package Index, is a web-based software catalogue and 
 distribution mechanism.

Further, as I read it, the previous revision makes the analogy to CPAN
which is exactly how Donald, Nick, Paul, and I are characterizing
PyPI.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Python-checkins] peps: PEP 470: use external indexes over link spidering

[Ian Cordasco]

On Thu, May 15, 2014 at 10:13 AM, Stefan Krah
stefan-use...@bytereef.org wrote:
 I'm not opposed to the Custom Additional Index if downloads can still be
 protected by a checksum stored on PyPI.

 What I find disturbing is that even after all these discussions the document
 contains many personal preferences, heavily biased statements about external
 servers and legal advice.

PEPs frequently include personal preferences and reasoning. PEP-0466
is an example of this.

 nick.coghlan python-check...@python.org wrote:
 +Author: Donald Stufft don...@stufft.io,
 +
 +Custom Additional Index
 +---
 +
 +Compared to the complex rules which a project must be aware of to prevent
 +themselves from being considered unsafely hosted setting up an index is 
 fairly
 +trivial and in the simplest case does not require anything more than a
 +filesystem and a standard web server such as Nginx.

 It's trivial to add the explicit URL with the checksum compared to configuring
 a new subdomain, so I don't like the reason given.

If we're arguing triviality of things, it's trivial to just upload
your package to PyPI, so I don't like your reason given. Then again my
dislike of your reason doesn't invalidate it, just as your dislike of
this doesn't invalidate it.

 +Deprecation and Removal of Link Spidering
 +=
 +After that switch, an email will be sent to projects which rely on hosting
 +external to PyPI. This email will warn these projects that externally hosted
 +files have been deprecated on PyPI and that in 6 months from the time of 
 that
 +email that all external links will be removed from the installer APIs. This
 +email *must* include instructions for converting their projects to be hosted
 +on PyPI and *must* include links to a script or package that will enable 
 them
 +to enter their PyPI credentials and package name and have it automatically
 +download and re-host all of their files on PyPI. This email *must also*
 +include instructions for setting up their own index page and registering 
 that
 +with PyPI.

 Please add: The email *must* include the PyPI terms and conditions.

I think a link to the Terms and Conditions is far more reasonable.

 +Five months after the initial email, another email must be sent to any 
 projects
 +still relying on external hosting. This email will include all of the same
 +information that the first email contained, except that the removal date 
 will
 +be one month away instead of six.

 Also here, please include the terms and conditions in the mail.

Again, a link to them should suffice.

 +* People are generally surprised that PyPI allows externally linking to 
 files
 +  and doesn't require people to host on PyPI. In contrast most of them are
 +  familiar with the concept of multiple software repositories such as is in
 +  use by many OSs.

 This is speculation and should be deleted.

This is not speculation. If you poll #python Freenode you'll see this
is a consensus opinion.

 +* PyPI is fronted by a globally distributed CDN which has improved the
 +  reliability and speed for end users. It is unlikely that any particular
 +  external host has something comparable. This can lead to extremely bad
 +  performance for end users when the external host is located in different
 +  parts of the world or does not generally have good connectivity.

 This is editorializing. In fact recently a former Debian project leader has
 called PyPI a bit flaky:

 https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html

Either this hasn't happened and you're intentionally linking to the
wrong message assuming people won't actually read it, or you linked to
the wrong message. I'd like to believe the latter but apparently there
are grand conspiracies in the Python Packaging community and I'm not
about to just trust you if there are, you could be one of the
conspirators.

 +  As a data point, many users reported sub DSL speeds and latency when
 +  accessing PyPI from parts of Europe and Asia prior to the use of the CDN.

 Irrelevant. Because the old PyPI server was overloaded, all external servers
 have to be, too?

No but link spidering seriously can degrade performance. Perhaps this
should be reworded to be more explicit, but the point stands.

 +* PyPI has monitoring and an on-call rotation of sysadmins whom can respond 
 to
 +  downtime quickly, thus enabling a quicker response to downtime. Again it 
 is
 +  unlikely that any particular external host will have this.

 That is quite general and irrelevant to the topic of this PEP (replacing one
 scheme of accommodating external hosts with another).

It isn't irrelevant. PyPI's availability and speed is superior to it's
past and to many other services and hosts. If anything happens to the
server you host cdecimal on, you're responsible for understanding that
and handling that. As a developer who uploads their software to PyPI
though you wouldn't have to worry about that. The 

Re: [Distutils] Need for respect

[Ian Cordasco]

Your point is invalid because you're ignoring that it is a
distribution mechanism. In order to distribute something, there needs
to be a catalogue of what you're distributing as well as the item
you're distributing. Businesses that serve as distributors both have
the physical item and a catalogue of what they have. Re-distributors
have catalogues of what others have. If it said web-based software
catalogue _or_ distribution mechanism I might concede your point, but
it doesn't.

On Thu, May 15, 2014 at 11:21 AM, Stefan Krah
stefan-use...@bytereef.org wrote:
 Ian Cordasco graffatcolmin...@gmail.com wrote:
 On Thu, May 15, 2014 at 10:27 AM, Stefan Krah
 stefan-use...@bytereef.org wrote:
 
  Vladimir Diaz vladimir.v.d...@gmail.com wrote:
  https://en.wikipedia.org/wiki/Python_Package_Index
 
  Description: While the PyPI website is maintained by the Python Software
  Foundation, its contents are uploaded by individual package maintainers. 
  Python
  package managers such as pip default to downloading packages from PyPI.
 
  Really? How about the previous revision:
 
  https://en.wikipedia.org/w/index.php?title=Python_Package_Indexoldid=575596902
 
 
  Or Jeremy Hylton's link:
 
  https://www.python.org/~jeremy/weblog/030924.html
 
 
  If anything, this edit is part of the ongoing reeducation and 
  indoctrination.

 There is no conspiracy. Python.org's wiki refers to PyPI as a
 distribution mechanism:

  PyPI, the Python Package Index, is a web-based software catalogue and 
  distribution mechanism.
^^

 Thank you for reinforcing my point.


 Stefan Krah


 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Need for respect

[Ian Cordasco]

On Thu, May 15, 2014 at 12:12 PM, Donald Stufft don...@stufft.io wrote:

 On May 15, 2014, at 12:21 PM, Stefan Krah stefan-use...@bytereef.org wrote:

 Ian Cordasco graffatcolmin...@gmail.com wrote:
 On Thu, May 15, 2014 at 10:27 AM, Stefan Krah
 stefan-use...@bytereef.org wrote:

 Vladimir Diaz vladimir.v.d...@gmail.com wrote:
 https://en.wikipedia.org/wiki/Python_Package_Index

 Description: While the PyPI website is maintained by the Python Software
 Foundation, its contents are uploaded by individual package maintainers. 
 Python
 package managers such as pip default to downloading packages from PyPI.

 Really? How about the previous revision:

 https://en.wikipedia.org/w/index.php?title=Python_Package_Indexoldid=575596902


 Or Jeremy Hylton's link:

 https://www.python.org/~jeremy/weblog/030924.html


 If anything, this edit is part of the ongoing reeducation and 
 indoctrination.

 There is no conspiracy. Python.org's wiki refers to PyPI as a
 distribution mechanism:

 PyPI, the Python Package Index, is a web-based software catalogue and 
 distribution mechanism.
   ^^


 I don't understand your point at all tbh. PyPI was started as an index, 
 however
 it's no longer primarily an index. If the only thing you have to stand on is
 the fact that a name chosen before the community at large decided to change
 it's purpose through their usage patterns then I'm not particularly convinced.

And if the name Index is confusing to you, we can always change
Python Package Index to Python Package Distribution Center or
something less confusing.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Windows 8 pkg_resources permissions bug

[Ian Cordasco]

We don't have the exact versions of setuptools or pip that are
problematic. If you have trouble reproducing this, we can ask our user
for clarification. :)

Thanks for the help!

On Mon, May 5, 2014 at 8:06 PM, Steve Dower steve.do...@microsoft.com wrote:
 My guess would be that something is setting permissions for the current
 user, and these installs were done by an admin user who is not the current
 user. This will result in explicit permissions on the file that may deny
 read access.

 I'll have to confirm tomorrow. Do you have the exact versions of setuptools
 and pip that are problematic? (My guess is that it won't matter. The code
 has probably been in there for a while if I'm right.)

 Cheers,
 Steve

 To-posted from my Windows Phone
 
 From: Ian Cordasco
 Sent: ‎5/‎5/‎2014 17:27
 To: Distutils list
 Subject: [Distutils] Windows 8 pkg_resources permissions bug

 Hey all,

 I searched the issues on the setuptools repository but I couldn't find
 any that seemed relevant to what a user of Flake8 has run into:
 https://bitbucket.org/tarek/flake8/issue/142/installation-on-windows-8-permissions

 It seems that installing something that relies on pkg_resources to
 load plugins will cause errors on Windows 8. Does anyone have any
 insight? Is this actually a bug? Are we just using pkg_resources
 incorrectly?

 Thanks in advance,
 Ian
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Windows 8 pkg_resources permissions bug

[Ian Cordasco]

On Tue, May 6, 2014 at 11:31 AM, Steve Dower steve.do...@microsoft.com wrote:
 So it isn't a trivial repro, which means there is some more context required 
 from anyone who sees this issue.

Thanks for putting the effort into reproducing it!

 I personally would be looking at the ACLs of the files that can't be 
 accessed. I expect to see that they are not inheriting permissions from their 
 parent and probably only have read-only permissions for whoever installed - 
 this is easy to check, because the user won't be able to open the file in any 
 other program either (or look at the permissions for that matter...). The 
 admin user will be able to see them and restore them.

 The logic in setuptools/pip/distutils/setup.py/whoever that may lead to 
 this is these files should _really_ be read-only and so they reset all 
 permissions and give the current user read access. If the administrative user 
 is the current user (which is pretty normal for Windows these days, because 
 even the admins spend most of their time restricted), this is fine for a 
 single-user system. However, where the admin user is a separate account 
 (name/SID) from the interactive user, this will result in the normal user 
 being denied even read access.

 My testing (with pip install flake8 in Python 2.7.6 64-bit) didn't show 
 anyone changing the permissions on any files. However, it is possible that 
 the user has manually secured their Python install (else why would they be 
 running pip install as admin?). AFAICT, Python 2.7 doesn't provide a way in 
 the stdlib to change ACLs (os.chmod is totally restricted to the read-only 
 flag, which isn't the problem here), and I didn't see anything in pip or 
 setuptools that could do it (though setuptools will use pywin32 if it is 
 there - it didn't seem related, but I also didn't test with it installed).

 My specific questions for the user would be:
 * Is your administrator account a different account from the one you normally 
 log in with? (Alternatively, do you see the UAC dialog appear? Do you have to 
 type a password or do you just click Yes?)
 * Can you open the file in Notepad or equivalent?
 * If you run cacls filename as administrator, what is the output?
 * If you run cacls C:\Python27\python.exe, what is the output?

I've added these questions to the issue but I'm not sure we'll hear
back from the user. They might not have an actual BitBucket account or
may have posted it anonymously by accident. If they don't check back,
they won't see the questions. Hopefully, if anyone else runs into
problems like this, they'll see those questions.

 This sort of case is the best argument for secure-by-default installations 
 into Program Files - no installer should even have to think about messing 
 with ACLs. The current CPython installers support this sort of install if you 
 change the directory, but maybe it is time for pip/setuptools to start 
 assuming that user site-packages is the default on Windows so we could 
 actually consider changing the default Python install location? (If anyone 
 really wants to dive into that discussion again, go ahead and change the 
 subject line :) )

Thanks again for all your help Steve!

Cheers!
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Windows 8 pkg_resources permissions bug

[Ian Cordasco]

Hey all,

I searched the issues on the setuptools repository but I couldn't find
any that seemed relevant to what a user of Flake8 has run into:
https://bitbucket.org/tarek/flake8/issue/142/installation-on-windows-8-permissions

It seems that installing something that relies on pkg_resources to
load plugins will cause errors on Windows 8. Does anyone have any
insight? Is this actually a bug? Are we just using pkg_resources
incorrectly?

Thanks in advance,
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))

[Ian Cordasco]

On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz n...@coderanger.net wrote:

 On Jul 14, 2013, at 9:45 AM, Steve Dower wrote:

 From: Paul Moore
 On 13 July 2013 10:05, Paul Moore p.f.mo...@gmail.com wrote:
 How robust is the process of upgrading pip using itself? Specifically on
 Windows, where these things typically seem less reliable.

 OK, I just did some tests. On Windows, pip install -U pip FAILS. The 
 reason
 for the failure is simple enough to explain - the pip.exe wrapper is held 
 open
 by the OS while it's in use, so that the upgrade cannot replace it.

 The result is a failed upgrade and a partially installed new version of 
 pip. In
 practice, the exe stubs are probably added fairly late in the install (at 
 least
 when installing from sdist, with a wheel that depends on the order of the 
 files
 in the wheel), so it's probably only a little bit broken, but a little bit
 broken is still broken :-(

 On the other hand, python -m pip install -U pip works fine because it 
 avoids
 the exe wrappers.

 There's a lot of scope for user confusion and frustration in all this. For
 standalone pip I've tended to recommend don't do that - manually 
 uninstall and
 reinstall pip, or recreate your virtualenv. It's not nice, but it's 
 effective.
 That sort of advice isn't going to be realistic for a pip bundled with 
 CPython.

 Does anyone have any suggestions?

 Unless I misunderstand how the exe wrappers work (they're all the same code 
 that looks for a .py file by the same name?) it may be easiest to somehow 
 mark them as non-vital, such that failing to update them does not fail the 
 installer. Maybe detect that it can't be overwritten, compare the 
 contents/hash with the new one, and only fail if it's changed (with an 
 instruction to use 'python -m...')?

 Spawning a separate process to do the install is probably no good, since 
 you'd have to kill the original one which is going to break command line 
 output.

 MoveFileEx (with its copy-on-reboot flag) is off the table, since it 
 requires elevation and a reboot. But I think that's the only supported API 
 for doing a deferred copy.

 If Windows was opening .exes with FILE_SHARE_DELETE then it would be 
 possible to delete the exe and create a new one by the same name, but I 
 doubt that will work and in any case could not be assumed to never change.

 So unless the exe wrapper is changing with each version, I think the best 
 way of handling this is to not force them to be replaced when they have not 
 changed.

 The usual way to do this is just move the existing executable to 
 pip.exe.deleteme or something, and then write out the new one. Then on every 
 startup (or maybe some level of special case for just pip upgrades?) try to 
 unlink *.deleteme. Not the simplest system ever, but it gets the job done.

I accidentally only emailed Paul earlier, but why can't we upgrade the
pip module with the exe and then replace the process (using something
in the os.exec* family) with `python -m pip update-exe` which could
then succeed since the OS isn't holding onto the exe file? I could be
missing something entirely obvious since I haven't developed
(directly) on or for Windows in at least 5 years.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))

[Ian Cordasco]

On Sun, Jul 14, 2013 at 2:10 PM, Noah Kantrowitz n...@coderanger.net wrote:

 On Jul 14, 2013, at 10:43 AM, Noah Kantrowitz wrote:


 On Jul 14, 2013, at 10:39 AM, Noah Kantrowitz wrote:


 On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote:

 On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz n...@coderanger.net 
 wrote:

 On Jul 14, 2013, at 9:45 AM, Steve Dower wrote:

 From: Paul Moore
 On 13 July 2013 10:05, Paul Moore p.f.mo...@gmail.com wrote:
 How robust is the process of upgrading pip using itself? Specifically on
 Windows, where these things typically seem less reliable.

 OK, I just did some tests. On Windows, pip install -U pip FAILS. The 
 reason
 for the failure is simple enough to explain - the pip.exe wrapper is 
 held open
 by the OS while it's in use, so that the upgrade cannot replace it.

 The result is a failed upgrade and a partially installed new version of 
 pip. In
 practice, the exe stubs are probably added fairly late in the install 
 (at least
 when installing from sdist, with a wheel that depends on the order of 
 the files
 in the wheel), so it's probably only a little bit broken, but a little 
 bit
 broken is still broken :-(

 On the other hand, python -m pip install -U pip works fine because it 
 avoids
 the exe wrappers.

 There's a lot of scope for user confusion and frustration in all this. 
 For
 standalone pip I've tended to recommend don't do that - manually 
 uninstall and
 reinstall pip, or recreate your virtualenv. It's not nice, but it's 
 effective.
 That sort of advice isn't going to be realistic for a pip bundled with 
 CPython.

 Does anyone have any suggestions?

 Unless I misunderstand how the exe wrappers work (they're all the same 
 code that looks for a .py file by the same name?) it may be easiest to 
 somehow mark them as non-vital, such that failing to update them does 
 not fail the installer. Maybe detect that it can't be overwritten, 
 compare the contents/hash with the new one, and only fail if it's 
 changed (with an instruction to use 'python -m...')?

 Spawning a separate process to do the install is probably no good, since 
 you'd have to kill the original one which is going to break command line 
 output.

 MoveFileEx (with its copy-on-reboot flag) is off the table, since it 
 requires elevation and a reboot. But I think that's the only supported 
 API for doing a deferred copy.

 If Windows was opening .exes with FILE_SHARE_DELETE then it would be 
 possible to delete the exe and create a new one by the same name, but I 
 doubt that will work and in any case could not be assumed to never 
 change.

 So unless the exe wrapper is changing with each version, I think the 
 best way of handling this is to not force them to be replaced when they 
 have not changed.

 The usual way to do this is just move the existing executable to 
 pip.exe.deleteme or something, and then write out the new one. Then on 
 every startup (or maybe some level of special case for just pip 
 upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it 
 gets the job done.

 I accidentally only emailed Paul earlier, but why can't we upgrade the
 pip module with the exe and then replace the process (using something
 in the os.exec* family) with `python -m pip update-exe` which could
 then succeed since the OS isn't holding onto the exe file? I could be
 missing something entirely obvious since I haven't developed
 (directly) on or for Windows in at least 5 years.

 Unfortunately windows doesn't actually offer the equivalent of a POSIX 
 exec(). The various functions in os don't actually replace the current 
 process, they just create a new one and terminate the old one. This means 
 the controlling terminal would see the pip process as ended, so it makes 
 showing output difficult at best.

 Check that, maybe I'm wrong, does anyone know if the P_OVERLAY flag unlocks 
 the original binary? /me drags out a windows VM …

 Ignore my ignoring, with os.execl command flow does return back to the 
 controlling terminal process (the new process continues in the background) 
 and with os.spawnl(os.P_OVERLAY, 'python-2') I just get a segfault on 3.3. 
 Yay for not completely misremembering, boo for this being so complicated.

I expected I was wrong, but I appreciate you looking at it to be certain.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Overriding dependency versions

[Ian Cordasco]

On Mon, May 13, 2013 at 10:12 AM, Reinout van Rees rein...@vanrees.org wrote:
 On 13-05-13 15:57, Ian Cordasco wrote:

 If I release a library dependent upon a particular API in one version
 of a dependency and before I release my next version I notice plans to
 break the existing API, why shouldn't I pin the version to protect
 users (or at least provide a maximum version) from getting horrible
 exceptions?


 Best example: if you pin somelibrary=1.2 in your library, none of your
 users can use the critical 1.2.1 bugfix release.

Thanks to you and Daniel. (I accidentally replied to Daniel off-list.)
I had just been advised by someone formerly a part of the distutils
(or distribute, I forget which) team that you either pin or don't. I
try not to, but there have been occasions where I found it necessary.
I'll certainly move forward a better developer for your explanations.

Thanks again,
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig