Re: Automation and other changes

[Donald Stufft]

> On Apr 2, 2017, at 1:26 PM, Donald Stufft <don...@stufft.io> wrote:
> 
> Sounds like between my starting to lean towards it, your preference, and the 
> fact it would trigger a new mail notification for Paul’s workflow too, that a 
> special command is the way to go. I will go ahead and implement that.


Ok! I have this implemented now, you can see it in action at 
https://github.com/pypa/pip/pull/4406 <https://github.com/pypa/pip/pull/4406>. 
I’m going to hold off on publishing docs because I want to switch to the user 
@pypa-bot instead of @BrownTruck but when I tried to create that user, GitHub 
flagged the account so it has to get manually reviewed first.

But basically, currently anyone can do “@BrownTruck request review” (must be on 
a line by itself, other stuff can be in the message though) and it will dismiss 
all of the current reviews, which moves it back into the “review queue” in the 
above work flow AND it triggers an email from the person to get email based 
workflow a notification that the PR is ready for more review.

Once I have it set up at @pypa-bot I’ll write the documentation for it, get it 
so that it comments about the command on the first request changes review (one 
per PR), and start sending periodic emails about things.

—
Donald Stufft

Re: Giving Pradyun Commit Access for Triaging

[Donald Stufft]

> On Jun 24, 2017, at 1:30 PM, Donald Stufft <don...@stufft.io> wrote:
> 
> I’m going to give Pradyun commit access on the pip repository for the limited 
> purpose of triaging issues.


This is done, I’ve created a new team called @pip-helpers which has write 
access to the repository for the sake of doing things like triage issues, but 
which does not have permissions to push to `master`. I’ve invited Pradyun into 
that team.

Welcome!

—
Donald Stufft

Re: NEWS entries for pip

[Donald Stufft]

> On May 2, 2017, at 4:08 PM, Paul Moore <p.f.mo...@gmail.com> wrote:
> 
> On 2 May 2017 at 20:50, Donald Stufft <don...@stufft.io> wrote:
>> It’s generally only a two step process if there isn’t an existing issue
>> you’re fixing. If you’re fixing an issue you can/should just use the issue’s
>> number instead of the PRs number. We can make it possible to do like
>> .feature and such too though if we think the random PR without an
>> issue case is big enough to warrant a special case.
> 
> Hmm, maybe. I tend to do all changes as PRs, not all of which come
> from an issue. But it's also true that it's not immediately obvious
> (to me) from the docs that it's OK to use the issue number rather than
> the PR number. I could try rewording the docs to that effect, but to
> do so will need the 2-step process as there's no issue for this :-)

Rewording the docs would be fine, and I think it would be a trivial fix since 
it’s just messing with development docs.

> 
> This originally came up for me because the discussion on
> https://github.com/pypa/pip/pull/4461 had got quite messy - I felt
> that it might be easier for me to offer to rework the OP's PR rather
> than keep asking for corrections - but having to ass the NEWS entry
> made me doing that more complex than I'd initially thought. NEWS
> management was making a pretty trivial change in the wording of an
> error into a bit of a nightmare, both for me and for the OP. I was
> *really * tempted to say drop the NEWS and I'll mark the change
> trivial - but that seemed wrong, given that the OP had taken the time
> to write a NEWS item.
> 
> I don't really have a good answer here. But this is my first
> experience with the new NEWS process - and it doesn't seem like an
> improvement over the old one, I'm afraid :-(
> 
> 


Would it be better if you could just name it no-issue-. in 
that case?


—
Donald Stufft

Remove access from inactive maintainers

[Donald Stufft]

Hi!

I was talking to some people today about some attack vectors, and one thing 
that got surfaced in that there are a few people able to cut a release to PyPI 
for pip/virtualenv/etc who have stepped back from being involved in the 
project. What I would like to do is remove access from these people *not* 
because we’d be “kicking them out”, but simply as an effort to reduce the 
accounts that are possible targets for compromising pip. I think the ideal way 
of doing this is to simply say that if they decide to come back, they can have 
their access reinstated without question.

I also think it’d make sense to extend this same policy to Github teams (not 
the organization itself, being a member of the organization doesn’t grant any 
special privileges).

With that in mind, my proposal is to remove:

* From pip on PyPI: Jannis Leidel, Brian Rosner, Carl Meyer, Ian Backing, 
Marcus Smith
* From virtualenv on PyPI: Jannis Leidel, Brian Rosner, Carl Meyer, Ian 
Backing, Marcus Smith
* From packaging: Marcus Smith

That leaves able to do releases being me on all 3, and Matt Iverson (Ivoz) on 
virtualenv. It’s not great to have a single bus factor on these projects in 
case something happens to me, so I’d like to add Paul Moore and Xavier 
Fernandez on all three projects as releasers as well (I’m fine actually 
continuing to do the releases generally, just as a backup) assuming they’re 
both agreeable.

Then On Github I’d like to remove:

* From the pip team: Brian Rosner, Ian Bicking, Hugo Lopes Tavares, Carl Meyer, 
Marcus Smith, 
* From the virtualenv team: Brian Rosner, Ian Bicking, Carl Meyer, Marcus Smith

Then there are currently 4 Owners of the Github Org PyPA, Myself, Brian Rosner, 
Carl Meyer, and Marcus Smith. For this I’d like to remove all but myself, and 
similarly to PyPI I’d like to add Paul and Xavier as owners so it’s not just me 
(also assuming both are agreeable).

This should remove access from anyone who hasn’t (that I could find) been an 
active participant in > 1 year, with the stipulation that if they decide to 
come back they will be granted their previous access back— so this is merely 
just a technical solution to limit access. If anyone has any problems with 
this, please speak up!

I’ve also made sure I’ve BCC’d anyone who I’ve mentioned as losing some kind of 
access to this email in case they’re not subscribed to pypa-dev so that they 
will be aware and can speak up themselves (BCC instead of CC so they don’t get 
spammed with any replies if they don’t care).

Absent any objections, I’ll take these actions in the next couple of days (and 
I’ll need PyPI usernames for Paul and Xavier).

—
Donald Stufft

Re: Remove access from inactive maintainers

[Donald Stufft]

> On Jun 6, 2017, at 2:53 PM, Jason R. Coombs <jar...@jaraco.com> wrote:
> 
> All fine by me.
> 
> Any reason I shouldn’t be an owner of PyPA and Setuptools shouldn’t inherit 
> similar permissions to the other projects? I really don’t want it to be a 
> special snowflake.
> 
> 

I don’t have a problem with you being an owner on the GH org. PyPA has always 
been in a bit of a weird place that it started out as a pip/virtualenv only org 
that got expanded out to covering the whole spectrum. In that vein, I was still 
thinking of who was active within pip for managing that.

So yea, totally fine with you being an owner. It doesn’t really affect the 
project itself much, being an owner mostly just gives you the 
rights/responsibility of managing teams/repos/etc when a new project gets added.

—
Donald Stufft

Re: New pip core developer: Pradyun Gedam

[Donald Stufft]

> On Oct 5, 2017, at 4:51 PM, Paul Moore  wrote:
> 
> Many of you will have seen the work Pradyun has been doing on the pip
> tracker recently. He's been doing a fantastic job, and as a result,
> we've offered him core developer status - and I'm pleased to say that
> he's accepted :-)
> 
> Welcome, Pradyun - thanks for all the work you've been doing, and
> here's to plenty more ;-)
> 
> Paul


Welcome!

Re: Disallow packages with the same name as stdlib modules

[Donald Stufft]

Sorry for the delay in response.

So we actually *do* disallow package names with the same name as stdlib 
modules, however because there are a number of them that exist today and are 
useful (asyncio, ssl, etc) the way we’ve implemented this is that *new* 
projects cannot be created with the same name as stdlib modules, but existing 
projects can continue to use their names. This also allows the PyPI admins to 
selectively give someone the same name as a stdlib module if needed.

Just to close the loop on this, I believe the ones identified here have all be 
removed from PyPI along with several others by the same author. If you come 
across any others feel free to point them out.

> On Oct 20, 2017, at 1:44 PM, Erik Bray  wrote:
> 
> Hi all,
> 
> Sorry if this has come up before--I don't remember if it has. A recent
> question on StackOverflow [1] alerted to me to the fact that there is
> a package named "os" on PyPI: https://pypi.python.org/pypi/os
> 
> *Thankfully* it is:
> 
> a) Malformed--the package tarball isn't built correctly and it doesn't
> install with pip
> b) Not (currently!) evil: It just raises a RuntimeError telling you
> not to "pip install os"
> 
> That said, I think such packages should be prevented from being
> uploaded at all.  Naturally, the list of stdlib modules is a moving
> target, but not *that* fast-moving.
> 
> Conversely, I don't think new modules added to the stdlib should use
> the name of a package on PyPI, or at least should be prevented from
> being uploaded for Python versions equal to or later than the version
> in which that module was added to the stdlib.
> 
> Thanks,
> Erik
> 
> 
> [1] 
> https://stackoverflow.com/questions/46853112/python-pip-install-os-windows-errno-2

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Donald Stufft]

> On Oct 21, 2017, at 10:30 PM, Nick Coghlan  wrote:
> 
> However, none of that impacts the question of whether `pip.main()` runs code 
> in the current process or implicitly runs it in a subprocess - `pip` doesn't 
> import the modules it installs either way, so it will all look the same as 
> far as the import system is concerned.


That of course also ignores things like “foo.py optionally imports bar.py if it 
is available”, with something like:

try:
import bar
except ImportError:
bar = None


If you then did ``import foo``, noticed the optional features powered by bar 
weren’t added, you would also have to reload ``foo`` (and track down any 
references to it!). Reloading modules in Python is hard :/


The additional thing here is that the import system isn’t the only cache at 
play either— pkg_resources builds a cache of installed packages at import time 
too and that has to get invalidated somehow as well. This took pip like 3 or 4 
versions to get right because we tried to detect what version of pip was 
installed _after_ we installed all the versions in order to stop doing the 
outdated warning with ``pip install -U pip`` and I’m still not entirely 
convinced we’re not erroneously spitting out the warning in some cases.

The end result of all of this is unless you’re really really careful and design 
things just right and know exactly how not only yourself, but all your 
dependencies are written (and track how they change in different versions!) and 
also how things that use _you_ are going to import you and your dependencies… 
it’s basically playing whack a mole and is almost never worth the effort.

Re: PyPI JSON API redirect loop for all unpublished packages

[Donald Stufft]

> On May 16, 2018, at 5:06 PM, Doug Hellmann  wrote:
> 
> 
> 
> We had an issue testing redirect rules in the OpenStack documentation
> build, so we built a tool to test the .htaccess files. I don't know
> if whereto would be directly useful (maybe you're not using Apache?),
> but I thought I'd mention it in case someone thinks it is. I would also
> happily take patches to make it read some other input format, if needed.
> 
> https://docs.openstack.org/whereto/latest/
> 
> 
> 


Unfortunately, it was an issue with the app itself rather than the web server 
(we don’t actually use a web server to handle any logic besides generic 
buffering and passing on requests to our app server).

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Donald Stufft]

> On Oct 20, 2017, at 9:30 AM, Paul Moore  wrote:
> 
> On 20 October 2017 at 14:26, Matthew Brett  wrote:
>> Thanks for the heads-up.
>> 
>> Will y'all be doing a PyPI pre-release so we can test with `pip
>> install --pre -U pip`?
> 
> We've not yet decided on that. Traditionally I don't think we have
> done so, but I'm inclined to think it's a good idea. It might not be
> until noticeably closer to the release, though…
> 

I used to cut pre-releases for pip, and after awhile I gave up on doing them 
because it felt like nobody ever actually reported any issues with them 
anyways, and it wasn’t until we cut the final release that we started finding 
bugs with them. I don’t have any problem with us starting to issue them again 
though and seeing if we start catching issues earlier this time.

Re: statistics on downloads/request.logs

[Donald Stufft]

https://packaging.python.org/guides/analyzing-pypi-package-downloads/ 


> On Mar 15, 2018, at 7:16 PM, kenneth topp  wrote:
> 
> Hello,
> 
> Are there any statistics of top downloaded pacakges/ api calls?  If not, can 
> we make 30/90 days of anonymized logs available so projects can be created to 
> analyze the data?  I'm interested in most popular packages.
> 
> I searched the forum and didn't see any threads on this topic, any pointers 
> would be appreciated.
> 
> Thanks,
> 
> Ken

Re: Impending silent breakage of pip / macOS likely to cause severe confusion

[Donald Stufft]

> On Apr 6, 2018, at 4:28 PM, Matthew Brett  wrote:
> 
> 
> Ouch - it does look like we're in a very difficult situation, which
> might apply to a significant number of users.
> 
> I think the worst option is what we have ramping up in the brownout,
> which is the silent failure to upgrade or find packages.   Yes, you
> get something informative with the -v option if you think to try it,
> but I think the confusion from the not -v state is of much greater
> harm than the help from the -v message.
> 
> The better option, I believe, is just to go straight to the SSL error.
> At least that will set people Googling.  They might be annoyed, but at
> least they won't be mystified.
> 
> Ideally, that would happen as late as possible, to give people the
> maximum chance to upgrade with the usual pip upgrade commands. One of
> the nasty features of this one is that it breaks the normal pip
> upgrade method.  Can we push back the SSL error until it's forced upon
> us in June?
> 

Not without delaying the launch of Warehouse until after June (at which point 
we will have already run out of MOSS grant money and won’t have people with 
paid, dedicated time on hand to sort out any issues that crop up from the 
launch). This is because portions of Warehouse are already mandatory TLSv1.2 
and that can’t be rolled back (Warehouse is new enough on Fastly it never had 
TLSv1.0/1.1 support).

It’s not great, but it’s the least bad option we have right now.

Re: Impending silent breakage of pip / macOS likely to cause severe confusion

[Donald Stufft]

> On Apr 6, 2018, at 12:30 PM, Matthew Brett <matthew.br...@gmail.com> wrote:
> 
> Hi,
> 
> On Fri, Apr 6, 2018 at 4:49 PM, Donald Stufft <don...@stufft.io 
> <mailto:don...@stufft.io>> wrote:
>> 
>> On Apr 6, 2018, at 11:25 AM, Matthew Brett <matthew.br...@gmail.com> wrote:
>> 
>> 
>> As of 8th April, pip will break, largely in silence.  As used with
>> default command line options, pip will stop seeing anything on pypi.
>> It will tell the user that it can't find packages that do exist on
>> pypi, that that users are up to date for packages that have later
>> versions on pypi - including pip.  There are no messages to warn the
>> user what has happened or how to fix it.  It is easy to think you've
>> upgraded to the latest version, when you have not.
>> 
>> 
>> This isn’t quite accurate:
>> 
>> This is not a blanket issue for anyone on macOS < 10.13. It *only* affects
>> people using a Python that links against the ancient OpenSSL provided by
>> Apple, of which the #1 cause is System Python on macOS < 10.13, however it
>> is also the 2.7 Python.org macOS installer on any version of Python. All
>> other forms of getting Python, to my knowledge, link against a newer version
>> of OpenSSL (or against LibreSSL in System Python on macOS 10.13). [1]
> 
> I think that list is for current installers - then there's the tail of
> older installs, of which I am one (a Python.org <http://python.org/> 3.5 
> install).   Only
> relevant, because this is going to be a complex patchwork of affected
> users.
> 
>> This affects a small percentage of users, Of the downloads from PyPI in the
>> last 7 days that originated from a macOS system, 6% of them would have
>> failed, 94% of them would have succeeded.
> 
> 6% of very confused users is a lot.   But - I guess some proportion of
> that 100% is continuous integration installs?   These will tend to
> have very up to date Pythons.  Is there any way to estimate how many
> non-bot users will be affected?

Not that I am aware of. Note— that is 6% of downloads, not 6% of users, that 
could have been a whole lot users each downloading a single package, or one 
user downloading thousands, we can’t tell.

> 
>> Once we are at 100% unavailability, we can ask our CDN to disable TLSv1.0
>> and TLSv1.1 completely, which will raise an OpenSSL error message instead of
>> a HTTP error message. That will look like (ignore the index-url, used that
>> because it already has TLSv1.0/1.1 disabled completely):
>> 
>>$ /usr/local/bin/python2.7 -m pip install --index-url
>> https://files.pythonhosted.org/ --upgrade pip
>>Could not fetch URL https://files.pythonhosted.org/pip/: There was a
>> problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION]
>> tlsv1 alert protocol version (_ssl.c:661) - skipping
>>Requirement already up-to-date: pip in
>> /Library/Frameworks/Python.framework/Versions/2.7.14_10_6/lib/python2.7/site-packages
>>You are using pip version 9.0.1, however version 9.0.3 is available.
>>You should consider upgrading via the 'pip install --upgrade pip'
>> command.
> 
> Yes - that's better than the current silent break, but still it's a
> shame that the message is so obscure given the long lead-in.
> 
>> Finally, even If the above weren’t true, there really isn’t much we can do
>> either way. We can’t go back in time and change the already released version
>> of pip, and we only have so many ways that PyPI can signal an error to pip
>> one is using an HTTP status code (what the brownouts do now) and one is
>> rejecting the connection with a TLS error (what the above snippet does).
>> Beyond that, any improvements require getting users to upgrade their version
>> of pip, and if we can get them to upgrade for a better error message,
>> presumably we can get them to upgrade for the SecureTransport fallback in
>> 9.0.3+.
> 
> I was hoping against hope that there was another way of generating an
> informative error from the server that would force itself on the
> user's attention, perhaps by subverting another mechanism.  Is there
> really no way for pypi to transmit an informative error when it
> detects a request from an affected system?

No, there’s not. pip makes HTTP requests and there’s no place for extra 
metadata attached those requests except in the HTTP status code (which as you 
noted, pip swallows by default because historically we didn’t know if the URL 
was expected to work or not). The simple API wasn’t really designed, it evolved 
out of the primordial ooze.

Re: Impending silent breakage of pip / macOS likely to cause severe confusion

[Donald Stufft]

> On Apr 6, 2018, at 5:06 PM, Matthew Brett  wrote:
> 
> OK - so our hard deadline is the planned Warehouse launch on April
> 16th?   I would argue for going straight to the SSL error at that
> point, and turning off the current brownout and April 8th TLS 1.0 shut
> down.  Is that possible?   Do other Macolytes agree with me that that
> would be less confusing?  In the mean time, would it be possible to
> put out some big announcements following up on the originals, giving
> the SSL error, to seed Google searches, and prime memories?


We’ve modified the plan so that instead of the brownout style error lasting 
until the 16th, we’re going to switch to the hard failure tomorrow with the 
100% brownout failure happening today (and yesterday). We didn’t want to move 
straight to the hard failure incase we needed to roll it back for some reason. 
We don’t want to wait until the 16th to avoid lumping too many changes onto a 
single day (so we don’t have to deal with potential fallout of too many 
different changes on a single day).

Hopefully that works for everyone.

Re: Impending silent breakage of pip / macOS likely to cause severe confusion

[Donald Stufft]

> On Apr 6, 2018, at 11:25 AM, Matthew Brett  wrote:
> 
> 
> As of 8th April, pip will break, largely in silence.  As used with
> default command line options, pip will stop seeing anything on pypi.
> It will tell the user that it can't find packages that do exist on
> pypi, that that users are up to date for packages that have later
> versions on pypi - including pip.  There are no messages to warn the
> user what has happened or how to fix it.  It is easy to think you've
> upgraded to the latest version, when you have not.
> 

This isn’t quite accurate:

This is not a blanket issue for anyone on macOS < 10.13. It *only* affects 
people using a Python that links against the ancient OpenSSL provided by Apple, 
of which the #1 cause is System Python on macOS < 10.13, however it is also the 
2.7 Python.org  macOS installer on any version of Python. 
All other forms of getting Python, to my knowledge, link against a newer 
version of OpenSSL (or against LibreSSL in System Python on macOS 10.13). [1]

This affects a small percentage of users, Of the downloads from PyPI in the 
last 7 days that originated from a macOS system, 6% of them would have failed, 
94% of them would have succeeded.

Once we are at 100% unavailability, we can ask our CDN to disable TLSv1.0 and 
TLSv1.1 completely, which will raise an OpenSSL error message instead of a HTTP 
error message. That will look like (ignore the index-url, used that because it 
already has TLSv1.0/1.1 disabled completely):

$ /usr/local/bin/python2.7 -m pip install --index-url 
https://files.pythonhosted.org/ --upgrade pip
Could not fetch URL https://files.pythonhosted.org/pip/: There was a 
problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] 
tlsv1 alert protocol version (_ssl.c:661) - skipping
Requirement already up-to-date: pip in 
/Library/Frameworks/Python.framework/Versions/2.7.14_10_6/lib/python2.7/site-packages
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.


Finally, even If the above weren’t true, there really isn’t much we can do 
either way. We can’t go back in time and change the already released version of 
pip, and we only have so many ways that PyPI can signal an error to pip one is 
using an HTTP status code (what the brownouts do now) and one is rejecting the 
connection with a TLS error (what the above snippet does). Beyond that, any 
improvements require getting users to upgrade their version of pip, and if we 
can get them to upgrade for a better error message, presumably we can get them 
to upgrade for the SecureTransport fallback in 9.0.3+.


[1] You can see more information about which Python.org  
macOS installers are affected here: 
https://github.com/pypa/warehouse/issues/3293#issuecomment-378468534 


- Donald

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Donald Stufft]

> On Apr 14, 2018, at 4:57 PM, Matthew Brett  wrote:
> 
> Is the suggestion to use the `_internal` import, or carry a copy of
> the pep425tags code myself, that I have to keep up to date with the
> internal pip copy?  That would also involve me copying the `glibc`
> part of the code.  I see that the `wheel` package has an old copy of
> that code too, which doesn't deal with manylinux wheels.You
> probably saw that `pip-tools` ended up vendoring the whole of pip9
> [1].

The best solution is to figure out what APIs people need, and either add them 
to packaging and have pip consume that as well as anyone else, or make a new 
library for the same.

If that’s unacceptable, vendoring or version pinning is probably the best 
option.

Re: suggestion: using "black" for Warehouse formatting

[Donald Stufft]

So I considered that (and right now, it has a show stopper bug which means we 
can’t start using it), but given (A) pinning to a specific version (which means 
we’re not going to be open to _new_ bugs or changes in behavior and (B) we’re 
still running flake8 (at least in my branch) but I think the risk is fairly 
low, particularly since its not a runtime library.

At worst we get crappy behavior and just disable it, we won’t have to revert 
the changes it has made since they still follow our flake8 requirements.

> On Mar 22, 2018, at 4:14 PM, Ian Stapleton Cordasco 
>  wrote:
> 
> If you look at the issues I'm not sure it's presently stable enough for 
> Warehouse. There are quite a few bugs which is normal for such a new project
> 
> Sent from my phone with my typo-happy thumbs. Please excuse my brevity
> 
> On Mar 22, 2018 12:18, "Sumana Harihareswara"  > wrote:
> black > is an 
> opinionated code formatter.
> It is currently a pre-release in alpha.
> 
> https://github.com/pypa/warehouse/pull/3367 
> 
> 
> Donald would like to add black to our linter and format all Warehouse
> code with black going forward.
> 
> Comment on the pull request if you have thoughts.
> --
> Sumana Harihareswara
> Changeset Consulting
> https://changeset.nyc