[Distutils] Re: Packaging optional, arch-dependent, pre-built libraries

[Wes Turner]

Is there some easy way to solve this specifically with cibuildwheel?

https://github.com/joerick/cibuildwheel :

> - Builds manylinux, macOS 10.9+, and Windows wheels for CPython and PyPy
> - Works on GitHub Actions, Azure Pipelines, Travis CI, AppVeyor,
CircleCI, and GitLab CI
>   - Bundles shared library dependencies on Linux and macOS through
auditwheel and delocate
>   - Runs your library's tests against the wheel-installed version of your
library

On Mon, Apr 5, 2021, 12:18 Tzu-ping Chung  wrote:

> If a file is not built or linked against, a dll in your wheel is
> essentially a plain data file from Python packaging’s perspective, no
> different from e.g. a text file. So you’re looking in the wrong direction
> for solutions.
>
> I believe the issue PyInstaller has with your package is that, since
> PyInstaller compiles a program into an executable,
> ctypes.util.find_library() won’t work (since there is no actual dll to
> find). If you know for sure the dll will be available, you can copy the
> binary to a temporary location (the “official” way to do this is through
> importlib.resources.path[1]), and use the path to load the dll directly
> instead.
>
>
> [1]: https://importlib-resources.readthedocs.io/en/latest/
>
> --
> Tzu-ping Chung (@uranusjr)
> uranu...@gmail.com
> https://uranusjr.com
>
> On 03/4/2021, at 06:57, Vincent Pelletier  wrote:
>
> Hello,
>
> I'm the author of python-libusb1, a pure-python ctypes wrapper for
> libusb1.
>
> Until recently, I had been purely relying on OS-linker-provided libusb1
> (distro-installed on GNU/Linux and *BSD, fink/macports/... on OSX, ...).
>
> Then, I've been requested to bundle the libusb1 dll on windows (x86 and
> x86_64 wheels) because otherwise distributions seems exceedingly
> painful for applications using my module. With some extra code to
> setup.py to fetch, unzip and copy[1] the dlls, plus a now even more
> multi-stage distribution process (sdist, both windows wheels, in
> addition to the existing sign and twine steps), and it ipso facto works.
>
> Now, I'm asked to add pyinstaller compatibility, as it on its own
> overlooks the dll. Which makes me feel that I am maybe not using the
> best possible way to bundle these.
>
> From my reading of distutils and setuptools, my understanding is that a
> package is that non-pure-python packages contain:
> - stuff they built themselves (build_ext & friends)
> - third-party libraries that the stuff they built themselves is linked
>  against
> Having nothing to build, I cannot seem to reach the library inclusion
> step.
>
> What is the recommended way of bundling a third-party-built
> arch-dependent library in an otherwise pure-python package ?
>
> [1]
> https://github.com/vpelletier/python-libusb1/blob/49f7f846bdd3c3d0f2ec3a01c23ed69885cf63a4/setup.py#L48-L58
>
> Regards,
> --
> Vincent Pelletier
> GPG fingerprint 983A E8B7 3B91 1598 7A92 3845 CAC9 3691 4257 B0C1
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/J6DDMSAC73TVHFMBIEIT6Z6ZB7HGSQGJ/
>
>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/OBAR3NYBPXFG3G2LU6YWA2CCVQO2UFNA/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/NDFXCQEXJ3R6OPLPDMWRYM656JOCC4Q7/

[Distutils] Re: pip and missing shared system system library

[Wes Turner]

On Sun, Aug 9, 2020, 3:28 PM Wes Turner  wrote:

> Are you requesting an implementation of autotools / autoconf / pkg-config
> / libtool in Python, in setuptools?
>
> Existing workarounds for building and distributing portable binaries:
>
> W/ shared library dependencies:
> - auditwheel & manylinux
>

"""
`auditwheel show`: shows external shared libraries that the wheel depends
on (beyond the libraries included in the manylinux policies), and checks
the extension modules for the use of versioned symbols that exceed the
manylinux ABI.

`auditwheel repair`: copies these external shared libraries into the wheel
itself, and automatically modifies the appropriate RPATH entries such that
these libraries will be picked up at runtime. This accomplishes a similar
result as if the libraries had been statically linked without requiring
changes to the build system. Packagers are advised that bundling, like
static linking, may implicate copyright concerns.
"""

https://github.com/pypa/auditwheel#overview

- package managers which support arbitrary binary packages in languages
> other than python:
>   - conda
>   - RPM / DEB / ...
> - bdist_rpm
> - bdist_deb
> - FPM
>
> W/ static dependencies:
> - zipapp
> - bazel / buck build / pants build (BUILD files)
> - py2app, py2exe, pyinstaller,
>   https://github.com/vinta/awesome-python#distribution
>
> On Sun, Aug 9, 2020, 3:05 PM David Mathog  wrote:
>
>> On Sun, Aug 9, 2020 at 10:21 AM Ned Deily  wrote:
>> > Just to be clear, pkg-config is not part of any Posix standard, AFAIK,
>> so you cannot depend on it being available.
>>
>> Understood.  However, if that is not employed what reasonable method
>> remains for implementing "Requires-External"?  The only thing I can
>> think of is to specify exact library or program names, like
>>
>> Requires-External gcc
>> Requires-External libpng.so
>>
>> and those could be found by searching the whole directory tree.  That
>> might even be efficient if updatedb/locate are available.  However
>> going that way, how would one determine version compatibility on a
>> library?  Doing it through the package manager may be possible, but it
>> is a multistep process:
>>
>> 1.  lookup libpng.so -> PATHPNG
>> 2.  rpm -q --whatprovides $PATHPNG -> name of package
>> 3.  analyze "name of package" for version information
>>
>> Much easier one suspects to install pkg-config on systems which do not
>> yet have it than to completely reimplement it.
>>
>> Does OS X have something which is equivalent to pkg-config, or is
>> there just no way to look up this sort of information on that OS?
>>
>> Regards,
>>
>> David Mathog
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/distutils-sig@python.org/message/BCYVPMEGXLU7YQJUCCQDV5BT7E22EH7M/
>>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/EBZ6LGL5ANUKH4MM6XQD5U5TFVRQBDGK/

[Distutils] Re: pip and missing shared system system library

[Wes Turner]

Are you requesting an implementation of autotools / autoconf / pkg-config /
libtool in Python, in setuptools?

Existing workarounds for building and distributing portable binaries:

W/ shared library dependencies:
- auditwheel & manylinux
- package managers which support arbitrary binary packages in languages
other than python:
  - conda
  - RPM / DEB / ...
- bdist_rpm
- bdist_deb
- FPM

W/ static dependencies:
- zipapp
- bazel / buck build / pants build (BUILD files)
- py2app, py2exe, pyinstaller,
  https://github.com/vinta/awesome-python#distribution

On Sun, Aug 9, 2020, 3:05 PM David Mathog  wrote:

> On Sun, Aug 9, 2020 at 10:21 AM Ned Deily  wrote:
> > Just to be clear, pkg-config is not part of any Posix standard, AFAIK,
> so you cannot depend on it being available.
>
> Understood.  However, if that is not employed what reasonable method
> remains for implementing "Requires-External"?  The only thing I can
> think of is to specify exact library or program names, like
>
> Requires-External gcc
> Requires-External libpng.so
>
> and those could be found by searching the whole directory tree.  That
> might even be efficient if updatedb/locate are available.  However
> going that way, how would one determine version compatibility on a
> library?  Doing it through the package manager may be possible, but it
> is a multistep process:
>
> 1.  lookup libpng.so -> PATHPNG
> 2.  rpm -q --whatprovides $PATHPNG -> name of package
> 3.  analyze "name of package" for version information
>
> Much easier one suspects to install pkg-config on systems which do not
> yet have it than to completely reimplement it.
>
> Does OS X have something which is equivalent to pkg-config, or is
> there just no way to look up this sort of information on that OS?
>
> Regards,
>
> David Mathog
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/BCYVPMEGXLU7YQJUCCQDV5BT7E22EH7M/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/7TCZJT2RDE36BJKUDU6JVTIUYHGVDVQY/

[Distutils] Re: twine upload & network robustness

[Wes Turner]

warehouse.forklift.legacy._is_valid_dist_file()
https://github.com/pypa/warehouse/blob/master/warehouse/forklift/legacy.py#L603

https://github.com/pypa/warehouse/blob/master/docs/api-reference/legacy.rst#upload-api
:

> The API endpoint served at upload.pypi.org/legacy/ is Warehouse’s
emulation of the legacy PyPI upload API. This is the endpoint that tools
such as twine use to upload distributions to PyPI.

https://warehouse.pypa.io/api-reference/legacy/#upload-api

^^ this page is not in the top-level warehouse TOC

https://warehouse.pypa.io/application/#file-and-directory-structure





On Fri, Aug 7, 2020, 12:34 PM Dustin Ingram  wrote:

> The latter.
>
> On Fri, Aug 7, 2020 at 4:37 AM Robin Becker  wrote:
>
>> Because of covid-19 I am working a lot from home so my network is likely
>> less robust.
>>
>> I was wondering about what happens if I upload with twine to pypi and
>> halfway through my network fails.
>>
>> Is the uploaded filename remembered so I cannot retry when my network
>> comes back?
>>
>> Or is the upload somehow atomic so that it has to complete(be checked)
>> before the file is entered into pypi?
>> --
>> Robin Becker
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/distutils-sig@python.org/message/SWHFPMYFVRLGXYRPOCTBQS7FKCUX3MUB/
>>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/T2ZIWXYADJKOX2D3OVR4EJMT6ENC5E2Y/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/FTOAM5POWBMI6GDVTOK5WZCENEREYWQ6/

[Distutils] Re: Archive this list & redirect conversation elsewhere?

[Wes Turner]

How to subscribe to all threads of discourse

On Tue, Aug 4, 2020 at 6:02 PM Brett Cannon  wrote:

>
>
> On Thu, Jul 30, 2020 at 8:41 AM Wes Turner  wrote:
>
>> I confess that I don't even know how to subscribe to all threads of a
>> discourse.
>>
>> - [ ] How to subscribe to all threads of discourse
>>
>
> Go to the category you care about, e.g.
> https://discuss.python.org/c/packaging/14, and if you look in the right
> side next to "+ New Topic" you will see a bell. you can click that and
> choose to what level you want to follow new topics (only new threads,
> notification of all comments, direct notification of all comments, etc.).
>
> -Brett
>

1. Go to the category page: e.g. https://discuss.python.org/c/packaging/14
2. Click the bell at the top right
3. Select "Watching"

Does this send a new email whenever someone edits a post that's already
been emailed out?

(edit) If you change your notification settings for a particular thread by
clicking on the bell, that setting is independent of your category-level
setting. So, you can "Watching" all on the category and "Mute" a particular
thread or vice-versa.


>
>
>>
>> So, I'd miss security or release announcements only posted to discourse
>> and not distutils-sig (or pypa-dev, which IMHO has the more appropriate
>> scope name in that packaging and PyPI are somewhat inseparable)
>>
>> Is there some convenient way to cryptographically-sign Discourse posts?
>> Maybe with e.g. OpenPGP.js? Or is that unnecessary these days.
>>
>> Discourse in Python would be great to have.
>>
>> https://github.com/discourse/discourse (Rails)
>>
>> https://gitlab.com/mailman/mailman
>>
>> https://gitlab.com/mailman/postorius (Django)
>>
>> Having mailing list discussions archived in one's searchable inbox is
>> underrated, IMHO
>>
>> Being able to link to specific messages within a thread using permalinks
>> is very useful; though most won't even read it
>>
>> On Thu, Jul 30, 2020, 8:24 AM Jeremy Stanley  wrote:
>>
>>> On 2020-07-30 11:51:24 +0100 (+0100), Paul Moore wrote:
>>> [...]
>>> > 1. Will dropping distutils-sig mean that people who prefer
>>> > interacting here lose their voice in packaging discussions?
>>>
>>> Probably not. I think the fact that most of the list's prior
>>> conversation has already moved to Discourse means this is already
>>> the case, and so closing this ML is probably more a reflection of
>>> the reality that those voices are effectively absent in relevant
>>> conversations now.
>>>
>>> > 2. Will having one fewer "recommended place" to start discussions
>>> > make it easier for new participants to get involved?
>>> [...]
>>>
>>> Maybe. But as I've seen in many other communities, discussions start
>>> organically in a variety of places and platforms, and are rarely
>>> constrained by community consensus "recommendations" of venue.
>>> --
>>> Jeremy Stanley
>>> --
>>> Distutils-SIG mailing list -- distutils-sig@python.org
>>> To unsubscribe send an email to distutils-sig-le...@python.org
>>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/distutils-sig@python.org/message/PCUVODMMWRAPGXA6PUJ5KHE5YU23T42O/
>>>
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/distutils-sig@python.org/message/6ULNQ4JH4EKYUKHW7JRWZA4RTU32SDUL/
>>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/VUIDZ36V6HYTPUIKAGH3MDEEUPC4IFKF/

[Distutils] Re: Archive this list & redirect conversation elsewhere?

[Wes Turner]

I confess that I don't even know how to subscribe to all threads of a
discourse.

- [ ] How to subscribe to all threads of discourse

So, I'd miss security or release announcements only posted to discourse and
not distutils-sig (or pypa-dev, which IMHO has the more appropriate scope
name in that packaging and PyPI are somewhat inseparable)

Is there some convenient way to cryptographically-sign Discourse posts?
Maybe with e.g. OpenPGP.js? Or is that unnecessary these days.

Discourse in Python would be great to have.

https://github.com/discourse/discourse (Rails)

https://gitlab.com/mailman/mailman

https://gitlab.com/mailman/postorius (Django)

Having mailing list discussions archived in one's searchable inbox is
underrated, IMHO

Being able to link to specific messages within a thread using permalinks is
very useful; though most won't even read it

On Thu, Jul 30, 2020, 8:24 AM Jeremy Stanley  wrote:

> On 2020-07-30 11:51:24 +0100 (+0100), Paul Moore wrote:
> [...]
> > 1. Will dropping distutils-sig mean that people who prefer
> > interacting here lose their voice in packaging discussions?
>
> Probably not. I think the fact that most of the list's prior
> conversation has already moved to Discourse means this is already
> the case, and so closing this ML is probably more a reflection of
> the reality that those voices are effectively absent in relevant
> conversations now.
>
> > 2. Will having one fewer "recommended place" to start discussions
> > make it easier for new participants to get involved?
> [...]
>
> Maybe. But as I've seen in many other communities, discussions start
> organically in a variety of places and platforms, and are rarely
> constrained by community consensus "recommendations" of venue.
> --
> Jeremy Stanley
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/PCUVODMMWRAPGXA6PUJ5KHE5YU23T42O/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/6ULNQ4JH4EKYUKHW7JRWZA4RTU32SDUL/

[Distutils] Re: Improving Communication

[Wes Turner]

- [ ] Communications flowchart
- [ ] Bot for mailing list <> Discourse trackbacks

On Sun, May 6, 2018, 5:33 AM Pradyun Gedam  wrote:

> Sorry for poking an old thread. I actually wanted to respond earlier but
> life got in the way.
>
> Just a small point though.
>
> On Sat, Apr 21, 2018 at 7:37 AM Donald Stufft  wrote:
>
>> Currently in the packaging space, we have a number of avenues for
>> communication, which are:
>>
>> - distutils-sig
>> - pypa-dev
>> - virtualenv-users
>> - Other project specific mailing lists
>> - IRC
>> - gitter
>> - Various issue trackers spread across multiple platforms.
>> - Probably more places I’m not remembering.
>>
>> The result of this is that all discussion ends up being super fractured
>> amongst the various places. Sometimes that is exactly what you want (for
>> instance, someone who is working on the wheel specs probably doesn’t care
>> about deprecation policy and internal module renaming in pip) and sometimes
>> that ends up being the opposite of what you want (for instance, when you’re
>> describing something that touches PyPI, setuptools, flit, pip, etc all at
>> once).
>>
>> Theoretically the idea is that distutils-sig is where cross project
>> reaching stuff goes, IRC/gitter is where real time discussion goes, and the
>> various project mailing lists and issue trackers are where the project
>> specific bits go. The problem is that often times doesn’t actually happen
>> in practice except for the largest and most obvious of changes.
>>
>
> ​For real time discussions, something like pypa/discuss and
> pypa/dev-discuss (or some other spelling) on gitter, bridged to #pypa and
> #pypa-dev would allow us to merge those two channels. That would be nice.
>
> It seems that Julia does something like this:
> https://groups.google.com/forum/#!topic/julia-users/ImKYzqHXA90
>
> I think our current “communications stack” kind of sucks, and I’d love to
>> figure out a better way for us to handle this that solves the sort of weird
>> “independent but related” set of projects we have here.
>>
>> From my POV, a list of our major problems are:
>>
>> * Discussion gets fractured across a variety of platforms and locations,
>> which can make it difficult to actually keep up with what’s going on but
>> also to know how to loop in someone relevant if their input would be
>> valuable. You have to more or less simultaneously know someone’s email,
>> Github username, IRC nick, bitbucket username, etc to be able to bring
>> threads of discussion to people’s attention.
>>
>> * It’s not always clear to users where a discussion should go, often
>> times they’ll come to one location and need to get redirected to another
>> location. If any discussion did happen in the incorrect location, it tends
>> to need to get restarted in the new location (and depending on the specific
>> platform, it may be impossible to actually redirect everyone over to the
>> proper location, so you again, end up fractured with the discussion
>> happening in two places).
>>
>> * A lot of the technology in this stack is particularly old, and lacks a
>> lot of the modern day affordances that newer things have. An example is
>> being able to edit a discussion post to fix typos that can hinder the
>> ability of others to actually understand whats being talked about. In your
>> typical mailing list or IRC there’s no mechanism by which you can edit an
>> already sent message, so your only option is to either let the problem ride
>> and hope it doesn’t trip up too many people, or send an additional message
>> to correct the error. However these show up as additional, later messages
>> which someone might not even see until they’ve already been thoroughly
>> confused by the first message (since people tend to read email/IRC in a
>> linear fashion).
>>   - There is a lot of things in this one, other things are things like
>> being able to control in a more fine grained manner what email you’re going
>> to get.
>>   - Holy crap, formatting and typography to make things actually readable
>> and not a big block of plaintext.
>>
>> * We don’t have a clear way for users to get help, leaving users to treat
>> random issues, discussion areas, etc as a support forum, rather than some
>> place that’s actually optimized for that. Some of this ties back into some
>> of the earlier things too, where it’s hard to actually redirect discussions
>>
>> These aren’t *new* problems, and often times the existing members of a
>> community are the least effected becasue they’ve already spent effort
>> learning the ins and outs and also curating a (typically custom) workflow
>> that they’ve grown accustomed too. The problem with that is that often
>> times that means that new users are left out, and the community gets
>> smaller and smaller as time goes on as people leave and aren’t replaced
>> with new blood, because they’re driven off but the issues with the stack.
>>
>> A big part of the place this is coming from, is me sitting back and
>> 

[Distutils] Re: pip resolver work chugging along

[Wes Turner]

On Tue, Mar 24, 2020 at 5:24 PM Tzu-ping Chung  wrote:

> To expand a little on the topic, there are multiple abstraction layers
> required to make the new resolver useful as a separate package.
>
> ResolveLib (mentioned in Paul’s message) is an dependency resolver
> implementation in abstract, entirely divorced of Python packaging. While it
> can be used by other tools, the implementer would need to provide the
> necessary parts to “teach” it how understand Python packages.
>
> The ongoing pip resolver work is exactly that, to integrate ResolveLib
> into pip by wiring it onto pip’s internal representation of Python
> packages. This is unfortunately not really reusable for other tools, since
> pip’s current model on Python packages is too deeply entangled into various
> parts of the code base.
>
> Some extra work is needed to make ResolveLib really useful as a reusable
> Python dependency resolver for other projects. One way would be to
> implement a wrapper library to provide the same Python package semantics to
> ResolveLib, but with more reusable models such as pypa/packaging,
> importlib-metadata, etc. The current pip resolver work would be able to
> offer much knowledge on dealing with various hairy problems, but the code
> written for pip wouldn’t be directly reusable.
>
> TP
>

"New Resolver: technical choices" compares various resolvers.
https://github.com/pypa/pip/issues/7406

FWICS, more test cases should be the immediate objective.

Factoring out to make a generic resolver would be redundant? Would existing
interfaces for e.g. libsolv (dnf) and pycosat (conda) be sufficient?


>
>
> > On 25/3/2020, at 03:02, Paul Moore  wrote:
> >
> > It's already available as a separate package:
> > https://pypi.org/project/resolvelib/
> >
> > Paul
> >
> > On Tue, 24 Mar 2020 at 18:52, Brett Cannon  wrote:
> >>
> >> I couldn't find this in the blog post but is the plan to make the
> resolver a separate package so other tools can use it? Or is the plan
> perhaps to get it working in pip first and to break it out later?
> >> --
> >> Distutils-SIG mailing list -- distutils-sig@python.org
> >> To unsubscribe send an email to distutils-sig-le...@python.org
> >> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> >> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/CLWGXEPNNMZ6YI5WWJ3ZKWWY5WVQOCAE/
> > --
> > Distutils-SIG mailing list -- distutils-sig@python.org
> > To unsubscribe send an email to distutils-sig-le...@python.org
> > https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> > Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/43NVR5W7YKVFBDY5FSMBWQPLKKG2V2EP/
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/VLSD5OXWR6YLAFXQZWCZSCLXPNUUBSU4/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/W2OKN24JMKO5TTMLCIJ247KDVFFOA6XW/

[Distutils] Re: pip resolver work chugging along

[Wes Turner]

 A few more project management resources (A bit OT, but intended to be
helpful:)

## Ways to integrate GitHub with other data sources:
### GitHub APIs:
- https://developer.github.com/v3/
- https://developer.github.com/v4/
- https://developer.github.com/v3/libraries/
- https://github.com/cli/cli
- https://github.com/github/hub
  - https://hub.github.com/#developer

### IFTTT
- https://ifttt.com/github
  - Create a Github issue when there's a new vulnerability published

### Zapier
- https://zapier.com/apps/github/integrations
  - Create a GitHub issue when a new survey is posted <--

### Bots
- https://github.com/python/miss-islington
- https://probot.github.io/apps/
  - https://probot.github.io/apps/polls/
  - https://probot.github.io/apps/triage-new-issues/
> Adds `triage` label to newly-created issues and then removes it when
other label(s) are added

### Zulip
- https://zulipchat.com/integrations/version-control
  - https://zulipchat.com/integrations/doc/github
  - https://zulipchat.com/integrations/doc/github_detail

On Tue, Mar 24, 2020 at 5:18 PM Wes Turner  wrote:

> Issue and Pull Request Templates:
> - https://github.com/devspace/awesome-github-templates
> - Sometimes, a blank issue template option seems most welcoming
>
> When users have issues with the resolver, the most helpful thing for them
> to do may be to add a test case. IDK what the best way to do that would be?
> - https://github.com/sarugaku/resolvelib/tree/master/tests/functional
>   - Create a template test case directory? Or e.g. a YAML file to call
> pytest.mark.parametrize with?
>
> Ways to store comments in GitHub issues:
> - https://github.com/utterance/utterances
> - https://github.com/imsun/gitment
>
> On Tue, Mar 24, 2020 at 4:21 PM Sumana Harihareswara 
> wrote:
>
>> On 3/24/20 4:13 PM, Sumana Harihareswara wrote:
>> > Sounds like you'll be submitting a pull request for that documentation
>> &
>> > README change -- go ahead and ping me as @brainwane for review when you
>> > do that! Thanks.
>>
>> Sorry, that last bit was kinda snarky. Am making this change myself now.
>> --
>> Sumana Harihareswara
>> Changeset Consulting
>> https://changeset.nyc
>>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/ZCRDE3U5UMPSUTQ7QJJA2O7TWSPDFGD5/

[Distutils] Re: pip resolver work chugging along

[Wes Turner]

Issue and Pull Request Templates:
- https://github.com/devspace/awesome-github-templates
- Sometimes, a blank issue template option seems most welcoming

When users have issues with the resolver, the most helpful thing for them
to do may be to add a test case. IDK what the best way to do that would be?
- https://github.com/sarugaku/resolvelib/tree/master/tests/functional
  - Create a template test case directory? Or e.g. a YAML file to call
pytest.mark.parametrize with?

Ways to store comments in GitHub issues:
- https://github.com/utterance/utterances
- https://github.com/imsun/gitment

On Tue, Mar 24, 2020 at 4:21 PM Sumana Harihareswara 
wrote:

> On 3/24/20 4:13 PM, Sumana Harihareswara wrote:
> > Sounds like you'll be submitting a pull request for that documentation &
> > README change -- go ahead and ping me as @brainwane for review when you
> > do that! Thanks.
>
> Sorry, that last bit was kinda snarky. Am making this change myself now.
> --
> Sumana Harihareswara
> Changeset Consulting
> https://changeset.nyc
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/75EXJS57JSFRV3RFG575XBTT7LBGEE5R/

[Distutils] Re: pip resolver work chugging along

[Wes Turner]

I think it may be most likely that you'll get quality feedback through
GitHub Issues.
Is there an issue template with at least some of the 23 questions in the
survey?

Survey:
https://tools.simplysecure.org/survey/index.php?r=survey/index=989272=en

Here's the "Resolver refactorings" project board:
https://github.com/pypa/pip/projects/5

A note in the README and the docs regarding testing the new resolver might
be good too.

On Tue, Mar 24, 2020 at 3:55 PM Wes Turner  wrote:

> IIRC, the issue for this (or one of the issues for this great work) is:
> "New Resolver: Rollout, Feedback Loops and Development Flow"
> https://github.com/pypa/pip/issues/6536
>
> On Tue, Mar 24, 2020 at 3:03 PM Paul Moore  wrote:
>
>> It's already available as a separate package:
>> https://pypi.org/project/resolvelib/
>>
>> Paul
>>
>> On Tue, 24 Mar 2020 at 18:52, Brett Cannon  wrote:
>> >
>> > I couldn't find this in the blog post but is the plan to make the
>> resolver a separate package so other tools can use it? Or is the plan
>> perhaps to get it working in pip first and to break it out later?
>> > --
>> > Distutils-SIG mailing list -- distutils-sig@python.org
>> > To unsubscribe send an email to distutils-sig-le...@python.org
>> > https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>> > Message archived at
>> https://mail.python.org/archives/list/distutils-sig@python.org/message/CLWGXEPNNMZ6YI5WWJ3ZKWWY5WVQOCAE/
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/distutils-sig@python.org/message/43NVR5W7YKVFBDY5FSMBWQPLKKG2V2EP/
>>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/CBFSJ7GNDA6IBHNUXUI2747EFQTGCBSY/

[Distutils] Re: pip resolver work chugging along

[Wes Turner]

IIRC, the issue for this (or one of the issues for this great work) is:
"New Resolver: Rollout, Feedback Loops and Development Flow"
https://github.com/pypa/pip/issues/6536

On Tue, Mar 24, 2020 at 3:03 PM Paul Moore  wrote:

> It's already available as a separate package:
> https://pypi.org/project/resolvelib/
>
> Paul
>
> On Tue, 24 Mar 2020 at 18:52, Brett Cannon  wrote:
> >
> > I couldn't find this in the blog post but is the plan to make the
> resolver a separate package so other tools can use it? Or is the plan
> perhaps to get it working in pip first and to break it out later?
> > --
> > Distutils-SIG mailing list -- distutils-sig@python.org
> > To unsubscribe send an email to distutils-sig-le...@python.org
> > https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> > Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/CLWGXEPNNMZ6YI5WWJ3ZKWWY5WVQOCAE/
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/43NVR5W7YKVFBDY5FSMBWQPLKKG2V2EP/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/ESIGG7TR7ZZ4VEWI32GBIH3WTAKYV47N/

[Distutils] Re: The base docker images for manylinux appear to be private

[Wes Turner]

On Tue, Feb 25, 2020, 10:03 PM Wes Turner  wrote:

>
> Is there a reason the new manylinux does not just extend centos:6?
>

https://github.com/pypa/manylinux/blob/master/docker/glibc/README.rst :

> Summary: Because of
https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html,
this a CentOS 6.10 Docker image that rebuilds glibc without vsyscall is
necessary to reliably run manylinux2010 on 64-bit hosts. This requires
building the image on a system with vsyscall=emulate but allows the
resulting container to run on systems with vsyscall=none or
vsyscall=emulate.
>
> vsyscall is an antiquated optimization for a small number of
frequently-used system calls. A vsyscall-enabled Linux kernel maps a
read-only page of data and system calls into a process' memory at a fixed
address. These system calls can then be invoked by dereferencing a function
pointers to fixed offsets in that page, saving a relatively expensive
context switch. [1]
>
> Unfortunately, because the code and its location in memory are fixed and
well-known, the vsyscall mechanism has become a source of gadgets for ROP
attacks (specifically, Sigreturn-Oriented Programs). [2] Linux 3.1
introduced vsyscall emulation that prevents attackers from jumping into the
middle of the system calls' code at the expense of speed, as well as the
ability to disable it entirely. [3] [4] The vsyscall mechanism could not be
eliminated at the time because glibc versions earlier than 2.14 contained
hard-coded references to the fixed memory address, specifically in time(2).
[5] These segfault when attempting to issue a vsyscall-optimized system
call against a kernel that has disabled it.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/HDNTKOTF2QNNPVGFVHYWP6CXOWN4AXO3/

[Distutils] Re: The base docker images for manylinux appear to be private

[Wes Turner]

I agree. The manylinux docker images should be entirely reproducible from
source.

Is there a reason the new manylinux does not just extend centos:6?


https://github.com/pypa/manylinux/blame/master/docker/Dockerfile-x86_64

https://discuss.python.org/t/manylinux2010-docker-image-now-available/1471

(There are also centos:6 ARM and ARM64 builds:
https://hub.docker.com/_/centos )

FWIW, I recently learned about vmware/tern:
> Tern is an inspection tool to find the metadata of the packages installed
in a container image.
https://github.com/vmware/tern

On Tue, Feb 25, 2020, 6:09 PM Sylvain Corlay 
wrote:

> It seems that the base images used for manylinux wheels are not publicly
> available.
>
> For example, https://quay.io/pypa/manylinux2010_centos-6-no-vsyscall gives
> a 403 error ("you are not authorized to view this resource").
>
> I think that it would make sense for reproducibility purposes to make
> these images public. (I have opened an issue about this on the GitHub
> repository: https://github.com/pypa/manylinux/issues/481).
>
> Best,
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/EMU5KOCLFACJ4HIORKBYAEZBBNN5VEPH/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/PY7GCMCF2MGUFXJPLQITCWLJNJ26JG5G/

[Distutils] Re: Packaging Summit 2020: Save the Date!

[Wes Turner]

On Tue, Jan 14, 2020, 8:47 PM Dustin Ingram  wrote:

> Thank you Paul for organizing!
>

Thanks!

In the meantime,

Source: https://github.com/pypa
Homepage: https://www.pypa.io/en/latest/
- Goals
- Specifications
- Roadmap
- ** Presentations & Activities **
  https://www.pypa.io/en/latest/presentations/

Python Packaging User Guide
https://packaging.python.org/

- PyPA Specifications
  https://packaging.python.org/specifications/

- Project Summaries (for pypa/ and non-pypa tools and libraries)
  https://packaging.python.org/key_projects/

...

And - though it's not yet time for the call for topics - I'd like to draw
attention to and second a call for review of pipgrep (which depends upon
the PubGrub implementation in mixology, which is extracted from poetry) as
a candidate for the new dependency resolver

New Resolver: Technical Choices (2019-)
https://github.com/pypa/pip/issues/7406

Pip needs a dependency resolver (2013-)
https://github.com/pypa/pip/issues/988#issuecomment-572797302

> Please give it a go and report any issues you find :)
>
> https://github.com/ddelange/pipgrip



>
> On Tue, Jan 14, 2020 at 7:37 PM Paul Ganssle  wrote:
> >
> > Greetings one and all!
> >
> > Each year millions of eyes around the world eagerly wait at their
> computer with baited breath to learn the timing of the most fabulous,
> extravagant, decadent and hyperbolically oversold event of the year: The
> Python Packaging Summit. While most of the details aren't finalized (and in
> fact basically no details are finalized), in the interest of giving those
> who would like to attend a chance at some additional notice on their travel
> plans, we have established that the date of the summit: Thursday, 16 April
> 2020 (the day before the first PyCon talk day).
> >
> > For the past two years we've unofficially held our mini-summit on the
> first day of PyCon Sprints, but given that this summit is of particular
> interest to maintainers - many of whom would like to spend the first day of
> sprints actually sprinting - we have decided to run the summit on the
> tutorials day parallel to the Education Summit. We have applied for a spot
> in the Hatchery program, but even if we are not accepted as an official
> PyCon evenat, we will make arrangements to run the summit off-site on that
> day.
> >
> > For those unaware of the nature of the summit, you may want to pay
> attention to the "hyperbolically oversold" portion of my description less
> than the "decadent" and "extravagant" portions. The purpose of the summit
> is to get a bunch of people interested in Python Packaging together to
> discuss and coordinate on important topics in packaging and how we can move
> them forward. If you are on the fence about joining and want to know more,
> here are some detailed notes taken at the summit last year:
> https://docs.google.com/document/d/1Wz2-ECkicJgAmQDxMFivWmU2ZunKvPZ2UfQ59zDGj7g/edit#heading=h.7rw2gk16okic
> >
> > As I mentioned earlier, we have finalized essentially no details of this
> summit other than the date. I do not know how many people will be able to
> attend or by what mechanism we will prioritize attendance at the summit if
> there is limited space.
> >
> > In the coming months please keep an eye out for:
> >
> > Registration details (if such a thing is deemed necessary)
> > A call for topic suggestions
> > Voting on how to prioritize topic suggestions
> >
> > Looking forward to seeing you all there!
> > Paul
> >
> > --
> > Distutils-SIG mailing list -- distutils-sig@python.org
> > To unsubscribe send an email to distutils-sig-le...@python.org
> > https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> > Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/ZEZNY2MCUN3S3JGUFF6U6OWVEOKW2UAF/
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/C5Y7A4QLVBKNRHIKF2667UFMYG4LTNZD/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/PNVTOHWUCRPMDIJPSNBHQS6N22Z2KEBR/

[Distutils] Re: Linux binary wheels?

[Wes Turner]

FWIW, conda supports the e.g. armv7l aarch32 and armv8 aarch64 / "ARM64"
platforms. Third-party-built packages are the norm there; where there are
channels like conda-forge and rpi. What does it mean to sign a CI build
from a given unsigned git tag?

"Build conda packages for ARM"
https://github.com/conda-forge/conda-forge.github.io/issues/269

Raspbian is built for armv7.

Raspberry Pi 2, 3, and 4 are armv8 CPUs (with hw SIMD, AES, SHA-1,
SHA-256), so they support armv7l (aarch32) and armv8 (aarch64)

Raspberry Pi Zero W have armv6 CPUs.

Raspberry Pi 4 have max 4GB of RAM, so that may be a reason for the
community to support aarch64.

On Tuesday, August 20, 2019, Tzu-ping Chung  wrote:

>
> On 20 Aug 2019, at 23:47, Nick Timkovich  wrote:
>
> On Tue, Aug 20, 2019, at 5:05 AM Matthew Brett 
> wrote:
>
>> ...  Unless you meant wheels for non-Intel platforms, in which case,
>> please do say more about you need.
>
>
> Minor tangent: I've seen some people use https://www.piwheels.org/ for
> Raspberry Pi (ARM 6/7), but could the ARM binaries be uploaded to PyPI?
>
> I think I'm conflating the wheel building spec (is manylinux amd64
> specific, or as long as the libraries are on any architecture?),
> toolchains, environment (sounds like Piwheels provides a platform to build
> them on), and package hosting (can PyPI host arbitrary archs?) in that one
> sentence.
>
>
> This issue may be of relevant: https://github.com/
> pypa/warehouse/issues/3668
>
> And there are even more layers to this problem. Wheels on piwheels are
> currently maintained by RPi folks; if they are going into PyPI, either
> package maintainers need to take over uploading (and even building) them,
> or PyPI needs a way to allow (qualified) people to upload stuffs for
> packages they don’t own. And maintainers might decide that ARM is not their
> supported platform anyway, and get us back to where we started.
>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/archives/list/distutils-sig@
> python.org/message/OXSUW73EO5DTUO34EFURN3KHCDAKNS4Z/
>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/WLASKLKOIT2TEHXWEKFAQVFXSP74LSKB/

[Distutils] Re: Linux binary wheels?

[Wes Turner]

How does this proposal differ from manylinux2010?

https://github.com/pypa/manylinux/blob/master/README.rst#example

PEP 513: manylinux1
https://www.python.org/dev/peps/pep-0513/

PEP 571: The manylinux2010 Platform Tag (latest, as of 2019)
https://www.python.org/dev/peps/pep-0571/



On Monday, August 19, 2019, Dan Stromberg  wrote:

> Hi folks.
> I have a pair of ideas about Linux binary wheels, which are currently (I
> heard) unsupported.
> It seems like it should be possible to support Linux binary wheels using
> one or both of these technologies:
> * https://build.opensuse.org/ is a service that builds packages for a
> variety of Linuxes
> * Docker could be used to automate the building of wheels for a handful of
> Linuxes with minimal dependencies.  It seems like if you get
> Debian/Ubuntu/Mint, Fedora/CentOS, openSUSE and perhaps one or two others,
> that would cover almost all Linuxes and Linux users.
>
> I'm up to my hears in commitments already, but I sincerely someone will
> grab onto one or both of these possibilities and run with them.
> Thanks for reading.
>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/HN7KH55YBWV2PCGKZBI23NAYWLW5OQQI/

[Distutils] Re: Seeking a new maintainer for packaging.python.org and Twine.

[Wes Turner]

Thanks for Twine!

On Friday, July 19, 2019, Pradyun Gedam  wrote:

> Thanks Thea for all of the work you've done!
>
> Best,
> Pradyun
>
> On Sat, 20 Jul 2019 at 4:03 AM, Thea Flowers via Distutils-SIG <
> distutils-sig@python.org> wrote:
>
>> Hi friends!
>> I'm stepping away from several things in the Python community. I've
>> served as the maintainer of packaging.python.org and Twine for a while,
>> but it's time for me to move on.
>>
>> I'm looking for someone to take on primary responsibility for
>> packaging.python.org and another person to help share the load for Twine.
>>
>> If you or someone you know is interested, please reply to this thread and
>> let me know.
>>
>> -Thea Flowers
>> magicalg...@google.com
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>> Message archived at https://mail.python.org/archives/list/distutils-sig@
>> python.org/message/M7VRNT5KP4YQ6UPVI4MN4IIWM2Z3IXCH/
>>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/DU3EAXLHOS6G3TCO6BHHQCNI4S5CCTDR/

[Distutils] Re: Docker builds, ~/.pip/cache, and O(1) or O(n) bandwidth

[Wes Turner]

On Tuesday, May 7, 2019, Daniel Holth  wrote:

> Have you tried using buildkit and the RUN --mount option? I've done extra
> stuff here (downloading rpms in a second image first) but I think you could
> just use the cache option.
>
> It would also be easy to use a second image and COPY in old docker.
>
> https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86
>

Thanks. Is this a multi-stage Dockerfile (with multiple FROM instructions)?

Is there a guide for a PyPA recommended way to create network-efficient
reproducible (manylinux2010) wheels?


>
> On Tue, May 7, 2019, 11:59 Alex Becker  wrote:
>
>> You can use a local PyPI mirror, e.g. devpi, and point your docker builds
>> at that, basically tricking docker by going through the (local) network
>> stack instead of the filesystem.
>>
>> On Tue, May 7, 2019 at 8:12 AM Wes Turner  wrote:
>>
>>> What is the best way to build docker images without constantly
>>> re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n)
>>> for every build)
>>>
>>> (AFAIK, nobody has any issue with the amount of bandwidth PyPI uses)
>>>
>>> Thus far, Docker doesn't want to support a build-time -v option (that
>>> could be used to bind-mount .pip/cache in at build time):
>>>
>>> "build time only -v option"
>>> https://github.com/moby/moby/issues/14080
>>>
>>> Buildah *does* support a build-time -v option (and can also do rootless
>>> builds without a docker socket)
>>> --
>>> Distutils-SIG mailing list -- distutils-sig@python.org
>>> To unsubscribe send an email to distutils-sig-le...@python.org
>>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>>> Message archived at https://mail.python.org/archives/list/distutils-sig@
>>> python.org/message/IL6MGTSAA4X5UZDERCAAAFGGX5DGRPF3/
>>>
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>> Message archived at https://mail.python.org/archives/list/distutils-sig@
>> python.org/message/2MNMGYBNFOOVGXA2JQ2PLDTZITZXANJU/
>>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/WHEVRTBBCTRKJX6BKT43DXJMOLFUEUB7/

[Distutils] Docker builds, ~/.pip/cache, and O(1) or O(n) bandwidth

[Wes Turner]

What is the best way to build docker images without constantly
re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n)
for every build)

(AFAIK, nobody has any issue with the amount of bandwidth PyPI uses)

Thus far, Docker doesn't want to support a build-time -v option (that could
be used to bind-mount .pip/cache in at build time):

"build time only -v option"
https://github.com/moby/moby/issues/14080

Buildah *does* support a build-time -v option (and can also do rootless
builds without a docker socket)
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MGTSAA4X5UZDERCAAAFGGX5DGRPF3/

[Distutils] Re: Adding namespace support to PyPi (continuation from PyPA Summit/Sprint)

[Wes Turner]

On Tue, May 7, 2019 at 10:01 AM Wes Turner  wrote:

> What do namespaces offer over forking, diffing, reviewing the latest
> commits, and installing from your GH fork URL commit hash?


IIUC, one of the primary objectives for namespaces is to enable a user to
store state like ("We've reviewed this and consider this version acceptable
for our purposes")

PyPI already hosts 1 copy of each build of each version of each package for
everyon.


> When I try to install 'westurner/pip' and 'pip' is already installed, what
> should it do?
>
> Should pypa/setuptools_scm include the namespace in the version tag?
>
> If my concerns include integrity and availability in a risky environment,
> why rely upon PyPI for hosting (manylinux2010 binary) wheels at all? (You
> can run warehouse and/or devpi on-prem and keep that upgraded and baselined)
>

Or by privately hosting just a directory of packages and setting
PIP_INDEX_URL.


> Will pip throw an exception if my local warehouse or devpi package isn't
> signed with 'a key recognized by upstream'?
>

Will the TUF implementation need any changes to support namespaces?

"Roadmap update for TUF support"
https://github.com/pypa/warehouse/issues/5247

> "PyPI security work: multifactor auth progress & help needed"
>
https://discuss.python.org/t/pypi-security-work-multifactor-auth-progress-help-needed/1042/10

(Docker Notary does TUF today)


>
> On Monday, May 6, 2019, Dave Ashby via Distutils-SIG <
> distutils-sig@python.org> wrote:
>
>> Good evening distutils-sig-
>>
>> TL;DR: today during the Python Packaging Summit/Sprint I broached the
>> topic of introducing namespaces on PyPi with several of the luminaries in
>> the PyPA ecosystem. After much discussion I think we concluded that there’s
>> general support for the idea, but quite a number of devils in the details
>> that would need worked out. I’ve tried to capture much of the relevant
>> discussion on in the associated thread on discuss.python.org
>> <https://discuss.python.org/t/namespace-support-in-pypi/1609> (including
>> a first-pass at a draft PEP
>> <https://gist.github.com/vadave/f1f2d07f5e355c6263fc111aae634ea5>)
>> Please join in the conversation about whether/how namespaces should be
>> implemented in PyPi.
>>
>>
>>
>> Detail:
>>
>> The namespaces discussion had actually come up before the transition to
>> warehouse[1], but at the time it was deferred due to the effort associated
>> with that migration. Now that Warehouse is in a better place, it seems like
>> it’s time to reinvigorate this discussion. Most other programming languages
>> have some type of namespace construct that can be used to help better
>> manage the supply chain integrity. As I see it namespaces have the
>> potential to provide significant value to **both** package maintainers
>> and package consumers. For package maintainers it potentially offers a
>> higher-level of control over packages they are responsible for maintaining.
>> For package consumers it potentially offers a cleaner mechanism for “at a
>> glance” recognition of the source of a particular package (e.g. whether the
>> package is vendor-developed vs. community-developed or has a formal
>> relationship with some larger group like PyData).
>>
>>
>>
>> With all that said, my personal goal is for this PEP to serve kind of a
>> foundational role. I **don’t want** to try to exhaustively define all
>> the different scenarios that namespaces can/should be used to enable. I **do
>> want** namespaces to become a clear security-boundary that can be used
>> in “high compliance” environments to make security-related risk decisions.
>> And lastly, I **definitely don’t want** to replace the current global
>> namespace…..I view this as a complement, not a replacement to the current
>> situation.
>>
>>
>>
>> Finally, as this is my first post to this list I’ll offer the obligatory
>> “new guy” introduction. I started using Python about 4 years ago, so I
>> still consider myself a relative newbie. For my day job I’m an independent
>> consultant working with what I call “high compliance” customers…..folks
>> that really, really care about security and have lots of process baked
>> around how they manage security risk within their organization. I’m based
>> in Fairfax, VA. Most of what I do from day-to-day **is not** writing
>> python code, but rather working issues around IT architecture and
>> governance (with a general focus around enabling infrastructure such as
>> cloud services and devops tooling).
>>
>>
>>
>> Thanks to all th

[Distutils] Re: Adding namespace support to PyPi (continuation from PyPA Summit/Sprint)

[Wes Turner]

What do namespaces offer over forking, diffing, reviewing the latest
commits, and installing from your GH fork URL commit hash?

When I try to install 'westurner/pip' and 'pip' is already installed, what
should it do?

Should pypa/setuptools_scm include the namespace in the version tag?

If my concerns include integrity and availability in a risky environment,
why rely upon PyPI for hosting (manylinux2010 binary) wheels at all? (You
can run warehouse and/or devpi on-prem and keep that upgraded and baselined)

Will pip throw an exception if my local warehouse or devpi package isn't
signed with 'a key recognized by upstream'?

On Monday, May 6, 2019, Dave Ashby via Distutils-SIG <
distutils-sig@python.org> wrote:

> Good evening distutils-sig-
>
> TL;DR: today during the Python Packaging Summit/Sprint I broached the
> topic of introducing namespaces on PyPi with several of the luminaries in
> the PyPA ecosystem. After much discussion I think we concluded that there’s
> general support for the idea, but quite a number of devils in the details
> that would need worked out. I’ve tried to capture much of the relevant
> discussion on in the associated thread on discuss.python.org
>  (including
> a first-pass at a draft PEP
> ) Please
> join in the conversation about whether/how namespaces should be implemented
> in PyPi.
>
>
>
> Detail:
>
> The namespaces discussion had actually come up before the transition to
> warehouse[1], but at the time it was deferred due to the effort associated
> with that migration. Now that Warehouse is in a better place, it seems like
> it’s time to reinvigorate this discussion. Most other programming languages
> have some type of namespace construct that can be used to help better
> manage the supply chain integrity. As I see it namespaces have the
> potential to provide significant value to **both** package maintainers
> and package consumers. For package maintainers it potentially offers a
> higher-level of control over packages they are responsible for maintaining.
> For package consumers it potentially offers a cleaner mechanism for “at a
> glance” recognition of the source of a particular package (e.g. whether the
> package is vendor-developed vs. community-developed or has a formal
> relationship with some larger group like PyData).
>
>
>
> With all that said, my personal goal is for this PEP to serve kind of a
> foundational role. I **don’t want** to try to exhaustively define all the
> different scenarios that namespaces can/should be used to enable. I **do
> want** namespaces to become a clear security-boundary that can be used in
> “high compliance” environments to make security-related risk decisions. And
> lastly, I **definitely don’t want** to replace the current global
> namespace…..I view this as a complement, not a replacement to the current
> situation.
>
>
>
> Finally, as this is my first post to this list I’ll offer the obligatory
> “new guy” introduction. I started using Python about 4 years ago, so I
> still consider myself a relative newbie. For my day job I’m an independent
> consultant working with what I call “high compliance” customers…..folks
> that really, really care about security and have lots of process baked
> around how they manage security risk within their organization. I’m based
> in Fairfax, VA. Most of what I do from day-to-day **is not** writing
> python code, but rather working issues around IT architecture and
> governance (with a general focus around enabling infrastructure such as
> cloud services and devops tooling).
>
>
>
> Thanks to all the folks that offered their time and opinions on this topic
> today. I look forward to continued dialogue on this, and welcome any ideas
> or cautions folks may have about making this a reality. And remember:
> “Namespaces are one honking great idea – let’s do more of those!”
>
> -d
>
>
>
>
>
> [1] Original feature request for adding namepsaces support:
> https://github.com/pypa/warehouse/issues/2589
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/KWWOYSH7NGPTDXACWGJ3LBJ2SKCLMPCV/

[Distutils] Re: PyPI security work: multifactor auth progress & help needed

[Wes Turner]

Is webauthn the multi-factor / 2FA spec to implement now? It's now
approved; so while you experts are working on it it may be worth a look to
just implement webauthn while we have funding for experts

https://www.w3.org/TR/webauthn/

Discourse mentions FIDO. FIDO2 is webauthn, AFAIU.

There are a number of implementations:

https://pypi.org/search/?q=webauthn

https://github.com/topics/webauthn

On Friday, March 22, 2019, Sumana Harihareswara  wrote:

> Work has started on the Open Technology Fund-supported project to improve
> Warehouse security, accessibility, and internationalization. More details
> in today's progress report:
>
> https://discuss.python.org/t/pypi-security-work-multifactor-auth-progress-
> help-needed/1042/2
>
>
> best,
> Sumana Harihareswara
> Warehouse project manager
> Changeset Consulting
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/archives/list/distutils-sig@
> python.org/message/3E64P4GNVFSG4JA42OITJUCYU5H3QLAZ/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/EPQTPF6HKANBRKCE33D7BLNL4VBY2MWC/

[Distutils] Re: API for SHA-256 fingerprints

[Wes Turner]

On Tuesday, February 12, 2019, Jeremy Stanley  wrote:

> On 2019-02-12 18:42:29 -0500 (-0500), Wes Turner wrote:
> [...]
> > All it has to be is an archive containing a setup.py.
> >
> > "MD5 considered harmful today:
> > Creating a rogue CA certificate" (2008)
> > https://www.win.tue.nl/hashclash/rogue-ca/
>
> You keep trotting out PKI examples as if they have anything
> whatsoever to do with checksumming, but I'm quickly getting the
> distinct impression you don't actually know the difference so I'll
> stop now as we've gone well off-topic for this list.
> --
> Jeremy Stanley
>

 you hash the file.
the hash is compared against a list.
if the hash matches, it's considered valid.

In 2008,
they were able to generate a file that has the same MD5 hash as one in a
list of considered-good hashes,
which is also a valid x.509 cert.

How is that at all different from generating an archive with a setup.py
that has the same hash as something listed on PyPI?

Trotting ... "Westminster Dad Show" https://youtu.be/2S2gQjTURvU ... Now
you've suggested that I'm FUD'ing: is there a difference between finding an
x.509 cert hash and a .tgz/.zip with a setup.py or setup.pyc hash? Maybe
there's something fundamental that I've misunderstood?

(So sorry to interrupt)
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/XDKGVO4PIDC6LCKJY3TFDNG275W7PBLD/

[Distutils] Re: API for SHA-256 fingerprints

[Wes Turner]

On Tuesday, February 12, 2019, Jeremy Stanley  wrote:

> On 2019-02-12 17:02:25 -0500 (-0500), Wes Turner wrote:
> > On Tuesday, February 12, 2019, Wes Turner  wrote:
> [...]
> > > It is possible to find a nonce value that causes an arbitrary package
> to
> > > have the same MD5 hash as the actual package.
> >
> > e.g. browsers MUST NOT rely upon MD5 for x.509 certificate SSL/TLS/HTTPS
> > fingerprints for exactly this reason.
> [...]
>
> I fear we're verging far into armchair crypto here, but you're
> either making buzzword soup or have a severely flawed understanding
> of the algorithms involved. There is no nonce in an IETF RFC 1321
> (colloquially "MD5 checksum") implementation, so please at least
> attempt to frame your assertions using terms found in the canonical
> literature.
>
> Creating a malicious package which computes to the same MD5 checksum
> as an existing package of your choice would require that the second
> preimage resistance of the MD5 algorithm is broken, or that you got
> (time complexity 2^128) "lucky." Uses of MD5 elsewhere which mix in
> attacker-controlled inputs to generate the reference output are
> another story entirely, but as with the any of the information
> security field the actual risk depends on your threat model.
>
> I'm not about to recommend MD5 to anyone these days, don't get me
> wrong. There are (at least marginally, again depending on your
> threat model) better alternatives which require no additional effort
> if you're designing a system from scratch. But let's not
> mischaracterize the qualities of any algorithm, as it makes it
> difficult for someone who does understand the differences to take us
> seriously.


All it has to be is an archive containing a setup.py.

"MD5 considered harmful today:
Creating a rogue CA certificate" (2008)
https://www.win.tue.nl/hashclash/rogue-ca/


> --
> Jeremy Stanley
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/WQPMWBYVTJ3UOHZXDI7P3ULISVINV42P/

[Distutils] Re: API for SHA-256 fingerprints

[Wes Turner]

On Tuesday, February 12, 2019, Trishank Kuppusamy <
trishank.kuppus...@datadoghq.com> wrote:

> On Tue, Feb 12, 2019 at 5:32 PM Cooper Ry Lees  wrote:
>
>> TUF should be handled via a grant from Facebook this year once Ernest and
>> I get this underway:
>> https://pyfound.blogspot.com/2018/12/upcoming-pypi-
>> improvements-for-2019.html
>>
>> We will take all the help we can get, but we'll have Project management
>> and some funds!
>>
>
> Happy to hear, and happy to pitch in when the time comes.
>

Same. Good to hear
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/WG7LI2GDDX27S6UDUMLWHLSGVLXNN3QZ/

[Distutils] Re: API for SHA-256 fingerprints

[Wes Turner]

On Tuesday, February 12, 2019, Wes Turner  wrote:

>
>
> On Tuesday, February 12, 2019, Jeremy Stanley  wrote:
>
>> On 2019-02-12 13:37:20 -0500 (-0500), Wes Turner wrote:
>> > MD5 is no longer suitable for verifying package integrity.
>> >
>> > https://en.wikipedia.org/wiki/MD5#Security
>> >
>> > > The security of the MD5 hash function is severely compromised. A
>> > > collision attack exists [...] there is also a chosen-prefix
>> > > collision attack
>> [...]
>>
>> The difference between collision (or chosen-prefix collision) and
>> preimage (or second preimage) attacks is still very relevant. With
>> MD5 you can't trust that someone who provided you with an input and
>> a hash of that input hasn't carefully crafted that input so that
>> there is also a second input which results in the same hash. Or in
>> package terms, you can't trust that the package you've received
>> wasn't part of a contrived scheme on the part of someone you've
>> already decided to trust. You can still rest assured (for now
>> anyway) that the package you receive is the same one the person or
>> system providing the MD5 checksum intended for you to receive.
>
>
> It is possible to find a nonce value that causes an arbitrary package to
> have the same MD5 hash as the actual package.
>

e.g. browsers MUST NOT rely upon MD5 for x.509 certificate SSL/TLS/HTTPS
fingerprints for exactly this reason.



>
>
>>
>> But because trying to explain this nuance to people is considerably
>> harder than just saying "MD5 bad" it's simply not worth trying to
>> have the discussion most of the time, and so easier instead to
>> replace it with a more modern alternative and move on with your
>> life.
>> --
>> Jeremy Stanley
>>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/HOJIB7HRCYAP225AN5RHTBNVDTYDH6TQ/

[Distutils] Re: API for SHA-256 fingerprints

[Wes Turner]

On Tuesday, February 12, 2019, Jeremy Stanley  wrote:

> On 2019-02-12 13:37:20 -0500 (-0500), Wes Turner wrote:
> > MD5 is no longer suitable for verifying package integrity.
> >
> > https://en.wikipedia.org/wiki/MD5#Security
> >
> > > The security of the MD5 hash function is severely compromised. A
> > > collision attack exists [...] there is also a chosen-prefix
> > > collision attack
> [...]
>
> The difference between collision (or chosen-prefix collision) and
> preimage (or second preimage) attacks is still very relevant. With
> MD5 you can't trust that someone who provided you with an input and
> a hash of that input hasn't carefully crafted that input so that
> there is also a second input which results in the same hash. Or in
> package terms, you can't trust that the package you've received
> wasn't part of a contrived scheme on the part of someone you've
> already decided to trust. You can still rest assured (for now
> anyway) that the package you receive is the same one the person or
> system providing the MD5 checksum intended for you to receive.


It is possible to find a nonce value that causes an arbitrary package to
have the same MD5 hash as the actual package.


>
> But because trying to explain this nuance to people is considerably
> harder than just saying "MD5 bad" it's simply not worth trying to
> have the discussion most of the time, and so easier instead to
> replace it with a more modern alternative and move on with your
> life.
> --
> Jeremy Stanley
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/Q6TVV5BTGXGGQBHTR33ICTA2GEKZSBPF/

[Distutils] Re: API for SHA-256 fingerprints

[Wes Turner]

On Tuesday, February 12, 2019, Wes Turner  wrote:

>
>
> On Tuesday, February 12, 2019, Alex Becker  wrote:
>
>> Also note that the simple API only includes a single hash for each file,
>> and may use md5 hashes instead of sha256 (technically it may use any of the
>> hash algorithms guaranteed by hashlib, but I've only seen those two). The
>> JSON API will give you *all* the hashes warehouse has for the file, which
>> may be more useful.
>>
>
> MD5 is no longer suitable for verifying package integrity.
>
> https://en.wikipedia.org/wiki/MD5#Security
>
> > The security of the MD5 hash function is severely compromised. A
> collision attack exists that can find collisions within seconds on a
> computer with a 2.6 GHz Pentium 4 processor (complexity of 224.1).[18]
> Further, there is also a chosen-prefix collision attack that can produce a
> collision for two inputs with specified prefixes within hours, using
> off-the-shelf computing hardware (complexity 239).[19]
>
>

[...]


>
> File has a .md5_digest, .sha256_digest, and .blake2_256_digest
>
> https://github.com/pypa/warehouse/search?q=md5_digest doesn't show
> selection of a hash with precedence; so IDK where that functionality is?
>

Oh, there it is in
https://github.com/pypa/warehouse/blob/master/warehouse/templates/legacy/api/simple/detail.html#L22
: the simple index *only* includes the sha256 hash.


>
>
>> Best,
>>
>> Alex Becker
>>
>
>> On Tue, Feb 12, 2019 at 9:58 AM Paul Moore  wrote:
>>
>>> On Tue, 12 Feb 2019 at 16:28, Eric Peterson
>>>  wrote:
>>> >
>>> > Brilliant, that's exactly what I was looking for—both the simple API
>>> and json API look very useful. Thanks!
>>>
>>> Just a quick note, the simple API is required for every index server
>>> to support, whereas the JSON API is not (yet?) standardised and may
>>> not be supported anywhere other than PyPI (I don't know about devpi,
>>> for example). This may not matter for your use case, but is useful to
>>> know more generally.
>>>
>>> Paul
>>> --
>>> Distutils-SIG mailing list -- distutils-sig@python.org
>>> To unsubscribe send an email to distutils-sig-le...@python.org
>>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>>> Message archived at https://mail.python.org/archiv
>>> es/list/distutils-sig@python.org/message/ZOU33JCVN32DWHRU5M
>>> JYGOV52BE5JIR3/
>>>
>>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/DIRIF65RN4DQX5QAVTQTZPQACI2F7U6A/

[Distutils] Re: API for SHA-256 fingerprints

[Wes Turner]

On Tuesday, February 12, 2019, Alex Becker  wrote:

> Also note that the simple API only includes a single hash for each file,
> and may use md5 hashes instead of sha256 (technically it may use any of the
> hash algorithms guaranteed by hashlib, but I've only seen those two). The
> JSON API will give you *all* the hashes warehouse has for the file, which
> may be more useful.
>

MD5 is no longer suitable for verifying package integrity.

https://en.wikipedia.org/wiki/MD5#Security

> The security of the MD5 hash function is severely compromised. A
collision attack exists that can find collisions within seconds on a
computer with a 2.6 GHz Pentium 4 processor (complexity of 224.1).[18]
Further, there is also a chosen-prefix collision attack that can produce a
collision for two inputs with specified prefixes within hours, using
off-the-shelf computing hardware (complexity 239).[19]


>
>
> Most likely (someone more familiar with Warehouse could answer this)
> Warehouse will select sha256 whenever it is available, so the simple API
> may be just as good for you. But it's something to consider.
>

https://github.com/pypa/warehouse/blob/master/warehouse/legacy/api/simple.py

https://github.com/pypa/warehouse/blob/master/tests/unit/legacy/api/test_simple.py

https://github.com/pypa/warehouse/blob/master/warehouse/packaging/models.py

File has a .md5_digest, .sha256_digest, and .blake2_256_digest

https://github.com/pypa/warehouse/search?q=md5_digest doesn't show
selection of a hash with precedence; so IDK where that functionality is?


> Best,
>
> Alex Becker
>

> On Tue, Feb 12, 2019 at 9:58 AM Paul Moore  wrote:
>
>> On Tue, 12 Feb 2019 at 16:28, Eric Peterson
>>  wrote:
>> >
>> > Brilliant, that's exactly what I was looking for—both the simple API
>> and json API look very useful. Thanks!
>>
>> Just a quick note, the simple API is required for every index server
>> to support, whereas the JSON API is not (yet?) standardised and may
>> not be supported anywhere other than PyPI (I don't know about devpi,
>> for example). This may not matter for your use case, but is useful to
>> know more generally.
>>
>> Paul
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>> Message archived at https://mail.python.org/archives/list/distutils-sig@
>> python.org/message/ZOU33JCVN32DWHRU5MJYGOV52BE5JIR3/
>>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/XLH7JLJHQUOSGBJVU4AVBMAFFQBRABGT/

[Distutils] Re: API for SHA-256 fingerprints

[Wes Turner]

On Tuesday, February 12, 2019, Eric Peterson <
epeter...@interactivebrokers.com> wrote:

>
> [...]. I am wondering if there is a programmatic way to access the SHA-256
> for a file (besides just scraping the web page)? Ideally there would be
> some way to construct a URL based on the name of the file that, when
> called, would return the fingerprint.


Because you'd be retrieving the SHA-256 over the same channel as the
release archive and said checksum is not signed, the SHA-256 should not be
considered sufficient for ensuring release integrity.

(Because if the bad guy is MITM'ing the release archive retrieval, they
could also be MITM'ing the SHA-256 retrieval)

Ways to mitigate such risk:

- retrieve SHA-256 cryptographic hash checksums over a different channel
- cryptographically sign the SHA-256 checksums with a key and retrieve the
corresponding key over a different channel

Re: GPG and PyPI:
https://github.com/pypa/warehouse/issues/3810#issuecomment-405975460

>From https://python-security.readthedocs.io/packages.html#pypi :

> - PEP 458 – Surviving a Compromise of PyPI (27-Sep-2013)
> - PEP 480 – Surviving a Compromise of PyPI: The Maximum Security Model
(8-Oct-2014)
> - Making PyPI security independent of SSL/TLS by Nick Coghlan

... The Update Framework (TUF) is in part derived from Thandy (the tor
updater). There's an automotive derivative of TUF called Uptane.
https://theupdateframework.github.io/

"Roadmap update for TUF support"
https://github.com/pypa/warehouse/issues/5247

"TUF deployment roadmap for PyPI"
https://github.com/theupdateframework/tuf/issues/816#

SHA-256 is not sufficient. GPG was removed because insufficient.
Does TUF need funding, person-hours, new code, or code-review?




> Thanks,
> Eric
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/archives/list/distutils-sig@
> python.org/message/FLNOENK2525RMHGL7SV2SBUXKSOJHSEZ/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/YT4CDSHIOWUN6W5SF7NY5TCLJ5TXFWRF/

[Distutils] Re: pip + safety

[Wes Turner]

> One way to avoid disclosing user environments to a third party is to
build this into PyPI instead

Warehouse:

- [ ] clarify pyup/safety-db license terms re: PyPI and attribution and
noncommercial
- [ ] packaging.tasks.?: retrieve pyup JSON
- [ ] packaging.views.release_detail: add the data to the package release
view json
- [ ] packaging.tasks.?: invalidate cached release JSON when pyup json
changes

https://github.com/pypa/warehouse/blob/master/warehouse/packaging/views.py

https://github.com/pypa/warehouse/blob/master/warehouse/packaging/tasks.py

Pip

- [ ] display release vuln info as retrieved from release_detail JSON
 - should this prompt or fail?

PyPA

- is there a PyPI release JSON spec that should or must be updated first?
(So that other package indexes and tools can also implement this JSON key
and insecure_full.json-style vuln info)

On Tuesday, February 12, 2019, Wes Turner  wrote:

> https://github.com/pypa/pipenv/blob/master/pipenv/patched/safety/safety.py
> :
>
> def write_to_cache(db_name, data):
> # cache is in: ~/safety/cache.json
> # ...
> def fetch_database(...)
>
> On Tuesday, February 12, 2019, Wes Turner  wrote:
>
>> Good call. I didn't realize that that's how safety works.
>>
>> Is this the same data that pipenv/safety retrieves from pyup?
>> https://github.com/pyupio/safety-db/blob/master/data/insecure_full.json
>>
>> https://pipenv.readthedocs.io/en/latest/advanced/#detection-
>> of-security-vulnerabilities :
>>
>> > Note
>> > In order to enable this functionality while maintaining its permissive
>> copyright license, pipenv embeds an API client key for the backend Safety
>> API operated by pyup.io rather than including a full copy of the
>> CC-BY-NC-SA licensed Safety-DB database. This embedded client key is shared
>> across all pipenv check users, and hence will be subject to API access
>> throttling based on overall usage rather than individual client usage.
>> >
>> > You can also use your own safety API key by setting the environment
>> variable PIPENV_PYUP_API_KEY.
>>
>> https://github.com/pypa/pipenv/blob/master/pipenv/patched/safety/cli.py
>> vulns = safety.check(packages=packages, key=key, db_mirror=db,
>> cached=cache, ignore_ids=ignore)
>>
>> https://github.com/pypa/pipenv/blob/master/tasks/vendoring/__init__.py
>> def update_safety
>>
>> On Monday, February 11, 2019, Tzu-ping Chung  wrote:
>>
>>> One way to avoid disclosing user environments to a third party is to
>>> build this into PyPI instead. The API could generate the warning for pip to
>>> display.
>>>
>>> This only covers packages on PyPI, of course, but trying to audit local
>>> and self-hosted packages is is a fools errand anyway IMO since there is no
>>> practical way for any tool to reliably know what *actually* is installed.
>>>
>>> --
>>> Tzu-ping Chung (@uranusjr)
>>> uranu...@gmail.com
>>> Sent from my iPhone
>>>
>>> On 12 Feb 2019, at 11:34, Wes Turner  wrote:
>>>
>>> Would something like this require:
>>>
>>> - a pip extension/plugin/post-install hook API
>>> - a post-install hook that discloses all installed packages and versions
>>> (from pypi.org, mirrors, local directory) in exchange for checking and
>>> online security DB
>>> - a way to specify a key to e.g. pyup
>>>
>>> GItHub and GitLab offer similar functionality:
>>>
>>> https://github.blog/2018-07-12-security-vulnerability-alerts-for-python/
>>>   https://help.github.com/articles/about-security-alerts-for
>>> -vulnerable-dependencies/
>>>
>>> https://docs.gitlab.com/ee/user/project/merge_requests/depen
>>> dency_scanning.html
>>>   https://gitlab.com/gitlab-org/security-products/dependency-s
>>> canning#supported-languages-and-package-managers
>>>
>>> https://pyup.io
>>>
>>> https://github.com/pyupio/safety-db
>>>
>>> > pipenv check relies on safety and Safety-DB to check for known
>>> vulnerabilities in locked components
>>>
>>>
>>> On Monday, February 11, 2019, Julian Berman 
>>> wrote:
>>>
>>>> Hi.
>>>>
>>>> I recently found myself installing a node.js package, and in the
>>>> process noticed that (sometime recently?) it started automatically warning
>>>> about known vulnerabilities during installation of package.jsons (see
>>>> https://docs.npmjs.com/cli/audit).
>>>>
>>>> At work, we run safety (https://pypi.org/project/safe

[Distutils] Re: pip + safety

[Wes Turner]

https://github.com/pypa/pipenv/blob/master/pipenv/patched/safety/safety.py :

def write_to_cache(db_name, data):
# cache is in: ~/safety/cache.json
# ...
def fetch_database(...)

On Tuesday, February 12, 2019, Wes Turner  wrote:

> Good call. I didn't realize that that's how safety works.
>
> Is this the same data that pipenv/safety retrieves from pyup?
> https://github.com/pyupio/safety-db/blob/master/data/insecure_full.json
>
> https://pipenv.readthedocs.io/en/latest/advanced/#detection-
> of-security-vulnerabilities :
>
> > Note
> > In order to enable this functionality while maintaining its permissive
> copyright license, pipenv embeds an API client key for the backend Safety
> API operated by pyup.io rather than including a full copy of the
> CC-BY-NC-SA licensed Safety-DB database. This embedded client key is shared
> across all pipenv check users, and hence will be subject to API access
> throttling based on overall usage rather than individual client usage.
> >
> > You can also use your own safety API key by setting the environment
> variable PIPENV_PYUP_API_KEY.
>
> https://github.com/pypa/pipenv/blob/master/pipenv/patched/safety/cli.py
> vulns = safety.check(packages=packages, key=key, db_mirror=db,
> cached=cache, ignore_ids=ignore)
>
> https://github.com/pypa/pipenv/blob/master/tasks/vendoring/__init__.py
> def update_safety
>
> On Monday, February 11, 2019, Tzu-ping Chung  wrote:
>
>> One way to avoid disclosing user environments to a third party is to
>> build this into PyPI instead. The API could generate the warning for pip to
>> display.
>>
>> This only covers packages on PyPI, of course, but trying to audit local
>> and self-hosted packages is is a fools errand anyway IMO since there is no
>> practical way for any tool to reliably know what *actually* is installed.
>>
>> --
>> Tzu-ping Chung (@uranusjr)
>> uranu...@gmail.com
>> Sent from my iPhone
>>
>> On 12 Feb 2019, at 11:34, Wes Turner  wrote:
>>
>> Would something like this require:
>>
>> - a pip extension/plugin/post-install hook API
>> - a post-install hook that discloses all installed packages and versions
>> (from pypi.org, mirrors, local directory) in exchange for checking and
>> online security DB
>> - a way to specify a key to e.g. pyup
>>
>> GItHub and GitLab offer similar functionality:
>>
>> https://github.blog/2018-07-12-security-vulnerability-alerts-for-python/
>>   https://help.github.com/articles/about-security-alerts-
>> for-vulnerable-dependencies/
>>
>> https://docs.gitlab.com/ee/user/project/merge_requests/depen
>> dency_scanning.html
>>   https://gitlab.com/gitlab-org/security-products/dependency-s
>> canning#supported-languages-and-package-managers
>>
>> https://pyup.io
>>
>> https://github.com/pyupio/safety-db
>>
>> > pipenv check relies on safety and Safety-DB to check for known
>> vulnerabilities in locked components
>>
>>
>> On Monday, February 11, 2019, Julian Berman  wrote:
>>
>>> Hi.
>>>
>>> I recently found myself installing a node.js package, and in the process
>>> noticed that (sometime recently?) it started automatically warning about
>>> known vulnerabilities during installation of package.jsons (see
>>> https://docs.npmjs.com/cli/audit).
>>>
>>> At work, we run safety (https://pypi.org/project/safety/) on all our
>>> projects (which has both free and paid versions). It's great.
>>>
>>> I know there's a ton of wonderful work happening at the minute to
>>> improve underlying scaffolding + specification to enable tools other than
>>> setuptools + pip to thrive, so maybe this is the wrong moment, but I
>>> figured I'd ask anyways :) -- what are opinions on running a similar thing
>>> during pip install?
>>>
>>> -J
>>>
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>> Message archived at https://mail.python.org/archiv
>> es/list/distutils-sig@python.org/message/GSTL47B4CREYHKOS5I
>> 47WOPQURBKTOAY/
>>
>>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/ZOB7CT4HI6VTG7KWP2TFH63Z6VWZBCZX/

[Distutils] Re: pip + safety

[Wes Turner]

Good call. I didn't realize that that's how safety works.

Is this the same data that pipenv/safety retrieves from pyup?
https://github.com/pyupio/safety-db/blob/master/data/insecure_full.json

https://pipenv.readthedocs.io/en/latest/advanced/#detection-of-security-vulnerabilities
:

> Note
> In order to enable this functionality while maintaining its permissive
copyright license, pipenv embeds an API client key for the backend Safety
API operated by pyup.io rather than including a full copy of the
CC-BY-NC-SA licensed Safety-DB database. This embedded client key is shared
across all pipenv check users, and hence will be subject to API access
throttling based on overall usage rather than individual client usage.
>
> You can also use your own safety API key by setting the environment
variable PIPENV_PYUP_API_KEY.

https://github.com/pypa/pipenv/blob/master/pipenv/patched/safety/cli.py
vulns = safety.check(packages=packages, key=key, db_mirror=db,
cached=cache, ignore_ids=ignore)

https://github.com/pypa/pipenv/blob/master/tasks/vendoring/__init__.py
def update_safety

On Monday, February 11, 2019, Tzu-ping Chung  wrote:

> One way to avoid disclosing user environments to a third party is to build
> this into PyPI instead. The API could generate the warning for pip to
> display.
>
> This only covers packages on PyPI, of course, but trying to audit local
> and self-hosted packages is is a fools errand anyway IMO since there is no
> practical way for any tool to reliably know what *actually* is installed.
>
> --
> Tzu-ping Chung (@uranusjr)
> uranu...@gmail.com
> Sent from my iPhone
>
> On 12 Feb 2019, at 11:34, Wes Turner  wrote:
>
> Would something like this require:
>
> - a pip extension/plugin/post-install hook API
> - a post-install hook that discloses all installed packages and versions
> (from pypi.org, mirrors, local directory) in exchange for checking and
> online security DB
> - a way to specify a key to e.g. pyup
>
> GItHub and GitLab offer similar functionality:
>
> https://github.blog/2018-07-12-security-vulnerability-alerts-for-python/
>   https://help.github.com/articles/about-security-alerts-for-vulnerable-
> dependencies/
>
> https://docs.gitlab.com/ee/user/project/merge_requests/
> dependency_scanning.html
>   https://gitlab.com/gitlab-org/security-products/dependency-
> scanning#supported-languages-and-package-managers
>
> https://pyup.io
>
> https://github.com/pyupio/safety-db
>
> > pipenv check relies on safety and Safety-DB to check for known
> vulnerabilities in locked components
>
>
> On Monday, February 11, 2019, Julian Berman  wrote:
>
>> Hi.
>>
>> I recently found myself installing a node.js package, and in the process
>> noticed that (sometime recently?) it started automatically warning about
>> known vulnerabilities during installation of package.jsons (see
>> https://docs.npmjs.com/cli/audit).
>>
>> At work, we run safety (https://pypi.org/project/safety/) on all our
>> projects (which has both free and paid versions). It's great.
>>
>> I know there's a ton of wonderful work happening at the minute to improve
>> underlying scaffolding + specification to enable tools other than
>> setuptools + pip to thrive, so maybe this is the wrong moment, but I
>> figured I'd ask anyways :) -- what are opinions on running a similar thing
>> during pip install?
>>
>> -J
>>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/archives/list/distutils-sig@
> python.org/message/GSTL47B4CREYHKOS5I47WOPQURBKTOAY/
>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/TSHUCVYXEYM5YMLQTNMRP76JK4GD3T2A/

[Distutils] Re: pip + safety

[Wes Turner]

Would something like this require:

- a pip extension/plugin/post-install hook API
- a post-install hook that discloses all installed packages and versions
(from pypi.org, mirrors, local directory) in exchange for checking and
online security DB
- a way to specify a key to e.g. pyup

GItHub and GitLab offer similar functionality:

https://github.blog/2018-07-12-security-vulnerability-alerts-for-python/

https://help.github.com/articles/about-security-alerts-for-vulnerable-dependencies/

https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html

https://gitlab.com/gitlab-org/security-products/dependency-scanning#supported-languages-and-package-managers

https://pyup.io

https://github.com/pyupio/safety-db

> pipenv check relies on safety and Safety-DB to check for known
vulnerabilities in locked components


On Monday, February 11, 2019, Julian Berman  wrote:

> Hi.
>
> I recently found myself installing a node.js package, and in the process
> noticed that (sometime recently?) it started automatically warning about
> known vulnerabilities during installation of package.jsons (see
> https://docs.npmjs.com/cli/audit).
>
> At work, we run safety (https://pypi.org/project/safety/) on all our
> projects (which has both free and paid versions). It's great.
>
> I know there's a ton of wonderful work happening at the minute to improve
> underlying scaffolding + specification to enable tools other than
> setuptools + pip to thrive, so maybe this is the wrong moment, but I
> figured I'd ask anyways :) -- what are opinions on running a similar thing
> during pip install?
>
> -J
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/GSTL47B4CREYHKOS5I47WOPQURBKTOAY/

[Distutils] Re: History of python packaging

[Wes Turner]

https://packaging.python.org
https://github.com/pypa/python-packaging-user-guide
- A history.rst document would be a useful addition

On Sunday, October 14, 2018, Dustin Ingram  wrote:

> Hi Konstantin,
>
> You might find my PyCon talk "How Python Packaging Works" useful for a
> general timeline:
>
>   https://www.youtube.com/watch?v=AQsZsgJ30AE
>
> The section "How did we get here" from Nick Coghlan's essay on a core
> packaging API is quite a bit more detailed:
>
>   http://python-notes.curiousefficiency.org/en/latest/pep_ideas/core_
> packaging_api.html#how-did-we-get-here
>
> D.
> On Sat, Oct 13, 2018 at 9:54 AM Константин 
> wrote:
> >
> > Hello!
> >
> > My name is Konstantin and I am going to participate in the PyPA event
> [1] next week. To understand the packaging landscape better, I've decided
> to create an article for myself generally, but I think it could be
> interesting for others as well because I've found that
> https://www.pypa.io/en/latest/history/ is not enough for me to understand
> how packaging works in general.
> >
> > If anyone has an interest, I would really appreciate any comments on it,
> either technical or just stylistic. https://medium.com/@knikitin/
> the-brief-history-of-package-distribution-in-python-5da35092964f , before
> I will publish it for a wider audience.
> >
> > Thank you!
> >
> > [1] https://generalassemb.ly/education/bloomberg-open-
> source-weekend-pypa/london/58402
> >
> >
> > --
> > Distutils-SIG mailing list -- distutils-sig@python.org
> > To unsubscribe send an email to distutils-sig-le...@python.org
> > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> > Message archived at https://mail.python.org/mm3/
> archives/list/distutils-sig@python.org/message/
> TVUQALCEYERESUZJHI46PDO2WMVIIWMC/
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/mm3/
> archives/list/distutils-sig@python.org/message/
> Q4K7IKGL6A3PHLFHZA6GCNMD4FLFV3I5/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/IC6YKHHFU7KHVEYC5AC67YMWFR7MCISI/

[Distutils] Re: History of python packaging

[Wes Turner]

https://www.pypa.io/en/latest/history/
- Doesn't mention that:

>  Significant parts of the implementation of setuptools were funded by the
Open Source Applications Foundation, to provide a plug-in infrastructure
for the Chandler PIM application.

https://setuptools.readthedocs.io/en/latest/history.html#credits

On Sunday, October 14, 2018, Wes Turner  wrote:

> https://packaging.python.org
> https://github.com/pypa/python-packaging-user-guide
> - A history.rst document would be a useful addition
>
> On Sunday, October 14, 2018, Dustin Ingram  wrote:
>
>> Hi Konstantin,
>>
>> You might find my PyCon talk "How Python Packaging Works" useful for a
>> general timeline:
>>
>>   https://www.youtube.com/watch?v=AQsZsgJ30AE
>>
>> The section "How did we get here" from Nick Coghlan's essay on a core
>> packaging API is quite a bit more detailed:
>>
>>   http://python-notes.curiousefficiency.org/en/latest/pep_
>> ideas/core_packaging_api.html#how-did-we-get-here
>>
>> D.
>> On Sat, Oct 13, 2018 at 9:54 AM Константин 
>> wrote:
>> >
>> > Hello!
>> >
>> > My name is Konstantin and I am going to participate in the PyPA event
>> [1] next week. To understand the packaging landscape better, I've decided
>> to create an article for myself generally, but I think it could be
>> interesting for others as well because I've found that
>> https://www.pypa.io/en/latest/history/ is not enough for me to
>> understand how packaging works in general.
>> >
>> > If anyone has an interest, I would really appreciate any comments on
>> it, either technical or just stylistic. https://medium.com/@knikitin/t
>> he-brief-history-of-package-distribution-in-python-5da35092964f , before
>> I will publish it for a wider audience.
>> >
>> > Thank you!
>> >
>> > [1] https://generalassemb.ly/education/bloomberg-open-source-
>> weekend-pypa/london/58402
>> >
>> >
>> > --
>> > Distutils-SIG mailing list -- distutils-sig@python.org
>> > To unsubscribe send an email to distutils-sig-le...@python.org
>> > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
>> > Message archived at https://mail.python.org/mm3/ar
>> chives/list/distutils-sig@python.org/message/TVUQALCEYERESUZ
>> JHI46PDO2WMVIIWMC/
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
>> Message archived at https://mail.python.org/mm3/ar
>> chives/list/distutils-sig@python.org/message/Q4K7IKGL6A3PHLF
>> HZA6GCNMD4FLFV3I5/
>>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/UDAWT6AYOS72MOFIO5ZBG564FVMSTPUS/

[Distutils] Re: setuptools API compatibility

[Wes Turner]

Setuptools setup.py scripts *are* generally forward compatible; though some
features are deprecated from time to time.

https://setuptools.readthedocs.io/en/latest/history.html
- Ctrl-F "Deprecated"

https://setuptools.readthedocs.io/en/latest/roadmap.html

https://cloud.google.com/appengine/docs/standard/python3/specifying-dependencies#declaring_and_managing_dependencies

I hadn't realized that AppEngine Python 3.7 now supports pip
requirements.txt files and C extensions.

On Friday, October 5, 2018,  wrote:

> Hi,
>
> Does setuptools make any API stability guarantees i.e. should all setup.py
> files that work with setuptools version X also work with setuptools version
> Y where  Y > X?
>
> I'm asking because the App Engine Python 3.7 runtime keeps it's version of
> setuptools up-to-date and I'm wondering if that is problematic due to API
> compatibility i.e. that package installation might fail due to an
> incompatibility introduced in setuptools.
>
> Cheers,
> Brian
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/mm3/
> archives/list/distutils-sig@python.org/message/
> 6UN6P7UCZVSS3XKM7CVNW7KRLQEYGPUW/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/VDWKRPFGPVFMTB3Y4I4VR2VGIX4HTOZH/

[Distutils] Re: SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register

[Wes Turner]

echo 2 > /sys/kernel/debug/x86/ibrs_enabled

(Coss-posting to distutils-sig, as C extensions may be the most likely
abuse vector)

# Forwarded message
From: Wes Turner 
Date: Mon, Sep 17, 2018 at 3:41 PM
Subject: Re: SEC: Spectre variant 2: GCC: -mindirect-branch=thunk
-mindirect-branch-register

Cc: Python-Dev 

On Mon, Sep 17, 2018 at 2:58 PM Wes Turner  wrote:

I thought I read that RH has a kernel flag for userspace?

"Controlling the Performance Impact of Microcode and Security Patches for
CVE-2017-5754 CVE-2017-5715 and CVE-2017-5753 using Red Hat Enterprise
Linux Tunables"
https://access.redhat.com/articles/3311301

> Indirect Branch Restricted Speculation (ibrs)
> [...] When ibrs_enabled is set to 1 (spectre_v2=ibrs) the kernel runs
with indirect branch restricted speculation, which protects the kernel
space from attacks (even from hyperthreading/simultaneous multi-threading
attacks). When IBRS is set to 2 (spectre_v2=ibrs_always), both userland and
kernel runs with indirect branch restricted speculation. This protects
userspace from hyperthreading/simultaneous multi-threading attacks as well,
and is also the default on certain old AMD processors (family 10h, 12h and
16h). This feature addresses CVE-2017-5715, variant #2.
> [...]
> echo 2 > /sys/kernel/debug/x86/ibrs_enabled

https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown/MitigationControls
> echo 2 > /proc/sys/kernel/ibrs_enabled will turn on IBRS in both
userspace and kernel

...
On Mon, Sep 17, 2018 at 5:26 AM Antoine Pitrou  wrote:
If you want to push this forward, I suggest you measure performance of
Python compiled with and without the Spectre mitigation options, and
report the results here.  That will help vendors and packagers decide
whether they want to pursue the route of enabling those options.

"Speculative Execution Exploit Performance Impacts - Describing the
performance impacts to security patches for CVE-2017-5754 CVE-2017-5753 and
CVE-2017-5715"
https://access.redhat.com/articles/3307751

- Revised worst-case peformance impact: 4-8%

On Sun, Sep 16, 2018 at 8:29 PM Wes Turner  wrote:

> Are all current Python builds and C extensions vulnerable to Spectre
> variants {1, 2, *}?
>
> There are now multiple threads:
>
> "SEC: Spectre variant 2: GCC: -mindirect-branch=thunk
> -mindirect-branch-register"
> -
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/4BGE226DB5EWIAT5VCJ75QD5ASOVJZCM/
> -
> https://mail.python.org/pipermail/python-ideas/2018-September/053473.html
> - https://mail.python.org/pipermail/python-dev/2018-September/155199.html
>
>
> Original thread (that I forwarded to security@):
> "[Python-ideas] Executable space protection: NX bit,"
> https://mail.python.org/pipermail/python-ideas/2018-September/053175.html
> > ~ Do trampolines / nested functions in C extensions switch off the NX
> bit?
>
> On Sunday, September 16, 2018, Nathaniel Smith  wrote:
>
>> On Wed, Sep 12, 2018, 12:29 Joni Orponen  wrote:
>>
>>> On Wed, Sep 12, 2018 at 8:48 PM Wes Turner  wrote:
>>>
>>>> Should C extensions that compile all add
>>>> `-mindirect-branch=thunk -mindirect-branch-register` [1] to mitigate
>>>> the risk of Spectre variant 2 (which does indeed affect user space
>>>> applications as well as kernels)?
>>>>
>>>
>>> Are those available on GCC <= 4.2.0 as per PEP 513?
>>>
>>
>> Pretty sure no manylinux1 compiler is ever going to get these mitigations.
>>
>> For manylinux2010 on x86-64, we can easily use a much newer compiler: RH
>> maintains a recent compiler, currently gcc 7.3, or if that doesn't work for
>> some reason then the conda folks have be apparently figured out how to
>> build the equivalent from gcc upstream releases.
>>
>
> Are there different CFLAGS and/or gcc compatibility flags in conda builds
> of Python and C extensions?
>
> Where are those set in conda builds?
>
> What's the best way to set CFLAGS in Python builds and C extensions?
>
> export CFLAGS="-mindirect-branch=thunk -mindirect-branch-register"
> ./configure
> make
>
> ?
>
> Why are we supposed to use an old version of GCC that doesn't have the
> retpoline patches that only mitigate Spectre variant 2?
>
>
>>
>> Unfortunately, the manylinux2010 infrastructure is not quite ready... I'm
>> pretty sure it needs some volunteers to push it to the finish line, though
>> unfortunately I haven't had enough time to keep track.
>>
>
> "PEP 571 -- The manylinux2010 Platform Tag"
> https://www.python.org/dev/peps/pep-0571/
>
> "Tracking issue for manylinux2010 rollout"
> https://github.com/pypa/manylinux/issues/179
>
> Are all current Python builds and C extensions vulnerable to Spectre
> variants {1, 2, *}?
>
>>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/ZAJQCMOPD3NFVUGG4MFVIY34C3HP45SF/

[Distutils] Re: SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register

[Wes Turner]

Are all current Python builds and C extensions vulnerable to Spectre
variants {1, 2, *}?

There are now multiple threads:

"SEC: Spectre variant 2: GCC: -mindirect-branch=thunk
-mindirect-branch-register"
-
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/4BGE226DB5EWIAT5VCJ75QD5ASOVJZCM/
- https://mail.python.org/pipermail/python-ideas/2018-September/053473.html
- https://mail.python.org/pipermail/python-dev/2018-September/155199.html


Original thread (that I forwarded to security@):
"[Python-ideas] Executable space protection: NX bit,"
https://mail.python.org/pipermail/python-ideas/2018-September/053175.html
> ~ Do trampolines / nested functions in C extensions switch off the NX bit?

On Sunday, September 16, 2018, Nathaniel Smith  wrote:

> On Wed, Sep 12, 2018, 12:29 Joni Orponen  wrote:
>
>> On Wed, Sep 12, 2018 at 8:48 PM Wes Turner  wrote:
>>
>>> Should C extensions that compile all add
>>> `-mindirect-branch=thunk -mindirect-branch-register` [1] to mitigate the
>>> risk of Spectre variant 2 (which does indeed affect user space applications
>>> as well as kernels)?
>>>
>>
>> Are those available on GCC <= 4.2.0 as per PEP 513?
>>
>
> Pretty sure no manylinux1 compiler is ever going to get these mitigations.
>
> For manylinux2010 on x86-64, we can easily use a much newer compiler: RH
> maintains a recent compiler, currently gcc 7.3, or if that doesn't work for
> some reason then the conda folks have be apparently figured out how to
> build the equivalent from gcc upstream releases.
>

Are there different CFLAGS and/or gcc compatibility flags in conda builds
of Python and C extensions?

Where are those set in conda builds?

What's the best way to set CFLAGS in Python builds and C extensions?

export CFLAGS="-mindirect-branch=thunk -mindirect-branch-register"
./configure
make

?

Why are we supposed to use an old version of GCC that doesn't have the
retpoline patches that only mitigate Spectre variant 2?


>
> Unfortunately, the manylinux2010 infrastructure is not quite ready... I'm
> pretty sure it needs some volunteers to push it to the finish line, though
> unfortunately I haven't had enough time to keep track.
>

"PEP 571 -- The manylinux2010 Platform Tag"
https://www.python.org/dev/peps/pep-0571/

"Tracking issue for manylinux2010 rollout"
https://github.com/pypa/manylinux/issues/179

Are all current Python builds and C extensions vulnerable to Spectre
variants {1, 2, *}?

>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/PCAV2LWACYHLIPHOADPAA6NY3TSUWWHZ/

[Distutils] Re: SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register

[Wes Turner]

"What is a retpoline and how does it work?"
https://stackoverflow.com/questions/48089426/what-is-a-retpoline-and-how-does-it-work

On Wednesday, September 12, 2018, Wes Turner  wrote:

> On Wednesday, September 12, 2018, Joni Orponen 
> wrote:
>
>> On Wed, Sep 12, 2018 at 8:48 PM Wes Turner  wrote:
>>
>>> Should C extensions that compile all add
>>> `-mindirect-branch=thunk -mindirect-branch-register` [1] to mitigate the
>>> risk of Spectre variant 2 (which does indeed affect user space applications
>>> as well as kernels)?
>>>
>>
>> Are those available on GCC <= 4.2.0 as per PEP 513?
>>
>
> AFAIU, only
> GCC 7.3 and 8 have the retpoline (indirect-branch=thunk) support enabled
> by the `-mindirect-branch=thunk -mindirect-branch-register` CFLAGS.
>
>>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/QCUJZLMBHJ7VDIAMSO6XUSTQY6GE4RLN/

[Distutils] Re: SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register

[Wes Turner]

On Wednesday, September 12, 2018, Joni Orponen 
wrote:

> On Wed, Sep 12, 2018 at 8:48 PM Wes Turner  wrote:
>
>> Should C extensions that compile all add
>> `-mindirect-branch=thunk -mindirect-branch-register` [1] to mitigate the
>> risk of Spectre variant 2 (which does indeed affect user space applications
>> as well as kernels)?
>>
>
> Are those available on GCC <= 4.2.0 as per PEP 513?
>

AFAIU, only
GCC 7.3 and 8 have the retpoline (indirect-branch=thunk) support enabled by
the `-mindirect-branch=thunk -mindirect-branch-register` CFLAGS.

>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/FMC5MIQDFXJD3MQHLLSYJRFRZQ2KPORI/

[Distutils] Re: Adopting virtualenv package maintenance

[Wes Turner]

"What is the difference between venv, pyvenv, pyenv, virtualenv,
virtualenvwrapper, pipenv, etc?"
https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe

virtualenvwrapper still depends on virtualenv, AFAIU


https://github.com/pypa/virtualenv/blob/master/CONTRIBUTING.rst


https://westurner.github.io/dotfiles/venv.html
https://github.com/westurner/dotfiles/blob/develop/scripts/venv_ipyconfig.py
https://github.com/westurner/dotfiles/blob/develop/scripts/venv_root_prefix.sh
https://github.com/westurner/dotfiles/blob/develop/scripts/venv_cdaliases.sh
https://github.com/westurner/dotfiles/blob/develop/scripts/venv_ipymagics.py
https://github.com/westurner/dotfiles/blob/develop/scripts/venv_relabel.sh
- SELinux file contexts


On Thursday, September 6, 2018, Bernat Gabor  wrote:

> I'm maintainer of tox and tox is heavily relying on virtualenv. I would be
> up for being a co maintainer, but for now I don't have availability for
> heavy undertakings. I've had a discussion with Donald at Pycon us about it
> and there certainly is a desire and place for virtualenv. the issue is
> mostly qualified people willing to invest time into it.
>
> On Thu, Sep 6, 2018, 21:44 Alex Becker  wrote:
>
>> Another +1 to the utility of a maintainer. I am also working on package
>> management and have found that venv is not a full replacement for
>> virtualenv--for example I don't believe the environment can be entered
>> programatically, while virtualenv provides activate_this.py which can be
>> exec'd. I'm sure there are many other limitations, so I don't think python
>> can give up on virtualenv soon.
>>
>> On Thu, Sep 6, 2018 at 12:48 AM Tzu-ping Chung 
>> wrote:
>>
>>> I don’t have the authority to do it, but I would really love to see
>>> virtualenv getting maintenance.
>>>
>>> Pipenv still depends on virtualenv because for not only Python 2. I am
>>> already working on switching
>>> to venv for Python 3.5+, but we will continue to need virtualenv for
>>> quite a while due to some
>>> compatibility issues, even on Python 3. We occasionally get bug reports
>>> about virtual environment
>>> creation, and those generally are due to some long standing bugs in
>>> virtualenv. It would be awesome
>>> if we could point reporters somewhere to actually get things fixed,
>>> instead of say sorry with a shrug.
>>>
>>> I am not suggesting in particular who should be the maintainer(s) (but
>>> also not objecting to Sorin’s
>>> proposal), but virtualenv really needs a maintainer.
>>>
>>> TP
>>>
>>>
>>> > On 05/9, 2018, at 16:56, Sorin Sbarnea 
>>> wrote:
>>> >
>>> > As it seems that virtualenv package is in need of some maintenance
>>> effort, focused mostly on doing reviews, closing or merging them and
>>> eventually doing a new release once a month.
>>> >
>>> > I know that virtualenv is in deprecation mode as its would be no
>>> longer needed when Python2 will no longer be used. The reality is that
>>> Python 2.x will still be in production after January 1st, 2020 because
>>> there are deployed products with LTS contracts which will need some time to
>>> get updated to newer versions that use py3. This automatically translates
>>> to the need to have a working virtualenv for testing them. I am part of the
>>> OpenStack team and I am sure that, even if I like it or not, I would have
>>> to deal with some amount of py2 even after the magic date.
>>> >
>>> > The current situation with virtualenv is pretty bad because there are
>>> lots of open pull-requests which are not reviewed or merged, mostly because
>>> there is nobody available to do that boring extra work. I had few changes
>>> that were improving the CI testing of virtualenv which soon will be one
>>> year old,... most of them without any feedback. Even finding whom to ping
>>> by email or irc was a challenge as I got two responses: no response at all
>>> or someone else telling me that they are not maintainers of the virtualenv
>>> package. Example https://groups.google.com/forum/#!topic/pypa-dev/
>>> YMVsRbNoVpg
>>> >
>>> > For these reasons I would like to become a maintainer for virtualenv,
>>> preferably working with two others on keeping it alive for a couple of
>>> years till we could organize a big wake ceremony for it.
>>> >
>>> > It would be preferable if two others would join the maintenance
>>> "taskforce" because merging a change should almost always involve at least
>>> two reviewers.
>>> >
>>> > While I cannot make any guarantees regarding dealing with all reported
>>> bugs, I can commit on assuring that there are no PRs that are not reviewed
>>> for longer than 30 days (aiming for one week). Now there are ~75 open PRs.
>>> I have being doing open source for a long time and I respect all the time
>>> and efforth put by project maintainers and at the same time I always tried
>>> to do my best dealign with incoming PRs because if someone spended his time
>>> trying to make a contribution that is 

[Distutils] Re: Environment markers for GPU/CUDA availibility

[Wes Turner]

Would warehouse need to be extended to support additional non-exclusive
environment markers?

On Monday, September 3, 2018, Nick Coghlan  wrote:

> On Mon., 3 Sep. 2018, 5:48 am Ronald Oussoren, 
> wrote:
>
>>
>> What’s the problem with including GPU and non-GPU variants of code in a
>> binary wheel other than the size of the wheel? I tend to prefer binaries
>> that work “everywhere", even if that requires some more work in building
>> binaries (such as including multiple variants of extensions to have
>> optimised code for different CPU variants, such as SSE and non-SSE variants
>> in the past).
>>
>
> As far as I'm aware, binary artifact size *is* the problem. It's just that
> once you're  automatically building and pushing an artifact (or an image
> containing that artifact) to thousands or tens of thousands of managed
> systems, the wasted bandwidth from pushing redundant implementations of the
> same functionality becomes more of a concern than the convenience of being
> able to use the same artifact across multiple platforms.
>
> Cheers,
> Nick.
>
>
>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/NIFRYUZTI2256FMK7YUP6VLNNBXWMCLI/

[Distutils] Re: Environment markers for GPU/CUDA availibility

[Wes Turner]

What are the conditionals/criteria?

- non Von Neumann (hardly debuggable)?
-

GPUs
- CUDA support
-

TPUs
-

If the GPU card is detected but the drivers aren't installed, what should
it do?

On Friday, August 31, 2018, Tzu-ping Chung  wrote:

> I’m not knowledgable about GPUs, but from limited conversations with
> others,
> it is important to first decide what exactly the problem area is. Unlike
> currently
> available environment markers, there’s currently not a very reliable way to
> programmatically determine even if there is a GPU, let alone what that GPU
> can
> actually do (not every GPU can be used by Tensorflow, for example).
>
> IMO it would likely be a good route to first implement some interface for
> GPU
> environment detection in Python. This interface can then be used in
> projects like
> tensorflow-auto-detect. Projects like Tensorflow can also detect directly
> what
> implementation it should use, like many projects do platform-specific
> things by
> detection os.name of sys.platform. Once we’re sure we have all the things
> needed
> for detection, markers can be drafted based on the detection interface.
>
> TP
>
>
> > On 01/9/2018, at 03:57, Dustin Ingram  wrote:
> >
> > Hi all, trying to pull together a few separate discussions into a
> > single thread here.
> >
> > The main issue is that currently PEP 508 does not provide environment
> > markers for GPU/CUDA availability, which leads to problems for
> > projects that want to provide distributions for environments with and
> > without GPU support.
> >
> > As far as I can tell, there's been multiple suggestions to bring this
> > issue to distutils-sig, but no one has actually done it.
> >
> > Relevant issues:
> >
> > (closed) "How should Python packages depending on TensorFlow structure
> > their requirements?"
> > https://github.com/tensorflow/tensorflow/issues/7166
> >
> > (closed) "Adding gpu or cuda specification in PEP 508"
> > https://github.com/python/peps/issues/581
> >
> > (closed) "More support for conditional installation"
> > https://github.com/pypa/pipenv/issues/1353
> >
> > (no response) "Adding gpu or cuda markers in PEP 508"
> > https://github.com/pypa/interoperability-peps/issues/68
> >
> > There is now a third-party project which attempts to amend this for
> > tensorflow (https://github.com/akatrevorjay/tensorflow-auto-detect)
> > but this approach is somewhat fragile (depends on version numbers
> > being in sync), doesn't directly scale to all similar projects, and
> > would require maintainers for a given project to maintain _three_
> > separate projects, instead of just one.
> >
> > I'm not intimately familiar with PEP 508, so my questions for this list:
> >
> > * Is the demand sufficient to justify supporting this use case?
> > * Is it possible to add support for GPU Environment markers?
> > * If so, what would need to be done?
> > * If implemented, what should the transition look like for projects
> > like tensorflow?
> >
> > Thanks!
> > D.
> > --
> > Distutils-SIG mailing list -- distutils-sig@python.org
> > To unsubscribe send an email to distutils-sig-le...@python.org
> > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> > Message archived at https://mail.python.org/mm3/
> archives/list/distutils-sig@python.org/message/
> LXLF4YSC4WUZOYRX65DW7CESIX7UUBK5/
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/mm3/
> archives/list/distutils-sig@python.org/message/
> 3VLRDFVQS7E7BBAYF6WQQ5TM2QJMVEDX/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/FXPQ4HBMX2A4LZSFHMCGA6RW6RQVBIMJ/

[Distutils] Re: pipenv and pip

[Wes Turner]

On Monday, August 20, 2018, Paul Moore  wrote:

> On Mon, 20 Aug 2018 at 10:54, Wes Turner  wrote:
> >
> > What stable API would be worth maintaining in pip for others to use?
>
> That's probably the sort of question that can only be usefully
> answered by projects like pipenv identifying the functionality they
> need and proposing something. Which is of course one of the reasons we
> (the pip devs) advise against "just using pip's internals", because it
> means we never get that information in any useful form.
>
> > Who is offering to maintain a stable API in/with/for pip and the Python
> community ad infinitum?
>
> That's the crux of the problem - basically the answer is "no-one".
> What we advocate is for generally useful functionality to be split out
> into standalone libraries, and then pip, as well as other consumers,
> can use those libraries. We already have that with the packaging
> library and the script wrappers (part of distlib). The new resolver is
> being developed as a standalone library (zazo) as is the PEP 517 hook
> wrapper functionality (pep517). There's no reason this model couldn't
> work in other areas. (But even then, the question "who's offering to
> write these libraries" still applies :-()


Thanks!


>
> >> > As we handle some resolution, which isn’t really something pip does,
> there is no cli interface to achieve this. I maintain a library (as of last
> week) which provides compatibility shims between pip versions 8-current. It
> is a good idea to use the cli, but we already spend enough resources
> forking subprocesses into the background that it is a lot more efficient to
> use the internals, which I track quite closely. The preference toward cli
> interaction is largely to allow internal api breakage which we don’t mind.
> >
> > What is the URL of this library of which you are speaking?
>
> I know "security by obscurity" doesn't work, but I'm happier if
> details of this library *aren't* widely known - it's not something I'd
> want to encourage people using, nor is it supported by pip, as it's
> basically a direct interface into pip's internal functions, papering
> over the name changes that we did in pip 10 specifically to dissuade
> people from doing this.


If someone was committing to identifying useful API methods, parameters,
and return values;
writing a ~PEP;
implementing said API;
and maintaining backwards compatible shims for some reason;
would something like `pip.api` be an appropriate namespace?
(now that we're on version 18 with a faster release cycle)?


>
> Paul
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/HTD4EF5WIQHMOZAJU4POMA6TI3WSEUMO/

[Distutils] Re: pipenv and pip

[Wes Turner]

What stable API would be worth maintaining in pip for others to use?

"[Distutils] Announcement: Pip 10 is coming, and will move all internal
APIs"
https://groups.google.com/forum/m/#!topic/pypa-dev/JVTfS6ZdAuM



On Monday, August 20, 2018, Chris Jerdonek  wrote:

> Thanks. Is the state of affairs as you described them what you're
> planning for the future as well, or do you anticipate any changes
> worthy of note?
>
> Also, are any of the bugs filed in pipenv's tracker due to bugs or
> rough spots in pip -- is there a way to find those, like by using a
> label? It would be good to be able to know about those so pip can
> improve and become more useful. It doesn't seem like any bugs have
> been filed in pip's tracker in the past year by any of pipenv's top
> contributors. That seems a bit surprising to me given pipenv's heavy
> reliance on pip (together with the fact that I know pip has its share
> of issues), or is there another way you have of communicating
> regarding things that interconnect with pip?


Label ideas?
- 'stable-api'
-

Who is offering to maintain a stable API in/with/for pip and the Python
community ad infinitum?


> Thanks,
> --Chris
>
>
>
> On Mon, Aug 20, 2018 at 12:51 AM, Dan Ryan  wrote:
> > Sure I can grab that— we patch pip because we use some internals to
> handle resolution and we have some bugs around that currently. They aren’t
> upstreamed because they aren’t actually present in pip, only in pipenv.
> Pipenv crosses back and forth across the virtualenv boundary during the
> process. Pipenv relies on piptools and vendors a patched version of pip to
> ensure consistency as well as to provide a few hacks around querying the
> index.  We do have a bit of reimplementation around some kinds of logic,
> with the largest overlap being in parsing of requirements.
> >
> > As we handle some resolution, which isn’t really something pip does,
> there is no cli interface to achieve this. I maintain a library (as of last
> week) which provides compatibility shims between pip versions 8-current. It
> is a good idea to use the cli, but we already spend enough resources
> forking subprocesses into the background that it is a lot more efficient to
> use the internals, which I track quite closely. The preference toward cli
> interaction is largely to allow internal api breakage which we don’t mind.


What is the URL of this library of which you are speaking?


> >
> > For the most part, we have open channels of communication as necessary.
> We rely as heavily as we can on pip, packaging, and setuptools to connect
> the dots, retrieve package info, etc.


An issue label and something like a PEP would likely survive the ravages of
10 years of tools tooling around with community packaging commitments.


> >
> > Dan Ryan // pipenv maintainer
> > gh: @techalchemy
> >
> >> On Aug 20, 2018, at 2:41 AM, Chris Jerdonek 
> wrote:
> >>
> >> Hi,
> >>
> >> Can someone explain to me the relationship between pipenv and pip,
> >> from the perspective of pipenv's maintainers?
> >>
> >> For example, does pipenv currently reimplement anything that pip tries
> >> to do, or does it simply call out to pip through the CLI or through
> >> its internal API's? Does it have any preferences or future plans in
> >> this regard? How about upstreaming to pip fixes or things that would
> >> help pipenv?
> >>
> >> I've been contributing to pip more lately, and I had a look at
> >> pipenv's repository for the first time today.
> >> https://github.com/pypa/pipenv
> >>
> >> Given that pip's code was recently made internal, I was a bit
> >> surprised to see that pipenv vendors and patches pip:
> >> https://github.com/pypa/pipenv/tree/master/pipenv/patched/notpip
> >> Before I had always assumed that pipenv used pip's CLI (because that's
> >> what pip says you should do).
> >>
> >> I also noticed that some bugs in pipenv's tracker seem closely related
> >> to pip's behavior, but I don't recall seeing any bugs or PR's in pip's
> >> tracker reported from pipenv maintainers.
> >>
> >> Without knowing a whole lot more than what I've stated, one concern I
> >> have is around fragmentation, duplication of work, and repeating
> >> mistakes (or introducing new ones) if a lot of work is going into
> >> pipenv without coordinating with pip. Is this in any way similar to
> >> the beginning of what happened with distutils, setuptools, and
> >> distribute that we are still recovering from?
> >>
> >> --Chris
> >> --
> >> Distutils-SIG mailing list -- distutils-sig@python.org
> >> To unsubscribe send an email to distutils-sig-le...@python.org
> >> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> >> Message archived at https://mail.python.org/mm3/
> archives/list/distutils-sig@python.org/message/
> 2QECNWSHNEW7UBB24M2K5BISYJY7GMZF/
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> 

[Distutils] Re: Is ensurepip still a thing?

[Wes Turner]

Is there a build script that automates vendoring in the latest pip? I see
checkpip.py, which only checks the version.

https://docs.python.org/3/library/ensurepip.html

https://www.python.org/dev/peps/pep-0453/#updating-the-private-copy-of-pip

https://github.com/python/cpython/blob/master/Tools/scripts/checkpip.py

https://github.com/pypa/get-pip/blob/master/tasks/generate.py

On Monday, August 6, 2018, Chris Barker via Distutils-SIG <
distutils-sig@python.org> wrote:

> I'm updating some instructions for my students, in which the first thing I
> do is have them run ensurepip:
>
> $ python3 -m ensurepip --upgrade
>
> which resulted in:
>
> $  python3 -m ensurepip --upgrade
> Looking in links: /var/folders/ym/tj87fc850yd6526nbrn14rxmgn
> /T/tmpwc8nd6oj
> Requirement already up-to-date: setuptools in /Library/Frameworks/Python.
> framework/Versions/3.7/lib/python3.7/site-packages (39.0.1)
> Requirement already up-to-date: pip in /Library/Frameworks/Python.
> framework/Versions/3.7/lib/python3.7/site-packages (10.0.1)
>
> (this is after a brand-new python 3.7 install on OS-X from python.org)
>
> All good. But then I use pip, and get (after a successful install):
>
> You are using pip version 10.0.1, however version 18.0 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
>
> Huh? shouldn't ensurepip have updated it for me already??
>
> Or should I simply suggest the
>
> pip install --upgrade pip
>
> command and not bother with ensurepip anymore?
>
> BTW -- shouldn't that be:
>
> python3 -m pip install --upgrade pip
>
> to make sure they get the "right" pip?
>
> KInda hard to keep up with the latest ...
>
> Thanks,
>
> -CHB
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/UYO7QQ45LRMPOZYVQORZYJNSQ7NHKHLT/

[Distutils] Re: Packaging Advice for EFF's Certbot

[Wes Turner]

Yw!

* Pytest, tox, and devpi handle plugins with pluggy; which works with OS
and Python package installs AFAICT:
https://pluggy.readthedocs.io/en/latest/

* IDK what's needed to remove a package from a repo because it's
dangerously out of date.

* Arrow recently gained lots of build/test/packaging scripts (that run on
TravisCI and AppVeyor):
- https://github.com/apache/arrow/blob/master/.travis.yml
  - TIL about 'after_success:'
- https://github.com/apache/arrow/tree/master/ci


* A few certbot links that you're probably already aware of:
- https://github.com/certbot/certbot/blob/master/.travis.yml
- https://github.com/certbot/certbot/blob/master/tox.ini


* And the TravisCI deployment docs:
  - https://docs.travis-ci.com/user/deployment/
  - https://docs.travis-ci.com/user/deployment/releases/ (GitHub releases)
  - https://docs.travis-ci.com/user/deployment/pypi/
  - https://docs.travis-ci.com/user/deployment/launchpad/
  - [ ] PPA
  - [ ] EPEL
  - [ ] Copr (Fedora RPM builds for yum/dnf)
  - [ ] Conda-forge


On Wednesday, August 1, 2018, Brad Warren  wrote:

>
> >> On Jul 24, 2018, at 4:36 AM, Nick Coghlan  wrote:
> >>
> >> However, there *are* folks that have been working on allowing
> >> applications to be defined primarily as Python projects, and then have
> >> the creation of wrapper native installers be a pushbutton exercise,
> >> rather than requiring careful human handholding.
> >
> > But it sounds like they also want to be able to install/remove/upgrade
> > *parts* of the Python project, for their plugin support.
> That’s correct. We currently have 18 official plugins for Certbot with
> plans to add more and a few dozen third-party plugins.
> > Do any of these tools allow that?
> This is a good question. If we went with something like dh-virtualenv or
> packaged virtualenvs with fpm, would we be able to have separate packages
> for our plugins that installed things in the same virtualenv? I haven’t
> looked into this yet, but I wouldn’t expect this to work.
> > That's the thing that really made me think about conda.
> My biggest concern with conda right now is I believe we (or our users)
> would be on their own for setting up a cron job or systemd timer for
> renewal. Is this correct?
>
> > On Jul 24, 2018, at 11:20 PM, Chris Jerdonek 
> wrote:
> >
> > Just to be clear, I wasn't meaning to promote or recommend the Docker
> > option I described.
> Sure! After reading your 2nd email describing how this would work in more
> detail, I think this would require a pretty major rewrite to how Certbot
> currently works. Given the other downsides, I’m not sure this is the best
> approach for us, but I appreciate you throwing out the idea regardless just
> in case it was!
>
> > On Jul 26, 2018, at 3:20 AM, Ben Finney via Distutils-SIG <
> distutils-sig@python.org> wrote:
> > Just focus on Certbot, and cheer from the sidelines as the OS
> distributions
> > do the work of third party packages.
> >
> > Yes, that's a different set of problems (for example, keeping Certbot
> > compatible with those versions of the libraries the OS repositories
> > provide).
> We’ve done the work to maintain compatibility with the older versions of
> our dependencies available in the official OS repos where we are packaged.
> The source of our problems with official OS packaging is described below
> and in the Google Doc linked in my initial email.
>
> > On Jul 28, 2018, at 8:53 AM, matt...@woodcraft.me.uk wrote:
> >
> > I wouldn't be too put off by the idea of Debian politics. Certbot should
> be a good fit for stable-updates:
>
> We thought so too and getting updates like this was our main packaging
> plan when we launched in 2015. Unfortunately, it hasn’t gone well and is
> the main reason we’re seeking our own solution. Perhaps we did something
> incorrectly, but as Nathaniel pointed out, Certbot is broken in Debian
> Stretch and has been since January. The same and many other problems affect
> the packages in Ubuntu Xenial. We’ve also struggled to find people to help
> maintain our PPA for older, non-EOL’d versions of Ubuntu.
>
> If anyone reading this would like to help us solve these problems or know
> someone who would, please reach out off-list. While these packages exist in
> OS repos, some users will continue to use them regardless of the
> alternative packaging approach we take. Unless the current issues are
> resolved and we’re confident new ones in the future will be fixed quickly
> as well, I think we need to offer alternative packaging so we can provide
> our users some means of getting a working version of Certbot.
>
> > On Jul 28, 2018, at 12:00 PM, Wes Turner  wrote:
> >
> > Took a minute mo

[Distutils] Re: Packaging Advice for EFF's Certbot

[Wes Turner]

Took a minute more to read the gdoc link.

The CI build should produce every {OS,} package as a build artifact (along
with coverage and static analysis reports). #goeswithoutsaying

Someone can probably point to a good GitLab CI config that produces all of
the requisite artifacts and uploads them to the appropriate repos (with an
optional manual approval step).


## a new package updater tool
- IIRC, cloudmatrix/esky has transactional updates but is no longer updated.
  - what are other good ways way to do auto-updating transactional updates?

- It makes sense to *start with TUF* (which is derived from the Tor updater)

- Polling for updates costs time and money.

## systemd/cron config
Official repos with e.g. Puppet, Salt, Ansible configs would be
advantageous because:
- certbot should be run on an appropriately secured baseline (with
filesystem labels)
- otherwise there are like 10 not even forks of unofficial configs and no
consensus

These also need to be part of the CI build.


On Thursday, July 26, 2018, Wes Turner  wrote:

>
>
> >> If we didn’t want to trust any binaries built by someone else or
> proprietary code, how much work would that be?
>
> - Docker Notary (The Update Framework)
> - PEP 458, PEP 480 (TUF)
> - Host GPG .asc(s) for things you just found
>
>
> ## To build the whole toolchain yourself?
> Build, Package, Install, Upgrade/Replace_with_new_container
>
> - Makefile
> - Conda-forge recipes with CI (meta.yaml, build.sh)
>   - [x] platform / architecture compilation optimizations
>   - [x] support for Windows (build.bat)
>   - [x] support for Mac & Linux (build.sh)
>
> - conda constructor
>   - build an installer
> - tar up the whole virtualenvwrapper virtualenv and unpack that into the
> exact same path in a container
>   - and check the .ASC
>
>
> ## To sign trusted releases:
> - See: Warehouse & GPG, Wheel & signatures
> - *The Update Framework*
>   - There are plans to merge support for TUF (which is designed to
> withstand package repo compromise) into Warehouse at some point, AFAIU.
>
>
> TUF: The Update Framework
> 
> | Wikipedia: https://en.wikipedia.org/wiki/The_Update_Framework_(TUF)
> | Homepage: https://theupdateframework.github.io/
> | Src: https://github.com/theupdateframework
> | Spec: https://github.com/theupdateframework/
> specification/blob/master/tuf-spec.md
>
> - Docker Notary supports TUF
> - TUF is mostly written in Python
> - Python PEP458:
>   - https://www.pypa.io/en/latest/roadmap/#pypi-integrate-tuf
>
> - "PEP 458 -- Surviving a Compromise of PyPI" https://www.python.org/dev/
> peps/pep-0458/"
>
> - "PEP 480 -- Surviving a Compromise of PyPI: The Maximum Security Model"
> https://www.python.org/dev/peps/pep-0480/
>
> - https://github.com/pypa/pip/issues/1035
>   "Implement "hook" support for package signature verification."
>
>
> ## To build native packages for your package and the whole toolchain?
> - fpm
> - release keys (note that any key in the ring can sign for any package)
> - docker/kubernetes
> - conda-forge (Appveyor, CircleCI, Travis CI)
>   - OSX instances
>
>
> fpm
> =
> | Src: https://github.com/jordansissel/fpm
> | Docs: https://fpm.readthedocs.io/en/latest/
> | Docs: https://fpm.readthedocs.io/en/latest/source/virtualenv.html
>
> - { deb, rpm, osxpkg, tar, }
>
>
> ## To host package repositories yourselves:
>
> Pulp
> =
> | Homepage: https://pulpproject.org/
> | Src: https://github.com/pulp/
>
>
> ## To sign releases uploaded to e.g. GitHub (which are CDN'd, AFAIU):
> - Wheel had a native signing mechanism that's since been removed
> - [ ] Attach a GPG .asc signature to the GitHub release
>   - https://wiki.debian.org/Creating%20signed%20GitHub%20releases
>
>
> ## To sign commits and merges:
> - Configure a per-project GPG agent and keyring config
> - Trust local datetimes
> - Remember that `git push -f` can occur.
>
>
> ## Include SELinux filesystem labels with the package:
> - This isn't possible with Python packaging (without extending setup.py
> with a custom command with a build dependency or calling out to a shell
> script that may be locatable with distutils.spawn.find_executable)
>
> On Thu, Jul 26, 2018 at 1:41 PM Nathaniel Smith  wrote:
>
>> On Thu, Jul 26, 2018, 05:48 Ben Finney via Distutils-SIG <
>> distutils-sig@python.org> wrote:
>>
>>> Brad Warren  writes:
>>>
>>> > Our main use case has been the long tail of individuals or small teams
>>> > of sysadmins who maintain servers and need or want help and automation
>>> > around maintaining a secure TLS confi

[Distutils] Re: Summary of PyPI overhaul in new LWN article

[Wes Turner]

Hey, what's the latest on this?

- Python PEP458:
  - https://www.pypa.io/en/latest/roadmap/#pypi-integrate-tuf


On Wed, Apr 18, 2018 at 5:43 PM Trishank Kuppusamy <
trishank.kuppus...@datadoghq.com> wrote:

>
> https://github.com/pypa/pip/compare/10.0.0...DataDog:trishankatdatadog/10.0.0.tuf-in-toto
>
> --
> https://keybase.io/trishankdatadog
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/2S4DSBNVKO4CWP4KYGVYFVQ7KP4JHJOT/

[Distutils] Re: Packaging Advice for EFF's Certbot

[Wes Turner]

>> If we didn’t want to trust any binaries built by someone else or
proprietary code, how much work would that be?

- Docker Notary (The Update Framework)
- PEP 458, PEP 480 (TUF)
- Host GPG .asc(s) for things you just found


## To build the whole toolchain yourself?
Build, Package, Install, Upgrade/Replace_with_new_container

- Makefile
- Conda-forge recipes with CI (meta.yaml, build.sh)
  - [x] platform / architecture compilation optimizations
  - [x] support for Windows (build.bat)
  - [x] support for Mac & Linux (build.sh)

- conda constructor
  - build an installer
- tar up the whole virtualenvwrapper virtualenv and unpack that into the
exact same path in a container
  - and check the .ASC


## To sign trusted releases:
- See: Warehouse & GPG, Wheel & signatures
- *The Update Framework*
  - There are plans to merge support for TUF (which is designed to
withstand package repo compromise) into Warehouse at some point, AFAIU.


TUF: The Update Framework

| Wikipedia: https://en.wikipedia.org/wiki/The_Update_Framework_(TUF)
| Homepage: https://theupdateframework.github.io/
| Src: https://github.com/theupdateframework
| Spec:
https://github.com/theupdateframework/specification/blob/master/tuf-spec.md

- Docker Notary supports TUF
- TUF is mostly written in Python
- Python PEP458:
  - https://www.pypa.io/en/latest/roadmap/#pypi-integrate-tuf

- "PEP 458 -- Surviving a Compromise of PyPI"
https://www.python.org/dev/peps/pep-0458/;

- "PEP 480 -- Surviving a Compromise of PyPI: The Maximum Security Model"
https://www.python.org/dev/peps/pep-0480/

- https://github.com/pypa/pip/issues/1035
  "Implement "hook" support for package signature verification."


## To build native packages for your package and the whole toolchain?
- fpm
- release keys (note that any key in the ring can sign for any package)
- docker/kubernetes
- conda-forge (Appveyor, CircleCI, Travis CI)
  - OSX instances


fpm
=
| Src: https://github.com/jordansissel/fpm
| Docs: https://fpm.readthedocs.io/en/latest/
| Docs: https://fpm.readthedocs.io/en/latest/source/virtualenv.html

- { deb, rpm, osxpkg, tar, }


## To host package repositories yourselves:

Pulp
=
| Homepage: https://pulpproject.org/
| Src: https://github.com/pulp/


## To sign releases uploaded to e.g. GitHub (which are CDN'd, AFAIU):
- Wheel had a native signing mechanism that's since been removed
- [ ] Attach a GPG .asc signature to the GitHub release
  - https://wiki.debian.org/Creating%20signed%20GitHub%20releases


## To sign commits and merges:
- Configure a per-project GPG agent and keyring config
- Trust local datetimes
- Remember that `git push -f` can occur.


## Include SELinux filesystem labels with the package:
- This isn't possible with Python packaging (without extending setup.py
with a custom command with a build dependency or calling out to a shell
script that may be locatable with distutils.spawn.find_executable)

On Thu, Jul 26, 2018 at 1:41 PM Nathaniel Smith  wrote:

> On Thu, Jul 26, 2018, 05:48 Ben Finney via Distutils-SIG <
> distutils-sig@python.org> wrote:
>
>> Brad Warren  writes:
>>
>> > Our main use case has been the long tail of individuals or small teams
>> > of sysadmins who maintain servers and need or want help and automation
>> > around maintaining a secure TLS configuration.
>>
>> For what it's worth, I certainly concur that most people in the group
>> you describe will thank you to not have a bundle of custom dependencies
>> from outside the OS repository, and instead make Certbot work with
>> (and/or be advocates to introduce) the dependencies as OS-provided
>> library packages.
>>
>
> I wonder if they have any user survey or anything that would speak to this.
>
> FWIW my impression is that the kinds of sysadmins who know enough to have
> opinions about packaging hygiene also know enough to set up one of
> certbot's many simpler, less magical competitors. Certbot's target audience
> has always been folks who didn't really understand any of this stuff and
> wanted to just hit a button and have things somehow work out by whatever
> means necessary. Which is great, those people deserve secure
> communications. But the point is that you can't make everything your number
> one priority, and when they have to choose, certbot has chosen to
> prioritize "make it work" over "fit nicely into a traditional distro", and
> I think that's the right decision for what they are.
>
> > This is definitely our preferred approach to building native packages
>> > right now. To be honest, no one on my team has any experience building
>> > .debs and .rpms and we’re happy to learn what we need to if we go with
>> > this approach, but the more reliable automation around the process we
>> > can use the better.
>>
>> For Debian, instead of becoming packaging experts yourselves, instead we
>> ask that you make Certbot easier for *us* to package and maintain in
>> Debian. See https://wiki.debian.org/UpstreamGuide>; other OS
>> projects 

[Distutils]Re: Handing over default BDFL-Delegate responsibilities for packaging interoperability PEPs to Paul Moore

[Wes Turner]

Thanks Nick and Paul!

Are there a few links which best describe the state of python packaging
interoperability PEPs?

https://www.pypa.io/en/latest/
https://www.pypa.io/en/latest/roadmap/

https://packaging.python.org/

https://github.com/pypa/interoperability-peps
https://github.com/pypa/interoperability-peps/issues

Are there any things you think should be prioritized going forward?

On Friday, July 6, 2018, Nathaniel Smith  wrote:

> Nick, thanks so much for your service in an often thankless job. It is
> appreciated! And Paul, thanks for taking this on!
>
> On Fri, Jul 6, 2018, 19:08 Nick Coghlan  wrote:
>
>> Hi folks,
>>
>> Since 2013, I've been the default BDFL-Delegate for packaging
>> interoperability PEPs. In that time, the Python packaging ecosystem
>> has moved forward in a lot of different areas, with pip being shipped
>> by default with CPython, the wheel binary packaging format reaching
>> ever-increasing heights of popularity, the cross-distro manylinux ABI
>> compatibility specification being developed, the new pyproject.toml
>> based sdist format being defined, the PSF's Packaging Working Group
>> being formed, the Python Packaging User Guide being developed, and
>> various aspects of the packaging metadata being enhanced to improve
>> the general user experience of the Python packaging ecosystem.
>>
>> The role of the BDFL-Delegate in that process is partly about making
>> arbitrary decisions when arbitrary decisions need to be made ("The
>> bikeshed shall be green!"), but also about helping to guide
>> discussions in productive directions, as well as determining when more
>> complex PEP level proposals have reached a sufficient level of
>> consensus that it makes sense to provisionally accept them and move on
>> to publishing reference implementations.
>>
>> While it's been a fascinating ~5 years, I've decided that it's time
>> for me to hand over those responsibilities to another PyPA
>> contributor. With Guido's approval, I've asked Paul Moore if he'd be
>> willing to take on the role, and Paul has graciously accepted the
>> additional responsibility.
>>
>> Paul's a long term pip contributor, and also a CPython core developer,
>> with a lot of practical experience in getting Python (and Python
>> packaging) to work well in Windows environments. He's also a familiar,
>> calm, and constructive presence in design discussions within
>> distutils-sig, pip and other PyPA projects, which is an important
>> characteristic when taking on BDFL-Delegate responsibilities.
>>
>> I'd like to personally thank Paul for being willing to take on this
>> task, and I look forward to many more productive design discussions!
>>
>> Cheers,
>> Nick.
>>
>> P.S. I'm not stepping down from Python packaging related activities
>> entirely, as I'll still be involved in Python Packaging User Guide and
>> pipenv maintenance, and will continue as a member of the PSF's
>> Packaging Working Group. However, the final sign-off for packaging
>> interoperability PEPs will now rest with Paul or someone else that
>> he appoints, rather than with me :)
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
>> Message archived at https://mail.python.org/mm3/
>> archives/list/distutils-sig@python.org/message/
>> QT7SKORCF6OKWO3OVP5KO6XNGU2AR6TU/
>>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/MSQ4XAFSZ4N5KZVM3LQJYS74U4RV7TMS/

[Distutils] Re: sudo pip install: install pip files into /usr/local on Linux?

[Wes Turner]

I like to be able to diff the dotfiles in my home directory. Sometimes,
just copying from one machine to another clobbers useful/necessary machine
defaults (eg. $PATH).

I don't want to add my entire home directory to version control because
slow, so I wrote a shell script to copy existing files out of the way and
then symlink from e.g. ~/.bashrc to ~/-dotfiles/etc/bash. That way, I can
'cdd[otfiles]' (cd $__DOTFILES) and 'git status etc/.bashrc' and 'git diff'.

It's a shell script in order to avoid having a dependency on python on eg
mobile/embedded systems, but it does the job. I'm sure there are various
other approaches to managing dotfiles within version control.

https://github.com/westurner/dotfiles/blob/develop/scripts/bootstrap_dotfiles.sh#L278

Is being able to diff shell configuration a security advantage? You'd need
to be pretty sophisticated to be making changes on disk; such as dropping a
script or a binary with execute permissions into a directory at the top of
the $PATH.

On Friday, June 1, 2018, Nick Coghlan  wrote:

> On 1 June 2018 at 02:11, Wes Turner  wrote:
>
>> On Thursday, May 31, 2018, Matthias Klose  wrote:
>>
>>> On 26.05.2018 14:59, Nick Coghlan wrote:
>>>
>>> > Yep, after all the other entries. I actually think Debian/Ubuntu may
>>> have
>>> > changed their default set up as well somewhere along the line, but
>>> even if
>>> > they did, it potentially wouldn't change the settings for existing user
>>> > profiles (depending on exactly how they implemented it).
>>>
>>> It's on the path by default in Debian and Ubuntu, only for new users
>>> (~/.profile).
>>
>>
>> I believe ~/.profile is copied from /etc/skel/.profile on most systems.
>>
>
> Right, but distro upgrades are now regularly reliable enough that folks
> may go for years without creating a fresh user profile for themselves. For
> me personally, even when I do set up a fresh machine with a clean Fedora
> install, I'll rsync my old home directory over the top of the new one.
>
> This kind of thing means that even when distros change their default
> settings, a *lot* of users will still have the old default.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/5KZ7H5C5BT3YQLSM35B5X2QY7XFM63AI/

[Distutils] Re: sudo pip install: install pip files into /usr/local on Linux?

[Wes Turner]

On Thursday, May 31, 2018, Matthias Klose  wrote:

> On 26.05.2018 14:59, Nick Coghlan wrote:
> > On Sat., 26 May 2018, 4:25 am Donald Stufft,  wrote:
> >
> >>
> >>
> >> On May 25, 2018, at 12:44 PM, Thomas Kluyver 
> wrote:
> >>
> >> It's more annoying for scripts - on common Linux distributions, the user
> >> scripts location ~/.local/bin is not on PATH by default.
> >>
> >>
> >>
> >> It’s on $PATH by default in Fedora I think.
> >>
> >
> > Yep, after all the other entries. I actually think Debian/Ubuntu may have
> > changed their default set up as well somewhere along the line, but even
> if
> > they did, it potentially wouldn't change the settings for existing user
> > profiles (depending on exactly how they implemented it).
>
> It's on the path by default in Debian and Ubuntu, only for new users
> (~/.profile).


I believe ~/.profile is copied from /etc/skel/.profile on most systems.


> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/mm3/
> archives/list/distutils-sig@python.org/message/
> GCDA3XQSGF5PFZXR3KTMTPUB67UBFUFW/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/L63R6R3PJZPQ7AJ6LWDB2PZMIBFJ6AIF/

[Distutils] Re: sudo pip install: install pip files into /usr/local on Linux?

[Wes Turner]

For secured production applications, the user running the app should not be
able to preempt system binaries or overwrite user-writeable config in $HOME.

We tend to compromise on the side of developer-friendliness over secure by
default.

Is pip a tool for development or a tool for production deployments? Pip is
definitely a tool for development. There are lots of packaging systems for
production deployments which can handle e.g. file permissions and modifying
/etc config. Pip is sometimes a tool used for production deployment.

On Friday, May 25, 2018, Thomas Kluyver <tho...@kluyver.me.uk> wrote:

> On Fri, May 25, 2018, at 6:58 PM, Wes Turner wrote:
>
> ~/.local/bin is user-writeable. If ~/.local was on PATH or by default, it
> could potentially preempt/modify the behavior of system libraries and
> binaries; which is a security risk.
>
>
> I've heard this argument before, and it doesn't stand up, because files
> like .profile and .bashrc are user writable, and you can use those to add a
> directory to PATH (among many other things). You may be able to come up
> with some corner case where it's possible to modify ~/.local/bin but not
> ~/.profile, but it's pretty clear that this is a post-hoc rationalisation,
> not a real reason.
>
> It's like that, I strongly suspect, just because that's how it's been
> forever, and the people who are inconvenienced by it know how to work
> around it.
>
> Thomas
>
--
Distutils-SIG mailing list
distutils-sig@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/YDTCJJHIZWP3S6SMHQJ5YARUT5I3FW4Z/

[Distutils] Re: sudo pip install: install pip files into /usr/local on Linux?

[Wes Turner]

On Friday, May 25, 2018, Wes Turner <wes.tur...@gmail.com> wrote:

>
>
> On Friday, May 25, 2018, Thomas Kluyver <tho...@kluyver.me.uk> wrote:
>
>> On Fri, May 25, 2018, at 5:11 PM, Victor Stinner wrote:
>> > As an user, I want to use "sudo pip install" because packages
>> > installed in /usr (or /usr/local) are accessible without having to
>> > touch PYTHONPATH: the install directory is part of the default
>> > sys.path.
>>
>> This is also true for "pip install --user", at least on the systems I'm
>> familiar with. There are options to disable 'user site packages', but it's
>> enabled by default.
>>
>> It's more annoying for scripts - on common Linux distributions, the user
>> scripts location ~/.local/bin is not on PATH by default.
>
>
> ~/.local/bin is user-writeable. If ~/.local was on PATH or by default, it
> could potentially preempt/modify the behavior of system libraries and
> binaries; which is a security risk.
>

*If ~/.local was on $PATH or sys.path by default


>
> ~/.local/bin/bash could be wrapper script that logs commands, for example.
> If it's first in the path (as e.g. Homebrew does, IIRC), it's run when bare
> `bash` or `/usr/bin/env bash` are executed.
>
> pipsi creates isolated virtualenvs per-install which are isolated from
> other library installs, but each env then must be upgraded separately.
>
>
>>
>> Thomas
>> --
>> Distutils-SIG mailing list
>> distutils-sig@python.org
>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
>> Message archived at https://mail.python.org/mm3/ar
>> chives/list/distutils-sig@python.org/message/2XEH7R56Y63D72B
>> ZBAWRRJ33HPIKCWZH/
>>
>
--
Distutils-SIG mailing list
distutils-sig@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/C6WNHWNSMZE6ERPE2ZAPSKQJUPRBJDUJ/

[Distutils] Re: sudo pip install: install pip files into /usr/local on Linux?

[Wes Turner]

On Friday, May 25, 2018, Thomas Kluyver  wrote:

> On Fri, May 25, 2018, at 5:11 PM, Victor Stinner wrote:
> > As an user, I want to use "sudo pip install" because packages
> > installed in /usr (or /usr/local) are accessible without having to
> > touch PYTHONPATH: the install directory is part of the default
> > sys.path.
>
> This is also true for "pip install --user", at least on the systems I'm
> familiar with. There are options to disable 'user site packages', but it's
> enabled by default.
>
> It's more annoying for scripts - on common Linux distributions, the user
> scripts location ~/.local/bin is not on PATH by default.


~/.local/bin is user-writeable. If ~/.local was on PATH or by default, it
could potentially preempt/modify the behavior of system libraries and
binaries; which is a security risk.

~/.local/bin/bash could be wrapper script that logs commands, for example.
If it's first in the path (as e.g. Homebrew does, IIRC), it's run when bare
`bash` or `/usr/bin/env bash` are executed.

pipsi creates isolated virtualenvs per-install which are isolated from
other library installs, but each env then must be upgraded separately.


>
> Thomas
> --
> Distutils-SIG mailing list
> distutils-sig@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/mm3/
> archives/list/distutils-sig@python.org/message/
> 2XEH7R56Y63D72BZBAWRRJ33HPIKCWZH/
>
--
Distutils-SIG mailing list
distutils-sig@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/F5JW4JU4A7GMLXZ6S2A6UE5ZARVSTOD5/

[Distutils] Re: sudo pip install: install pip files into /usr/local on Linux?

[Wes Turner]

Maybe `sudo pip install` should:

- create a chroot && mkdir --prefix
- drop privileges*
- pip install
- chown -R root:root
- mv chroot/prefix/* prefix/

In most cases, the user does not need to run the (unreviewed, unsigned)
code as root; neither should they run the (unreviewed, unsigned) setup.py
as root.

All a wheel needs to be installed is an unzip.*

On Friday, May 25, 2018, Victor Stinner  wrote:

> As an user, I want to use "sudo pip install" because packages
> installed in /usr (or /usr/local) are accessible without having to
> touch PYTHONPATH: the install directory is part of the default
> sys.path.
>
> Steve Dower also proposed the idea of a "default virtual environment"
> somewhere in the $HOME directory which would be in the default
> sys.path.
>
> Victor
>
> 2018-05-23 21:47 GMT+02:00 Alex Walters :
> > I think the obvious, if socially hard solution, is to make pip panic
> when it
> > sees its being run as root (without, perhaps, a flag to tell pip "No, I
> > really mean it, run as root"), and default to --user.  It is not a good
> idea
> > to install packages system wide with pip for reasons more than just
> > clobbering apt/dnf installed packages.  I still think the best idea for
> > getting a python program to run system wide is either A: symlink from a
> > inside a venv  into something on $PATH, B: just set a shebang to the
> python
> > in a venv, or C: bundle your application into a .deb or .rpm and use the
> > system package manager to install it.
> >
> >> -Original Message-
> >> From: Victor Stinner 
> >> Sent: Wednesday, May 23, 2018 11:22 AM
> >> To: distutils-sig@python.org
> >> Subject: [Distutils] sudo pip install: install pip files into /usr/local
> > on Linux?
> >>
> >> Hi,
> >>
> >> pip is currently not well integrated on Linux: it conflicts  with the
> >> system package manager like apt or rpm. When pip writes files  into
> >> /usr, it can replace files written by the system package manager  and
> >> so create different kind of issues. For example, if you check the
> >> system integry, you will likely see that some Python files have been
> >> modified.
> >>
> >> I would like to open a discussion to see how each Linux vendor handles
> >> the issue, and see if a common solution can be designed.
> >>
> >> Debian uses /usr for apt-get install and /usr/local for distutils and
> >> "sudo pip".
> >>
> >> Fedora  decided to change pip to install files into /usr/local by
> >> default,  instead of /usr, so "sudo pip install" doesn't replace files
> >> installed  by dnf (Fedora package manager):
> >> https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
> >>
> >> It  gives you 3 main places to install Python code: /usr (managed by
> >> dnf),  /usr/local (managed by sudo pip), $HOME/.local (managed by pip
> >> --user).
> >>
> >> Would it make sense to make the Fedora/Debian change upstream? At
> >> least, give an opt-in option for Linux vendors to use /usr/local?
> >>
> >> I  propose to make the change upstream because there are still issues,
> >> and  I don't want to be alone to have to fix them :-) It should be
> >> easier if  we agree on a filesystem layout and an implementation, so
> >> we can  collaborate on issues!
> >>
> >>
> >> Issues with the current Fedora implementation:
> >>
> >> (1)  When Python is embedded in an application, there is an issue with
> >> the  current heuristic to decide if /usr/local should be added to
> >> sys.path:
> >>
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1532287
> >>
> >> (2)  On Fedora, "sudo pip install -U" currently removes old code from
> >> /usr  and install the new one in /usr/local. We should leave /usr
> >> unchanged,  since only dnf should touch /usr.
> >>
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24
> >>
> >> The implementation is made of a single patch on the Python site module:
> >>
> >> https://src.fedoraproject.org/rpms/python3/blob/master/f/00251-change-
> >> user-install-location.patch
> >>
> >> --
> >>
> >> There are two issues related to the "sudo pip" change, but they
> >> already exist when pip is installed in $HOME/.local:
> >>
> >> (3) Priority issue between PATH and PYTHONPATH directories.
> >>
> >> When  the user runs "pip", the pip binary may come from /usr,
> >> /usr/local or  $HOME/.local/bin, but the Python pip module ("import
> >> pip") may come from  a different path. Which binary and which module
> >> should be used?
> >>
> >> Obvisouly, users can replace these two environment variables...
> >>
> >> (4)  Related to (3). Running "pip" may run pip binary of one pip
> >> version,  but pick the "pip" Python module of another pip version.
> >>
> >> For example, pip9 binary from /usr/bin/pip, but pip10 module from
> >> /usr/local.
> >>
> >>
> >> Fedora works around issue (4) with a downstream patch on pip:
> >>
> >> https://src.fedoraproject.org/rpms/python-pip/blob/master/f/pip9-allow-
> >> pip10-import.patch
> >>
> >> --

[Distutils] Re: sudo pip install: install pip files into /usr/local on Linux?

[Wes Turner]

On Wednesday, May 23, 2018, Michael Sarahan <msara...@anaconda.com> wrote:

>
>
> On Wed, May 23, 2018 at 3:45 PM, Wes Turner <wes.tur...@gmail.com> wrote:
>
>>
>>
>> On Wednesday, May 23, 2018, Michael Sarahan <msara...@anaconda.com>
>> wrote:
>>
>>> Thanks for starting this discussion, Victor!  This topic is something
>>> we're very interested in at Anaconda.
>>>
>>> I'd like to generalize the problem statement to the question of "how can
>>> we make pip behave well when it is sharing package management with
>>> something else?  Similarly, how can we make the something else behave well
>>> with pip?"
>>>
>>> We all share the pain of trying to have two package managers effectively
>>> manage the same space.  The alternate-folder-for-pip approach is a good
>>> idea, but ultimately has issues that you pointed out.
>>>
>>> After several great conversations with many people at PyCon last week, I
>>> came to the conclusion that conda and pip probably won't ever interoperate
>>> very well.  In order for them to do so, conda must respect all of pip's
>>> constraints during its solving of dependencies, and likewise, pip must
>>> respect conda's constraints.  We are investigating the first of those
>>> options and having some promising initial results, but the inverse is not
>>> something that seems feasible.  It amounts to pip having a solver that is
>>> at least as good as the best supported package manager, and pip learning
>>> about *all* of any other package managers that it claims to be compatible
>>> with.  That doesn't seem like a viable project.
>>>
>> What's the status on this? AFAIU, depsolving setup.py packages is blocked
>> because it's necessary to execute setup.py to get the conditional
>> requirements?
>>
>
> Yes, executing setup.py is a blocker.  Wheels are improving this.  The
> main complicating factor is conditional or optional dependencies, as you
> say, and how to express or execute the branch logic required.  setup.py
> need not be executed all the time - only enough to gather the metadata to
> feed an index.  It's much more of a security concern than a time concern.
> A first rough approach that didn't handle optional or conditional
> requirements seems reasonable as a proof of concept.
>
>
>>
>> https://www.pypa.io/en/latest/roadmap/#pip-dependency-resolution
>>
>>>
>>> Ultimately, I believe a better approach is for the PyPA to define a
>>> minimal set of functionality and interfaces to PyPI that any package
>>> manager claiming to manage python packages must implement.  Pip can be a
>>> reference implementation of that specification.  Any distributor (Red Hat,
>>> Canonical, Homebrew, Anaconda) could then have their own implementations
>>> that use their solvers, but also can install software from PyPI at user's
>>> request, or as a fall-through when a native package is unavailable.
>>>
>>  Metadata compatibility and adapter registration could help solve for
>> this; though there's no money and not much demand. #PEP426JSONLD
>>
>
> With adapter registration, what would the adapters be?  I'm just not sure
> what pip's role should be.  If you define interfaces for solving and
> installing/uninstalling/upgrading the solved-for set of packages, maybe
> that's enough?
>

Plugins/adapters/interface implementations.
http://zopeinterface.readthedocs.io/en/latest/adapter.html

Perhaps ironically and conveniently, the well-regarded pluggy system for
plugins does not depend on setuptools entry_points:
https://pluggy.readthedocs.io/en/latest/

Some way to avoid adding plugin/adapter registration overhead in site.py
would be necessary.


>  #PEP426JSONLD looks great, but do you see that as a glue between PyPI and
> pip, pip and other tools, or other tools and PyPI?  I am impressed and
> encouraged by your research into the topic, and I'd like to know if there's
> a way that we can help with it if it would further our goals of having
> conda be able to either interop with pip happily or install directly from
> PyPI.
>

I piggybacked onto PEP426 because if we were going to substantially change
metadata, we might as well make it linked data; ideally with a
cross-language spec that would unfortunately take years to get compliance
from EVERY vendor for/with.

Metadata interoperability wouldn't be strictly necessary to achieve what I
think you're describing; but otherwise there'd be such a disjoint
dependency graph that determining what we've installed here and where we
need to be would be frustratingly complex.


>
>

[Distutils] Re: org-mode README file formats

[Wes Turner]

Pypa/readme is what Warehouse uses to render rst and markdown
long_descriptions to safe HTML.
https://github.com/pypa/readme_renderer

There are a number of org-mode implementations; even for a few for Vim and
Sublime.
https://en.m.wikipedia.org/wiki/Org-mode#Integration

``conda install pandoc`` is the quickest way to install pandoc, AFAIU. The
last time I installed pandoc with os packages it pulled in a bunch of
Haskell packages.

Python-only is the status quo for pypa/readme.


On Wednesday, May 9, 2018, Thomas Kluyver  wrote:

> On Wed, May 9, 2018, at 7:22 PM, Diane Trout wrote:
>
> Does the warehouse actually read the readme file directly? or does it
> preferentially look at long_description metadata?
>
>
> I think it only looks at the long_description metadata; as far as I know
> README is just another file in the archive.
>
> If you do want to convert it, though, please make sure it can fail
> gracefully if pandoc isn't there, because it would be frustrating to have
> problems installing a package from source just because it couldn't convert
> the README format.
>
--
Distutils-SIG mailing list
distutils-sig@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/NAG6QPAKUSVNGVF6SE4T7FZBFV4QSRCE/

[Distutils] Re: proposing Python package index upload API spec (potential PEP)

[Wes Turner]

I move to define PyPA and Warehouse HTTP APIs with OpenAPI definitions.

https://swagger.io/specification/
https://github.com/OAI/OpenAPI-Specification/blob/OpenAPI.next/versions/3.0.0.md

> OPENAPI SPECIFICATION
>
> The OpenAPI specification (formerly known as the Swagger Specification)
is a powerful definition format to describe RESTful APIs. The specification
creates a RESTful interface for easily developing and consuming an API by
effectively mapping all the resources and operations associated with it.
It’s easy-to-learn, language agnostic, and both human and machine readable.

> Introduction
>
> The OpenAPI Specification (OAS) defines a standard, language-agnostic
interface to RESTful APIs which allows both humans and computers to
discover and understand the capabilities of the service without access to
source code, documentation, or through network traffic inspection. When
properly defined, a consumer can understand and interact with the remote
service with a minimal amount of implementation logic.
>
> An OpenAPI definition can then be used by documentation generation tools
to display the API, code generation tools to generate servers and clients
in various programming languages, testing tools, and many other use cases.

- It's possible to generate WAF (Web Application Firewall) rules from a
Swagger/OpenAPI definition.
- There's an online editor.
- There are many integrations for various languages and frameworks.

An OpenAPI spec/schema can be written by hand as YAML or JSON,
autogenerated from e.g. decorators or docstrings or framework integration.

https://swagger.io/docs/specification/basic-structure/ has a YAML example.

Here's the online editor with validation:
https://swagger.io/swagger-editor/

https://swagger.io/open-source-integrations/ #python

- pyramid-swagger
   https://github.com/striglia/pyramid_swagger
> Convenient tools for using Swagger to define and validate your interfaces
in a Pyramid webapp.
> http://pyramid-swagger.readthedocs.org/

Request/response validation do have overhead and may be best done in a WAF,
but there's no reason not to serve e.g. /swagger.json or /swagger.yml with
the API spec for this app.

OTOH, DevPi and Pulp Project support the v1 upload API.

An initial OpenAPI schema definition document for just the v1 upload API
e.g. path, request, and response shouldn't take  much time at all.


On Wednesday, May 9, 2018, Nick Coghlan  wrote:

> On 9 May 2018 at 04:09, Sumana Harihareswara  wrote:
>
>> As a new Twine maintainer I've been running into questions like:
>>
>> * Now that Warehouse doesn't use "register" anymore, can we deprecate it
>> from distutils, setuptools, and twine? Are any other package indexes or
>> upload tools using it? https://github.com/pypa/twine/issues/311
>> * It would be nice if Twine could depend on a package index providing an
>> HTTP 201 response in response to a successful upload, and fail on 200 (a
>> response some non-package-index servers will give to an arbitrary POST
>> request).
>>
>> I do not see specifications to guide me here, e.g., in the official
>> guidance on hosting one's own package index
>> https://packaging.python.org/guides/hosting-your-own-index/ . PEP 301
>> was long enough ago that it's due an update, and PEP 503 only concerns
>> browsing and download, not upload.
>>
>> I suggest that I write a PEP specifying an API for uploading to a Python
>> package index. This PEP would partially supersede PEP 301 and would
>> document the Warehouse reference implementation. I would write it in
>> collaboration with the Warehouse maintainers who will develop the reference
>> implementation per pypa/warehouse/issues/284 and maybe add a header
>> referring to compliance with this new standard. And I would consult with
>> the maintainers of packaging and distribution tools such as zest.releaser,
>> flit, poetry, devpi, pypiserver, etc.
>>
>> Per Nick Coghlan's formulation, my specific goal here would be close to:
>>
>> > Documenting what the current upload API between twine & warehouse
>> actually is, similar to the way PEP 503 focused on describing the status
>> quo, without making any changes to it. That way, other servers (like devpi)
>> and other upload clients have the info they need to help ensure
>> interoperability.
>>
>> Since Warehouse is trying to redo its various APIs in the next several
>> months, I think it might be more useful to document and work with the new
>> upload API, but I'm open to feedback on this.
>>
>
> That's effectively what PEP 517 did for the legacy setup.py-centric sdist
> format, with just a single paragraph referencing the previous de facto
> standard and giving that version a name: https://www.python.org/dev/
> peps/pep-0517/#source-trees
>
> The equivalent here would be to call the existing upload interface the "v1
> upload API", and cite the relevant Warehouse endpoint URL as the reference
> implementation for it.
>
> While I initially wasn't a fan of that 

[Distutils] This list will soon be migrating to Mailman 3

[Wes Turner]

I think the mm footer should be preceded by a signature delimiter ('-- ').
Is there any reason not to?

If there's not an HTML footer option yet, it's probably because you'd need
to change the tag structure and/or CSS.

message footer


footer

Copying and pasting from HTML into Gmail generally causes the message to be
imperceptibly upgraded to text/html.
Is there a way to tell whether a draft message is text/plain or multipart
text/html or?
Is there a way to keep messages as plaintext (text/plain) only; as people
on these lists seem to prefer?

- Desktop: Ctrl-Shift-V pastes as text/plain (Linux, )
- Android: Copy/paste to a memo app, back (strips the formatting), open
memo, Copy (Omni Notes, )
- iOS:

In addition to indicating that the entire message has transmitted and been
replied to in full,
redundant and obnoxious email footers are often deleted when replying
inline;
the '-- ' helps indicate what can/should (?) be trimmed from the reply
chain in order to keep it under 40KB. Is 40KB still the mm message size
limit?

On Tuesday, May 1, 2018, Mark Sapiro
<m...@msapiro.net> wrote:

On 04/30/2018 11:37 AM, Wes Turner wrote:
> >
> > - Is there a way to hide these list footer links in an email signature
> > block?
> > - If there are two '^-- \n$', do email clients fold at the first?
>
>
> Note that the MM 2 list footer had somewhat different content, but the
> same structure as this one.
>
> In any case, if there is consensus that the 'line' at the beginning of
> the footer should be changed to a '-- ' separator, I can do that. Let me
> know.
>
> Here's more detail.
>
> This gets very tricky. If the original post is a simple text/plain
> message, Mailman will just append the footer to the message body. In
> that case, if the original had a signature with a '-- ' separator and
> the footer also had one, Thunderbird treats everything from the first
> separator on as a signature block. I didn't check any other MUAs.
>
> However, if the original post is more complicated such that after
> possible content filtering (not done on this list) it is other than a
> simple text/plain message, the footer is added as a separate MIME part.
>
> For example, the message to which I'm replying had the original structure
>
> multipart/alternative
> text/plain
> plain text alternative
> text/html
> rich text alternative
>
> and the post delivered from Mailman had the structure
>
> multipart/mixed
> multipart/alternative
> text/plain
> plain text alternative
> text/html
> rich text alternative
> text/plain
> the footer
>
> There is much variation in how different MUAs render that.
>
> --
> Mark Sapiro <m...@msapiro.net>The highway is for gamblers,
> San Francisco Bay Area, Californiabetter use your sense - B. Dylan
> ___
> Distutils-SIG mailing list
> distutils-sig@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/mm3/ar
> chives/list/distutils-sig@python.org/message/3P73LBXITY67G5K
> K775OONBZDKPEZAG5/
>
___
Distutils-SIG mailing list
distutils-sig@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/G2HMVQI47XPZE5PYHPJ344PNYS6ZJ5NX/

[Distutils] Re: This list will soon be migrating to Mailman 3

[Wes Turner]

 Looking good.

The signature footer from the message from you that ends with "I think it's
fixed now." appears to collapse in the Gmail interface,
but no longer does with the new, helpful footer?

- Is there a way to hide these list footer links in an email signature
block?
- If there are two '^-- \n$', do email clients fold at the first?

https://en.wikipedia.org/wiki/Signature_block#Signatures_in_Usenet_postings

> This latter prescription, which goes by many names, including "sig
dashes", "signature cut line", "sig-marker", "sig separator" and "signature
delimiter", allows software to automatically mark or remove the sig block
as the receiver desires.

-- 
One


-- 
Two


On Monday, April 30, 2018, Mark Sapiro  wrote:

> On 04/30/2018 10:02 AM, Joni Orponen wrote:
> >
> > This is something I've wanted in most mailing lists for over a decade
> > now. Thank you very much.
> >
> > Did this break the existing mail archive links now, though?
>
>
> No. The prior archive is still there and all prior URLs still work. The
> messages have also (with a few minor exceptions) been imported into the
> new HyperKitty archive and going forward, new posts will only be
> archived in the HyperKitty archive at
> 
>
> --
> Mark Sapiro The highway is for gamblers,
> San Francisco Bay Area, Californiabetter use your sense - B. Dylan
> ___
> Distutils-SIG mailing list
> distutils-sig@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/mm3/
> archives/list/distutils-sig@python.org/message/
> NY6M25J5SWM45V45K2CQJYM6PBXUAYCG/
>
___
Distutils-SIG mailing list
distutils-sig@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/YVG2VEN5TDMW5VJYJAQ547QKYUD6LEJ7/

[Distutils] Re: This list will soon be migrating to Mailman 3

[Wes Turner]

On Sunday, April 29, 2018, Mark Sapiro <m...@msapiro.net> wrote:

> On 04/29/2018 10:54 AM, Wes Turner wrote:
> >
> > --
> > ___
> > %(real_name)s maillist  -  To unsubscribe send an email to
> > distutils-sig-le...@python.org
> > %(web_page_url)slistinfo/%(_internal_name)s
> >
> >
> > Should these parameters be interpolated with the values of variables?
>
>
> Yes they should. That was my oversight. The list migration tool attempts
> to convert the old msg_footer and digest_footer attributes into the
> appropriate Mailman 3 equivalents but some things don't convert.
>
> I think it's fixed now.


Cool. Thanks again! Is there a way to add links to messages in the footer,
or is there an email client that makes it easy to get the is it a message
id header?


>
> --
> Mark Sapiro <m...@msapiro.net>The highway is for gamblers,
> San Francisco Bay Area, Californiabetter use your sense - B. Dylan
>
___
Distutils-SIG mailing list
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/

[Distutils] Re: This list will soon be migrating to Mailman 3

[Wes Turner]

On Sunday, April 29, 2018, Mark Sapiro  wrote:

> On 04/27/2018 09:19 PM, Nick Coghlan wrote:
> >
> > The list migration to Mailman 3 has now been requested, and is expected
> > to start around 1600 UTC, Sunday April 29th. Mark Sapiro will be
> > handling the migration for us (Thank you, Mark!).
>
>
> The migration is complete. There were a few anomalies in importing the
> archives. There were some older posts that didn't have Message-ID and/or
> Date: headers. This has caused a few messages to be missing and some
> older messages to be archived as if posted today.
>
> There were also a few with unescaped '^From ' lines in the body which
> causes the archived message to be truncated at that point.
>
> There may be other minor issues with the imported archives, but
> everything should be working at this point.


Thanks!

Something like these links may be helpful for the upgrade:

Old archive:
https://mail.python.org/pipermail/distutils-sig/
https://mail.python.org/pipermail/distutils-sig/2018-April/thread.html

New archive:
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/2018/4/


>
> --
> ___
> %(real_name)s maillist  -  To unsubscribe send an email to
> distutils-sig-le...@python.org
> %(web_page_url)slistinfo/%(_internal_name)s


Should these parameters be interpolated with the values of variables?

Mailman, HyperKitty for developers:

https://www.gnu.org/software/mailman/devs.html
https://gitlab.com/groups/mailman
https://gitlab.com/mailman/mailman
https://gitlab.com/mailman/hyperkitty
___
%(real_name)s maillist  -  To unsubscribe send an email to 
distutils-sig-le...@python.org
%(web_page_url)slistinfo/%(_internal_name)s

Re: [Distutils] Improving Communication

[Wes Turner]

On Saturday, April 21, 2018, James Bennett  wrote:

> Pulling in a sort-of success story from another large project, I like the
> general way things happen in Django.
>
> For developers proposing an idea or fixing a bug:
>
> * There's IRC (#django-dev) for quick, synchronous-ish discussion, useful
> for someone to find a sounding board for an idea
>

#pypa (Freenode manages these)


> * There's a dev mailing list where proposals can be discussed a bit more
> formally
>

https://mail.python.org/mailman/listinfo/distutils-sig (still not sure who
volunteered to manage these)

https://groups.google.com/forum/m/#!forum/pypa-dev (or this)


> * There's the public ticket tracker as a place to follow work being done
>

https://github.com/pypa/setuptools/issues
https://github.com/pypa/pip/issues
https://github.com/pypa/warehouse/issues
https://github.com/pypa/twine/issues
https://github.com/pypa/virtualenv/issues

https://github.com/pypa/packaging-problems/issues



>
> And for users seeking help or general discussion:
>
> * There's IRC (#django) and a users mailing list
>

GitHub.com/pypa//new
  label: Question

#pypa
distutils-sig


> * There's an official-ish subreddit moderated by committers
> * And there's the rest of the internet, including Stack Overflow, which we
> can't moderate but which many experienced people in the community do pay
> attention to
>

https://stackoverflow.com/questions/tagged/pip
https://stackoverflow.com/questions/tagged/pypi
https://stackoverflow.com/questions/tagged/virtualenv
https://stackoverflow.com/questions/tagged/python-packaging

Thanks to the people who host and handle these questions.


> We've avoided using GitHub issues for Django, preferring the workflow
> tools we get from our own customized Trac instance.
>
> I wonder whether something similar -- a real-time chat for
> discussion/sounding board/etc., mailing list to bring things to once
> they've been thought about a bit, public tracker for following
> work/archival purposes -- would work for packaging.
>

Should there, in addition to #pypa IRC and gitter, be a zulip topic (?) for
pypa?

AFAIU, Zulip started at Dropbox, is written in Python, supports boots, and
is functionally similar to Slack, Gitter, and MatterMost?


>
> (I am also wary of too much "process"; Django has a fair bit, and more
> than I'd ideally like, but my experience has been that process and
> participation are generally inversely correlated)
>

Better yet, fork and send a pull request (PR)?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Improving Communication

[Wes Turner]

On Saturday, April 21, 2018, Wayne Werner <waynejwer...@gmail.com> wrote:

>
>
> On Fri, Apr 20, 2018, 9:54 PM Wes Turner <wes.tur...@gmail.com> wrote:
>
>>
>>
>> On Friday, April 20, 2018, Donald Stufft <don...@stufft.io> wrote:
>>
>>>
>>>
>>> * A lot of the technology in this stack is particularly old, and lacks a
>>> lot of the modern day affordances that newer things have. An example is
>>> being able to edit a discussion post to fix typos that can hinder the
>>> ability of others to actually understand whats being talked about. In your
>>> typical mailing list or IRC there’s no mechanism by which you can edit an
>>> already sent message, so your only option is to either let the problem ride
>>> and hope it doesn’t trip up too many people, or send an additional message
>>> to correct the error. However these show up as additional, later messages
>>> which someone might not even see until they’ve already been thoroughly
>>> confused by the first message (since people tend to read email/IRC in a
>>> linear fashion).
>>
>>
>> Issues, Pull Requests, Wikis, and Team Discussions are all editable.
>>
>
> Though one thing I've noticed is that a lot of issues get created and
> responded to via email clients that hide the existing message so someone
> will say "I agree" or worse, +1, and then there's 400 line of email. I
> don't know if GH has fixed the display on those things yet.
>
> FWIW.
>

There's now mobile reaction support; so you can click 'view on github' in
the email footer and thumbs up/down a message.

GitHub just added Hide and Edit comment options for maintainers:

https://blog.github.com/2018-04-18-new-tools-for-open-source-maintainers/#minimized-comments

Trimming replies is a lot of work on a mobile device:
- With a keyboard, shift- selects complete lines
- You can also , + to select text

Generally, I try and follow up and clean up formatting and remove any email
signature footer after email-replying to add a comment to a GitHub issue.

There are mobile GitHub apps; though sometimes not replying until I have a
real keyboard is the better option... Far preferable to it.


>
> -W
>
>>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Improving Communication

[Wes Turner]

On Friday, April 20, 2018, Donald Stufft  wrote:

> Currently in the packaging space, we have a number of avenues for
> communication, which are:
>
> - distutils-sig
> - pypa-dev
> - virtualenv-users
> - Other project specific mailing lists
> - IRC
> - gitter
> - Various issue trackers spread across multiple platforms.
> - Probably more places I’m not remembering.


Is there something like a collaboration plan with a decision tree / flow
chart / list of what types of communications are best communicated on which
platform?

GitHub Team Discussions [1]
- threaded
- editable
- trackbackable (I think?)
- @user notifications
- email replies
- cross-repo
- public / team-only
- Unfortunately, they're not real time like IRC/Gitter

[1] https://blog.github.com/2017-11-20-introducing-team-discussions/


>
> The result of this is that all discussion ends up being super fractured
> amongst the various places. Sometimes that is exactly what you want (for
> instance, someone who is working on the wheel specs probably doesn’t care
> about deprecation policy and internal module renaming in pip) and sometimes
> that ends up being the opposite of what you want (for instance, when you’re
> describing something that touches PyPI, setuptools, flit, pip, etc all at
> once).
>
> Theoretically the idea is that distutils-sig is where cross project
> reaching stuff goes, IRC/gitter is where real time discussion goes, and the
> various project mailing lists and issue trackers are where the project
> specific bits go. The problem is that often times doesn’t actually happen
> in practice except for the largest and most obvious of changes.


"Please refer to the communications flowchart: [url]"


>
> I think our current “communications stack” kind of sucks, and I’d love to
> figure out a better way for us to handle this that solves the sort of weird
> “independent but related” set of projects we have here.


Do GitHub project boards help?

Team email and web notifications on GitHub work like this:
@pypa/core



>
> From my POV, a list of our major problems are:
>
> * Discussion gets fractured across a variety of platforms and locations,
> which can make it difficult to actually keep up with what’s going on but
> also to know how to loop in someone relevant if their input would be
> valuable. You have to more or less simultaneously know someone’s email,
> Github username, IRC nick, bitbucket username, etc to be able to bring
> threads of discussion to people’s attention.


A collaboration plan might include this contact information for each team
member


>
> * It’s not always clear to users where a discussion should go, often times
> they’ll come to one location and need to get redirected to another
> location. If any discussion did happen in the incorrect location, it tends
> to need to get restarted in the new location (and depending on the specific
> platform, it may be impossible to actually redirect everyone over to the
> proper location, so you again, end up fractured with the discussion
> happening in two places).


IDK how many times we've discussed upgrading mailman to add per message
footer links to relayed messages. Google Groups has this feature on by
default.


>
> * A lot of the technology in this stack is particularly old, and lacks a
> lot of the modern day affordances that newer things have. An example is
> being able to edit a discussion post to fix typos that can hinder the
> ability of others to actually understand whats being talked about. In your
> typical mailing list or IRC there’s no mechanism by which you can edit an
> already sent message, so your only option is to either let the problem ride
> and hope it doesn’t trip up too many people, or send an additional message
> to correct the error. However these show up as additional, later messages
> which someone might not even see until they’ve already been thoroughly
> confused by the first message (since people tend to read email/IRC in a
> linear fashion).


Issues, Pull Requests, Wikis, and Team Discussions are all editable.

[EDIT]

I EDITED THIS.


>   - There is a lot of things in this one, other things are things like
> being able to control in a more fine grained manner what email you’re going
> to get.


One can unsubscribe from issue notifications


>   - Holy crap, formatting and typography to make things actually readable
> and not a big block of plaintext.


This is a fenced code block in Markdown and GFM: (GitHub Flavored Markdown)

```python
__import__('this')
```

These create a trackback on the mentioned issue:

  #1
  pypa/pip#1


This sends notifications to each mentioned user according to their GitHub
notification settings:

  /cc @westurner @dstufft @pypa/core


> * We don’t have a clear way for users to get help, leaving users to treat
> random issues, discussion areas, etc as a support forum, rather than some
> place that’s actually optimized for that. Some of this ties back into some
> of the earlier things too, where 

Re: [Distutils] File handling- tab separated files

[Wes Turner]

This is the python distutils list for discussions regarding python
packaging.

For help with using the Python standard library, try the docs and/or the
python tutor mailing list.

https://www.google.com/search?q=site:docs.python.org+tsv

https://www.python.org/community/lists/#tutor
https://markmail.org/search/?q=list:org.python.tutor+tsv

https://www.reddit.com/r/learnpython/
https://www.reddit.com/r/learnpython/search?q=tsv

What you describe should be possible by setting delimiter='\t' with
csv.reader with the CSV module in the Python standard library:

https://docs.python.org/3/library/csv.html#csv.reader
https://docs.python.org/3/library/csv.html#csv.Dialect.delimiter

Various third party packages are faster at reading delimited files or have
additional useful functionality:

http://docs.python-tablib.org/en/master/

https://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html
https://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html#numpy.genfromtxt

https://pandas.pydata.org/pandas-docs/stable/io.html#csv-text-files
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html

On Wednesday, April 18, 2018, Niharika Jakhar 
wrote:

> Hello
>
> I am having trouble opening and reading a Tab separated file(big data). I
> stored the data the data in a list and then used the split function (\t) to
> tried to print a list of 0 to 100 position. But it is resulting in random
> values from the files.
>
> Please let me know, if you deal with such queries, if not, can you please
> suggest me some good source in order to solve this problem.
>
> Thanking you in advance
>
> Best regards
> NIHARIKA
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pip 10.0 has been released

[Wes Turner]

Thanks!

On Saturday, April 14, 2018, Thomas Kluyver  wrote:

> Thanks and congratulations to everyone who has worked on pip!
>
> On Sat, Apr 14, 2018, at 1:47 PM, Paul Moore wrote:
> > On behalf of the PyPA, I am pleased to announce that pip 10.0 has just
> > been released. This release has been the culmination of many months of
> > work by the community.
> >
> > To install pip 10.0, you can run
> >
> > python -m pip install --upgrade pip
> >
> > or use get-pip, as described in
> > https://pip.pypa.io/en/latest/installing. If you are using a version
> > of pip supplied by your distribution vendor, vendor-supplied upgrades
> > will be available in due course (or you can use pip 10 in a virtual
> > environment).
> >
> > (One minor issue with using get-pip on Windows - when you download
> > get-pip.py, rename it to something that doesn't include "pip" in the
> > name, such as "gp.py", as the standard name triggers a check in pip
> > that aborts the run - this is being tracked in
> > https://github.com/pypa/pip/issues/5219).
> >
> > Highlights of the new release:
> >
> > * Python 2.6 is no longer supported - if you need pip on Python 2.6,
> > you should stay on pip 9, which is the last version to support Python
> > 2.6.
> > * Support for PEP 518, which allows projects to specify what packages
> > they require in order to build from source. (PEP 518 support is
> > currently limited, with full support coming in future versions - see
> > the documentation for details).
> > * Significant improvements in Unicode handling for non-ASCII locales on
> Windows.
> > * A new "pip config" command.
> > * The default upgrade strategy has become "only-if-needed"
> > * Many bug fixes and minor improvements.
> >
> > In addition, the previously announced reorganisation of pip's
> > internals has now taken place. Unless you are the author of code that
> > imports the pip module (or a user of such code), this change will not
> > affect you. If you are affected, please report the issue to the author
> of the
> > offending code (refer them to
> > https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html
> > for the details of the announcement).
> >
> > Thanks to everyone who put so much effort into the new release. Many
> > of the contributions came from community members, whether in the form
> > of code, participation in design discussions, or bug reports. The pip
> > development team is extremely grateful to everyone in the community
> > for their contributions.
> >
> > Thanks,
> > Paul
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Summary of PyPI overhaul in new LWN article

[Wes Turner]

>From "TUF, Warehouse, Pip, PyPA, ld-signatures, ed25519"

https://mail.python.org/pipermail/distutils-sig/2018-March/032081.html :

> Are there pypa/warehouse github issues for implementing the TUF trust
root support in warehouse; and client support in pip (or a module that pip
and other tools can use)?

Read and review these PEPs:

"PEP 458 -- Surviving a Compromise of PyPI"
https://www.python.org/dev/peps/pep-0458/;

"PEP 480 -- Surviving a Compromise of PyPI: The Maximum Security Model"
https://www.python.org/dev/peps/pep-0480/

On Thursday, April 12, 2018, Trishank Kuppusamy <
trishank.kuppus...@datadoghq.com> wrote:

> On Wed, Apr 11, 2018 at 10:30 PM, Sumana Harihareswara 
> wrote:
>
>> Today, LWN published my new article "A new package index for Python".
>> https://lwn.net/Articles/751458/ In it, I discuss security, policy, UX
>> and developer experience changes in the 15+ years since PyPI's founding,
>> new features (and deprecated old features) in Warehouse, and future
>> plans. Plus: screenshots!
>>
>> If you aren't already an LWN subscriber, you can use this subscriber
>> link for the next week to read the article despite the LWN paywall.
>> https://lwn.net/SubscriberLink/751458/81b2759e7025d6b9/
>
>
> Thanks for the summary, and all your hard work, Sumana :)
>
> Happy to see this bit about TUF in future horizons:
>
> Warehouse's signature handling demonstrates a shift in Python's thinking
>> regarding key management and package signatures. Ideally, package users,
>> software distributors, and package distribution tools would regularly use
>> signatures to verify Python package integrity. For the most part, however,
>> they don't, and there are major infrastructural barriers to them
>> effectively doing so. Therefore, GPG/PGP signatures for packages are no
>> longer visible in PyPI's web interface. Project maintainers can still
>> attach signatures to their release uploads, and those signatures still
>> appear in the Simple Project API as described in PEP 503. Stufft has made
>> no secret of his opinion that "package signing is not the Holy Grail";
>> current discussion among packaging-tools developers leans toward removing
>> signing features from another part of the Python packaging ecology (the
>> wheel library) and working toward implementing The Update Framework
>> instead. Relatedly, Warehouse, unlike legacy PyPI, does not provide an
>> interface for users to manage GPG or SSH public keys.
>
>
>  We would love to help with this efforts any way we can.
>
> --
> curl https://keybase.io/trishankdatadog/pgp_keys.asc | gpg --import
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] providing a way for pip to communicate extra info to users

[Wes Turner]

A MOTD from anything but a signed package would be user-supplied input.
Shell/terminal command ^[escaping would be necessary:
https://stackoverflow.com/questions/6534556/how-to-remove-and-all-of-the-escape-sequences-in-a-file-using-linux-shell-sc

Impact:

Are additional requests and variable messages really necessary? Can static
error messages simply say 'check /news for more information'? (thus saving:
millions of requests per year and additional MOTD package signing overhead
and bandwidth)?



On Thursday, April 12, 2018, Justin Cappos  wrote:

> FYI: TUF has a custom metadata field in the targets metadata that could
> potentially be used for this purpose.  We can explain more if there is
> interest...
>
> On Thu, Apr 12, 2018 at 8:26 AM, Nathaniel Smith  wrote:
>
>> From the TUF perspective it seems like it would be straightforward to
>> make the MOTD a "package", whose "contents" is the MOTD text, and that we
>> "upgrade" it to get the latest text before displaying anything.
>>
>> -n
>>
>> On Thu, Apr 12, 2018, 05:10 Nick Coghlan  wrote:
>>
>>> On 12 April 2018 at 07:01, Paul Moore  wrote:
>>> > HTTPS access to the index server is fundamental to pip - if an
>>> > attacker can subvert that, they don't need to mess with a message,
>>> > they can just replace packages. So I don't see that displaying a
>>> > message that's available from that same index server is an additional
>>> > vulnerability, surely? But I'm not a security expert - I'd defer to
>>> > someone like Donald to comment on the security aspects of any proposal
>>> > here.
>>>
>>> Right now it doesn't create any additional vulnerabilities, since
>>> we're relying primarily on HTTPS for PyPI -> installer security.
>>>
>>> However, that changes once PEP 458 gets implemented, as that will
>>> switch the primary package level security mechanism over to TUF, which
>>> includes a range of mechanisms designed to detect tampering with the
>>> link to PyPI (including freeze attacks that keep you from checking for
>>> new packages, or attempting to lie about which versions are
>>> available).
>>>
>>> So the scenario we want to avoid is one where an attacker can present
>>> a notice that says "Please ignore that scary security warning your
>>> installer is giving you, we're having an issue with the metadata
>>> generation process on the server. To resolve the problem, please force
>>> upgrade pip".
>>>
>>> That's a solvable problem (e.g. only check for the MOTD *after*
>>> successfully retrieving a valid metadata file), but it's still
>>> something to take into account.
>>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distributing packages to offline machines

[Wes Turner]

On Friday, April 6, 2018, Ben Finney  wrote:

> Nick Coghlan  writes:
>
> > Keep a requirements.txt file or `Pipfile` in source control, then run
> > CI jobs based on that repo […]
>
> What is a “Pipfile”?


https://docs.pipenv.org/

https://docs.pipenv.org/basics/#example-pipfile-pipfile-lock


>
> --
>  \  “Software patents provide one more means of controlling access |
>   `\  to information. They are the tool of choice for the internet |
> _o__) highwayman.” —Anthony Taylor |
> Ben Finney
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distributing packages to offline machines

[Wes Turner]

Have you already tried `pip download --platform`?

https://pip.pypa.io/en/stable/reference/pip_download/#cmdoption-platform


It may be worth setting up devpi (maybe in a container) and caching the
packages; particularly for CI:

https://packaging.python.org/guides/index-mirrors-and-caches/

AFAIU, there's no way to specify a limited set of packages and platforms to
mirror with bandersnatch?

On Wednesday, April 4, 2018, Eric Gorr  wrote:

> I had a question about distributing python packages to offline machines
> when the offline machine is running a different OS then a machine with an
> internet connection. The packages I am concerned with are third party upon
> which mine depend.
>
> Based on what I have learned so far, there are three solutions.
>
> (a) Use a CI to run a fleet of machines for each OS one needs to target to
> obtain the OS specific wheels.
>
> (b) 'pip download  --no-binary :all:' -- the intention here
> is to grab the source distribution without any OS specific code included.
>
> (c) use https://warehouse.pypa.io/api-reference/json to look for
> distributed wheels for the target OS and python version and download them
> directly. (This may make for a nice flag to add to pip somewhere...the
> ability to specify what wheel one wants when it isn’t for the machine pip
> is running on)
>
> The issue I see with (a) is the shear amount work it would take to setup
> and maintain such a system.
>
> The issue I see with (b) is that it is not 100% reliable as some packages
> are tricky to install and may not work well with 'pip download’.
>
> I have not played around with (c) yet, so I do not know how well it will
> work, but it seems like a viable solution.
>
> I was just wondering if anyone had any comments on this as I think though
> the ways to solve this problem.
>
> Thank you.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Removing wheel signing features from the wheel library

[Wes Turner]

On Thursday, March 22, 2018, Justin Cappos <jcap...@nyu.edu> wrote:

> You don't need a traditional CA for use with TUF or need to worry about a
> single PKI.  TUF is built to be resilient to the compromise of single
> servers / keys.
>
> Typically you would ship the metadata about what keys to trust (TUF's
> "root metadata") with the software installation tool.  This single set of
> pre-shared metadata means you can securely obtain the rest of the
> software.  (I'm happy to go into more detail, but wanted to avoid this
> becoming a barrage of TUF details unless everyone is interested.)
>
> If you don't want to ship the metadata with the tool, you can also have it
> work in a trust-on-first-use model.  This is what Docker does in their
> deployment of TUF.
>

[Distutils] TUF, Warehouse, Pip, PyPA, ld-signatures, ed25519
https://mail.python.org/pipermail/distutils-sig/2018-March/032081.html

I split the thread. Thanks for the explanation.


>
> On Thu, Mar 22, 2018 at 4:40 PM, Wes Turner <wes.tur...@gmail.com> wrote:
>
>>
>>
>> On Thursday, March 22, 2018, Daniel Holth <dho...@gmail.com> wrote:
>>
>>> The feature was a building block that was intended to be used in much
>>> the same way that SHA package hashes are used, providing similar security
>>> to the ssh-style TOFU model, but less security than other imaginable public
>>> key systems. The idea was that it could provide more security in practice
>>> because the signatures could exist and be present with the archive, unlike
>>> gpg which provides loads of security in theory only. Unfortunately wheel
>>> signatures were never built up. I don't think anyone was tricked into
>>> believing the primitive provided security on its own.
>>>
>>
>> The hashes serve as file integrity check but provide no assurance that
>> they are what the author intended to distribute because there is no
>> cryptographic signature.
>>
>> File hashes help detect bit flips -- due to solar flares -- in storage or
>> transit, but do not mitigate against malicious package modification to
>> packages in storage or transit.
>>
>> AFAIU, TUF (The Update Framework) has a mechanism for limiting which
>> signing keys are valid for which package? Are pre-shared keys then still
>> necessary, or do we then rely on a PKI where one compromised CA cert can
>> then forge any other cert?
>>
>> https://theupdateframework.github.io/
>>
>>
>>>
>>> On Thu, Mar 22, 2018 at 2:21 PM Nathaniel Smith <n...@pobox.com>ared
>>> wrote:
>>>
>>>> Even if no maintenance were required, it's still a feature that
>>>> promises to provide security but doesn't. This kind of feature has negative
>>>> value.
>>>>
>>>> I'd also suggest adding a small note to the PEP documenting that the
>>>> signing feature didn't work out, and maybe linking to Donald's package
>>>> signing blog post. I know updating PEPs isn't the most common thing, but
>>>> it's the main documentation of the wheel format and it'll save confusion
>>>> later.
>>>>
>>>> On Mar 22, 2018 10:57 AM, "Wes Turner" <wes.tur...@gmail.com> wrote:
>>>>
>>>>> What maintenance is required?
>>>>>
>>>>> Here's a link to the previous discussion of this issue:
>>>>>
>>>>> "Remove or deprecate wheel-signing features"
>>>>> https://github.com/pypa/wheel/issues/196
>>>>>
>>>>> What has changed? There is still no method for specifying a keyring;
>>>>> whereas with GPG, all keys in the ring are trusted.
>>>>>
>>>>> On Thursday, March 22, 2018, Nick Coghlan <ncogh...@gmail.com> wrote:
>>>>>
>>>>>> On 22 March 2018 at 22:35, <alex.gronh...@nextday.fi> wrote:
>>>>>>
>>>>>>> I am not changing the format of RECORD, I'm simply removing the
>>>>>>> cryptographic signing and verifying functionality, just the way you
>>>>>>> described. Hash checking will stay. As we agreed earlier, those
>>>>>>> features could be deprecated or removed from the PEP entirely.
>>>>>>>
>>>>>>
>>>>>> Cool, that's what I thought you meant, but I figured I should double
>>>>>> check since our discussion was a while ago now :)
>>>>>>
>>>>>> Cheers,
>>>>>> Nick.
>>>>>>
>>>>>> --
>>>>>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>>>>>
>>>>>
>>>>> ___
>>>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>>>
>>>>> ___
>>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>>
>>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Removing wheel signing features from the wheel library

[Wes Turner]

On Thursday, March 22, 2018, <alex.gronh...@nextday.fi> wrote:

> to, 2018-03-22 kello 16:40 -0400, Wes Turner kirjoitti:
>
>
>
> On Thursday, March 22, 2018, Daniel Holth <dho...@gmail.com> wrote:
>
> The feature was a building block that was intended to be used in much the
> same way that SHA package hashes are used, providing similar security to
> the ssh-style TOFU model, but less security than other imaginable public
> key systems. The idea was that it could provide more security in practice
> because the signatures could exist and be present with the archive, unlike
> gpg which provides loads of security in theory only. Unfortunately wheel
> signatures were never built up. I don't think anyone was tricked into
> believing the primitive provided security on its own.
>
>
> The hashes serve as file integrity check but provide no assurance that
> they are what the author intended to distribute because there is no
> cryptographic signature.
>
> File hashes help detect bit flips -- due to solar flares -- in storage or
> transit, but do not mitigate against malicious package modification to
> packages in storage or transit.
>
>
> I've been wondering about something – zip files already contain CRC based
> checksums for each the stored file. What benefit is there in storing a
> RECORD file which basically duplicates this functionality?
>

AFAIU, there's no good way to do post-install hash verification like
`debsums` and `rpm -V` with zip CRCs (though it's probably possible if
there's a structured log of installed packages).


>
>
> AFAIU, TUF (The Update Framework) has a mechanism for limiting which
> signing keys are valid for which package? Are pre-shared keys then still
> necessary, or do we then rely on a PKI where one compromised CA cert can
> then forge any other cert?
>
> https://theupdateframework.github.io/
>
>
>
> On Thu, Mar 22, 2018 at 2:21 PM Nathaniel Smith <n...@pobox.com>ared wrote:
>
> Even if no maintenance were required, it's still a feature that promises
> to provide security but doesn't. This kind of feature has negative value.
>
> I'd also suggest adding a small note to the PEP documenting that the
> signing feature didn't work out, and maybe linking to Donald's package
> signing blog post. I know updating PEPs isn't the most common thing, but
> it's the main documentation of the wheel format and it'll save confusion
> later.
>
> On Mar 22, 2018 10:57 AM, "Wes Turner" <wes.tur...@gmail.com> wrote:
>
> What maintenance is required?
>
> Here's a link to the previous discussion of this issue:
>
> "Remove or deprecate wheel-signing features"
> https://github.com/pypa/wheel/issues/196
>
> What has changed? There is still no method for specifying a keyring;
> whereas with GPG, all keys in the ring are trusted.
>
> On Thursday, March 22, 2018, Nick Coghlan <ncogh...@gmail.com> wrote:
>
> On 22 March 2018 at 22:35, <alex.gronh...@nextday.fi> wrote:
>
> I am not changing the format of RECORD, I'm simply removing the
> cryptographic signing and verifying functionality, just the way you
> described. Hash checking will stay. As we agreed earlier, those
> features could be deprecated or removed from the PEP entirely.
>
>
> Cool, that's what I thought you meant, but I figured I should double check
> since our discussion was a while ago now :)
>
> Cheers,
> Nick.
>
> ___
> Distutils-SIG maillist  -  
> Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] TUF, Warehouse, Pip, PyPA, ld-signatures, ed25519

[Wes Turner]

TUF, Warehouse, Pip, PyPA, ld-signatures, ed

"PEP 480 -- Surviving a Compromise of PyPI"
https://www.python.org/dev/peps/pep-0458/

"PEP 480 -- Surviving a Compromise of PyPI: The Maximum Security Model"
https://www.python.org/dev/peps/pep-0480/



I need to spend time reviewing these PEPs. Backseat driving here; sorry:

Are there pypa/warehouse github issues for implementing the TUF trust root
support in warehouse; and client support in pip (or a module that pip and
other tools can use)?

Warehouse is already a SPOF.
That's a hefty responsibility that contributions should support.

Would [offline] package mirrors and the CDN still work for/with TUF keys?


ld-signatures has some normative language that could be useful.

ld-signatures uses URIs for signature suites (a canonicalization algorithm,
a message digest algorithm, and a signature algorithm) and JSONLD. That
should be pretty future proof in regards to the NIST post-quantum
algorithms call that's under review at this time.

Blockcerts builds upon ld-signatures.

There's a compact form of JSONLD.
JSON[LD] can also be serialized as BSON (and RDFHDT).

"Linked Data Signatures 1.0" (draft)
https://w3c-dvcg.github.io/ld-signatures/

"Ed25519 Signature 2018" (draft)
https://w3c-dvcg.github.io/lds-ed25519-2018/
- canonicalizationAlgorithm: https://w3id.org/security#URDNA2015
- digestAlgorithm: http://w3id.org/digests#sha512
- signatureAlgorithm: http://w3id.org/security#ed25519


https://theupdateframework.github.io/

https://github.com/theupdateframework/specification/blob/master/tuf-spec.md#the-update-framework-specification


On Thursday, March 22, 2018, Trishank Kuppusamy <
trishank.kuppus...@datadoghq.com> wrote:

> Hi Wes,
>
> On Thu, Mar 22, 2018 at 4:40 PM, Wes Turner <wes.tur...@gmail.com> wrote:
>
>>
>> The hashes serve as file integrity check but provide no assurance that
>> they are what the author intended to distribute because there is no
>> cryptographic signature.
>>
>> File hashes help detect bit flips -- due to solar flares -- in storage or
>> transit, but do not mitigate against malicious package modification to
>> packages in storage or transit.
>>
>> AFAIU, TUF (The Update Framework) has a mechanism for limiting which
>> signing keys are valid for which package? Are pre-shared keys then still
>> necessary, or do we then rely on a PKI where one compromised CA cert can
>> then forge any other cert?
>>
>
> Yes, you are right, the hashes need to be signed: otherwise you have
> integrity, but no authenticity.
>
> We wrote PEPs 458 <https://www.python.org/dev/peps/pep-0458/> and 480
> <https://www.python.org/dev/peps/pep-0480/> to discuss how TUF might be
> deployed on PyPI / Warehouse. The PEPs go into details about public key
> distribution. The TLDR is that is that clients (i.e., pip) need to be
> shipped with one self-signed root metadata file, and the rest of the PKI is
> bootstrapped from there. PyPI would act as an authority that distributes,
> revokes, and replaces public keys for packages.
>
> More details on security vs usability also available in our Diplomat
> <https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/kuppusamy>
> paper.
>
> If the community is interested, we'd love to discuss how we could help
> make this happen.
>
> Thanks,
> Trishank
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Removing wheel signing features from the wheel library

[Wes Turner]

On Thursday, March 22, 2018, Daniel Holth <dho...@gmail.com> wrote:

> The feature was a building block that was intended to be used in much the
> same way that SHA package hashes are used, providing similar security to
> the ssh-style TOFU model, but less security than other imaginable public
> key systems. The idea was that it could provide more security in practice
> because the signatures could exist and be present with the archive, unlike
> gpg which provides loads of security in theory only. Unfortunately wheel
> signatures were never built up. I don't think anyone was tricked into
> believing the primitive provided security on its own.
>

The hashes serve as file integrity check but provide no assurance that they
are what the author intended to distribute because there is no
cryptographic signature.

File hashes help detect bit flips -- due to solar flares -- in storage or
transit, but do not mitigate against malicious package modification to
packages in storage or transit.

AFAIU, TUF (The Update Framework) has a mechanism for limiting which
signing keys are valid for which package? Are pre-shared keys then still
necessary, or do we then rely on a PKI where one compromised CA cert can
then forge any other cert?

https://theupdateframework.github.io/


>
> On Thu, Mar 22, 2018 at 2:21 PM Nathaniel Smith <n...@pobox.com>ared wrote:
>
>> Even if no maintenance were required, it's still a feature that promises
>> to provide security but doesn't. This kind of feature has negative value.
>>
>> I'd also suggest adding a small note to the PEP documenting that the
>> signing feature didn't work out, and maybe linking to Donald's package
>> signing blog post. I know updating PEPs isn't the most common thing, but
>> it's the main documentation of the wheel format and it'll save confusion
>> later.
>>
>> On Mar 22, 2018 10:57 AM, "Wes Turner" <wes.tur...@gmail.com> wrote:
>>
>>> What maintenance is required?
>>>
>>> Here's a link to the previous discussion of this issue:
>>>
>>> "Remove or deprecate wheel-signing features"
>>> https://github.com/pypa/wheel/issues/196
>>>
>>> What has changed? There is still no method for specifying a keyring;
>>> whereas with GPG, all keys in the ring are trusted.
>>>
>>> On Thursday, March 22, 2018, Nick Coghlan <ncogh...@gmail.com> wrote:
>>>
>>>> On 22 March 2018 at 22:35, <alex.gronh...@nextday.fi> wrote:
>>>>
>>>>> I am not changing the format of RECORD, I'm simply removing the
>>>>> cryptographic signing and verifying functionality, just the way you
>>>>> described. Hash checking will stay. As we agreed earlier, those
>>>>> features could be deprecated or removed from the PEP entirely.
>>>>>
>>>>
>>>> Cool, that's what I thought you meant, but I figured I should double
>>>> check since our discussion was a while ago now :)
>>>>
>>>> Cheers,
>>>> Nick.
>>>>
>>>> --
>>>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>>>
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Removing wheel signing features from the wheel library

[Wes Turner]

What maintenance is required?

Here's a link to the previous discussion of this issue:

"Remove or deprecate wheel-signing features"
https://github.com/pypa/wheel/issues/196

What has changed? There is still no method for specifying a keyring;
whereas with GPG, all keys in the ring are trusted.

On Thursday, March 22, 2018, Nick Coghlan  wrote:

> On 22 March 2018 at 22:35,  wrote:
>
>> I am not changing the format of RECORD, I'm simply removing the
>> cryptographic signing and verifying functionality, just the way you
>> described. Hash checking will stay. As we agreed earlier, those
>> features could be deprecated or removed from the PEP entirely.
>>
>
> Cool, that's what I thought you meant, but I figured I should double check
> since our discussion was a while ago now :)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] It should be possible to set a custom pypirc location

[Wes Turner]

"Custom location for .pypirc file"
https://stackoverflow.com/questions/37845125/custom-location-for-pypirc-file

- https://github.com/pypa/setuptools/blob/master/
setuptools/package_index.py#L997


https://github.com/pypa/setuptools/issues/new

These are the pip.conf file paths:
https://pip.pypa.io/en/stable/user_guide/#config-file

... pip combines all of the pip.conf/pip.ini files that it finds.



On Friday, March 9, 2018, Joel Croteau  wrote:

> Hardcoding ~/.pypirc as the only possible location for repo configs makes
> certain use cases, like deploying to a custom repo from Jenkins, difficult.
> It should be possible to set a custom pypirc location, either through an
> environment variable, or an argument to setup.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] print module

[Wes Turner]

https://www.google.com/search?q=Which+module+do+I+import+to+use+print%3F
(the StackOverflow result answers your question)

https://www.google.com/search?q=python+print+function (first result)

https://docs.python.org/3/search.html?q=Print (second result -> print
function)

https://docs.python.org/2/search.html?q=Print (second result -> print
statement)

https://docs.python.org/3/library/__future__.html?highlight=print (what is
a PEP?)


This list is for distutils-related discussions.
https://mail.python.org/mailman/listinfo/distutils-sig

This list is for Python language and standard library questions:

https://mail.python.org/mailman/listinfo/tutor

This forum is also for similar questions:

https://www.reddit.com/r/learnpython

https://www.reddit.com/r/learnpython/search?q=print+function


On Monday, January 29, 2018, Kevin Urban  wrote:

>
> Just starting out unable to use print function on a mac. Which module do I
> import to use print?
>

>
> Kevin Urban
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating/Removing OpenID/Google login support for PyPI

[Wes Turner]

python-social-auth supports OAuth 1, OAuth 2, OpenID, SAML with many auth
providers and python trsmeworks; including Pyramid, BitBucket, Google,
GitHub, GitLab,

https://python-social-auth.readthedocs.io/en/latest/

http://python-social-auth.readthedocs.io/en/latest/backends/

https://github.com/python-social-auth/social-app-pyramid/

There's likely someone with more experience with a different authentication
abstraction API?

https://github.com/uralbash/awesome-pyramid/#authentication lists quite a
few authentication and authorization systems which may also be useful for
implementing TUF?

On Friday, January 12, 2018, Donald Stufft  wrote:

> As folks are likely aware, legacy PyPI currently supports logging in using
> OpenID and Google Auth while Warehouse does not. After much deliberation,
> I’ve decided that Warehouse will not be implementing OpenID or Google
> logins, and once we shutdown legacy PyPI, OpenID/ and Google logins to PyPI
> will no longer be possible.
>
> This decision was made for a few reasons:
>
> * Very few people actually are using OpenID or Google logins as it is. In
> one month we had ~15k logins using the web form, ~5k using basic auth, and
> 62 using Google and 7 using OpenID. This is a several orders of magnitude
> difference.
> * Regardless of how you log into PyPI (Password or Google/OpenID) you’re
> required to have a password added to your account to actually upload
> anything to PyPI. This negates much of the benefit of a federated
> authentication for PyPI as it stands.
> * Keeping these requires ongoing maintenance to deal with any changes in
> the specification or to update as Google deprecates/changes things.
> * Adding support for them to Warehouse requires additional work that could
> better be used elsewhere, where it would have a higher impact.
>
> - Donald
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] RFC: PEP 566 - Metadata for Python Software Packages 1.3

[Wes Turner]

On Monday, December 11, 2017, Dustin Ingram  wrote:

> After working a bit on an implementation of the "JSON-compatible
> Metadata" section in this PEP, I'm noticing that it might be more
> useful if it had the following step instead for canonicalizing the
> field names:
>
> > #. All transformed keys should be reduced to lower case. Hyphens should
> be
> >replaced with underscores, but otherwise should retain all other
> characters;
>
> Thus a field like `Description-Content-Type` would become
> `description_content_type` and the resulting data structure would be
> slightly more useful (as it could be passed as `**kwargs` to a
> function with PEP 8 style parameter names)


Good idea.


>
> If no one has any objections to this, I'll update the draft accordingly.
>
> Also, if there is no other feedback on the current draft, I'l be
> formally submitting it for Daniel's review this week.


(Copied and pasted this from above)

>From "[distutils] Multiple package authors" [1]

- How should multiple author-email and maintainer-email addresses be
specified?

- Should we add security-email and/or security-disclosure-instructions?


[1] http://markmail.org/thread/xmwfktnsbmpakv6b


>
> Thanks,
> D.
>
> On Sat, Dec 9, 2017 at 8:27 AM, Daniel Holth  wrote:
> > I don't know why the parentheses were included in the older pep. They are
> > widely deployed. We probably can get rid of them or make them optional
> in a
> > practical parser.
> >
> >
> > On Sat, Dec 9, 2017, 02:03 Nick Coghlan  wrote:
> >>
> >> On 9 December 2017 at 02:42, Thomas Kluyver 
> wrote:
> >> > Dustin asked me to bring this issue to this thread:
> >> >
> >> > Metadata version 1.2 (PEP 345) says that version specifiers within a
> >> > Requires-Dist field should go in parentheses: "zope.interface
> (>3.5.0)".
> >> > The metadata spec on PyPUG repeats this.
> >> >
> >> > However, PEP 508 says that the parentheses are not needed, and tools
> >> > writing dependency specifications should not create them. Its
> >> > recommended format is therefore "zope.interface >3.5.0".
> >> >
> >> > Should the metadata 1.3 PEP note that this has changed? Or do we only
> >> > need to update the metadata spec on PyPUG?
> >>
> >> The new PEP already delegates to PEP 508 for the version specifier
> >> format, so I think that can just be updated in PyPUG.
> >>
> >> Cheers,
> >> Nick.
> >>
> >> --
> >> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> >> ___
> >> Distutils-SIG maillist  -  Distutils-SIG@python.org
> >> https://mail.python.org/mailman/listinfo/distutils-sig
> >
> >
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] RFC: PEP 566 - Metadata for Python Software Packages 1.3

[Wes Turner]

>From "[distutils] Multiple package authors" [1]

- How should multiple author-email and maintainer-email addresses be
specified?

- Should we add security-email and/or security-disclosure-instructions?


[1] http://markmail.org/thread/xmwfktnsbmpakv6b

On Wednesday, December 6, 2017, Nick Coghlan  wrote:

> On 6 December 2017 at 02:39, Dustin Ingram 
> wrote:
> > Hi all,
> >
> > I'd appreciate your feedback on the following Metadata 1.3 PEP.
> >
> > The goal here is not to provide a full specification for all fields as
> > in previous PEPs, but to:
> >
> > * Motivate and describe the addition of the new fields or changes to
> > existing fields;
> > * Point to the Core Metadata Specifications reference document as the
> > canonical source for the specification.
>
> Hi folks,
>
> While I'd normally step up as BDFL-Delegate for an interoperability
> specification like this one, I'm actually going to be offline for most
> of the rest of the year, which I figure wouldn't be especially
> conducive to running an effective and timely review process :)
>
> So while I'll note that I'm personally happy with the PEP as it
> currently stands (based on my review of Dustin's earlier drafts), I've
> also asked Daniel Holth if he'd be willing to handle the task of final
> review and approval for this PEP as the original author of the wheel
> specification, and Daniel's graciously agreed to do so.
>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Multiple package authors

[Wes Turner]

There are author-email and maintainer-email fields.

You could also or instead use a mailing list address for the author-email
or maintainer-email fields. Newlines work (just like file\nnames)?

With a mailing list, package maintainers can share responsibility (*) and
hand off correspondence without forwards and indentation.

Google Groups is free; are there alternatives:
https://support.google.com/groups/answer/2464926

You can use a third party service to create e.g. GitHub or GitLab issues
via email; however, security sensitive information (vulnerabilities,
credentials, personal information) may require additional caution and
admonitions.

If not otherwise specified in the long description, presumably the
author-email and/or maintainer-email address(es) are the correct place to
send fair disclosure information.

This could be a separate thread/issue and an additional package metadata
field maybe for Package Metadata 1.3? Sorry, a BIT OT.

security-email?

On Thursday, December 7, 2017, Barry Warsaw  wrote:

> I think I implicitly knew this, but as I've just released a package (to
> be announced soon) that actually has multiple authors, I found out first
> hand that PyPI rejects uploads where the author-email field isn't a
> completely valid email address, and that there is no support for
> multiple author emails.
>
> As it turns out, you can kludge this into your pyproject.toml or
> setup.py file.  flit for example separates multiple emails with a
> newline, but you could also separate them with commas.  You don't notice
> the problem until PyPI rejects the upload (with a 400 IIRC).
>
> I filed this issue with flit: https://github.com/takluyver/flit/issues/153
>
> It looks like Thomas agrees that at least flit will eventually validate
> its fields so you error early.  It was a bit of a PITA to do my upload
> because I didn't notice the problem until after I'd tagged the repo.
>
> Multiple package authors doesn't seem like that fringe of a use case;
> are there any plans, documents, PEPs, musings, grumbles about supporting
> multiple package authors explicitly?
>
> Cheers,
> -Barry
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pythonhosted.org doc upload no longer works

[Wes Turner]

Is there a recommended way to redirect docs that weren't updated before new
uploads were prevented?

(The mailing list thread about this change is further up in the thread)

On Monday, November 20, 2017, Ronald Oussoren 
wrote:

>
> On 20 Oct 2017, at 14:42, Ronald Oussoren  > wrote:
>
>
> On 7 Oct 2017, at 09:12, Giampaolo Rodola'  > wrote:
>
> Any news about this please? This is becoming a problem as my outdated doc
> keeps being around and I have no way to remove it.
>
>
> I have the same problem with my projects: the documentation on
> pythonhosted.org can no longer be updated, and it is also not possible to
> change that documentation into a link to a new location.
>
>
> Sigh… I still have outdated documentation on pythonhosted.org and no way
> to replace this with a link to the new location. As the documentation on
> pythonhosted is the top hit on Google I’d rather not remove the
> documentation there without any replacement.
>
> Ronald
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where to get help changing ownership of Pypi package?

[Wes Turner]

There's a "package claim" label in the pypa/pypi-legacy Github issue
tracker:

https://github.com/pypa/pypi-legacy/issues

What is the documented procedure for handling transfer of package ownership?

On Sunday, November 5, 2017, Stuart Axon via Distutils-SIG <
distutils-sig@python.org> wrote:

> Hi,
>I realise this is the wrong place, but does anyone know who could help
> resolve ownership of the Pypi package PyCairo
>
> https://sourceforge.net/p/pypi/support-requests/728/
>
>
> There are also a bunch of similar tickets, I guess nobody checks these any
> more ? :(
>
> S++
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Python-ideas] Add processor generation to wheel metadata

[Wes Turner]

Maybe the anaconda team has some insight on a standard way to capture (&
configure) compiler versions and flags in package metadata?

From
https://www.anaconda.com/blog/developer-blog/announcing-the-release-of-anaconda-distribution-5-0/
:

> The Anaconda 5.0 release used very modern compilers to rebuild almost
everything (~99.5%) provided in the installers for x86 Linux and MacOS.
This enables Anaconda users to get the benefits of the latest compilers—
still allowing support for older operating systems—back to MacOS 10.9 and
CentOS 6. Our own builds of GCC 7.2 (Linux) and Clang 4.0.1 (MacOS) are
used, and every reasonable security flag has been enabled. CFLAGS and
CXXFLAGS are no longer managed by each package; instead compiler activation
sets them globally.
>
> The packages built with the new compilers are in a different channel from
packages built the old way, and as we build out this new channel, we will
eventually be able to change the default experience to only using these
packages. Interested in using this approach to build your own conda
packages? Stay tuned for a more developer-focused blog post!

On Tuesday, October 31, 2017, Ivan Pozdeev via Distutils-SIG <
distutils-sig@python.org> wrote:

> Missed the fact that Nathaniel didn't quote the entire original message.
> Here it is:
> --
>
> Generally, packages are compiled for the same processor generation as the
> corresponding Python.
> But not always -- e.g. NumPy opted for SSE2 even for Py2 to work around
> some compiler bug
> (https://github.com/numpy/numpy/issues/6428).
> I was bitten by that at an old machine once and found out that there is no
> way for `pip' to have checked for that.
> Besides, performance-oriented packages like the one mentioned could
> probably benefit from newer instructions.
>
> Regarding identifiers:
> gcc, cl and clang all have their private directories of generation
> identifiers:
> https://gcc.gnu.org/onlinedocs/gcc-4.7.1/gcc/i386-
> and-x86_002d64-Options.html
> https://msdn.microsoft.com/en-us/library/7t5yh4fd.aspx
> https://clang.llvm.org/doxygen/Driver_2ToolChains_
> 2Arch_2X86_8cpp_source.html
>
> Linux packages typically use gcc's ones. Clang generally follows in gcc's
> footsteps and accepts cl's IDs, too, as aliases.
>
> So, using the IDs of whatever compiler is used to build the package (i.e.
> most likely the canonical compiler for CPython for that platform) looks
> like the simple(r) way - we can just take the value of the "march"
> argument.
>
>
> The tricky part is mapping the system's processor to an ID when checking
> compatibility: the logic will have to keep a directory (that's the job of
> `wheel' package maintainers though, I guess).
> I also guess that there are cases where there's no such thing as _the_
> system's processor.
>
> --
> Regards,
> Ivan
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheel 1.0 roadmap

[Wes Turner]

REQ: feedback re: "Remove or deprecate wheel signing features #196"
https://github.com/pypa/wheel/issues/196

Is the current implementation incomplete without signature verification?
According to the spec?

```
The spec includes this feature. So, even though this verify() function is
incomplete, it would be wrong to just remove it without also removing it
from the spec.

- https://www.python.org/dev/peps/pep-0427/#signed-wheel-files
- https://www.python.org/dev/peps/pep-0491/#signed-wheel-files

I don't have the information needed to explain what completely implemented
signatures are useful for. Does the spec explain this?

> A wheel installer is not required to understand digital signatures but
MUST verify the hashes in RECORD against the extracted file contents. When
the installer checks file hashes against RECORD, a separate signature
checker only needs to establish that RECORD matches the signature.
```

On Sunday, October 29, 2017, Alex Grönholm  wrote:

> I am planning for a 1.0.0 release of the "wheel" library. I would like to
> start using semver from this point onwards, which in the case of wheel
> means that its command line interface should be well defined and remain
> backwards compatible. As part of this effort, I've rewritten the
> documentation (currently in the "docs-update" branch on Github) to conform
> to the PyPA guidelines. Wheel also had some generated API documentation on
> ReadTheDocs, but as discussed privately with Daniel Holth and Nick Coghlan,
> wheel should not have a public API going forward so I've deleted that
> documentation.
>
> I've also taken a hard look at wheel's features and would like to remove
> those which I consider to be either useless or harmful. I've added these
> tasks as issues on Github.
>
> All the issues that I'd like to get resolved by 1.0.0 have been tagged
> with the proper milestone marker here: https://github.com/pypa/wheel/
> milestone/1
>
> Feedback is very welcome!
>
> ps. Daniel, if you're reading this, would you mind giving the new docs a
> once-over? Also, if you can suggest where to put the "story" page, I'll
> link it back to the main index file.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Wes Turner]

- Are there issue tickets which contain the search-indexed ERROR_STRINGS
users may encounter due to this change?

- Does it make sense to add an update regarding this necessary security
upgrade to https://status.python.org (which can be subscribed to and
followed on http://www.twitter.com/PythonStatus )?

On Thursday, October 26, 2017, Donald Stufft  wrote:

> Historically PyPI was only available over either HTTP or unvalidated
> HTTPS, and over time we’ve been pushing more and more traffic onto HTTPS.
> In Warehouse the decision was made to *not* redirect “API” URLs from HTTP
> to HTTPS, but to rather return an error accessing them from HTTP. This is
> because while logged in views have HSTS to ensure HTTPS in the browser (and
> with humans manually entering them into the URL bar regularly they are more
> error prone) APIs which are typically accessed by automated clients with an
> URL configured or hardcoded typically do not respect HSTS, so if you had a
> script that did ``curl http://pypi.python.org/simple/``, it would
> silently get redirects to https and appear to “work”, but you wouldn’t get
> any of the security properties of TLS because an attacker would just
> intercept the request prior to the redirect happening.
>
> Today I’ve backported this changed to the current production deployment of
> PyPI, which means that you can no longer access /simple/ and /packages/
> over HTTP and you will have to directly go to HTTPS. For most people this
> should have no effect, because most tooling should be defaulting to HTTPS
> anyways, however if you’re using a significantly old version of tooling, it
> may still be defaulting to the HTTP url and will now stop functioning.
>
> The recommended remediation is to upgrade your tooling to versions that
> support verified TLS connections and which default to the proper HTTPS URLs.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Wes Turner]

On Friday, October 27, 2017, Paul Moore  wrote:

> On 27 October 2017 at 22:22, Alex Domoradov  > wrote:
> > I got it. And what I should do with old system? For e.g. we still use
> ubuntu
> > 12.04. Is there any way to upgrade pip/setuptools?
>
> Well, if Ubuntu aren't offering an upgrade, you can do
>
> pip install -i https://pypi.python.org/simple/ ...
>
> Or install your own copy of pip/setuptools, I guess (get-pip --user,
> see https://pip.pypa.io/en/stable/installing/#installing-with-get-pip-py).
> Paul


Does this upgrade system pip, regardless of the package checksums:

$ sudo pip install -i https://pypi.python.org/simple/ -U pip setuptools


If that doesn't work (or **isn't advisable because `sudo pip` is
dangerous**),
you can also configure the index URL with a config file or an environment
variable:

https://pip.pypa.io/en/latest/user_guide/#config-file lists the paths for
Windows, MacOS, and Linux.

/etc/pip.conf
~/.pip/pip.conf
$VIRTUAL_ENV/pip.conf

```
[global]

index-url = https://pypi.python.org/simple/
```

$ export PIP_INDEX_URL="https://pypi.python.org/simple/;

Setuptools (easy_install) uses ~/.pydistutils.cfg :

```
[easy_install]
index_url = https://pypi.python.org/simple/
```

Buildout uses buildout.cfg and ~/.buildout/default.cfg :

```
[buildout]
index = https://pypi.python.org/simple/
```

"What to do when PyPi goes down" (2010)
https://jacobian.org/writing/when-pypi-goes-down/

This URL also works as the -i/--index-url?

https://pypi.org/simple/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Re PEP 508

[Wes Turner]

https://docs.python.org/devguide/pullrequest.html

https://github.com/python/peps/issues/new

https://github.com/python/peps/blob/master/pep-0508.txt


On Sunday, October 22, 2017, Michael Laing  wrote:

> I have a correction and a couple of possible improvements to the parsley
> grammar.
>
> Is there a convenient way to submit them?
>
> Thanks,
>
> ml
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org 
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Wes Turner]

On Saturday, October 21, 2017, Nick Coghlan  wrote:

> On 20 October 2017 at 23:42, Donald Stufft  > wrote:
>
>> On Oct 20, 2017, at 9:35 AM, Nick Coghlan > > wrote:
>> The interoperability spec is going to state that conflict resolution when
>> the same name within a group is declared by multiple packages is the
>> responsibility of the group consumer, so documenting the format should
>> actually improve this situation, since it allows for the development of
>> competing conflict resolution strategies in different runtime libraries.
>>
>> I think it makes it *worse*, because now the behavior isn’t just a
>> entrypoints weirdness, but now it changes based on which runtime library
>> you use (which isn’t something that end users are likely to have much
>> insight into) and it represents a footgun that package authors are unlikely
>> to be aware of. If mycoolentrypointslib comes out that is faster, but
>> changes some subtle behavior like this it’ll break people, but that is
>> unlikely going to be an effect that people expect to happen just because
>> they switched between two things both implementing the same standard.
>>
>> So effectively this means that not only is the fact you’re using
>> entrypoints part of your API, but now which entry point library you’re
>> using at runtime is now also part of your API.
>>
>
> The semantics of conflict resolution across different projects is a
> concern that mainly affects app developers a large established plugin base,
> and even with pkg_resources the question of whether or not multiple
> projects re-using the same entrypoint name is a problem depends on how the
> application uses that information.
>
> With console_scripts and gui_scripts, name conflicts can definitely be a
> problem, since different projects will end up fighting over the same
> filename for their executable script wrapper.
>
> For other use cases (like some of the ones Doug described for stevedore),
> it's less of a concern, because the names never get collapsed into a single
> flat namespace the way script wrappers do.
>
> Cheers,
> Nick.
>
> P.S. Thanks for your comments on the PR - they're helping to make sure we
> accurately capture the status quo. I'm also going to file an issue on the
> setuptools issue tracker to make sure Jason is aware of what we're doing,
> and get his explicit OK with the idea of making the format a PyPA
> interoperability specification (if he isn't, we'll demote Thomas's document
> to being a guide for tool developers aiming for pkg_resources
> interoperability).
>

What are the URIs for this PR and issue?


>
> --
> Nick Coghlan   |   ncogh...@gmail.com
>    |   Brisbane,
> Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Wes Turner]

On Friday, October 20, 2017, Doug Hellmann  wrote:

> Excerpts from Wes Turner's message of 2017-10-20 10:41:02 -0400:
> > On Friday, October 20, 2017, Donald Stufft  > wrote:
> >
> > >
> > >
> > > On Oct 20, 2017, at 9:35 AM, Nick Coghlan  
> > > ');>>
> wrote:
> > >
> > > On 20 October 2017 at 23:19, Donald Stufft  
> > > ');>>
> wrote:
> > >
> > >> One that I was helping someone debug just the other day is that
> they’re
> > >> super non-debuggable and the behavior when you have two things
> providing
> > >> the same entry point is basically  (If I remember correctly, the
> > >> behavior is that the first thing found is the one that “wins”, which
> means
> > >> the ordering of sys.path and the names of the projects supply it is
> what
> > >> determines it). This got exposed to the end user that they installed
> > >> something that they thought was going to add support for something,
> but
> > >> which silently did nothing because two different project happened to
> pick
> > >> the same name for their entry point (not the group, it was two things
> > >> providing plugins for the same system).
> > >>
> > >
> > > While I agree with this, I think that's a combination of pkg_resources
> > > itself being hard to debug in general, and the fact that pkg_resources
> > > doesn't clearly define the semantics of how it resolves name conflicts
> > > within an entry point group - as far as I know, it's largely an
> accident of
> > > implementation.
> > >
> > > The interoperability spec is going to state that conflict resolution
> when
> > > the same name within a group is declared by multiple packages is the
> > > responsibility of the group consumer, so documenting the format should
> > > actually improve this situation, since it allows for the development of
> > > competing conflict resolution strategies in different runtime
> libraries.
> > >
> > >
> > > I think it makes it *worse*, because now the behavior isn’t just a
> > > entrypoints weirdness, but now it changes based on which runtime
> library
> > > you use (which isn’t something that end users are likely to have much
> > > insight into) and it represents a footgun that package authors are
> unlikely
> > > to be aware of. If mycoolentrypointslib comes out that is faster, but
> > > changes some subtle behavior like this it’ll break people, but that is
> > > unlikely going to be an effect that people expect to happen just
> because
> > > they switched between two things both implementing the same standard.
> > >
> > > So effectively this means that not only is the fact you’re using
> > > entrypoints part of your API, but now which entry point library you’re
> > > using at runtime is now also part of your API.
> > >
> >
> > When should the check for duplicate entry points occur?
> >
> > - At on_install() time (+1)
> > - At runtime
> >
> > Is a sys.path-like OrderedDict preemptive strategy preferable or just as
> > dangerous as importlib?
>
> Having "duplicate" entry points is not necessarily an error. It's
> a different usage pattern.  The semantics of dropping a named plugin
> into a namespace are defined by the application and plugin-point.
> Please do not build assumptions about uniqueness into the underlying
> implementation.


I think that, at least with console_scripts, we already assume uniqueness:
if there's another package which provides a 'pip' console_script, for
example, there's not yet an error message?

Would it be helpful to at least spec that iterated entrypoints are in
sys.path order? And then what about entrypoints coming from the same path
in sys.path: alphabetical? Whatever hash randomization does with it?

Whenever I feel unsure about my data model, I tend to sometimes read the
OWL spec: here, the OWL spec has owl:cardinality OR owl:minCardinality OR
owl:maxCardinality. Some entrypoints may have 0, only one, or n "instances"?

We should throw an error if a given console_script entrypoint has more than
one "instance" (exceeds maxCardinality xsd:string = 1).


>
> The stevedore library wraps up pkg_resources with several such
> patterns. For example, it supports "give me all of the plugins in
> a namespace" (find all the extensions to your app), "give me all
> of the plugins named $name in a namespace" (find the hooks for a
> specific event defined by the app), and "give me *the* plugin named
> $name in a namespace" (load a driver for talking to a backend).
>
> https://docs.openstack.org/stevedore/latest/reference/index.html


https://github.com/openstack/stevedore/blob/master/stevedore/extension.py

https://github.com/openstack/stevedore/blob/master/stevedore/tests/test_extension.py

These tests mention saving discovered entry points in a cache?
___

Re: [Distutils] Documentation link on PyPI.org

[Wes Turner]

On Friday, October 20, 2017, Ronald Oussoren <ronaldousso...@mac.com> wrote:

>
> On 28 Aug 2017, at 00:12, Brett Cannon <br...@python.org
> <javascript:_e(%7B%7D,'cvml','br...@python.org');>> wrote:
>
> If you search the archive of this mailing list you will notice I asked
> this exact question about a month or 2 ago (I think). The answer I got was
> it is used on PyPI.org, but I don't know how to set it with setuptools
> (flit will probably add support after PEP 517).
>
>
> To get back to this: I’m probably doing something wrong, but I haven’t
> managed yet to make anything show up on PyPI.org when I use “Project-URL”
> in metadata.
>
> There’s a pyobjc-core 4.0.1b1 wheel on PyPI.org that includes two
> Project-URL lines in its metadata, but that doesn’t add anything to the
> sidebar. I’ve manually verified that the metadata is present in the wheel
> file (as I’m using a replacement egg_info command in my setup.py file to
> generate this part of the metadata).
>

https://github.com/pypa/warehouse/search?utf8=✓="project-url"=Issues

https://github.com/pypa/warehouse/search?q="Project-Url"=Commits=✓


>
> Ronald
>
> P.S. I tried to upload to test.pypi.org, but that didn’t work due to an
> SSL error:
>
> $ twine upload -r testpypi dist/pyobjc_core-4.0.1b1-cp27-
> cp27m-macosx_10_6_intel.whl
> Uploading distributions to https://test.pypi.org/legacy/
> Uploading pyobjc_core-4.0.1b1-cp27-cp27m-macosx_10_6_intel.whl
> SSLError: HTTPSConnectionPool(host='test.pypi.org', port=443): Max
> retries exceeded with url: /legacy/ (Caused by SSLError(SSLError(1, u'[SSL:
> TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)'),))
>

https://www.google.com/search?q=TLSV1_ALERT_PROTOCOL_VERSION

Looks like TLSv1 (TLS 1.0) is deprecated.

https://www.google.com/search?q=TLSV1_ALERT_PROTOCOL_VERSION+twine

https://github.com/pypa/twine/issues/273




>
>
> On Sun, Aug 27, 2017, 13:01 Ronald Oussoren <ronaldousso...@mac.com
> <javascript:_e(%7B%7D,'cvml','ronaldousso...@mac.com');>> wrote:
>
>> On 27 Aug 2017, at 17:33, Wes Turner <wes.tur...@gmail.com
>> <javascript:_e(%7B%7D,'cvml','wes.tur...@gmail.com');>> wrote:
>>
>> You can add a URL in the long_description (e.g. to https://read-the-docs.
>> readthedocs.org/ )
>>
>>
>> I know, but that adds the URL to the project description and not the
>> sidebar. PEP 345 appears to specify metadata that could be used for this,
>> and warehouse seems to contain to make use of this (or at least a database
>> model that allows for specifying more links than just the default one).
>>
>> Adding links to the sidebar would be nicer with the new design on
>> PyPI.org <http://pypi.org/>, but there doesn’t seem to be a clean way to
>> add “Project-URL” metadata as described in PEP 345 to a wheel file. I could
>> tweak my setup.py files to add Project-URL metadata, but before I do that
>> I’d like to confirm that (a) that metadata is actually used by warehouse
>> and (b) there is no cleaner way to add this metadata using setuptools.
>>
>> Ronald
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> <javascript:_e(%7B%7D,'cvml','Distutils-SIG@python.org');>
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Wes Turner]

On Friday, October 20, 2017, Donald Stufft  wrote:

>
>
> On Oct 20, 2017, at 9:35 AM, Nick Coghlan  > wrote:
>
> On 20 October 2017 at 23:19, Donald Stufft  > wrote:
>
>> One that I was helping someone debug just the other day is that they’re
>> super non-debuggable and the behavior when you have two things providing
>> the same entry point is basically  (If I remember correctly, the
>> behavior is that the first thing found is the one that “wins”, which means
>> the ordering of sys.path and the names of the projects supply it is what
>> determines it). This got exposed to the end user that they installed
>> something that they thought was going to add support for something, but
>> which silently did nothing because two different project happened to pick
>> the same name for their entry point (not the group, it was two things
>> providing plugins for the same system).
>>
>
> While I agree with this, I think that's a combination of pkg_resources
> itself being hard to debug in general, and the fact that pkg_resources
> doesn't clearly define the semantics of how it resolves name conflicts
> within an entry point group - as far as I know, it's largely an accident of
> implementation.
>
> The interoperability spec is going to state that conflict resolution when
> the same name within a group is declared by multiple packages is the
> responsibility of the group consumer, so documenting the format should
> actually improve this situation, since it allows for the development of
> competing conflict resolution strategies in different runtime libraries.
>
>
> I think it makes it *worse*, because now the behavior isn’t just a
> entrypoints weirdness, but now it changes based on which runtime library
> you use (which isn’t something that end users are likely to have much
> insight into) and it represents a footgun that package authors are unlikely
> to be aware of. If mycoolentrypointslib comes out that is faster, but
> changes some subtle behavior like this it’ll break people, but that is
> unlikely going to be an effect that people expect to happen just because
> they switched between two things both implementing the same standard.
>
> So effectively this means that not only is the fact you’re using
> entrypoints part of your API, but now which entry point library you’re
> using at runtime is now also part of your API.
>

When should the check for duplicate entry points occur?

- At on_install() time (+1)
- At runtime

Is a sys.path-like OrderedDict preemptive strategy preferable or just as
dangerous as importlib?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Wes Turner]

On Friday, October 20, 2017, Thomas Kluyver  wrote:

> On Fri, Oct 20, 2017, at 01:36 PM, Donald Stufft wrote:
>
> Entry points have a lot of problems and I know of multiple systems that
> have either moved away from them, had to hack around how bad they are, have
> refused to implement them because of previous pain felt by them, are
> looking for ways to eliminate them, or which just regret ever supporting
> them.
>
>
> The fate of the PR notwithstanding, I'd be interested in hearing more
> about what problems projects have experienced with entry points, if you
> have time to describe some examples. We're looking at using them in more
> places than we already do, so it would be useful to hear about drawbacks we
> might not have thought about, and about what other options projects have
> moved to.
>
> Thomas
>

 What were the issues with setuptools entry points here (in 2014, when you
two were opposed to adding them to sendibly list ipython extensions)?

https://github.com/ipython/ipython/pull/4673

https://github.com/ipython/ipython/compare/master...westurner:setuptools_entry_points
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Wes Turner]

On Thursday, October 19, 2017, Donald Stufft  wrote:

>
> On Oct 19, 2017, at 5:26 PM, Tres Seaver  > wrote:
>
> Having the packaging
> system register those services at installation time (even if it doesn't
> care otherwise about them) seems pretty reasonable to me.
>
>
> It does register them at installation time, using an entirely generic
> feature of “you can add any file you want to a dist-info directory and we
> will preserve it”. It doesn’t need to know anything else about them other
> then it’s a file that needs preserved.
>

When I think of 'register at installation time', I think of adding them to
a single { locked JSON || SQLite DB || ...}; because that's the only way
there'd be a performance advantage?

Why would we write a .txt, transform it to {JSON || SQL INSERTS}, and then
write it to a central registrar?

(BTW, pipsi does console script entry points with isolated virtualenvs
linked into from ~/.local/bin (which is generally user-writable)).
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Wes Turner]

def get_env_json_path():
  directory = $VIRTUAL_ENV || ?
  return os.path.join(directory, ENV_JSON_FILENAME)

def on_install(pkg_json):
  env_json_path = get_env_json_path()
  env_json = json.load(env_json_path)
  env_json['pkgs’][pkgname] = pkg_json
  with open(env_json_path, 'w') as f:
f.write(env_json)

def read_cached_entry_points():
  env_json_path = get_env_json_path()
  env_json = json.load(env_json_path)
  entry_points = flatten(**{ pkg['entry_points'] for pkg in
env_json['pigs']})
  return entry_points


Would this introduce a need for a new and confusing rescan_metadata()
(pkg.on_install() for pkg in pkgs)?

On Wednesday, October 18, 2017, Nick Coghlan  wrote:

> On 19 October 2017 at 12:16, Daniel Holth  > wrote:
>
>> We said "you won't have to install setuptools" but actually "you don't
>> have to use it" is good enough. If you had 2 pkg-resources implementations
>> running you might wind up scanning sys.path extra times...
>>
> True, but that's where Thomas's suggestion of attempting to define a
> standardised caching convention comes in: right now, there's no middle
> ground between "you must use pkg_resources" and "every helper library must
> scan for the raw entry-point metadata itself".
>
> If there's a defined common caching mechanism, and support for it is added
> to new versions of pkg_resources, then the design constraint becomes "If
> you end up using multiple entry-point scanners, you'll want a recent
> setuptools/pkg_resource, so you don't waste too much time on repeated
> metadata scans".
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com
>    |   Brisbane,
> Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig