[Distutils] Re: Shutting down distutils-sig, for real

[Daniel Holth]

What a wonderful new year's gift 

On Thu, Dec 30, 2021, 11:05 AM Jeremy Stanley  wrote:

> On 2021-12-30 09:55:34 -0500 (-0500), Andrew M. Kuchling wrote:
> [...]
> > let's shut the list down for 2021!
> [...]
>
> No objection from me at this point. While I find Discourse to
> perform suboptimally as a mailing list server, all the packaging
> discussions seem to have moved there (or into GitHub issues, which
> are an even worse substitute for a mailing list), so it'll have to
> do.
>
> Thanks for all your hard work as a ML moderator over the years, and
> to everyone else who made this list great!
> --
> Jeremy Stanley
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/IYIWFQ34DSZCPULQRFBBM2KSDBRLCB56/
>
-- 
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/GTHXWASMQLXJNO4OVMGFW6NNIMTBUWQD/

[Distutils] Re: Making setup.py run external command to build

[Daniel Holth]

You would have to have a high tolerance for learning SCons. I'm aware that
this is not for everyone. Then you could write a SConstruct with dependent
tasks in a normal build system way. e.g.

target = env.Command("a task", ...)
platlib = env.Whl("platlib", target, root=".")
whl = env.WhlFile(platlib)

and have each step re-build only when its sources change.

Here is a stackoverflow answer about custom distutils commands.
https://stackoverflow.com/questions/1710839/custom-distutils-commands
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/QMFJZ4O6XFUE3FLRSH5J4BBJT436F43L/

[Distutils] Re: Making setup.py run external command to build

[Daniel Holth]

enscons is an alternative to distutils.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/EJQKJHKKWD77YMPTSZFJBYUFJNKTQFZX/

[Distutils] Re: Making setup.py run external command to build

[Daniel Holth]

I've been working on ensconced which lets you use a SConstruct to build. It
is easier than customizing distutils.

https://github.com/dholth/pysdl2-cffi/blob/master/SConstruct is the most
code-generation-y project. Its build might be a little out of date and it
has an independent setup.py building the project with setuptools.

https://github.com/dholth/bioopenssl doesn't have C extensions but is an
example of a stub setup.py (only necessary for the develop command) and
pyproject.toml (which pip can use without a setup.py via PEP 517).

On Fri, Dec 11, 2020 at 10:09 AM Julian Smith  wrote:

> Hello
>
> I have a project with a fairly involved build process where we generate
> C++ via a python-clang-based code analyser, generates Python bindings
> for the resulting C++ API with SWIG, and finally compile and link to
> create various .so's and .py files.
>
> I'd like to package things up with a setup.py script, but from what i
> have read, it looks like distutils expects to compile, link and run
> swig itself, using distutils.core.setup()'s ext_modules arg?
>
> Is there a way to write setup.py so that it instead runs an external
> command (e.g. 'cd foo && make all') when it is asked to build?
>
> Thanks for any help,
>
> - Jules
>
> --
> http://op59.net
>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/AJDJU5DYMDBPGZ75P35M5647O5VH22FL/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/FBC7LEB3WUK2XTOKYXKQMO3J7WA47SNE/

[Distutils] Re: Shipping a Python C extension + a library it uses as source

[Daniel Holth]

If you're adventurous we could try to get it built and packaged with SCons.
This extension generates code and then complies a cffi module:
https://bitbucket.org/dholth/pysdl2-cffi/raw/a31bf1d1100bc4573ccadcd574f8fdc5cc841e1c/SConstruct

On Tue, Jun 11, 2019, 16:13 Spencer Baugh  wrote:

>
> Hi distutils-sig,
>
> I've developed a small C library and a much larger Python native
> extension that uses that library. The native extension has no
> dependencies other than that small C library.
>
> While the library is useful on its own, I'm thinking it would be good to
> include the library's source directly with my native extension's source
> when I upload to PyPI, so that it can be easily built for arbitrary
> platforms without me having to create wheels for them.
>
> Is this a known approach? Have people done this before?
>
> For reference, the C library, which currently builds with autotools, is
> here:
> https://github.com/catern/rsyscall/tree/master/c
> And the Python library and native extension, which currently builds with
> setuptools and uses cffi, is here:
> https://github.com/catern/rsyscall/tree/master/python
>
> Thanks for any guidance!
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/VB5DIPW3CUQVKBLXSDACRP23FAZWJPLS/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/ISK7YNCNACHXYQJ7XRLQ6D2MQ4YX6ZM5/

[Distutils] Re: enscons 0.7.0 released with PEP517 support

[Daniel Holth]

Try python -m enscons.setup2toml and then run scons in your favorite python
package.

On Tue, May 21, 2019, 10:14 Nick Coghlan  wrote:

> On Sun, 12 May 2019 at 09:19, Daniel Holth  wrote:
> >
> > And now for something completely different
> >
> > I've implemented PEP517 support for the SCons-powered build system
> > "enscons", quite possibly the most different pip build backend
> > available. It should be able to build itself and rsalette (another
> > enscons-using package) using pep517.
>
> Very cool :)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/ROOFOZ75IDNVKTLROWDJMRJCWXSVRQN3/

[Distutils] enscons 0.7.0 released with PEP517 support

[Daniel Holth]

And now for something completely different

I've implemented PEP517 support for the SCons-powered build system
"enscons", quite possibly the most different pip build backend
available. It should be able to build itself and rsalette (another
enscons-using package) using pep517. Note the newly required
env.Default() that enscons needs to return the target filename to pip.
Due to SCons limitations enscons may not be able to build .tar.gz
sdists on Windows.

enscons is an experimental Python build system that runs on top of
SCons, a Makefile replacement. It is able to build C. It should be
much easier to customize than a setuptools-based system. I originally
wrote it so that I could experiment with the wheel format without
having to hack on setuptools.

It might be useful to you if you have a package that has a complex
dependency tree of generated code between the source and whatever ends
up in the wheel, if you want to build more than one wheel from a
single source tree, or if you have other needs that just don't fit
into setuptools.

enscons also has a setup.py emulation that supports "setup.py develop".

The pep517 package is pretty cool. Thanks!

I would call these pep517 powered builds "pluggable".

Daniel Holth
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/V57I7I5EHR7Q6MA43XYU3YP4KCGDQ32V/

[Distutils] Re: Docker builds, ~/.pip/cache, and O(1) or O(n) bandwidth

[Daniel Holth]

Also you must set an environment variable for docker build to be able to
use --mount

On Tue, May 7, 2019, 12:28 Daniel Holth  wrote:

> That one is multistage, great feature, but I think cache mounts would work
> fine with any number of stages. The mount option is a pretty new feature.
>
> On Tue, May 7, 2019, 12:24 Wes Turner  wrote:
>
>>
>>
>> On Tuesday, May 7, 2019, Daniel Holth  wrote:
>>
>>> Have you tried using buildkit and the RUN --mount option? I've done
>>> extra stuff here (downloading rpms in a second image first) but I think you
>>> could just use the cache option.
>>>
>>> It would also be easy to use a second image and COPY in old docker.
>>>
>>> https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86
>>>
>>
>> Thanks. Is this a multi-stage Dockerfile (with multiple FROM
>> instructions)?
>>
>> Is there a guide for a PyPA recommended way to create network-efficient
>> reproducible (manylinux2010) wheels?
>>
>>
>>>
>>> On Tue, May 7, 2019, 11:59 Alex Becker  wrote:
>>>
>>>> You can use a local PyPI mirror, e.g. devpi, and point your docker
>>>> builds at that, basically tricking docker by going through the (local)
>>>> network stack instead of the filesystem.
>>>>
>>>> On Tue, May 7, 2019 at 8:12 AM Wes Turner  wrote:
>>>>
>>>>> What is the best way to build docker images without constantly
>>>>> re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n)
>>>>> for every build)
>>>>>
>>>>> (AFAIK, nobody has any issue with the amount of bandwidth PyPI uses)
>>>>>
>>>>> Thus far, Docker doesn't want to support a build-time -v option (that
>>>>> could be used to bind-mount .pip/cache in at build time):
>>>>>
>>>>> "build time only -v option"
>>>>> https://github.com/moby/moby/issues/14080
>>>>>
>>>>> Buildah *does* support a build-time -v option (and can also do
>>>>> rootless builds without a docker socket)
>>>>> --
>>>>> Distutils-SIG mailing list -- distutils-sig@python.org
>>>>> To unsubscribe send an email to distutils-sig-le...@python.org
>>>>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>>>>> Message archived at
>>>>> https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MGTSAA4X5UZDERCAAAFGGX5DGRPF3/
>>>>>
>>>> --
>>>> Distutils-SIG mailing list -- distutils-sig@python.org
>>>> To unsubscribe send an email to distutils-sig-le...@python.org
>>>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>>>> Message archived at
>>>> https://mail.python.org/archives/list/distutils-sig@python.org/message/2MNMGYBNFOOVGXA2JQ2PLDTZITZXANJU/
>>>>
>>>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/JXVF2IQKGCAGLH2A6KYOIQARJGXVKTI5/

[Distutils] Re: Docker builds, ~/.pip/cache, and O(1) or O(n) bandwidth

[Daniel Holth]

That one is multistage, great feature, but I think cache mounts would work
fine with any number of stages. The mount option is a pretty new feature.

On Tue, May 7, 2019, 12:24 Wes Turner  wrote:

>
>
> On Tuesday, May 7, 2019, Daniel Holth  wrote:
>
>> Have you tried using buildkit and the RUN --mount option? I've done extra
>> stuff here (downloading rpms in a second image first) but I think you could
>> just use the cache option.
>>
>> It would also be easy to use a second image and COPY in old docker.
>>
>> https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86
>>
>
> Thanks. Is this a multi-stage Dockerfile (with multiple FROM instructions)?
>
> Is there a guide for a PyPA recommended way to create network-efficient
> reproducible (manylinux2010) wheels?
>
>
>>
>> On Tue, May 7, 2019, 11:59 Alex Becker  wrote:
>>
>>> You can use a local PyPI mirror, e.g. devpi, and point your docker
>>> builds at that, basically tricking docker by going through the (local)
>>> network stack instead of the filesystem.
>>>
>>> On Tue, May 7, 2019 at 8:12 AM Wes Turner  wrote:
>>>
>>>> What is the best way to build docker images without constantly
>>>> re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n)
>>>> for every build)
>>>>
>>>> (AFAIK, nobody has any issue with the amount of bandwidth PyPI uses)
>>>>
>>>> Thus far, Docker doesn't want to support a build-time -v option (that
>>>> could be used to bind-mount .pip/cache in at build time):
>>>>
>>>> "build time only -v option"
>>>> https://github.com/moby/moby/issues/14080
>>>>
>>>> Buildah *does* support a build-time -v option (and can also do rootless
>>>> builds without a docker socket)
>>>> --
>>>> Distutils-SIG mailing list -- distutils-sig@python.org
>>>> To unsubscribe send an email to distutils-sig-le...@python.org
>>>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>>>> Message archived at
>>>> https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MGTSAA4X5UZDERCAAAFGGX5DGRPF3/
>>>>
>>> --
>>> Distutils-SIG mailing list -- distutils-sig@python.org
>>> To unsubscribe send an email to distutils-sig-le...@python.org
>>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/distutils-sig@python.org/message/2MNMGYBNFOOVGXA2JQ2PLDTZITZXANJU/
>>>
>>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/UWNYYGT7BGCEOM4JP7WMMPFQJZ2HBGHC/

[Distutils] Re: Docker builds, ~/.pip/cache, and O(1) or O(n) bandwidth

[Daniel Holth]

Have you tried using buildkit and the RUN --mount option? I've done extra
stuff here (downloading rpms in a second image first) but I think you could
just use the cache option.

It would also be easy to use a second image and COPY in old docker.

https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86

On Tue, May 7, 2019, 11:59 Alex Becker  wrote:

> You can use a local PyPI mirror, e.g. devpi, and point your docker builds
> at that, basically tricking docker by going through the (local) network
> stack instead of the filesystem.
>
> On Tue, May 7, 2019 at 8:12 AM Wes Turner  wrote:
>
>> What is the best way to build docker images without constantly
>> re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n)
>> for every build)
>>
>> (AFAIK, nobody has any issue with the amount of bandwidth PyPI uses)
>>
>> Thus far, Docker doesn't want to support a build-time -v option (that
>> could be used to bind-mount .pip/cache in at build time):
>>
>> "build time only -v option"
>> https://github.com/moby/moby/issues/14080
>>
>> Buildah *does* support a build-time -v option (and can also do rootless
>> builds without a docker socket)
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MGTSAA4X5UZDERCAAAFGGX5DGRPF3/
>>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/2MNMGYBNFOOVGXA2JQ2PLDTZITZXANJU/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/HUYZYFQVUGBVCQWT5AEJUTXQW4VJTZ6L/

[Distutils] Re: psycopg2/psycopg2-binary

[Daniel Holth]

I was frustrated with this very wheel. I had hoped I could provide a
locally compiled version in my local wheeldir on pip's path to be
automatically picked up, but pip prefers the manylinux tag (not compiled on
my own machine) to the standard local Linux tag (compiled on my machine).

On Sat, Apr 6, 2019, 18:16 Bert JW Regeer  wrote:

>
>
> On Apr 6, 2019, at 23:10, Nathaniel Smith  wrote:
>
> On Sat, Apr 6, 2019 at 2:14 PM Bert JW Regeer  wrote:
>
>
> Hey all,
>
> You may have seen some hub hub around psycopg2 and no longer shipping
> binary wheels by default [1][2] (and in fact using psycopg2-binary if you
> want wheels), and I wanted to bring it up here because it demonstrates a
> problem area with the current state of packaging in Python:
>
> There is no good way for a new package to specify that it provides what
> another package would provide, and setuptools currently checks that all
> distributions are found before running the console scripts (so if a console
> script has a setup.py that depends on psycopg2 and you install
> psycopg2-binary it fails) [3]
>
> So currently if you pip install psycopg2-binary and then install a project
> that uses psycopg2 as a dependency it will install psycopg2 over top of
> psycopg2-binary.
>
> The author of psycopg2 stopped distributing binaries for psycopg2 because
> of issues with the version of SSL that Python was compiled for/what was
> used by psycopg2 and it causing all kinds of issues.
>
> I don't have a proposal or a fix, but this is going to be an issue not
> just for psycopg2 but also for other projects that potentially distribute
> wheels that are built against a different version of OpenSSL.
>
> I see two things that should get some thought:
>
> 1. How to have a package provide for another package (there are keywords
> but AFAIK they are currently ignored by pip)
> 2. How to handle/deal with shared libraries that are not versioned
>
>
> The psycopg2 authors originally misdiagnosed the problem, and haven't
> updated their docs since the problem was diagnosed further, so a lot
> of people are confused about this whole psycopg2-binary thing :-(
>
> There is no problem with shipping openssl in wheels. Lots of projects
> do it fine. The reason psycopg2 is having problems is because of an
> easily-fixable bug in psycopg2:
>
> - Old versions of OpenSSL need some annoying configuration applied to
> make them thread-safe
> - libpq (which psycopg2 uses) normally does this configuration in one way
> - the Python ssl module also normally does this configuration in a
> different way
> - If libpq and the stdlib ssl module are both linked against the
> *same* copy of openssl, then they can end up fighting with each other
> - So the psycopg2 code has a special hack to unconditionally disable
> libpq's thread-safety code, because the psycopg2 developers assumed
> that psycopg2 and the stdlib ssl would *always* share the same copy of
> openssl, and the stdlib ssl module would take care of the
> thread-safety stuff
> - Then they started distributing psycopg2 binaries with their own copy
> of openssl in them, and of course they got crashes, because they've
> turned off thread-safety, and now that they have their own copy of
> openssl, no-one else is fixing it for them
>
> So all they need to do to fix their wheels is either:
>
> - somehow disable this patch in their wheel builds:
>
> https://github.com/psycopg/psycopg2/commit/a59704cf93e8594dfe59cf12d416e82a816953a4
>
> Or else:
>
> - switch to building their wheels against a newer version of openssl,
> since newer versions of openssl are now thread-safe by default (thank
> goodness)
>
> Either way, it's totally fixable, and only the psycopg2 devs can fix it.
>
> They've known this since July, but said they don't have energy to fix
> it: https://github.com/psycopg/psycopg2/issues/543#issuecomment-408352209
>
> I sympathize with that, but I wish they would tell people "hey, this
> is a bug in psycopg2, we need help", instead of blaming python
> packaging and trying to force all their downstreams to do wacky stuff
> with dependencies to work around their bug.
>
> It would indeed be nice if Python packages had better support for
> Provides:, but psycopg2 is not really a good motivating use case.
>
>
> No, it is not a good motivating use case, but it is an issue that exists
> _today_ hence why I brought it up.
>
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
>
> Thank you for your information Nathaniel. It is a shame that this fix was
> not used so that they could continue to ship psycopg2 wheels. Sadly the
> maintainers have made more work for themselves by not following up on your
> suggestions :-/
>
> Bert
>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> 

[Distutils] Re: PEP-582 concerns

[Daniel Holth]

I like the pep at first glance. I have long thought that virtualenv was a
weird solution to an artificial problem notwithstanding that all
programming problems are artificial. Virtualenv looks good only because a
global interpreter centric environment is bad. A program centric
alternative is welcome.

On Wed, Feb 20, 2019, 08:57 Nathaniel Smith  I'd caution against folks getting too worked up about PEP 582. I know it's
> been getting a lot of attention on social media recently, but, it's a draft
> that hasn't even been submitted for discussion yet. Most PEPs at this stage
> never end up going anywhere. And in general, when people start digging in
> on positions for and against something it always leads to worse decisions,
> and the earlier this happens the worse it gets.
>
> It has some interesting ideas and also some real limitations. I think it's
> a good signal that there are folks interested in helping make the python
> dev workflow easier, including changing the interpreter if that turns out
> to be the right thing to do. That's really all it means so far.
>
> I wonder if we should stick a header on the PEP draft saying something
> like this? There's a lot of scattershot responses happening and I think a
> lot of the people reacting are lacking context.
>
> -n
>
> On Wed, Feb 20, 2019, 04:40 Alex Walters  wrote:
>
>> I have 2 main concerns about PEP 582 that might just be me
>> misunderstanding
>> the pep.
>>
>> My first concern is the use of CWD, and prepending ./_pypackages_ for
>> scripts.  For example, if you were in a directory with a _pypackages_
>> subdirectory, and had installed the module "super.important.module".  My
>> understanding is that any scripts you run will have
>> "super.important.module"
>> available to it before the system site-packages directory.  Say you also
>> run
>> "/usr/bin/an_apt-ly_named_python_script" that uses
>> "super.important.module"
>> (and there is no _pypackages_ subdirectory in /usr/bin).  You would be
>> shadowing "super.important.module".
>>
>> In this case, this adds no more version isolation than "pip install
>> --user",
>> and adds to the confoundment factor for a new user.  If this is a
>> misunderstanding of the pep (which it very well might be!), then ignore
>> that
>> concern.  If it's not a misunderstanding, I think that should be
>> emphasized
>> in the docs, and perhaps the pep.
>>
>> My second concern is a little more... political.
>>
>> This pep does not attempt to cover all the use-cases of virtualenvs -
>> which
>> is understandable.  However, this also means that we have to teach new
>> users
>> *both* right away in order to get them up and running, and teach them the
>> complexities of both, and when to use one over the other.  Instead of
>> making
>> it easier for the new user, this pep makes it harder.  This also couldn't
>> have come at a worse time with the growing use of pipenv which provides a
>> fully third way of thinking about application dependencies (yes, pipenv
>> uses
>> virtualenvs under the hood, but it is a functionally different theory of
>> operation from a user standpoint compared to traditional pip/virtualenv or
>> this pep).
>>
>> Is it really a good idea to do this pep at this time?
>>
>> In a vacuum, I like this pep.  Aside from the (possible) issue of
>> unexpected
>> shadowing, it's clean and straight forward.  It's easy to teach.  But it
>> doesn't exist in a vacuum, and we have to teach the methods it is intended
>> to simplify anyways, and it exists in competition with other solutions.
>>
>> I am not a professional teacher; I don't run python training courses.  I
>> do,
>> however, volunteer quite a bit of time on the freenode channel.  I get
>> that
>> the audience there is self-selecting to those who want to donate their
>> time,
>> and those who are having a problem (sometimes, those are the same people).
>> This is the kind of thing that generates a lot of confusion and
>> frustration
>> to the new users I interact with there.
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/distutils-sig@python.org/message/SFMFKTQVKTONCYNN7UEKLFAQ2VRKXEHK/
>>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/5AGJJPMSWI7VIXXOBBYBH7PYINQI7HWM/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/T33UIQLXGDZJ2K6KAPGHLSIIILBYQFHF/

[Distutils] Re: Announcing Wheelodex

[Daniel Holth]

I like it

On Thu, Oct 18, 2018, 13:06 Dustin Ingram  wrote:

> What a great tool! I can see myself using this to explain to folks why
> the metadata for a given release is broken or incorrect. It would be
> really nice (for me) to be able to link to specific lines in the
> metadata file, but I figure most people don't need that feature.
>
> Very interesting to see the list of projects that declare a dependency
> on `pip` (and thus are probably importing `pip`, although I'm sure
> most don't actually declare the dependency):
> https://www.wheelodex.org/projects/pip/rdepends/
> On Thu, Oct 18, 2018 at 11:09 AM John Thorvald Wodder II
>  wrote:
> >
> > I've created a website that is likely to be of interest to people here:
> Wheelodex , a site for browsing the metadata
> of wheels on PyPI.
> >
> > It allows you to find out what projects a wheel depends on, what other
> projects depend on a given project, what commands & other entry points a
> wheel defines, what files are in a wheel, etc.  You can even search for
> wheels containing a given module or file, or browse a list of all commands
> & other entry points defined by wheels.  There's also a basic API for
> getting wheel data as JSON: .
> >
> > I'm open to suggestions on what else to do with the data.  I'm also open
> to suggestions on how to make the interface look less sucky.
> >
> > -- John Wodder
> > --
> > Distutils-SIG mailing list -- distutils-sig@python.org
> > To unsubscribe send an email to distutils-sig-le...@python.org
> > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> > Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/IHJU6RAUBFIEEDVR46BDT76F7XOHQ7OM/
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/KEG3RM7TCGRUO3XKMFZIZHFSDTRBBAZN/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/SY3NYNS3QG2QDGRVKNJMTHNQDO5T2C62/

[Distutils] Re: setuptools configuration in pyproject.toml

[Daniel Holth]

Yes! bdist_wheel is supposed to be how you install packages built with
setuptools on platforms without setuptools, to give you time to write its
replacement.

On Tue, Sep 25, 2018, 15:39 Thomas Kluyver  wrote:

> On Tue, Sep 25, 2018, at 9:13 AM, Paul Moore wrote:
> > I personally don't see much advantage in the "dump everything in one
> > file" philosophy. so from a personal POV I'd prefer setuptools' config
> > to remain in setup.cfg, but if they have reasons for wanting to move,
> > the PEP allows them that option.
>
> I've come across quite a few people who want to avoid 'clutter' in the top
> level of the repository. By the time you've got packaging config, test
> config, CI config, editor config and so on, I can see where they're coming
> from. I think all this mass of different files can also be confusing to
> newcomers. So I try to avoid creating extra files where it's easy to do so.
>
> But in the case of setuptools, I wouldn't push for it to use
> pyproject.toml. It will have to continue supporting setup.py and setup.cfg
> forever, so all you could do is add yet another place that metadata might
> live, making it more complicated to understand.
>
> Of course, I'm biased, because if setuptools uses pyproject.toml, flit
> looks kind of redundant. ;-)
>
> Thomas
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/N6SW5IMDROORV7ZZZ3GBMQQAYODOSFAV/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/W6GRFXA43LZXYMGLPAT3KIMHS5XKJZMJ/

[Distutils] Re: setuptools configuration in pyproject.toml

[Daniel Holth]

You could probably implement this outside of setuptools as an extension.
Clients would add a load-toml line to setup.py. Do build requirements work
yet?

One obstacle might be reconciling the all-strings nature of .cfg with typed
toml.

On Mon, Sep 24, 2018, 12:06 RonnyPfannschmidt <
opensou...@ronnypfannschmidt.de> wrote:

> I believe contributions in the directions would be welcome,
> for now having something like tool.setuptools.{metadata,options,command.*}
> might be interesting to grow and experiment with
>
> but whats really missing is a setuptools conributor with more than just
> thinly stretched time.
>
> -- Ronny
> Am 24.09.18 um 17:30 schrieb Bernat Gabor:
>
> I'm aware this might be a controversial subject, so let's have the initial
> discussion about it here first for full disclosure and see what people
> think about it. Should setuptools support pyproject.toml as configuration
> source or not (alternative to setup.cfg which it already does -
> https://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files)?
>
>
> The main benefit of having this would be to decrease configuration files
> and have build dependencies and other types of dependencies in one
> location. Furthermore many other packaging projects (flit, poetry) already
> do define their dependencies inside pyproject.toml; so would create one
> unified location where to look for such in the future.
>
> The counter-argument is that "a big part of pyproject.toml was keeping
> that file clean" and would furthermore increase the size of that file down
> the line.
>
> So what do people think? Should we encourage or discourage to have a
> single python project file?
>
> I'm personally supporting build/code quality tools supporting
> pyproject.toml as their main configuration file.
>
> Thanks
>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to 
> distutils-sig-leave@python.orghttps://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at 
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/C3JEBOCQEILLPXK4FDQPADCFO6WWW6JT/
>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/VBC35YTBWZYJCFPTRQRLSFBTV5FZV4OR/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/KV72IYDNA2M3WXHKWEV5RPX4UJ4OACA7/

[Distutils] Re: disable building wheel for a package

[Daniel Holth]

A corner case is where the package is importable because it is on
$PYTHONPATH and so the $virtual_env at runtime is different than the one at
install time. That is why it might be useful to store the data directory
per-package.
wheel.install.get_install_paths(package_name) shows where things would be
installed according to wheel. It comes from a call to distutils. pip has
its own implementation. On my machine data goes into $virtualenv in a
virtualenv and into /usr on system python.

On Fri, Sep 14, 2018 at 11:26 AM Paul Moore  wrote:

> On Fri, 14 Sep 2018 at 16:03, Daniel Holth  wrote:
> >
> > It can be hard to predict where data went at runtime.
>
> I don't think it's "hard to predict". I *do* think it's badly
> documented/not standardised. See my previous note - pip installs into
> the install_data location that setuptools/distutils chooses. Ideally:
>
> a) Setuptools and/or distutils would document where that is clearly
> and in an easy to find location.
> b) There should be a standard defining "schemes" like this so that the
> choices aren't implementation-defined (by pip, setuptools and
> distutils in some weird combination).
>
> Of course "there should be..." means "someone who cares enough needs
> to invest time in..." :-(
>
> >  Maybe we could record its path in .dist-info during the install. I
> think it may also not be clear how to put files into wheel's data directory
> from setup.py.
>
> Better would be to have a supported library that exposes the logic pip
> uses (or as I said above, the standard-defined logic) to determine
> such paths. See https://github.com/pypa/pip/issues/5191
>
> > If we added more categories it would allow the installer to override
> e.g. just the config directory rather than copying a Unix-like tree under
> data/ onto the install path.
>
> That's a reasonable but somewhat unrelated suggestion. I don't think
> it's needed for the OP's use case, but it may well be helpful for
> others.
>
> Paul
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/AS47NGBRJLEHFT5RANSCGLNSPZGVJTIW/

[Distutils] Re: disable building wheel for a package

[Daniel Holth]

It can be hard to predict where data went at runtime. Maybe we could record
its path in .dist-info during the install. I think it may also not be clear
how to put files into wheel's data directory from setup.py.

If we added more categories it would allow the installer to override e.g.
just the config directory rather than copying a Unix-like tree under data/
onto the install path.

On Fri, Sep 14, 2018, 10:46 Tzu-ping Chung  wrote:

> I’m wondering though, why? The wheel format already offers a data
> directory, couldn’t it be used for most of the things mentioned?
>
>
> On 14/9/2018, at 22:25, Daniel Holth  wrote:
>
> No one wants wheel to be able to install things outside of the virtualenv.
> What people have repeatedly asked for is the ability to install things
> somewhere besides $VIRTUAL_ENV/lib/python#.#/site-packages/, places like
> $VIRTUAL_ENV/etc/ for example.
> Should all the config files, documentation, data, man pages, licenses,
> images, go into $VIRTUAL_ENV/lib/python#.#/site-packages/? Or can we do a
> better job letting people put files in $VIRTUAL_ENV/xyz?
>
> On Fri, Sep 14, 2018 at 9:51 AM sashk  wrote:
>
>>
>>
>> 14.09.2018, 08:37, "Paul Moore" :
>>
>> On Fri, 14 Sep 2018 at 12:43, Jeroen Demeyer  wrote:
>>
>>  On 2018-09-14 12:55, Alex Grönholm wrote:
>>  > I'm curious: what data does it attempt to install and where? Have you
>>  > created a ticket for this somewhere?
>>
>>  The OP mentioned absolute paths. However, it really sounds like a bad
>>  idea to hard-code an absolute installation path. Let's consider it a
>>  feature that wheel doesn't support that.
>>
>>
>> The OP hasn't said, but I assumed that it was expecting to install
>> something in a "standard" Unix location like /etc.
>>
>> No, I'm not installing anything into standard unix locations. My package
>> is for internal use, so we had a luxury to write it specifically for use
>> with virtual environment.
>>
>> We need to install Jupyter kernels (and other files) into
>> $VIRTUAL_ENV/etc/jupyter and $VIRTUAL_ENV/share/jupyter paths. This was
>> done with the help of data_files, and works unless we build wheel, because
>> of use of absolute paths.
>>
>> Do I understand correctly, that when using relative paths in the
>> data_files and installing package into virtual environment,  installation
>> prefix is sys.prefix and it is the same as $VIRTUAL_ENV?
>>
>> Thanks.
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
>> Message archived at
>> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/AR65F7SMLRN54FKZ6EI6LKZZDCVFNKUX/
>>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/YSMYRZDS3DESU6YOCVSGP7ZMGQJUAK77/
>
>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/TKHX2FE47UQWG3H6SVANC2QNVOYUSNRV/

[Distutils] Re: Three clarification questions about PEP 425 and PyPy3

[Daniel Holth]

Package version resolution is a completely different and np-complete
problem. https://en.wikipedia.org/wiki/Boolean_satisfiability_problem

On Thu, Aug 30, 2018 at 2:51 PM Chris Jerdonek 
wrote:

> Broad question: is there a sense in which the same “resolver” logic could
> be used for both choosing the best tag, and choosing what versions of
> packages to install from abstract requirements, or are these fundamentally
> different paradigms that shouldn’t be thought of along the same lines?
>
> —Chris
>
> On Thu, Aug 30, 2018 at 11:42 AM Paul Moore  wrote:
>
>> On Thu, 30 Aug 2018 at 19:30, Donald Stufft  wrote:
>> > I find it helpful to generally not think of compatibility tags as hard
>> “this wheel is supported on this platform”, but more along the lines of “if
>> I built a wheel in the specified environment, I would get the same
>> results”. Those results may or may not work. So for example, if you have a
>> pure Python wheel that *only* works on Python 3.5+, but it produces the
>> same output when building the wheel on Python 3.3 (even if it doesn’t
>> ultimately work), then a “py3” wheel is the right tag to use. Arguably
>> you’d even use py2.py3 since you’d produce the same output (but in practice
>> most don’t since that’s extra work).
>>
>> I also quite like Daniel's description of the list of supported tags
>> as "try these, in this order - the first one you match is the best
>> chance you have of getting something that works". It may still not
>> work, but there's nothing that the package builder declared as being
>> any better.
>>
>> The thing I most dislike about the tag system is that if pip (or any
>> installer) misses out a particular tag combination, it gets totally
>> ignored. I sort of wish that there was some level of wildcard
>> matching, so that weird combinations get picked up at least
>> *somewhere* in the priority list. But I never managed to design a
>> workable scheme for doing that, so it may not even be possible.
>>
>> Paul
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
>> Message archived at
>> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/HU2K3CEZPJF7SWRMPGZ5NCULMZIB65NO/
>>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/ZZEHWOFEHILLMNXRHXOHKBB3FQZYQNUY/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/HPGE3SSX7BX7WOTWV3LAQX6UYVCILQAT/

[Distutils] Re: Three clarification questions about PEP 425 and PyPy3

[Daniel Holth]

As far as I know no one has released a compiler that turns f-strings into
not-f-strings. If something like that existed, then you could have two
wheels for the same software, one that had been passed through the compiler
to remove f-strings and gain compatibility with older Python. If you were
doing something like that then the pyXY tag becomes useful. Python-Requires
is more useful for the ordinary case.
The arch-only tags are an example of something we had to add because a
previously theoretical case became real.
It is a little silly but harmless to go all the way down to py30. Python
3.2 or 2.6 were perhaps the oldest Pythons bdist_wheel could ever run on
(2012).

On Thu, Aug 30, 2018 at 2:01 PM Nathaniel Smith  wrote:

> On Thu, Aug 30, 2018, 08:23 Nick Coghlan  wrote:
>
>> On Thu, 30 Aug 2018 at 09:58, Brett Cannon  wrote:
>> > On Wed, 29 Aug 2018 at 15:54 Nathaniel Smith  wrote:
>> >> This is a tricky decision. Any time a new Python comes out, some
>> >> existing wheels will continue to work fine, and some will be broken.
>> >> One goal is to avoid installing broken wheels. But, there's also
>> >> another consideration: if we're too conservative, then with every
>> >> release we create a bunch of make-work as projects have to re-roll old
>> >> wheels that would have worked fine, and some percentage of projects
>> >> won't do this (e.g. b/c they're abandoned), and we lose them forever.
>> >> Also, for the py3x tags in particular, if the wheel fails on py3(x+1),
>> >> then the sdist probably will too, so it's not like we have any useful
>> >> fallback.
>> >
>> > Right, but isn't that what the py3-none-any tag is meant to represent?
>> If someone doesn't use that tag then I would take that as there is some
>> version-specific stuff in that wheel.
>>
>> The problem is that "py3-none-any" doesn't specify a *minimum*
>> version, so if a project starts using a new feature like f-strings,
>> they *have* to declare "py36-...".
>>
>
> That's the theory, but I think these tags are useless in practice.
>
> If you're on py35 and pip sees a wheel with py36 as the tag, then it falls
> back to building from the sdist. For ABI-related tags this makes sense,
> because given an sdist and an appropriate compiler, you have a good chance
> of being able to generate wheels for some arbitrary platform, even one that
> the original authors never heard of. But... the python dialect tags are
> different. If your wheel uses f-strings, then your sdist probably does too,
> so all the tag does is move around the error to happen somewhere else.
>
> To avoid this, you have to put a Python-Requires header in your metadata.
> It's the only thing that works for sdists. And it also works for wheels.
> And it's strictly more expressive than the wheel tag version (you can write
> arbitrary restrictions like ">= 3.5.2, != 3.6.1". Note that 3.5.2 actually
> is a common minimum version for lots of async libraries, because it had a
> breaking change in the core async/await protocols).
>
> So I don't think there's any case where the pyXY tags are actually useful.
> You're always better off using Python-Requires.
>
> -n
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/K72BHTS6S3JX2U7BWYVDK2EZB3ZY3VI4/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/NY77HNUPMLLZIDZRBMHHEALSCHUXNPTE/

[Distutils] Re: Three clarification questions about PEP 425 and PyPy3

[Daniel Holth]

It's not an intuitive system. We have wheel tags to choose the best
alternative wheel or fall back to sdist. So py3-none-any is fine for
f-strings if no other candidate wheel (a list of all available wheels for
the same version number of a package) has been compiled to not require
f-strings. The tag only has to tell you which wheel is most likely to work.

No sdist or wheel is ever guaranteed to work, for any number of reasons.

On Aug 30, 2018 11:25, "Nick Coghlan"  wrote:

On Thu, 30 Aug 2018 at 09:58, Brett Cannon  wrote:
> On Wed, 29 Aug 2018 at 15:54 Nathaniel Smith  wrote:
>> This is a tricky decision. Any time a new Python comes out, some
>> existing wheels will continue to work fine, and some will be broken.
>> One goal is to avoid installing broken wheels. But, there's also
>> another consideration: if we're too conservative, then with every
>> release we create a bunch of make-work as projects have to re-roll old
>> wheels that would have worked fine, and some percentage of projects
>> won't do this (e.g. b/c they're abandoned), and we lose them forever.
>> Also, for the py3x tags in particular, if the wheel fails on py3(x+1),
>> then the sdist probably will too, so it's not like we have any useful
>> fallback.
>
> Right, but isn't that what the py3-none-any tag is meant to represent? If
someone doesn't use that tag then I would take that as there is some
version-specific stuff in that wheel.

The problem is that "py3-none-any" doesn't specify a *minimum*
version, so if a project starts using a new feature like f-strings,
they *have* to declare "py36-...".

So even though it isn't what PEP 425 actually says, in practice it's
turned out to be more useful to interpret the Python version tag as
being "version X.Y or later", and only interpret the ABI tag strictly.
That philosophy also makes the "abi3" ABI tag more coherent, since it
means that the "pyXY" part also specifies the minimum required ABI
version.

The marker for "exact version required" could then be to nominate a
specific Python implementation, rather than using the "py" prefix - so
a hypothetical wheel builder could use "cp36-none-any" for a
bytecode-only wheel archive that *only* ran on CPython 3.6, and
wouldn't be portable to other versions or implementations.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/FJGFRPQUMAZZ35TONE7JPTGK74PHFKFI/
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/NVREUFIWOY6A3ML3VM27KLOTHZ4EWF66/

[Distutils] Re: Three clarification questions about PEP 425 and PyPy3

[Daniel Holth]

That's right. No one writes 2to3 but for python 3.6 -> 3.7. The js people
do. If we got into it we could have wheel tags for that sort of thing. In
practice only a few classes of tags are used.

On Mon, Aug 27, 2018, 22:06 Nathaniel Smith  wrote:

> I think the answer to all of these questions is "well, no-one's ever
> really looked that closely".
>
> There's a theory behind the tags; they're supposed to be a reasonably
> expressive language for talking about Python dialect compatibility,
> Python C ABI compatibility, and platform ABI compatibility,
> respectively. But in practice so far only a small fixed set of tag
> combinations actually gets used, so there's plenty of room for weird
> stuff to accumulate in the corners where no-one looks.
>
> I've never been able to figure out a use case for the interpreter tags
> in the first field ("cp36", "pp3", etc). IIUC, the theory is that
> they're supposed to mean "the Python code in this package is not
> actually portable Python, but uses an interpreter-specific dialect of
> the language". (This is very different from the second field, where
> tags like "cp36m" tell you about the required C ABI -- that one is
> obviously super useful.) I guess if you had a package that like,
> absolutely depended on GC being refcount-based, you could use a cp3
> tag to indicate that, or if you had a pure-Python, python 2 package,
> that required 'dict' to be ordered, maybe that's 'pp2-none-any'? But
> this never seems to actually happen in practice. It seems like an idea
> that sounded plausible early on, and then never got fleshed out or
> revisited.
>
> The distutils folks have never sat down to seriously think about
> non-CPython implementations, where the language version and the
> implementation version are separate things.
>
> The pypy folks have never sat down to seriously think about API/ABI
> stability. Generally at the Python dialect level they try to match a
> given version of (C)Python, and at the ABI level every new release is
> a new ABI.
>
> My guess is you shouldn't spend too much effort on trying to slavishly
> reproduce pip's logic, and that if you wanted to go clean up pip's
> logic (and maybe extract it into a reusable library?) then the devs
> would be perfectly happy that someone was doing it...
>
> -n
>
> On Mon, Aug 27, 2018 at 6:28 PM, Brett Cannon  wrote:
> > And to help in getting a reply, here is the trimmed-down results for
> CPython
> > 3.7 to compare against:
> >
> > [('cp37', 'cp37m', 'macosx_10_13_x86_64'),
> > …
> >  ('cp37', 'abi3', 'macosx_10_13_x86_64'),
> > …
> >   ('cp37', 'none', 'macosx_10_13_x86_64'),
> > …
> >  ('cp36', 'abi3', 'macosx_10_13_x86_64'),
> > …
> >   ('cp35', 'abi3', 'macosx_10_13_x86_64'),
> > …
> >   ('cp34', 'abi3', 'macosx_10_13_x86_64'),
> > …
> >  ('cp33', 'abi3', 'macosx_10_13_x86_64'),
> > …
> >  ('cp32', 'abi3', 'macosx_10_13_x86_64'),
> > …
> >  ('py3', 'none', 'macosx_10_13_x86_64'),
> > …
> >  ('cp37', 'none', 'any'),
> >  ('cp3', 'none', 'any'),
> >  ('py37', 'none', 'any'),
> >  ('py3', 'none', 'any'),
> >  ('py36', 'none', 'any'),
> >  ('py35', 'none', 'any'),
> >  ('py34', 'none', 'any'),
> >  ('py33', 'none', 'any'),
> >  ('py32', 'none', 'any'),
> >  ('py31', 'none', 'any'),
> >  ('py30', 'none', 'any')]
> >
> > So, it re-iterate the questions:
> >
> > What is ('pp3', 'none', 'any') supposed to represent for PyPy3? Since the
> > version of the interpreter is PyPy3 6.0 the lack of major version number
> > seems like a bug more than a purposeful interpreter version (and there's
> > only a single project -- cliquet -- that has a wheel that's compatible
> with
> > that tag triple and it's not even for their latest release).
> > Why does CPython have (*, 'none', 'any') from the version of the
> interpreter
> > down to Python 3.0 plus generically Python 3 while PyPy3 only gets
> generic
> > Python 3?
> > Why isn't (*, 'none', platform) listed from Python 3.7 to 3.0 for either
> > CPython or PyPy3? I understand not iterating through all versions when an
> > ABI is involved (without knowing exactly which versions are compatible
> like
> > abi3), but this triple seems safe to iterate through as a fallback just
> as
> > much as (*, 'none', 'any'). Maybe because it's too ambiguous to know how
> > important such a fallback would be between e.g. ('py36', 'none',
> > 'macosx_10_13_x86_64') and ('py37', 'none', 'any'), and so why bother
> when
> > the older version triples are there just for a safety net to have at
> least
> > some chance of a match?
> > I still think ('py360', 'none', 'any') is a bug. ;)
> >
> >
> > P.S.: The ('py3', 'none', 'macosx_10_13_x86_64') triple being between
> e.g.
> > ('pp360', 'none', 'macosx_10_13_x86_64') and ('pp360', 'none', 'any') is
> > really messing with my head and making the code to generate supported
> > triples a bit less elegant. ;)
> >
> > On Sat, 25 Aug 2018 at 

[Distutils] Re: Accessing extras during install command

[Daniel Holth]

It was never stored. It would probably not be hard to check whether a
particular extras dependencies were installed by using pkg_resources?

On Tue, Aug 21, 2018, 11:01 Joni Orponen  wrote:

> Hello,
>
> I'd like to run different external programs during the install command
> which are provided by (potentially) installed extras, but I cannot find
> where the currently selected extras are stored.
>
> The only thing I quickly found by going through the distutils and
> setuptools codebases is an old TODO from easy_install indicating it might
> not have been implemented yet:
>
>
> https://github.com/pypa/setuptools/blob/afba2d89c4436451ad4d7e19228584246a9aaadc/setuptools/command/easy_install.py#L1106
>
> I can hack this by try excepting on ImportError, but is there a more
> elegant (and intended) way?
>
> --
> Joni Orponen
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/Z3LLXCCWWFHNRIHQTVFSHSYEJYNXSVXX/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/OOEAJPHQTUHKQ6JNNNEYV6V4IDBM2WOO/

[Distutils]Re: Introducing XAR - SquashFS based mountable executables - Calling OS/Distro Maintainers

[Daniel Holth]

This is very cool. I'm glad to see attention towards packaging useful
applications and not just libraries for web development in a virtualenv,
and the limitations of zip compression are felt by our Python applications
comprised of many separately compressed files.
Is bdist_xar calling out to pip to install all of a package's dependencies
(or all that are not already installed) into the xar as part of the build?

On Tue, Jul 17, 2018 at 10:48 PM Barry Warsaw  wrote:

>
> >> * Is there any practical operational or performance limits on the
> number of mounted XARs you can have?  E.g. what's the impact of deploying
> say a few thousand XARs on any particular machine (generally, of Linux and
> macOS - I'm not as concerned about Windows :)
> > We have tiers (think of that as a collection of servers for a particular
> service) that have 100s mounted at times. Have not hit 1000s - But have not
> seen any huge problems with 100s. What do your servers do that would need
> 1000s of XARs running at once? Seems like that could be optimized.
>
> I’m not thinking about just our server deployed applications, but our
> command line tools too.  Combined, that’s roughly in the ballpark of 1000
> or so Python applications (more if we count other languages).
>
> >
> >>
> >> * Do you just put the XAR mountpoint's bin on everyone's $PATH or do
> you symlink those bins into a standard $PATH location?
> > Standard PATH location via RPM installs.
>
> Cool, thanks!
>
> -Barry
>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/MKJYW7PI42BW5E2UYYP6Y4YS366KSJTP/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/WAG3I6AZOXF6ZSVBUHEW6ACIED46HB2R/

[Distutils]Re: The Wheel specification and compatibility tags on Windows

[Daniel Holth]

According to a link in this chain, virtualenv lately copies python3.dll (+
python3.7 e.g.) on Windows. So hopefully the extension links with that dll.

On Tue, Jul 17, 2018, 15:51 Nathaniel Smith  wrote:

> The promise behind the limited ABI is exactly that if your extension works
> on 3.x, it will also work on 3.y, for y >= x.
>
> One thing to watch out for: normally extension modules on Linux and MacOS
> don't try to link to libpython. Instead they trust that someone else will
> make sure they only get loaded into a compatible python interpreter, and
> that all the symbols they need from python will be injected by the host
> process.
>
> On Windows, the way the dynamic loader works, you can't do this: every
> extension module has to explicitly say "I'm getting PyNumber_Add from the
> dll named: somethingsomething.dll"
>
> But traditionally, version X.Y of python includes a pythonXY.dll, so
> there's no consistent name across releases. So even if your library uses
> only the limited ABI and all of your imports could just as well come from
> python36.dll or python37.dll... you need some way to express that, and only
> Windows has this problem.
>
> I feel bad sending this without doing my own research, but I don't have
> access to a Windows box right now. Does anyone know if this problem has
> been solved? Is it still true that Windows python dlls always include the
> python version in their name?
>
> -n
>
> On Tue, Jul 17, 2018, 09:38 Paul Moore  wrote:
>
>> On 17 July 2018 at 16:59, Cosimo Lupo  wrote:
>> > I would like to revive this 5 year old thread and see if we can stir
>> things
>> > up a bit.
>> >
>> > Basically the problem is that, in the current state of the PEPs and
>> build
>> > tools, it is still not possible to build and distribute a Windows wheel
>> that
>> > includes an extension module compiled with Py_LIMITED_API.
>> > Setuptools can successfully build such extension module on Windows (with
>> > `.pyd` file extension and no extra specifiers in the module filename),
>> and
>> > these seems to work at least on CPython 3.5 and above. However the
>> > `--py-limited-api cpXX` option of bdist_wheel command fails on Windows
>> > because it attempts to use the `abi3` tag but the latter is not in the
>> list
>> > of compatible tags for that platform.
>> > One can work around this by creating a wheel with `none` as the abi
>> tag, and
>> > `cp35.cp36.cp37` as the python implementation tag but this feels a bit
>> > hackish.
>> >
>> > Here are some unresolved questions from the old distutils-sig thread,
>> > quoting Paul Moore:
>> >
>> >> 2. How should tools determine which ABIs a given Python supports?
>> >> (This is the get_supported function in wheel and distlib). The "Use"
>> >> section of the PEP (http://www.python.org/dev/peps/pep-0425/#id1)
>> >> gives a Linux-based example, but nothing normative and nothing that is
>> >> understandable to a Windows user.
>> >
>> > And from Vinay Sajip
>> >
>> >> For Windows, we (eventually) need some low-level sysconfig-supported
>> way
>> >> to get the ABI information in an analogous way to how it happens on
>> POSIX:
>> >> and
>> >> because that's not currently there, distlib doesn't provide any ABI
>> >> information
>> >> on Windows other than "none".
>> >
>> > Other related links:
>> > https://github.com/pypa/pip/issues/4445
>> >
>> https://mail.python.org/pipermail/distutils-sig/2018-January/031856.html
>> >
>> > So.. what needs to be done here to allow distributing/installing Windows
>> > wheels with Py_LIMITED_API support?
>>
>> IMO, the question I posed back then remains key. Vinay's response is
>> fair, but I don't think that waiting for core Python to provide
>> something via sysconfig is practical (it's not happened yet, so why
>> would we expect things to change?). So I think the next step is
>> probably for someone to propose an algorithm that can be used.
>> Actually, what I'd like to see is a full end to end proposal of the
>> process someone would use to build and install a limited-ABI
>> extension. That would probably tease out a number of issues.
>>
>> I imagine the steps would be something like this:
>>
>> 1. Create an extension. Presumably you'd need to #define
>> PY_LIMITED_ABI in the source.
>> 2. Build a wheel from it - how would you do that? I gather it's
>> possible to do this with plain setuptools - would it be necessary to
>> do this with setuptools/bdist_wheel, or should there be a way to
>> request a limited ABI build via pip? If we do want to be able to
>> request this from a build tools like pip, do we need something in PEP
>> 517? Are we only looking at the prebuilt wheel case, or do we need to
>> support building from source?
>> 3. What tags would the built wheel have?
>> 4. Install that wheel - `pip install xxx`. Pip needs to be able to
>> enumerate the full list of valid tags here (cp37-abi3, cp3-abi3, ...)
>> There are also questions like - if there's a limited ABI wheel and a
>> full ABI (version specific) wheel, which 

[Distutils]Re: The Wheel specification and compatibility tags on Windows

[Daniel Holth]

So it appears the "supports abi3" check on Windows is simply python 3.5+?
There is also the question of which visual studio CRT is in use, which
could break compat apart from the Python ABI.

This is the change to add abi3 to wheel including an extension using it.
Must pass a define, which should probably be and argument to Extension (),
as well as the flag.
https://bitbucket.org/pypa/wheel/pull-requests/69/add-py-limited-api-flag-to-control-abi3/diff

bdist_wheel could be more clever and enumerate all the extensions looking
for the py_limited_api flag.

On Tue, Jul 17, 2018, 12:58 Phil Thompson 
wrote:

> On 17 Jul 2018, at 5:37 pm, Paul Moore  wrote:
> >
> > On 17 July 2018 at 16:59, Cosimo Lupo  wrote:
> >> I would like to revive this 5 year old thread and see if we can stir
> things
> >> up a bit.
> >>
> >> Basically the problem is that, in the current state of the PEPs and
> build
> >> tools, it is still not possible to build and distribute a Windows wheel
> that
> >> includes an extension module compiled with Py_LIMITED_API.
> >> Setuptools can successfully build such extension module on Windows (with
> >> `.pyd` file extension and no extra specifiers in the module filename),
> and
> >> these seems to work at least on CPython 3.5 and above. However the
> >> `--py-limited-api cpXX` option of bdist_wheel command fails on Windows
> >> because it attempts to use the `abi3` tag but the latter is not in the
> list
> >> of compatible tags for that platform.
> >> One can work around this by creating a wheel with `none` as the abi
> tag, and
> >> `cp35.cp36.cp37` as the python implementation tag but this feels a bit
> >> hackish.
> >>
> >> Here are some unresolved questions from the old distutils-sig thread,
> >> quoting Paul Moore:
> >>
> >>> 2. How should tools determine which ABIs a given Python supports?
> >>> (This is the get_supported function in wheel and distlib). The "Use"
> >>> section of the PEP (http://www.python.org/dev/peps/pep-0425/#id1)
> >>> gives a Linux-based example, but nothing normative and nothing that is
> >>> understandable to a Windows user.
> >>
> >> And from Vinay Sajip
> >>
> >>> For Windows, we (eventually) need some low-level sysconfig-supported
> way
> >>> to get the ABI information in an analogous way to how it happens on
> POSIX:
> >>> and
> >>> because that's not currently there, distlib doesn't provide any ABI
> >>> information
> >>> on Windows other than "none".
> >>
> >> Other related links:
> >> https://github.com/pypa/pip/issues/4445
> >>
> https://mail.python.org/pipermail/distutils-sig/2018-January/031856.html
> >>
> >> So.. what needs to be done here to allow distributing/installing Windows
> >> wheels with Py_LIMITED_API support?
> >
> > IMO, the question I posed back then remains key. Vinay's response is
> > fair, but I don't think that waiting for core Python to provide
> > something via sysconfig is practical (it's not happened yet, so why
> > would we expect things to change?). So I think the next step is
> > probably for someone to propose an algorithm that can be used.
> > Actually, what I'd like to see is a full end to end proposal of the
> > process someone would use to build and install a limited-ABI
> > extension. That would probably tease out a number of issues.
> >
> > I imagine the steps would be something like this:
> >
> > 1. Create an extension. Presumably you'd need to #define
> > PY_LIMITED_ABI in the source.
> > 2. Build a wheel from it - how would you do that? I gather it's
> > possible to do this with plain setuptools - would it be necessary to
> > do this with setuptools/bdist_wheel, or should there be a way to
> > request a limited ABI build via pip? If we do want to be able to
> > request this from a build tools like pip, do we need something in PEP
> > 517? Are we only looking at the prebuilt wheel case, or do we need to
> > support building from source?
> > 3. What tags would the built wheel have?
> > 4. Install that wheel - `pip install xxx`. Pip needs to be able to
> > enumerate the full list of valid tags here (cp37-abi3, cp3-abi3, ...)
> > There are also questions like - if there's a limited ABI wheel and a
> > full ABI (version specific) wheel, which takes precedence?
> >
> > I don't honestly know how well the limited ABI actually achieves its
> > goals - is "cp3-abi3-win_x86_64" a realistic tag to apply? Can limited
> > ABI wheels really be used on any version of Python 3? That's a
> > question for python-dev, rather than distutils-sig, but if we take the
> > position that this is what's promised, and it later turns out not to
> > be true, we've got a lot of wheel renaming to do when Python 3.10
> > comes out and it doesn't support the same limited ABI as 3.x for x <
> > 10...
> >
> > Also, does the limited ABI work on other platforms? If it does, we
> > should ensure that the Windows support works the same. If it doesn't,
> > do we want a Windows-only solution (why is that OK?) or should we
> > extend to (say) manylinux or OSX (at 

Re: [Distutils] Removing wheel signing features from the wheel library

[Daniel Holth]

The feature was a building block that was intended to be used in much the
same way that SHA package hashes are used, providing similar security to
the ssh-style TOFU model, but less security than other imaginable public
key systems. The idea was that it could provide more security in practice
because the signatures could exist and be present with the archive, unlike
gpg which provides loads of security in theory only. Unfortunately wheel
signatures were never built up. I don't think anyone was tricked into
believing the primitive provided security on its own.

On Thu, Mar 22, 2018 at 2:21 PM Nathaniel Smith  wrote:

> Even if no maintenance were required, it's still a feature that promises
> to provide security but doesn't. This kind of feature has negative value.
>
> I'd also suggest adding a small note to the PEP documenting that the
> signing feature didn't work out, and maybe linking to Donald's package
> signing blog post. I know updating PEPs isn't the most common thing, but
> it's the main documentation of the wheel format and it'll save confusion
> later.
>
> On Mar 22, 2018 10:57 AM, "Wes Turner"  wrote:
>
>> What maintenance is required?
>>
>> Here's a link to the previous discussion of this issue:
>>
>> "Remove or deprecate wheel-signing features"
>> https://github.com/pypa/wheel/issues/196
>>
>> What has changed? There is still no method for specifying a keyring;
>> whereas with GPG, all keys in the ring are trusted.
>>
>> On Thursday, March 22, 2018, Nick Coghlan  wrote:
>>
>>> On 22 March 2018 at 22:35,  wrote:
>>>
 I am not changing the format of RECORD, I'm simply removing the
 cryptographic signing and verifying functionality, just the way you
 described. Hash checking will stay. As we agreed earlier, those
 features could be deprecated or removed from the PEP entirely.

>>>
>>> Cool, that's what I thought you meant, but I figured I should double
>>> check since our discussion was a while ago now :)
>>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 566 - Package metadata version 2.1

[Daniel Holth]

I accept PEP 566. Thank you for doing this work.

Daniel

On Sun, Feb 18, 2018 at 5:05 PM Dustin Ingram <di@di.codes> wrote:

> I've updated the PEP to include the message body as Description.
>
>   https://www.python.org/dev/peps/pep-0566/
>
> I don't think there are any other outstanding issues, so this should
> be ready for Daniel's review.
>
> Thanks,
> D.
>
> On Thu, Feb 15, 2018 at 2:09 PM, Trishank Kuppusamy
> <trishank.kuppus...@datadoghq.com> wrote:
> > On Thu, Feb 15, 2018 at 2:19 PM, Daniel Holth <dho...@gmail.com> wrote:
> >>
> >>
> >> Hashes do not belong in this PEP, which is intended to do just a little
> >> more than document the status quo. The document does provide for future
> >> enhancements to the spec without using the PEP process.
> >>
> >> Personally I am not a fan of putting concrete requirements or hashes of
> >> specific archives at this level.
> >
> >
> > Fair enough, just throwing the idea out there!
> >
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 566 - Package metadata version 2.1

[Daniel Holth]

Doing fine.

Hashes do not belong in this PEP, which is intended to do just a little
more than document the status quo. The document does provide for future
enhancements to the spec without using the PEP process.

Personally I am not a fan of putting concrete requirements or hashes of
specific archives at this level.

On Thu, Feb 15, 2018 at 1:44 PM Trishank Kuppusamy <
trishank.kuppus...@datadoghq.com> wrote:

> Hi Daniel, long time no speak, how you doing? :)
>
> Maybe slightly off-topic, but I wonder if it the PEP allows for specifies
> hashes of external requirements? Given a good copy of hashes, this would be
> useful to survive a compromise of any package index.
>
> Does this make sense? Please let me know if you have questions, and thanks!
>
> On Thu, Feb 15, 2018 at 1:31 PM, Daniel Holth <dho...@gmail.com> wrote:
>
>> I agree but have simply not had time. Edit it to add something like
>> "Instead of a description header, the description may be provided in the
>> message body, e.g. after a completely blank line to end the headers,
>> followed by the long description with no indentation or other special
>> formatting needed". Write something about putting the body back into a
>> description key in the json version. Just delete the example parsing code
>> which doesn't parse message bodies. I don't recall any other issues that
>> would prevent approval.
>>
>> On Thu, Feb 15, 2018 at 11:14 AM Thomas Kluyver <tho...@kluyver.me.uk>
>> wrote:
>>
>>> I'd like to once again prod this PEP towards completion:
>>>
>>> https://www.python.org/dev/peps/pep-0566/
>>>
>>> The version numbering question has been decided in favour of calling it
>>> 2.1.
>>>
>>> The remaining question I'm aware of is whether to make the body text (in
>>> the email format of the metadata file) officially represent the package
>>> long description. I'm in favour of doing so: at least twine and flit
>>> already use this for metadata in wheels.
>>>
>>> Thomas
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 566 - Package metadata version 2.1

[Daniel Holth]

I agree but have simply not had time. Edit it to add something like
"Instead of a description header, the description may be provided in the
message body, e.g. after a completely blank line to end the headers,
followed by the long description with no indentation or other special
formatting needed". Write something about putting the body back into a
description key in the json version. Just delete the example parsing code
which doesn't parse message bodies. I don't recall any other issues that
would prevent approval.

On Thu, Feb 15, 2018 at 11:14 AM Thomas Kluyver 
wrote:

> I'd like to once again prod this PEP towards completion:
>
> https://www.python.org/dev/peps/pep-0566/
>
> The version numbering question has been decided in favour of calling it
> 2.1.
>
> The remaining question I'm aware of is whether to make the body text (in
> the email format of the metadata file) officially represent the package
> long description. I'm in favour of doing so: at least twine and flit
> already use this for metadata in wheels.
>
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 566 - metadata 1.3 changes

[Daniel Holth]

>1.2 and not 2.0 is correct.

I took a look at pkg_resources. It doesn't read Metadata-Version at all. It
only cares about Version, and in wheels Requires-Dist and Provides-Extra.
Everything else is ignored. So PEP 566 won't break anything there as long
as someone checks that pkg_resources can handle the optional parenthesis
which seems likely.

On Thu, Jan 18, 2018 at 11:37 AM Dustin Ingram  wrote:

> > Given that, I think it would be reasonable to finally Withdraw PEP 426
> > (rather than continuing to defer it), and have PEP 566 define metadata
> > version 2.1, so that it's unambiguously the latest metadata version.
>
> I'm amenable to any version number that is > 1.2 and not 2.0.
>
> D.
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 566 - metadata 1.3 changes

[Daniel Holth]

5

On Tue, Jan 16, 2018 at 12:08 PM Alex Grönholm 
wrote:

> Whichever we choose, the metadata version should match the PEP version,
> which it currently does not.
>
> Nathaniel Smith kirjoitti 16.01.2018 klo 18:58:
>
> On Jan 12, 2018 8:00 AM, "Alex Grönholm"  wrote:
>
> On the same note, wheel currently writes "2.0" as its metadata version.
> Shouldn't this be changed into 1.3 (along with ditching metadata.json)?
>
>
> Should wheel change to emit 1.3, or should the PEP change to become 2.0? I
> know there were great hopes for "metadata 2.0", but given that there are
> bazillions of packages out there with a metadata version of 2.0, we're
> never going to be able to meaningfully use that version number for anything
> else, and it's confusing if when reading package metadata the ordering is
> 1.2 < 2.0 == 1.3 < 1.4. So maybe we should declare that this update is 2.0
> or 2.1, the next one will be 2.1 or 2.2, etc., and if anyone asks why the
> major version bump, well, it's hardly the strangest thing we've done for
> compatibility :-). (And in the unlikely event that PEP 426 lurches back to
> life, we can make it 3.0.)
>
> -n
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 566 - metadata 1.3 changes

[Daniel Holth]

This is the old text.


Describing the distribution
===

The distribution metadata should include a longer description of the
distribution that may run to several paragraphs. Software that deals
with metadata should not assume any maximum size for the description.

The recommended location for the description is in the metadata payload,
separated from the header fields by at least one completely blank line
(that is, two successive line separators with no other characters
between them, not even whitespace).

Alternatively, the description may be provided in the `Description`__
metadata header field. Providing both a ``Description`` field and a
payload is an error.

__ `Description (optional, deprecated)`_

The distribution description can be written using reStructuredText
markup [1]_.  For programs that work with the metadata, supporting
markup is optional; programs may also display the contents of the
field as plain text without any special formatting.  This means that
authors should be conservative in the markup they use.


On Tue, Jan 16, 2018 at 6:44 AM Thomas Kluyver <tho...@kluyver.me.uk> wrote:

> Allow me to prod this topic again. ;-)
>
> I'm happy with PEP 566 as it stands.
>
> Do we want to specify 'email body is long description' in this PEP? It
> appears to have at least some real world support, but I'm not familiar
> enough with the email metadata format to write a proper description of it.
>
> Thanks,
> Thomas
>
> On Fri, Jan 12, 2018, at 4:02 PM, Daniel Holth wrote:
>
> Yes, after the PEP is prep'd.
>
> On Fri, Jan 12, 2018 at 11:00 AM Alex Grönholm <alex.gronh...@nextday.fi>
> wrote:
>
> On the same note, wheel currently writes "2.0" as its metadata version.
> Shouldn't this be changed into 1.3 (along with ditching metadata.json)?
>
>
> Thomas Kluyver kirjoitti 12.01.2018 klo 17:26:
> > On Wed, Jan 10, 2018, at 11:42 PM, Nick Coghlan wrote:
> >> On 11 January 2018 at 00:54, Daniel Holth <dho...@gmail.com> wrote:
> >>> AFAICT the only missing feature from old-Metadata-2.0 is "description
> as
> >>> message body", which places readable description text after the
> key/value
> >>> pairs.
> >> Do pip/PyPI/et al currently support that?
> > It looks like twine supports it, at least for wheels:
> >
> https://github.com/pypa/twine/blob/f74eae5506300387572c65c9dbfe240d927788c2/twine/wheel.py#L99
> >
> > I don't think pip needs to support it (does pip do anything with
> descriptions?). I haven't looked at PyPI's code, but I'd guess it uses the
> metadata sent with the upload by tools like twine and flit.
> >
> > Thomas
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
> *___*
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 566 - metadata 1.3 changes

[Daniel Holth]

Yes, after the PEP is prep'd.

On Fri, Jan 12, 2018 at 11:00 AM Alex Grönholm <alex.gronh...@nextday.fi>
wrote:

> On the same note, wheel currently writes "2.0" as its metadata version.
> Shouldn't this be changed into 1.3 (along with ditching metadata.json)?
>
>
> Thomas Kluyver kirjoitti 12.01.2018 klo 17:26:
> > On Wed, Jan 10, 2018, at 11:42 PM, Nick Coghlan wrote:
> >> On 11 January 2018 at 00:54, Daniel Holth <dho...@gmail.com> wrote:
> >>> AFAICT the only missing feature from old-Metadata-2.0 is "description
> as
> >>> message body", which places readable description text after the
> key/value
> >>> pairs.
> >> Do pip/PyPI/et al currently support that?
> > It looks like twine supports it, at least for wheels:
> >
> https://github.com/pypa/twine/blob/f74eae5506300387572c65c9dbfe240d927788c2/twine/wheel.py#L99
> >
> > I don't think pip needs to support it (does pip do anything with
> descriptions?). I haven't looked at PyPI's code, but I'd guess it uses the
> metadata sent with the upload by tools like twine and flit.
> >
> > Thomas
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 566 - metadata 1.3 changes

[Daniel Holth]

AFAICT the only missing feature from old-Metadata-2.0 is "description as
message body", which places readable description text after the key/value
pairs.

On Wed, Jan 10, 2018 at 8:45 AM Thomas Kluyver  wrote:

> I hope everyone has had a good break. :-)
>
> I'd like to see PEP 566 move forwards. From the last thread, I think
> people were mostly happy with it as stands, but there were some proposals
> to introduce further metadata changes. I'd suggest that we finalise the PEP
> as it currently stands, and save up other changes for a future metadata
> revision.
>
> One change I would like to the current text is to make  more explicit that
> the format of several fields allowing version specifiers (Requires-Dist,
> Provides-Dist, Obsoletes-Dist, Requires-External) has changed, as PEP 508
> removes the need for parentheses around the version specifier.
>
> The section on version specifiers currently refers to PEP 440, which
> doesn't mention the parentheses at all, and says they are otherwise
> unchanged from PEP 345, which says parentheses are required. I would like
> to add some text to that section, such as:
>
> "Following PEP 508, version specifiers no longer need to be surrounded by
> parentheses in the fields Requires-Dist, Provides-Dist, Obsoletes-Dist or
> Requires-External, so e.g. `requests >= 2.8.1` is now a valid value. The
> recommended format is without parentheses, but tools parsing metadata
> should also be able to handle version specifiers in parentheses."
>
> Thanks,
> Thomas
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] RFC: PEP 566 - Metadata for Python Software Packages 1.3

[Daniel Holth]

I don't know why the parentheses were included in the older pep. They are
widely deployed. We probably can get rid of them or make them optional in a
practical parser.

On Sat, Dec 9, 2017, 02:03 Nick Coghlan  wrote:

> On 9 December 2017 at 02:42, Thomas Kluyver  wrote:
> > Dustin asked me to bring this issue to this thread:
> >
> > Metadata version 1.2 (PEP 345) says that version specifiers within a
> > Requires-Dist field should go in parentheses: "zope.interface (>3.5.0)".
> > The metadata spec on PyPUG repeats this.
> >
> > However, PEP 508 says that the parentheses are not needed, and tools
> > writing dependency specifications should not create them. Its
> > recommended format is therefore "zope.interface >3.5.0".
> >
> > Should the metadata 1.3 PEP note that this has changed? Or do we only
> > need to update the metadata spec on PyPUG?
>
> The new PEP already delegates to PEP 508 for the version specifier
> format, so I think that can just be updated in PyPUG.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Caching entry points - performance

[Daniel Holth]

Neat

On Thu, Nov 30, 2017 at 1:25 PM Thomas Kluyver  wrote:

> To pick up the caching discussion again, I've started to experiment with
> a couple of different caching techniques.
>
> The headline results: a cold-start scan of entry points goes from about
> 4.5 seconds with no caching, to 0.45 seconds with caching. There's only
> a small difference (for me) between having a cache file per sys.path
> directory and a single per-user cache file. I've posted lots more
> details here: https://github.com/takluyver/entrypoints/issues/16
>
> I keep my sys.path quite short, and the difference between the two
> approaches presumably increases the more sys.path entries you have. If
> anyone has time to test the different techniques, I'd be interested to
> know what your results look like. I've put (Linux) instructions on the
> same issue.
>
> If you'd like to try with a different format or caching mechanism, feel
> free to add some scripts and see how they compare.
>
> I haven't yet tried to integrate either technique into the entrypoints
> module.
>
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Daniel Holth]

I agree. The "malware" problem is really a "how do I understand which hooks
run in each environment" problem. The hooks could slow down or confuse,
frustrate people in ways that were unrelated to any malicious intent.

The caching could just be a more efficient, lossless representation of the
*.dist/egg-info data model.
Would something as simple as a file per sys.path with the 'last modified by
installer' date be helpful? You could check those to determine whether your
cache was out of date.

Another option would be to try to investigate whether the per-sys-path
operations that 'import x' has to do anyway can be cached and shared with
pkg_resources?

On Thu, Oct 26, 2017 at 8:21 AM Nick Coghlan  wrote:

> On 26 October 2017 at 18:33, Thomas Kluyver  wrote:
>
>> Nathaniel raises the point that it may be easier to convince other
>> package managers to regenerate an entry points cache than to call arbitrary
>> Python hooks on install.
>>
>
> At least for RPM, we have file triggers now, whereby system packages can
> register a hook to say "Any time another package touches a file under  of interest> I want to know about it".
>
> That means the exact semantics of any RPM integration would likely end up
> just living in a file trigger, so it wouldn't matter to much whether that
> trigger was "refresh these predefined caches" or "run any installed hooks
> based on the defined Python level metadata".
>
> However, I expect it would be much easier to define a "optionally export
> data for caching in a more efficient key value store" API than it would be
> to define an API for arbitrary pre-/post- [un]install hooks. In particular,
> a caching API is much easier to *repair*, since the "source of truth"
> remains the installation DB itself - the cache is just to speed up runtime
> lookups.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Daniel Holth]

I like the idea of lifecycle hooks but I worry about the malware problem;
would there need to be a blacklist / whitelist / disable system?
(ignore-scripts=true is now a recommended part of anyone's npm
configuration) That is why we have avoided any kind of (package specific)
hooks to wheel. However hooks would be a very elegant way to avoid worrying
about core pip functionality since it wouldn't be core functionality.

On Fri, Oct 20, 2017 at 4:41 PM Nathaniel Smith  wrote:

> On Oct 19, 2017 11:10, "Donald Stufft"  wrote:
>
>
> EXCEPT, for the fact that with the desire to cache things, it would be
> beneficial to “hook” into the lifecycle of a package install. However I
> know that there are other plugin systems out there that would like to also
> be able to do that (Twisted Plugins come to mind) and that I think outside
> of plugin systems, such a mechanism is likely to be useful in general for
> other cases.
>
> So heres a different idea that is a bit more ambitious but that I think is
> a better overall idea. Let entrypoints be a setuptools thing, and lets
> define some key lifecycle hooks during the installation of a package and
> some mechanism in the metadata to let other tools subscribe to those hooks.
> Then  a caching layer could be written for setuptools entrypoints to make
> that faster without requiring standardization, but also a whole new, better
> plugin system could to, Twisted plugins could benefit, etc [1].
>
>
> In this hypothetical system, how do installers like pip find the list of
> hooks to call? By looking up an entrypoint? (Sorry if this was discussed
> downthread; I didn't see it but I admit I only skimmed.)
>
> -n
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Daniel Holth]

I prefer a single more generic mechanism that packaging happens to use
instead of making special mechanisms for scripts or other callables that
packaging might some day be interested in. One API, I can type
pkg_resources.iter_entry_points('console_scripts') to enumerate the scripts
and perhaps invoke them without the wrappers, or I can look other plugins.

+1 on simply documenting what we have first.

How long does pkg_resources take to import for you folks?

On Thu, Oct 19, 2017 at 2:10 PM Donald Stufft  wrote:

>
>
> > On Oct 19, 2017, at 12:14 PM, Thomas Kluyver 
> wrote:
> >
> > On Thu, Oct 19, 2017, at 04:10 PM, Donald Stufft wrote:
> >> I’m in favor, although one question I guess is whether it should be a a
> >> PEP or an ad hoc spec. Given (2) it should *probably* be a a PEP (since
> >> without (2), its just another file in the .dist-info directory and that
> >> doesn’t actually need standardized at all). I don’t think that this will
> >> be a very controversial PEP though, and should be pretty easy.
> >
> > I have opened a PR to document what is already there, without adding any
> > new features. I think this is worth doing even if we don't change
> > anything, since it's a de-facto standard used for different tools to
> > interact.
> >
> > https://github.com/pypa/python-packaging-user-guide/pull/390
> >
> > We can still write a PEP for caching if necessary.
>
> I think documenting what’s there is a reasonable goal, but if we’re going
> to add caching we should just PEP the whole thing changing it from a defect
> standard to an actual standard + caching. Generally we should only use
> non-PEP “specs” in places where we’re just trying to document what exists
> already, but where we’re not really happy with the current solution or we
> plan to alter it eventually.
>
> For this, I think the entry points solution is generally a good one with
> some alterations (namely, the addition of caching)…. Although now that I
> think about it, maybe this isn’t really a packaging problem at all and I’m
> not sure that it benefits from standardization at all.
>
> So stepping back a second, here’s what entrypoints provides today:
>
> 1. A way to implement a interface that some other package can provide
> implementations for.
> 2. A way to specify script wrappers that will be automatically generated.
> 3. A way to define extras that must be installed in order for a particular
> entry point to be available.
>
> Off the bat I’m going to say we don’t need to worry about (2) in this
> hypothetical system, because I think the fact it is implemented currently
> via this system is mostly a historic accident, and it’s not something we
> should be looking at in the future. Script wrappers should have some
> dedicated metadata, not piggybacking off of the plugin system.
>
> For (3) I don’t believe that what extras were installed is recorded
> anywhere, so I’m going to guess that this works by looking up what extras
> are *available* for a particular package and then seeing if all of the
> requirements of that distribution are satisfied. Assuming that’s the case
> then that’s not really something that requires deep integration with the
> packaging toolchain, it just needs the APIs to look those things up.
>
> Finally we come to (1), which is in my opinion the meet of what you’re
> hoping to achieve here (and what most people are using entry points for
> outside of console scripts. What I notice about (1) is that it really has
> absolutely nothing to do with packaging at all. It would likely use some of
> the APIs provided by the packaging toolchain (for instance, the ability to
> add custom files to a .dist-info directory, the ability to iterate over
> installed packages, etc) but as a whole pip, setuptools, twine, PyPI, etc
> none of these things need to know anything about it.
>
> EXCEPT, for the fact that with the desire to cache things, it would be
> beneficial to “hook” into the lifecycle of a package install. However I
> know that there are other plugin systems out there that would like to also
> be able to do that (Twisted Plugins come to mind) and that I think outside
> of plugin systems, such a mechanism is likely to be useful in general for
> other cases.
>
> So heres a different idea that is a bit more ambitious but that I think is
> a better overall idea. Let entrypoints be a setuptools thing, and lets
> define some key lifecycle hooks during the installation of a package and
> some mechanism in the metadata to let other tools subscribe to those hooks.
> Then  a caching layer could be written for setuptools entrypoints to make
> that faster without requiring standardization, but also a whole new, better
> plugin system could to, Twisted plugins could benefit, etc [1].
>
> One thing that I like about all of our work recently in packaging is a lot
> of it has been about making it so there isn’t just one standard set of
> tools, and I think that providing lifecycle hooks is 

Re: [Distutils] Entry points: specifying and caching

[Daniel Holth]

We said "you won't have to install setuptools" but actually "you don't have
to use it" is good enough. If you had 2 pkg-resources implementations
running you might wind up scanning sys.path extra times...

On Wed, Oct 18, 2017, 20:53 Nick Coghlan <ncogh...@gmail.com> wrote:

> On 19 October 2017 at 04:18, Alex Grönholm <alex.gronh...@nextday.fi>
> wrote:
>
>> Daniel Holth kirjoitti 18.10.2017 klo 21:06:
>>
>>
>> http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata
>>
>>
>> http://setuptools.readthedocs.io/en/latest/pkg_resources.html?highlight=pkg_resources#creating-and-parsing
>>
>> It is not very complicated. It looks like the characters are mostly
>> 'python identifier' rules with a little bit of 'package name' rules.
>>
>> I am also concerned about the amount of parsing on startup. A hard
>> problem for certain, since no one likes outdated cache problems either. It
>> is also unpleasant to have too much code with a runtime dependency on
>> 'packaging'.
>>
>> Wasn't someone working on implementing pkg_resources in the standard
>> library at some point?
>>
>
> The idea has been raised, but we've been hesitant for the same reason
> we're inclined to take distutils out: packaging APIs need to be free to
> evolve in line with packaging interoperability standards, rather than with
> the Python language definition.
>
> Barry Warsaw & Brett Cannon recently mentioned something to me about
> working on a potential runtime alternative to pkg_resources that could be
> installed without also installing setuptools, but I don't know any of the
> specifics (and I'm not sure either of them follows distutils-sig).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Daniel Holth]

On Wed, Oct 18, 2017 at 2:57 PM Paul Moore  wrote:

> On 18 October 2017 at 19:42, Thomas Kluyver  wrote:
> > On Wed, Oct 18, 2017, at 05:59 PM, Paul Moore wrote:
> >> > I've always used the setuptools documentation as a reference. Are you
> >> > suggesting moving that information to a different location to
> >> > allow/encourage other tools to implement it as a standard?
> >>
> >> I've never used entry points myself (other than the console script
> >> entry points supported by packaging) but a quick Google search found
> >>
> http://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins
> >> as the only obvious candidate for documentation (and a bit later I
> >> thought of looking under pkg_resources and found
> >>
> http://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points
> ).
> >> This doesn't really say how the entry point data is stored in the
> >> project metadata, so it's not clear how I'd read that data in my own
> >> code (the answer is of course to use pkg_resources, but the point of
> >> documenting it as a standard is to allow alternative implementations).
> >
> > I have in fact made an alternative implementation (PyPI package
> > entrypoints) by 'reverse engineering' the format. A simple text-based
> > format doesn't really justify the term 'reverse engineering', but for
> > instance it wasn't obvious to me that the names were case sensitive,
> > whereas Python's standard config parser treats keys as case-insensitive.
> >
> > Daniel:
> >>
> http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata
> >
> > Thanks, this link is closer than any I found to a specification. There
> > are docs on how to create entry points in setup.py and how to use them
> > with pkg_resources, but that's the only bit I've seen that describes the
> > interchange file format.
>
> Agreed, I hadn't found that, either.
>
> > I think we can probably expand on it a bit, though! I'll try to put
> > together something for packaging.python.org.
>
> One thing that immediately strikes me is that the encoding of the file
> is unspecified...
> Paul
>

Now that's an easy one to clear up, since there is only one worthwhile
encoding.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Daniel Holth]

On Wed, Oct 18, 2017 at 2:18 PM Alex Grönholm <alex.gronh...@nextday.fi>
wrote:

> Daniel Holth kirjoitti 18.10.2017 klo 21:06:
>
>
> http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata
>
>
> http://setuptools.readthedocs.io/en/latest/pkg_resources.html?highlight=pkg_resources#creating-and-parsing
>
> It is not very complicated. It looks like the characters are mostly
> 'python identifier' rules with a little bit of 'package name' rules.
>
> I am also concerned about the amount of parsing on startup. A hard problem
> for certain, since no one likes outdated cache problems either. It is also
> unpleasant to have too much code with a runtime dependency on 'packaging'.
>
> Wasn't someone working on implementing pkg_resources in the standard
> library at some point?
>

I'm just saying it is good to avoid importing it unless you really need to.
Same reason we removed it from entry point script wrappers.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Daniel Holth]

http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata

http://setuptools.readthedocs.io/en/latest/pkg_resources.html?highlight=pkg_resources#creating-and-parsing

It is not very complicated. It looks like the characters are mostly 'python
identifier' rules with a little bit of 'package name' rules.

I am also concerned about the amount of parsing on startup. A hard problem
for certain, since no one likes outdated cache problems either. It is also
unpleasant to have too much code with a runtime dependency on 'packaging'.

On Wed, Oct 18, 2017 at 1:00 PM Paul Moore  wrote:

> On 18 October 2017 at 17:48, Doug Hellmann  wrote:
> > Excerpts from Thomas Kluyver's message of 2017-10-18 15:52:00 +0100:
> >> We're increasingly using entry points in Jupyter to help integrate
> >> third-party components. This brings up a couple of things that I'd like
> >> to do:
> >>
> >> 1. Specification
> >>
> >> As far as I know, there's no document describing the details of entry
> >> points; it's a de-facto standard established by setuptools. It seems to
> >> work quite well, but it's worth writing down what is unofficially
> >> standardised. I would like to see a document on
> >> https://packaging.python.org/specifications/ saying:
> >>
> >> - Where build tools should put entry points in wheels
> >> - Where entry points live in installed distributions
> >> - The file format (including allowed characters, case sensitivity...)
> >>
> >> I guess I'm volunteering to write this, although if someone else wants
> >> to, don't let me stop you. ;-)
> >>
> >> I'd also be happy to hear that I'm wrong, that this specification
> >> already exists somewhere. If it does, can we add a link from
> >> https://packaging.python.org/specifications/ ?
> >
> > I've always used the setuptools documentation as a reference. Are you
> > suggesting moving that information to a different location to
> > allow/encourage other tools to implement it as a standard?
>
> I've never used entry points myself (other than the console script
> entry points supported by packaging) but a quick Google search found
>
> http://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins
> as the only obvious candidate for documentation (and a bit later I
> thought of looking under pkg_resources and found
> http://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points
> ).
> This doesn't really say how the entry point data is stored in the
> project metadata, so it's not clear how I'd read that data in my own
> code (the answer is of course to use pkg_resources, but the point of
> documenting it as a standard is to allow alternative implementations).
> Also, it's not clear how a tool like flit might implement entry points
> - again, because the specifications don't describe how the metadata is
> stored.
>
> +1 from me on moving the entry point specification to
> https://packaging.python.org/specifications/
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[Daniel Holth]

Enscons uses parts of distutils to get compiler flags and so on but does
not use Extension() to do the actual compiling. There might be a cleaner
way to do it that I was not able to find. There could be a cleaner
separation between parts of distutils related to how Python itself was
compiled and the part that does the actual compiling. The sysconfig module
attempts to do at least part of this.

distutils also knows about some compiler tricks like generating symbol
tables for Windows, that are not easy to re-use or invoke separately.
Someone could catalog these tricks and techniques and refactor the useful
ones into a reusable library. I don't have a great sense of how to do it.

https://bitbucket.org/dholth/enscons/src/tip/enscons/cpyext.py?at=default=file-view-default

On Wed, Sep 27, 2017 at 11:36 PM Nick Coghlan  wrote:

> On 28 September 2017 at 06:00, xoviat  wrote:
> > That's actually an interesting idea though: for Python 3.7 distutils ->
> > _distutils (and then setuptools is required for building). For/against?
>
> distutils works fine for its original purpose (building components for
> the system Python in Linux distros), so we still need to avoid
> breaking that. setuptools is only essential if you want full support
> for modern *Python* level packaging features (PEP 376 install
> metadata, venv compatibility, wheel files, etc), and a lot of Linux
> system components simply don't worry about those things, and rely on
> their system level equivalents instead (e.g. the RPM/deb databases,
> chroots and containers, RPM/deb files)
>
> However, what *could* be interesting is a proposal to move distutils
> to the "ensurepip" model, where rather than maintaining distutils
> directly as part of CPython, the CPython build process instead runs
> setuptools/distutils from a bundled wheel file. Doing that would
> entail having setuptools actually start installing a copy of distutils
> into site-packages: older CPython releases would ignore it by default
> (since the stdlib version would shadow it), while 3.7+ would offer
> either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools"
> (bikeshed to be painted via the PEP process) to install the
> setuptools-provided version.
>
> I'm not claiming actually doing that would be particularly easy - I
> just think it's the most viable path to get us away from the current
> version coupling between the build infrastructure in distutils and the
> runtime support infrastructure in the rest of the standard library,
> and to avoid maintaining two distinct copies of distutils indefinitely
> (one in the stdlib, one in setuptools).
>
> That approach wouldn't even entail any *new* bundling at the CPython
> level, as while it's currently formally an implementation detail
> (pending potential removal in a post-PEP-517 world), setuptools is
> already bundled as part of the support infrastructure for ensurepip:
> https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New wheel release (v0.30.0)

[Daniel Holth]

Thanks.

This release also drops support for Python 2.6. It seems hardly anyone
noticed.

On Mon, Sep 11, 2017, 01:24 Nick Coghlan  wrote:

> On 11 September 2017 at 08:21, Alex Grönholm 
> wrote:
> > I have now made my first wheel release as the new maintainer.
> >
> > This release contains bug fixes and documentation updates. I decided to
> make
> > a release now to get a fix out for some long standing issues.
>
> Thank you for taking this on, and thanks to Daniel for being open to
> granting you the power to make new releases! :)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Migrating interoperability specs to packaging.python.org

[Daniel Holth]

Well, none of the metadata generated by bdist wheel conforms to an accepted
pep. But if you rely on the json file then you won't be interoperable with
wheels from any other generator.

On Mon, Sep 4, 2017, 10:06 Alex Grönholm <alex.gronh...@nextday.fi> wrote:

> Yes, I see the inclusion of a metadata file which conforms to an
> unaccepted PEP as potentially dangerous.
>
> Perhaps I should disable it in the next release?
>
> Daniel Holth kirjoitti 04.09.2017 klo 17:03:
>
> Some people enjoy using metadata.json which tracked pep 426 but I have
> been meaning to take it out, and perhaps keep the key/value to json
> converter as a command.
>
> On Mon, Sep 4, 2017, 09:33 Nick Coghlan <ncogh...@gmail.com> wrote:
>
>> Some time ago, I started the process [1] of adjusting how
>> distutils-sig uses the PEP process so that the reference
>> specifications will live on packaging.python.org, and we use the PEP
>> process to manage *changes* to those specifications, rather than
>> serving as the specifications themselves (that is, adopting a process
>> closer to the URL-centric way the Python language reference is
>> managed, rather than using the RFCstyle PEP-number-centric model the
>> way we do now).
>>
>> I never actually finished that work, and as a result, it's currently
>> thoroughly unclear [2] that Description-Content-Type and
>> Provides-Extra are defined at
>> https://packaging.python.org/specifications/#core-metadata rather than
>> in a PEP.
>>
>> I'm currently at the CPython core development sprint in San Francisco,
>> and I'm thinking that finalising that migration [3] and updating the
>> affected PEPs accordingly (most notably, PEP 345) is likely to be a
>> good use of my time.
>>
>> However, I'm also wondering if it may still be worthwhile writing a
>> metadata 1.3 PEP that does the following things:
>>
>> 1. Explicitly notes the addition of the two new fields
>> 2. Describes the process change for packaging interoperability
>> specifications
>> 3. Defines a canonical transformation between the human-readable
>> key:value format and a more automation friendly JSON format
>>
>> That PEP would then essentially be the first one to use the new
>> process: it would supersede PEP 345 as the latest metadata
>> specification, but it would *also* redirect readers to the relevant
>> URL on packaging.python.org as the canonical source of the
>> specification, rather than being the reference documentation in its
>> own right.
>>
>> Cheers,
>> Nick.
>>
>> [1] https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061
>> [2] https://github.com/python/peps/issues/388
>>
>> P.S. Daniel, if you're currently thinking "I proposed defining an
>> incremental metadata 1.3 tweak years ago!", aye, you did. My
>> subsequent additions to PEP 426 were a classic case of second-system
>> syndrome: https://en.wikipedia.org/wiki/Second-system_effect (which we
>> suspected long ago, hence that PEP's original deferral)
>>
>> Fortunately, the disciplining effect of working with a primarily
>> volunteer contributor base has prevented my over-engineered
>> change-for-change's-sake-rather-than-to-solve-actual-user-problems
>> version from becoming reality ;)
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
> ___
> Distutils-SIG maillist  -  
> Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Migrating interoperability specs to packaging.python.org

[Daniel Holth]

Some people enjoy using metadata.json which tracked pep 426 but I have been
meaning to take it out, and perhaps keep the key/value to json converter as
a command.

On Mon, Sep 4, 2017, 09:33 Nick Coghlan  wrote:

> Some time ago, I started the process [1] of adjusting how
> distutils-sig uses the PEP process so that the reference
> specifications will live on packaging.python.org, and we use the PEP
> process to manage *changes* to those specifications, rather than
> serving as the specifications themselves (that is, adopting a process
> closer to the URL-centric way the Python language reference is
> managed, rather than using the RFCstyle PEP-number-centric model the
> way we do now).
>
> I never actually finished that work, and as a result, it's currently
> thoroughly unclear [2] that Description-Content-Type and
> Provides-Extra are defined at
> https://packaging.python.org/specifications/#core-metadata rather than
> in a PEP.
>
> I'm currently at the CPython core development sprint in San Francisco,
> and I'm thinking that finalising that migration [3] and updating the
> affected PEPs accordingly (most notably, PEP 345) is likely to be a
> good use of my time.
>
> However, I'm also wondering if it may still be worthwhile writing a
> metadata 1.3 PEP that does the following things:
>
> 1. Explicitly notes the addition of the two new fields
> 2. Describes the process change for packaging interoperability
> specifications
> 3. Defines a canonical transformation between the human-readable
> key:value format and a more automation friendly JSON format
>
> That PEP would then essentially be the first one to use the new
> process: it would supersede PEP 345 as the latest metadata
> specification, but it would *also* redirect readers to the relevant
> URL on packaging.python.org as the canonical source of the
> specification, rather than being the reference documentation in its
> own right.
>
> Cheers,
> Nick.
>
> [1] https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061
> [2] https://github.com/python/peps/issues/388
>
> P.S. Daniel, if you're currently thinking "I proposed defining an
> incremental metadata 1.3 tweak years ago!", aye, you did. My
> subsequent additions to PEP 426 were a classic case of second-system
> syndrome: https://en.wikipedia.org/wiki/Second-system_effect (which we
> suspected long ago, hence that PEP's original deferral)
>
> Fortunately, the disciplining effect of working with a primarily
> volunteer contributor base has prevented my over-engineered
> change-for-change's-sake-rather-than-to-solve-actual-user-problems
> version from becoming reality ;)
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] why do we still put version numbers into dist-info folders

[Daniel Holth]

A long time ago pkg_resources was able to sometimes get the version number
only from the .egg-info folder name, and skip opening the metadata at all
when that data was not needed. We might always do listdir() for path in
sys.path and parse every METADATA on the first pkg_resources import.

On Fri, Sep 1, 2017 at 4:32 PM Alex Grönholm 
wrote:

> +1 for getting rid of version numbers in the dist-info folders.
>
>
> RonnyPfannschmidt kirjoitti 01.09.2017 klo 20:32:
> > Hi everyone,
> >
> > a while now i have wondered - why put version numbers into the dist info
> > folders
> > not only makes it lookup more expensive (need to search for a
> > distribution->folder map)
> > its also not serving a purpose, as the rest of the package is
> > unversioned at the root of site-packages.
> >
> > in addition all those versions in folder names, that serve no purpose,
> > and create duplication/conflicts now creep into other peps as well,
> >
> > please lets take a note at setuptools - for editable installs it wont
> > put a version in to the egg info folder
> >
> > and imho thats the correct way to deal with data thats not actually
> > versioned at that level where its being stored.
> >
> >
> > -- Ronny
> >
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

I don't think this is a big deal. Compatibility is a guess, wheel tags do
not and are not intended to guarantee compatibility especially when the
wheel was built locally for caching purposes and not intentionally
published. Anything can happen during a build. PEP 517 won't change this.

So right now pip has an odd way of segregating its wheel cache per
interpreter that involves obscure setup.py arguments. That can be fixed
without re-tagging or asking the build system to re-tag wheels or otherwise
interacting with the new PEP.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

Don't be afraid of extras. They have not been PEP'd but are required for a
useful Python packaging system. install_requires is just a shorthand for
extras with the "" empty string as their name. Too bad we couldn't agree on
; vs : as the delimiter though.

On Thu, Aug 31, 2017 at 2:04 PM Nathaniel Smith  wrote:

> On Thu, Aug 31, 2017 at 8:41 AM, Chris Barker - NOAA Federal
>  wrote:
> > The package manager should manage the package, not built it, or change
> it.
> >
> > Surely the build system should know how to correctly name the wheel it
> builds.
>
> It's probably worth mentioning the specific problem that motivated pip
> to start doing this.
>
> It used to be standard, and is still quite common, for setup.py
> scripts to contain stuff like:
>
> install_requires = [...]
> if sys.version_info < (3, 4):
> install_requires += [...]
> if platform.python_implementation() == "PyPy":
> install_requires += [...]
>
> setup(..., install_requires=install_requires)
>
> This kind of logic in setup.py worked fine in the old days when all
> you did was 'setup.py install', but then wheels came along and
> retroactively turned lots of setup.py scripts from working into
> broken. The problem is that with this kind of setup.py, setuptools has
> *no idea* that the install_requires you gave it would have been
> different if you had run setup.py with a different version of Python,
> so when it has to assign Python tags to a built wheel it guesses wrong
> and uses ones that are too general.
>
> The right way to do this is to use PEP 508 environment markers:
> https://www.python.org/dev/peps/pep-0508/#environment-markers
> or the non-standard extras hack:
>
> https://wheel.readthedocs.io/en/latest/#defining-conditional-dependencies
> Both of these let you export the whole requirements-choosing logic
> into the wheel metadata, so that it can be evaluated at install time
> instead of build time.
>
> But it will take a while for existing setup.py files transition to
> using those, and in the mean time pip can't assume that a random wheel
> generated by 'setup.py bdist_wheel' has accurate Python tags.
>
> Hopefully new legacy-free backends will get this right from the start.
> For example flit makes it impossible to get this wrong.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

On Thu, Aug 31, 2017 at 12:58 PM Chris Barker  wrote:

> On Thu, Aug 31, 2017 at 8:57 AM, Paul Moore  wrote:
>
>> > As to using pip to build wheels -- there is good reason to do that
>> > now, but in s post PEP 517 world, one would call the build system
>> > directly to build a wheel-- after all, all pip should be  doing is
>> > calling the build system anyway.
>>
>> I disagree - "pip wheel" will still be useful, for example to obtain a
>> wheel from PyPI either by downloading or download & build. Also just
>> to have a unified interface that works regardless of the project
>> backend - if a project switches from setuptools to flit, for example,
>> it would be good if deployment and test scripts didn't have to change.
>>
>
> Isn't that why we have PEP 517? I unified interface to build tools?
>
> So I still expect pip wheel to be useful in a post-PEP 517 world.
>
>
> Maybe so -- but all pip should be doing is passing off the job to the
> back-end.
>
> Again, the package manager, well, manages the packages. It shouldn't be
> concerned with how to build them.
>

pip has always been under appreciated as a build tool. Suppose the next
version of pip dropped support for sdists and could only install wheels.
Then it would not be a build tool. Very little changes when it switches to
supporting sdists from its own setup.py machinery to a more pluggable
mechanism. It is a little controversial that it can produce wheels for
re-use but the feature is very practical. You could put 'pip wheel' under a
different command line tool but why bother.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

I'm not against renaming a wheel to adjust the tags, but it is a strange
caching strategy. wheel is due for a 'retag' feature that does a more
rigorous job of changing tags.

The build system should name its wheels correctly. Normally the pyNN tag
would be used for a pure distribution and the cpNN tag when there are
extensions. If you want to affect the tagging when called through PEP 517,
edit the source's pyproject.toml / setup.cfg / setup.py.

It will probably always make sense to use pip to build wheels when you want
to build wheels for a project and all its dependencies. Then you can keep
these wheels in a folder for repeatable offline deployment.

On Thu, Aug 31, 2017 at 11:48 AM Chris Barker - NOAA Federal <
chris.bar...@noaa.gov> wrote:

> > that neither pip nor the setuptools backend should not change the tags
> > it applies to wheels by default).
>
> I'm a bit confused -- are we talking about the backwards compatible
> path to the future -- or the end-game?
>
> In short -- I'm sure we'll have to do some hacky stuff to keep
> backwards compatibility, but the end game should be a clean separation
> of concerns :
>
> Pip (or any front end) should never "build a wheel", and it certainly
> shouldn't have to know enough about what's in a wheel to be re-naming
> it for generic python vs cpython.
>
> The package manager should manage the package, not built it, or change it.
>
> Surely the build system should know how to correctly name the wheel it
> builds.
>
> As to using pip to build wheels -- there is good reason to do that
> now, but in s post PEP 517 world, one would call the build system
> directly to build a wheel-- after all, all pip should be  doing is
> calling the build system anyway.
>
> -CHB
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

Then end the debate by letting the PEP authors decide the return type, and
write a paragraph explaining why the other options were rejected. It is not
going to make a big difference.

On Mon, Aug 28, 2017 at 3:59 PM Thomas Kluyver  wrote:

> On Mon, Aug 28, 2017, at 08:50 PM, Paul Moore wrote:
> > My main motivation for wavering is that I thought agreeing to trust
> > the backend would simplify many of the decisions, and it's immensely
> > frustrating to me that we're still debating the same question in the
> > "return None" thread.
>
> The difference I see with the "return None" question is that there we
> have an alternative (return NotImplemented) which is just as simple for
> both sides, but avoids the identified issue with a buggy backend. The
> only argument there seems to be for using None is about semantics - and
> that's not a great argument, because 'practicality beats purity'.
>
> With the questions over sdist/wheel consistency, there's a complexity
> cost, for the spec and for frontends, in deciding that they can't. So
> we're weighing a trade-off: do we force ourselves to resolve the
> notimplemented question so that frontends can do sdist-wheel+fallback,
> or do we leave it up to frontends and risk some bugs which we might
> otherwise have prevented.
>
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

On Mon, Aug 28, 2017 at 3:48 PM xoviat  wrote:

> > But I'm suspicious of the rationale that *there will be fewer frontends so
> they should have more responsibility*.
>
> To be fair, pip is currently struggling to keep up with project
> requirements as it is, which has caused some frustration in the community
> (not that the frustration isn't wrong, but I don't think it's not going to
> resolve any problems). I don't see how having even more frontends will
> resolve these manpower issues.
>

Simple. It would be different people.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PBR/distutils2 with pep517 Support Was: Conditionless setup.py

[Daniel Holth]

Imo PBR is entirely a setuptools creature, without special concerns to
operate in pep517 land. If I were them I'd rewrite it to not require
setup.py and call that pbr2.

On Mon, Aug 28, 2017, 12:44 Chris Barker  wrote:

> I've thought for ages that we could  transition to a more sane system by
> convention:
>
> "your setup.py, after being imported, will have a "setup_params" attribute
> that is a dict that can be passed to setup()."
>
> So tools that want metadata, etc. without actually running an install
> could do;
>
> import setup
>
> name = setup.setup_params['name']
> requirements = setup.setup_params['install_requires']
>
> etc...
>
> i.e -- it does need to run the code in setup.py, but it does NOT need to
> run setup install, or build, or whatever else.
>
>
> A typical simple setup.py with static meta-data would be:
>
> from setuptools import setup
>
> setup_params = {name:'sample',
> version:'1.2.0',
> description:'A sample Python project',
> url:'https://github.com/pypa/sampleproject',
> author:'The Python Packaging Authority',
> license:'MIT',
> packages:['the_package'],
> install_requires=['peppercorn'],
> ...
> }
>
> setup(**setup_params)
>
> But a setup.py could still have arbitrary code to build up the dict of
> parameters if need be.
>
> -CHB
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

Return none makes sense, but I don't think the outcome of that or the
"configurable src path" decisions will affect the success of the proposal.

On Sun, Aug 27, 2017, 05:59 Paul Moore  wrote:

> On 26 August 2017 at 23:13, Nathaniel Smith  wrote:
> > This whole discussion is about handling a specific case: suppose you
> > have a frontend like pip that when given a source directory and asked
> > to build a wheel, wants to implement that as:
> >   - build sdist
> >   - unpack sdist
> >   - build wheel from unpacked sdist
>
> While I can't speak for Donald or the other pip devs, I'd like to
> point out that in
> https://mail.python.org/pipermail/distutils-sig/2017-August/031288.html
> I confirmed that I'm basically persuaded now that pip should just
> "trust the backend" and use build_wheel directly. So this whole debate
> isn't relevant for pip. We'll need to wait for other
> currently-hypothetical frontends like tox or twine which might use the
> build_wheel hook, before we'll have any practical experience to
> confirm if there's an issue with "return None" as the way for backends
> to signal that they weren't able to build a wheel.
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

I'm bored with that

On Sat, Aug 26, 2017, 11:59 xoviat  wrote:

> Is everyone on board with that?
>
> On Aug 26, 2017 4:29 AM, "Paul Moore"  wrote:
>
>> On 26 August 2017 at 03:17, Guido van Rossum  wrote:
>> > In pretty much any other context, if you have an operation that returns
>> an
>> > regular value or an error value, the error value should be None.
>> (Exceptions
>> > include e.g. returning a non-negative int or -1 for errors, or True for
>> > success and False for errors.)
>>
>> So, given that build_sdist returns the path of the newly built sdist,
>> the correct way to signal "I didn't manage to build a sdist" would be
>> to return None.
>>
>> Now that it's put this way, it seems glaringly obvious to me that this
>> is the correct thing to do.
>> Paul
>>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Conditionless setup.py

[Daniel Holth]

It's important to realize that build is a Turing-complete activity. We
write programs to build programs. Where setup.py gets in trouble is by
making both the metadata and the build script the same thing, making it
harder to reason about that metadata. That is why we are working towards
being able to express the metadata in pyproject.toml, while letting you
have any kind of build script you please. We expect that many more projects
will be able to deal with this limitation compared to e.g. always having to
express the whole build with static configuration of something like
distutils2 or PBR.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Daniel Holth]

It would be simpler if the only requirement was to produce a correct wheel.

On Thu, Aug 24, 2017, 20:07 Greg Ewing  wrote:

> Paul Moore wrote:
> > 4. The point of all this is that the definition of build_directory
> > says "If build_directory is None, the backend may do an 'in place'
> > build which modifies the source directory and may produce different
> > results from those that would be obtained by exporting an sdist
> > first".
>
> Is it really necessary for pip to specify exactly *where* to do
> an out-of-place build? Why should it care, as long as the end
> result is a clean wheel?
>
> --
> Greg
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Daniel Holth]

The distutils build/ folder is what moves if you pass build_dir to the
proposed function.

On Thu, Aug 24, 2017, 14:39 xoviat <xov...@gmail.com> wrote:

> > It looks like a lot of trouble for a feature that is designed to solve a
> problem for a very thin intersection of the Venn diagram of people who:
>
> For the record, I don't agree at all that build_wheel_incremental should
> be specified here. The suggestion is simply a compromise that I can
> tolerate because it's optional.
>
> > For better or worse, editable installs exist and don't need to be
> defined by PEP 517 just to handle this case. I'm not convinced it's worth
> the additional complexity of the spec just for that.
>
> The editable installs in my opinion should have been specified in this PEP
> because people are clearly trying to work around a lack of consensus on
> that issue so that they don't have to wait for the next PEP and waste their
> time waiting for uncached builds. I can't say I disagree with this but it's
> trying to fit a square peg into a round hole.
>
> > Is there any backend doing this "caching elsewhere" today? Is any
> backend planning to?
>
> Yes. Distutils caches in the build folder. It's very useful when working
> on complex projects with Extensions.
>
> > Can we arrive at some more concrete examples of how that would work?
> Like how would pip be invoked, how would the backend react.
>
> The reason that I'm willing to live with a separate optional function is
> that I don't think it will ever be implemented. It will be a note in Python
> history until install_editable is specified.
>
> 2017-08-24 13:30 GMT-05:00 Leonardo Rochael Almeida <leoroch...@gmail.com>
> :
>
>>
>> On 24 August 2017 at 15:13, Thomas Kluyver <tho...@kluyver.me.uk> wrote:
>>
>>> On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote:
>>>
>>> On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver <tho...@kluyver.me.uk>
>>> wrote:
>>>
>>> Is there a reason to ask for an 'unclean' build, though? There may be a
>>> performance optimisation from reusing cached data,
>>>
>>>
>>> For the same reason you would ever ask for incremental builds, to more
>>> quickly iterate while hacking, imagining that you are using the PEP 517
>>> interface to develop, perhaps to have a uniform interface to patch
>>> something when you are not familiar with exactly the build system it uses.
>>>
>>> It looks like a lot of trouble for a feature that is designed to solve a
>> problem for a very thin intersection of the Venn diagram of people who:
>>
>>  a. wants to quickly iterate while hacking a package
>>
>>  b. doesn't want to bother learning how to trigger the incremental build
>> system of that package or how to install it in editable mode.
>>
>> For better or worse, editable installs exist and don't need to be defined
>> by PEP 517 just to handle this case. I'm not convinced it's worth the
>> additional complexity of the spec just for that.
>>
>> Yup, this is what I meant about using cached data for performance. But I
>>> don't think this requires builds to modify the source directory. The build
>>> system can cache data elsewhere on the filesystem, whether or not it's
>>> given an explicit build_directory.
>>>
>>
>> Is there any backend doing this "caching elsewhere" today? Is any backend
>> planning to?
>>
>>
>>> I think we can specify one kind of build that is both 'clean' (doesn't
>>> modify the source directory) and incremental (can use cached data to speed
>>> up the build). If that's workable, it seems like it would save a lot of
>>> headaches rather than trying to specify them as two different options.
>>>
>>> We have explicitly excluded editable installs (i.e. inplace builds) from
>>> this PEP, though we can add a hook for that in a later PEP. I agree with
>>> you that this PEP should allow for fast incremental (but non-inplace)
>>> builds if possible, though.
>>>
>>
>> Can we arrive at some more concrete examples of how that would work? Like
>> how would pip be invoked, how would the backend react. And how (and for
>> whom) this would be better than either doing an editable install or
>> straight clean install?
>>
>> I fear we're complicating the spec for something that will be hard to
>> happen in practice.
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Daniel Holth]

On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver <tho...@kluyver.me.uk>
wrote:

> On Thu, Aug 24, 2017, at 05:26 PM, Daniel Holth wrote:
>
> by including the build_dir parameter, a nearly universal build system
> concept, pip gets an elegant way to ask for either a clean or unclean build.
>
>
> Is there a reason to ask for an 'unclean' build, though? There may be a
> performance optimisation from reusing cached data,
>

For the same reason you would ever ask for incremental builds, to more
quickly iterate while hacking, imagining that you are using the PEP 517
interface to develop, perhaps to have a uniform interface to patch
something when you are not familiar with exactly the build system it uses.


> but building in a separate directory doesn't preclude caching
> intermediates somewhere else.
>

The argument is necessarily a hint to the build system? If it works it can
do whatever.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Daniel Holth]

build_dir:

bdist_wheel sometimes has a problem with unclean builds. admittedly this
could be fixed.
pip copies entire source tree, including unclean build/ directory, to tmpdir
bdist_wheel runs in tmpdir, including copied leftover build/ data in wheel

by including the build_dir parameter, a nearly universal build system
concept, pip gets an elegant way to ask for either a clean or unclean build.

sdist optional:

enscons can use .hg manifest instead of MANIFEST.in. what is this git
nonsense?
user has decided to build from an unpacked sdist
impossible to create new sdist from unpacked sdist



On Thu, Aug 24, 2017 at 11:49 AM xoviat  wrote:

> If the entire idea of copying out-of-tree is to work around setuptools
> deficiencies, then perhaps it would be a better idea to push this onto the
> setuptools build backend rather than bring these problems into PEP 517?
>
> 2017-08-24 10:32 GMT-05:00 xoviat :
>
>> May I ask what is wrong *in principle* with the setuptools "build" folder
>> (other than the fact that it does not contain all tree changes)?
>>
>> 2017-08-24 10:27 GMT-05:00 xoviat :
>>
>>> That is actually the first time that build_dir makes sense to me now.
>>> Thank you.
>>>
>>> 2017-08-24 10:24 GMT-05:00 Paul Moore :
>>>
 On 24 August 2017 at 16:20, xoviat  wrote:
 >>  I *do* care about telling backends we don't want "different
 > results from those that would be obtained by exporting an sdist
 > first".
 >
 > I completely agree with this statement, but I don't believe that it
 can or
 > should be accomplished with this parameter. Let me just quote the
 process
 > that I proposed:
 >
 >
 >> Proposed process:
 > - Frontend copies source tree to temporary directory

 That step's the problem. If the frontend does that it can potentially
 be copying a lot of unneeded stuff (VCS history, for example). We
 tried that with pip and it was a major issue. That problem is the
 *whole point* of all the discussions about the various proposals that
 ended up with build_tree.

 > - Frontend invokes build-sdist to build an sdist
 > - Frontend extracts sdist to new temporary directory
 > - Frontend reloads backend from sdist directory and invokes
 build-wheel

 Paul

>>>
>>>
>>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Bootstrapping setuptools

[Daniel Holth]

Isn't this a special case of needing . on sys.path when building a package?
Then the same version of setuptools that is being packaged builds itself.

I've experimented with a different kind of bootstrapping where pip installs
setuptools on demand when the first setuptools requiring package is built.
You kindof need pkg-resources as a separate package for this. It is a cool
parlor trick but it's simpler and harmless to just always install
setuptools like we do.

On Tue, Aug 22, 2017, 09:14 Jim Fulton  wrote:

> On Mon, Aug 21, 2017 at 5:39 PM, xoviat  wrote:
>
>> This statement comes from something that Donald said:
>>
>> > The unvendoring means that setuptools and the project code are now
>> competing over who gets to define what an acceptable version is for these
>> libraries to be installed with.
>>
>> As if this isn't going to be true for any other build system, and
>> setuptools won't at all be in any way special in a PEP 517 world. So you
>> are always going to have complaints about "well build system X required a
>> newer version but my project requires an older version" when the real
>> problem is that people are requiring an older version/ projects do not have
>> a stable API.
>>
>
> This boils down to implementation details.   pip dealt with this
> issue and buildout stumbled because of differences in the way they managed
> their paths.  Buildout could have coped (eventually).
>
> Jim
>
>
>> 2017-08-21 16:27 GMT-05:00 Jim Fulton :
>>
>>>
>>>
>>> On Mon, Aug 21, 2017 at 5:17 PM, xoviat  wrote:
>>>
 Of course, to be frank, the principle failure with this plan is that
 third-party tools (buildout, os packagers) will not be compliant with PEP
 517 even after it is adopted, and will then complain about having to update
 their build systems.

>>>
>>> I don't understand this statement. If buildout needs to be compliant, it
>>> will be (eventually).  I can imagine a future where everything is
>>> distributed with wheels and buildout wouldn't need to build anything.
>>>
>>> Jim
>>>
>>>

 2017-08-21 16:05 GMT-05:00 xoviat :

> Previously, the attempt to move setuptools off of vendored
> dependencies failed because it was not done correctly: install_requires 
> was
> set to the vendored dependencies but not setup_requires, which would have
> been required to correctly specify dependencies. However, setup_requires
> would have introduced circular dependencies that are impossible to
> correctly resolve, so that would have simply shifted an impossible problem
> onto pip.
>
> The principle issue then, is that setuptools requires setuptools to
> build itself. And although, this problem was previously not worth solving
> because of 'setup.py install', it is now not a difficult problem to solve
> if we rethink slightly what should be included in the setuptools
> respository:
> - Rather than re-generating egg_info on each setup.py invocation, we
> can cache dist-info in the setuptools respository.
> - We can implement a simple entry_points script that updates dist-info
> based on the contents of setup.py, which is only runnable of course after
> setuptools is installed
> - We can implement the reference build backend for setuptools: simply
> copy the files and dist-info into a new compliant zip archive
>
> Viola! Setuptools is no longer required to build setuptools, and the
> installation process is fully compliant with Python PEPs.
>


 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


>>>
>>>
>>> --
>>> Jim Fulton
>>> http://jimfulton.info
>>>
>>
>>
>
>
> --
> Jim Fulton
> http://jimfulton.info
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build frontend responsibilities

[Daniel Holth]

It's probably a tiny wrapper running the hook in its own subprocess.
Probably few modules loaded.

On Sat, Aug 19, 2017, 14:31 xoviat  wrote:

> Ah the joy of Python 2.7; it seems I've forgotten its perils: unloading is
> not possible which means that we would need to come up with another
> solution to this problem. Perhaps setting aside a namespace for the build
> frontend in the subprocess?
>
> 2017-08-19 14:23 GMT-04:00 xoviat :
>
>> I assume that the outstanding issues mentioned are related to sys.path in
>> the build tree. My view on that is the following: the build frontend should
>> be responsible using any mechanism that it chooses for invoking the build
>> backend, which must be imported with '' at the front of sys.path. This
>> obviously means that if the build frontend experiences faults before it
>> manages to import and invoke the backend, then it's the build frontend's
>> fault.
>>
>> The only potential issue remaining that I can think of then is: what
>> about modules that are already imported when the build backend is called?
>> WRT standard library modules, we could simply say that build backends must
>> be prepared to function in any environment where standard library modules
>> are already imported, which I think is a reasonable requirement. But what
>> about imports that aren't standard library modules but are fairly common,
>> like pip imports? The simplest way to simplify the interface is probably to
>> say that all non-standard library modules must be removed from sys.modules
>> (unloaded) before the build backend is called.
>>
>> What do others think?
>>
>> 2017-08-18 23:41 GMT-04:00 Nick Coghlan :
>>
>>> On 19 August 2017 at 05:28, Thomas Kluyver  wrote:
>>> > We've probably all wished that the discussion could be brought to a
>>> swift
>>> > conclusion. But there are real questions to work out, and people have
>>> many
>>> > other things to pay attention to. I'm frustrated by how long it's
>>> taking as
>>> > well, but there's no magic button anyone can press to make it go
>>> quickly.
>>>
>>> Technically, commercial redistributors do have a magic button we could
>>> press called "ongoing funding for sustaining engineering in the
>>> upstream Python ecosystem" (since that kind of funding can also cover
>>> needs-driven UX improvements), but alas, whether or not we ever
>>> actually press that is conditional on potential customers pressing the
>>> "customer demand" button hard enough and often enough to light up the
>>> "viable business opportunity" sign :P
>>>
>>> I'm actually genuine curious to see how those commercial dynamics
>>> start changing as the end date for community support of the Python 2.x
>>> series gets closer :)
>>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>>
>>
>>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build frontend responsibilities

[Daniel Holth]

Apart from the issues, can we get some prototype implementations?

On Fri, Aug 18, 2017, 13:24 xoviat  wrote:

> Thomas:
>
>
> What are the specific issues that need to be worked out?
>
>
> Regards,
>
>
> xoviat
>
> 2017-08-18 3:09 GMT-05:00 Thomas Kluyver :
>
>> On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote:
>> > The other aspect I'm not clear on is whether or not both PEP authors
>> > are otherwise happy for the current version to be *considered* for
>> > acceptance
>>
>> Nathaniel has mentioned to me a list of issues he sees that we still
>> need to work through. So I think we're going to be working on this for a
>> while yet.
>>
>> Thomas
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Daniel Holth]

Works for me

El mar., 1 de ago. de 2017 11:32, Thomas Kluyver 
escribió:

> Are we content to say that sys.path includes the source directory where
> the hook is run? Shall I prepare a PR against the PEP for that?
>
> On Sun, Jul 30, 2017, at 02:12 PM, Nick Coghlan wrote:
> > On 30 July 2017 at 02:46, Nathaniel Smith  wrote:
> > > Or am I worrying about a non-issue and it's fine if flit imports click
> from
> > > the source tree?
> >
> > Don't worry about it too much, as the problem here isn't really any
> > worse than it is for normal runtime dependencies of any other project
> > that relies on having the current directory first in sys.path. It just
> > so happens that the project in question in this case is a Python
> > project's build system.
> >
> > Due to the preference for a flat module namespace as the default,
> > there are plenty of ways to hit name shadowing problems in Python, and
> > as Donald notes, build systems have other motives to vendor their
> > dependencies rather than installing them normally.
> >
> > Just switching the path order as has been suggested also doesn't solve
> > the problem, as it merely inverts the issue: having "some_name"
> > installed in site-packages would break source installations of
> > packages that expected to be able to import "some_name" from their own
> > root directory.
> >
> > If the problem does come up in practice, then there are a number of
> > ways for affected projects to work around it in their project
> > directory structure:
> >
> > 1. Use a top-level "src" directory (we may want to reserve "src" on PyPI)
> > 2. Use a top-level "tools" directory (we may want to reserve "tools" on
> > PyPI)
> > 3. Add a leading or trailing underscore to the local directory name
> > (as while that's legal for Python imports, it's prohibited for PyPI
> > project names, and hence will often sidestep naming conflicts with
> > published packages)
> >
> > Beyond that, the only approaches I'm aware of that systematically
> > avoid this kind of problem at the language design level are to either
> > use URL-based imports (ala Java or Go), or else to have separate
> > syntax for "system-only" and "local resolution permitted" imports (ala
> > C and C++), and Guido opted not to pursue either of those strategies
> > for Python.
> >
> > Cheers,
> > Nick.
> >
> > --
> > Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Daniel Holth]

Yes vendoring would be a solution but it's too soon to say whether the
problem is significant. Setuptools and pip have a different problem which
is that the package manager isn't available yet. In pep517 the package
manager is available.

On Sat, Jul 29, 2017, 17:50 Alex Grönholm <alex.gronh...@nextday.fi> wrote:

> Daniel Holth kirjoitti 30.07.2017 klo 00:48:
>
> I think the proposal is that flit depends on click depends on flit and
> neither one has a wheel and must be built from sdists. Then you have a
> circular build problem. So don't do that. I put this in the same category
> as accidentally conflicting with a stdlib module; it is confusing when it
> happens but it's also fairly avoidable.
>
> Sure but vendorizing the dependencies would work around the problem, yes?
> Like how setuptools does?
>
>
> On Sat, Jul 29, 2017, 17:38 Alex Grönholm <alex.gronh...@nextday.fi>
> wrote:
>
>> Donald Stufft kirjoitti 29.07.2017 klo 23:47:
>>
>>
>> On Jul 29, 2017, at 12:46 PM, Nathaniel Smith <n...@pobox.com> wrote:
>>
>> I guess the most obvious example of when this would occur is: suppose
>> click switches to using flit for builds, and then flit switches to using
>> click for command line parsing. Now there's a bit of a chicken and egg
>> problem where 'pip install click' will end up importing flit with the click
>> source tree on the path, and this tree of course contains a directory named
>> 'click', so unless special measures are taken flit will end up importing
>> the code it's trying to build.
>>
>> But of course this can happen for lots of reasons; most packages have
>> names that you wouldn't expect to randomly encounter at the root of a
>> source tree very often, but with 100,000+ packages on pypi I expect it will
>> happen eventually.
>>
>> This doesn't happen with setuptools because setuptools traditionally has
>> few or no dependencies, but obviously we're changing that; the whole idea
>> here is that now your build system has full access to pypi.
>>
>>
>>
>> This is something to be discouraged anyways, because it creates a sort of
>> broken situation (the same situation that setuptools itself had). The
>> problem is that if you’re starting from only sdists, you have a circular
>> dependency that cannot be broken. You can’t build click, because  click
>> requires flit, but you can’t install flit, because flit requires click. The
>> only way to fix this is to either have an already built wheel that you can
>> use (which obviously was either built with a flit that didn’t require
>> click, or a click that didn’t require flit, or it’s provenance can be
>> traced back to that) or do some hacks that will hopefully resolve the
>> situation good enough to get your first wheel built.
>>
>> Setuptools tried to depend on things, and it broke shit for a lot of
>> people because of this. You basically can’t depend on anything as a build
>> system that uses you as a build system. You can only depend on things that
>> use other, different build systems in the entire dependency tree. Likely
>> the best thing for build systems to do is either have no dependencies, or
>> to have minimal dependencies that promise to only use setuptools (or
>> another build tool, which one doesn’t matter, just as long as it has no
>> dependencies) forever (and have setuptools or this other build tool promise
>> to never take a dependency).
>>
>> Or vendorize their dependencies? To me it seems unrealistic for a build
>> system to have no dependencies at all. Or perhaps this is exactly what you
>> meant :)
>>
>>
>> —
>> Donald Stufft
>>
>>
>>
>>
>>
>> ___
>> Distutils-SIG maillist  -  
>> Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Daniel Holth]

Pip has to vendor its dependencies primarily because it can't pip install
them. Not the same problem as build systems.

On Sat, Jul 29, 2017, 18:00 Alex Grönholm <alex.gronh...@nextday.fi> wrote:

> Daniel Holth kirjoitti 30.07.2017 klo 00:58:
>
> Yes vendoring would be a solution but it's too soon to say whether the
> problem is significant. Setuptools and pip have a different problem which
> is that the package manager isn't available yet. In pep517 the package
> manager is available.
>
> What package manager are you referring to?
>
>
> On Sat, Jul 29, 2017, 17:50 Alex Grönholm <alex.gronh...@nextday.fi>
> wrote:
>
>> Daniel Holth kirjoitti 30.07.2017 klo 00:48:
>>
>> I think the proposal is that flit depends on click depends on flit and
>> neither one has a wheel and must be built from sdists. Then you have a
>> circular build problem. So don't do that. I put this in the same category
>> as accidentally conflicting with a stdlib module; it is confusing when it
>> happens but it's also fairly avoidable.
>>
>> Sure but vendorizing the dependencies would work around the problem, yes?
>> Like how setuptools does?
>>
>>
>> On Sat, Jul 29, 2017, 17:38 Alex Grönholm <alex.gronh...@nextday.fi>
>> wrote:
>>
>>> Donald Stufft kirjoitti 29.07.2017 klo 23:47:
>>>
>>>
>>> On Jul 29, 2017, at 12:46 PM, Nathaniel Smith <n...@pobox.com> wrote:
>>>
>>> I guess the most obvious example of when this would occur is: suppose
>>> click switches to using flit for builds, and then flit switches to using
>>> click for command line parsing. Now there's a bit of a chicken and egg
>>> problem where 'pip install click' will end up importing flit with the click
>>> source tree on the path, and this tree of course contains a directory named
>>> 'click', so unless special measures are taken flit will end up importing
>>> the code it's trying to build.
>>>
>>> But of course this can happen for lots of reasons; most packages have
>>> names that you wouldn't expect to randomly encounter at the root of a
>>> source tree very often, but with 100,000+ packages on pypi I expect it will
>>> happen eventually.
>>>
>>> This doesn't happen with setuptools because setuptools traditionally has
>>> few or no dependencies, but obviously we're changing that; the whole idea
>>> here is that now your build system has full access to pypi.
>>>
>>>
>>>
>>> This is something to be discouraged anyways, because it creates a sort
>>> of broken situation (the same situation that setuptools itself had). The
>>> problem is that if you’re starting from only sdists, you have a circular
>>> dependency that cannot be broken. You can’t build click, because  click
>>> requires flit, but you can’t install flit, because flit requires click. The
>>> only way to fix this is to either have an already built wheel that you can
>>> use (which obviously was either built with a flit that didn’t require
>>> click, or a click that didn’t require flit, or it’s provenance can be
>>> traced back to that) or do some hacks that will hopefully resolve the
>>> situation good enough to get your first wheel built.
>>>
>>> Setuptools tried to depend on things, and it broke shit for a lot of
>>> people because of this. You basically can’t depend on anything as a build
>>> system that uses you as a build system. You can only depend on things that
>>> use other, different build systems in the entire dependency tree. Likely
>>> the best thing for build systems to do is either have no dependencies, or
>>> to have minimal dependencies that promise to only use setuptools (or
>>> another build tool, which one doesn’t matter, just as long as it has no
>>> dependencies) forever (and have setuptools or this other build tool promise
>>> to never take a dependency).
>>>
>>> Or vendorize their dependencies? To me it seems unrealistic for a build
>>> system to have no dependencies at all. Or perhaps this is exactly what you
>>> meant :)
>>>
>>>
>>> —
>>> Donald Stufft
>>>
>>>
>>>
>>>
>>>
>>> ___
>>> Distutils-SIG maillist  -  
>>> Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Daniel Holth]

What you would do is put your project in src/ and choose a build system
that allows the same, if you happen to be developing such a conflicting
project.

On Sat, Jul 29, 2017 at 1:29 PM Paul Moore  wrote:

> On 29 July 2017 at 17:46, Nathaniel Smith  wrote:
> > So one consequence of this is that every time a new release of flit (for
> > example) adds a new dependency on X, or one of flit's dependencies adds a
> > new dependency on X, then this is technically a backwards incompatible
> > change, because any existing packages that use flit and happen to have a
> top
> > level directory named X will stop working.
> [...]
> > Or am I worrying about a non-issue and it's fine if flit imports click
> from
> > the source tree?
>
> This sounds like a pretty rare issue, and one that I'd be inclined to
> assume isn't going to be a problem in practice - but if it were
> considered worth worrying about, why not just put the project root at
> the *end* of sys.path, so that packages installed normally will take
> precendence?
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Daniel Holth]

On Fri, Jul 28, 2017 at 7:12 PM Nathaniel Smith  wrote:

> On Fri, Jul 28, 2017 at 1:58 PM, Thomas Kluyver 
> wrote:
> > I think this was actually a question that someone (Nathaniel?) brought
> up:
> > is the project source directory on sys.path when the backend is loaded? I
> > would lean yes, so that it's possible to have a custom backend in the
> > project tree, but I think whoever brought it up before was leaning no.
>
> You're probably thinking of
>
> https://github.com/njsmith/peps/blob/90f7adc130cbae741b1abd3a9ad407ba6731ea9f/pep-0517.txt#L976-L987
>
> As noted there, my suggestion would be that we don't put the project
> directory on sys.path by default, but that we do provide an explicit
> configuration key that lets projects name a directory in the project
> tree that should be prepended to sys.path. So example:
>
> [build-system]
> python-path = ["."]
> build-backend = "my_custom_backend"# refers to
> my_custom_backend.py in the project root
>
> # - or -
>
> [build-system]
> python-path = ["tools"]
> build-backend = "my_custom_backend"# refers to
> tools/my_custom_backend.py
>
> Rationale:
>
> - Having the project source directory on sys.path when you don't
> expect it is confusing and error-prone
> - If you do want it on sys.path then it's easy to add a line to
> pyproject.toml, explicit > implicit
> - I think it's an anti-pattern how every piece of tooling wants to
> claim part of the top-level directory (.travis.yml, appveyor.yml,
> .readthedocs.yml, CONTRIBUTING.yml, ...) -- pyproject.toml has to go
> there because it's the root where all our tooling starts from, but we
> should support projects that want to tuck away the rest of their
> tooling into some subdirectory :-)
>
> Given how much trouble we're having with PEP 517 already, it might
> make more sense to have PEP 517 just mandate that the directory not be
> on sys.path, and make a little follow-up PEP just for python-path.
>

But can you not also make bridges out of stone?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Daniel Holth]

Yes, the cwd strongly should be on sys.path, so many possibilities, and we
are used to that behavior in setup.py.

On Fri, Jul 28, 2017 at 4:59 PM Thomas Kluyver <tho...@kluyver.me.uk> wrote:

> On Fri, Jul 28, 2017, at 09:37 PM, Daniel Holth wrote:
>
> See how trivial it would be to put the delegated sdist generator into the
> build-backend within the confines of the current PEP?
>
>
> I agree that delegation within the backend is a reasonable answer to this.
>
> The build-backend could point to a .py in the current directory that
> implements itself with different tools,
>
>
> I think this was actually a question that someone (Nathaniel?) brought up:
> is the project source directory on sys.path when the backend is loaded? I
> would lean yes, so that it's possible to have a custom backend in the
> project tree, but I think whoever brought it up before was leaning no.
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Daniel Holth]

See how trivial it would be to put the delegated sdist generator into the
build-backend within the confines of the current PEP? The build-backend
could point to a .py in the current directory that implements itself with
different tools, or a delegating build backend could parse a separate
source-backend out of pyproject.toml for kicks. So why worry?

On Fri, Jul 28, 2017 at 4:30 PM Leonardo Rochael Almeida <
leoroch...@gmail.com> wrote:

> Hi Thomas,
>
> On 28 July 2017 at 16:53, Thomas Kluyver  wrote:
>
>>
>> [...] I have a nagging concern about something that someone mentioned
>> ages ago: does it make sense for building sdists and building wheels to be
>> part of the same backend?
>>
>> [...]
>>
>
>> So I'd like us to circle back round and reconsider allowing projects to
>> specify 'use tool X to make wheels, and tool Y to make sdists'. If everyone
>> else thinks that's unnecessary, I think we'd all be glad to finish this
>> discussion up, but this concern has been growing in my mind for a while,
>> and I want to get it out there before we finalise the PEP.
>>
>
> Great, now you've planted your nagging concern on my mind as well ;-)
>
> What would using different backends for sdist and wheel look like?
> Something like this?
>
> # pyproject.toml
> [build-system]
> requires = ["sdister", "wheeler"]
> source-backend = "sdister.api:main"
> build-backend = "wheeler.api:main"
>
> If it's something like the above (or even, if it's a different section
> (`source-system`? I'm trying hard to avoid the `sdist` word), then perhaps
> this could be developed in a future pep without much issue. considering
> what Daniel just reminded us about tools being able to claim ignorance on
> how to develop sdists.
>
> But if it's also that simple, maybe it could be added to this pep (/me
> ducks).
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Daniel Holth]

I'm not worried about it. If you can generate sdist the file format is
simple, if you cannot then the build tool asks you for a wheel. Build tools
can have dependencies too and could depend on a nifty sdist generating
library. It seems to me that the whole spec lets you start by figuring out
how to build wheels, and then proceed to implement the rest in order for
your project to be lauded by PyPA. Thus build systems of differing
complexity and distribution are accommodated.

On Fri, Jul 28, 2017 at 3:53 PM Thomas Kluyver <tho...@kluyver.me.uk> wrote:

> On Fri, Jul 28, 2017, at 04:16 PM, Daniel Holth wrote:
>
> It looks like we've run out of things to say about PEP 517, except, how
> soon can we get it into pip?
>
>
> I admire your optimism! ;-)
>
> While I partly hope that I get a unanimous disagreement, as it would be
> simpler, I have a nagging concern about something that someone mentioned
> ages ago: does it make sense for building sdists and building wheels to be
> part of the same backend?
>
> Flit now makes both sdists and wheels, but for a long time it only made
> wheels, and the two parts are largely separate: it wouldn't take much work
> to use flit's sdist machinery but build the wheel with a different tool
> (e.g. if it had compiled parts).
>
> Requiring one backend to build both formats may result in a significantly
> higher barrier to entry for backend developers: I don't know if I would
> have started writing flit if PEP 517 had already been finalised and I had
> to make both wheels and sdists to comply with it. They're also, at least to
> my mind, quite different kinds of thing: an sdist is almost like an archive
> of a VCS tag, whereas a wheel is the end result of any build steps the
> project needs.
>
> So I'd like us to circle back round and reconsider allowing projects to
> specify 'use tool X to make wheels, and tool Y to make sdists'. If everyone
> else thinks that's unnecessary, I think we'd all be glad to finish this
> discussion up, but this concern has been growing in my mind for a while,
> and I want to get it out there before we finalise the PEP.
>
> Thomas
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] status check on PEP 517

[Daniel Holth]

It looks like we've run out of things to say about PEP 517, except, how
soon can we get it into pip? These function signatures will serve us well,
significantly lowering the barrier to entry for new pip-compatible build
systems.

On Thu, Jul 20, 2017 at 3:36 PM Brett Cannon  wrote:

> On Thu, 20 Jul 2017 at 03:56 Nathaniel Smith  wrote:
>
>> On Thu, Jul 20, 2017 at 3:22 AM, Paul Moore  wrote:
>> > On 20 July 2017 at 10:46, Nathaniel Smith  wrote:
>> >> To make this concrete: I'm *pretty* sure (?) that at this point all
>> >> the basic elements in my "simplified" rewrite are things that we now
>> >> have consensus are needed in some form, so maybe we can use that as a
>> >> kind of "minimal core" reference point:
>> >>
>> >>
>> https://github.com/njsmith/peps/commits/517-refactor-streamline/pep-0517.txt
>> >
>> > I don't have time this week to look at this, sorry, but a couple of
>> points:
>> >
>> > 1) This links to a commit in your repo, not to the actual document.
>>
>> Ugh, sorry! Pasted the wrong link. This is the correct one:
>>
>>
>> https://github.com/njsmith/peps/blob/517-refactor-streamline/pep-0517.txt
>>
>> > Also, I'd *really* prefer a rendered version if I'm going to review
>> > this.
>>
>> Annoyingly, the PEPs repo uses a .txt extension for .rst files, thus
>> cleverly preventing github from doing anything useful by default...
>>
>
> We will happily accept help in converting plaintext PEPs into reST ones so
> we can universally change the file extensions to .rst (see
> https://github.com/python/peps/issues/1) :)
>
> -Brett
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintenance of the "wheel" package

[Daniel Holth]

Fine by me, probably inevitable, obviously I'm not a fan of git. Please
take care of the useful pending pull requests on bitbucket first.

On Tue, Jul 25, 2017 at 9:05 AM Alex Grönholm 
wrote:

> Looking at the commit history, the maintenance of "wheel" has more or less
> stalled since 2016. I would like to volunteer myself as a co-maintainer of
> this project, if allowed. Here's my list of things I would like to do with
> it:
>
>- Move the project from Bitbucket to the PyPA organization on Github
>- Add a setup() keyword (and documentation for it) for license_file
>(currently only usable via the deprecated [metadata] section in setup.cfg)
>- Refactor the test suite to make better use of py.test
>- Clean up code style for PEP 8 compliance
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] requirements for variable python

[Daniel Holth]

Do you just need a conditional dependency?

  extras_require={  ':python_version=="2.7"': ['functools32'],


On Wed, Jul 19, 2017 at 11:20 AM Robin Becker  wrote:

> Is there a way to build a universal wheel which needs an additional module
> when
> python 2.7 is being used.
>
> Currently the run requirements for the module are
>
> python-dateutil
> jsonschema
>
> but in addition jsonschema needs functools32 for python 2.7.
>
> Is there a way to get the extra requirement used when python 2.7 is the
> installer? Or does this package not qualify for universalness.
> --
> Robin Becker
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

Yes, build_directory does not do the same thing as sdist -> unpack ->
build. It would be more likely to be useful if you are not going through
the sdist step for any reason on a tree that has been built before.

The assumption being that the back end might use src/*.py as its source,
but build/** as the source for the wheel, so that any extra files that wind
up in build/ are automatically included in the wheel. bdist_wheel does
this. SCons does not work this way, it keeps track of all the files that
should be in build/, adds only those files to the wheel, and ignores all
extra files in build/.

Perhaps you have a system that uses 'hg manifest' to create the sdist and
src/*.py to build, but you forgot to add src/newfile.py to version control.
Then sdist -> unpack -> build would tell you something.

Can we imagine a situation where a built file shows up in src/ and the
wheel but not in the sdist?

Seems like you will need to show the backend's stdout on an error whatever
the hook's return value is.

Let's say you are not using 'pip install .' for development, then you get
to validate your release artifacts some other way.

On Mon, Jul 17, 2017 at 9:54 AM Nathaniel Smith <n...@pobox.com> wrote:

> On Mon, Jul 17, 2017 at 6:36 AM, Daniel Holth <dho...@gmail.com> wrote:
> > Here's my own summary.
> >
> > pip can do build_sdist -> unpack sdist -> build wheel if it wants to,
> > serving as some kind of linter if you happen to run 'pip install .'
> during
> > development.
> >
> > build_directory provides a way to ask for a clean build, the lack of
> which
> > causes problems in bdist_wheel sometimes. It is one way to try harder to
> > generate a correct wheel, not try harder to generate a correct sdist.
>
> I would say -- build_directory provides a way to ask that the source
> tree be left clean for next time. This is unfortunately not quite the
> same. From pip's point of view, the advantages of build_sdist are
> twofold: (1) it validates the sdist building path, (2) it really does
> give a clean build.
>
> This is discussed more here:
> https://mail.python.org/pipermail/distutils-sig/2017-July/031020.html
>
> (I know you've read it, but putting the link in for future archive readers
> etc.)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Notes on pip's requirements for PEP 517

[Daniel Holth]

A minor detail for pip strategy option #2 is that sdists do not have to
have PKG-INFO.

On Mon, Jul 17, 2017 at 9:02 AM Nathaniel Smith  wrote:

> Hi all,
>
> I happened to talk to Donald on IRC last night, and said I'd write up
> some notes on that discussion. (He's also seen these notes, but is
> partially-offline dealing with jury duty.) [Edit: but apparently still
> replying to email on the list, so some of this repeats things he's
> posted since I sent it to him to look at, but oh well :-). That's life
> in the fast-paced world of F/OSS development I guess.]
>
> One thing we tried to get a handle on is what, exactly, are
> constraints that PEP 517 is trying to deal with.
>
> As far as pip goes, these are some of the situations that Donald's
> worried about handling gracefully when it comes to 'pip install .':
>
> a) The source tree is on read-only media
>
> b) People do an editable install in a source tree, and then follow it
> up with a regular install ('pip install -e . && pip install .').
> Apparently this can cause problems because of the residual metadata
> from the editable install confusing setuptools.
>
> c) Someone does 'pip install .' on a complex package, then discovers
> that while the build succeeded, it's missing some optional feature
> because some necessary system library was missing, so they install
> that library and re-try the 'pip install .'. Will the build backend
> notice that the library has appeared, or does it only do environment
> sniffing the first time it's run in a given tree?
>
> d) Mounting the same source tree into multiple different docker
> containers: 'docker run ubuntu -v .:/io pip install /io && docker run
> alpine -v .:/io install /io'. Will the build backend notice that these
> two systems have incompatible C toolchains and C ABIs, so you can't
> share .o files between them? In principle a build system could notice
> this by detecting that the compiler and system headers have changed,
> but in traditional build systems it's common to intentionally *not*
> include these in your build cache key, because you don't want to
> rebuild the world after every apt upgrade. (For example, gcc provides
> the -M switch to generate make rules to trigger rebuilds on header
> changes, and the -MM switch to do the same but ignoring changes in
> system headers; lots of projects choose to use -MM. This is related to
> the thing where incremental builds are traditionally a bit sloppy and
> intended for experts only.)
>
> e) Build systems that have different sources of truth for what goes
> into an sdist versus what goes into a wheel, and thus can easily end
> up in a situation where a direct VCS->wheel build gives a different
> result than a VCS->sdist->wheel build. The most prominent offender
> here is distutils/setuptools with its MANIFEST.in mess, but Donald is
> nervous that other systems might also reproduce this mistake.
>
> f) And finally Donald feels "it's just more hygenic to have ``pip
> install .`` not modify ., similiarly to how [he] would be upset if
> ``pip install foo-1.0.tar.gz`` modified foo-1.0.tar.gz in some way".
>
> Of course, no system can avoid every problem; the overall goal here is
> harm reduction and minimizing spurious bugs filed on pip, not
> perfection.
>
> None of these cases arise for 'pip install name' or 'pip install
> sdist.tar.gz'; it's really only 'pip install .' on an user-provided
> source tree.
>
> --
>
> For reference, here's my analysis of how these particular desiderata
> relate to some possible approaches:
>
> - Pip's current system for handling builds from existing source trees
> (copytree + setup.py bdist_wheel -or- setup.py install) handles (a),
> (c), (d), (f), but not (b) or (e), unless someone has previously done
> an in-place build in the tree, in which case it handles (a) and (f)
> but not (b), (c), (d), or (e). Which is kind of unfortunate, since (a)
> and (f) are probably the least important items.
>
> - When sdist->unpack->wheel is possible, it automatically handles all
> of these cases.
>
> - If a build backend *only* does in-place builds (like meson) and
> *does not support* in-place or editable installs, then having an
> out-of-place build_wheel hook automatically takes care of everything
> except (e).
>
> - Otherwise, an out-of-place build_wheel hook acts handles (a), (c),
> (d), (f) but not (b) or (e), unless someone has previously done an
> in-place build in the tree, in which case it handles (a) and (f) but
> not (b), (c), (d), or (e) (assuming that the build system can get
> confused between artifacts left behind by in-place builds when doing
> an out-of-place build, which appears to be a common feature of
> existing systems).
>
>   Notice that this means that as far as this score card is concerned,
> out-of-place build_wheel is identical to pip's current copytree +
> in-place build_wheel. The key insight here is that an out-of-tree
> build is nicer for the *next* person to use this source tree, 

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

Here's my own summary.

pip can do build_sdist -> unpack sdist -> build wheel if it wants to,
serving as some kind of linter if you happen to run 'pip install .' during
development.

build_directory provides a way to ask for a clean build, the lack of which
causes problems in bdist_wheel sometimes. It is one way to try harder to
generate a correct wheel, not try harder to generate a correct sdist.

An important class of "can't build an sdist" problems happen in an unpacked
sdist, when the build system expects VCS metadata to get the sdist
manifest. Perhaps you are patching a dependency and don't need to do new
source releases in this case, but it would be really annoying if 'pip
install .' refused to work. You can probably get a source distribution when
you really need one, i.e. during ordinary development of your own package
from a VCS.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

I agree that sdist consistency is not enforceable. Very little is. What if
we deleted every unenforceable part of the PEP? No explanations of what
backends should do. Every parameter is a hint. If you put the output file
where requested then you are a good back end. Would that work better?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

I proposed the build directory parameter because the copytree hook made no
sense to me. It is not a perfect substitute but perhaps a configurable
build directory is nice on its own without having to satisfy all older
arguments in favor of copytree. I think true in-place builds are the
oddball (for example 2to3 or any build where sources have the same name as
outputs needs a build directory to put the translated .py files, otherwise
it would overwrite the source). What people think of as in-place builds in
distutils are usually just builds into the default build directory.

IIRC the main reason we are worried sdist generation would fail is when you
are trying to do new development from an unpacked sdist. Suggest not
worrying that this use case is a perfect experience. It's good enough for
quick patching, but if you need to do a new source release then you
probably have time to meet the requirements of your build backend.

Surely the pluggable build systems will compete on reliability, rather than
repeatedly making MANIFEST.in errors you will be able to just switch
build-backend.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

The PEP process is awful. Sorry. Thank you for helping to make this happen.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

On Tue, Jul 11, 2017 at 9:58 AM Thomas Kluyver  wrote:

> To summarise the current state of this discussion as I understand it,
> the hooks we are currently thinking of defining are:
>
> def get_build_wheel_requires(config_settings)
> def prepare_wheel_metadata(metadata_directory, config_settings)
> def build_wheel(wheel_directory, config_settings, build_directory=None,
> metadata_directory=None)
>
> def get_build_sdist_requires(config_settings)
> def build_sdist(sdist_directory, config_settings[,
> build_directory=None])  # Last param under discussion
>
> I know we were also discussing a different naming scheme, but I've
> forgotten what names were proposed, and don't have time to find the
> relevant email again. Could someone produce a copy of the list above
> with the proposed names substituted?
>

Pro_duced.

Mandatory backend hooks:

- build_sdist(sdist_directory, config_settings={})
- build_wheel(wheel_directory, config_settings={}, build_directory=None,
metadata_directory=None)

Optional backend hooks:

- get_requires_for_build_sdist(config_settings={})
- get_requires_for_build_wheel(config_settings={})
- prepare_metadata_for_build_wheel(metadata_directory, config_settings={})
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

On Mon, Jul 10, 2017, 10:58 C Anthony Risinger 
wrote:

> On Jul 10, 2017 8:59 AM, "Nick Coghlan"  wrote:
>
> On 10 July 2017 at 22:08, Jeremy Stanley  wrote:
> > On 2017-07-10 20:33:16 +1000 (+1000), Nick Coghlan wrote:
> > [...]
> >> we don't really *want* folks to be adding generated files to their
> >> sdist that they aren't keeping under source control - we'd prefer
> >> that such activities were postponed to "build_wheel" now that we
> >> have separate source and precompiled distribution formats.
> > [...]
> >
> > This is a mildly naive view. The software I'm familiar with is
> > actually attempting to reflect metadata _from_ the source revision
> > control _into_ the sdist because while it's "tracked" there it's not
> > tracked as normal files (version information from tags, change logs
> > from the commit history, contributor lists from commit authorship).
> > The metadata in question is lost by just blindly tarring up tracked
> > files, so needs some mechanism to export and inject as untracked
> > files (from the source revision control perspective) for inclusion
> > in the sdist.
>
> Right, we know there will be *some* generated files (as Daniel notes,
> there will typically at least be the metadata files in a
> setuptools/distutils generated sdist), as well as potentially
> modifications to some files based on the version history.
>
> That isn't the question though - the question is whether we want to
> actively support folks moving "compilation" like activities
> (minification, pyx->C conversion, etc) to the sdist generation stage
> by adding the optional "build_directory" option to "build_sdist" as
> well.
>
> And that's the part where we decided the answer is "No", we only want
> to support the following configurations:
>
> build_sdist:
> working directory -> target directory
>
> build_wheel:
> working directory -> target directory
> working directory -> build directory -> target directory
>
> In all cases the frontend provides a target directory that is distinct
> from the current working directory, so backends have a place to put
> both generated intermediate artifacts *and* the final assembled
> archive file.
>
>
> When preparing a redistributable archive, we don't want people to first
> generate difficult or inconvenient artifacts? I always thought that was a
> major feature of an archive, to reduce the content down to common
> denominators for verification, reproducibility, and build simplicity, at
> the expense of not being fully representative of the original build
> capabilities and likely an irreversible step.
>
> I know there's been a lot of discussion here, and I probably missed it,
> but early on we talked about things like Cython and cffi being part of
> sdist generation (in setuptools at least).
>
> Are these things now expected to be deferred to the wheel building stage,
> thus adding deps to the final build server, or are we just saying there is
> less utility in supporting an explicit "artifacts directory" during sdist
> generation?
>

Just that there is less utility in supporting a separate artifacts
directory during sdist generation, correct.

If the latter (and I think this is the case), that does seem reasonable, I
> just wanted to confirm that such "compliations" are still expected, or at
> least permissible, in downstream redistributables (sdists).
>

Personally if you want to do I want to help make it easier to do.

-- 
>
> C Anthony
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

A huge benefit to using non distutils build systems is making it easy to
generate files at any step. I just don't think it's worth it to force a
particular build directory at the generate sdist phase. Foolish
consistentcy.

On Mon, Jul 10, 2017, 10:20 Paul Moore  wrote:

> On 10 July 2017 at 14:58, Nick Coghlan  wrote:
> > That isn't the question though - the question is whether we want to
> > actively support folks moving "compilation" like activities
> > (minification, pyx->C conversion, etc) to the sdist generation stage
> > by adding the optional "build_directory" option to "build_sdist" as
> > well.
> >
> > And that's the part where we decided the answer is "No", we only want
> > to support the following configurations:
>
> I'm not sure I follow this comment (or if I do, I don't agree with it
> :-)). I would expect projects that use Cython to have the step to
> build the C files from the Cython sources as part of the "build sdist"
> step, so that the sdist contains standard C sources, and end users can
> build from sdist with a C compiler installed, but without needing to
> install Cython. That's the standard approach these days, and I'd hope
> it will be supported under PEP 517.
>
> Paul
>
> PS Completely off-topic, but since when has gmail's web interface
> stopped allowing you to highlight a section of a mail and hit "Reply"
> to get just that section quoted? It seems to have changed very
> recently, or is it some setting change I might have accidentally made?
> It's intensely annoying, as it makes it much more time consuming to
> avoid top-posting :-( (and please, no suggestions that I use an
> alternative client, my usage pattern makes that impractical).
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

Sdists contain generated PKG-INFO and .egg-info.

I'd prefer to let the build backend manage any sdist build directory. It
doesn't provide the same benefit to pip as the configurable wheel build
directory.

On Mon, Jul 10, 2017, 08:09 Jeremy Stanley  wrote:

> On 2017-07-10 20:33:16 +1000 (+1000), Nick Coghlan wrote:
> [...]
> > we don't really *want* folks to be adding generated files to their
> > sdist that they aren't keeping under source control - we'd prefer
> > that such activities were postponed to "build_wheel" now that we
> > have separate source and precompiled distribution formats.
> [...]
>
> This is a mildly naive view. The software I'm familiar with is
> actually attempting to reflect metadata _from_ the source revision
> control _into_ the sdist because while it's "tracked" there it's not
> tracked as normal files (version information from tags, change logs
> from the commit history, contributor lists from commit authorship).
> The metadata in question is lost by just blindly tarring up tracked
> files, so needs some mechanism to export and inject as untracked
> files (from the source revision control perspective) for inclusion
> in the sdist.
> --
> Jeremy Stanley
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

On Mon, Jul 10, 2017, 04:54 Steve Dower  wrote:

> One nice thing about providing a “put your work in this directory” setting
> for all tasks is that only the front end has to know how and where to
> create it, and how and when to clean it up later. Users may want to
> configure this across all projects, regardless of the backend in use.
>
>
>
> Permitting this directory to be the source tree implicitly requires
> backends to support “in place” builds (i.e. you should put output files in
> a matching structure under that directory in case it really is the source
> tree). In this case, front ends need to be responsible for (not) running
> rmtree and backends should not blindly delete everything (or else they’ll
> get bug reports from very upset users).
>
Not sure about this particular idea. It might have to be a special case and
implementation in the backend driver, usually build in a directory and copy
file into the source tree.


> Cheers,
>
> Steve
>
>
>
> Top-posted from my Windows phone at EuroPython
>
>
>
> *From: *Thomas Kluyver 
> *Sent: *Monday, July 10, 2017 9:14
> *To: *distutils-sig@python.org
>
>
> *Subject: *Re: [Distutils] A possible refactor/streamlining of PEP 517
>
>
>
> On Mon, Jul 10, 2017, at 07:01 AM, Nick Coghlan wrote:
>
> > So I think we have pretty solid evidence that the reason the
>
> > procedural "build directory preparation" hook wasn't sitting well with
>
> > people was because that isn't the way build systems typically model
>
> > the concept, while a "build directory" setting is very common (even if
>
> > that "setting" is "the current working directory when configuring or
>
> > running the build").
>
>
>
> Hooray! :-)
>
>
>
> Do we want to also provide a build_directory for the build_sdist hook?
>
> In principle, I don't think making an sdist should involve a build step,
>
> but I know that some projects do perform steps like cython code gen or
>
> JS minification before making the sdist. I think this was a workaround
>
> to ease installation before wheel support was widespread, and I'd be
>
> inclined to discourage it now, so my preference would be no
>
> build_directory parameter for build_sdist. Backends which insist on
>
> generating intermediates at that point can make a temp dir themselves.
>
>
>
> Then I guess that the choice between building a wheel directly and
>
> attempting to build an sdist first (with direct fallback) is one for
>
> frontends, and doesn't need to be specified.
>
>
>
> Thomas
>
> ___
>
> Distutils-SIG maillist  -  Distutils-SIG@python.org
>
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distutils question

[Daniel Holth]

Great, thanks! It looks like out of tree builds are supported fine even in
distutils. Properly written extensions will also find the build directory
as a property of the build command. There should be no barrier to using the
single build wheel function for the pep.

On Sun, Jul 9, 2017, 10:33 Jeremy Kloth <jeremy.kl...@gmail.com> wrote:

> On Sat, Jul 8, 2017 at 9:09 PM, Daniel Holth <dho...@gmail.com> wrote:
>
>> Unrelated to pep 517, remind me whether when invoking setup.py build -b
>> dir bdist, if the -b argument passed to the build command is supposed to
>> affect the build command run as a subcommand of bdist? Asking for a friend
>> 
>>
>
> "setup.py build -b {dirname} bdist" will first run the build command (and
> its sub commands build_*) then invoke the bdist command.  When bdist
> requests build to be executed, distutils will see that it is already run
> and simply continue on.  Even if a reinit is requested (for build), the
> supplied options will be applied again when the build command is finalized
> prior to running.
>
> --
> Jeremy Kloth
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Distutils question

[Daniel Holth]

Unrelated to pep 517, remind me whether when invoking setup.py build -b dir
bdist, if the -b argument passed to the build command is supposed to affect
the build command run as a subcommand of bdist? Asking for a friend 

On Sat, Jul 8, 2017, 11:17 Jeremy Kloth <jeremy.kl...@gmail.com> wrote:

> On Fri, Jul 7, 2017 at 9:45 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> > On 7 July 2017 at 23:23, Daniel Holth <dho...@gmail.com> wrote:
> >> FYI distutils supports out of tree builds too. It is the -b argument to
> >> 'setup.py build'.
> >
> > Sort of. That's short for "--bdist-dir" and tells distutils/setuptools
> > not to use the "dist/" subdirectory for either build trees or the
> > build artifacts. It doesn't say anything about where intermediate
> > artifacts generated by compilers etc should end up.
>
> No, Daniel is correct.  "setup.py *build* -b" is short for
> --build-base which is where all build artifacts go, even those from
> compilers.  It seems you are confusing it with "setup.py *bdist* -b"
> which is indeed short for "--bdist-dir".
>
> --
> Jeremy Kloth
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

On Fri, Jul 7, 2017 at 9:23 AM Daniel Holth <dho...@gmail.com> wrote:

> FYI distutils supports out of tree builds too. It is the -b argument to
> 'setup.py build'.
>

And it works in bdist_wheel by adding half a dozen lines. It copies the -b
argument to the 'build' subcommand, so effectively you can put the ./build/
directory wherever you want. This will probably also solve other problems
people sometimes have with 'unclean' bdist_wheel builds. As a bonus it no
longer works in Python 2.6.

+1 on Thomas' added wording.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

FYI distutils supports out of tree builds too. It is the -b argument to
'setup.py build'.

On Fri, Jul 7, 2017 at 7:08 AM Paul Moore  wrote:

> Also, please note that the proposal doesn't *prohibit* in-place
> builds, quite the opposite, it allows backends to decide when asked
> how to implement both in-place and out of place builds (where the
> current tree allows backends to decide how to do in place builds and
> how to copy trees, and leaves the frontend to decide how to implement
> out of place builds, typically via something like a tree copy and
> subsequent in-place build).
>
> On 7 July 2017 at 12:05, Paul Moore  wrote:
> > On 7 July 2017 at 11:30, Nathaniel Smith  wrote:
> >> Is it absolutely necessary to get this into the first PEP?
> >
> > As far as I'm concerned, it's no more than a restating (and
> > simplification?) of all the discussions around building out of tree
> > via creating an sdist and unpacking it, or having the various prepare
> > files hooks. It's always been there and always been a requirement.
> >
> >> This proposal creates substantial complications for build systems that
> default to doing in-place builds
> >
> > Well, we're focused on build systems that will get a PEP 517
> > interface, so far the only concrete examples we have are flit (which
> > has no problem with this), enscons (which has no problem with it) and
> > probably setuptools, in some form (which needs out-of-tree builds
> > based on our experience with it in pip, although I concede that you
> > would argue that point).
> >
> > Also, out of tree builds are something that pip is planning on
> > implementing, and for us it's therefore necessary to have hooks that
> > let us do that. The current version of the PEP provides those hooks,
> > and we're planning on using them to do out of tree builds. This new
> > suggestion is basically backend authors saying "you don't need to do
> > the out of place builds, we're willing to take responsibility for
> > that". If you're saying that you're not happy with that, then all that
> > will end up happening is we revert to the previous approach, and pip
> > implements out of tree builds based on that (and I guess you have the
> > same argument all over again on the PR for the pip change...)
> >
> > Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

It might be more natural to pass a build directory for intermediate build
artefacts along with the wheel output directory to the build wheel hook.
This would remove pip from an awkward position of managing a copy step in
the middle of a build and would be more like out of tree builds in other
build systems. For example in automake you do out of tree builds by making
a new build directory and running the configure script from that directory
instead of the source directory. With a fresh directory old builds don't
get in the way.

On Thu, Jul 6, 2017, 15:37 Thomas Kluyver  wrote:

> On Thu, Jul 6, 2017, at 07:19 PM, Paul Moore wrote:
> > That's a good point - and provides a good contrast to my perspective
> > as a pip developer that *pip* gets issues raised that aren't really
> > pip's problem. I think it's in everyone's best interests to ensure
> > that the user's experience is as unambiguous as possible in saying
> > where any given problem lies.
>
> Working on Jupyter, I've also seen plenty of misattributed issues from
> people who think that we make the whole ecosystem they're using in the
> notebook. I'm all for making it as unambiguous as possible, but I also
> believe that in many cases it's impossible to be totally clear which
> part has gone wrong, especially for users unfamiliar with the stack. So
> my priority is to minimise user-facing failures, because we're all
> likely to get bug reports.
>
> > One thought occurs to me in that context - in my view, we should be
> > clearly presenting to the user that it's *pip's* role to do the
> > install, and flit's responsibility is to build wheels. I know that
> > flit includes an install command, but I view that as a temporary
> > workaround for the fact that PEP 517 isn't implemented yet. I'd be
> > interested to know if you agree with that.
>
> Yes-ish. There are three parts:
>
> - flit installfrom (e.g. to install from a Github repo): entirely a
> stopgap measure, I'm very happy to pass this responsibility over to pip.
> - flit install (local install): I'll probably recommend pip once that
> works, but may leave it in place a bit longer. It already works by
> building a wheel and asking pip to install it.
> - flit install --symlink (development install): stays around at least
> until there's a standardised approach for this that I'm happy with.
>
> > But essentially, we're promoting "pip install
> > " as the canonical install command, and "pip wheel
> > " as the canonical "build a wheel" command - backend
> > specific commands should be for specialised use only, as I see it.
>
> Depending on exactly what you mean by 'specialised'. I don't see flit as
> 'a PEP 517 backend', but rather 'a tool which provides a PEP 517
> backend'. I will continue to recommend that developers invoke flit
> directly to build and publish packages, but it should be transparent to
> typical downstream users.
>
> > Personally I wouldn't have a major problem with this, although I don't
> > think Donald would agree, as there's questions that he's raised around
> > potential inconsistencies between sdists and wheels built direct from
> > the source tree that are unanswered in this model. My biggest concern,
> > though, is that if we take this view, then it's critical that we have
> > a reliable and efficient means of *copying* source trees.
>
> So we have two alternative proposals for this bit:
>
> 1. Try to make an sdist, fall back to copytree if not supported
> + Provides some measure of built-in checking for the sdist problem
> + Reuses existing sdist machinery
> - Fallback may be slow
>
> 2. Separate hook for efficient copy
> + Single mechanism is more predictable than primary+fallback
> + Reliably efficient
> - Requires extra backend code
>
> I'm willing to implement the necessary for either (but preferably not
> both!). I think 2 is perhaps a bit more user friendly - it's not going
> to be inexplicably slow because you've hit the fallback case.
>
> > Tox may have more stringent requirements - currently it requires the
> > ability to build a sdist to install from, and I'm inclined to think
> > that this is a deliberate design choice rather than merely a
> > convenience. I'm guessing that no-one has particularly explored the
> > question of how tox would interact with flit-based projects yet?
>
> I don't think so.
>
> > Would
> > it be acceptable to say that tox only works on a full checkout with
> > VCS tools present (i.e, what flit needs to build a sdist) for
> > flit-based projects? I don't really know.
>
> I'd be willing to explore whether we can do better than that, but I see
> tox as a developer tool, so I wouldn't consider it a show-stopper if it
> required the presence of a VCS.
>
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org

[Distutils] how sdist generation works in enscons

[Daniel Holth]

Here's how sdist generation works in enscons.

enscons, a build tool that exists to prototype new Python packaging
features, is just a set of tools for SCons that makes it easier to generate
wheels and sdists. If targets with certain names exist (sdist, bdist_wheel)
then the provide setup.py shim and new PEP 517 wrapper will interoperate
with pip.

For enscons itself, the sdist build rule looks like

sdist = env.SDist(source=FindSourceFiles() + ['PKG-INFO', 'setup.py',
'README.rst', 'CHANGES'])env.Alias('sdist', sdist)


Enscons would be able to build its own sdist outside a repository, and it
would include the listed files plus everything that is used to build the
other targets (the wheel).

A different package runs 'hg manifest' in a subprocess to feed to
env.SDist(), a natural thing to do in this kind of build system. That one
would not be able to build another sdist from an unpacked sdist.

Enscons itself doesn't have a way to know whether sdist generation will
succeed other than trying to run 'SCons sdist' against the user provided
build script.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

-1 on _for_ ; why should a common prefix plus extra typing be any clearer
than common suffix? Or rearrange the words without _for_.

For the --config options, there's a double dash on the key and the value, I
found that confusing. I suppose the theory, not thoroughly explained, is
that the value is sent to the command line of the build backend. I think
"--build-option=toves=slithy" would be more clear.
--build-option="--build-option1"

A backwards compatible setup.py shim will have to convert
prepare_wheel_metadata
to .egg-info for pip, which is pretty easy.

I didn't anticipate get_requires_* to require running the whole (designed
for a single invocation per process) build backend. Suppose anything's
possible.

On Mon, Jul 3, 2017 at 10:04 AM Nick Coghlan  wrote:

> On 1 July 2017 at 20:53, Nathaniel Smith  wrote:
> > Hi all,
> >
> > I just attempted an experimental refactor/streamlining of PEP 517, to
> > match what I think it should look like :-). I haven't submitted it as
> > a PR to the PEPs repository yet since I don't know if others will
> > agree with the changes, but I've pasted the full text below, or you
> > can see the text online at:
> >
> >
> https://github.com/njsmith/peps/blob/517-refactor-streamline/pep-0517.txt
> >
> > and the diff at:
> >
> >
> https://github.com/python/peps/compare/master...njsmith:517-refactor-streamline?expand=1
> >
> > Briefly, the changes are:
> >
> > - Rearrange text into (hopefully) better signposted sections with
> >   better organization
>
> This is definitely needed, so thanks for that.
>
> > - Clarify a number of details that have come up in discussion (e.g.,
> >   be more explicit that the hooks are run with the process working
> >   directory set to the source tree, and why)
>
> This is again very helpful (we also need to be clearer on which sets
> of dependencies must be installed prior to running the various hooks)
>
> > - Drop prepare_wheel_metadata and prepare_wheel_build_files (for now);
> >   add detailed rationale for why we might want to add them back later.
>
> I want prepare_wheel_metadata there as a straightforward way for
> backends to expose the equivalent of "setup.py egg_info". Losing that
> would be a significant regression relative to the status quo, since
> pip relies on it during the dependency resolution process (and a
> backtracking resolver will be even more dependent on being able to do
> that efficiently).
>
> For the build preparation hook, Thomas already rejected promising that
> build_sdist will always work in flit when given only an unpacked
> sdist, while Donald and Paul rejected relying on full tree copies as a
> fallback for build_sdist failing.
>
> Since I'm not interested in rehashing that debate, the build
> preparation hook also stays.
>
> However, I'd be fine with renaming these hooks using the
> "prepare_X_for_Y" scheme you suggest below:
>
> - prepare_metadata_for_build_wheel
> - prepare_input_for_build_wheel
>
> That makes it clearer where they fit into the overall wheel building
> process, as well as which particular step they implement within that
> process.
>
> > - Add an "extensions" hook namespace to allow prototyping of future
> >   extensions.
>
> No, on grounds of YAGNI. You can't reject the two hooks we've actually
> identified as needed (aka the matrix multiplication operators of this
> situation), and then turn around and argue in favour of an overly
> general scheme that supports arbitrary extension hooks (which would be
> somewhat akin to having taken the PEP 225 approach to resolving the
> syntactic ambiguity between elementwise multiplication and matrix
> multiplication).
>
> We're not fumbling around based on a complete absence of information
> here - we're formalising a backend build interface based on more than
> a decade of experience with the successes and challenges of the
> existing system based on distutils, setuptools and PyPI.
>
> Besides, if frontends want to invoke arbitrary interfaces that are
> specific to particular backends, we already have ways for them to do
> that: Python's regular import system, and plugin management approaches
> like pkg_resources entry points.
>
> > - Rename get_build_*_requires -> get_requires_for_build_* to make the
> >   naming parallelism more obvious
>
> This sounds like a good change to me.
>
> > - Add the option to declare an operation unsupported by returning
> >   NotImplemented
>
> No. Supported operations for a backend must be identifiable without
> installing any dependencies, and without running any backend code.
>
> To tighten that requirement up even further: if the backend's
> capabilities can't be accurately determined using
> "inspect.getattr_static", then the backend is not compliant with the
> spec. The build frontend/backend API is not a space where we want
> people to try to be clever - we want them to be completely dull and
> boring and use the most mundane code they can possibly come up with.
>
> However, 

Re: [Distutils] Finishing up PEP 517

[Daniel Holth]

Sounds good, I'll bring it up there.

On Thu, Jun 29, 2017, 17:54 Bill Deegan <b...@baddogconsulting.com> wrote:

> Daniel,
>
> Perhaps the better way to go about it than to call scons N times is to
> have N variant dirs? where there's a variant dir for each build type you
> want, and then youd call
>
> scons build_wheel
> which is aliased Alias('build_wheel','variant_dir for building wheel path')
>
> Maybe bring this up on scons users mailing list and we'll see if it can be
> resolved?
>
> On Wed, Jun 28, 2017 at 6:42 PM, Daniel Holth <dho...@gmail.com> wrote:
>
>> I was able to implement PEP 517 build_wheel and build_sdist for enscons
>> (on bitbucket), and a click cli calling any backend mentioned in
>> pyproject.toml. Pretty simple. Not sure what to do with the config
>> dictionary.
>>
>> SCons is not designed to be called twice in the same process with
>> different arguments. It would be easier for me to know that pip would only
>> invoke one PEP 517 function per subprocess, or to know all target
>> directories in advance. Otherwise the enscons.api subprocess has to invoke
>> SCons in another subprocess. SCons also builds all targets (wheel and sdist
>> in same invocation) by default.
>>
>> "Alakazam!"
>>
>> On Wed, Jun 28, 2017 at 3:52 PM Thomas Kluyver <tho...@kluyver.me.uk>
>> wrote:
>>
>>> On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote:
>>>
>>> Is there a prototype implementation of pep 517 yet?
>>>
>>>
>>> - Flit has a PR with a prototype backend implementation, though it's not
>>> up to date with all the changes the PEP has undergone. I'll update it when
>>> we've agreed on a spec - it's still a fast moving target right now.
>>> - I have a prototype module frontends could use to call hooks here:
>>> https://github.com/takluyver/pep517 . It's mostly up to date, except
>>> for the issue with using sdist as a fallback for copying files for a wheel.
>>>
>>> Re: magic strings - like Nathaniel said, I haven't noticed them as part
>>> of any proposal so far.
>>>
>>> Thomas
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Daniel Holth]

The other thing I noticed is that build-system requires has no hyphen but
build-backend does.

On Wed, Jun 28, 2017, 22:28 Daniel Holth <dho...@gmail.com> wrote:

> That would fit with setup.py, which also likes to sys.exit() after one
> command.
>
> On Wed, Jun 28, 2017, 22:25 Nick Coghlan <ncogh...@gmail.com> wrote:
>
>> On 29 June 2017 at 11:42, Daniel Holth <dho...@gmail.com> wrote:
>> > I was able to implement PEP 517 build_wheel and build_sdist for enscons
>> (on
>> > bitbucket), and a click cli calling any backend mentioned in
>> pyproject.toml.
>> > Pretty simple. Not sure what to do with the config dictionary.
>>
>> That's designed to allow frontends to tunnel custom settings from the
>> user through to the backend, so if you don't support any config
>> settings yet, it probably makes sense to just error out with "Unknown
>> config setting" if the dictionary is non-empty. Alternatively, you
>> could silently ignore it.
>>
>> > SCons is not designed to be called twice in the same process with
>> different
>> > arguments. It would be easier for me to know that pip would only invoke
>> one
>> > PEP 517 function per subprocess, or to know all target directories in
>> > advance. Otherwise the enscons.api subprocess has to invoke SCons in
>> another
>> > subprocess. SCons also builds all targets (wheel and sdist in same
>> > invocation) by default.
>>
>> That's an interesting point of tension between supporting imperative
>> frontends like pip (which have a strong opinion about the order in
>> which steps should be executed) and declarative build systems like
>> Scons (which are more "produce the defined artifact set").
>>
>> However, I think the way to go for now would be to say:
>>
>> - each hook call should be made in a fresh subprocess (so backends
>> don't need to use a second subprocess "just to be on the safe side")
>> - in a *future* API extension, we may add an optional "build_all" hook
>> (whereby the backend produced both an sdist and all wheels it knew how
>> to create for the current platform), but anything like that will be
>> out of scope for the initial PEP 517 API
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Daniel Holth]

That would fit with setup.py, which also likes to sys.exit() after one
command.

On Wed, Jun 28, 2017, 22:25 Nick Coghlan <ncogh...@gmail.com> wrote:

> On 29 June 2017 at 11:42, Daniel Holth <dho...@gmail.com> wrote:
> > I was able to implement PEP 517 build_wheel and build_sdist for enscons
> (on
> > bitbucket), and a click cli calling any backend mentioned in
> pyproject.toml.
> > Pretty simple. Not sure what to do with the config dictionary.
>
> That's designed to allow frontends to tunnel custom settings from the
> user through to the backend, so if you don't support any config
> settings yet, it probably makes sense to just error out with "Unknown
> config setting" if the dictionary is non-empty. Alternatively, you
> could silently ignore it.
>
> > SCons is not designed to be called twice in the same process with
> different
> > arguments. It would be easier for me to know that pip would only invoke
> one
> > PEP 517 function per subprocess, or to know all target directories in
> > advance. Otherwise the enscons.api subprocess has to invoke SCons in
> another
> > subprocess. SCons also builds all targets (wheel and sdist in same
> > invocation) by default.
>
> That's an interesting point of tension between supporting imperative
> frontends like pip (which have a strong opinion about the order in
> which steps should be executed) and declarative build systems like
> Scons (which are more "produce the defined artifact set").
>
> However, I think the way to go for now would be to say:
>
> - each hook call should be made in a fresh subprocess (so backends
> don't need to use a second subprocess "just to be on the safe side")
> - in a *future* API extension, we may add an optional "build_all" hook
> (whereby the backend produced both an sdist and all wheels it knew how
> to create for the current platform), but anything like that will be
> out of scope for the initial PEP 517 API
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Daniel Holth]

I was able to implement PEP 517 build_wheel and build_sdist for enscons (on
bitbucket), and a click cli calling any backend mentioned in
pyproject.toml. Pretty simple. Not sure what to do with the config
dictionary.

SCons is not designed to be called twice in the same process with different
arguments. It would be easier for me to know that pip would only invoke one
PEP 517 function per subprocess, or to know all target directories in
advance. Otherwise the enscons.api subprocess has to invoke SCons in
another subprocess. SCons also builds all targets (wheel and sdist in same
invocation) by default.

"Alakazam!"

On Wed, Jun 28, 2017 at 3:52 PM Thomas Kluyver <tho...@kluyver.me.uk> wrote:

> On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote:
>
> Is there a prototype implementation of pep 517 yet?
>
>
> - Flit has a PR with a prototype backend implementation, though it's not
> up to date with all the changes the PEP has undergone. I'll update it when
> we've agreed on a spec - it's still a fast moving target right now.
> - I have a prototype module frontends could use to call hooks here:
> https://github.com/takluyver/pep517 . It's mostly up to date, except for
> the issue with using sdist as a fallback for copying files for a wheel.
>
> Re: magic strings - like Nathaniel said, I haven't noticed them as part of
> any proposal so far.
>
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Daniel Holth]

Is there a prototype implementation of pep 517 yet?

On Wed, Jun 28, 2017, 06:58 Robert Collins 
wrote:

>
>
> Re: returning magic strings to indicate exceptions. Please no. Just no.
> Have any pep build host add a small library to Python path with any symbols
> we want to define. Hardly an implementation hurdle.
>
> Rob
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Daniel Holth]

My impressions on reading what is hopefully the current version of the PEP

Are there different paths to get to sdist/wheel for example tree -> prepare
sdist/wheel files -> build sdist and tree -> build sdist, tree -> build
wheel depending on what [pip] decides to do? Should the frontend do both
and compare the result to make sure it is the same?

I'm not entirely clear on what the prepare hooks should do.

The rationalizations interleaved between the descriptions of each hook
distract from what the hooks should actually do. Suggested replacement
rationalization: "Some people like to build [format]. This hook builds
[format]".

I will make an effort to implement this for enscons, probably with the most
direct tree -> wheel, tree -> sdist hooks and without any optional hooks.

Thanks,

Daniel

On Mon, Jun 26, 2017 at 8:36 AM Nathaniel Smith  wrote:

> On Jun 25, 2017 12:46 AM, "Nick Coghlan"  wrote:
>
> On 25 June 2017 at 17:41, Nathaniel Smith  wrote:
> > Maybe you're right and there are exactly 2 front-end use cases and it
> > will turn out that the current PEP addresses them perfectly. I don't
> > have a crystal ball; I'm making an argument from ignorance.
>
> I'm not - we have two concrete potential consumers of the interface
> (pip and tox, aka "build to use" and "build to test"), and I'm
> designing the interface to cover their needs (i.e. out-of-tree wheel
> builds and actual sdists).
>
> If we discover other use cases later, we'll worry about them then (and
> the easy of doing so is the nicest benefit of defining this as a
> Python API), but the temptation to design in hyper-flexibility now
> falls under YAGNI (You Ain' Gonna Need It).
>
>
> My proposal also covers their needs AFAICT? At least I thought Donald said
> he thought the would work for pip. And you can't use YAGNI to argue for a
> more complicated proposal, that's cheating :-).
>
> -n
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig