Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 10 May 2016 at 07:08, Chris Barker  wrote:
> But I started this whole line of conversation because it seemed that there
> was desire for:
>
> Ability to isolate the build environment.
> Ability to better handle/manage non-python dependencies

I don't care about the first one - between disposable VMs and Linux
containers, we're already spoiled for choice when it comes to
supporting isolated build environments, and every project still
gaining net new contributors gets a natural test of this whenever
someone tries to set up their own local build environment.

I do care about the second one - Tennessee Leeuwenburg's draft PEP for
that is something he put together at the PyCon Australia sprints, and
it's the cornerstone of eventually being able to publish to PyPI and
have RPMs, Debian packages, conda packages, homebrew packages, etc,
"just happen" without any need for human intervention, even if your
package has external binary dependencies.

The fact that people would potentially be able to do "pip wheel" more
easily (since they'd be presented with a clean "you need , but
don't have it" error message rather than a cryptic build failure) is a
nice bonus, but it's not the reason I personally care about the
feature - I care about making more efficient use of distro packager's
time, by only asking them to do things a computer couldn't be doing
instead. The more complete we're able to make upstream dependency
metadata, the less people will need to manually tweak the output of
pyp2rpm (and similar tools for other platforms).

>> (twice, actually, anchored on
>> /usr/bin/python and /usr/bin/python3), it should eventually be
>> feasible to have the upstream->conda pipeline fully automated as well.
>
> yeah -- there's been talk for ages of automatically building conda packages
> (on the fly, maybe) from PyPi packages. But currently on conda-forge we've
> decided to NOT try to do that -- it's turned out in practice that enough
> pypi packages end up needing some hand-tweaking to build. So teh planned
> workflow is now:
>
> Auto-build a conda build script for a PyPi package
> Test it
> Tweak it as required
> Add it to conda-forge.
>
> Then -- *maybe* write a tool that auto-updates the PyPi based packages in a
> chron job or whatever.
>
> So not quite a automated conda-PyPi bridge, but not a bad start.

Yep, this is around the same level the Linux distros are generally at
- a distro level config gets generated *once* (perhaps with a tool
like pyp2rpm), but the result of that process then needs to be hand
edited when new upstream releases come out, even if nothing
significant has changed except the application code itself (no new
build dependencies, no new build steps, etc).

Utilities like Fedora's rebase-helper can help automate those updates,
but I still consider the ideal arrangement to be for the upstream
metadata to be of a sufficient standard that post-generation manual
tweaking ceases to be necessary in most cases.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Sun, May 8, 2016 at 5:31 AM, Nick Coghlan  wrote:

> > any python developer is going to
> > run into these issues eventually...
>
> Aye, I know - conda was one of the systems I evaluated as a possible
> general purpose tool for user level package management in Fedora and
> derivatives (see
>
> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement#Directions_to_be_Explored
> for details).
>
> Hence my comment about conda not currently solving *system integrator*
> problems in the general case - it's fine as a platform for running
> software *on top of* Fedora and for DIY integration within an
> organisation, but as a tool for helping to *build Fedora*, it turned
> out to introduce a whole slew of new problems


Sure -- I don't think it was ever intended to support rpm et. all -- it's
to be used INSTEAD of rpm et. all -- for the most part, rpm has solved the
problems that conda is trying to solve -- but only for rpm-based Linux
systems. And I'm going to guess that Continuum didn't want to:

build packages for rpm systems (which ones?)
build packages for deb-based systems (which ones?)
build packages for gentoo
build packages for arch..
.
build packages for homebrew

build packages for cygwin
build packages for Windows.. OOPS, there IS not package manager for
Windows!!

And, AFAICT, none of those package management systems support
"environments", either.

Clearly -- a tool like conda was required to meet Continuum's goals -- and
those goals are very much the same as PyPa's goals, actually. (except the
curated collection of packages part, but conda itself is not about the
Curation...)

However, the kinds of enhancements we're now considering upstream in
> pip should improve things for conda users as well - just as some folks
> in Fedora are exploring the feasibility of automatically rebuilding
> the whole of PyPI as RPMs


yes -- that's a good analogy -- for python packages, conda relies entirely
on distutils/setuptools/pip -- so yes, making those tools better and more
flexible is great.

but I'm still confused about what "the kinds of enhancements we're now
considering upstream in pip" are.

Here are a few:

More flexibility about the build system used
Easier to get metadata without actually initiating a build

These are great!

But I started this whole line of conversation because it seemed that there
was desire for:

Ability to isolate the build environment.
Ability to better handle/manage non-python dependencies

These are what I was referring to as mission-creep, and that overlap with
conda (and other tools).


> (twice, actually, anchored on
> /usr/bin/python and /usr/bin/python3), it should eventually be
> feasible to have the upstream->conda pipeline fully automated as well.


yeah -- there's been talk for ages of automatically building conda packages
(on the fly, maybe) from PyPi packages. But currently on conda-forge we've
decided to NOT try to do that -- it's turned out in practice that enough
pypi packages end up needing some hand-tweaking to build. So teh planned
workflow is now:

Auto-build a conda build script for a PyPi package
Test it
Tweak it as required
Add it to conda-forge.

Then -- *maybe* write a tool that auto-updates the PyPi based packages in a
chron job or whatever.

So not quite a automated conda-PyPi bridge, but not a bad start.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 8 May 2016 at 04:15, Chris Barker  wrote:
> On Sat, May 7, 2016 at 6:51 AM, Nick Coghlan  wrote:
>>
>> On 7 May 2016 01:55, "Chris Barker"  wrote:
>> > So my point is about scope-creep -- if you want the PyPa tools to solve
>> > all these problems, then you are re-inventing conda -- better to simply
>> > adopt conda (or fork it and fix issues that I'm sure are there)
>>
>> conda doesn't solve these problems either - it solves the *end user*
>> problem for data analysts (install the Python library they want to use),
>
> I really need to make this clear -- there is NOTHING "data analyst" specific
> about these problems -- they do come up more in the computational
> programming domain, but there are any number of other application that have
> the same problems (pyQT has come up in this conversation, yes?) -- and we're
> finding conda to be a good solution for our web development needs, too -- a
> conda environment  is kinda like a lighter-weight, platform independent
> docker container. And of course, there is more an more data analysis going
> on behind web services these days too -- any python developer is going to
> run into these issues eventually...

Aye, I know - conda was one of the systems I evaluated as a possible
general purpose tool for user level package management in Fedora and
derivatives (see
https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement#Directions_to_be_Explored
for details).

The problem with it is a practical one related to barriers to
adoption: to push any of the three systems I looked at, we'd face
significant challenges on the system administrator side (providing
capabilities comparable to what they're used to with RPMs) *and* on
the developer side (interoperating with the existing language specific
tooling for all of the language runtimes provided in Fedora).

Hence my comment about conda not currently solving *system integrator*
problems in the general case - it's fine as a platform for running
software *on top of* Fedora and for DIY integration within an
organisation, but as a tool for helping to *build Fedora*, it turned
out to introduce a whole slew of new problems (like needing to add
support to COPR/Koji/Pulp), without giving us any significant
capabilities that can't be addressed just as well by increasing the
level of automation in the upstream -> RPM pipeline and simplifying
the end user experience for container images.

However, the kinds of enhancements we're now considering upstream in
pip should improve things for conda users as well - just as some folks
in Fedora are exploring the feasibility of automatically rebuilding
the whole of PyPI as RPMs (twice, actually, anchored on
/usr/bin/python and /usr/bin/python3), it should eventually be
feasible to have the upstream->conda pipeline fully automated as well.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Chris Barker]

On Sat, May 7, 2016 at 6:17 AM, Greg Ewing 
wrote:

> Do you expect that
>
>> projects ... should (somehow) contain simplified instructions on how to
>> build the various C/Fortran extensions supplied in the bundle as
>> source code?
>>
>
> Essentially, yes. I'm not sure how achievable it would
> be, but ideally that's what I'd like.
>

I think we've all come to conclusion that that's simply not possible --
build configuration cannot be purely declarative in the general case -- if
you are building a package you are going to be running arbitrary code (and
really, if you're running a compiler, you're doing that anyway, so there
isn't an additional security issue here -- if you trust the code, you might
as well trust the build script)

> On a unix system, most of the time they would all be in

> well-known locations. If I install something in an unusual
> place or in an unusual way, I'm going to have to tell
> something about it anyway. I don't see how an executable
> setup file provided by the package author is going to
> magically figure out all the weird stuff I've done.
>

You can look at any number of packages -- essentially, they do what
configure scripts do with autotools -- various hacky ways to find the stuff
it's looking for.

and users really want this -- maybe on a "normal" *nix system there isn't
much to it, but on OS-X there sure is -- people may have hand installed the
dependencies, or they may have used fink, or macports, or homebrew, or...
and, except for the hand-install case, they may have no idea what the heck
they did and where it is.

I don't know if there are conventions for such things on
> Windows. I suspect not, in which case manual input is
> going to be needed one way or another.
>

Windows is even worse -- not only no conventions, but also no common
package manager, either (at least OS-X is limited to four :-) )

Not all of it, only the parts that strictly have to be
> performed as part of the build step of the install
> process, to use your terminology. That's a fairly
> restricted subset of the whole problem of compiling
> software.
>

I don't think it's possible (or desirable) to make a clear distinction
between "the build step of the install process" and building in general.

Could a purely declarative config file be flexible
> enough to handle this? I don't know. The distutils
> API is actually pretty declarative already if you
> use it in a straightforward way.
>

indeed -- and for the common cases, that works fine, but there's always
SOMETHING That some weird case is going to need.

You could, I suppose, separate out the configuration step from the build
step -- analogous to ./configure, vs make.

So the configure step would generate a purely declarative config file of
some sort, and then the build step would use that. In the simple case,
there might be no need for a configure step at all.

though I'm not sure even this is possible -- for instance, numpy used a
heavily enhanced distutils to do it's thing. Cython extends distutils to
understand the "build the C from the pyx" step, etc This is still used
decoratively, but it's third party code that is doing part of the build
step -- so in order to make the build system extendable, you need to it run
code

Anyway, I thought it was clear that we need to make the distinction between
building and installing/packaging, etc clear -- both form an API
perspective and a user perspective. So at this point, we need to establish
and API to the build system (probably compatible with what we have
(setup.py build) but we leave it up to the build system to figure out how
to do it's thing -- maybe someone will come up with a purely declarative
system -- who knows?

>  Running Pyrex to generate .c files

> from .pyx files is one that I've encountered.
> (I encouraged that one myself by including a distutils
> extension for it, which I later decided had been a
> mistake.)
>

I don't think it was a mistake :-) -- that's got to be done some time --
why add another layer???

That's nice, but it wouldn't help me when I encounter
> a package that *hadn't* been set up to use gregsbuild. :-(
>

sure -- but we can't have a one-build-system-to-rule-them-all until we can
first have a way to have ANY build system other than setuptools :-)

Key issue here:

Right now, in order to make things easier for users, and due to history,
the whole build/package/install processes are fully intermingled. I think
this is a BAD THING, for two reasons:

1) you can't replace any of the tools individually (except I suppose the
packaging part, as it's the last step)

2) user's don't know what's going on, or what's going to happen when they
try to "intsall" something:

It's great that you can "just run pip install" and it often "just works" --
but when it doesn't it's kind of ugly. And there are security concerns:
When a user runs:

pip install some_package

They don't know what's going to happen. pip goes and looks on PyPi to see
if 

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Sat, May 7, 2016 at 6:51 AM, Nick Coghlan  wrote:

> On 7 May 2016 01:55, "Chris Barker"  wrote:
> > So my point is about scope-creep -- if you want the PyPa tools to solve
> all these problems, then you are re-inventing conda -- better to simply
> adopt conda (or fork it and fix issues that I'm sure are there)
>
> conda doesn't solve these problems either - it solves the *end user*
> problem for data analysts (install the Python library they want to use),
>
I really need to make this clear -- there is NOTHING "data analyst"
specific about these problems -- they do come up more in the computational
programming domain, but there are any number of other application that have
the same problems (pyQT has come up in this conversation, yes?) -- and
we're finding conda to be a good solution for our web development needs,
too -- a conda environment  is kinda like a lighter-weight, platform
independent docker container. And of course, there is more an more data
analysis going on behind web services these days too -- any python
developer is going to run into these issues eventually...

> by ignoring the system integrator one (interoperate with the system
> integrator's existing platform level package management systems, of which
> we all tend to have our own with no plans for consolidation)
>
> That's by design, though - conda was specifically created as a language
> independent cross-platform platform, not as a cross-platform plugin
> management system for Python runtimes.
>
exactly!

> For a long time I was convinced the existence of conda and Linux
> containers as end user tools meant we wouldn't ever need to solve these
> problems at the language runtime layer, but it's since become clear to me
> that there's significant value in solving these problems in a way that
> doesn't care about how you obtained your Python runtime.
>
yup -- that would be the scope creep I'm referring too :-)

But now I'm confused about what problems we're talking about. From above:

"""
interoperate with the system integrator's existing platform level package
management systems
"""

you mean rpm, .deb, homebrew, etc?  distutils did have (does it still)
bdist_rpm (and, for that matter bdist_wininst) -- which I see as an attempt
to inter-operate with the system integrator's platform. But I think it
turns out that this is a hopeless task -- there are just too many different
systems to consider and maintain -- much better to let the external systems
handle that. And it got really ugly when you wanted to mingle virtualenv
with rpm, etc

The "solution" to that is to actually not do it -- use Docker or Conda if
you want an isolated environment.

What all this means to me is that  we really do need to keep the building
separate from the packaging -- so that the building tools can be used /
supported by the systems integrator. But people shouldn't be using rpm to
manage their system, and expect to install binary wheels...

The trick is that the pip/wheel system is really handy -- certainly for
Windows and OS-X that don't provide a standard system package manager. It
is nice to be able to "just pip install" various packages. But the problem
is that it is a half way solution -- it only works well for python-centered
packages -- pure python is trivial of course, but custom written C
extensions work great too -- but as soon as you get external dependencies,
it all goes to heck. We can improve that -- defining the manylinux platform
is a great step, but to really get there, there needs to be a way to handle
third party libs (which we could probably do with the existing tools by
establishing conventions), but then you get to non-python support tools:
Fortran compilers, other language run times, all sorts of stuff. Oh, and
then we want isolated environments, and they need to isolate these other
non-python tools, so 

I think it's WAY beyond the current PEP under discussion, but we really
should map out a "where we'd like to get" plan.

"significant value in solving these problems in a way that doesn't care
about how you obtained your Python runtime."

I don't get this -- I don't think it's possible to not care about the
python run-time -- we've only gotten this far because most folks have more
or less standardized on the python.org builds as the standard python build.
Even when you get your python elsewhere (Anaconda, for instance) they take
pains to keep it compatible.

I think we should count on python itself being managed by the same package
manager as the packages -- or at least conforming to a standard (which I
think we have with teh ABI tags, yes?)

BTW, I only recently learned that there are more "other" python builds out
there than I thought. Apparently the Visual Effect folks are on the habit
of building their own python2.7 on Windows, so they can use a newer MS
compiler, and then build their extensions to be compatible (all so you can
use modern C++ in the extension code). I don't 

Re: [Distutils] moving things forward

[Paul Moore]

On 7 May 2016 at 14:17, Greg Ewing  wrote:
> I don't know if there are conventions for such things on
> Windows. I suspect not, in which case manual input is
> going to be needed one way or another.

There aren't. You typically need to specify the exact locations of all
non-system libraries you use when you link. Judging by the various
complex systems in use (autoconf, pkg_config, ...) I don't think it's
as simple as you're suggesting on Unix either.

The complexities of configuring and building C libraries are a hard
problem, and one that people have been trying to solve for years. It's
not an area we (the Python packaging community) have any interest or
intention of getting involved in. Distutils went there and provided a
solution that did a fantastic job of solving the simpler parts of the
problem[1], but these days, the limitations of that approach are clear
- and we'd much rather enable specialists to build better (or more
specific) solutions and plug them into pip, than (as non-experts) try
to write those solutions for them.

> That's nice, but it wouldn't help me when I encounter
> a package that *hadn't* been set up to use gregsbuild. :-(

And ultimately that's the key issue that pip has to deal with - we
have to provide support for the many thousands of packages on PyPI and
elsewhere (including in closed-source in-house projects that we aren't
even theoretically capable of finding out about) that as you say
haven't been set up to use gregsbuild.

You suggest a declarative way of specifying compilation details. But
I'm not clear how *that* would help when I encounter a project that
hasn't been set up to use that new system, either.
Paul

[1] I was around before distutils, and it's incredibly easy these
days, with such a high awareness of the limitations of distutils, to
forget what a huge improvement it was over the previous situation. I
strongly believe that distutils - and the way it enabled all of
Python's subsequent packaging infrastructure - is one of the key
reasons why Python has become as popular as it is today. The debt we
owe to Greg Ward, who wrote distutils, is huge.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 7 May 2016 01:55, "Chris Barker"  wrote:
> So my point is about scope-creep -- if you want the PyPa tools to solve
all these problems, then you are re-inventing conda -- better to simply
adopt conda (or fork it and fix issues that I'm sure are there)

conda doesn't solve these problems either - it solves the *end user*
problem for data analysts (install the Python library they want to use), by
ignoring the system integrator one (interoperate with the system
integrator's existing platform level package management systems, of which
we all tend to have our own with no plans for consolidation)

That's by design, though - conda was specifically created as a language
independent cross-platform platform, not as a cross-platform plugin
management system for Python runtimes.

For a long time I was convinced the existence of conda and Linux containers
as end user tools meant we wouldn't ever need to solve these problems at
the language runtime layer, but it's since become clear to me that there's
significant value in solving these problems in a way that doesn't care
about how you obtained your Python runtime.

Cheers,
Nick.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Greg Ewing]

Paul Moore wrote:


Do you expect that
projects ... should (somehow) contain simplified instructions on how to
build the various C/Fortran extensions supplied in the bundle as
source code?


Essentially, yes. I'm not sure how achievable it would
be, but ideally that's what I'd like.


would the user need to put all the various details of his system into
a configuration file somewhere (and update that file whenever he
installs a new library, updates his compiler, or whatever)?


On a unix system, most of the time they would all be in
well-known locations. If I install something in an unusual
place or in an unusual way, I'm going to have to tell
something about it anyway. I don't see how an executable
setup file provided by the package author is going to
magically figure out all the weird stuff I've done.

I don't know if there are conventions for such things on
Windows. I suspect not, in which case manual input is
going to be needed one way or another.


How would
this cope with (for example) projects on Windows that *have* to be
compiled with mingw, and not with MSVC?


An option to specify which compiler to use would be a fairly
obvious thing that the config format should provide. As
would a way to include different option settings for
different platforms. Combine them in the obvious way.


This sounds to me more like an attempt to replace the "build" part of
distutils/setuptools with a more declarative system.


Not all of it, only the parts that strictly have to be
performed as part of the build step of the install
process, to use your terminology. That's a fairly
restricted subset of the whole problem of compiling
software.

(Ideally, I would make it *very* restricted, such as
only compiling C code (and possibly C++, not sure).
For anything else you would have to write a C wrapper
or use a tool that generates C. But that might be
a step too far.)

Could a purely declarative config file be flexible
enough to handle this? I don't know. The distutils
API is actually pretty declarative already if you
use it in a straightforward way.

The trouble is that there's nothing to prevent, or
even discourage, writing a setup.py that works in
non-straightforward ways. I've seen some pretty
convoluted setup.py code that worked great when
your system happened to match one of the possibilites
that the author had in mind, but was very difficult
to deal with otherwise. If I'd been able to just
set a few compile options and library paths directly,
it would have been a lot easier.

There's also nothing to prevent the setup.py from
doing things that don't strictly need to be done at
install time. Running Pyrex to generate .c files
from .pyx files is one that I've encountered.
(I encouraged that one myself by including a distutils
extension for it, which I later decided had been a
mistake.)


the proposals currently on the table
would allow you to ask pip to use that build system rather than
setuptools:

>

[buildsystem]
requires=gregsbuild
build_command=gregsbuild --make-wheel


That's nice, but it wouldn't help me when I encounter
a package that *hadn't* been set up to use gregsbuild. :-(

--
Greg
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Paul Moore]

tl;dr version

I think you're right that terminology can be confusing. I think the
definitions people typically work to are:

1. The "packaging" or "release" process - the process (run on a
developer's machine) of creating files that get published for users to
download and install.
2. The "install" process - the process (run on a user's machine) of
taking a published file and making it available in their environment.
This consists of separate steps:
  2a. Optional, and to be avoided wherever possible (by distribution
of wheels) - the "build" step that takes a published file and
configures (compiles) it for the user's environment
  2b. The "install" step (confusion alert! - the "install" step is
only one step of the "install" *process*) that puts the files in the
correct places on the user's machine.

We're not interested in trying to dictate the "packaging" process -
pip isn't involved in that process at all (see flit for a system that
lets projects build releases in a completely different way).

Sigh. Even the tl;dr version is too long :-)

On 7 May 2016 at 01:55, Greg Ewing  wrote:
> Chris Barker wrote:
>>
>> But I think there is consensus here that build systems need to be
>> customisable -- which means arbitrary code may have to be run.
>
> I think different people are using the word "build" in
> different ways here.
>
> To my mind, "building" is what the developer of a package
> does, and a "build system" is what he uses to do it. I
> don't care how much arbitrary code gets run during that
> process.

That is correct, and I agree with you that making a build process like
this declarative is not particularly useful. However...

> But when I do "python setup.py install" or "pip install"
> or whatever the recommended way is going to be, from my
> point of view I'm not "building" the package, I'm
> *installing* it.

Unfortunately, "python setup.py install" does not work that way - it
builds the project and then installs the files. So whether you want to
or not, you're building. That's basically why we're trying to make
"pip install foo" the canonical way of installing packages. So let's
ignore "setup.py install" for the rest of this discussion.

Now, for "pip install foo", *if the foo project provides a wheel
compatible with your system* then what you expect is what you get - a
pure install with no build step.

The problem lies with projects that don't supply wheels, only source.
Or unusual systems that we can't expect projects to have wheels for.
Or local checkouts ("pip install ."). In those cases, it's necessary
to do a build before you can install.

So while we're aiming for 80% or more of the time "pip install" to do
a pure install from a binary distribution, we can't avoid the fact
that occasionally the install will need to run an implicit build step.

> Confusion arises because the process of installation may
> require running a C compiler to generate extension modules.
> But figuring out how to do that shouldn't require
> running arbitrary code supplied by the developer. All the
> tricky stuff should have been done before the package
> was released.

I'm not sure I follow what you are suggesting here. Do you expect that
projects should be able to publish something (it's not exactly a
sdist, but it's not a wheel either as it doesn't contain everything
built) should (somehow) contain simplified instructions on how to
build the various C/Fortran extensions supplied in the bundle as
source code? That's an interesting idea, but how would it work in
practice? The bundles would need to be platform specific, I assume? Or
would the user need to put all the various details of his system into
a configuration file somewhere (and update that file whenever he
installs a new library, updates his compiler, or whatever)? How would
this cope with (for example) projects on Windows that *have* to be
compiled with mingw, and not with MSVC?

> If it's having trouble finding some library or header
> file or whatever on my system, I'd much rather have a
> nice, clear declarative config file that I can edit to
> tell it where to find them, than some overly clever
> piece of python code that's great when it works but
> a pain to unravel when it doesn't.

This sounds to me more like an attempt to replace the "build" part of
distutils/setuptools with a more declarative system. While that may be
a worthwhile goal (I genuinely have no opinion on that) it's largely
orthogonal to the current discussions. Except in the sense that if you
were to build such a system, the proposals currently on the table
would allow you to ask pip to use that build system rather than
setuptools:

# Invented syntax, because syntax is what we *haven't* agreed on yet :-)
[buildsystem]
requires=gregsbuild
build_command=gregsbuild --make-wheel

Then if a user on a system for which the project doesn't have a binary
wheel installed tries to install the project, your build system will
be downloaded, and the "gregsbuild 

Re: [Distutils] moving things forward

[Greg Ewing]

Chris Barker wrote:
But I think there is consensus here that build systems need to be 
customisable -- which means arbitrary code may have to be run.


I think different people are using the word "build" in
different ways here.

To my mind, "building" is what the developer of a package
does, and a "build system" is what he uses to do it. I
don't care how much arbitrary code gets run during that
process.

But when I do "python setup.py install" or "pip install"
or whatever the recommended way is going to be, from my
point of view I'm not "building" the package, I'm
*installing* it.

Confusion arises because the process of installation may
require running a C compiler to generate extension modules.
But figuring out how to do that shouldn't require
running arbitrary code supplied by the developer. All the
tricky stuff should have been done before the package
was released.

If it's having trouble finding some library or header
file or whatever on my system, I'd much rather have a
nice, clear declarative config file that I can edit to
tell it where to find them, than some overly clever
piece of python code that's great when it works but
a pain to unravel when it doesn't.

--
Greg
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Fri, May 6, 2016 at 9:39 AM, Donald Stufft  wrote:

> On May 6, 2016, at 11:54 AM, Chris Barker  wrote:
>
> So my point is about scope-creep -- if you want the PyPa tools to solve
> all these problems, then you are re-inventing conda -- better to simply
> adopt conda (or fork it and fix issues that I'm sure are there….)
>
>
> Adopting Conda is unlikely to ever happen for the defaulting tooling.
>

I didn't put the emphasis right in that sentence -- I was not actually
suggesting that conda be adopted -- what I was suggesting was the we DON'T
expand the scope of pip et al to be able to do everything that conda does.

That being said, I think there are some misunderstanding here that may be
relevant to this discussion:


> The problems that the default tooling attempt to solve are significantly
> harder than the problems that Conda attempts to solve
>

kinda-sorta -- conda is a packaging system. period, end of story (actually
it is an isolated environment system, too -- is that packaging? )-- it is
NOT a build system. Right now, you can invoke pip to:

* Find and install a binary package (wheel) -- this conda does
* Find, download, build and install install a source package -- nope, conda
doesn't do anything like that.
* Build and install a local package from source
* Install a local package from source in "develop mode" (editable mde) --
actually conda does have that -- though I'm not sure who well it works, of
it it's python specific.

But the fact that this all is (apparently) done by one tool is actually a
problem -- we have a tangled mess of binary and source and building and
installing, and it isn't obvious to the user what they are getting ) often
they don't care -- as long as it works :-) ) And, under the hood, it's all
driven by setuptools, which also has duplicate and slightly different
functionality that can be accidentally invoked (fun with easy_install!).

So I thought the goal now was to clean up an untangle this mess, yes?


and switching to it would be a regression.
>

only in the sense the switching to pip (and getting rid of setuptools)
would be a regression


> The primary benefit of Conda’s packages over Wheels are that they have a
> curated repository with people who are ensuring that things build correctly
> and since they don’t rely on authors to do it correctly, they don’t have to
> wait for, or convince authors to do this fresh new thing.
>

That simply is not true. Let's not mix social issues from technical ones.
Continuum set out to create a curated set of packages, yes -- but they
didn't put the enormous amount of work into conda for no reason -- they did
it because pip + wheel didn't meet their needs (and it doesn't meet a lot
of our needs, either). And, as I understand it, they came to the Python
community, and said something along the lines of "the packaging tooling
doesn't meet the needs of the scientific community, can we contribute
changes, improvements" -- and were told something along the lines of: "We
don't want the standard tooling to solve those problems -- go make what you
need, and then maybe we'll talk"

(I can dig out the message from Travis Oliphant about this if you like)

And they did go out and make something else, and they have released it as
an open source project, and a number of folks are adopting it to solve
their problems, too.


> The problem is, none of those benefits are something that would apply if
> we decided to throw away the 588,074 files that are currently able to be
> installed on PyPI.
>

conda aside, I think there is a real hang-up here. yes, of course you don't
want to throw away all that -- we absolutely need to maintain the current
packaging, building structure to support existing projects. But that
doesn't mean something new and better can't be built to be used in parallel
-- as folks move forward with pacakge development, they can adopt the new
system or they can keep using teh same old setup.py they have. MAybe
decades later something will get depreciated. But this approach is what got
us in the ugly mess to begin with.


Us deciding that Conda is the thing to use isn’t going to magically create
> an army of volunteers to go through and take all 80,000 packages on PyPI
> and ensure that we get a correctly compiled package on every platform for
> each version. If we could do that we could just convince everyone to go out
> and build binary packages, we could just do that with Wheels without
> requiring forklifting an entire ecosystem.
>

Again, conda packages and wheels are not the same thing -- there are
technical differences, and those differences really are why continuum uses
it for Anaconda, and why a lot of folks are adopting it for community led
efforts, like conda-forge. And there are a LOT of packages on conda-forge
despite it being very new and only a small handful of people contributing.

https://conda-forge.github.io/feedstocks.html


> While wheels are optimized for the pure 

Re: [Distutils] moving things forward

[Ethan Furman]

On 05/06/2016 09:48 AM, Leonardo Rochael Almeida wrote:

On 6 May 2016 at 13:15, Chris Barker wrote:



"python literals" is perfectly well defined -- both by the language
reference, and by "can be parsed by ast.literal_eval" and it addresses

>> the limitations of JSON and is fully declarative.


There is actually prior art for that kind of use. Odoo uses such a
language for its addons system, including package dependencies. See
example file here:

https://github.com/OCA/maintainer-tools/blob/master/template/module/__openerp__.py

Notice the `depends` key, that lists other addons, and the
`external_dependencies` key that can list both python distribution
dependencies as well as external program dependencies.


This is one of the very few things Odoo got right.  Let's not look to 
them for other examples of good coding practices.


--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Leonardo Rochael Almeida]

On 6 May 2016 at 13:15, Chris Barker  wrote:

> On Fri, May 6, 2016 at 5:41 AM, Nick Coghlan  wrote:
>
>> [...]
>
>
>
>> So rather than saying "the bootstrapping dependency declaration file
>> is Python-but-not-really", it's easier to say "it's an ini-file format
>> that can be parsed with the configparser module" or "it's JSON" (I'm
>> ruling out any options that don't have a stdlib parser in Python 2.7)
>>
>
> Last time, I promise :-)
>
> "python literals" is perfectly well defined -- both by the language
> reference, and by "can be parsed by ast.literal_eval"
>

> and it addresses the limitations of JSON and is fully declarative.
>

There is actually prior art for that kind of use. Odoo uses such a language
for its addons system, including package dependencies. See example file
here:

https://github.com/OCA/maintainer-tools/blob/master/template/module/__openerp__.py

Notice the `depends` key, that lists other addons, and the
`external_dependencies` key that can list both python distribution
dependencies as well as external program dependencies.

ISTM the reluctance in adopting this idea comes from the desire of using
other programming languages to parse the bootstraping information (imagine
an sdist -> deb converter implemented in go. It will have to exec python
eventually, during the build process, but not before it has prepared the
build environment).

And even though, unlike JSON, the ConfigParser format of setup.cfg is not
officially defined anywhere, there are sufficiently compatible ini parsers
in other languages that the idea still has merit.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Donald Stufft]

> On May 6, 2016, at 11:54 AM, Chris Barker  wrote:
> 
> So my point is about scope-creep -- if you want the PyPa tools to solve all 
> these problems, then you are re-inventing conda -- better to simply adopt 
> conda (or fork it and fix issues that I'm sure are there….)


Adopting Conda is unlikely to ever happen for the defaulting tooling. The 
problems that the default tooling attempt to solve are significantly harder 
than the problems that Conda attempts to solve and switching to it would be a 
regression. The primary benefit of Conda’s packages over Wheels are that they 
have a curated repository with people who are ensuring that things build 
correctly and since they don’t rely on authors to do it correctly, they don’t 
have to wait for, or convince authors to do this fresh new thing.

The problem is, none of those benefits are something that would apply if we 
decided to throw away the 588,074 files that are currently able to be installed 
on PyPI. Us deciding that Conda is the thing to use isn’t going to magically 
create an army of volunteers to go through and take all 80,000 packages on PyPI 
and ensure that we get a correctly compiled package on every platform for each 
version. If we could do that we could just convince everyone to go out and 
build binary packages, we could just do that with Wheels without requiring 
forklifting an entire ecosystem.

While wheels are optimized for the pure Python case, there is nothing 
preventing them from being used to install anything else (Erlang or R or even 
Python itself). The pynativelib stuff is proof of the ability to do just that— 
and in distutils-sig land we tend to care a lot more about how these things 
will affect our downstream consumers like Debian and such than Conda needs to 
(or should need to!).

Now, this isn’t to say that Conda is bad or anything, but it’s use as a 
replacement for the current ecosystem is about as interesting as suggesting we 
all adopt RPM, or apt-get, or Choclatey, or Ports, or Pacman or whatever flavor 
of downstream you wish to insert here.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Fri, May 6, 2016 at 5:41 AM, Nick Coghlan  wrote:

> The "Python-with-imports" case is the status quo with setup.py, and we
> already know that's a pain because you need to set up an environment
> that already has the right dependencies installed to enable your
> module level imports in order to run the script and find out what
> dependencies you need to install to let you run the script in the
> first place.
>

good point -- this is really key.

The "Python-without-imports" approach would just be confusing -


I agree -- I never suggested that -- it's full python or fully declarative.


> So rather than saying "the bootstrapping dependency declaration file
> is Python-but-not-really", it's easier to say "it's an ini-file format
> that can be parsed with the configparser module" or "it's JSON" (I'm
> ruling out any options that don't have a stdlib parser in Python 2.7)
>

Last time, I promise :-)

"python literals" is perfectly well defined -- both by the language
reference, and by "can be parsed by ast.literal_eval"

and it addresses the limitations of JSON and is fully declarative.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Fri, May 6, 2016 at 5:55 AM, Nick Coghlan  wrote:

> > And maybe it's good to keep "new style" configuration clearly separate.
>
> Part of my motivation for suggesting re-using setup.cfg is the
> proliferation of packaging related config sprawl in project root
> directories - setup.py won't be going anywhere any time soon for most
> projects, some folks already have a setup.cfg (e.g. to specify
> universal wheel creation), and there's also MANIFEST.in to control
> sdist generation.


yeah -- ugly, but will  one more file make a difference? now that I think
about it -- IIUC the goals here, we want to separate packaging from
building -- I'd argue that setuup.cfg is about building -- we really should
keep the package configuration  separate. Some day, some how, we hoping to
have a new (or multiple build systems) they won't use setup.cfg to
configure the build. So we probably shouldn't marry ourselves to setup.cfg
for package configuration.

The other issue is social -- if we stick with setup.cfg, we are implying
that this is all about tweaking distutils/setuptools, not doing something
different -- seemingly keeping the mingling of building and packaging
forevermore

But this isn't that big a deal -- enough bike-shedding, time to make a
declaration.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Thu, May 5, 2016 at 6:37 PM, Robert Collins 
wrote:

>
> Thats good. It occurs to me that scientific builds may be univerally
> broken because folk want to avoid easy-install and the high cost of
> double builds of things. E.g. adding bootstrap_requires will let folk
> fix things?
>

scientific builds are universally "broken" because their dependencies are
often non-python (and oftne complex and hard to build) external libs.

that is simply a problem that setuptools and pip were never designed to
address.


> But the main question is : why are these packages staying inaccurate?
> Even in the absence of a systematic solution I'd expect bug reports
> and pull requests to converge on good dependencies.
>

the problem is not that people haven't declared the proper dependencies in
their packages -- it's because it's impossible to declare the proper
dependencies in their packages...

The solution is to do one of three things:

1) punt -- and let folks use Canopy, or conda, or third party collections
of wheels like the Gohlke repository, or let folks build it themselves (a
lot of packages have all kinds of custom code to go look in macports, or
homebrew, or fink locations for libs on OS-X for instance.

2) build binary wheels that statically link in everything needed (or
include the dlls in the package)

3) make binary wheels out of C libs, so that other packages can depend on
them --  this is kind of a kludgy hack (abuse?) of pip/wheel but should be
do-able.

Some folks are making a valiant effort to do a combination of (2) and (3)
for Windows, OS-X, and now many linux -- but it is too bad that the tools
don't make it easy. And so far, all we really have are the core scipy stack
(and pyQT5???)

In short -- building fully supported binary packages is possible, but
requires a fair bit of expertise and a valiant effort on the part of
someone...

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Thu, May 5, 2016 at 4:32 PM, Nathaniel Smith  wrote:

> > You do know that we're on our way to re-writing conda, now, don't you?
> :-)
> >
> > I think we need to be careful of scope-creep...
>
> conda did not invent the idea of creating separate python environments
> for different tasks :-)


I'm not suggesting conda invented anything -- I'm suggesting it has
implemented many of the things being talked about here -- the truth is
conda was designed to solve exactly the problems that scientific python
packages had that pip+wheel+setuptools do not currently solve.

So my point is about scope-creep -- if you want the PyPa tools to solve all
these problems, then you are re-inventing conda -- better to simply adopt
conda (or fork it and fix issues that I'm sure are there)

On the other hand, improving the PyPa tools while maintaining their current
scope is a lovely idea -- but that means leaving isolation of build
environments etc, to external tools: Docker, VMs, conda..

Actually, conda has a good lesson here: conda is about packaging, NOT
building -- I was quite disappointed that conda provided NO support for
cross platform building at all -- but after using it a bit I realized that
that was a great decision -- if you want to support a wide variety of
packages, you really need to let the package authors use whatever the heck
build system they want -- you really don't want to have to implement (or
even debug) everyone else's builds.

And IIUC, that is the direction we are trying to go with pip now -- making
pip+wheel build-system independent -- good goal.

Which actually give me an idea: it seems we are very bogged down in
struggles with keeping backward compatibility, and coming up with a API for
an arbitrary build system. Maybe we can take a lesson from conda and
essentially punt:

conda works by reading a meta.yaml -- a purely declarative package
configuration. The actual building is done by calling a build script --
conda itself does not need to know or care what the heck is in that build
script -- all it needs to know is that it can be run. Why not keep the API
that simple for pip?

we could do total backward compatible, and simply say: pip will call
"python setup.py install".

And that's it, end of story -- individual packages could do ANYTHING in
that setup.py script -- current packages would be using setuptools, but pip
wouldn't need to care at all what was in there -- only know that it would
install the package.

And, I suppose, a setup.py bdist_wheel for building wheels, and setup.py
sdist for creating source distributions), though I'm not sure that those
need to be a standardized API -- or at least not the SAME standardized API.


-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Chris Barker]

On Thu, May 5, 2016 at 10:45 PM, Greg Ewing 
wrote:


> Even if python is available, you might not want to run
> arbitrary code just to install a package.
>
> If a config file can contain executable code, then it
> can contain bugs. Debugging is something the developer of
> a package should have to do, not the user. In my experience,
> fixing someone else's buggy setup.py is about as much fun
> as pulling one's own teeth out with a blunt screwdriver.


well, in my experience, debugging my OWN buggy setup.py is about equally
pleasant!

But I think there is consensus here that build systems need to be
customisable -- which means arbitrary code may have to be run. So I don't
think this is going to help anyone avoiding dealing with buggy builds :-(

or are we talking about a config file that would be delivered with a binary
wheel? IN which case, yes, it should not have any executable code in it.

And anyway, it seems folks want to go with static config anyway -- clearly
separating configuration from customization, so I'll stop now.

I'd still like to be able to use Python literals, though :-)

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 6 May 2016 at 06:41, Chris Barker  wrote:
> On Wed, May 4, 2016 at 8:09 PM, Nick Coghlan  wrote:
>> 3. The ongoing popularity of setup.cfg shows that while ini-style may
>> not be perfect for this use case, it clearly makes it over the
>> threshold of "good enough"
>
> it's only popular because it's what's there -- if we're using that standard,
> we could make the same argument about setuptools ;-)

That's exactly the course change we made following the release of
Python 3.3 - rather than trying to replace setuptools directly, the
scope of our ambitions was narrowed to eliminating the need to run
"./setup.py install", while keeping setuptools as the default build
system used to enable things like generating wheel files ("./setup.py
bdist_wheel" doesn't work if you're importing setup from
distutils.core - you have to be importing it from setuptools. If you
have a setup.py file that imports from distutils.core, you have to use
"pip wheel" if you want to get a wheel file)

>> So when I ask myself now "What's the *simplest* thing we could do that
>> will make things better than the status quo?", then the answer I come
>> up with today is your original idea: bless setup.cfg (or at least a
>> subset of it) as a standardised interface.
>
> IIUC, we would be changin, or at least adding to the current setup.dfg spec.
> So this is a change, no matter how you slice it, saying "configuration will
> be specified in setup.something, in some other format, is the least
> significant part of all this change.
>
> And maybe it's good to keep "new style" configuration clearly separate.

Part of my motivation for suggesting re-using setup.cfg is the
proliferation of packaging related config sprawl in project root
directories - setup.py won't be going anywhere any time soon for most
projects, some folks already have a setup.cfg (e.g. to specify
universal wheel creation), and there's also MANIFEST.in to control
sdist generation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 6 May 2016 at 06:30, Chris Barker  wrote:
> On Wed, May 4, 2016 at 7:45 PM, Nick Coghlan  wrote:
>> Usually that enforcement is
>> handled by making the configuration declarative - it's in some passive
>> format like an ini file or JSON, and if it gets too repetitive then
>> you introduce a config generator, rather than making the format itself
>> more sophisticated.
>
>
> OK -- that's more or less my thought -- if it's  python that gets run, then
> you've got your config generator built in -- why not?

The immediate reason is because Python allows imports, and if imports
are permitted in the config script, people will use them, and if
they're not permitted, they'll complain about their absence.

The "Python-with-imports" case is the status quo with setup.py, and we
already know that's a pain because you need to set up an environment
that already has the right dependencies installed to enable your
module level imports in order to run the script and find out what
dependencies you need to install to let you run the script in the
first place.

The "Python-without-imports" approach would just be confusing - while
it would avoid the dependency bootstrapping problem, it would only be
kinda-sorta-Python rather than actual Python.

So rather than saying "the bootstrapping dependency declaration file
is Python-but-not-really", it's easier to say "it's an ini-file format
that can be parsed with the configparser module" or "it's JSON" (I'm
ruling out any options that don't have a stdlib parser in Python 2.7)

The "future benefit" reason is that it's a lot easier to be confident
that reading a config file from an uploaded artifact isn't going to
compromise a web service, so a future version of PyPI can readily pull
information out of the config file and republish it via an API. Once
you have that kind of information available via an API, you can
resolve it before downloading *anything* (which is especially useful
when your goal is dependency graph analysis rather than downloading
the whole of PyPI and running a Python script from every package you
downloaded).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Greg Ewing]

Chris Barker wrote:
This is python packaging, I'm still trying to see why you'd need to read 
the config without python available.


Even if python is available, you might not want to run
arbitrary code just to install a package.

If a config file can contain executable code, then it
can contain bugs. Debugging is something the developer of
a package should have to do, not the user. In my experience,
fixing someone else's buggy setup.py is about as much fun
as pulling one's own teeth out with a blunt screwdriver.

--
Greg
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Daniel Holth]

>From my point of view mandatory build isolation will make building thinks
slow and bad, besides being totally unrelated to the idea of a working
bootstrap requirements feature.

On Thu, May 5, 2016 at 9:37 PM Robert Collins 
wrote:

> On 5 May 2016 at 21:47, Nathaniel Smith  wrote:
> > On Wed, May 4, 2016 at 11:57 PM, Robert Collins
> ...
> >>> I don't think I've ever seen a package that had accurate
> >>> setup_requires (outside the trivial case of packages where
> >>> setup_requires=[] is accurate). Scientific packages in particular
> >>> universally have undeclared setup requirements.
> >>
> >> Are those requirements pip installable?
> >
> > Either they are, or they will be soon.
>
> Thats good. It occurs to me that scientific builds may be univerally
> broken because folk want to avoid easy-install and the high cost of
> double builds of things. E.g. adding bootstrap_requires will let folk
> fix things?
>
> But the main question is : why are these packages staying inaccurate?
> Even in the absence of a systematic solution I'd expect bug reports
> and pull requests to converge on good dependencies.
>
> ...
> >> So the 10x thing is defining how the thing doing the isolation (e.g.
> >> pip) should handle things that can't be installed but are already
> >> available on the system.
> >>
> >> And that has to tunnel all the way out to the user, because its
> >> context specific, its not an attribute of the dependencies per se
> >> (since new releases can add or remove this situation), nor of the
> >> consuming thing (same reason).
> >
> > # User experience today on i386
> > $ pip install foo
> > <... error: missing pyqt5 ...>
> > $ apt install python-pyqt5
> > $ pip install foo
> >
> > # User experience with build isolation on i386
> > $ pip install foo
> > <... error: missing pyqt5 ...>
> > $ apt install python-pyqt5
> > $ pip install --no-isolated-environment foo
>
> So thats one way that isolation can be controlled yes.
>
> > It'd even be straightforward for pip to notice that the requirement
> > that it failed to satisfy is already satisfied by the ambient
> > environment, and suggest --no-isolated-environment as a solution.
> >
> >> Ultimately, its not even an interopability question: pip could do
> >> isolated builds now, if it chose, and it has no ramifications as far
> >> as PEPs etc are concerned.
> >
> > That's not true. In fact, it seems dangerously naive :-/
> >
> > If pip just went ahead and flipped a switch to start doing isolated
> > builds now, then everything would burst into flame and there would be
> > a howling mob in the bug tracker. Sure, there's no PEP saying we
> > *can't* do that, but in practice it's utterly impossible.
>
> We've done similarly omg it could be bad changes before - e.g.
> introducing wheel caching. A couple of iterations later and we're in
> good shape. Paul seems to think we could introduce it gracefully - opt
> in, then default with fallback, then solely opt-out.
>
> > If we roll out this feature without build isolation, then next year
> > we'll still be in the exact same situation we are today -- we'll have
> > the theoretical capability of enabling build isolation, but everything
> > would break, so in practice, we won't be able to.
> >
> > The reason I'm being so intense about this is that AFAICT these are all
> true:
> >
> > Premise 1: Without build isolation enabled by default, then in
> > practice everyone will putter along putting up with broken builds all
> > the time. It's *incredibly* easy to forget to declare a build
> > dependency, it's the kind of mistake that every new user makes, and
> > experienced users too.
>
> I know lots of projects that already build in [mostly] clean
> environments all the time - e.g. anything using tox[*], most things
> that use Travis, Appveyor, Jenkins etc. Yes, if its not there by
> default it requires effort to opt-in, and there will be a tail of some
> sort. I don't disagree with the effect of the premise, but I do
> disagree about the intensity.
>
> [*]: yes, to runs setup.py sdist outside the environment, so
> setup_requires doesn't get well exercised. IIRC tox is adding a
> feature to build in a venv so they will be exercised.
>
> > Premise 2: We can either enable build isolation together with the new
> > static bootstrap requirements, or we can never enable build isolation
> > at all, ever.
>
> I don't agree with this one at all. We can enable it now if we want:
> yes, its a behavioural change, and we'd need to phase it in, but the
> logistics are pretty direct.
>
> > Conclusion: If we want to ever reach a state where builds are
> > reliable, we need to tie build isolation to the new static metadata.
> >
> > If you have some clever plan for how we could practically transition
> > to build isolation without having them opt-in via a new feature, then
> > that would be an interesting counter-argument; or an alternative plan
> > for how to reach a point where build requirements 

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 5 May 2016 at 21:47, Nathaniel Smith  wrote:
> On Wed, May 4, 2016 at 11:57 PM, Robert Collins
...
>>> I don't think I've ever seen a package that had accurate
>>> setup_requires (outside the trivial case of packages where
>>> setup_requires=[] is accurate). Scientific packages in particular
>>> universally have undeclared setup requirements.
>>
>> Are those requirements pip installable?
>
> Either they are, or they will be soon.

Thats good. It occurs to me that scientific builds may be univerally
broken because folk want to avoid easy-install and the high cost of
double builds of things. E.g. adding bootstrap_requires will let folk
fix things?

But the main question is : why are these packages staying inaccurate?
Even in the absence of a systematic solution I'd expect bug reports
and pull requests to converge on good dependencies.

...
>> So the 10x thing is defining how the thing doing the isolation (e.g.
>> pip) should handle things that can't be installed but are already
>> available on the system.
>>
>> And that has to tunnel all the way out to the user, because its
>> context specific, its not an attribute of the dependencies per se
>> (since new releases can add or remove this situation), nor of the
>> consuming thing (same reason).
>
> # User experience today on i386
> $ pip install foo
> <... error: missing pyqt5 ...>
> $ apt install python-pyqt5
> $ pip install foo
>
> # User experience with build isolation on i386
> $ pip install foo
> <... error: missing pyqt5 ...>
> $ apt install python-pyqt5
> $ pip install --no-isolated-environment foo

So thats one way that isolation can be controlled yes.

> It'd even be straightforward for pip to notice that the requirement
> that it failed to satisfy is already satisfied by the ambient
> environment, and suggest --no-isolated-environment as a solution.
>
>> Ultimately, its not even an interopability question: pip could do
>> isolated builds now, if it chose, and it has no ramifications as far
>> as PEPs etc are concerned.
>
> That's not true. In fact, it seems dangerously naive :-/
>
> If pip just went ahead and flipped a switch to start doing isolated
> builds now, then everything would burst into flame and there would be
> a howling mob in the bug tracker. Sure, there's no PEP saying we
> *can't* do that, but in practice it's utterly impossible.

We've done similarly omg it could be bad changes before - e.g.
introducing wheel caching. A couple of iterations later and we're in
good shape. Paul seems to think we could introduce it gracefully - opt
in, then default with fallback, then solely opt-out.

> If we roll out this feature without build isolation, then next year
> we'll still be in the exact same situation we are today -- we'll have
> the theoretical capability of enabling build isolation, but everything
> would break, so in practice, we won't be able to.
>
> The reason I'm being so intense about this is that AFAICT these are all true:
>
> Premise 1: Without build isolation enabled by default, then in
> practice everyone will putter along putting up with broken builds all
> the time. It's *incredibly* easy to forget to declare a build
> dependency, it's the kind of mistake that every new user makes, and
> experienced users too.

I know lots of projects that already build in [mostly] clean
environments all the time - e.g. anything using tox[*], most things
that use Travis, Appveyor, Jenkins etc. Yes, if its not there by
default it requires effort to opt-in, and there will be a tail of some
sort. I don't disagree with the effect of the premise, but I do
disagree about the intensity.

[*]: yes, to runs setup.py sdist outside the environment, so
setup_requires doesn't get well exercised. IIRC tox is adding a
feature to build in a venv so they will be exercised.

> Premise 2: We can either enable build isolation together with the new
> static bootstrap requirements, or we can never enable build isolation
> at all, ever.

I don't agree with this one at all. We can enable it now if we want:
yes, its a behavioural change, and we'd need to phase it in, but the
logistics are pretty direct.

> Conclusion: If we want to ever reach a state where builds are
> reliable, we need to tie build isolation to the new static metadata.
>
> If you have some clever plan for how we could practically transition
> to build isolation without having them opt-in via a new feature, then
> that would be an interesting counter-argument; or an alternative plan
> for how to reach a point where build requirements are accurate without
> being enforced; or ...?

As Paul said, add it off by default, phase it in over a couple of releases.
0: opt-in
1: opt-out but when builds fail disable and retry (and warn)
2: opt out

>> pyqt5 not having i386 is just a trivial egregious case. ARM32 and 64
>> is going to be super common, Power8 another one, let alone less common
>> but still extant and used architectures like PPC, itanium, or new ones
>> like x86_32 [If I remember 

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Daniel Holth]

Clearly it should spin up a Gentoo VM from scratch each time. No half
measures.

On Thu, May 5, 2016, 19:32 Nathaniel Smith  wrote:

> On Thu, May 5, 2016 at 3:18 PM, Chris Barker 
> wrote:
> > On Thu, May 5, 2016 at 2:10 AM, Nathaniel Smith  wrote:
> >>
> >> ...The main thing I want to point out though, is that all of these
> >> problems you're raising are complications caused entirely by wanting
> >> to avoid build isolation in the name of simplicity. If each package
> >> gets its own isolated build environment, then it can depend on
> >> whatever it wants without any danger of collision with the ambient
> >> environment.
> >
> >
> > You do know that we're on our way to re-writing conda, now, don't you?
> :-)
> >
> > I think we need to be careful of scope-creep...
>
> conda did not invent the idea of creating separate python environments
> for different tasks :-)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nathaniel Smith]

On Thu, May 5, 2016 at 3:18 PM, Chris Barker  wrote:
> On Thu, May 5, 2016 at 2:10 AM, Nathaniel Smith  wrote:
>>
>> ...The main thing I want to point out though, is that all of these
>> problems you're raising are complications caused entirely by wanting
>> to avoid build isolation in the name of simplicity. If each package
>> gets its own isolated build environment, then it can depend on
>> whatever it wants without any danger of collision with the ambient
>> environment.
>
>
> You do know that we're on our way to re-writing conda, now, don't you? :-)
>
> I think we need to be careful of scope-creep...

conda did not invent the idea of creating separate python environments
for different tasks :-)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Chris Barker]

On Thu, May 5, 2016 at 3:21 PM, Greg Ewing 
wrote:

> Chris Barker wrote:
>
> OK -- that's more or less my thought -- if it's  python that gets run,
>> then you've got your config generator built in -- why not?
>>
>
> The difference is that the config generator only gets run
> once, when the config info needs to get produced. If the
> config file itself is executable, it needs to be run each
> time the config info gets *used*


exactly -- maybe some run-time configuration specialization? though maybe
that's exactly what we want to avoid.



> -- which could be by a
> tool for which running arbitrary python code is awkward or
> undesirable.


This is python packaging, I'm still trying to see why you'd need to read
the config without python available.

But I think the key point is -- do we want to enforce that the config data
is static -- it WILL not change depending on when or how or where it is
read?

If so, then yeah, no runnable code.

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

Last post! sorry to have not kept up last night

> to call the new feature setup_requires but some prefer to eliminate
> > uncertainty by calling it bootstrap_requires.
>
> The main advantage of a new feature name is that when someone searches
> the internet for "python bootstrap_requires", they won't find a
> decade+ worth of inapplicable documentation of the setuptools feature
> :)


new name for sure -- but I don't like bootstrap-requires -- is this
bootstrapping isn't it jsut building?

why not build_requires?

BTW, conda has:

requires:
   build:
   run:
   test:

(I think that's it)

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Thu, May 5, 2016 at 2:47 AM, Nathaniel Smith  wrote:

> > When you introduce isolation, the build will only have the standard
> > library + whatever is declared as a dep: and pyqt5 has no source on
> > PyPI.
>

so build isolation isolates you from arbitrary system libs? now you NEED to
ship all your non-trivial dependent libs. which I guess is the point, but
again, we're re-implementing conda here.

And the only time where it makes sense to force this build requirement is
when you are building binary wheels to put up on PyPi. So maybe part of
bdist_wheel??

OH -- and you need to make sure there is an isolated test environment too
-- which is a bit of pain, particularly when the tests aren't installed as
part of the package.

And, of course this means a version of virtualenv that isolates you from
system libs -- that might be quite hard to do, actually.

and again, conda :-)

-CHB





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Greg Ewing]

Chris Barker wrote:

OK -- that's more or less my thought -- if it's  python that gets run, 
then you've got your config generator built in -- why not?


The difference is that the config generator only gets run
once, when the config info needs to get produced. If the
config file itself is executable, it needs to be run each
time the config info gets *used* -- which could be by a
tool for which running arbitrary python code is awkward or
undesirable.

--
Greg
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Thu, May 5, 2016 at 2:10 AM, Nathaniel Smith  wrote:

> ...The main thing I want to point out though, is that all of these
> problems you're raising are complications caused entirely by wanting
> to avoid build isolation in the name of simplicity. If each package
> gets its own isolated build environment, then it can depend on
> whatever it wants without any danger of collision with the ambient
> environment.


You do know that we're on our way to re-writing conda, now, don't you? :-)

I think we need to be careful of scope-creep...

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Wed, May 4, 2016 at 11:57 PM, Robert Collins 
wrote:

>
> No. Old pip new package will break, new pip old package is entirely safe
> AFAICT.


That's the goal, yes?

So I think we need to get less obsessed with backward compatibility:

pip will retain (for along time) the ability to install, build, whatever
any packages that it currently works for -- nothing on PyPi breaks.

New versions of pip can "detect" that a given package is "new style", and
it will do the right thing with that -- maybe following a completely
different code path.

If we take this approach, then we can design the new pip interface however
we want, rather than trying to keep compatibility.

but we need to be abel to provide the build tool to go with it, yes?

I think the way to do that is to have a new build tool that is forked from
setuptools, and is essentially the same thing (in version 1), but with the
functionality that it "shouldn't" have stripped out -- i.e. no easy_install!

Pacakge developers that want to adpt the new system only need to change an
import ln their setup.py:

from setuptools-lite import setup

and then do the right things with meta-data, and it will work with new
versions of pip.

 Right now we have a swamp
> >>> we need to drain,


exactly -- but rather than trying to drain that swamp, let's just move over
to the dry land next door

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Alex Grönholm]

OK, so which setup() arguments do we want to leave out of the static 
metadata?


05.05.2016, 23:53, Daniel Holth kirjoitti:
This is a recurring point of confusion. setup.py is not ever going 
away. In general it is necessary for you to be able to write software 
to build your software, and there is no intention to take that feature 
away.


People repeatedly come to the conclusion that static metadata means 
the entire build is static. It's only the dependencies that need to be 
static to enable better dependency resolution in pip. The build does 
not need to be static.


The proposed feature means you will be able to have a simpler setup.py 
or no setup.py it by using something like flit or pbr that are 
configured with .cfg or .ini. setup.py is not going away.


Static metadata means the list of dependencies, author name, trove 
classifiers is static. Not the build itself.


Enforced staticness of the build script is right out.

On Thu, May 5, 2016 at 4:34 PM Alex Grönholm > wrote:


I think it would be best to gather a few extreme examples of
setup.py files from real world projects and figure out if they can
be implemented in a declarative fashion. That at least would help
us identify the pain points.

For starters, gevent's setup.py looks like it needs a fair bit of
custom logic:
https://github.com/gevent/gevent/blob/master/setup.py

05.05.2016, 23:30, Chris Barker kirjoitti:

On Wed, May 4, 2016 at 7:45 PM, Nick Coghlan > wrote:

This configuration vs customisation distinction is probably worth
spelling out for folks without a formal software engineering or
computer science background, so:


fair enough -- good to be clear on the terms.

Configuration is different: you're choosing amongst a set of
possibilities that have been constrained in some way, and those
constraints are structurally enforced. 



That's a key point here -- I guess I'm skeptical that we can have
the flexibility we need with a purely configuration-based system
-- we probably don't WANT to constrain the options completely. If
you think about it, while distutils has it's many, many flaws,
what made it possible for it to be as useful as it is, and last
as long as it has because is CAN be customized -- users are NOT
constrained to the built-in functionality.

I suspect the idea of this thread is to keep the API to a build
system constrained -- and let the build systems themselves be as
customizable as the want to be. And I haven't thought it out
carefully, but I have a feeling that we're going to hit a wall
that way .. but maybe not.

Usually that enforcement is
handled by making the configuration declarative - it's in
some passive
format like an ini file or JSON, and if it gets too
repetitive then
you introduce a config generator, rather than making the
format itself
more sophisticated.


OK -- that's more or less my thought -- if it's  python that gets
run, then you've got your config generator built in -- why not?

The big advantage of configuration over customisation is that you
substantially increase the degrees of freedom in how
*consumers* of
that configuration are implemented - no longer do you need a full
Python runtime (or whatever), you just need an ini file
parser, or a
JSON decoder, and then you can look at just the bits you care
about
for your particular use case and ignore the rest.


Sure -- but do we care? this is about python packaging -- is it
too big a burden to say you need python to read the configuration?

-CHB

-- 


Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov 


___
Distutils-SIG maillist  -Distutils-SIG@python.org 

https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  - Distutils-SIG@python.org

https://mail.python.org/mailman/listinfo/distutils-sig



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Daniel Holth]

C extensions, py-modules, ...

On Thu, May 5, 2016, 17:05 Alex Grönholm  wrote:

> OK, so which setup() arguments do we want to leave out of the static
> metadata?
>
>
> 05.05.2016, 23:53, Daniel Holth kirjoitti:
>
> This is a recurring point of confusion. setup.py is not ever going away.
> In general it is necessary for you to be able to write software to build
> your software, and there is no intention to take that feature away.
>
> People repeatedly come to the conclusion that static metadata means the
> entire build is static. It's only the dependencies that need to be static
> to enable better dependency resolution in pip. The build does not need to
> be static.
>
> The proposed feature means you will be able to have a simpler setup.py or
> no setup.py it by using something like flit or pbr that are configured with
> .cfg or .ini. setup.py is not going away.
>
> Static metadata means the list of dependencies, author name, trove
> classifiers is static. Not the build itself.
>
> Enforced staticness of the build script is right out.
>
> On Thu, May 5, 2016 at 4:34 PM Alex Grönholm 
> wrote:
>
>> I think it would be best to gather a few extreme examples of setup.py
>> files from real world projects and figure out if they can be implemented in
>> a declarative fashion. That at least would help us identify the pain points.
>>
>> For starters, gevent's setup.py looks like it needs a fair bit of custom
>> logic:
>> https://github.com/gevent/gevent/blob/master/setup.py
>>
>> 05.05.2016, 23:30, Chris Barker kirjoitti:
>>
>> On Wed, May 4, 2016 at 7:45 PM, Nick Coghlan  wrote:
>>
>>
>>> This configuration vs customisation distinction is probably worth
>>> spelling out for folks without a formal software engineering or
>>> computer science background, so:
>>>
>>
>> fair enough -- good to be clear on the terms.
>>
>>
>>> Configuration is different: you're choosing amongst a set of
>>> possibilities that have been constrained in some way, and those
>>> constraints are structurally enforced.
>>
>>
>> That's a key point here -- I guess I'm skeptical that we can have the
>> flexibility we need with a purely configuration-based system -- we probably
>> don't WANT to constrain the options completely. If you think about it,
>> while distutils has it's many, many flaws, what made it possible for it to
>> be as useful as it is, and last as long as it has because is CAN be
>> customized -- users are NOT constrained to the built-in functionality.
>>
>> I suspect the idea of this thread is to keep the API to a build system
>> constrained -- and let the build systems themselves be as customizable as
>> the want to be. And I haven't thought it out carefully, but I have a
>> feeling that we're going to hit a wall that way .. but maybe not.
>>
>>
>>> Usually that enforcement is
>>> handled by making the configuration declarative - it's in some passive
>>> format like an ini file or JSON, and if it gets too repetitive then
>>> you introduce a config generator, rather than making the format itself
>>> more sophisticated.
>>>
>>
>> OK -- that's more or less my thought -- if it's  python that gets run,
>> then you've got your config generator built in -- why not?
>>
>>
>>
>>> The big advantage of configuration over customisation is that you
>>> substantially increase the degrees of freedom in how *consumers* of
>>> that configuration are implemented - no longer do you need a full
>>> Python runtime (or whatever), you just need an ini file parser, or a
>>> JSON decoder, and then you can look at just the bits you care about
>>> for your particular use case and ignore the rest.
>>>
>>
>> Sure -- but do we care? this is about python packaging -- is it too big a
>> burden to say you need python to read the configuration?
>>
>> -CHB
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR(206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115   (206) 526-6317   main reception
>>
>> chris.bar...@noaa.gov
>>
>>
>>
>> ___
>> Distutils-SIG maillist  -  
>> Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

We've spent a huge amount of effort on reaching the point where pretty
> much everything *can* be made pip installable. Heck, *PyQt5*, which is
> my personal benchmark for a probably-totally-unpackageable package,
> announced last week that they now have binary wheels on pypi for all
> of Win/Mac/Linux:
>

that's thanks to binary wheels being a simple archive -- by definition, if
you can build and install it *somehow* you can make a binary wheel out of
it -- which is a great thing -- so nice to have wheel completely
independent of the build system.

isn't that where we are trying to go -- have pip be independent of the
build system?

> If the BDFL delegate makes a call - fine. I read
> > Nick's earlier email in the thread as such a thing TBH :).
>
> Oh sure, I think everyone agrees that the file format choice is not a
> make-or-break decision


Am I the only one that thinks we're conflating two things here? choice of
file format is separate from "use setup.cfg" -- I think we should use
something NEW -- rather than try to patch on top of existing setup.cfg
files -- that way we're free to change the rules, and no one (and no tools)
will be confused about what version of what system is in play.

In short -- we should do something new, not just extend setup.cfg, and not
because we want a new file format.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Daniel Holth]

This is a recurring point of confusion. setup.py is not ever going away. In
general it is necessary for you to be able to write software to build your
software, and there is no intention to take that feature away.

People repeatedly come to the conclusion that static metadata means the
entire build is static. It's only the dependencies that need to be static
to enable better dependency resolution in pip. The build does not need to
be static.

The proposed feature means you will be able to have a simpler setup.py or
no setup.py it by using something like flit or pbr that are configured with
.cfg or .ini. setup.py is not going away.

Static metadata means the list of dependencies, author name, trove
classifiers is static. Not the build itself.

Enforced staticness of the build script is right out.

On Thu, May 5, 2016 at 4:34 PM Alex Grönholm 
wrote:

> I think it would be best to gather a few extreme examples of setup.py
> files from real world projects and figure out if they can be implemented in
> a declarative fashion. That at least would help us identify the pain points.
>
> For starters, gevent's setup.py looks like it needs a fair bit of custom
> logic:
> https://github.com/gevent/gevent/blob/master/setup.py
>
> 05.05.2016, 23:30, Chris Barker kirjoitti:
>
> On Wed, May 4, 2016 at 7:45 PM, Nick Coghlan  wrote:
>
>
>> This configuration vs customisation distinction is probably worth
>> spelling out for folks without a formal software engineering or
>> computer science background, so:
>>
>
> fair enough -- good to be clear on the terms.
>
>
>> Configuration is different: you're choosing amongst a set of
>> possibilities that have been constrained in some way, and those
>> constraints are structurally enforced.
>
>
> That's a key point here -- I guess I'm skeptical that we can have the
> flexibility we need with a purely configuration-based system -- we probably
> don't WANT to constrain the options completely. If you think about it,
> while distutils has it's many, many flaws, what made it possible for it to
> be as useful as it is, and last as long as it has because is CAN be
> customized -- users are NOT constrained to the built-in functionality.
>
> I suspect the idea of this thread is to keep the API to a build system
> constrained -- and let the build systems themselves be as customizable as
> the want to be. And I haven't thought it out carefully, but I have a
> feeling that we're going to hit a wall that way .. but maybe not.
>
>
>> Usually that enforcement is
>> handled by making the configuration declarative - it's in some passive
>> format like an ini file or JSON, and if it gets too repetitive then
>> you introduce a config generator, rather than making the format itself
>> more sophisticated.
>>
>
> OK -- that's more or less my thought -- if it's  python that gets run,
> then you've got your config generator built in -- why not?
>
>
>
>> The big advantage of configuration over customisation is that you
>> substantially increase the degrees of freedom in how *consumers* of
>> that configuration are implemented - no longer do you need a full
>> Python runtime (or whatever), you just need an ini file parser, or a
>> JSON decoder, and then you can look at just the bits you care about
>> for your particular use case and ignore the rest.
>>
>
> Sure -- but do we care? this is about python packaging -- is it too big a
> burden to say you need python to read the configuration?
>
> -CHB
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
>
>
> ___
> Distutils-SIG maillist  -  
> Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Wed, May 4, 2016 at 8:09 PM, Nick Coghlan  wrote:


> I know I'm one of the folks that has historically been dubious of the
> "just use setup.cfg" idea, due to the assorted problems with the
> ini-style format not extending particularly well to tree-structured
> data (beyond the single level of file sections).
>

me too :-)


> 1. We've repeatedly run up against the "JSON is good for programs

talking to each other, but lousy as a human-facing interface" problem
>

yeah, JSON is annoying that way -- but not much worse than INI -- except
for lack of comments.

(side note, I'd really like it if the json module would accept "JSON
extended with comments" as an option...)



> 3. The ongoing popularity of setup.cfg shows that while ini-style may
> not be perfect for this use case, it clearly makes it over the
> threshold of "good enough"
>

it's only popular because it's what's there -- if we're using that
standard, we could make the same argument about setuptools ;-)



> So when I ask myself now "What's the *simplest* thing we could do that
> will make things better than the status quo?", then the answer I come
> up with today is your original idea: bless setup.cfg (or at least a
> subset of it) as a standardised interface.
>

IIUC, we would be changin, or at least adding to the current setup.dfg
spec. So this is a change, no matter how you slice it, saying
"configuration will be specified in setup.something, in some other format,
is the least significant part of all this change.

And maybe it's good to keep "new style" configuration clearly separate.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Alex Grönholm]

I think it would be best to gather a few extreme examples of setup.py 
files from real world projects and figure out if they can be implemented 
in a declarative fashion. That at least would help us identify the pain 
points.


For starters, gevent's setup.py looks like it needs a fair bit of custom 
logic:

https://github.com/gevent/gevent/blob/master/setup.py

05.05.2016, 23:30, Chris Barker kirjoitti:
On Wed, May 4, 2016 at 7:45 PM, Nick Coghlan > wrote:


This configuration vs customisation distinction is probably worth
spelling out for folks without a formal software engineering or
computer science background, so:


fair enough -- good to be clear on the terms.

Configuration is different: you're choosing amongst a set of
possibilities that have been constrained in some way, and those
constraints are structurally enforced. 



That's a key point here -- I guess I'm skeptical that we can have the 
flexibility we need with a purely configuration-based system -- we 
probably don't WANT to constrain the options completely. If you think 
about it, while distutils has it's many, many flaws, what made it 
possible for it to be as useful as it is, and last as long as it has 
because is CAN be customized -- users are NOT constrained to the 
built-in functionality.


I suspect the idea of this thread is to keep the API to a build system 
constrained -- and let the build systems themselves be as customizable 
as the want to be. And I haven't thought it out carefully, but I have 
a feeling that we're going to hit a wall that way .. but maybe not.


Usually that enforcement is
handled by making the configuration declarative - it's in some passive
format like an ini file or JSON, and if it gets too repetitive then
you introduce a config generator, rather than making the format itself
more sophisticated.


OK -- that's more or less my thought -- if it's  python that gets run, 
then you've got your config generator built in -- why not?


The big advantage of configuration over customisation is that you
substantially increase the degrees of freedom in how *consumers* of
that configuration are implemented - no longer do you need a full
Python runtime (or whatever), you just need an ini file parser, or a
JSON decoder, and then you can look at just the bits you care about
for your particular use case and ignore the rest.


Sure -- but do we care? this is about python packaging -- is it too 
big a burden to say you need python to read the configuration?


-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov 


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Wed, May 4, 2016 at 7:45 PM, Nick Coghlan  wrote:


> This configuration vs customisation distinction is probably worth
> spelling out for folks without a formal software engineering or
> computer science background, so:
>

fair enough -- good to be clear on the terms.


> Configuration is different: you're choosing amongst a set of
> possibilities that have been constrained in some way, and those
> constraints are structurally enforced.


That's a key point here -- I guess I'm skeptical that we can have the
flexibility we need with a purely configuration-based system -- we probably
don't WANT to constrain the options completely. If you think about it,
while distutils has it's many, many flaws, what made it possible for it to
be as useful as it is, and last as long as it has because is CAN be
customized -- users are NOT constrained to the built-in functionality.

I suspect the idea of this thread is to keep the API to a build system
constrained -- and let the build systems themselves be as customizable as
the want to be. And I haven't thought it out carefully, but I have a
feeling that we're going to hit a wall that way .. but maybe not.


> Usually that enforcement is
> handled by making the configuration declarative - it's in some passive
> format like an ini file or JSON, and if it gets too repetitive then
> you introduce a config generator, rather than making the format itself
> more sophisticated.
>

OK -- that's more or less my thought -- if it's  python that gets run, then
you've got your config generator built in -- why not?



> The big advantage of configuration over customisation is that you
> substantially increase the degrees of freedom in how *consumers* of
> that configuration are implemented - no longer do you need a full
> Python runtime (or whatever), you just need an ini file parser, or a
> JSON decoder, and then you can look at just the bits you care about
> for your particular use case and ignore the rest.
>

Sure -- but do we care? this is about python packaging -- is it too big a
burden to say you need python to read the configuration?

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Wed, May 4, 2016 at 3:28 PM, Paul Moore  wrote:

> On 4 May 2016 at 23:11, Chris Barker  wrote:
> > so it could be purely declarative, but users could also put code in
> there to
> > customize the configuration on the fly, too.
>
> That basically repeats the mistake that was made with setup.py. We
> explicitly don't want an executable format for specifying build
> configuration.
>

I don't think it's the same thing -- setup.py is supposed to actually CALL
setup() when it is imported or run. It is not declarative in the least.

what I'm suggesting is that the API be purely declarative:

"These values will be declared" (probably a single dict of stuff)

but if you make it an executable python file, then users can write code to
actually create their setup.

Alternatively, you make the API purely declarative, and then folks have to
write external scripts that create/populate the configuration files.

I guess where I'm coming from is that I'm not sure we CAN make a purely,
completely declarative API to a build system -- folks are always going to
find some weird corner case where they need to do something special. making
the configuration file executable is an easy way to allow this. Otherwise,
you need to provide hooks to plug in custom functionality, which is jsut
alot more awkward, though I suppose more robust.

I've found it very handy to use an executable python file for configuration
for a number of my projects -- it's really nice to just throw in a little
calculation for something on the fly -- even if it could be done purely
declaratively


Anyway, if you stick with purely declarative, you can still use Python
literals as an alternative to JSON (Or INI, or, ). Python literals are
richer, more flexible, and, of course, familiar to anyone developing a
python package :-)

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Paul Moore]

On 5 May 2016 at 13:36, Daniel Holth  wrote:
> Here's the kind of thing that you should expect. Someone will write
>
> setup.cfg:
>
> [bootstrap_requires]
> pbr
>
> pip installs pbr in a directory that is added to PYTHONPATH for that build.

Ah, so we don't install into the environment, just to a temporary
location which we add to PYTHONPATH. That is indeed a much better
approach than what I assumed, as it avoids any messing with the user's
actual environment. (We may need to be a bit more clever than just
adding the temporary directory to PYTHONPATH, to ensure that what we
install *overrides* anything in the user's environment, such as a
newer version of setuptools, but that's just implementation details).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 5 May 2016 at 22:36, Daniel Holth  wrote:
> Pedantic note
>
> setup_requires is a setuptools parameter used to install packages after
> setup() is called. Even though very many people expect or want those
> packages to be installed before setup.py executes. I think it is reasonable
> to call the new feature setup_requires but some prefer to eliminate
> uncertainty by calling it bootstrap_requires.

The main advantage of a new feature name is that when someone searches
the internet for "python bootstrap_requires", they won't find a
decade+ worth of inapplicable documentation of the setuptools feature
:)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Daniel Holth]

Here's the kind of thing that you should expect. Someone will write

setup.cfg:

[bootstrap_requires]
pbr

pip installs pbr in a directory that is added to PYTHONPATH for that build.

The package builds.

And there was much rejoicing.

The big gain from this simple feature is that people will be able to re-use
code and have actual abstractions in setup.py. People have been desperate
for this feature for years.

Imagine that someone forgets to declare their build dependency. Do enough
hand-holding to build a locally cached wheel of the package and submit a
patch to the author. Easy.

In exchange for this potential inconvenience you will get dramatically
easier to write setup.py.

Here is my own implementation of the feature in 34 lines to be prepended to
setup.py: https://bitbucket.org/dholth/setup-requires/src/tip/setup.py , I
see Nick linked to it already, it only executes pip in a subprocess if the
requirements are not already installed but your regularly scheduled
setup.py can just go at the bottom of the file and runs in the same process.

Pedantic note

setup_requires is a setuptools parameter used to install packages after
setup() is called. Even though very many people expect or want those
packages to be installed before setup.py executes. I think it is reasonable
to call the new feature setup_requires but some prefer to eliminate
uncertainty by calling it bootstrap_requires.

On Thu, May 5, 2016 at 6:42 AM Paul Moore  wrote:

> On 5 May 2016 at 10:10, Nathaniel Smith  wrote:
> > ...The main thing I want to point out though, is that all of these
> > problems you're raising are complications caused entirely by wanting
> > to avoid build isolation in the name of simplicity. If each package
> > gets its own isolated build environment, then it can depend on
> > whatever it wants without any danger of collision with the ambient
> > environment.
>
> Understood. But that doesn't mean they can *only* be solved by build
> isolation. One relatively naive approach might be:
>
> 1. Collect build requirements. If they are satisfied, fine, move on to
> build.
> 2. If the only unsatisfied requirements are new installs, install them
> and move on to build.
> 3. If there are unsatisfied requirements needing an upgrade, do the
> upgrades (possibly require a flag for the pip invocation to allow
> silent upgrades), and move on to build.
> 4. Anything more complicated (downgrades, conflicts) report the issue
> and stop to let the user resolve it manually.
>
> Sure, build isolation might be better (probably is, IMO, but that's
> clearly a topic for some serious debate) but the whole point here is
> to do something incremental, and "good enough to make things better",
> not to stall on a completely perfect solution.
>
> > Premise 1: Without build isolation enabled by default, then in
> > practice everyone will putter along putting up with broken builds all
> > the time. It's *incredibly* easy to forget to declare a build
> > dependency, it's the kind of mistake that every new user makes, and
> > experienced users too.
>
> That's certainly possible, but "some missing dependencies" is still
> better than "no way to specify dependencies at all". And all it needs
> is an end user to hit the problem, raise an issue, and the dependency
> gets added and we're closer to perfection. Why is that worse than the
> current status quo?
>
> > Premise 2: We can either enable build isolation together with the new
> > static bootstrap requirements, or we can never enable build isolation
> > at all, ever.
>
> I have no idea why you feel that's the case. Why can't we add build
> isolation later? You clearly have more experience in this area than I
> do, so I'm genuinely trying to see what I'm missing here. You mention
> "we can't add it later without breaking everything". But the same
> applies now surely? And why can't we introduce the feature gradually -
> initially as opt-in via an "--isolated-build" flag, then in a later
> release change the default to "--isolated-build" with
> "--no-isolated-build" as a fallback for people who haven't fixed their
> builds yet, and ultimately removing "--no-isolated-build" and making
> isolated builds the only option? That's the standard approach we use,
> and I see no reason it wouldn't work here for introducing isolated
> builds at *any* time.
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Paul Moore]

On 5 May 2016 at 10:10, Nathaniel Smith  wrote:
> ...The main thing I want to point out though, is that all of these
> problems you're raising are complications caused entirely by wanting
> to avoid build isolation in the name of simplicity. If each package
> gets its own isolated build environment, then it can depend on
> whatever it wants without any danger of collision with the ambient
> environment.

Understood. But that doesn't mean they can *only* be solved by build
isolation. One relatively naive approach might be:

1. Collect build requirements. If they are satisfied, fine, move on to build.
2. If the only unsatisfied requirements are new installs, install them
and move on to build.
3. If there are unsatisfied requirements needing an upgrade, do the
upgrades (possibly require a flag for the pip invocation to allow
silent upgrades), and move on to build.
4. Anything more complicated (downgrades, conflicts) report the issue
and stop to let the user resolve it manually.

Sure, build isolation might be better (probably is, IMO, but that's
clearly a topic for some serious debate) but the whole point here is
to do something incremental, and "good enough to make things better",
not to stall on a completely perfect solution.

> Premise 1: Without build isolation enabled by default, then in
> practice everyone will putter along putting up with broken builds all
> the time. It's *incredibly* easy to forget to declare a build
> dependency, it's the kind of mistake that every new user makes, and
> experienced users too.

That's certainly possible, but "some missing dependencies" is still
better than "no way to specify dependencies at all". And all it needs
is an end user to hit the problem, raise an issue, and the dependency
gets added and we're closer to perfection. Why is that worse than the
current status quo?

> Premise 2: We can either enable build isolation together with the new
> static bootstrap requirements, or we can never enable build isolation
> at all, ever.

I have no idea why you feel that's the case. Why can't we add build
isolation later? You clearly have more experience in this area than I
do, so I'm genuinely trying to see what I'm missing here. You mention
"we can't add it later without breaking everything". But the same
applies now surely? And why can't we introduce the feature gradually -
initially as opt-in via an "--isolated-build" flag, then in a later
release change the default to "--isolated-build" with
"--no-isolated-build" as a fallback for people who haven't fixed their
builds yet, and ultimately removing "--no-isolated-build" and making
isolated builds the only option? That's the standard approach we use,
and I see no reason it wouldn't work here for introducing isolated
builds at *any* time.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 5 May 2016 at 19:47, Nathaniel Smith  wrote:
> The reason I'm being so intense about this is that AFAICT these are all true:
>
> Premise 1: Without build isolation enabled by default, then in
> practice everyone will putter along putting up with broken builds all
> the time. It's *incredibly* easy to forget to declare a build
> dependency, it's the kind of mistake that every new user makes, and
> experienced users too.
>
> Premise 2: We can either enable build isolation together with the new
> static bootstrap requirements, or we can never enable build isolation
> at all, ever.
>
> Conclusion: If we want to ever reach a state where builds are
> reliable, we need to tie build isolation to the new static metadata.

OK, I think I see where we're talking past each other here.

Yes, being able to do isolated builds is important, but we don't need
to invent a Python specific solution to build isolation, as build
isolation can already be handled by running a build in a fresh VM, or
in a container, and continuous integration systems already let people
do exactly that.

This means that even if the original publisher of a package doesn't
regularly run a "Can I reliably rebuild this from source on a clean
system?" check, plenty of consumers of their software will, and those
folks will complain if the build dependencies are wrong. (And folks
consuming pre-built binaries won't care in the first place).

Longer term, as an example of what increasing automation makes
possible, folks in Fedora are exploring what would be involved in
doing automatic mass rebuilds of PyPI as RPM packages [1], and I
assume they'll eventually get to a point where the problems in the
automation pipeline are ironed out, so they'll instead be looking at
problems like expressing external deps in the upstream metadata [2],
as well as finding errors in the dependency definitions of individual
packages.

The only thing that absolutely *has* to be handled centrally by
distutils-sig is ensuring that build requirements can be expressed
accurately enough to allow for fully automated builds on a clean
system. Everything else (including quality assurance on build
dependencies) is more amenable to distributed effort.

Cheers,
Nick.

[1] 
http://miroslav.suchy.cz/blog/archives/2016/04/21/wip_rebuilding_all_pypi_modules_as_rpm_packages/index.html
[2] https://github.com/pypa/interoperability-peps/pull/30

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 5 May 2016 at 14:22, Nathaniel Smith  wrote:
> On Wed, May 4, 2016 at 6:28 AM, Nick Coghlan  wrote:
>> On 4 May 2016 at 23:00, Daniel Holth  wrote:
>>> +1 It would be great to start with a real setup_requires and probably would
>>> not interfere with later build system abstractions at all.
>>
>> If we're going to go down that path, perhaps it might make sense to
>> just define a standard [setup_requires] section in setup.cfg?
>>
>> Quite a few projects already have one of those thanks to distutiils2,
>> d2to1 and pbr, which means the pragmatic approach here might be to ask
>> what needs to change so the qualifier can be removed from this current
>> observation in the PBR docs: "The setup.cfg file is an ini-like file
>> that can mostly replace the setup.py file."
>>
>> The build system abstraction config could then also just be another
>> setup.cfg section.
>
> I'm sympathetic to the general approach, but on net I think I prefer a
> slightly different proposal.
>
> Downsides to just standardizing [setup_requires]:
>
> - if projects have existing ones, that's actually kinda good / kinda
> bad -- pip has never been respecting these before, so if we suddenly
> start doing that then existing stuff could break. I don't know how
> likely this is, but experience suggests that *something* will break
> and make someone angry.

(Robert & Nathaniel worked through this while I was writing it, but I
figured it was worth clarifying my expectations anyway)

I'm not aware of any current widely adopted systems that use a
"setup_requires" or comparable bootstrapping section in setup.cfg -
they pass setup_requires to the setup() call in setup.py. Examples:

pbr: http://docs.openstack.org/developer/pbr/#setup-py
d2to1: https://pypi.python.org/pypi/d2to1

I didn't find any clear examples like those for numpy.distutils, but
it seems it works the same way.

That's the whole problem: as long as people have to do their build
dependency bootstrapping *from* setup.py, module level imports in
setup.py don't work, and any replaced packages (such as a newer or
older setuptools) need to be injected into an already running Python
process.

The closest thing that we have today to a bootstrapping system that's
independent of easy_install that I'm aware of is Daniel Holth's
concept demonstrator that uses setup.py to read a multi-valued key
from setup.cfg, uses pip to install the extracted modules, and then
goes on to execute a "real-setup.py" in a subprocess:
https://bitbucket.org/dholth/setup-requires/src

The fact that the "Python-style ini format" used for setup.cfg is an
implementation defined format defined by the standard library's
configparser module rather than a properly specified language
independent format is certainly a downside, but we recently decided
we're OK with assuming "Python related build tools will be implemented
in Python (or at least have a Python shim)" when opting for a Python
API rather than a CLI as the preferred build system abstraction.

Changing that one assumption (i.e. that we're actually OK with using
Python-centric interoperability formats where they make pragmatic
sense) means that a number of other previously rejected ideas (like
"let's just use a new section or multi-valued key in setup.cfg") can
also be reconsidered.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nathaniel Smith]

On Wed, May 4, 2016 at 11:57 PM, Robert Collins
 wrote:
> On 5 May 2016 at 18:32, Nathaniel Smith  wrote:
>> On Wed, May 4, 2016 at 10:42 PM, Robert Collins
>>...
>>> Yes, things will break: anyone using this will need a new pip, by
>>> definition. Not everyone will be willing to wait 10 years before using
>>> it :).
>>
>> Just to clarify (since we seem to agree): I meant that if pip starts
>> interpreting an existing setup.cfg thing, then the new-pip/old-package
>> situation could break, which would be bad.
>
> No. Old pip new package will break, new pip old package is entirely safe 
> AFAICT.

We're talking past each other... I'm saying, *if* pip started
reinterpreting some existing thing as indicating setup-requirements,
*then* things would break. You're saying, pip isn't going to do that,
so they won't. So we're good :-)

 - IMO an extremely valuable aspect of this new declarative
 setup-requirements thing is that it gives us an opportunity to switch
 to enforcing the accuracy of this metadata. Right now we have a swamp
 we need to drain, where there's really no way to know what environment
 any given setup.py needs to run. Sometimes there are setup_requires,
 sometimes not; if there are setup_requires then sometimes they're
>>>
>>> Huh? I've not encountered any of this, ever. I'd love some examples to
>>> go look at. The only issue I've ever had with setup_requires is the
>>> easy_install stuff it ties into.
>>
>> I don't think I've ever seen a package that had accurate
>> setup_requires (outside the trivial case of packages where
>> setup_requires=[] is accurate). Scientific packages in particular
>> universally have undeclared setup requirements.
>
> Are those requirements pip installable?

Either they are, or they will be soon.

> ..
>>> I'm very much against forcing isolated build environments as part of
>>> this effort. I get where you are coming from, but really it conflates
>>> two entirely separate things, and you'll *utterly* break building
>>> anything with dependencies that are e.g. SWIG based unless you
>>> increase the size of the PEP by about 10-fold. (Thats not hyperbole, I
>>> think).
>>
>> Okay, now's my turn to be baffled :-). I literally have no idea what
>> you're talking about here. What would this 10x longer PEP be talking
>> about? Why would this be needed?
>
> Take an i386 linux machine, and build something needing pyqt5 on it
> :). Currently, you apt-get/yum/dnf/etc install python-pyqt5, then run
> pip install. If using a virtualenv you enable system site packages.
>
> When you introduce isolation, the build will only have the standard
> library + whatever is declared as a dep: and pyqt5 has no source on
> PyPI.
>
> So the 10x thing is defining how the thing doing the isolation (e.g.
> pip) should handle things that can't be installed but are already
> available on the system.
>
> And that has to tunnel all the way out to the user, because its
> context specific, its not an attribute of the dependencies per se
> (since new releases can add or remove this situation), nor of the
> consuming thing (same reason).

# User experience today on i386
$ pip install foo
<... error: missing pyqt5 ...>
$ apt install python-pyqt5
$ pip install foo

# User experience with build isolation on i386
$ pip install foo
<... error: missing pyqt5 ...>
$ apt install python-pyqt5
$ pip install --no-isolated-environment foo

It'd even be straightforward for pip to notice that the requirement
that it failed to satisfy is already satisfied by the ambient
environment, and suggest --no-isolated-environment as a solution.

> Ultimately, its not even an interopability question: pip could do
> isolated builds now, if it chose, and it has no ramifications as far
> as PEPs etc are concerned.

That's not true. In fact, it seems dangerously naive :-/

If pip just went ahead and flipped a switch to start doing isolated
builds now, then everything would burst into flame and there would be
a howling mob in the bug tracker. Sure, there's no PEP saying we
*can't* do that, but in practice it's utterly impossible.

If we roll out this feature without build isolation, then next year
we'll still be in the exact same situation we are today -- we'll have
the theoretical capability of enabling build isolation, but everything
would break, so in practice, we won't be able to.

The reason I'm being so intense about this is that AFAICT these are all true:

Premise 1: Without build isolation enabled by default, then in
practice everyone will putter along putting up with broken builds all
the time. It's *incredibly* easy to forget to declare a build
dependency, it's the kind of mistake that every new user makes, and
experienced users too.

Premise 2: We can either enable build isolation together with the new
static bootstrap requirements, or we can never enable build isolation
at all, ever.

Conclusion: If we want to ever reach a state where builds are
reliable, 

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nathaniel Smith]

On Thu, May 5, 2016 at 2:02 AM, Paul Moore  wrote:
> On 5 May 2016 at 07:57, Robert Collins  wrote:
>> We've a history in this group of biting off too much and things not
>> getting executed. We're *still* in the final phases of deploying
>> PEP-508, and it was conceptually trivial. I'm not arguing that we
>> shouldn't make things better, I'm arguing that tying two separate
>> things together because we *can* seems, based on the historical
>> record, to be unwise.
>
> This is a very good point, and ties in nicely with Nick's comment
> about taking small steps to make things better than they currently
> are.
>
> On that basis, I'd be +1 on a simple proposal to add a new "install
> this stuff before we do the build" capability that sits in setup.cfg.
> Let's keep build isolation off the table for now.
>
> There's probably enough substantive detail (I'll do my best to avoid
> bikeshedding over trivia :-)) to thrash out in that simple proposal.
> For example, if package foo specifies that it needs a new version of
> setuptools to build, is it OK for "pip install foo" to automatically
> upgrade setuptools, or should it fail with an error "your setuptools
> is too old"? If it does auto-upgrade, then if the build of foo fails,
> is it OK that we won't be able to revert the upgrade of setuptools?
> How should we handle cases where a package specifies that the it needs
> an *older* version of setuptools? I'd expect we simply bail and report
> an error for that one - it should never really happen, so why waste
> time on "clever" solutions?

Uh, realistically speaking I'm pretty sure that setuptools will at
some point in the future contain at least 1 regression. I.e. it will
totally happen that at some point someone will pin an old version of
setuptools in their build requirements.

...The main thing I want to point out though, is that all of these
problems you're raising are complications caused entirely by wanting
to avoid build isolation in the name of simplicity. If each package
gets its own isolated build environment, then it can depend on
whatever it wants without any danger of collision with the ambient
environment.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Paul Moore]

On 5 May 2016 at 07:57, Robert Collins  wrote:
> We've a history in this group of biting off too much and things not
> getting executed. We're *still* in the final phases of deploying
> PEP-508, and it was conceptually trivial. I'm not arguing that we
> shouldn't make things better, I'm arguing that tying two separate
> things together because we *can* seems, based on the historical
> record, to be unwise.

This is a very good point, and ties in nicely with Nick's comment
about taking small steps to make things better than they currently
are.

On that basis, I'd be +1 on a simple proposal to add a new "install
this stuff before we do the build" capability that sits in setup.cfg.
Let's keep build isolation off the table for now.

There's probably enough substantive detail (I'll do my best to avoid
bikeshedding over trivia :-)) to thrash out in that simple proposal.
For example, if package foo specifies that it needs a new version of
setuptools to build, is it OK for "pip install foo" to automatically
upgrade setuptools, or should it fail with an error "your setuptools
is too old"? If it does auto-upgrade, then if the build of foo fails,
is it OK that we won't be able to revert the upgrade of setuptools?
How should we handle cases where a package specifies that the it needs
an *older* version of setuptools? I'd expect we simply bail and report
an error for that one - it should never really happen, so why waste
time on "clever" solutions?

Anyway, we can have these sorts of debate when we get down to details.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Paul Moore]

On 5 May 2016 at 00:58, Ethan Furman  wrote:
> Somebody will have to distill that PEP, I have only an small inkling of what
> it's trying to say.

The relevant point for this discussion is "you can request that
particular packages are installed before the build step for your
package, and precisely what gets installed can depend on the Python
version".

If you're not intending to use a build system other than setuptools,
this is probably not relevant to you (other than to require a new
enough version of setuptools that has the bug fixes you need).

> As for my specific use case:  I have Python3-only files in my distribution,
> so they should only be installed on Python3 systems. Python2 systems
> generate useless errors.

That's a build system issue, so not directly relevant here.
Specifically, assuming it's the issue you reported here previously,
it's a problem in setuptools (so when it's fixed there, with this new
proposal you can specify that your build requires setuptools >= X.Y,
to ensure the bug fix is present).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 5 May 2016 at 18:32, Nathaniel Smith  wrote:
> On Wed, May 4, 2016 at 10:42 PM, Robert Collins
>...
>> Yes, things will break: anyone using this will need a new pip, by
>> definition. Not everyone will be willing to wait 10 years before using
>> it :).
>
> Just to clarify (since we seem to agree): I meant that if pip starts
> interpreting an existing setup.cfg thing, then the new-pip/old-package
> situation could break, which would be bad.

No. Old pip new package will break, new pip old package is entirely safe AFAICT.

>>> - IMO an extremely valuable aspect of this new declarative
>>> setup-requirements thing is that it gives us an opportunity to switch
>>> to enforcing the accuracy of this metadata. Right now we have a swamp
>>> we need to drain, where there's really no way to know what environment
>>> any given setup.py needs to run. Sometimes there are setup_requires,
>>> sometimes not; if there are setup_requires then sometimes they're
>>
>> Huh? I've not encountered any of this, ever. I'd love some examples to
>> go look at. The only issue I've ever had with setup_requires is the
>> easy_install stuff it ties into.
>
> I don't think I've ever seen a package that had accurate
> setup_requires (outside the trivial case of packages where
> setup_requires=[] is accurate). Scientific packages in particular
> universally have undeclared setup requirements.

Are those requirements pip installable?
..
>> I'm very much against forcing isolated build environments as part of
>> this effort. I get where you are coming from, but really it conflates
>> two entirely separate things, and you'll *utterly* break building
>> anything with dependencies that are e.g. SWIG based unless you
>> increase the size of the PEP by about 10-fold. (Thats not hyperbole, I
>> think).
>
> Okay, now's my turn to be baffled :-). I literally have no idea what
> you're talking about here. What would this 10x longer PEP be talking
> about? Why would this be needed?

Take an i386 linux machine, and build something needing pyqt5 on it
:). Currently, you apt-get/yum/dnf/etc install python-pyqt5, then run
pip install. If using a virtualenv you enable system site packages.

When you introduce isolation, the build will only have the standard
library + whatever is declared as a dep: and pyqt5 has no source on
PyPI.

So the 10x thing is defining how the thing doing the isolation (e.g.
pip) should handle things that can't be installed but are already
available on the system.

And that has to tunnel all the way out to the user, because its
context specific, its not an attribute of the dependencies per se
(since new releases can add or remove this situation), nor of the
consuming thing (same reason).

Ultimately, its not even an interopability question: pip could do
isolated builds now, if it chose, and it has no ramifications as far
as PEPs etc are concerned.
...
> What are these things that aren't pip-installable and why isn't the
> solution to fix that? I definitely don't want to break things that
> work now, but providing new features that incentivize folks to clean
> up their stuff is a good thing, surely? Yeah, it means that the
> bootstrap-requirements stuff will take some time and cleanup to
> spread, but that's life.

We've a history in this group of biting off too much and things not
getting executed. We're *still* in the final phases of deploying
PEP-508, and it was conceptually trivial. I'm not arguing that we
shouldn't make things better, I'm arguing that tying two separate
things together because we *can* seems, based on the historical
record, to be unwise.

> We've spent a huge amount of effort on reaching the point where pretty
> much everything *can* be made pip installable. Heck, *PyQt5*, which is
> my personal benchmark for a probably-totally-unpackageable package,
> announced last week that they now have binary wheels on pypi for all
> of Win/Mac/Linux:
>
>   https://pypi.python.org/pypi/PyQt5/5.6
>
> I want to work towards a world where this stuff just works, not keep
> holding the entire ecosystem back with compromise hacks to work around
> a minority of broken packages.

Sure, but the underlying problem here is that manylinux is a 90%
solve: its great for the common cases but it doesn't actually solve
the actual baseline problem: we can't represent the actual system
dependencies needed to rebuild many Python packages.

pyqt5 not having i386 is just a trivial egregious case. ARM32 and 64
is going to be super common, Power8 another one, let alone less common
but still extant and used architectures like PPC, itanium, or new ones
like x86_32 [If I remember the abbreviation right - no, its not i386].

Solve that underlying problem - great, then isolation becomes an
optimisation question for things without manylinux wheels. But if we
don't solve it then isolation becomes a 'Can build X at all' question,
which is qualitatively different.

I'm all for solving the underlying problem, but not at the cost of
*not 

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nathaniel Smith]

On Wed, May 4, 2016 at 10:42 PM, Robert Collins
 wrote:
> On 5 May 2016 at 16:22, Nathaniel Smith  wrote:
> ...
>> I'm sympathetic to the general approach, but on net I think I prefer a
>> slightly different proposal.
>>
>> Downsides to just standardizing [setup_requires]:
>
> If I write a PEP, it won't be standardising setup_requires, it will be
> standardising bootstrap requirements. The distinction is nuance but
> critical:
>
>  - we don't expect setuptools to ever need to honour it (of course, it
> could, should it choose to, and ease of adoption may suggest that it
> should)
>  - as a new feature, making it opt in allows folk to adopt it when
> they are ready; if it was literally just a new home for
> setup_requires, we may see build systems auto-populating it, and the
> potential swamp you describe below would then apply.

The main argument I was making there was that it needs to be a new
opt-in thing, so if we're agreed on that then great :-).

>> - if projects have existing ones, that's actually kinda good / kinda
>> bad -- pip has never been respecting these before, so if we suddenly
>> start doing that then existing stuff could break. I don't know how
>> likely this is, but experience suggests that *something* will break
>> and make someone angry. (I'm still blinking at getting angry
>> complaints arguing that uploading a wheel to pypi, where before there
>> was only an sdist, should be treated as a compatibility-breaking
>> change that requires a new version etc.)
>
> Yes, things will break: anyone using this will need a new pip, by
> definition. Not everyone will be willing to wait 10 years before using
> it :).

Just to clarify (since we seem to agree): I meant that if pip starts
interpreting an existing setup.cfg thing, then the new-pip/old-package
situation could break, which would be bad.

>> - IMO an extremely valuable aspect of this new declarative
>> setup-requirements thing is that it gives us an opportunity to switch
>> to enforcing the accuracy of this metadata. Right now we have a swamp
>> we need to drain, where there's really no way to know what environment
>> any given setup.py needs to run. Sometimes there are setup_requires,
>> sometimes not; if there are setup_requires then sometimes they're
>
> Huh? I've not encountered any of this, ever. I'd love some examples to
> go look at. The only issue I've ever had with setup_requires is the
> easy_install stuff it ties into.

I don't think I've ever seen a package that had accurate
setup_requires (outside the trivial case of packages where
setup_requires=[] is accurate). Scientific packages in particular
universally have undeclared setup requirements.

> ...
>> instead of punishing them -- is if we make the rule be "projects that
>> use the declarative setup-requirements feature also get isolated build
>> environments". (Then the messaging is: "see, this helps you check that
>> you actually set it up right! if your new metadata works for you in
>> testing, it'll also work for your users!) But this again would mean we
>> can't reuse the existing [setup_requires] section.
>
> I'm very much against forcing isolated build environments as part of
> this effort. I get where you are coming from, but really it conflates
> two entirely separate things, and you'll *utterly* break building
> anything with dependencies that are e.g. SWIG based unless you
> increase the size of the PEP by about 10-fold. (Thats not hyperbole, I
> think).

Okay, now's my turn to be baffled :-). I literally have no idea what
you're talking about here. What would this 10x longer PEP be talking
about? Why would this be needed?

> Working around that will require a bunch of UX work, and its
> transitive: you have to expose
> how-to-workaround-the-fact-that-all-our-deps-are-not-installable all
> the way up the chain. That's probably a good thing to do (e.g. see
> bindep, or the aborted callout-to-system-packages we proposed after
> PyCon AU last year), but tying these two things together is not
> necessary, though I can certainly see the appeal; the main impact I
> see is that it will just impede adoption.

What are these things that aren't pip-installable and why isn't the
solution to fix that? I definitely don't want to break things that
work now, but providing new features that incentivize folks to clean
up their stuff is a good thing, surely? Yeah, it means that the
bootstrap-requirements stuff will take some time and cleanup to
spread, but that's life.

We've spent a huge amount of effort on reaching the point where pretty
much everything *can* be made pip installable. Heck, *PyQt5*, which is
my personal benchmark for a probably-totally-unpackageable package,
announced last week that they now have binary wheels on pypi for all
of Win/Mac/Linux:

  https://pypi.python.org/pypi/PyQt5/5.6

I want to work towards a world where this stuff just works, not keep
holding the entire ecosystem back with compromise hacks to work 

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 5 May 2016 at 16:22, Nathaniel Smith  wrote:
...
> I'm sympathetic to the general approach, but on net I think I prefer a
> slightly different proposal.
>
> Downsides to just standardizing [setup_requires]:

If I write a PEP, it won't be standardising setup_requires, it will be
standardising bootstrap requirements. The distinction is nuance but
critical:

 - we don't expect setuptools to ever need to honour it (of course, it
could, should it choose to, and ease of adoption may suggest that it
should)
 - as a new feature, making it opt in allows folk to adopt it when
they are ready; if it was literally just a new home for
setup_requires, we may see build systems auto-populating it, and the
potential swamp you describe below would then apply.

> - if projects have existing ones, that's actually kinda good / kinda
> bad -- pip has never been respecting these before, so if we suddenly
> start doing that then existing stuff could break. I don't know how
> likely this is, but experience suggests that *something* will break
> and make someone angry. (I'm still blinking at getting angry
> complaints arguing that uploading a wheel to pypi, where before there
> was only an sdist, should be treated as a compatibility-breaking
> change that requires a new version etc.)

Yes, things will break: anyone using this will need a new pip, by
definition. Not everyone will be willing to wait 10 years before using
it :).

> - IMO an extremely valuable aspect of this new declarative
> setup-requirements thing is that it gives us an opportunity to switch
> to enforcing the accuracy of this metadata. Right now we have a swamp
> we need to drain, where there's really no way to know what environment
> any given setup.py needs to run. Sometimes there are setup_requires,
> sometimes not; if there are setup_requires then sometimes they're

Huh? I've not encountered any of this, ever. I'd love some examples to
go look at. The only issue I've ever had with setup_requires is the
easy_install stuff it ties into.
...
> instead of punishing them -- is if we make the rule be "projects that
> use the declarative setup-requirements feature also get isolated build
> environments". (Then the messaging is: "see, this helps you check that
> you actually set it up right! if your new metadata works for you in
> testing, it'll also work for your users!) But this again would mean we
> can't reuse the existing [setup_requires] section.

I'm very much against forcing isolated build environments as part of
this effort. I get where you are coming from, but really it conflates
two entirely separate things, and you'll *utterly* break building
anything with dependencies that are e.g. SWIG based unless you
increase the size of the PEP by about 10-fold. (Thats not hyperbole, I
think).

Working around that will require a bunch of UX work, and its
transitive: you have to expose
how-to-workaround-the-fact-that-all-our-deps-are-not-installable all
the way up the chain. That's probably a good thing to do (e.g. see
bindep, or the aborted callout-to-system-packages we proposed after
PyCon AU last year), but tying these two things together is not
necessary, though I can certainly see the appeal; the main impact I
see is that it will just impede adoption.

The reality, AFAICT, is that most projects with undeclared build deps
today get corrected fairly quickly: a bug is filed, folk fix it, and
we move on. A robotic system that isolates everything such that folk
*cannot* fix it is much less usable, and I'm very much in favour of
pragmatism here.

> - setup.cfg is kind of a terrible format for standardizing things
> because the only definition of the format is "read the ConfigParser
> source". You cannot parse setup.cfg from a non-Python language. And
> the *only* benefit is that it already exists; teaching pip to read
> pybuild.json or pybuild.toml instead would be completely trivial. So
> if we aren't going to try and grandfather in existing [setup_requires]
> sections, then we might as well switch to a better file format at the
> same time -- this is not the hard part.

The patch to read a single list-valued key out of setup.cfg is trivial
and shallow. We've not managed to settle on consensus on a file format
choice in a year of debate. I hold zero confidence we will going
forward either. If the BDFL delegate makes a call - fine. I read
Nick's earlier email in the thread as such a thing TBH :).

> So I like the idea of splitting out the declarative setup-requirements
> PEP from the new build system hook interface PEP, but I think that the
> way we should do this is by defining a new pybuild.whatever file like
> the ones in PEP 516 / PEP 517 (they're identical in this regard) that
> *only* has schema-version and bootstrap-requirements, and then we can
> add the build backend key as a second step.

I think the risk there is that that presumes the answer, and without
the abstract build effort actually moving forward we may be years
before we actually 

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nathaniel Smith]

On Wed, May 4, 2016 at 6:28 AM, Nick Coghlan  wrote:
> On 4 May 2016 at 23:00, Daniel Holth  wrote:
>> +1 It would be great to start with a real setup_requires and probably would
>> not interfere with later build system abstractions at all.
>
> If we're going to go down that path, perhaps it might make sense to
> just define a standard [setup_requires] section in setup.cfg?
>
> Quite a few projects already have one of those thanks to distutiils2,
> d2to1 and pbr, which means the pragmatic approach here might be to ask
> what needs to change so the qualifier can be removed from this current
> observation in the PBR docs: "The setup.cfg file is an ini-like file
> that can mostly replace the setup.py file."
>
> The build system abstraction config could then also just be another
> setup.cfg section.

I'm sympathetic to the general approach, but on net I think I prefer a
slightly different proposal.

Downsides to just standardizing [setup_requires]:

- if projects have existing ones, that's actually kinda good / kinda
bad -- pip has never been respecting these before, so if we suddenly
start doing that then existing stuff could break. I don't know how
likely this is, but experience suggests that *something* will break
and make someone angry. (I'm still blinking at getting angry
complaints arguing that uploading a wheel to pypi, where before there
was only an sdist, should be treated as a compatibility-breaking
change that requires a new version etc.)

- IMO an extremely valuable aspect of this new declarative
setup-requirements thing is that it gives us an opportunity to switch
to enforcing the accuracy of this metadata. Right now we have a swamp
we need to drain, where there's really no way to know what environment
any given setup.py needs to run. Sometimes there are setup_requires,
sometimes not; if there are setup_requires then sometimes they're
real, sometimes they're lies. We'd like to reach the point where for a
random package on pypi, either we can tell it's a legacy package, or
else we can be confident that the setup-requirements it declares are
actually accurate and sufficient to building the package. This is
important because it unblocks ecosystem improvements like automated
wheel builders, autoconfiguring CI systems, etc. And the only way
we'll get there, AFAICT, is if at some point we start enforcing build
isolation, so that by default a build *can't* import anything that
wasn't declared as a setup-requirement. And the only way I can see for
us to make this transition smoothly -- without any horrible flag days,
and with a nice carrot so projects feel like we're giving them a gift
instead of punishing them -- is if we make the rule be "projects that
use the declarative setup-requirements feature also get isolated build
environments". (Then the messaging is: "see, this helps you check that
you actually set it up right! if your new metadata works for you in
testing, it'll also work for your users!) But this again would mean we
can't reuse the existing [setup_requires] section.

- setup.cfg is kind of a terrible format for standardizing things
because the only definition of the format is "read the ConfigParser
source". You cannot parse setup.cfg from a non-Python language. And
the *only* benefit is that it already exists; teaching pip to read
pybuild.json or pybuild.toml instead would be completely trivial. So
if we aren't going to try and grandfather in existing [setup_requires]
sections, then we might as well switch to a better file format at the
same time -- this is not the hard part.

So I like the idea of splitting out the declarative setup-requirements
PEP from the new build system hook interface PEP, but I think that the
way we should do this is by defining a new pybuild.whatever file like
the ones in PEP 516 / PEP 517 (they're identical in this regard) that
*only* has schema-version and bootstrap-requirements, and then we can
add the build backend key as a second step.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

Ok so, if i draft a pep for said proposal, will it die under the weight of
a thousand bike sheds?
On 5 May 2016 3:09 PM, "Nick Coghlan"  wrote:

> On 5 May 2016 at 06:28, Robert Collins  wrote:
> > the only reason I got involved in build system discussions was
> > pushback 18months or so back when I implemented a proof of concept for
> > pip that just used setup.cfg. I'd be very happy to ignore all the
> > build system stuff and just do bootstrap requirements in setup.cfg.
>
> I know I'm one of the folks that has historically been dubious of the
> "just use setup.cfg" idea, due to the assorted problems with the
> ini-style format not extending particularly well to tree-structured
> data (beyond the single level of file sections).
>
> However, my point of view has changed over that time:
>
> 1. We've repeatedly run up against the "JSON is good for programs
> talking to each other, but lousy as a human-facing interface" problem
> 2. By way of PEPs 440 and 508, we've got a lot more experience in
> figuring out how to effectively bless de facto practices as properly
> documented standards (even when it makes the latter more complicated
> than we'd like)
> 3. The ongoing popularity of setup.cfg shows that while ini-style may
> not be perfect for this use case, it clearly makes it over the
> threshold of "good enough"
> 4. Folks that *really* want a different input format (whether that's
> YAML, TOML, or something else entirely) will still be free to treat
> setup.cfg as a generated file, just as setup.py can be a generated
> file today
>
> The last couple of years have also given me a whole range of
> opportunities (outside distutils-sig) to apply the mantra "the goal is
> to make things better than the status quo, not to make them perfect",
> and that has meant getting better at distinguishing what I would do
> given unlimited development resources from what makes the most sense
> given the development resources that are actually available.
>
> So when I ask myself now "What's the *simplest* thing we could do that
> will make things better than the status quo?", then the answer I come
> up with today is your original idea: bless setup.cfg (or at least a
> subset of it) as a standardised interface.
>
> Don't get me wrong, I still think that answer has significant
> downsides - I've just come around to the view that "is likely to be
> easy to implement and adopt" are upsides that can outweigh a whole lot
> of downsides :)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 5 May 2016 at 06:28, Robert Collins  wrote:
> the only reason I got involved in build system discussions was
> pushback 18months or so back when I implemented a proof of concept for
> pip that just used setup.cfg. I'd be very happy to ignore all the
> build system stuff and just do bootstrap requirements in setup.cfg.

I know I'm one of the folks that has historically been dubious of the
"just use setup.cfg" idea, due to the assorted problems with the
ini-style format not extending particularly well to tree-structured
data (beyond the single level of file sections).

However, my point of view has changed over that time:

1. We've repeatedly run up against the "JSON is good for programs
talking to each other, but lousy as a human-facing interface" problem
2. By way of PEPs 440 and 508, we've got a lot more experience in
figuring out how to effectively bless de facto practices as properly
documented standards (even when it makes the latter more complicated
than we'd like)
3. The ongoing popularity of setup.cfg shows that while ini-style may
not be perfect for this use case, it clearly makes it over the
threshold of "good enough"
4. Folks that *really* want a different input format (whether that's
YAML, TOML, or something else entirely) will still be free to treat
setup.cfg as a generated file, just as setup.py can be a generated
file today

The last couple of years have also given me a whole range of
opportunities (outside distutils-sig) to apply the mantra "the goal is
to make things better than the status quo, not to make them perfect",
and that has meant getting better at distinguishing what I would do
given unlimited development resources from what makes the most sense
given the development resources that are actually available.

So when I ask myself now "What's the *simplest* thing we could do that
will make things better than the status quo?", then the answer I come
up with today is your original idea: bless setup.cfg (or at least a
subset of it) as a standardised interface.

Don't get me wrong, I still think that answer has significant
downsides - I've just come around to the view that "is likely to be
easy to implement and adopt" are upsides that can outweigh a whole lot
of downsides :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 5 May 2016 at 08:28, Paul Moore  wrote:
> On 4 May 2016 at 23:11, Chris Barker  wrote:
>> so it could be purely declarative, but users could also put code in there to
>> customize the configuration on the fly, too.
>
> That basically repeats the mistake that was made with setup.py. We
> explicitly don't want an executable format for specifying build
> configuration.

This configuration vs customisation distinction is probably worth
spelling out for folks without a formal software engineering or
computer science background, so:

Customisation is programming: you're writing plugins in a Turing
complete language (where "Turing complete" is a computer science
classification that says "any program that can be expressed at all,
can be expressed in this language", although it doesn't make any
promises about readability). As such, Python "config files" like
setup.py in distutils and settings.py in Django are actually software
*customisation*  interfaces - the only way to reliably interact with
them is to run them and see what happens. Anything you do via static
analysis rather than code execution is a *heuristic* that may fail on
some code that "works" when executed.

Configuration is different: you're choosing amongst a set of
possibilities that have been constrained in some way, and those
constraints are structurally enforced. Usually that enforcement is
handled by making the configuration declarative - it's in some passive
format like an ini file or JSON, and if it gets too repetitive then
you introduce a config generator, rather than making the format itself
more sophisticated.

The big advantage of configuration over customisation is that you
substantially increase the degrees of freedom in how *consumers* of
that configuration are implemented - no longer do you need a full
Python runtime (or whatever), you just need an ini file parser, or a
JSON decoder, and then you can look at just the bits you care about
for your particular use case and ignore the rest.

Regards,
Nick.

P.S. Fun fact: XML (as formally specified) is a customisation
language, rather than a configuration language. If you want to safely
use it as a configuration language, you need to use a library like
defusedxml to cut it down to a non-Turing complete subset of the spec.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Daniel Holth]

The only part that needs to be static is the metadata. The actual build can
have code without hindering our dependency resolution aspirations.

On Wed, May 4, 2016, 19:59 Ethan Furman  wrote:

> On 05/04/2016 04:29 PM, Alex Grönholm wrote:
>
> > Different files for what? Something not covered by PEP 508?
>
> Somebody will have to distill that PEP, I have only an small inkling of
> what it's trying to say.
>
> As for my specific use case:  I have Python3-only files in my
> distribution, so they should only be installed on Python3 systems.
> Python2 systems generate useless errors.
>
> --
> ~Ethan~
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Ethan Furman]

On 05/04/2016 04:29 PM, Alex Grönholm wrote:


Different files for what? Something not covered by PEP 508?


Somebody will have to distill that PEP, I have only an small inkling of 
what it's trying to say.


As for my specific use case:  I have Python3-only files in my 
distribution, so they should only be installed on Python3 systems. 
Python2 systems generate useless errors.


--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Alex Grönholm]

Different files for what? Something not covered by PEP 508?

05.05.2016, 02:23, Ethan Furman kirjoitti:

On 05/04/2016 03:28 PM, Paul Moore wrote:

On 4 May 2016 at 23:11, Chris Barker wrote:




That basically repeats the mistake that was made with setup.py. We
explicitly don't want an executable format for specifying build
configuration.


Executable code or not, we need to be able to specify different files 
depending on the python version.


--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Ethan Furman]

On 05/04/2016 03:28 PM, Paul Moore wrote:

On 4 May 2016 at 23:11, Chris Barker wrote:




That basically repeats the mistake that was made with setup.py. We
explicitly don't want an executable format for specifying build
configuration.


Executable code or not, we need to be able to specify different files 
depending on the python version.


--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Paul Moore]

On 4 May 2016 at 23:11, Chris Barker  wrote:
> so it could be purely declarative, but users could also put code in there to
> customize the configuration on the fly, too.

That basically repeats the mistake that was made with setup.py. We
explicitly don't want an executable format for specifying build
configuration.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Wed, May 4, 2016 at 3:33 AM, Donald Stufft  wrote:

> I'd actually prefer not using JSON for something that is human
> editable/writable because I think it's a pretty poor format for that case.
> It
> is overly restrictive in what it allows (for instance, no trailing comma
> gets
> me every time) and the lack of comments I think make it a poor format for
> that.
>

yup -- these are really annoying when JSON is used for a config format.

but INI pretty much sucks, too.

What about PYSON (my term) -- python literals -- could be evaluated with
ast.literal_eval to be safe, and would give us comments, and trailing
commas, and python's richer data types.

or just plain Python -- the file would be imported and we'd specify
particular variables that needed to be defined -- maybe as simple as:

config = a_big_dict_with_lots_of_stuff_in_it.

so it could be purely declarative, but users could also put code in there
to customize the configuration on the fly, too.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 4 May 2016 at 22:33, Donald Stufft  wrote:
>
..> I also believe that we can't provide a replacement for setup.py
without either
> purposely declaring we no longer support something that people used from it or
> providing a way to support that in the new, setup.py-less format.

The thunk I wrote being insufficient?

> One thing I remarked to Nataniel yesterday was that it might be a good idea to
> drop the build system aspect of these for right now (since afaict all the
> invested parties are currently overloaded and/or have a lack of time) and 
> focus
> soley on the part of the proposal that enables us to get a real setup_requires
...

the only reason I got involved in build system discussions was
pushback 18months or so back when I implemented a proof of concept for
pip that just used setup.cfg. I'd be very happy to ignore all the
build system stuff and just do bootstrap requirements in setup.cfg.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Daniel Holth]

Just call it Steve

On Wed, May 4, 2016, 16:25 Robert Collins  wrote:

> On 4 May 2016 at 19:39, Nick Coghlan  wrote:
> > On 4 May 2016 at 16:03, Robert Collins 
> wrote:
> >> The edits I'd expect to make if the conclusions I suggested in
> >> https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
> >> are adopted are:
> >>
> >>  - change to a Python API
> >>  - BFDL call on the file format and name
> >>
> >> There is no need to issue a new sdist thing, because sdists today are
> >> *already* documented across PEPs 241, 314 and 345.
> >
> > I already +1'ed using a Python API, but on the file name & format
> > side, we have the following candidates and prior art floating around:
> >
> > pypa.json in PEP 516
> > pypackage.json in PEP 517
> > pydist.json in PEP 426
> > METADATA (Key: Value) in sdists and wheels
> > WHEEL (Key: Value) in wheels
> >
> > My impression is that we're generally agreed on wanting to move from
> > Key:Value to JSON as the baseline for interoperability formats, so my
> > suggestion is to use the name "pybuild.json".
> >
> > The problem I have with pypa/pypackage/pydist is that they're all too
> > broad - we're moving towards an explicitly multi-stage pipeline (tree
> > -> sdist -> wheel -> installed) and additional metadata gets added at
> > each step. The "pybuild.json" metadata specifically covers how to get
> > from a source tree or sdist to a built wheel file, so I think it makes
> > sense to use a name that reflects that.
>
> I don't think we have anything resembling consensus on that pipeline idea.
>
> pybuild.json would be fine as a name though :).
>
> -Rob
>
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 4 May 2016 at 19:39, Nick Coghlan  wrote:
> On 4 May 2016 at 16:03, Robert Collins  wrote:
>> The edits I'd expect to make if the conclusions I suggested in
>> https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
>> are adopted are:
>>
>>  - change to a Python API
>>  - BFDL call on the file format and name
>>
>> There is no need to issue a new sdist thing, because sdists today are
>> *already* documented across PEPs 241, 314 and 345.
>
> I already +1'ed using a Python API, but on the file name & format
> side, we have the following candidates and prior art floating around:
>
> pypa.json in PEP 516
> pypackage.json in PEP 517
> pydist.json in PEP 426
> METADATA (Key: Value) in sdists and wheels
> WHEEL (Key: Value) in wheels
>
> My impression is that we're generally agreed on wanting to move from
> Key:Value to JSON as the baseline for interoperability formats, so my
> suggestion is to use the name "pybuild.json".
>
> The problem I have with pypa/pypackage/pydist is that they're all too
> broad - we're moving towards an explicitly multi-stage pipeline (tree
> -> sdist -> wheel -> installed) and additional metadata gets added at
> each step. The "pybuild.json" metadata specifically covers how to get
> from a source tree or sdist to a built wheel file, so I think it makes
> sense to use a name that reflects that.

I don't think we have anything resembling consensus on that pipeline idea.

pybuild.json would be fine as a name though :).

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Daniel Holth]

Agree.

On Wed, May 4, 2016, 09:28 Nick Coghlan  wrote:

> On 4 May 2016 at 23:00, Daniel Holth  wrote:
> > +1 It would be great to start with a real setup_requires and probably
> would
> > not interfere with later build system abstractions at all.
>
> If we're going to go down that path, perhaps it might make sense to
> just define a standard [setup_requires] section in setup.cfg?
>
> Quite a few projects already have one of those thanks to distutiils2,
> d2to1 and pbr, which means the pragmatic approach here might be to ask
> what needs to change so the qualifier can be removed from this current
> observation in the PBR docs: "The setup.cfg file is an ini-like file
> that can mostly replace the setup.py file."
>
> The build system abstraction config could then also just be another
> setup.cfg section.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 4 May 2016 at 23:00, Daniel Holth  wrote:
> +1 It would be great to start with a real setup_requires and probably would
> not interfere with later build system abstractions at all.

If we're going to go down that path, perhaps it might make sense to
just define a standard [setup_requires] section in setup.cfg?

Quite a few projects already have one of those thanks to distutiils2,
d2to1 and pbr, which means the pragmatic approach here might be to ask
what needs to change so the qualifier can be removed from this current
observation in the PBR docs: "The setup.cfg file is an ini-like file
that can mostly replace the setup.py file."

The build system abstraction config could then also just be another
setup.cfg section.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Daniel Holth]

+1 It would be great to start with a real setup_requires and probably would
not interfere with later build system abstractions at all.

On Wed, May 4, 2016 at 6:33 AM Donald Stufft  wrote:

>
> > On May 4, 2016, at 3:39 AM, Nick Coghlan  wrote:
> >
> > On 4 May 2016 at 16:03, Robert Collins 
> wrote:
> >> The edits I'd expect to make if the conclusions I suggested in
> >> https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
> >> are adopted are:
> >>
> >> - change to a Python API
> >> - BFDL call on the file format and name
> >>
> >> There is no need to issue a new sdist thing, because sdists today are
> >> *already* documented across PEPs 241, 314 and 345.
> >
> > I already +1'ed using a Python API, but on the file name & format
> > side, we have the following candidates and prior art floating around:
> >
> > pypa.json in PEP 516
> > pypackage.json in PEP 517
> > pydist.json in PEP 426
> > METADATA (Key: Value) in sdists and wheels
> > WHEEL (Key: Value) in wheels
> >
> > My impression is that we're generally agreed on wanting to move from
> > Key:Value to JSON as the baseline for interoperability formats, so my
> > suggestion is to use the name "pybuild.json”.
>
> I'd actually prefer not using JSON for something that is human
> editable/writable because I think it's a pretty poor format for that case.
> It
> is overly restrictive in what it allows (for instance, no trailing comma
> gets
> me every time) and the lack of comments I think make it a poor format for
> that.
>
> I think JSON is great for what gets included *IN* a sdist or a wheel, but
> for
> what sits inside of a VCS checkout that we expect human beings to edit, I
> think
> not.
>
> I'm +1 on tying this to a new extension because I feel like it
> fundamentally
> changes what it means to be a Python sdist. It eliminates the setup.py
> which
> is just about the only thing you actually can depend on existing inside of
> a
> Python sdist and there are a lot of things out there that make the
> assumption
> that Python + ".tar.gz/.zip/etc" == has a setup.py and are going to be
> broken
> from it. A new extension means those tools will ignore it and we can bake
> in
> versioning of the format itself right from the start (even if the new
> format
> looks remarkably like the old format with a different name).
>
> I also believe that we can't provide a replacement for setup.py without
> either
> purposely declaring we no longer support something that people used from
> it or
> providing a way to support that in the new, setup.py-less format.
>
> One thing I remarked to Nataniel yesterday was that it might be a good
> idea to
> drop the build system aspect of these for right now (since afaict all the
> invested parties are currently overloaded and/or have a lack of time) and
> focus
> soley on the part of the proposal that enables us to get a real
> setup_requires
> that doesn't involve needing to do the tricky delayed import thing that the
> current implementation of setup_requires needs. That would net a pretty
> huge
> win I think since people would be able to use abstractions in their
> setup.py
> (even if they still use setuptools) through a simple ``import statement``,
> including the ability to specify what version of setuptools they need to
> build.
> People could still implement non setuptools build systems by mimicing the
> setup.py interface (though it will still continue to be less then amazingly
> documented/defined) but some of the Numpy folks from one of the previous
> threads stated that mimicing setup.py wasn't really the hard part of making
> numpy.distutils anyways. The benefit of that is not only a smaller chunk,
> but
> also the chunk that I think (and I could be wrong) that there's no real
> disagreement on about how to go about doing it (besides some bikshedding
> things
> like what the filename should be).
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Donald Stufft]

> On May 4, 2016, at 3:39 AM, Nick Coghlan  wrote:
> 
> On 4 May 2016 at 16:03, Robert Collins  wrote:
>> The edits I'd expect to make if the conclusions I suggested in
>> https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
>> are adopted are:
>> 
>> - change to a Python API
>> - BFDL call on the file format and name
>> 
>> There is no need to issue a new sdist thing, because sdists today are
>> *already* documented across PEPs 241, 314 and 345.
> 
> I already +1'ed using a Python API, but on the file name & format
> side, we have the following candidates and prior art floating around:
> 
> pypa.json in PEP 516
> pypackage.json in PEP 517
> pydist.json in PEP 426
> METADATA (Key: Value) in sdists and wheels
> WHEEL (Key: Value) in wheels
> 
> My impression is that we're generally agreed on wanting to move from
> Key:Value to JSON as the baseline for interoperability formats, so my
> suggestion is to use the name "pybuild.json”.

I'd actually prefer not using JSON for something that is human
editable/writable because I think it's a pretty poor format for that case. It
is overly restrictive in what it allows (for instance, no trailing comma gets
me every time) and the lack of comments I think make it a poor format for that.

I think JSON is great for what gets included *IN* a sdist or a wheel, but for
what sits inside of a VCS checkout that we expect human beings to edit, I think
not.

I'm +1 on tying this to a new extension because I feel like it fundamentally
changes what it means to be a Python sdist. It eliminates the setup.py which
is just about the only thing you actually can depend on existing inside of a
Python sdist and there are a lot of things out there that make the assumption
that Python + ".tar.gz/.zip/etc" == has a setup.py and are going to be broken
from it. A new extension means those tools will ignore it and we can bake in
versioning of the format itself right from the start (even if the new format
looks remarkably like the old format with a different name).

I also believe that we can't provide a replacement for setup.py without either
purposely declaring we no longer support something that people used from it or
providing a way to support that in the new, setup.py-less format.

One thing I remarked to Nataniel yesterday was that it might be a good idea to
drop the build system aspect of these for right now (since afaict all the
invested parties are currently overloaded and/or have a lack of time) and focus
soley on the part of the proposal that enables us to get a real setup_requires
that doesn't involve needing to do the tricky delayed import thing that the
current implementation of setup_requires needs. That would net a pretty huge
win I think since people would be able to use abstractions in their setup.py
(even if they still use setuptools) through a simple ``import statement``,
including the ability to specify what version of setuptools they need to build.
People could still implement non setuptools build systems by mimicing the
setup.py interface (though it will still continue to be less then amazingly
documented/defined) but some of the Numpy folks from one of the previous
threads stated that mimicing setup.py wasn't really the hard part of making
numpy.distutils anyways. The benefit of that is not only a smaller chunk, but
also the chunk that I think (and I could be wrong) that there's no real
disagreement on about how to go about doing it (besides some bikshedding things
like what the filename should be).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 4 May 2016 at 16:03, Robert Collins  wrote:
> The edits I'd expect to make if the conclusions I suggested in
> https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
> are adopted are:
>
>  - change to a Python API
>  - BFDL call on the file format and name
>
> There is no need to issue a new sdist thing, because sdists today are
> *already* documented across PEPs 241, 314 and 345.

I already +1'ed using a Python API, but on the file name & format
side, we have the following candidates and prior art floating around:

pypa.json in PEP 516
pypackage.json in PEP 517
pydist.json in PEP 426
METADATA (Key: Value) in sdists and wheels
WHEEL (Key: Value) in wheels

My impression is that we're generally agreed on wanting to move from
Key:Value to JSON as the baseline for interoperability formats, so my
suggestion is to use the name "pybuild.json".

The problem I have with pypa/pypackage/pydist is that they're all too
broad - we're moving towards an explicitly multi-stage pipeline (tree
-> sdist -> wheel -> installed) and additional metadata gets added at
each step. The "pybuild.json" metadata specifically covers how to get
from a source tree or sdist to a built wheel file, so I think it makes
sense to use a name that reflects that.

Cheers,
Nick.

P.S. If you search for "pybuild", there *are* some existing utilities
out there by that name, including a package builder for Debian. If you
search for "pybuild.json" instead, then the 3rd-ranked link, at least
for me, is Antoine Pitrou's original suggestion of that name back in
November.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 4 May 2016 at 05:10, Paul Moore  wrote:

> Nick - do you have the time to pick this up? Or does it need someone
> to step up as BDFL-delegate? Robert, Nathaniel, do you have time to
> spend on a final round of discussion on this, on the assumption that
> the goal will be a final decision at the end of it? Donald, do you
> have the time and interest to complete and publish your proposal?

I'm currently going through a redundancy process @ HPE - while I
remain convinced that sorting out the issues with packaging is crucial
for Python going forward, my time to work on it is now rather more
limited than it was. I'm not sure where I'm going to end up, nor how
much work time I'll have for this going forward: it may end up being a
personal-time-only thing. Planning wise, we have to work on 'personal
time only' going forward - which means I"m going to be very careful
about biting off more than I can chew :)

Right now, I don't see any point updating the PEP at this point.

The edits I'd expect to make if the conclusions I suggested in
https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
are adopted are:

 - change to a Python API
 - BFDL call on the file format and name

There is no need to issue a new sdist thing, because sdists today are
*already* documented across PEPs 241, 314 and 345.

So - if Nick is ready to rule, there is basically about an hour of
editing to switch the CLI to a Python API and update the file format
bits, and then we can call it provisional and encourage
implementations, update the thunk implementation and so on.

If there's more debate to be had, thats fine too, but editing the PEP
won't achieve that.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 4 May 2016 at 04:28, Nathaniel Smith  wrote:
> On Tue, May 3, 2016 at 10:10 AM, Paul Moore  wrote:
>> On 3 May 2016 at 17:47, Donald Stufft  wrote:
>>> It will likely get decided as part of the build system PEP, whenever that
>>> gets picked up again.
>>
>> Yes, but on 15th March
>> (https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html)
>> Robert posted
>>
>>> Just to set expectations: this whole process seems stalled to me; I'm
>>> going to context switch and focus on things that can move forward.
>>> Someone please ping me when its relevant to put effort in again :).
>>
>> And I think that's right. The whole build system PEP issue appears
>> stalled from a lack of someone willing (or with the time) to make a
>> call on the approach we take.
>
> No, no, Nick's not the blocker. I'm the blocker! (Sorry)

It's been an interesting couple of months, so even if you had got this
post together earlier, it's quite possible we would have ended up
blocked on me anyway :)

> Donald + Robert + I had a longish conversation about this on IRC a
> month ago [1]. I volunteered to summarize back to the mailing list,
> and then I flaked -- so I guess this is that belated email :-).
>
> Here's the tentative conclusions we came to:
>
> Blocker 1 is figuring out what to do about the sdist format. The
> debate is between keeping something that's basically the current
> format, versus somehow cleaning it up (e.g. Donald's "source wheel"
> ideas). To move forward:
> - I'll write up a PEP that attempts to just document/standardize the
> current de facto sdist format and send it to the mailing list
> (basically: filename convention, PKG-INFO + a list of which fields in
> PKG-INFO pypi actually cares about, presence of setup.py), and adds
> some sort of optional-defaulting-to-1 SDist-Version (I guess in a file
> called SDIST by analogy with WHEEL). And also contains a rationale
> section explaining the trade-offs of standardizing this versus
> creating a new extension.)
> - Donald will make his case for the new extension approach on the mailing list
> - We beg Nick to read over both things and make a ruling so we can move on

+1 for just documenting the sdist-we-have-today, and avoiding making
the build system decoupling proposals dependent on any changes to
that. (That's not to say defining a better source format isn't
desirable - it's just a nice-to-have future enhancement, rather than
being essential in the near term)

One of the big reasons I stopped working on metadata 2.0 is that
there's no way for us to force re-releases of everything on PyPI,
which means we need to accommodate already published releases if we
want a new approach to be widely adopted

That's why PEP 440 (particularly its normalisation scheme) was checked
for a high level of pragmatic compatibility against PyPI's existing
contents, why PEP 508's dependency specifier syntax is closer to that
defined by setuptools than it is the PEP 345 one, and why I think
better documenting the current sdist format is the right way forward
here.

> Blocker 2 is figuring out whether the new pip <-> build system "hook"
> interface should be command-line based (like the current draft of PEP
> 516) or Python function call based (like the current draft of PEP
> 517). It sounds like currently Donald and I are in favor of the python
> hooks approach, and Robert is indifferent between them and just wants
> to move forward, so we agreed that unless anyone objects we'll drop
> the command-line approach and go ahead with refining the Python
> function call approach. So... if you want to object then speak up now.

While I'd previously expressed a preference for a command line based
approach, the subsequent discussion persuaded me the Python API was at
least as viable, if not preferable, so I'd be happy to endorse that
approach.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Alex Grönholm]

I certainly have no problem with Daniel's suggestion (and it would be 
much better than my solution) but would involve yet more standards work. 
Who's going to do that and when?


03.05.2016, 22:39, Donald Stufft kirjoitti:

On May 3, 2016, at 3:35 PM, Daniel Holth  wrote:

Who cares exactly why it doesn't work? We know how to fix it by doing something 
different (put build dependencies in a static file and have them installed by 
pip before running setup.py).


Presumably Alex would like to know why we can’t implement his suggestion.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Donald Stufft]

> On May 3, 2016, at 3:35 PM, Daniel Holth  wrote:
> 
> Who cares exactly why it doesn't work? We know how to fix it by doing 
> something different (put build dependencies in a static file and have them 
> installed by pip before running setup.py).
> 

Presumably Alex would like to know why we can’t implement his suggestion.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Daniel Holth]

Who cares exactly why it doesn't work? We know how to fix it by doing
something different (put build dependencies in a static file and have them
installed by pip before running setup.py).

On Tue, May 3, 2016 at 3:33 PM Donald Stufft  wrote:

>
> On May 3, 2016, at 3:28 PM, Alex Grönholm 
> wrote:
>
> As I pointed out in my previous post, "extended commands" are not the only
> use case for setup_requires -- upgrading setuptools itself would enable
> support for PEP 508 style conditional requirements. This currently does not
> work because if you have such requirements in your setup(), setuptools will
> fail when parsing those before it even has a chance to act on the minimum
> setuptools requirement specification in setup_requires.
>
>
> I don’t think this would work anyways without some terrible hacks since by
> the time setup_requires have been installed setuptools has already been
> imported so the *old* version will be in sys.modules.
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Donald Stufft]

> On May 3, 2016, at 3:28 PM, Alex Grönholm  wrote:
> 
> As I pointed out in my previous post, "extended commands" are not the only 
> use case for setup_requires -- upgrading setuptools itself would enable 
> support for PEP 508 style conditional requirements. This currently does not 
> work because if you have such requirements in your setup(), setuptools will 
> fail when parsing those before it even has a chance to act on the minimum 
> setuptools requirement specification in setup_requires.

I don’t think this would work anyways without some terrible hacks since by the 
time setup_requires have been installed setuptools has already been imported so 
the *old* version will be in sys.modules.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Alex Grönholm]

As I pointed out in my previous post, "extended commands" are not the 
only use case for setup_requires -- upgrading setuptools itself would 
enable support for PEP 508 style conditional requirements. This 
currently does not work because if you have such requirements in your 
setup(), setuptools will fail when parsing those before it even has a 
chance to act on the minimum setuptools requirement specification in 
setup_requires.


03.05.2016, 21:33, Daniel Holth kirjoitti:
What happened is that only a half-dozen setuptools experts (I am not 
one of those six people) know how to write an extended command or 
whatever that would actually be able to take advantage of setup 
requirements as implemented by setuptools. Everyone else wants to 
"import x" at the top of setup.py and pass arguments to the setup() 
function. So it would be better to have the installer make that possible.


On Tue, May 3, 2016 at 2:29 PM Alex Grönholm > wrote:


No, setuptools parses the install requirements before acting on
setup requirements. That is the source of the problem. If
setuptools only parsed and acted on setup requirements before even
parsing install requirements, this wouldn't be an issue.


03.05.2016, 21:26, Leonardo Rochael Almeida kirjoitti:



On 3 May 2016 at 15:07, Alex Grönholm > wrote:

Having setuptools process the setup requirements before
parsing install requirements would be a good step forward.
Had that been done before, we could've just added a setup
requirement for a newer setuptools to enable PEP 508
conditional requirements.


Setuptools does process setup requirements before install
requirements. The "chicken and egg" issue with setuptools is
that, most of the time, setup requires are needed to calculate
information that is passed into the `setup()` call itself.

For example information on header files coming from the C api of
`numpy` which is used to build extensions.

This usually means importing code from the packages in "setup
requires" before setuptools has a chance to actually look at it.

A simple fix would be to allow `setup()` keywords to accept
functions as well as direct values and only invoke the functions
when the values are actually needed, but this idea never gained
traction.

Of course, even if this was implemented, it wouldn't help
directly with "setup requiring" a new version of setuptools
itself, unless setuptools detected this situation and reinvoked
setup.py from scratch.

Regards,

Leo


03.05.2016, 21:04, Daniel Holth kirjoitti:

We did separate build from install. Now we just want to be
able to build without [having to emulate] distutils; just
having some dependencies installed before setup.py runs
would also be a great boon.

I'm reading part of this conversation as "a simple
bdist_wheel bug is a reason to do a lot of work
standardizing file formats" which I find unfortunate.

If he is still up for it let Robert implement his own PEP as
the way forward for build system abstraction. The extra PEPs
are just delaying action.

On Tue, May 3, 2016 at 1:11 PM Paul Moore
> wrote:

On 3 May 2016 at 17:47, Donald Stufft > wrote:
> It will likely get decided as part of the build system
PEP, whenever that
> gets picked up again.

Yes, but on 15th March

(https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html)
Robert posted

> Just to set expectations: this whole process seems
stalled to me; I'm
> going to context switch and focus on things that can
move forward.
> Someone please ping me when its relevant to put effort
in again :).

And I think that's right. The whole build system PEP
issue appears
stalled from a lack of someone willing (or with the
time) to make a
call on the approach we take.

As far as I'm aware, the decision remains with Nick.
With the possible
exception of Donald's proposal (which AFAIK never got
formally
published as a PEP) everything that can be said on the
other proposals
has been said, and the remaining differences are ones of
choice of
approach rather than anything affecting capabilities.
(Robert's
message at

https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
summarised the state of the 3 proposals at 

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Paul Moore]

On 3 May 2016 at 19:28, Nathaniel Smith  wrote:
> No, no, Nick's not the blocker. I'm the blocker! (Sorry)
>
> Donald + Robert + I had a longish conversation about this on IRC a
> month ago [1]. I volunteered to summarize back to the mailing list,
> and then I flaked -- so I guess this is that belated email :-).

Not a problem - I'm glad my mail prompted some movement, but I
completely understand that other things can get in the way.

> Here's the tentative conclusions we came to:
>
> Blocker 1 is figuring out what to do about the sdist format. The
> debate is between keeping something that's basically the current
> format, versus somehow cleaning it up (e.g. Donald's "source wheel"
> ideas). To move forward:
> - I'll write up a PEP that attempts to just document/standardize the
> current de facto sdist format and send it to the mailing list
> (basically: filename convention, PKG-INFO + a list of which fields in
> PKG-INFO pypi actually cares about, presence of setup.py), and adds
> some sort of optional-defaulting-to-1 SDist-Version (I guess in a file
> called SDIST by analogy with WHEEL). And also contains a rationale
> section explaining the trade-offs of standardizing this versus
> creating a new extension.)
> - Donald will make his case for the new extension approach on the mailing list
> - We beg Nick to read over both things and make a ruling so we can move on

Even though I was one who wanted a properly defined sdist format, I'm
inclined to be OK with a "whatever works for now" approach, so I don't
see a problem with documenting and tidying up the current de facto
standard. If Donald comes up with a good proposal, that's great - but
if not, we can always revisit that side of things later.

> Blocker 2 is figuring out whether the new pip <-> build system "hook"
> interface should be command-line based (like the current draft of PEP
> 516) or Python function call based (like the current draft of PEP
> 517). It sounds like currently Donald and I are in favor of the python
> hooks approach, and Robert is indifferent between them and just wants
> to move forward, so we agreed that unless anyone objects we'll drop
> the command-line approach and go ahead with refining the Python
> function call approach. So... if you want to object then speak up now.

Cool. No objections from me.

> Then there are a bunch of details to work out about what hooks to
> provide exactly and what their semantics should be, but hopefully once
> we've settled the two issues above that will be an easier discussion
> to have.
>
> So yeah, basically the next step is for me [2] to write up a spec for
> how sdists currently (really) work.
[...]
> [2] Or if someone else wants to raise their hand and volunteer I
> wouldn't object, obviously I am a bit swamped right now :-)

I don't want to volunteer to take this on completely, as I'll probably
not have the time either, but if I can help in any way (research,
proofreading, writing parts of the document) let me know.

Thanks for the update!

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Brett Cannon]

Thanks for the update! Glad this is still moving forward. I'll continue to
prod the list if things stall again as I want to respond to "Python
packaging is broken" with "actually your knowledge is just outdated, go
read packaging.python.org". :)

On Tue, 3 May 2016 at 11:28 Nathaniel Smith  wrote:

> On Tue, May 3, 2016 at 10:10 AM, Paul Moore  wrote:
> > On 3 May 2016 at 17:47, Donald Stufft  wrote:
> >> It will likely get decided as part of the build system PEP, whenever
> that
> >> gets picked up again.
> >
> > Yes, but on 15th March
> > (https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html)
> > Robert posted
> >
> >> Just to set expectations: this whole process seems stalled to me; I'm
> >> going to context switch and focus on things that can move forward.
> >> Someone please ping me when its relevant to put effort in again :).
> >
> > And I think that's right. The whole build system PEP issue appears
> > stalled from a lack of someone willing (or with the time) to make a
> > call on the approach we take.
>
> No, no, Nick's not the blocker. I'm the blocker! (Sorry)
>
> Donald + Robert + I had a longish conversation about this on IRC a
> month ago [1]. I volunteered to summarize back to the mailing list,
> and then I flaked -- so I guess this is that belated email :-).
>
> Here's the tentative conclusions we came to:
>
> Blocker 1 is figuring out what to do about the sdist format. The
> debate is between keeping something that's basically the current
> format, versus somehow cleaning it up (e.g. Donald's "source wheel"
> ideas). To move forward:
> - I'll write up a PEP that attempts to just document/standardize the
> current de facto sdist format and send it to the mailing list
> (basically: filename convention, PKG-INFO + a list of which fields in
> PKG-INFO pypi actually cares about, presence of setup.py), and adds
> some sort of optional-defaulting-to-1 SDist-Version (I guess in a file
> called SDIST by analogy with WHEEL). And also contains a rationale
> section explaining the trade-offs of standardizing this versus
> creating a new extension.)
> - Donald will make his case for the new extension approach on the mailing
> list
> - We beg Nick to read over both things and make a ruling so we can move on
>
> Blocker 2 is figuring out whether the new pip <-> build system "hook"
> interface should be command-line based (like the current draft of PEP
> 516) or Python function call based (like the current draft of PEP
> 517). It sounds like currently Donald and I are in favor of the python
> hooks approach, and Robert is indifferent between them and just wants
> to move forward, so we agreed that unless anyone objects we'll drop
> the command-line approach and go ahead with refining the Python
> function call approach. So... if you want to object then speak up now.
>
> Then there are a bunch of details to work out about what hooks to
> provide exactly and what their semantics should be, but hopefully once
> we've settled the two issues above that will be an easier discussion
> to have.
>
> So yeah, basically the next step is for me [2] to write up a spec for
> how sdists currently (really) work.
>
> -n
>
> [1] Logs (split across two pages in the log viewer):
>   http://chat-logs.dcpython.org/day/pypa-dev/2016-03-30#18.23.46.njs
>   http://chat-logs.dcpython.org/day/pypa-dev/2016-03-31#00.29.32.lifeless
> [2] Or if someone else wants to raise their hand and volunteer I
> wouldn't object, obviously I am a bit swamped right now :-)
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Daniel Holth]

What happened is that only a half-dozen setuptools experts (I am not one of
those six people) know how to write an extended command or whatever that
would actually be able to take advantage of setup requirements as
implemented by setuptools. Everyone else wants to "import x" at the top of
setup.py and pass arguments to the setup() function. So it would be better
to have the installer make that possible.

On Tue, May 3, 2016 at 2:29 PM Alex Grönholm 
wrote:

> No, setuptools parses the install requirements before acting on setup
> requirements. That is the source of the problem. If setuptools only parsed
> and acted on setup requirements before even parsing install requirements,
> this wouldn't be an issue.
>
>
> 03.05.2016, 21:26, Leonardo Rochael Almeida kirjoitti:
>
>
>
> On 3 May 2016 at 15:07, Alex Grönholm  wrote:
>
>> Having setuptools process the setup requirements before parsing install
>> requirements would be a good step forward. Had that been done before, we
>> could've just added a setup requirement for a newer setuptools to enable
>> PEP 508 conditional requirements.
>>
>
> Setuptools does process setup requirements before install requirements.
> The "chicken and egg" issue with setuptools is that, most of the time,
> setup requires are needed to calculate information that is passed into the
> `setup()` call itself.
>
> For example information on header files coming from the C api of `numpy`
> which is used to build extensions.
>
> This usually means importing code from the packages in "setup requires"
> before setuptools has a chance to actually look at it.
>
> A simple fix would be to allow `setup()` keywords to accept functions as
> well as direct values and only invoke the functions when the values are
> actually needed, but this idea never gained traction.
>
> Of course, even if this was implemented, it wouldn't help directly with
> "setup requiring" a new version of setuptools itself, unless setuptools
> detected this situation and reinvoked setup.py from scratch.
>
> Regards,
>
> Leo
>
>
> 03.05.2016, 21:04, Daniel Holth kirjoitti:
>>
>> We did separate build from install. Now we just want to be able to build
>> without [having to emulate] distutils; just having some dependencies
>> installed before setup.py runs would also be a great boon.
>>
>> I'm reading part of this conversation as "a simple bdist_wheel bug is a
>> reason to do a lot of work standardizing file formats" which I find
>> unfortunate.
>>
>> If he is still up for it let Robert implement his own PEP as the way
>> forward for build system abstraction. The extra PEPs are just delaying
>> action.
>>
>> On Tue, May 3, 2016 at 1:11 PM Paul Moore  wrote:
>>
>>> On 3 May 2016 at 17:47, Donald Stufft  wrote:
>>> > It will likely get decided as part of the build system PEP, whenever
>>> that
>>> > gets picked up again.
>>>
>>> Yes, but on 15th March
>>> (https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html)
>>> Robert posted
>>>
>>> > Just to set expectations: this whole process seems stalled to me; I'm
>>> > going to context switch and focus on things that can move forward.
>>> > Someone please ping me when its relevant to put effort in again :).
>>>
>>> And I think that's right. The whole build system PEP issue appears
>>> stalled from a lack of someone willing (or with the time) to make a
>>> call on the approach we take.
>>>
>>> As far as I'm aware, the decision remains with Nick. With the possible
>>> exception of Donald's proposal (which AFAIK never got formally
>>> published as a PEP) everything that can be said on the other proposals
>>> has been said, and the remaining differences are ones of choice of
>>> approach rather than anything affecting capabilities. (Robert's
>>> message at
>>> https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
>>> summarised the state of the 3 proposals at the time).
>>>
>>> I think this is something that should be resolved - we don't appear to
>>> be gaining anything by waiting, and until we have a decision on the
>>> approach that's being taken, we aren't going to get anyone writing
>>> code for their preferred option.
>>>
>>> Nick - do you have the time to pick this up? Or does it need someone
>>> to step up as BDFL-delegate? Robert, Nathaniel, do you have time to
>>> spend on a final round of discussion on this, on the assumption that
>>> the goal will be a final decision at the end of it? Donald, do you
>>> have the time and interest to complete and publish your proposal?
>>>
>>> Paul
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>
>>
>> ___
>> Distutils-SIG maillist  -  
>> Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>>
>> 

Re: [Distutils] moving things forward

[Alex Grönholm]

No, setuptools parses the install requirements before acting on setup 
requirements. That is the source of the problem. If setuptools only 
parsed and acted on setup requirements before even parsing install 
requirements, this wouldn't be an issue.


03.05.2016, 21:26, Leonardo Rochael Almeida kirjoitti:



On 3 May 2016 at 15:07, Alex Grönholm > wrote:


Having setuptools process the setup requirements before parsing
install requirements would be a good step forward. Had that been
done before, we could've just added a setup requirement for a
newer setuptools to enable PEP 508 conditional requirements.


Setuptools does process setup requirements before install 
requirements. The "chicken and egg" issue with setuptools is that, 
most of the time, setup requires are needed to calculate information 
that is passed into the `setup()` call itself.


For example information on header files coming from the C api of 
`numpy` which is used to build extensions.


This usually means importing code from the packages in "setup 
requires" before setuptools has a chance to actually look at it.


A simple fix would be to allow `setup()` keywords to accept functions 
as well as direct values and only invoke the functions when the values 
are actually needed, but this idea never gained traction.


Of course, even if this was implemented, it wouldn't help directly 
with "setup requiring" a new version of setuptools itself, unless 
setuptools detected this situation and reinvoked setup.py from scratch.


Regards,

Leo


03.05.2016, 21:04, Daniel Holth kirjoitti:

We did separate build from install. Now we just want to be able
to build without [having to emulate] distutils; just having some
dependencies installed before setup.py runs would also be a great
boon.

I'm reading part of this conversation as "a simple bdist_wheel
bug is a reason to do a lot of work standardizing file formats"
which I find unfortunate.

If he is still up for it let Robert implement his own PEP as the
way forward for build system abstraction. The extra PEPs are just
delaying action.

On Tue, May 3, 2016 at 1:11 PM Paul Moore > wrote:

On 3 May 2016 at 17:47, Donald Stufft > wrote:
> It will likely get decided as part of the build system PEP,
whenever that
> gets picked up again.

Yes, but on 15th March
(https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html)
Robert posted

> Just to set expectations: this whole process seems stalled
to me; I'm
> going to context switch and focus on things that can move
forward.
> Someone please ping me when its relevant to put effort in
again :).

And I think that's right. The whole build system PEP issue
appears
stalled from a lack of someone willing (or with the time) to
make a
call on the approach we take.

As far as I'm aware, the decision remains with Nick. With the
possible
exception of Donald's proposal (which AFAIK never got formally
published as a PEP) everything that can be said on the other
proposals
has been said, and the remaining differences are ones of
choice of
approach rather than anything affecting capabilities. (Robert's
message at
https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
summarised the state of the 3 proposals at the time).

I think this is something that should be resolved - we don't
appear to
be gaining anything by waiting, and until we have a decision
on the
approach that's being taken, we aren't going to get anyone
writing
code for their preferred option.

Nick - do you have the time to pick this up? Or does it need
someone
to step up as BDFL-delegate? Robert, Nathaniel, do you have
time to
spend on a final round of discussion on this, on the
assumption that
the goal will be a final decision at the end of it? Donald,
do you
have the time and interest to complete and publish your proposal?

Paul
___
Distutils-SIG maillist  - Distutils-SIG@python.org

https://mail.python.org/mailman/listinfo/distutils-sig



___
Distutils-SIG maillist  -Distutils-SIG@python.org 

https://mail.python.org/mailman/listinfo/distutils-sig



___
Distutils-SIG maillist  - Distutils-SIG@python.org

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nathaniel Smith]

On Tue, May 3, 2016 at 10:10 AM, Paul Moore  wrote:
> On 3 May 2016 at 17:47, Donald Stufft  wrote:
>> It will likely get decided as part of the build system PEP, whenever that
>> gets picked up again.
>
> Yes, but on 15th March
> (https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html)
> Robert posted
>
>> Just to set expectations: this whole process seems stalled to me; I'm
>> going to context switch and focus on things that can move forward.
>> Someone please ping me when its relevant to put effort in again :).
>
> And I think that's right. The whole build system PEP issue appears
> stalled from a lack of someone willing (or with the time) to make a
> call on the approach we take.

No, no, Nick's not the blocker. I'm the blocker! (Sorry)

Donald + Robert + I had a longish conversation about this on IRC a
month ago [1]. I volunteered to summarize back to the mailing list,
and then I flaked -- so I guess this is that belated email :-).

Here's the tentative conclusions we came to:

Blocker 1 is figuring out what to do about the sdist format. The
debate is between keeping something that's basically the current
format, versus somehow cleaning it up (e.g. Donald's "source wheel"
ideas). To move forward:
- I'll write up a PEP that attempts to just document/standardize the
current de facto sdist format and send it to the mailing list
(basically: filename convention, PKG-INFO + a list of which fields in
PKG-INFO pypi actually cares about, presence of setup.py), and adds
some sort of optional-defaulting-to-1 SDist-Version (I guess in a file
called SDIST by analogy with WHEEL). And also contains a rationale
section explaining the trade-offs of standardizing this versus
creating a new extension.)
- Donald will make his case for the new extension approach on the mailing list
- We beg Nick to read over both things and make a ruling so we can move on

Blocker 2 is figuring out whether the new pip <-> build system "hook"
interface should be command-line based (like the current draft of PEP
516) or Python function call based (like the current draft of PEP
517). It sounds like currently Donald and I are in favor of the python
hooks approach, and Robert is indifferent between them and just wants
to move forward, so we agreed that unless anyone objects we'll drop
the command-line approach and go ahead with refining the Python
function call approach. So... if you want to object then speak up now.

Then there are a bunch of details to work out about what hooks to
provide exactly and what their semantics should be, but hopefully once
we've settled the two issues above that will be an easier discussion
to have.

So yeah, basically the next step is for me [2] to write up a spec for
how sdists currently (really) work.

-n

[1] Logs (split across two pages in the log viewer):
  http://chat-logs.dcpython.org/day/pypa-dev/2016-03-30#18.23.46.njs
  http://chat-logs.dcpython.org/day/pypa-dev/2016-03-31#00.29.32.lifeless
[2] Or if someone else wants to raise their hand and volunteer I
wouldn't object, obviously I am a bit swamped right now :-)

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Daniel Holth]

We did separate build from install. Now we just want to be able to build
without [having to emulate] distutils; just having some dependencies
installed before setup.py runs would also be a great boon.

I'm reading part of this conversation as "a simple bdist_wheel bug is a
reason to do a lot of work standardizing file formats" which I find
unfortunate.

If he is still up for it let Robert implement his own PEP as the way
forward for build system abstraction. The extra PEPs are just delaying
action.

On Tue, May 3, 2016 at 1:11 PM Paul Moore  wrote:

> On 3 May 2016 at 17:47, Donald Stufft  wrote:
> > It will likely get decided as part of the build system PEP, whenever that
> > gets picked up again.
>
> Yes, but on 15th March
> (https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html)
> Robert posted
>
> > Just to set expectations: this whole process seems stalled to me; I'm
> > going to context switch and focus on things that can move forward.
> > Someone please ping me when its relevant to put effort in again :).
>
> And I think that's right. The whole build system PEP issue appears
> stalled from a lack of someone willing (or with the time) to make a
> call on the approach we take.
>
> As far as I'm aware, the decision remains with Nick. With the possible
> exception of Donald's proposal (which AFAIK never got formally
> published as a PEP) everything that can be said on the other proposals
> has been said, and the remaining differences are ones of choice of
> approach rather than anything affecting capabilities. (Robert's
> message at
> https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
> summarised the state of the 3 proposals at the time).
>
> I think this is something that should be resolved - we don't appear to
> be gaining anything by waiting, and until we have a decision on the
> approach that's being taken, we aren't going to get anyone writing
> code for their preferred option.
>
> Nick - do you have the time to pick this up? Or does it need someone
> to step up as BDFL-delegate? Robert, Nathaniel, do you have time to
> spend on a final round of discussion on this, on the assumption that
> the goal will be a final decision at the end of it? Donald, do you
> have the time and interest to complete and publish your proposal?
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Alex Grönholm]

Having setuptools process the setup requirements before parsing install 
requirements would be a good step forward. Had that been done before, we 
could've just added a setup requirement for a newer setuptools to enable 
PEP 508 conditional requirements.


03.05.2016, 21:04, Daniel Holth kirjoitti:
We did separate build from install. Now we just want to be able to 
build without [having to emulate] distutils; just having some 
dependencies installed before setup.py runs would also be a great boon.


I'm reading part of this conversation as "a simple bdist_wheel bug is 
a reason to do a lot of work standardizing file formats" which I find 
unfortunate.


If he is still up for it let Robert implement his own PEP as the way 
forward for build system abstraction. The extra PEPs are just delaying 
action.


On Tue, May 3, 2016 at 1:11 PM Paul Moore > wrote:


On 3 May 2016 at 17:47, Donald Stufft > wrote:
> It will likely get decided as part of the build system PEP,
whenever that
> gets picked up again.

Yes, but on 15th March
(https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html)
Robert posted

> Just to set expectations: this whole process seems stalled to
me; I'm
> going to context switch and focus on things that can move forward.
> Someone please ping me when its relevant to put effort in again :).

And I think that's right. The whole build system PEP issue appears
stalled from a lack of someone willing (or with the time) to make a
call on the approach we take.

As far as I'm aware, the decision remains with Nick. With the possible
exception of Donald's proposal (which AFAIK never got formally
published as a PEP) everything that can be said on the other proposals
has been said, and the remaining differences are ones of choice of
approach rather than anything affecting capabilities. (Robert's
message at
https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
summarised the state of the 3 proposals at the time).

I think this is something that should be resolved - we don't appear to
be gaining anything by waiting, and until we have a decision on the
approach that's being taken, we aren't going to get anyone writing
code for their preferred option.

Nick - do you have the time to pick this up? Or does it need someone
to step up as BDFL-delegate? Robert, Nathaniel, do you have time to
spend on a final round of discussion on this, on the assumption that
the goal will be a final decision at the end of it? Donald, do you
have the time and interest to complete and publish your proposal?

Paul
___
Distutils-SIG maillist  - Distutils-SIG@python.org

https://mail.python.org/mailman/listinfo/distutils-sig



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Ian Cordasco]

On Tue, May 3, 2016 at 12:10 PM, Paul Moore  wrote:
> On 3 May 2016 at 17:47, Donald Stufft  wrote:
>> It will likely get decided as part of the build system PEP, whenever that
>> gets picked up again.
>
> Yes, but on 15th March
> (https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html)
> Robert posted
>
>> Just to set expectations: this whole process seems stalled to me; I'm
>> going to context switch and focus on things that can move forward.
>> Someone please ping me when its relevant to put effort in again :).
>
> And I think that's right. The whole build system PEP issue appears
> stalled from a lack of someone willing (or with the time) to make a
> call on the approach we take.
>
> As far as I'm aware, the decision remains with Nick. With the possible
> exception of Donald's proposal (which AFAIK never got formally
> published as a PEP) everything that can be said on the other proposals
> has been said, and the remaining differences are ones of choice of
> approach rather than anything affecting capabilities. (Robert's
> message at 
> https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
> summarised the state of the 3 proposals at the time).
>
> I think this is something that should be resolved - we don't appear to
> be gaining anything by waiting, and until we have a decision on the
> approach that's being taken, we aren't going to get anyone writing
> code for their preferred option.
>
> Nick - do you have the time to pick this up? Or does it need someone
> to step up as BDFL-delegate? Robert, Nathaniel, do you have time to
> spend on a final round of discussion on this, on the assumption that
> the goal will be a final decision at the end of it? Donald, do you
> have the time and interest to complete and publish your proposal?
>
> Paul


I was following that PEP and going to implement it in Twine for the
PyPA. If it would help, I can help Nick with this process. I read both
PEPs a while ago and I think updates have been made so I'd need to
read them again, but I can probably make some time for this.

--
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Paul Moore]

On 3 May 2016 at 17:47, Donald Stufft  wrote:
> It will likely get decided as part of the build system PEP, whenever that
> gets picked up again.

Yes, but on 15th March
(https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html)
Robert posted

> Just to set expectations: this whole process seems stalled to me; I'm
> going to context switch and focus on things that can move forward.
> Someone please ping me when its relevant to put effort in again :).

And I think that's right. The whole build system PEP issue appears
stalled from a lack of someone willing (or with the time) to make a
call on the approach we take.

As far as I'm aware, the decision remains with Nick. With the possible
exception of Donald's proposal (which AFAIK never got formally
published as a PEP) everything that can be said on the other proposals
has been said, and the remaining differences are ones of choice of
approach rather than anything affecting capabilities. (Robert's
message at 
https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
summarised the state of the 3 proposals at the time).

I think this is something that should be resolved - we don't appear to
be gaining anything by waiting, and until we have a decision on the
approach that's being taken, we aren't going to get anyone writing
code for their preferred option.

Nick - do you have the time to pick this up? Or does it need someone
to step up as BDFL-delegate? Robert, Nathaniel, do you have time to
spend on a final round of discussion on this, on the assumption that
the goal will be a final decision at the end of it? Donald, do you
have the time and interest to complete and publish your proposal?

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig