Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Nick Coghlan]

On 11 February 2016 at 17:50, Ralf Gommers  wrote:
> On Wed, Feb 10, 2016 at 2:43 PM, Paul Moore  wrote:
>>
>> On 10 February 2016 at 13:23, Nick Coghlan  wrote:
>> > On 10 February 2016 at 20:53, Paul Moore  wrote:
>> >> We don't have to solve the whole "sdist 2.0" issue right now. Simply
>> >> saying that in order to publish pypa.json-based source trees you need
>> >> to zip up the source directory, name the file "project-version.zip"
>> >> and upload to PyPI, would be sufficient as a short-term answer
>> >> (assuming that this *would* be a viable "source file" that pip could
>> >> use - and I must be clear that I *haven't checked this*!!!)
>
>
> This is exactly what pip itself does right now for "pip install .", so
> clearly it is viable.
>
>> until
>> >> something like Nathaniel's source distribution proposal, or a
>> >> full-blown sdist-2.0 spec, is available. We'd need to support whatever
>> >> stopgap proposal we recommend for backward compatibility in those new
>> >> proposals, but that's a necessary cost of not wanting to delay the
>> >> current PEP on those other ones.
>> >
>> > One of the reasons I went ahead and created the specifications page at
>> > https://packaging.python.org/en/latest/specifications/ was to let us
>> > tweak interoperability requirements as needed, without wasting
>> > people's time with excessive PEP wrangling by requiring a separate PEP
>> > for each interface affected by a proposal.
>> >
>> > In this case, the build system abstraction PEP should propose some
>> > additional text for
>> >
>> > https://packaging.python.org/en/latest/specifications/#source-distribution-format
>> > defining how to publish source archives containing a pypa.json file
>> > and the setup.py shim.
>
>
> The setup.py shim should be optional right? If a package author decides to
> not care about older pip versions, then the shim isn't needed.

Given how long it takes for new versions of pip to filter out through
the ecosystem, the shim's going to be needed for quite a while. Since
we have the power to make things "just work" even for folks on older
pip versions that assume use of the setuptools/distutils CLI, it makes
sense to nudge sdist creation tools in that direction.

The real pay-off here is getting setup.py out of most source repos and
replacing it with a declarative format - keeping it out of sdists is a
non-goal from my perspective.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[David Cournapeau]

On Thu, Feb 11, 2016 at 2:32 PM, Paul Moore  wrote:

> On 11 February 2016 at 14:02, David Cournapeau  wrote:
> >>> On Wed, Feb 10, 2016 at 1:52 PM, Paul Moore 
> wrote:
> 
>  We should probably also check with the flit people that the proposed
>  approach works for them. (Are there any other alternative build
>  systems apart from flit that exist at present?)
> >>>
> >>> I am not working on it ATM, but bento was fairly complete and could
> >>> interoperate w/ pip (a few years ago at least):
> >>> https://cournape.github.io/Bento/
> >>
> >> I plan to test with Bento (I'm still using it almost daily to work on
> >> Scipy) when an implementation is proposed for pip. The interface in the
> PEP
> >> is straightforward though, I don't see any fundamental reason why it
> >> wouldn't work for Bento if it works for flit.
> >
> > It should indeed work, I was just pointing at an alternative build
> system ;)
>
> Yes, I knew there was something other than flit, thanks for the reminder.
>
> > I am a bit worried about making a PEP for interfacing before we have a
> few
> > decent alternative implementations. Having the official interface is not
> > necessary to actually interoperate, even if it is ugly.
>
> Well, a lot of people have complained that setuptools is a problem.
>

You won't hear me defending setuptools/distutils, but the main pain points
of setuptools are not related to interoperability with `setup.py`.


> We've only really seen bento and now flit appear as alternatives, the
> only conclusion we've been able to draw is that the barrier to
> creating alternative build systems is the need to emulate setuptools.
> This PEP (hopefully!) removes that barrier, but I agree we need some
> validation that people who want to create alternative build systems
> (or have done so) can work with the interface in the PEP.
>

If this is indeed the main argument for the PEP, that it is IMO misguided.
Making new buildsystems for python is hard work (once you go beyond the
trivial packages), but interoperability w/ setup.py is not difficult

There is some value to the PEP even if it doesn't enable new build
> tools (we can fix the problem of install_requires triggering
> easy_install) but the key has to be the evolution of (one or more)
> replacements for setuptools.
>
> You suggest getting more alternative build systems before implementing
> the PEP. That would be nice, but how would we get that to happen?
> People have been wanting alternatives to setuptools for years, but
> no-one has delivered anything (as far as I know) except for you and
> the flit guys. So my preference is to implement the PEP (and remove
> the "behave like setuptools" pain point), and then wait to see what
> level of adoption flit/bento achieve with the simpler interface and
> automated use of the declared build tool. At that stage, all of flit,
> bento and setuptools, as well as any newly developed tools, will be
> competing on an equal footing, and either one will emerge as the
> victor, or people will be free to choose their preferred tool without
> concern about whether it is going to work with pip.
>

My main worry is about designing an interface before seeing multiple
implementations. This often causes trouble. If we think the issue is
allowing people to start working on the more "interesting" parts while
keeping compat w/ pip, then I would suggest working on an example
buildsystem with interoperability that people could steal for their own.

I played w/ that idea there: https://github.com/cournape/toydist. The main
point of that project was to progressively bootstrap itself by
re-implementing the basic features required by pip. E.g. right now, only
`python setup.py egg_info` is "distutils clean", but then you just need to
implement develop to get `pip install -e .` working. To get `pip install .`
to work, you only need to add install. Etc.

David


> I'm not sure what we gain by waiting (incremental improvements to the
> spec can me made over time - as long as the basic form of the PEP is
> acceptable to *current* tool developers, I think that's sufficient).
>
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[David Cournapeau]

On Thu, Feb 11, 2016 at 7:43 AM, Ralf Gommers 
wrote:

>
>
> On Wed, Feb 10, 2016 at 3:30 PM, David Cournapeau 
> wrote:
>
>>
>>
>>
>> On Wed, Feb 10, 2016 at 1:52 PM, Paul Moore  wrote:
>>
>
>>> We should probably also check with the flit people that the proposed
>>> approach works for them. (Are there any other alternative build
>>> systems apart from flit that exist at present?)
>>>
>>
>> I am not working on it ATM, but bento was fairly complete and could
>> interoperate w/ pip (a few years ago at least):
>> https://cournape.github.io/Bento/
>>
>
> I plan to test with Bento (I'm still using it almost daily to work on
> Scipy) when an implementation is proposed for pip. The interface in the PEP
> is straightforward though, I don't see any fundamental reason why it
> wouldn't work for Bento if it works for flit.
>

It should indeed work, I was just pointing at an alternative build system ;)

I am a bit worried about making a PEP for interfacing before we have a few
decent alternative implementations. Having the official interface is not
necessary to actually interoperate, even if it is ugly.

David


> Ralf
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Paul Moore]

On 11 February 2016 at 14:02, David Cournapeau  wrote:
>>> On Wed, Feb 10, 2016 at 1:52 PM, Paul Moore  wrote:

 We should probably also check with the flit people that the proposed
 approach works for them. (Are there any other alternative build
 systems apart from flit that exist at present?)
>>>
>>> I am not working on it ATM, but bento was fairly complete and could
>>> interoperate w/ pip (a few years ago at least):
>>> https://cournape.github.io/Bento/
>>
>> I plan to test with Bento (I'm still using it almost daily to work on
>> Scipy) when an implementation is proposed for pip. The interface in the PEP
>> is straightforward though, I don't see any fundamental reason why it
>> wouldn't work for Bento if it works for flit.
>
> It should indeed work, I was just pointing at an alternative build system ;)

Yes, I knew there was something other than flit, thanks for the reminder.

> I am a bit worried about making a PEP for interfacing before we have a few
> decent alternative implementations. Having the official interface is not
> necessary to actually interoperate, even if it is ugly.

Well, a lot of people have complained that setuptools is a problem.
We've only really seen bento and now flit appear as alternatives, the
only conclusion we've been able to draw is that the barrier to
creating alternative build systems is the need to emulate setuptools.
This PEP (hopefully!) removes that barrier, but I agree we need some
validation that people who want to create alternative build systems
(or have done so) can work with the interface in the PEP.

There is some value to the PEP even if it doesn't enable new build
tools (we can fix the problem of install_requires triggering
easy_install) but the key has to be the evolution of (one or more)
replacements for setuptools.

You suggest getting more alternative build systems before implementing
the PEP. That would be nice, but how would we get that to happen?
People have been wanting alternatives to setuptools for years, but
no-one has delivered anything (as far as I know) except for you and
the flit guys. So my preference is to implement the PEP (and remove
the "behave like setuptools" pain point), and then wait to see what
level of adoption flit/bento achieve with the simpler interface and
automated use of the declared build tool. At that stage, all of flit,
bento and setuptools, as well as any newly developed tools, will be
competing on an equal footing, and either one will emerge as the
victor, or people will be free to choose their preferred tool without
concern about whether it is going to work with pip.

I'm not sure what we gain by waiting (incremental improvements to the
spec can me made over time - as long as the basic form of the PEP is
acceptable to *current* tool developers, I think that's sufficient).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Ralf Gommers]

On Thu, Feb 11, 2016 at 11:16 AM, Nick Coghlan  wrote:

> On 11 February 2016 at 17:50, Ralf Gommers  wrote:
> > On Wed, Feb 10, 2016 at 2:43 PM, Paul Moore  wrote:
> >>
> >> On 10 February 2016 at 13:23, Nick Coghlan  wrote:
> >> > On 10 February 2016 at 20:53, Paul Moore  wrote:
> >> >> We don't have to solve the whole "sdist 2.0" issue right now. Simply
> >> >> saying that in order to publish pypa.json-based source trees you need
> >> >> to zip up the source directory, name the file "project-version.zip"
> >> >> and upload to PyPI, would be sufficient as a short-term answer
> >> >> (assuming that this *would* be a viable "source file" that pip could
> >> >> use - and I must be clear that I *haven't checked this*!!!)
> >
> >
> > This is exactly what pip itself does right now for "pip install .", so
> > clearly it is viable.
> >
> >> until
> >> >> something like Nathaniel's source distribution proposal, or a
> >> >> full-blown sdist-2.0 spec, is available. We'd need to support
> whatever
> >> >> stopgap proposal we recommend for backward compatibility in those new
> >> >> proposals, but that's a necessary cost of not wanting to delay the
> >> >> current PEP on those other ones.
> >> >
> >> > One of the reasons I went ahead and created the specifications page at
> >> > https://packaging.python.org/en/latest/specifications/ was to let us
> >> > tweak interoperability requirements as needed, without wasting
> >> > people's time with excessive PEP wrangling by requiring a separate PEP
> >> > for each interface affected by a proposal.
> >> >
> >> > In this case, the build system abstraction PEP should propose some
> >> > additional text for
> >> >
> >> >
> https://packaging.python.org/en/latest/specifications/#source-distribution-format
> >> > defining how to publish source archives containing a pypa.json file
> >> > and the setup.py shim.
> >
> >
> > The setup.py shim should be optional right? If a package author decides
> to
> > not care about older pip versions, then the shim isn't needed.
>
> Given how long it takes for new versions of pip to filter out through
> the ecosystem, the shim's going to be needed for quite a while. Since
> we have the power to make things "just work" even for folks on older
> pip versions that assume use of the setuptools/distutils CLI, it makes
> sense to nudge sdist creation tools in that direction.
>
> The real pay-off here is getting setup.py out of most source repos and
> replacing it with a declarative format - keeping it out of sdists is a
> non-goal from my perspective.
>

I don't feel too strongly about this, but:
- there's also a usability argument for no setup.py in sdists (people will
still unzip an sdist and run python setup.py install on it)
- it makes implementing something like 'flit sdist' more complicated;
without the shim it can be as simply as just zipping the non-hidden files
in the source tree.
- if flit decides not to implement sdist (good chance of that), then people
*will* still need to add the shim to their own source repos to comply to
this 'spec'.

Ralf
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Paul Moore]

On 11 February 2016 at 15:01, David Cournapeau  wrote:
>> We've only really seen bento and now flit appear as alternatives, the
>> only conclusion we've been able to draw is that the barrier to
>> creating alternative build systems is the need to emulate setuptools.
>> This PEP (hopefully!) removes that barrier, but I agree we need some
>> validation that people who want to create alternative build systems
>> (or have done so) can work with the interface in the PEP.
>
> If this is indeed the main argument for the PEP, that it is IMO misguided.
> Making new buildsystems for python is hard work (once you go beyond the
> trivial packages), but interoperability w/ setup.py is not difficult

I'll let the PEP authors comment on that one - what I said above is
really only my interpretation and I could easily be wrong.

> My main worry is about designing an interface before seeing multiple
> implementations. This often causes trouble. If we think the issue is
> allowing people to start working on the more "interesting" parts while
> keeping compat w/ pip, then I would suggest working on an example
> buildsystem with interoperability that people could steal for their own.

I understand your point, but I'm not clear what's stopping people
doing this right now? For all the complaints about how pip is tightly
coupled to setuptools, very few people seem to be providing
alternatives.

The pip developers in particular aren't looking to develop a build
system, so any work in that area will have to come from the community.
We hear the message that people wish they could use other build
systems - maybe that's actually just people saying "I wish someone
else would write a new build system" rather than anyone actually
wanting to write one themselves?

> I played w/ that idea there: https://github.com/cournape/toydist. The main
> point of that project was to progressively bootstrap itself by
> re-implementing the basic features required by pip. E.g. right now, only
> `python setup.py egg_info` is "distutils clean", but then you just need to
> implement develop to get `pip install -e .` working. To get `pip install .`
> to work, you only need to add install. Etc.

So someone could integrate flit with pip using this approach, is that
what you're saying? Or someone could write a new build system and use
that code as a basis for the "pip integration layer"?

If so, then yes, I'd like to see someone do that and provide feedback
to this PEP based on their experience. But I'm not sure I want the PEP
to stall (along with all work on decoupling pip from setuptools) until
someone gets round to it.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Paul Moore]

On 10 February 2016 at 13:43, Paul Moore  wrote:
>> In this case, the build system abstraction PEP should propose some
>> additional text for
>> https://packaging.python.org/en/latest/specifications/#source-distribution-format
>> defining how to publish source archives containing a pypa.json file
>> and the setup.py shim. At that point, it will effectively become the
>> spec for sdist 1.0, since that's never previously been officially
>> defined.
>>
>> The key difference from setuptools is that the setup.py shim will be a
>> standard one that flit (and other source archive creation tools) can
>> inject when building the sdist, rather than needing to be a custom
>> file stored in each project's source repository.
>
> Neat! That sounds like a sensible approach, and if the build system
> abstraction PEP adds this, then that addresses my remaining
> objections.

We should probably also check with the flit people that the proposed
approach works for them. (Are there any other alternative build
systems apart from flit that exist at present?)

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[David Cournapeau]

On Wed, Feb 10, 2016 at 1:52 PM, Paul Moore  wrote:

> On 10 February 2016 at 13:43, Paul Moore  wrote:
> >> In this case, the build system abstraction PEP should propose some
> >> additional text for
> >>
> https://packaging.python.org/en/latest/specifications/#source-distribution-format
> >> defining how to publish source archives containing a pypa.json file
> >> and the setup.py shim. At that point, it will effectively become the
> >> spec for sdist 1.0, since that's never previously been officially
> >> defined.
> >>
> >> The key difference from setuptools is that the setup.py shim will be a
> >> standard one that flit (and other source archive creation tools) can
> >> inject when building the sdist, rather than needing to be a custom
> >> file stored in each project's source repository.
> >
> > Neat! That sounds like a sensible approach, and if the build system
> > abstraction PEP adds this, then that addresses my remaining
> > objections.
>
> We should probably also check with the flit people that the proposed
> approach works for them. (Are there any other alternative build
> systems apart from flit that exist at present?)
>

I am not working on it ATM, but bento was fairly complete and could
interoperate w/ pip (a few years ago at least):
https://cournape.github.io/Bento/


>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Paul Moore]

On 10 February 2016 at 13:23, Nick Coghlan  wrote:
> On 10 February 2016 at 20:53, Paul Moore  wrote:
>> We don't have to solve the whole "sdist 2.0" issue right now. Simply
>> saying that in order to publish pypa.json-based source trees you need
>> to zip up the source directory, name the file "project-version.zip"
>> and upload to PyPI, would be sufficient as a short-term answer
>> (assuming that this *would* be a viable "source file" that pip could
>> use - and I must be clear that I *haven't checked this*!!!) until
>> something like Nathaniel's source distribution proposal, or a
>> full-blown sdist-2.0 spec, is available. We'd need to support whatever
>> stopgap proposal we recommend for backward compatibility in those new
>> proposals, but that's a necessary cost of not wanting to delay the
>> current PEP on those other ones.
>
> One of the reasons I went ahead and created the specifications page at
> https://packaging.python.org/en/latest/specifications/ was to let us
> tweak interoperability requirements as needed, without wasting
> people's time with excessive PEP wrangling by requiring a separate PEP
> for each interface affected by a proposal.
>
> In this case, the build system abstraction PEP should propose some
> additional text for
> https://packaging.python.org/en/latest/specifications/#source-distribution-format
> defining how to publish source archives containing a pypa.json file
> and the setup.py shim. At that point, it will effectively become the
> spec for sdist 1.0, since that's never previously been officially
> defined.
>
> The key difference from setuptools is that the setup.py shim will be a
> standard one that flit (and other source archive creation tools) can
> inject when building the sdist, rather than needing to be a custom
> file stored in each project's source repository.

Neat! That sounds like a sensible approach, and if the build system
abstraction PEP adds this, then that addresses my remaining
objections.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Ralf Gommers]

On Wed, Feb 10, 2016 at 3:30 PM, David Cournapeau 
wrote:

>
>
>
> On Wed, Feb 10, 2016 at 1:52 PM, Paul Moore  wrote:
>

>> We should probably also check with the flit people that the proposed
>> approach works for them. (Are there any other alternative build
>> systems apart from flit that exist at present?)
>>
>
> I am not working on it ATM, but bento was fairly complete and could
> interoperate w/ pip (a few years ago at least):
> https://cournape.github.io/Bento/
>

I plan to test with Bento (I'm still using it almost daily to work on
Scipy) when an implementation is proposed for pip. The interface in the PEP
is straightforward though, I don't see any fundamental reason why it
wouldn't work for Bento if it works for flit.

Ralf
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Ralf Gommers]

On Wed, Feb 10, 2016 at 2:43 PM, Paul Moore  wrote:

> On 10 February 2016 at 13:23, Nick Coghlan  wrote:
> > On 10 February 2016 at 20:53, Paul Moore  wrote:
> >> We don't have to solve the whole "sdist 2.0" issue right now. Simply
> >> saying that in order to publish pypa.json-based source trees you need
> >> to zip up the source directory, name the file "project-version.zip"
> >> and upload to PyPI, would be sufficient as a short-term answer
> >> (assuming that this *would* be a viable "source file" that pip could
> >> use - and I must be clear that I *haven't checked this*!!!)
>

This is exactly what pip itself does right now for "pip install .", so
clearly it is viable.

until
> >> something like Nathaniel's source distribution proposal, or a
> >> full-blown sdist-2.0 spec, is available. We'd need to support whatever
> >> stopgap proposal we recommend for backward compatibility in those new
> >> proposals, but that's a necessary cost of not wanting to delay the
> >> current PEP on those other ones.
> >
> > One of the reasons I went ahead and created the specifications page at
> > https://packaging.python.org/en/latest/specifications/ was to let us
> > tweak interoperability requirements as needed, without wasting
> > people's time with excessive PEP wrangling by requiring a separate PEP
> > for each interface affected by a proposal.
> >
> > In this case, the build system abstraction PEP should propose some
> > additional text for
> >
> https://packaging.python.org/en/latest/specifications/#source-distribution-format
> > defining how to publish source archives containing a pypa.json file
> > and the setup.py shim.
>

The setup.py shim should be optional right? If a package author decides to
not care about older pip versions, then the shim isn't needed.

Ralf
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Paul Moore]

On 10 February 2016 at 01:19, Robert Collins  wrote:
> On 10 February 2016 at 13:09, Paul Moore  wrote:
>> [I need to read and digest the rest of this, but it's late here, so
>> that will be tomorrow]
>
> OK, cool.

Right, I've been thinking about this a bit more, and I've re-read the
PEP. I wasn't at all clear in my concern - in my own mind, and as a
consequence in how I explained the issue. My apologies.

I don't have any problem with the basic proposed interface, and I see
what you mean that it doesn't need to require a "sdist" command. I was
wrong on that. But the proposal is written in terms of how pip "works
from a Python source tree". The example used throughout the PEP is a
local source directory. While that's an easy and understandable
example to use, it glosses over the detail of how pip will *get* such
a source tree.

What the PEP omits, it seems to me, is an explanation of how pip will
get a "source tree" in the form defined by the PEP. In essence, there
is no transition plan in the PEP. At present, pip sets up a source
tree by downloading a sdist from PyPI and unpacking it. The PEP
explains that sources without a pypa.json should be treated as current
setuptools-style sdists for backward compatibility, but offers no
guidance on how projects (or build tools, on behalf of projects that
use them) can transition to a pypa.json-based approach.

That's really what I'm trying to get at with my focus on "source
distributions". As a project author, how do I switch to this new
format? What do I publish? Publishing binaries in the form of wheels
is irrelevant - projects using flit or other build tools can do that
right now. This PEP is entirely about enabling them to publish
*source* using new build tools, and yet it doesn't cover the
publishing side at all.

So while the PEP is fine, it's incomplete from the user's point of
view, and my concern is that if we finalise it in its current form,
people will have to roll their own publishing solutions, and we'll
then be playing "catch up" trying to define a standard that takes into
account the approaches people have developed in the meantime. And as a
user, I'll then be left with projects I can't easily build wheels for.
It's all very well to say "the project should build wheels", and I
wish they would, but not all do or will - I care because one of my
ongoing side projects is an automated wheel build farm (for simple
cases only, don't anyone get their hopes up! ;-)) and being able to
build wheels in a standard way is key to that.

We don't have to solve the whole "sdist 2.0" issue right now. Simply
saying that in order to publish pypa.json-based source trees you need
to zip up the source directory, name the file "project-version.zip"
and upload to PyPI, would be sufficient as a short-term answer
(assuming that this *would* be a viable "source file" that pip could
use - and I must be clear that I *haven't checked this*!!!) until
something like Nathaniel's source distribution proposal, or a
full-blown sdist-2.0 spec, is available. We'd need to support whatever
stopgap proposal we recommend for backward compatibility in those new
proposals, but that's a necessary cost of not wanting to delay the
current PEP on those other ones.

Hope this is clearer.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Nick Coghlan]

On 10 February 2016 at 20:53, Paul Moore  wrote:
> We don't have to solve the whole "sdist 2.0" issue right now. Simply
> saying that in order to publish pypa.json-based source trees you need
> to zip up the source directory, name the file "project-version.zip"
> and upload to PyPI, would be sufficient as a short-term answer
> (assuming that this *would* be a viable "source file" that pip could
> use - and I must be clear that I *haven't checked this*!!!) until
> something like Nathaniel's source distribution proposal, or a
> full-blown sdist-2.0 spec, is available. We'd need to support whatever
> stopgap proposal we recommend for backward compatibility in those new
> proposals, but that's a necessary cost of not wanting to delay the
> current PEP on those other ones.

One of the reasons I went ahead and created the specifications page at
https://packaging.python.org/en/latest/specifications/ was to let us
tweak interoperability requirements as needed, without wasting
people's time with excessive PEP wrangling by requiring a separate PEP
for each interface affected by a proposal.

In this case, the build system abstraction PEP should propose some
additional text for
https://packaging.python.org/en/latest/specifications/#source-distribution-format
defining how to publish source archives containing a pypa.json file
and the setup.py shim. At that point, it will effectively become the
spec for sdist 1.0, since that's never previously been officially
defined.

The key difference from setuptools is that the setup.py shim will be a
standard one that flit (and other source archive creation tools) can
inject when building the sdist, rather than needing to be a custom
file stored in each project's source repository.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 27 January 2016 at 22:24, Paul Moore  wrote:
> On 27 January 2016 at 05:39, Robert Collins  wrote:
>>> - You and Paul were strongly in favor of splitting up "working directories"
>>> and "sdists"; Robert and I didn't like the idea. The argument in favor is
>>> that we have to support working directory -> sdist and sdist -> wheel, so
>>> adding working directory -> wheel as an additional pathway creates larger
>>> test matrices and more opportunities for things to go wrong; the argument
>>> against is that this requires a new sdist format definition (currently it's
>>> basically "a zipped up working dir"), and it breaks like incremental
>>> rebuilds, which are a critical feature for developer adoption.
>>
>> That was something that occurred in the rabbit-hole discussion of your
>> first draft; The PR Donald is proposing to accept is I think the
>> fourth major iteration? Two from you, two from me - building on
>> feedback each time. While I don't think Donalds position here has
>> changed, the draft in question doesn't alter any semantics of anything
>> - it captures a subset of the semantics such that the existing
>> behaviour of pip can be modelled on the resulting interface. In
>> particular, for this question, it retains 'develop', which is the
>> primary developer mode in use today: it has no implications for or
>> against incremental builds - it doesn't alter pips copy-before-build
>> behaviour, which pip already has for non-develop installs. E.g. its
>> moot; we can debate that further - and i'm sure we shall - but it has
>> no impact on the interface.
>
> I'm still not clear on what the position is here. The PEP as I read it
> still offers no means to ask the build system to produce a sdist, so
> I'm concerned as to what will happen here.


> In the absence of a "pip sdist" command I can see that you're saying
> that we can implement pip's functionality without caring about this.
> But fundamentally, people upload sdists and wheels to PyPI. A build
> system that doesn't support sdists (which this PEP allows) will
> therefore have to publish wheel-only builds to PyPI, and I am very
> much against the idea of PyPI hosting "binary only" distributions.

As you know - 
https://mail.python.org/pipermail/distutils-sig/2015-March/026013.html
- flit /already/ does this. If we want to require sdists as a thing,
PyPI needs to change to do that. I don't care whether it does or
doesn't myself: folk that want to not ship sources will always find a
way. I predict empty sdists, for instance.

> If project foo has no sdist, "pip wheel foo" for a platform where
> there's no compatible wheel available will fail, as there's no way for
> pip to locate the source.

Thats true; but pip isn't a build system itself - and so far pip has
no role in the uploading of wheels to PyPI. Nor does - last I checked
- twine build things for you. The pattern is that developers prepare
there artifacts, then ask twine to upload them.

> So can we please revisit the question of whether build systems will be
> permitted to refuse to generate sdists? Note that I don't care whether
> we formally define a new sdist format, or go with something adhoc, or
> whatever. All I care about is that the PEP states that build systems
> must support generating a file that can be uploaded to PyPI and used
> by pip to build a wheel as described above (not "git clone this
> location and do 'pip wheel .'"). I think that making build systems
> expose how you make such a file by requiring a "sdist" subcommand is a
> reasonable approach, but I'm OK with naming the subcommand
> differently.

I truely have no opinion here. I don't think it harms the abstract
build system to have it, but I do know that Donald very much does not
want pip to have to know or care about building sdists per se, and he
may see this as an attractive nuisance.

Who / what tool do we expect to use the sdist command in the abstract interface?

> Thanks,
> Paul
>
> PS I agree with Nathaniel, insofar as there were definitely a number
> of unresolved points remaining after the various public discussions,
> and I don't think it's reasonable to say that any proposal is "ready
> for implementation" without a public confirmation that those issues
> have been resolved.

I suspect it had been quescient too long and folk had paged out the
state. Certainly I don't think Donald intended to bypass any community
debate, and neither did I.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Paul Moore]

On 9 February 2016 at 21:38, Robert Collins  wrote:
>> In the absence of a "pip sdist" command I can see that you're saying
>> that we can implement pip's functionality without caring about this.
>> But fundamentally, people upload sdists and wheels to PyPI. A build
>> system that doesn't support sdists (which this PEP allows) will
>> therefore have to publish wheel-only builds to PyPI, and I am very
>> much against the idea of PyPI hosting "binary only" distributions.
>
> As you know - 
> https://mail.python.org/pipermail/distutils-sig/2015-March/026013.html
> - flit /already/ does this. If we want to require sdists as a thing,
> PyPI needs to change to do that. I don't care whether it does or
> doesn't myself: folk that want to not ship sources will always find a
> way. I predict empty sdists, for instance.

I know flit does this. We're obviously never going to be able to
*stop* people generating wheels any way they like. But there is value
in having a standard invocation that builds the wheel - anyone who has
ever used make is aware that a standardised build command has value.

At the moment, that standard is "pip wheel foo". Or maybe it's
"setup.py bdist_wheel", but I prefer the layer of build system
abstraction offered by pip.

>
>> If project foo has no sdist, "pip wheel foo" for a platform where
>> there's no compatible wheel available will fail, as there's no way for
>> pip to locate the source.
>
> Thats true; but pip isn't a build system itself - and so far pip has
> no role in the uploading of wheels to PyPI. Nor does - last I checked
> - twine build things for you. The pattern is that developers prepare
> there artifacts, then ask twine to upload them.

As someone who frequently builds wheels for multiple projects, where
those projects don't have complex build requirements but the projects
themselves do not upload wheels, I find significant value in being
able to just do "pip wheel foo". Maybe it's not pip's core
functionality, maybe in the long term it'll be replaced by a separate
utility, but the key point is that one command, given a project name,
locates it on PyPI, downloads it and builds the source into a wheel.

Having that capability is a *huge* benefit (I remember the days before
distutils and PyPI, when every single package build was a process of
locating files on obscure websites, and struggling with custom build
processes - I'm admittedly over-sensitive about the risks of going
back to that).

I don't care in the slightest how build systems implement
" sdist". They can write a custom "setup.py" script that
downloads the real sources from github and bulids them using flit, for
all I care.

What I do care about is ensuring that if a projects hasn't published a
wheel for some particular environment, and the project is *not*
actively trying to avoid publishing sources, then we provide them with
a means to publish those sources in such a way that "pip wheel
theproject" (or whatever command replaces it as the canonical "build a
wheel" command) can automate the process of generating a wheel for the
user's platform.

Of course projects who want to make their sources unavailable can, and
will. I'm not trying to force them not to.
Of course users can manually locate the sources and build from them. I
believe that is a major step back in usability.
Of course projects can upload wheels. They will never cover all platforms.

Longer term, do we want a standard "source distribution" format? I
thought we did (Nathaniel's proposals generated a lot of debate, but
no-one was saying "let's just leave projects to do whatever they want,
there's no point standardising"). If we do, then a standard process
for building a wheel from that format seems like an obvious thing to
expect. On the way to that goal, let's not start off by going
backwards and abandoning the standard we currently have in "pip wheel
project-with-source-on-pypi".

>> So can we please revisit the question of whether build systems will be
>> permitted to refuse to generate sdists? Note that I don't care whether
>> we formally define a new sdist format, or go with something adhoc, or
>> whatever. All I care about is that the PEP states that build systems
>> must support generating a file that can be uploaded to PyPI and used
>> by pip to build a wheel as described above (not "git clone this
>> location and do 'pip wheel .'"). I think that making build systems
>> expose how you make such a file by requiring a "sdist" subcommand is a
>> reasonable approach, but I'm OK with naming the subcommand
>> differently.
>
> I truely have no opinion here. I don't think it harms the abstract
> build system to have it, but I do know that Donald very much does not
> want pip to have to know or care about building sdists per se, and he
> may see this as an attractive nuisance.

As I said above, all I really care about is that "pip wheel foo"
either continue to work for any project with sources published on
PyPI, or that we have a 

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 10 February 2016 at 11:40, Paul Moore  wrote:
> On 9 February 2016 at 21:38, Robert Collins  wrote:
>>> In the absence of a "pip sdist" command I can see that you're saying
>>> that we can implement pip's functionality without caring about this.
>>> But fundamentally, people upload sdists and wheels to PyPI. A build
>>> system that doesn't support sdists (which this PEP allows) will
>>> therefore have to publish wheel-only builds to PyPI, and I am very
>>> much against the idea of PyPI hosting "binary only" distributions.
>>
>> As you know - 
>> https://mail.python.org/pipermail/distutils-sig/2015-March/026013.html
>> - flit /already/ does this. If we want to require sdists as a thing,
>> PyPI needs to change to do that. I don't care whether it does or
>> doesn't myself: folk that want to not ship sources will always find a
>> way. I predict empty sdists, for instance.
>
> I know flit does this. We're obviously never going to be able to
> *stop* people generating wheels any way they like. But there is value
> in having a standard invocation that builds the wheel - anyone who has
> ever used make is aware that a standardised build command has value.
>
> At the moment, that standard is "pip wheel foo". Or maybe it's
> "setup.py bdist_wheel", but I prefer the layer of build system
> abstraction offered by pip.

But 'pip wheel foo' already reuses existing wheels: things like flit
that upload universal wheels work with 'pip wheel foo' just fine (as
long as --no-binary has not been passed). Things that upload an sdist
will also still work, unchanged, with this proposal.

>>> If project foo has no sdist, "pip wheel foo" for a platform where
>>> there's no compatible wheel available will fail, as there's no way for
>>> pip to locate the source.
>>
>> Thats true; but pip isn't a build system itself - and so far pip has
>> no role in the uploading of wheels to PyPI. Nor does - last I checked
>> - twine build things for you. The pattern is that developers prepare
>> there artifacts, then ask twine to upload them.
>
> As someone who frequently builds wheels for multiple projects, where
> those projects don't have complex build requirements but the projects
> themselves do not upload wheels, I find significant value in being
> able to just do "pip wheel foo". Maybe it's not pip's core
> functionality, maybe in the long term it'll be replaced by a separate
> utility, but the key point is that one command, given a project name,
> locates it on PyPI, downloads it and builds the source into a wheel.
>
> Having that capability is a *huge* benefit (I remember the days before
> distutils and PyPI, when every single package build was a process of
> locating files on obscure websites, and struggling with custom build
> processes - I'm admittedly over-sensitive about the risks of going
> back to that).
>
> I don't care in the slightest how build systems implement
> " sdist". They can write a custom "setup.py" script that
> downloads the real sources from github and bulids them using flit, for
> all I care.

I'm struggling to connect 'pip wheel foo' to 'they can do an sdist'.
'pip wheel foo' does not invoke sdist. There is a bug proposing that
it should ( https://github.com/pypa/pip/issues/2195 and
https://github.com/pypa/pip/pull/3219 ) - but that proposal is
contentious - see the ongoing related debate about editable mode /
incremental builds and also the discussion on PR 3219.

> What I do care about is ensuring that if a projects hasn't published a
> wheel for some particular environment, and the project is *not*
> actively trying to avoid publishing sources, then we provide them with
> a means to publish those sources in such a way that "pip wheel
> theproject" (or whatever command replaces it as the canonical "build a
> wheel" command) can automate the process of generating a wheel for the
> user's platform.

Sure, they should have that. I don't understand why pip's
*abstraction* over project *building* should know about that. Issue
2195 aside, which seems to be entirely about avoiding one particular
developer failure mode, and other than that makes no sense to me.


> Of course projects who want to make their sources unavailable can, and
> will. I'm not trying to force them not to.
> Of course users can manually locate the sources and build from them. I
> believe that is a major step back in usability.
> Of course projects can upload wheels. They will never cover all platforms.
>
> Longer term, do we want a standard "source distribution" format? I
> thought we did (Nathaniel's proposals generated a lot of debate, but
> no-one was saying "let's just leave projects to do whatever they want,
> there's no point standardising"). If we do, then a standard process
> for building a wheel from that format seems like an obvious thing to
> expect. On the way to that goal, let's not start off by going
> backwards and abandoning the standard we currently have in "pip wheel
> 

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Nathaniel Smith]

On Tue, Feb 9, 2016 at 2:40 PM, Paul Moore  wrote:
> On 9 February 2016 at 21:38, Robert Collins  wrote:
>>> In the absence of a "pip sdist" command I can see that you're saying
>>> that we can implement pip's functionality without caring about this.
>>> But fundamentally, people upload sdists and wheels to PyPI. A build
>>> system that doesn't support sdists (which this PEP allows) will
>>> therefore have to publish wheel-only builds to PyPI, and I am very
>>> much against the idea of PyPI hosting "binary only" distributions.
>>
>> As you know - 
>> https://mail.python.org/pipermail/distutils-sig/2015-March/026013.html
>> - flit /already/ does this. If we want to require sdists as a thing,
>> PyPI needs to change to do that. I don't care whether it does or
>> doesn't myself: folk that want to not ship sources will always find a
>> way. I predict empty sdists, for instance.
>
> I know flit does this. We're obviously never going to be able to
> *stop* people generating wheels any way they like. But there is value
> in having a standard invocation that builds the wheel - anyone who has
> ever used make is aware that a standardised build command has value.
>
> At the moment, that standard is "pip wheel foo". Or maybe it's
> "setup.py bdist_wheel", but I prefer the layer of build system
> abstraction offered by pip.

Everyone agrees that we absolutely need to support "pip wheel
packagename" and and "pip wheel sdist.zip" and "pip wheel some-path/".
The (potential) disagreement is about whether the most primitive input
to "pip wheel" is an sdist or a directory tree.

I.e., option A is that "pip wheel some-path/" is implemented as "pip
sdist some-path/ && pip wheel some-path-sdist.zip", and option B is
"pip wheel sdist.zip" implemented as "unzip sdist.zip && pip wheel
sdist-tree/". Robert's draft PEP doesn't explicitly come down on
either side, but in practice I think that at least to start with it
would require the "option B" approach, since it doesn't provide any
"pip sdist" command.

>>> If project foo has no sdist, "pip wheel foo" for a platform where
>>> there's no compatible wheel available will fail, as there's no way for
>>> pip to locate the source.
>>
>> Thats true; but pip isn't a build system itself - and so far pip has
>> no role in the uploading of wheels to PyPI. Nor does - last I checked
>> - twine build things for you. The pattern is that developers prepare
>> there artifacts, then ask twine to upload them.
>
> As someone who frequently builds wheels for multiple projects, where
> those projects don't have complex build requirements but the projects
> themselves do not upload wheels, I find significant value in being
> able to just do "pip wheel foo". Maybe it's not pip's core
> functionality, maybe in the long term it'll be replaced by a separate
> utility, but the key point is that one command, given a project name,
> locates it on PyPI, downloads it and builds the source into a wheel.
>
> Having that capability is a *huge* benefit (I remember the days before
> distutils and PyPI, when every single package build was a process of
> locating files on obscure websites, and struggling with custom build
> processes - I'm admittedly over-sensitive about the risks of going
> back to that).
>
> I don't care in the slightest how build systems implement
> " sdist". They can write a custom "setup.py" script that
> downloads the real sources from github and bulids them using flit, for
> all I care.
>
> What I do care about is ensuring that if a projects hasn't published a
> wheel for some particular environment, and the project is *not*
> actively trying to avoid publishing sources, then we provide them with
> a means to publish those sources in such a way that "pip wheel
> theproject" (or whatever command replaces it as the canonical "build a
> wheel" command) can automate the process of generating a wheel for the
> user's platform.

Both of the options under discussion give us all of these properties.
You can absolutely implement a "pip wheel" command that handles all
these cases without needing any "pip sdist" command.

> Of course projects who want to make their sources unavailable can, and
> will. I'm not trying to force them not to.
> Of course users can manually locate the sources and build from them. I
> believe that is a major step back in usability.
> Of course projects can upload wheels. They will never cover all platforms.
>
> Longer term, do we want a standard "source distribution" format? I
> thought we did (Nathaniel's proposals generated a lot of debate, but
> no-one was saying "let's just leave projects to do whatever they want,
> there's no point standardising"). If we do, then a standard process
> for building a wheel from that format seems like an obvious thing to
> expect. On the way to that goal, let's not start off by going
> backwards and abandoning the standard we currently have in "pip wheel
> project-with-source-on-pypi".
>
>>> So 

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 10 February 2016 at 13:09, Paul Moore  wrote:
> [I need to read and digest the rest of this, but it's late here, so
> that will be tomorrow]

OK, cool.

> On 9 February 2016 at 23:19, Robert Collins  wrote:
 Who / what tool do we expect to use the sdist command in the abstract 
 interface?

>>> I do not accept any proposal that removes "pip wheel " without
>>> providing *any* replacement.
>>
>> But the current proposal *DOES NOT REMOVE IT*.
>
> By  I had in mind "project name", implying "download from
> PyPI". And by "remove" i meant "open up the possibility of people
> using tools that don't support easy creation of source artifacts that
> can be uploaded to PyPI, resulting in pip not being able to find
> something to download".

Sure - but Nathaniel and I both seem to think that the PEP doesn't
make it any easier to do that - and in fact should allow flit to start
uploading source artifacts (by allowing pip to consume it's sdists),
optionally with a setuptools_shim style setup.py.

>> We're clearly miscommunicating about something :).
>
> Yes, and that's probably my fault. I need to go back and reread the
> PEP and the thread.
>
> But as I said in my response to Nathaniel, it may be that all that is
> needed is some context in the PEP explaining how we require[1] people
> to upload source to PyPI in the new world where we support build
> systems which don't have a "sdist" command like setuptools does.
>
> Paul
>
> [1] I say "require" in the sense of "you have to follow these rules if
> pip is to be able to use your source", not "you must upload source" -
> although I hope that the number of people actually preferring to *not*
> include source in their PyPI uploads is vanishingly small...

So, I'm not against us making a statement like that, but I don't think
it belongs in this PEP - it should be in the main PyPI docs/rules,
surely?

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Paul Moore]

On 27 January 2016 at 05:39, Robert Collins  wrote:
>> - You and Paul were strongly in favor of splitting up "working directories"
>> and "sdists"; Robert and I didn't like the idea. The argument in favor is
>> that we have to support working directory -> sdist and sdist -> wheel, so
>> adding working directory -> wheel as an additional pathway creates larger
>> test matrices and more opportunities for things to go wrong; the argument
>> against is that this requires a new sdist format definition (currently it's
>> basically "a zipped up working dir"), and it breaks like incremental
>> rebuilds, which are a critical feature for developer adoption.
>
> That was something that occurred in the rabbit-hole discussion of your
> first draft; The PR Donald is proposing to accept is I think the
> fourth major iteration? Two from you, two from me - building on
> feedback each time. While I don't think Donalds position here has
> changed, the draft in question doesn't alter any semantics of anything
> - it captures a subset of the semantics such that the existing
> behaviour of pip can be modelled on the resulting interface. In
> particular, for this question, it retains 'develop', which is the
> primary developer mode in use today: it has no implications for or
> against incremental builds - it doesn't alter pips copy-before-build
> behaviour, which pip already has for non-develop installs. E.g. its
> moot; we can debate that further - and i'm sure we shall - but it has
> no impact on the interface.

I'm still not clear on what the position is here. The PEP as I read it
still offers no means to ask the build system to produce a sdist, so
I'm concerned as to what will happen here.

In the absence of a "pip sdist" command I can see that you're saying
that we can implement pip's functionality without caring about this.
But fundamentally, people upload sdists and wheels to PyPI. A build
system that doesn't support sdists (which this PEP allows) will
therefore have to publish wheel-only builds to PyPI, and I am very
much against the idea of PyPI hosting "binary only" distributions.

If project foo has no sdist, "pip wheel foo" for a platform where
there's no compatible wheel available will fail, as there's no way for
pip to locate the source.

So can we please revisit the question of whether build systems will be
permitted to refuse to generate sdists? Note that I don't care whether
we formally define a new sdist format, or go with something adhoc, or
whatever. All I care about is that the PEP states that build systems
must support generating a file that can be uploaded to PyPI and used
by pip to build a wheel as described above (not "git clone this
location and do 'pip wheel .'"). I think that making build systems
expose how you make such a file by requiring a "sdist" subcommand is a
reasonable approach, but I'm OK with naming the subcommand
differently.

Thanks,
Paul

PS I agree with Nathaniel, insofar as there were definitely a number
of unresolved points remaining after the various public discussions,
and I don't think it's reasonable to say that any proposal is "ready
for implementation" without a public confirmation that those issues
have been resolved.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Nick Coghlan]

On 27 January 2016 at 17:53, Robert Collins  wrote:
> Yeah - note that my PR punts on this - it says PEP 427 METADATA file,
> but as we know the contents of that are a snapshot of earlier drafts
> of PEP-426, so its really not all that well defined, and before folk
> can iterate on that further we'll need to do something about that -
> either by a rewrite of PEP 426 into a much more pithy thing now,
> without the enterprise feel and the moon-shot aspects, or by issuing a
> PEP that specifies exactly what is in METADATA.

As of about a minute ago, there's a slightly better reference for that
now: https://packaging.python.org/en/latest/specifications/#core-metadata

It gives us something to point to that says that in practice, the
current metadata spec is the file format and field definitions in PEP
345 + Provides-Extra + PEP 440 + PEP 508.

My thread about it didn't garner a lot of attention, so I figured it
made more sense to just go ahead and flip the switch, rather than
trying to encourage more interest.

More generally, it means any "the PEPs don't always match reality"
clarifications can now just go into the PyPUG specs section until such
time as we get around to updating the relevant spec.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 27 January 2016 at 16:37, Nathaniel Smith  wrote:
> On Tue, Jan 26, 2016 at 5:08 PM, Donald Stufft  wrote:
>> As many know there has been an effort to get a generalized interface that
>> a build system can implement so that we can break the hard dependency on
>> setuptools that Python packaging currently has. After a lot of deliberation
>> and threads back and forth (as well as video meetings!) I think that the
>> version of the PEP that is currently on the PR in
>> https://github.com/pypa/interoperability-peps/pull/54 looks like it’s
>> generally the right thing to move forward with. I made a few little comments
>> but overall I think it’s there and we’re ready to move forward on trying to
>> get a reference implementation done that can validate the decisions made in
>> that PEP (and then, hopefully finalize the PEP and merge those
>> implementations).
>>
>> So many thanks to everyone involved in hammering this out thus far :) I’m
>> nervous but excited about the possibility of making setuptools just another
>> build system.
>
> AFAICR at the time the set of public threads petered out (I guess because
> Robert and I both got busy with other things), there were several
> fundamental disagreements still unresolved:

There were two pep drafts in flight, but I would hesitate to call it a
fundamental disagreement :).

> - You and Paul were strongly in favor of splitting up "working directories"
> and "sdists"; Robert and I didn't like the idea. The argument in favor is
> that we have to support working directory -> sdist and sdist -> wheel, so
> adding working directory -> wheel as an additional pathway creates larger
> test matrices and more opportunities for things to go wrong; the argument
> against is that this requires a new sdist format definition (currently it's
> basically "a zipped up working dir"), and it breaks like incremental
> rebuilds, which are a critical feature for developer adoption.

That was something that occurred in the rabbit-hole discussion of your
first draft; The PR Donald is proposing to accept is I think the
fourth major iteration? Two from you, two from me - building on
feedback each time. While I don't think Donalds position here has
changed, the draft in question doesn't alter any semantics of anything
- it captures a subset of the semantics such that the existing
behaviour of pip can be modelled on the resulting interface. In
particular, for this question, it retains 'develop', which is the
primary developer mode in use today: it has no implications for or
against incremental builds - it doesn't alter pips copy-before-build
behaviour, which pip already has for non-develop installs. E.g. its
moot; we can debate that further - and i'm sure we shall - but it has
no impact on the interface.

> - Robert was in favor of a command-line based interface; I favored a hook
> based interface. I won't try to summarize the disagreement here b/c I'm
> unable to articulate any advantages of the command-line based interface :-)

it has the fairly large advantage that the developers of pip are in
favour of it - as the folk that will have to deal with and debug what
goes on, thats more than sufficient for me.

> - Robert was in favor of preserving the current 'setup.py develop' semantics
> as-is, and in particular keeping the current behavior where it's the build
> system's job to pick which directory the .egg-link file should be installed
> into and then do the actual work; I favored an interface where the build
> system says "here's the path to a directory with the package and metadata",
> and then the installation tool (pip) picks the location and installs the
> .egg-link file (or whatever) pointing to that directory. The argument in
> favor of the current semantics is that hey, we need to get this show on the
> road, and this will require less code added to pip. The information is
> against is that the current design gets the separation of concerns wrong,
> impossible to implement without relying on unstandardized features like
> egg-link, and also I don't think it actually works right currently (e.g. if
> the user supplies --user or --root to the pip install -e command, then what
> happens? But I could be wrong -- haven't had a chance to fully reverse
> engineer pip/setuptools interaction here).

That feels like an exaggeration of my position. I'd love to evolve
those semantics. But since such an evolution is a hard API break
across everything, I don't want to make fixing the much more
significant problems of setup_requires and having to use setuptools
conditional on that other thing: its a massive adoption hurdle I
suspect.

> My personal feeling is that on the first two points my position is
> objectively correct ;-), while the third is more confusing and more of a
> judgement call.
>
> I haven't looked at the current version of the PEP, so the above may be out
> of date, but I think it's accurate WRT what's been discussed on the mailing
> list. Have 

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 27 January 2016 at 20:30, Nathaniel Smith  wrote:

>>> - Robert was in favor of a command-line based interface; I favored a hook
>>> based interface. I won't try to summarize the disagreement here b/c I'm
>>> unable to articulate any advantages of the command-line based interface :-)
>>
>> it has the fairly large advantage that the developers of pip are in
>> favour of it - as the folk that will have to deal with and debug what
>> goes on, thats more than sufficient for me.
>
> Can you link to where they said this?

Donald specifically:

https://mail.python.org/pipermail/distutils-sig/2015-November/027727.html
https://mail.python.org/pipermail/distutils-sig/2015-November/027722.html


>>> - Robert was in favor of preserving the current 'setup.py develop' semantics
>>> as-is, and in particular keeping the current behavior where it's the build
>>> system's job to pick which directory the .egg-link file should be installed
>>> into and then do the actual work; I favored an interface where the build
>>> system says "here's the path to a directory with the package and metadata",
>>> and then the installation tool (pip) picks the location and installs the
>>> .egg-link file (or whatever) pointing to that directory. The argument in
>>> favor of the current semantics is that hey, we need to get this show on the
>>> road, and this will require less code added to pip. The information is
>>> against is that the current design gets the separation of concerns wrong,
>>> impossible to implement without relying on unstandardized features like
>>> egg-link, and also I don't think it actually works right currently (e.g. if
>>> the user supplies --user or --root to the pip install -e command, then what
>>> happens? But I could be wrong -- haven't had a chance to fully reverse
>>> engineer pip/setuptools interaction here).
>>
>> That feels like an exaggeration of my position. I'd love to evolve
>> those semantics. But since such an evolution is a hard API break
>> across everything, I don't want to make fixing the much more
>> significant problems of setup_requires and having to use setuptools
>> conditional on that other thing: its a massive adoption hurdle I
>> suspect.
>
> Are you concerned that this would make it impossible to do a simple
> shim to run old setuptools projects under the new interface? I think
> this part is fine: basically for the hook that I propose, a
> PEPXX->setup.py shim would implement it by running
>
>   setup.py build_ext -i && setup.py egg_info


> which sets up the current directory to be (a) importable, and (b) have
> metadata, and then pip would be responsible for installing a .egg-link
> pointing to this directory. IIUC this is exactly what 'develop'
> currently does, except that it creates the .egg-link itself.

And it needs to do command line entry point processing, not that that
is super hard.

My concern is mainly about unknown unknowns. We've been blocked on
lots of incremental improvement because we've been taking really big
bites. I firmly believe we should take small bites: do one thing, do
it well. E.g. PEP 508, which really did just one thing - fixed the
python 2.7.10 < 2.7.9 issue - but even it took weeks of drafting,
testing, and the patches for it are still wending through review into
setuptools, pip and packaging, let along projects further out on the
periphery like pbr.


The one thing I want to accomplish here is enabling alternative build
systems, because it seem(ed) like there was lots of pent up demand for
that; I specifically don't want to reengineer what pip needs from the
build systems, nor what they need to deliver *in this iteration*: the
tech debt we have here is enormous: there is /no/ standard that we can
point at and say 'implement that', because its literally 'whatever pip
calls setup.py with', and that is subject to change whenever someone
puts a clever patch into pip.

My view is that we won't get it right, no matter what. We have so many
things we're still working on getting agreement on in the long view -
like the copy-before-build question; should progress bars be a thing?
What about incremental builds? What about Wheel format 2? ... once we
have a standard, that standard can be evolved both compatibly, and
incompatibly, with a reference baseline we can compare too, rather
than a defacto thing. We probably want to batch up the changes, and
not dribble them out once every few months - but I definitely see more
iterations happening.

> Doh, and of course that was the other substantive disagreement:
> whether to mandate egg_info.

Yeah - note that my PR punts on this - it says PEP 427 METADATA file,
but as we know the contents of that are a snapshot of earlier drafts
of PEP-426, so its really not all that well defined, and before folk
can iterate on that further we'll need to do something about that -
either by a rewrite of PEP 426 into a much more pithy thing now,
without the enterprise feel and the moon-shot aspects, or by issuing a
PEP 

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Nathaniel Smith]

On Tue, Jan 26, 2016 at 9:39 PM, Robert Collins
 wrote:
> On 27 January 2016 at 16:37, Nathaniel Smith  wrote:
>> On Tue, Jan 26, 2016 at 5:08 PM, Donald Stufft  wrote:
>>> As many know there has been an effort to get a generalized interface that
>>> a build system can implement so that we can break the hard dependency on
>>> setuptools that Python packaging currently has. After a lot of deliberation
>>> and threads back and forth (as well as video meetings!) I think that the
>>> version of the PEP that is currently on the PR in
>>> https://github.com/pypa/interoperability-peps/pull/54 looks like it’s
>>> generally the right thing to move forward with. I made a few little comments
>>> but overall I think it’s there and we’re ready to move forward on trying to
>>> get a reference implementation done that can validate the decisions made in
>>> that PEP (and then, hopefully finalize the PEP and merge those
>>> implementations).
>>>
>>> So many thanks to everyone involved in hammering this out thus far :) I’m
>>> nervous but excited about the possibility of making setuptools just another
>>> build system.
>>
>> AFAICR at the time the set of public threads petered out (I guess because
>> Robert and I both got busy with other things), there were several
>> fundamental disagreements still unresolved:
>
> There were two pep drafts in flight, but I would hesitate to call it a
> fundamental disagreement :).

Yeah, that may be overstating it :-). I guess I mean something like
"basic design issues that people were still arguing about". (As
compared to stuff like "what file format should we use for the config
file" where it's worth talking about but fundamentally it just doesn't
really matter.)

>> - You and Paul were strongly in favor of splitting up "working directories"
>> and "sdists"; Robert and I didn't like the idea. The argument in favor is
>> that we have to support working directory -> sdist and sdist -> wheel, so
>> adding working directory -> wheel as an additional pathway creates larger
>> test matrices and more opportunities for things to go wrong; the argument
>> against is that this requires a new sdist format definition (currently it's
>> basically "a zipped up working dir"), and it breaks like incremental
>> rebuilds, which are a critical feature for developer adoption.
>
> That was something that occurred in the rabbit-hole discussion of your
> first draft; The PR Donald is proposing to accept is I think the
> fourth major iteration? Two from you, two from me - building on
> feedback each time. While I don't think Donalds position here has
> changed, the draft in question doesn't alter any semantics of anything
> - it captures a subset of the semantics such that the existing
> behaviour of pip can be modelled on the resulting interface. In
> particular, for this question, it retains 'develop', which is the
> primary developer mode in use today: it has no implications for or
> against incremental builds - it doesn't alter pips copy-before-build
> behaviour, which pip already has for non-develop installs. E.g. its
> moot; we can debate that further - and i'm sure we shall - but it has
> no impact on the interface.

I'm pretty sure there was another big subthread about this just before
things petered out, but yeah, fair enough. And as far as I'm concerned
there's only one possible answer here since if pip insists on
copy-before-build then all the projects I'm involved in will simply
refuse to use pip for building, so I certainly don't object to
anything in your draft in this regard :-).

>> - Robert was in favor of a command-line based interface; I favored a hook
>> based interface. I won't try to summarize the disagreement here b/c I'm
>> unable to articulate any advantages of the command-line based interface :-)
>
> it has the fairly large advantage that the developers of pip are in
> favour of it - as the folk that will have to deal with and debug what
> goes on, thats more than sufficient for me.

Can you link to where they said this?

>> - Robert was in favor of preserving the current 'setup.py develop' semantics
>> as-is, and in particular keeping the current behavior where it's the build
>> system's job to pick which directory the .egg-link file should be installed
>> into and then do the actual work; I favored an interface where the build
>> system says "here's the path to a directory with the package and metadata",
>> and then the installation tool (pip) picks the location and installs the
>> .egg-link file (or whatever) pointing to that directory. The argument in
>> favor of the current semantics is that hey, we need to get this show on the
>> road, and this will require less code added to pip. The information is
>> against is that the current design gets the separation of concerns wrong,
>> impossible to implement without relying on unstandardized features like
>> egg-link, and also I don't think it actually works right currently 

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

Cool - thank you.

Right now it looks like there is one edit (add PATH as a variable folk
can depend on being set) and one undecided (metadata file format).

I've pushed a shim that permits having a setup.py for non-setuptools
projects using solely the abstract interface to
https://pypi.python.org/pypi/setuptools_shim and
https://github.com/rbtcollins/setuptools_shim

It depends on https://github.com/pypa/packaging/pull/50 - the test
suite is hardcoded to locate that in ../packaging at the moment - but
I believe that that will be merging soon, so it should be generally
usable from there on out.

I'd love it if someone put together a flit implementation of the
abstract build system and checked that the shim does let pip work :).
Theory and practice and all that.

-Rob


On 27 January 2016 at 14:08, Donald Stufft  wrote:
> As many know there has been an effort to get a generalized interface that a 
> build system can implement so that we can break the hard dependency on 
> setuptools that Python packaging currently has. After a lot of deliberation 
> and threads back and forth (as well as video meetings!) I think that the 
> version of the PEP that is currently on the PR in 
> https://github.com/pypa/interoperability-peps/pull/54 looks like it’s 
> generally the right thing to move forward with. I made a few little comments 
> but overall I think it’s there and we’re ready to move forward on trying to 
> get a reference implementation done that can validate the decisions made in 
> that PEP (and then, hopefully finalize the PEP and merge those 
> implementations).
>
> So many thanks to everyone involved in hammering this out thus far :) I’m 
> nervous but excited about the possibility of making setuptools just another 
> build system.
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Nathaniel Smith]

On Tue, Jan 26, 2016 at 5:08 PM, Donald Stufft  wrote:
> As many know there has been an effort to get a generalized interface that
a build system can implement so that we can break the hard dependency on
setuptools that Python packaging currently has. After a lot of deliberation
and threads back and forth (as well as video meetings!) I think that the
version of the PEP that is currently on the PR in
https://github.com/pypa/interoperability-peps/pull/54 looks like it’s
generally the right thing to move forward with. I made a few little
comments but overall I think it’s there and we’re ready to move forward on
trying to get a reference implementation done that can validate the
decisions made in that PEP (and then, hopefully finalize the PEP and merge
those implementations).
>
> So many thanks to everyone involved in hammering this out thus far :) I’m
nervous but excited about the possibility of making setuptools just another
build system.

AFAICR at the time the set of public threads petered out (I guess because
Robert and I both got busy with other things), there were several
fundamental disagreements still unresolved:

- You and Paul were strongly in favor of splitting up "working directories"
and "sdists"; Robert and I didn't like the idea. The argument in favor is
that we have to support working directory -> sdist and sdist -> wheel, so
adding working directory -> wheel as an additional pathway creates larger
test matrices and more opportunities for things to go wrong; the argument
against is that this requires a new sdist format definition (currently it's
basically "a zipped up working dir"), and it breaks like incremental
rebuilds, which are a critical feature for developer adoption.

- Robert was in favor of a command-line based interface; I favored a hook
based interface. I won't try to summarize the disagreement here b/c I'm
unable to articulate any advantages of the command-line based interface :-)

- Robert was in favor of preserving the current 'setup.py develop'
semantics as-is, and in particular keeping the current behavior where it's
the build system's job to pick which directory the .egg-link file should be
installed into and then do the actual work; I favored an interface where
the build system says "here's the path to a directory with the package and
metadata", and then the installation tool (pip) picks the location and
installs the .egg-link file (or whatever) pointing to that directory. The
argument in favor of the current semantics is that hey, we need to get this
show on the road, and this will require less code added to pip. The
information is against is that the current design gets the separation of
concerns wrong, impossible to implement without relying on unstandardized
features like egg-link, and also I don't think it actually works right
currently (e.g. if the user supplies --user or --root to the pip install -e
command, then what happens? But I could be wrong -- haven't had a chance to
fully reverse engineer pip/setuptools interaction here).

My personal feeling is that on the first two points my position is
objectively correct ;-), while the third is more confusing and more of a
judgement call.

I haven't looked at the current version of the PEP, so the above may be out
of date, but I think it's accurate WRT what's been discussed on the mailing
list. Have there been some private video meetings where all these issues
got hashed out somehow, or...?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig