Re: [Distutils] Fwd: Re: PEP 517 again

[Nick Coghlan]

On 25 August 2017 at 17:47, Paul Moore  wrote:
> If anyone feels that it's *not* reasonable for users to expect "build
> a wheel" and "build a sdist and build a wheel from that" to produce
> the same results for a given source tree (assuming the backend
> supports both building wheels and sdists from the given source tree)
> then they need to speak up with a *specific* example of a use case, so
> we can include it in the PEP as a motivating example, and use it in
> future debates.

Just the NotImplementedError case: running build_wheel against a
source tree is permitted to succeed in cases where build_sdist raises
NotImplementedError, and in these cases, the result may *not* match
what you would have gotten if the necessary inputs and tools for
creating the sdist were available.

> At the moment too much discussion is getting bogged
> down with comments like "I think someone said that XXX needs this..."
>
> I'm *not* interested in trying to tie down "the correct behaviour" any
> more tightly than this.

Cool, we're on the same page then - the specific thing that was
bothering me was the request to start articulating "permitted side
effects" at the PEP level, and I don't think the PEP is the right
place for that.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Thomas Kluyver]

On Fri, Aug 25, 2017, at 11:56 AM, Thomas Kluyver wrote:
> Great, I think we're on the same page. I'm going to make a PR against
> the PEP while I still remember what page it is. ;-)

https://github.com/python/peps/pull/364
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Thomas Kluyver]

On Fri, Aug 25, 2017, at 11:45 AM, Paul Moore wrote:
> If it can't build a sdist then there's no opportunity for frontends to
> fall back to sdist->wheel anyway, so that's not an issue[1].
> 
> It's only if sdist->wheel and direct wheel are both possible that
> frontends have a choice, and I feel that we should mandate that
> backends ensure that it doesn't matter which route the frontend
> chooses.

Great, I think we're on the same page. I'm going to make a PR against
the PEP while I still remember what page it is. ;-)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Paul Moore]

On 25 August 2017 at 11:16, Thomas Kluyver  wrote:
>> I repeat my earlier comment here - do we have any actual cases where
>> we expect in-place wheel builds to *not* enforce the equivalence?
>
> Flit cannot strictly enforce it if it can't build an sdist: with the way
> I've chosen to do sdists, it's not possible to know what would be in an
> sdist in the case where we can't build one.

If it can't build a sdist then there's no opportunity for frontends to
fall back to sdist->wheel anyway, so that's not an issue[1].

It's only if sdist->wheel and direct wheel are both possible that
frontends have a choice, and I feel that we should mandate that
backends ensure that it doesn't matter which route the frontend
chooses.

Paul

[1] Excepting the case where the user modifies things to allow sdists
(e.g. by installing git) and expects to get the same result, which is
a quality of implementation point and I'm sure flit does its best to
get that right, so I'm not worried about that case.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Thomas Kluyver]

Thanks Paul,

On Fri, Aug 25, 2017, at 10:45 AM, Paul Moore wrote:
> I'd *also* expect that the wheel built (and hence what gets installed)
> would be the same as what you'd get if the user added write
> permissions and then ran the command. That's part of what I mean by
> direct wheel and sdist->wheel being equivalent. I'd completely support
> any user who complained that getting different results in those 2
> cases is a bug in the backend. But as Nick pointed out, this is
> probably more detail than we need in the PEP.

Agreed.

> > Frontends which wish to ensure this consistency may call build_sdist
> > (see below) first, and then call build_wheel in the unpacked sdist. But
> > if build_sdist fails with NotImplementedError, they will fall back to
> > calling build_wheel in the source directory, and rely on the backend to
> > produce an equivalent wheel.
> 
> This seems to contradict the previous point - I'd rephrase it as
> "Frontends which want to protect against backends violating this
> expectation may call build_sdist..." and at that point it starts to
> seem like excessive paranoia on the part of frontends.

Your wording is fine by me - I was struggling to find the right phrasing
there.

> I repeat my earlier comment here - do we have any actual cases where
> we expect in-place wheel builds to *not* enforce the equivalence? 

Flit cannot strictly enforce it if it can't build an sdist: with the way
I've chosen to do sdists, it's not possible to know what would be in an
sdist in the case where we can't build one.

The 'on-label' use case for flit is packages with no built or generated
parts. All of the modules and data files inside the package should be
checked in to version control, which means they would be included in an
sdist. I don't think I can technically stop people from running a
separate step to generate files before building a wheel, but I'd
consider that creative misuse of flit.

(There is a discussion issue open - #119 - about adding support for
build steps in flit. If we decide to do that, the situation will change,
but that's some way off even if we do do it)

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Paul Moore]

On 25 August 2017 at 09:38, Thomas Kluyver  wrote:
> On Fri, Aug 25, 2017, at 08:47 AM, Paul Moore wrote:
>> I've never cared about whether the backend modifies the source tree,
>> only that the direct-wheel / sdist->wheel equivalence holds.
>
> If a user types 'pip install .' in a directory where they don't have
> write permissions, what should happen?
>
> 1. The backend is capable of building without writing in that directory.
> 2. The frontend (pip) notices and pre-emptively copies to a writable tmp
> directory.
> 3. The user accepts that the operation may fail, depending on the
> backend.
>
> I'd go for option 1 or 3.

I'd go for option (1). I don't think we want to get into (2), with the
frontend trying to spot possible issues. I think (3) is always a
possibility, but users have a right to expect that backends will do
whatever they can to avoid it.

I'd *also* expect that the wheel built (and hence what gets installed)
would be the same as what you'd get if the user added write
permissions and then ran the command. That's part of what I mean by
direct wheel and sdist->wheel being equivalent. I'd completely support
any user who complained that getting different results in those 2
cases is a bug in the backend. But as Nick pointed out, this is
probably more detail than we need in the PEP.

> Here's a proposed update to the build_wheel section of the PEP:

(Apologies if I miss a change here, I don't have the original copy
right now to compare against).

> build_wheel(wheel_directory, config_settings=None,
> metadata_directory=None):
> ...
>
> Must build a .whl file, and place it in the specified wheel_directory.
> It must return the basename (not the full path) of the .whl file it
> creates, as a unicode string.
>
> If the build frontend has previously called
> prepare_metadata_for_build_wheel and depends on the wheel resulting from
> this call to have metadata matching this earlier call, then it should
> provide the path to the created .dist-info directory as the
> metadata_directory argument. If this argument is provided, then
> build_wheel MUST produce a wheel with identical metadata. The directory
> passed in by the build frontend MUST be identical to the directory
> created by prepare_metadata_for_build_wheel, including any unrecognized
> files it created.
>
> Backends which do not provide the prepare_metadata_for_build_wheel hook
> may either silently ignore the metadata_directory parameter to
> build_wheel, or else raise an exception when it is set to anything other
> than None.
>
> Wheels built by this hook should, as far as possible, be equivalent to
> first building an sdist and then building a wheel from that sdist.

This is the point that I care about.

> Frontends which wish to ensure this consistency may call build_sdist
> (see below) first, and then call build_wheel in the unpacked sdist. But
> if build_sdist fails with NotImplementedError, they will fall back to
> calling build_wheel in the source directory, and rely on the backend to
> produce an equivalent wheel.

This seems to contradict the previous point - I'd rephrase it as
"Frontends which want to protect against backends violating this
expectation may call build_sdist..." and at that point it starts to
seem like excessive paranoia on the part of frontends.

I repeat my earlier comment here - do we have any actual cases where
we expect in-place wheel builds to *not* enforce the equivalence? This
was originally a huge debate issue because of all the comments about
development builds and incremental compilation, and caching. Are the
people who insisted that backends be allowed to carry build artifacts
between compiles now accepting that they have to ensure build
consistency? Or have they just dropped out of the discussion? I'm
pretty sure Nathaniel was one of those people, but I'm equally sure
he's fed up of debating theoretical possibilities. I suspect he still
considers my assumptions fairly naive, though, as I have no experience
of the sort of complexities involved in building the scientific
stack...

Either way, pip *can* go sdist->wheel, but as things are worded here,
I don't see a real reason to bother. I'd be more inclined to push any
bug reports about getting different results to the backend, quoting
the PEP's requirement for build equivalence, and only implement
sdist->wheel in pip if the backends pushed back (at which point, we'd
need to reopen the debate on what the statement in the PEP about
equivalence actually meant... )

> The source directory may be read-only. Backends should therefore be
> prepared to build without creating or modifying any files in the source
> directory, but they may opt not to handle this case, in which case
> failures will be visible to the user. Frontends are not responsible for
> any special handling of read-only source directories.

This is a useful addition, particularly as it's a general point, not
tied to any specific backend implementation.

Personally, I'd 

Re: [Distutils] Fwd: Re: PEP 517 again

[Thomas Kluyver]

On Fri, Aug 25, 2017, at 08:47 AM, Paul Moore wrote:
> I've never cared about whether the backend modifies the source tree,
> only that the direct-wheel / sdist->wheel equivalence holds.

If a user types 'pip install .' in a directory where they don't have
write permissions, what should happen?

1. The backend is capable of building without writing in that directory.
2. The frontend (pip) notices and pre-emptively copies to a writable tmp
directory.
3. The user accepts that the operation may fail, depending on the
backend.

I'd go for option 1 or 3.


Here's a proposed update to the build_wheel section of the PEP:

build_wheel(wheel_directory, config_settings=None,
metadata_directory=None):
...

Must build a .whl file, and place it in the specified wheel_directory.
It must return the basename (not the full path) of the .whl file it
creates, as a unicode string.

If the build frontend has previously called
prepare_metadata_for_build_wheel and depends on the wheel resulting from
this call to have metadata matching this earlier call, then it should
provide the path to the created .dist-info directory as the
metadata_directory argument. If this argument is provided, then
build_wheel MUST produce a wheel with identical metadata. The directory
passed in by the build frontend MUST be identical to the directory
created by prepare_metadata_for_build_wheel, including any unrecognized
files it created.

Backends which do not provide the prepare_metadata_for_build_wheel hook
may either silently ignore the metadata_directory parameter to
build_wheel, or else raise an exception when it is set to anything other
than None.

Wheels built by this hook should, as far as possible, be equivalent to
first building an sdist and then building a wheel from that sdist.
Frontends which wish to ensure this consistency may call build_sdist
(see below) first, and then call build_wheel in the unpacked sdist. But
if build_sdist fails with NotImplementedError, they will fall back to
calling build_wheel in the source directory, and rely on the backend to
produce an equivalent wheel.

The source directory may be read-only. Backends should therefore be
prepared to build without creating or modifying any files in the source
directory, but they may opt not to handle this case, in which case
failures will be visible to the user. Frontends are not responsible for
any special handling of read-only source directories.

The backend may store intermediate artifacts in cache locations or
temporary directories. The presence or absence of any caches should not
make a material difference to the final result of the build.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Paul Moore]

On 25 August 2017 at 06:54, Nick Coghlan  wrote:
> On 25 August 2017 at 14:33, xoviat  wrote:
>> Just this morning, Paul said the following:
>>
>> That step's the problem. If the frontend does that it can potentially
>> be copying a lot of unneeded stuff (VCS history, for example). We
>> tried that with pip and it was a major issue. That problem is the
>> *whole point* of all the discussions about the various proposals that
>> ended up with build_tree.
>>
>> I took that to mean that we were trusting the backend to do the right thing.
>> And most people agreed with that. I don't personally care but there does
>> seem to be some miscommunication here.
>
> Folks are wanting the "right thing" to be defined in the spec, but
> that's the wrong place for it, since "the right thing" will be use
> case dependent. So I'm now saying "stop trying to define what the
> right thing is (beyond producing standards compliant artifacts), and
> instead just define a usable thing, run with it, and let consensus on
> 'the right thing' emerge over time".

I do believe that the PEP is the right place to make the point that
*users* (not just front ends) have the right to expect that the direct
wheel and sdist->wheel routes will produce the same results.
Originally, I would have expected this to be self-evident, but with
all the debates around incremental builds and similar, I now feel that
it's necessary to clarify that this is the expected behaviour. Too
many tools (not just pip, tox is another one) depend on the
equivalence for us to leave it open to interpretation.

If anyone feels that it's *not* reasonable for users to expect "build
a wheel" and "build a sdist and build a wheel from that" to produce
the same results for a given source tree (assuming the backend
supports both building wheels and sdists from the given source tree)
then they need to speak up with a *specific* example of a use case, so
we can include it in the PEP as a motivating example, and use it in
future debates. At the moment too much discussion is getting bogged
down with comments like "I think someone said that XXX needs this..."

I'm *not* interested in trying to tie down "the correct behaviour" any
more tightly than this.

> Donald and Paul originally didn't like that notion, but then Paul
> realised that pip had never actually tried implementing the "always
> build an sdist first" approach. That means pip's notion of "the right
> thing" can be adequately captured as "source tree -> sdist -> wheel"
> producing the same results as "source tree -> wheel", and pip can
> strongly encourage that behaviour in backend implementations by always
> trying build_sdist first, and only falling back to calling build_wheel
> directly if build_sdist raises NotImplementedError.

Precisely. Does any tool or participant in this discussion *not* see
this as the minimal definition of correct behaviour, though?

> Particular *frontends* (including pip) may further refine their
> definition of the "right thing" to categorically disallow source tree
> changes, but that's something they can do over time based on the bug
> reports they actually receive, rather than needing to be mandated a
> priori in the interface specification.

I've never cared about whether the backend modifies the source tree,
only that the direct-wheel / sdist->wheel equivalence holds.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

Okay, but I would like to hear Paul's response to that first.

On Aug 25, 2017 12:54 AM, "Nick Coghlan"  wrote:

> On 25 August 2017 at 14:33, xoviat  wrote:
> > Just this morning, Paul said the following:
> >
> > That step's the problem. If the frontend does that it can potentially
> > be copying a lot of unneeded stuff (VCS history, for example). We
> > tried that with pip and it was a major issue. That problem is the
> > *whole point* of all the discussions about the various proposals that
> > ended up with build_tree.
> >
> > I took that to mean that we were trusting the backend to do the right
> thing.
> > And most people agreed with that. I don't personally care but there does
> > seem to be some miscommunication here.
>
> Folks are wanting the "right thing" to be defined in the spec, but
> that's the wrong place for it, since "the right thing" will be use
> case dependent. So I'm now saying "stop trying to define what the
> right thing is (beyond producing standards compliant artifacts), and
> instead just define a usable thing, run with it, and let consensus on
> 'the right thing' emerge over time".
>
> Donald and Paul originally didn't like that notion, but then Paul
> realised that pip had never actually tried implementing the "always
> build an sdist first" approach. That means pip's notion of "the right
> thing" can be adequately captured as "source tree -> sdist -> wheel"
> producing the same results as "source tree -> wheel", and pip can
> strongly encourage that behaviour in backend implementations by always
> trying build_sdist first, and only falling back to calling build_wheel
> directly if build_sdist raises NotImplementedError.
>
> Particular *frontends* (including pip) may further refine their
> definition of the "right thing" to categorically disallow source tree
> changes, but that's something they can do over time based on the bug
> reports they actually receive, rather than needing to be mandated a
> priori in the interface specification.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Nick Coghlan]

On 25 August 2017 at 14:33, xoviat  wrote:
> Just this morning, Paul said the following:
>
> That step's the problem. If the frontend does that it can potentially
> be copying a lot of unneeded stuff (VCS history, for example). We
> tried that with pip and it was a major issue. That problem is the
> *whole point* of all the discussions about the various proposals that
> ended up with build_tree.
>
> I took that to mean that we were trusting the backend to do the right thing.
> And most people agreed with that. I don't personally care but there does
> seem to be some miscommunication here.

Folks are wanting the "right thing" to be defined in the spec, but
that's the wrong place for it, since "the right thing" will be use
case dependent. So I'm now saying "stop trying to define what the
right thing is (beyond producing standards compliant artifacts), and
instead just define a usable thing, run with it, and let consensus on
'the right thing' emerge over time".

Donald and Paul originally didn't like that notion, but then Paul
realised that pip had never actually tried implementing the "always
build an sdist first" approach. That means pip's notion of "the right
thing" can be adequately captured as "source tree -> sdist -> wheel"
producing the same results as "source tree -> wheel", and pip can
strongly encourage that behaviour in backend implementations by always
trying build_sdist first, and only falling back to calling build_wheel
directly if build_sdist raises NotImplementedError.

Particular *frontends* (including pip) may further refine their
definition of the "right thing" to categorically disallow source tree
changes, but that's something they can do over time based on the bug
reports they actually receive, rather than needing to be mandated a
priori in the interface specification.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

And one more thing: it would probably be prescient to require a
get_backend_info hook that returns the name and url of the backend so that
pip can direct people where to file bugs when there are problems.

On Aug 24, 2017 11:52 PM, "xoviat"  wrote:

> Actually after reading the email that Nathaniel referenced, I disagree
> with Nick's position on in-tree builds. It's absolutely necessary because
> the tree may be read only and it's enforceable by pip through a simple
> recursive list.
>
> On Aug 24, 2017 11:33 PM, "xoviat"  wrote:
>
>> Just this morning, Paul said the following:
>>
>> That step's the problem. If the frontend does that it can potentially
>> be copying a lot of unneeded stuff (VCS history, for example). We
>> tried that with pip and it was a major issue. That problem is the
>> *whole point* of all the discussions about the various proposals that
>> ended up with build_tree.
>>
>> I took that to mean that we were trusting the backend to do the right
>> thing. And most people agreed with that. I don't personally care but there
>> does seem to be some miscommunication here.
>>
>> On Aug 24, 2017 11:24 PM, "Nathaniel Smith"  wrote:
>>
>>> On Thu, Aug 24, 2017 at 9:17 PM, xoviat  wrote:
>>> > > I'm *not* OK with banning in-tree builds in the spec, since that's
>>> > > both unnecessary and unenforceable
>>> >
>>> > Well then either we can trust the backend or we cannot. If we can,
>>> then this
>>> > is both necessary and enforceable. If not, then we're back to pip
>>> copying
>>> > files. You can't make and argument that it's okay to trust build_sdist
>>> but
>>> > not build_wheel.
>>>
>>> I think at this point everyone has made their peace with the pip
>>> developers' decision that they want to keep copying files -- at least
>>> for now -- and that's just how it's going to be. This email has a more
>>> detailed discussion of the options, their "threat model", and the
>>> tradeoffs:
>>>
>>> https://mail.python.org/pipermail/distutils-sig/2017-July/03
>>> 1020.html
>>>
>>> I can see an argument for adding language saying that build_sdist
>>> SHOULD avoid modifying the source tree if possible, and MAY write
>>> scratch files to the sdist_directory.
>>>
>>> -n
>>>
>>> --
>>> Nathaniel J. Smith -- https://vorpus.org
>>>
>>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

Actually after reading the email that Nathaniel referenced, I disagree with
Nick's position on in-tree builds. It's absolutely necessary because the
tree may be read only and it's enforceable by pip through a simple
recursive list.

On Aug 24, 2017 11:33 PM, "xoviat"  wrote:

> Just this morning, Paul said the following:
>
> That step's the problem. If the frontend does that it can potentially
> be copying a lot of unneeded stuff (VCS history, for example). We
> tried that with pip and it was a major issue. That problem is the
> *whole point* of all the discussions about the various proposals that
> ended up with build_tree.
>
> I took that to mean that we were trusting the backend to do the right
> thing. And most people agreed with that. I don't personally care but there
> does seem to be some miscommunication here.
>
> On Aug 24, 2017 11:24 PM, "Nathaniel Smith"  wrote:
>
>> On Thu, Aug 24, 2017 at 9:17 PM, xoviat  wrote:
>> > > I'm *not* OK with banning in-tree builds in the spec, since that's
>> > > both unnecessary and unenforceable
>> >
>> > Well then either we can trust the backend or we cannot. If we can, then
>> this
>> > is both necessary and enforceable. If not, then we're back to pip
>> copying
>> > files. You can't make and argument that it's okay to trust build_sdist
>> but
>> > not build_wheel.
>>
>> I think at this point everyone has made their peace with the pip
>> developers' decision that they want to keep copying files -- at least
>> for now -- and that's just how it's going to be. This email has a more
>> detailed discussion of the options, their "threat model", and the
>> tradeoffs:
>>
>> https://mail.python.org/pipermail/distutils-sig/2017-July/031020.html
>>
>> I can see an argument for adding language saying that build_sdist
>> SHOULD avoid modifying the source tree if possible, and MAY write
>> scratch files to the sdist_directory.
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

Just this morning, Paul said the following:

That step's the problem. If the frontend does that it can potentially
be copying a lot of unneeded stuff (VCS history, for example). We
tried that with pip and it was a major issue. That problem is the
*whole point* of all the discussions about the various proposals that
ended up with build_tree.

I took that to mean that we were trusting the backend to do the right
thing. And most people agreed with that. I don't personally care but there
does seem to be some miscommunication here.

On Aug 24, 2017 11:24 PM, "Nathaniel Smith"  wrote:

> On Thu, Aug 24, 2017 at 9:17 PM, xoviat  wrote:
> > > I'm *not* OK with banning in-tree builds in the spec, since that's
> > > both unnecessary and unenforceable
> >
> > Well then either we can trust the backend or we cannot. If we can, then
> this
> > is both necessary and enforceable. If not, then we're back to pip copying
> > files. You can't make and argument that it's okay to trust build_sdist
> but
> > not build_wheel.
>
> I think at this point everyone has made their peace with the pip
> developers' decision that they want to keep copying files -- at least
> for now -- and that's just how it's going to be. This email has a more
> detailed discussion of the options, their "threat model", and the
> tradeoffs:
>
> https://mail.python.org/pipermail/distutils-sig/2017-July/031020.html
>
> I can see an argument for adding language saying that build_sdist
> SHOULD avoid modifying the source tree if possible, and MAY write
> scratch files to the sdist_directory.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Nathaniel Smith]

On Thu, Aug 24, 2017 at 9:17 PM, xoviat  wrote:
> > I'm *not* OK with banning in-tree builds in the spec, since that's
> > both unnecessary and unenforceable
>
> Well then either we can trust the backend or we cannot. If we can, then this
> is both necessary and enforceable. If not, then we're back to pip copying
> files. You can't make and argument that it's okay to trust build_sdist but
> not build_wheel.

I think at this point everyone has made their peace with the pip
developers' decision that they want to keep copying files -- at least
for now -- and that's just how it's going to be. This email has a more
detailed discussion of the options, their "threat model", and the
tradeoffs:

https://mail.python.org/pipermail/distutils-sig/2017-July/031020.html

I can see an argument for adding language saying that build_sdist
SHOULD avoid modifying the source tree if possible, and MAY write
scratch files to the sdist_directory.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

I'm *not* OK with banning in-tree builds in the spec, since that's
both unnecessary and unenforceable

Well then either we can trust the backend or we cannot. If we can, then
this is both necessary and enforceable. If not, then we're back to pip
copying files. You can't make and argument that it's okay to trust
build_sdist but not build_wheel.

On Aug 24, 2017 10:14 PM, "Nick Coghlan"  wrote:

> On 25 August 2017 at 10:08, Daniel Holth  wrote:
> > It would be simpler if the only requirement was to produce a correct
> wheel.
>
> +1
>
> To be clear on my current position:
>
> * due to the introduction of NotImplementedError, I'm now OK with
> explicitly out-of-tree builds being deferred to a later iteration of
> the API
> * I'm *not* OK with banning in-tree builds in the spec, since that's
> both unnecessary and unenforceable
>
> If a frontend really wants to ensure sdist/wheel consistency, then the
> way for it to do so is to build the sdist from the source tree, and
> then build the wheel from the sdist. Building a wheel directly from
> the source tree should ideally give the same results as building the
> sdist first, but it's explicitly OK for there to be cases where they
> end up being different (such as when build_sdist would raise
> NotImplementedError).
>
> As far as build_directory goes, the spec should just leave the
> question of artifact caching (or the lack thereof) up to the backend
> implementations for now - if/when we decide to add explicit artifact
> caching support to the API specification we can make a more informed
> decision at a future date based on what the initial generation of
> backends are actually doing.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Nick Coghlan]

On 25 August 2017 at 10:08, Daniel Holth  wrote:
> It would be simpler if the only requirement was to produce a correct wheel.

+1

To be clear on my current position:

* due to the introduction of NotImplementedError, I'm now OK with
explicitly out-of-tree builds being deferred to a later iteration of
the API
* I'm *not* OK with banning in-tree builds in the spec, since that's
both unnecessary and unenforceable

If a frontend really wants to ensure sdist/wheel consistency, then the
way for it to do so is to build the sdist from the source tree, and
then build the wheel from the sdist. Building a wheel directly from
the source tree should ideally give the same results as building the
sdist first, but it's explicitly OK for there to be cases where they
end up being different (such as when build_sdist would raise
NotImplementedError).

As far as build_directory goes, the spec should just leave the
question of artifact caching (or the lack thereof) up to the backend
implementations for now - if/when we decide to add explicit artifact
caching support to the API specification we can make a more informed
decision at a future date based on what the initial generation of
backends are actually doing.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

The current proposal is to remove build_dir and leave it up to the backend.

On Aug 24, 2017 7:07 PM, "Greg Ewing"  wrote:

> Paul Moore wrote:
>
>> 4. The point of all this is that the definition of build_directory
>> says "If build_directory is None, the backend may do an 'in place'
>> build which modifies the source directory and may produce different
>> results from those that would be obtained by exporting an sdist
>> first".
>>
>
> Is it really necessary for pip to specify exactly *where* to do
> an out-of-place build? Why should it care, as long as the end
> result is a clean wheel?
>
> --
> Greg
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Daniel Holth]

It would be simpler if the only requirement was to produce a correct wheel.

On Thu, Aug 24, 2017, 20:07 Greg Ewing  wrote:

> Paul Moore wrote:
> > 4. The point of all this is that the definition of build_directory
> > says "If build_directory is None, the backend may do an 'in place'
> > build which modifies the source directory and may produce different
> > results from those that would be obtained by exporting an sdist
> > first".
>
> Is it really necessary for pip to specify exactly *where* to do
> an out-of-place build? Why should it care, as long as the end
> result is a clean wheel?
>
> --
> Greg
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Greg Ewing]

Paul Moore wrote:

4. The point of all this is that the definition of build_directory
says "If build_directory is None, the backend may do an 'in place'
build which modifies the source directory and may produce different
results from those that would be obtained by exporting an sdist
first".


Is it really necessary for pip to specify exactly *where* to do
an out-of-place build? Why should it care, as long as the end
result is a clean wheel?

--
Greg
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

Actually I submitted a compliant PR 3 months ago when the PEP was first
marked as accepted. Then it was marked as draft and I thought that I would
just wait for distutils-sig to handle it. And then I came over here to find
what was going on.

On Aug 24, 2017 5:33 PM, "xoviat"  wrote:

> I'm handling the setuptools backend. I will make sure that it complies
> with the clean directory.
>
> On Aug 24, 2017 5:30 PM, "Paul Moore"  wrote:
>
>> On 24 August 2017 at 20:00, Thomas Kluyver  wrote:
>> > So for the time being, I'd prefer to say that the build_wheel() hook
>> should
>> > not create or modify files in the source directory. Backends may keep
>> their
>> > own caches elsewhere to speed up repeated builds, but any more
>> provisions
>> > for developer use cases should be left for a future PEP. And developers
>> can
>> > invoke enscons/flit/... directly in the meantime.
>>
>> That means a PEP 517 compatible interface to setuptools is more
>> complex, as it would *have* to specify a temporary build directory.
>> And I'm not sure whether there are any *other* files it dumps in the
>> source directory (egg info comes to mind, and PKG-INFO) and if so
>> whether it's easy to get setuptools to place them elsewhere. That's
>> fine, as far as I'm concerned, but it should be made explicit, IMO.
>> I'm not yet sure who (if anyone) is proposing to write a PEP 517
>> wrapper for setuptools (or if we'll need to special-case setuptools in
>> pip for "legacy" support) but we should make it as easy as possible
>> for anyone who does to get the details right, as it's likely to be a
>> very widely used backend.
>>
>> It would be a lot easier to handle the edge cases here if there were
>> someone writing a setuptools wrapper. Any takers? (I'm not sure I can
>> make the time to do so, and I *am* sure that if I did, I don't have
>> the time to support and maintain it going forward...)
>>
>> Paul
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

I'm handling the setuptools backend. I will make sure that it complies with
the clean directory.

On Aug 24, 2017 5:30 PM, "Paul Moore"  wrote:

> On 24 August 2017 at 20:00, Thomas Kluyver  wrote:
> > So for the time being, I'd prefer to say that the build_wheel() hook
> should
> > not create or modify files in the source directory. Backends may keep
> their
> > own caches elsewhere to speed up repeated builds, but any more provisions
> > for developer use cases should be left for a future PEP. And developers
> can
> > invoke enscons/flit/... directly in the meantime.
>
> That means a PEP 517 compatible interface to setuptools is more
> complex, as it would *have* to specify a temporary build directory.
> And I'm not sure whether there are any *other* files it dumps in the
> source directory (egg info comes to mind, and PKG-INFO) and if so
> whether it's easy to get setuptools to place them elsewhere. That's
> fine, as far as I'm concerned, but it should be made explicit, IMO.
> I'm not yet sure who (if anyone) is proposing to write a PEP 517
> wrapper for setuptools (or if we'll need to special-case setuptools in
> pip for "legacy" support) but we should make it as easy as possible
> for anyone who does to get the details right, as it's likely to be a
> very widely used backend.
>
> It would be a lot easier to handle the edge cases here if there were
> someone writing a setuptools wrapper. Any takers? (I'm not sure I can
> make the time to do so, and I *am* sure that if I did, I don't have
> the time to support and maintain it going forward...)
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Paul Moore]

On 24 August 2017 at 20:00, Thomas Kluyver  wrote:
> So for the time being, I'd prefer to say that the build_wheel() hook should
> not create or modify files in the source directory. Backends may keep their
> own caches elsewhere to speed up repeated builds, but any more provisions
> for developer use cases should be left for a future PEP. And developers can
> invoke enscons/flit/... directly in the meantime.

That means a PEP 517 compatible interface to setuptools is more
complex, as it would *have* to specify a temporary build directory.
And I'm not sure whether there are any *other* files it dumps in the
source directory (egg info comes to mind, and PKG-INFO) and if so
whether it's easy to get setuptools to place them elsewhere. That's
fine, as far as I'm concerned, but it should be made explicit, IMO.
I'm not yet sure who (if anyone) is proposing to write a PEP 517
wrapper for setuptools (or if we'll need to special-case setuptools in
pip for "legacy" support) but we should make it as easy as possible
for anyone who does to get the details right, as it's likely to be a
very widely used backend.

It would be a lot easier to handle the edge cases here if there were
someone writing a setuptools wrapper. Any takers? (I'm not sure I can
make the time to do so, and I *am* sure that if I did, I don't have
the time to support and maintain it going forward...)

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

So in summary:
- NotImplementedError
- Remove build_dir and specify that no modifications can be made to the
source directory

On the third issue:

I understand where Nathaniel is coming from in regards to sys.path. But
there should at least be an option for this because of for example, numpy:
numpy builds itself with it's own build system that is defined in the
source directory. If you remove '' from sys.path, then numpy cannot build
itself because there is no other numpy.distutils package.

2017-08-24 14:00 GMT-05:00 Thomas Kluyver :

> On Thu, Aug 24, 2017, at 07:30 PM, Leonardo Rochael Almeida wrote:
>
> It looks like a lot of trouble for a feature that is designed to solve a
> problem for a very thin intersection of the Venn diagram of people who:
>
>  a. wants to quickly iterate while hacking a package
>
>  b. doesn't want to bother learning how to trigger the incremental build
> system of that package or how to install it in editable mode.
>
> For better or worse, editable installs exist and don't need to be defined
> by PEP 517 just to handle this case. I'm not convinced it's worth the
> additional complexity of the spec just for that.
>
>
> This is a good point. Given that developers can use the build tool
> directly for development purposes, rather than going through the PEP 517
> interface, we don't need to prioritise that use case for the spec.
>
> So for the time being, I'd prefer to say that the build_wheel() hook
> should not create or modify files in the source directory. Backends may
> keep their own caches elsewhere to speed up repeated builds, but any more
> provisions for developer use cases should be left for a future PEP. And
> developers can invoke enscons/flit/... directly in the meantime.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Thomas Kluyver]

On Thu, Aug 24, 2017, at 07:30 PM, Leonardo Rochael Almeida wrote:
> It looks like a lot of trouble for a feature that is designed to
> solve a problem for a very thin intersection of the Venn diagram of
> people who:> 
>  a. wants to quickly iterate while hacking a package
> 
>  b. doesn't want to bother learning how to trigger the incremental
> build system of that package or how to install it in editable
> mode.> 
> For better or worse, editable installs exist and don't need to be
> defined by PEP 517 just to handle this case. I'm not convinced it's
> worth the additional complexity of the spec just for that.
This is a good point. Given that developers can use the build tool
directly for development purposes, rather than going through the PEP 517
interface, we don't need to prioritise that use case for the spec.
So for the time being, I'd prefer to say that the build_wheel() hook
should not create or modify files in the source directory. Backends may
keep their own caches elsewhere to speed up repeated builds, but any
more provisions for developer use cases should be left for a future PEP.
And developers can invoke enscons/flit/... directly in the meantime.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Daniel Holth]

The distutils build/ folder is what moves if you pass build_dir to the
proposed function.

On Thu, Aug 24, 2017, 14:39 xoviat  wrote:

> > It looks like a lot of trouble for a feature that is designed to solve a
> problem for a very thin intersection of the Venn diagram of people who:
>
> For the record, I don't agree at all that build_wheel_incremental should
> be specified here. The suggestion is simply a compromise that I can
> tolerate because it's optional.
>
> > For better or worse, editable installs exist and don't need to be
> defined by PEP 517 just to handle this case. I'm not convinced it's worth
> the additional complexity of the spec just for that.
>
> The editable installs in my opinion should have been specified in this PEP
> because people are clearly trying to work around a lack of consensus on
> that issue so that they don't have to wait for the next PEP and waste their
> time waiting for uncached builds. I can't say I disagree with this but it's
> trying to fit a square peg into a round hole.
>
> > Is there any backend doing this "caching elsewhere" today? Is any
> backend planning to?
>
> Yes. Distutils caches in the build folder. It's very useful when working
> on complex projects with Extensions.
>
> > Can we arrive at some more concrete examples of how that would work?
> Like how would pip be invoked, how would the backend react.
>
> The reason that I'm willing to live with a separate optional function is
> that I don't think it will ever be implemented. It will be a note in Python
> history until install_editable is specified.
>
> 2017-08-24 13:30 GMT-05:00 Leonardo Rochael Almeida 
> :
>
>>
>> On 24 August 2017 at 15:13, Thomas Kluyver  wrote:
>>
>>> On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote:
>>>
>>> On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver 
>>> wrote:
>>>
>>> Is there a reason to ask for an 'unclean' build, though? There may be a
>>> performance optimisation from reusing cached data,
>>>
>>>
>>> For the same reason you would ever ask for incremental builds, to more
>>> quickly iterate while hacking, imagining that you are using the PEP 517
>>> interface to develop, perhaps to have a uniform interface to patch
>>> something when you are not familiar with exactly the build system it uses.
>>>
>>> It looks like a lot of trouble for a feature that is designed to solve a
>> problem for a very thin intersection of the Venn diagram of people who:
>>
>>  a. wants to quickly iterate while hacking a package
>>
>>  b. doesn't want to bother learning how to trigger the incremental build
>> system of that package or how to install it in editable mode.
>>
>> For better or worse, editable installs exist and don't need to be defined
>> by PEP 517 just to handle this case. I'm not convinced it's worth the
>> additional complexity of the spec just for that.
>>
>> Yup, this is what I meant about using cached data for performance. But I
>>> don't think this requires builds to modify the source directory. The build
>>> system can cache data elsewhere on the filesystem, whether or not it's
>>> given an explicit build_directory.
>>>
>>
>> Is there any backend doing this "caching elsewhere" today? Is any backend
>> planning to?
>>
>>
>>> I think we can specify one kind of build that is both 'clean' (doesn't
>>> modify the source directory) and incremental (can use cached data to speed
>>> up the build). If that's workable, it seems like it would save a lot of
>>> headaches rather than trying to specify them as two different options.
>>>
>>> We have explicitly excluded editable installs (i.e. inplace builds) from
>>> this PEP, though we can add a hook for that in a later PEP. I agree with
>>> you that this PEP should allow for fast incremental (but non-inplace)
>>> builds if possible, though.
>>>
>>
>> Can we arrive at some more concrete examples of how that would work? Like
>> how would pip be invoked, how would the backend react. And how (and for
>> whom) this would be better than either doing an editable install or
>> straight clean install?
>>
>> I fear we're complicating the spec for something that will be hard to
>> happen in practice.
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

> It looks like a lot of trouble for a feature that is designed to solve a
problem for a very thin intersection of the Venn diagram of people who:

For the record, I don't agree at all that build_wheel_incremental should be
specified here. The suggestion is simply a compromise that I can tolerate
because it's optional.

> For better or worse, editable installs exist and don't need to be defined
by PEP 517 just to handle this case. I'm not convinced it's worth the
additional complexity of the spec just for that.

The editable installs in my opinion should have been specified in this PEP
because people are clearly trying to work around a lack of consensus on
that issue so that they don't have to wait for the next PEP and waste their
time waiting for uncached builds. I can't say I disagree with this but it's
trying to fit a square peg into a round hole.

> Is there any backend doing this "caching elsewhere" today? Is any backend
planning to?

Yes. Distutils caches in the build folder. It's very useful when working on
complex projects with Extensions.

> Can we arrive at some more concrete examples of how that would work? Like
how would pip be invoked, how would the backend react.

The reason that I'm willing to live with a separate optional function is
that I don't think it will ever be implemented. It will be a note in Python
history until install_editable is specified.

2017-08-24 13:30 GMT-05:00 Leonardo Rochael Almeida :

>
> On 24 August 2017 at 15:13, Thomas Kluyver  wrote:
>
>> On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote:
>>
>> On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver 
>> wrote:
>>
>> Is there a reason to ask for an 'unclean' build, though? There may be a
>> performance optimisation from reusing cached data,
>>
>>
>> For the same reason you would ever ask for incremental builds, to more
>> quickly iterate while hacking, imagining that you are using the PEP 517
>> interface to develop, perhaps to have a uniform interface to patch
>> something when you are not familiar with exactly the build system it uses.
>>
>> It looks like a lot of trouble for a feature that is designed to solve a
> problem for a very thin intersection of the Venn diagram of people who:
>
>  a. wants to quickly iterate while hacking a package
>
>  b. doesn't want to bother learning how to trigger the incremental build
> system of that package or how to install it in editable mode.
>
> For better or worse, editable installs exist and don't need to be defined
> by PEP 517 just to handle this case. I'm not convinced it's worth the
> additional complexity of the spec just for that.
>
> Yup, this is what I meant about using cached data for performance. But I
>> don't think this requires builds to modify the source directory. The build
>> system can cache data elsewhere on the filesystem, whether or not it's
>> given an explicit build_directory.
>>
>
> Is there any backend doing this "caching elsewhere" today? Is any backend
> planning to?
>
>
>> I think we can specify one kind of build that is both 'clean' (doesn't
>> modify the source directory) and incremental (can use cached data to speed
>> up the build). If that's workable, it seems like it would save a lot of
>> headaches rather than trying to specify them as two different options.
>>
>> We have explicitly excluded editable installs (i.e. inplace builds) from
>> this PEP, though we can add a hook for that in a later PEP. I agree with
>> you that this PEP should allow for fast incremental (but non-inplace)
>> builds if possible, though.
>>
>
> Can we arrive at some more concrete examples of how that would work? Like
> how would pip be invoked, how would the backend react. And how (and for
> whom) this would be better than either doing an editable install or
> straight clean install?
>
> I fear we're complicating the spec for something that will be hard to
> happen in practice.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Leonardo Rochael Almeida]

On 24 August 2017 at 15:13, Thomas Kluyver  wrote:

> On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote:
>
> On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver 
> wrote:
>
> Is there a reason to ask for an 'unclean' build, though? There may be a
> performance optimisation from reusing cached data,
>
>
> For the same reason you would ever ask for incremental builds, to more
> quickly iterate while hacking, imagining that you are using the PEP 517
> interface to develop, perhaps to have a uniform interface to patch
> something when you are not familiar with exactly the build system it uses.
>
> It looks like a lot of trouble for a feature that is designed to solve a
problem for a very thin intersection of the Venn diagram of people who:

 a. wants to quickly iterate while hacking a package

 b. doesn't want to bother learning how to trigger the incremental build
system of that package or how to install it in editable mode.

For better or worse, editable installs exist and don't need to be defined
by PEP 517 just to handle this case. I'm not convinced it's worth the
additional complexity of the spec just for that.

Yup, this is what I meant about using cached data for performance. But I
> don't think this requires builds to modify the source directory. The build
> system can cache data elsewhere on the filesystem, whether or not it's
> given an explicit build_directory.
>

Is there any backend doing this "caching elsewhere" today? Is any backend
planning to?


> I think we can specify one kind of build that is both 'clean' (doesn't
> modify the source directory) and incremental (can use cached data to speed
> up the build). If that's workable, it seems like it would save a lot of
> headaches rather than trying to specify them as two different options.
>
> We have explicitly excluded editable installs (i.e. inplace builds) from
> this PEP, though we can add a hook for that in a later PEP. I agree with
> you that this PEP should allow for fast incremental (but non-inplace)
> builds if possible, though.
>

Can we arrive at some more concrete examples of how that would work? Like
how would pip be invoked, how would the backend react. And how (and for
whom) this would be better than either doing an editable install or
straight clean install?

I fear we're complicating the spec for something that will be hard to
happen in practice.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

To be clear, is everyone okay with the following?

build_wheel(wheel_directory, config_settings=None,
metadata_directory=None): [REQUIRED]
build_wheel_incremental(wheel_directory, config_settings=None,
metadata_directory=None): [OPTIONAL]


I still think that the frontend should not provide a build_directory, but
if someone *really* wants the build_directory in the incremental function,
then I will "disagree and commit."

2017-08-24 13:13 GMT-05:00 Thomas Kluyver :

> On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote:
>
> On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver 
> wrote:
>
> Is there a reason to ask for an 'unclean' build, though? There may be a
> performance optimisation from reusing cached data,
>
>
> For the same reason you would ever ask for incremental builds, to more
> quickly iterate while hacking, imagining that you are using the PEP 517
> interface to develop, perhaps to have a uniform interface to patch
> something when you are not familiar with exactly the build system it uses.
>
>
> Yup, this is what I meant about using cached data for performance. But I
> don't think this requires builds to modify the source directory. The build
> system can cache data elsewhere on the filesystem, whether or not it's
> given an explicit build_directory.
>
> I think we can specify one kind of build that is both 'clean' (doesn't
> modify the source directory) and incremental (can use cached data to speed
> up the build). If that's workable, it seems like it would save a lot of
> headaches rather than trying to specify them as two different options.
>
> We have explicitly excluded editable installs (i.e. inplace builds) from
> this PEP, though we can add a hook for that in a later PEP. I agree with
> you that this PEP should allow for fast incremental (but non-inplace)
> builds if possible, though.
>
> Thomas
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Thomas Kluyver]

On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote:
> On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver
>  wrote:>> Is there a reason to ask for an 'unclean' 
> build, though? There may be
>> a performance optimisation from reusing cached data,> 
> For the same reason you would ever ask for incremental builds, to
> more quickly iterate while hacking, imagining that you are using the
> PEP 517 interface to develop, perhaps to have a uniform interface to
> patch something when you are not familiar with exactly the build
> system it uses.
Yup, this is what I meant about using cached data for performance. But I
don't think this requires builds to modify the source directory. The
build system can cache data elsewhere on the filesystem, whether or not
it's given an explicit build_directory.
I think we can specify one kind of build that is both 'clean'
(doesn't modify the source directory) and incremental (can use cached
data to speed up the build). If that's workable, it seems like it
would save a lot of headaches rather than trying to specify them as
two different options.
We have explicitly excluded editable installs (i.e. inplace builds) from
this PEP, though we can add a hook for that in a later PEP. I agree with
you that this PEP should allow for fast incremental (but non-inplace)
builds if possible, though.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

> For the same reason you would ever ask for incremental builds, to more
quickly iterate while hacking, imagining that you are using the PEP 517
interface to develop, perhaps to have a uniform interface to patch
something when you are not familiar with exactly the build system it uses.

And so I reiterate:

> I understand that, but what I disagree with is modifying build_wheel to
make up for a lack of consensus on editable installs.

I don't think the build_wheel hook should be used for iteration or
development, because the intended purpose is for production. And I don't
think we should even consider *any* specific deficiencies of setuptools or
wheel when designing a specification.

2017-08-24 12:20 GMT-05:00 Daniel Holth :

> On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver 
> wrote:
>
>> On Thu, Aug 24, 2017, at 05:26 PM, Daniel Holth wrote:
>>
>> by including the build_dir parameter, a nearly universal build system
>> concept, pip gets an elegant way to ask for either a clean or unclean build.
>>
>>
>> Is there a reason to ask for an 'unclean' build, though? There may be a
>> performance optimisation from reusing cached data,
>>
>
> For the same reason you would ever ask for incremental builds, to more
> quickly iterate while hacking, imagining that you are using the PEP 517
> interface to develop, perhaps to have a uniform interface to patch
> something when you are not familiar with exactly the build system it uses.
>
>
>> but building in a separate directory doesn't preclude caching
>> intermediates somewhere else.
>>
>
> The argument is necessarily a hint to the build system? If it works it can
> do whatever.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Daniel Holth]

On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver 
wrote:

> On Thu, Aug 24, 2017, at 05:26 PM, Daniel Holth wrote:
>
> by including the build_dir parameter, a nearly universal build system
> concept, pip gets an elegant way to ask for either a clean or unclean build.
>
>
> Is there a reason to ask for an 'unclean' build, though? There may be a
> performance optimisation from reusing cached data,
>

For the same reason you would ever ask for incremental builds, to more
quickly iterate while hacking, imagining that you are using the PEP 517
interface to develop, perhaps to have a uniform interface to patch
something when you are not familiar with exactly the build system it uses.


> but building in a separate directory doesn't preclude caching
> intermediates somewhere else.
>

The argument is necessarily a hint to the build system? If it works it can
do whatever.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Daniel Holth]

build_dir:

bdist_wheel sometimes has a problem with unclean builds. admittedly this
could be fixed.
pip copies entire source tree, including unclean build/ directory, to tmpdir
bdist_wheel runs in tmpdir, including copied leftover build/ data in wheel

by including the build_dir parameter, a nearly universal build system
concept, pip gets an elegant way to ask for either a clean or unclean build.

sdist optional:

enscons can use .hg manifest instead of MANIFEST.in. what is this git
nonsense?
user has decided to build from an unpacked sdist
impossible to create new sdist from unpacked sdist



On Thu, Aug 24, 2017 at 11:49 AM xoviat  wrote:

> If the entire idea of copying out-of-tree is to work around setuptools
> deficiencies, then perhaps it would be a better idea to push this onto the
> setuptools build backend rather than bring these problems into PEP 517?
>
> 2017-08-24 10:32 GMT-05:00 xoviat :
>
>> May I ask what is wrong *in principle* with the setuptools "build" folder
>> (other than the fact that it does not contain all tree changes)?
>>
>> 2017-08-24 10:27 GMT-05:00 xoviat :
>>
>>> That is actually the first time that build_dir makes sense to me now.
>>> Thank you.
>>>
>>> 2017-08-24 10:24 GMT-05:00 Paul Moore :
>>>
 On 24 August 2017 at 16:20, xoviat  wrote:
 >>  I *do* care about telling backends we don't want "different
 > results from those that would be obtained by exporting an sdist
 > first".
 >
 > I completely agree with this statement, but I don't believe that it
 can or
 > should be accomplished with this parameter. Let me just quote the
 process
 > that I proposed:
 >
 >
 >> Proposed process:
 > - Frontend copies source tree to temporary directory

 That step's the problem. If the frontend does that it can potentially
 be copying a lot of unneeded stuff (VCS history, for example). We
 tried that with pip and it was a major issue. That problem is the
 *whole point* of all the discussions about the various proposals that
 ended up with build_tree.

 > - Frontend invokes build-sdist to build an sdist
 > - Frontend extracts sdist to new temporary directory
 > - Frontend reloads backend from sdist directory and invokes
 build-wheel

 Paul

>>>
>>>
>>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

If the entire idea of copying out-of-tree is to work around setuptools
deficiencies, then perhaps it would be a better idea to push this onto the
setuptools build backend rather than bring these problems into PEP 517?

2017-08-24 10:32 GMT-05:00 xoviat :

> May I ask what is wrong *in principle* with the setuptools "build" folder
> (other than the fact that it does not contain all tree changes)?
>
> 2017-08-24 10:27 GMT-05:00 xoviat :
>
>> That is actually the first time that build_dir makes sense to me now.
>> Thank you.
>>
>> 2017-08-24 10:24 GMT-05:00 Paul Moore :
>>
>>> On 24 August 2017 at 16:20, xoviat  wrote:
>>> >>  I *do* care about telling backends we don't want "different
>>> > results from those that would be obtained by exporting an sdist
>>> > first".
>>> >
>>> > I completely agree with this statement, but I don't believe that it
>>> can or
>>> > should be accomplished with this parameter. Let me just quote the
>>> process
>>> > that I proposed:
>>> >
>>> >
>>> >> Proposed process:
>>> > - Frontend copies source tree to temporary directory
>>>
>>> That step's the problem. If the frontend does that it can potentially
>>> be copying a lot of unneeded stuff (VCS history, for example). We
>>> tried that with pip and it was a major issue. That problem is the
>>> *whole point* of all the discussions about the various proposals that
>>> ended up with build_tree.
>>>
>>> > - Frontend invokes build-sdist to build an sdist
>>> > - Frontend extracts sdist to new temporary directory
>>> > - Frontend reloads backend from sdist directory and invokes build-wheel
>>>
>>> Paul
>>>
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

May I ask what is wrong *in principle* with the setuptools "build" folder
(other than the fact that it does not contain all tree changes)?

2017-08-24 10:27 GMT-05:00 xoviat :

> That is actually the first time that build_dir makes sense to me now.
> Thank you.
>
> 2017-08-24 10:24 GMT-05:00 Paul Moore :
>
>> On 24 August 2017 at 16:20, xoviat  wrote:
>> >>  I *do* care about telling backends we don't want "different
>> > results from those that would be obtained by exporting an sdist
>> > first".
>> >
>> > I completely agree with this statement, but I don't believe that it can
>> or
>> > should be accomplished with this parameter. Let me just quote the
>> process
>> > that I proposed:
>> >
>> >
>> >> Proposed process:
>> > - Frontend copies source tree to temporary directory
>>
>> That step's the problem. If the frontend does that it can potentially
>> be copying a lot of unneeded stuff (VCS history, for example). We
>> tried that with pip and it was a major issue. That problem is the
>> *whole point* of all the discussions about the various proposals that
>> ended up with build_tree.
>>
>> > - Frontend invokes build-sdist to build an sdist
>> > - Frontend extracts sdist to new temporary directory
>> > - Frontend reloads backend from sdist directory and invokes build-wheel
>>
>> Paul
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

That is actually the first time that build_dir makes sense to me now. Thank
you.

2017-08-24 10:24 GMT-05:00 Paul Moore :

> On 24 August 2017 at 16:20, xoviat  wrote:
> >>  I *do* care about telling backends we don't want "different
> > results from those that would be obtained by exporting an sdist
> > first".
> >
> > I completely agree with this statement, but I don't believe that it can
> or
> > should be accomplished with this parameter. Let me just quote the process
> > that I proposed:
> >
> >
> >> Proposed process:
> > - Frontend copies source tree to temporary directory
>
> That step's the problem. If the frontend does that it can potentially
> be copying a lot of unneeded stuff (VCS history, for example). We
> tried that with pip and it was a major issue. That problem is the
> *whole point* of all the discussions about the various proposals that
> ended up with build_tree.
>
> > - Frontend invokes build-sdist to build an sdist
> > - Frontend extracts sdist to new temporary directory
> > - Frontend reloads backend from sdist directory and invokes build-wheel
>
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Paul Moore]

On 24 August 2017 at 16:23, xoviat  wrote:
>>  I don't recall you having said "we do sdist->wheel then fall back to
> requesting wheels directly"
>
> You're correct. I did not say that because build_sdist is required.

Wait, what? Flit specifically needs to be allowed to refuse to build
sdists (as I understand it), for example if the user doesn't have git
installed. I thought that's what the NotImplemented exception was for?

You're right that the PEP doesn't seem to mention this.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

> I know Daniel has been involved in the discussion and I don't think
he's raised any such objection about the PEP (and he developed
enscons, so he has direct experience of writing backends).

Daniel raised that specific objection.

2017-08-24 10:21 GMT-05:00 Paul Moore :

> On 24 August 2017 at 16:15, xoviat  wrote:
> >> 2. I'm not completely clear on how pip's implementation will work - I
> > think the intention is to always build a sdist and build a wheel from
> > that, unless the backend reports it can't build a sdist, in which case
> > we ask it to build a wheel directly.
> >
> > This was the exact process that I proposed, but was told that:
> >
> >
> >> I agree that the way you want to do packaging is fundamentally
> >> incompatible
> > with build systems that do not resemble distutils. Perhaps since this is
> > distutils sig some here are too used to distutils as the only model of
> how
> > packaging might work. Beware that, when fighting monsters, you yourself
> do
> > not become a monster... for when you gaze long into the abyss. The abyss
> > gazes also into you.
>
> That wasn't me that said that.
>
> > There seems to be some miscommunication about the actual process being
> > proposed.
>
> Possibly - or there's some confusion about what you're proposing. I
> certainly have found your suggestions difficult to follow, and I don't
> recall you having said "we do sdist->wheel then fall back to
> requesting wheels directly". I also don't see how that's incompatible
> with systems that don't resemble distutils, so I suspect that whoever
> you quoted (which I think might have been Daniel) understood whatever
> you did say to mean something different.
>
> I know Daniel has been involved in the discussion and I don't think
> he's raised any such objection about the PEP (and he developed
> enscons, so he has direct experience of writing backends).
>
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Paul Moore]

On 24 August 2017 at 16:20, xoviat  wrote:
>>  I *do* care about telling backends we don't want "different
> results from those that would be obtained by exporting an sdist
> first".
>
> I completely agree with this statement, but I don't believe that it can or
> should be accomplished with this parameter. Let me just quote the process
> that I proposed:
>
>
>> Proposed process:
> - Frontend copies source tree to temporary directory

That step's the problem. If the frontend does that it can potentially
be copying a lot of unneeded stuff (VCS history, for example). We
tried that with pip and it was a major issue. That problem is the
*whole point* of all the discussions about the various proposals that
ended up with build_tree.

> - Frontend invokes build-sdist to build an sdist
> - Frontend extracts sdist to new temporary directory
> - Frontend reloads backend from sdist directory and invokes build-wheel

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

>  I don't recall you having said "we do sdist->wheel then fall back to
requesting wheels directly"

You're correct. I did not say that because build_sdist is required.

2017-08-24 10:21 GMT-05:00 Paul Moore :

> On 24 August 2017 at 16:15, xoviat  wrote:
> >> 2. I'm not completely clear on how pip's implementation will work - I
> > think the intention is to always build a sdist and build a wheel from
> > that, unless the backend reports it can't build a sdist, in which case
> > we ask it to build a wheel directly.
> >
> > This was the exact process that I proposed, but was told that:
> >
> >
> >> I agree that the way you want to do packaging is fundamentally
> >> incompatible
> > with build systems that do not resemble distutils. Perhaps since this is
> > distutils sig some here are too used to distutils as the only model of
> how
> > packaging might work. Beware that, when fighting monsters, you yourself
> do
> > not become a monster... for when you gaze long into the abyss. The abyss
> > gazes also into you.
>
> That wasn't me that said that.
>
> > There seems to be some miscommunication about the actual process being
> > proposed.
>
> Possibly - or there's some confusion about what you're proposing. I
> certainly have found your suggestions difficult to follow, and I don't
> recall you having said "we do sdist->wheel then fall back to
> requesting wheels directly". I also don't see how that's incompatible
> with systems that don't resemble distutils, so I suspect that whoever
> you quoted (which I think might have been Daniel) understood whatever
> you did say to mean something different.
>
> I know Daniel has been involved in the discussion and I don't think
> he's raised any such objection about the PEP (and he developed
> enscons, so he has direct experience of writing backends).
>
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Paul Moore]

On 24 August 2017 at 16:15, xoviat  wrote:
>> 2. I'm not completely clear on how pip's implementation will work - I
> think the intention is to always build a sdist and build a wheel from
> that, unless the backend reports it can't build a sdist, in which case
> we ask it to build a wheel directly.
>
> This was the exact process that I proposed, but was told that:
>
>
>> I agree that the way you want to do packaging is fundamentally
>> incompatible
> with build systems that do not resemble distutils. Perhaps since this is
> distutils sig some here are too used to distutils as the only model of how
> packaging might work. Beware that, when fighting monsters, you yourself do
> not become a monster... for when you gaze long into the abyss. The abyss
> gazes also into you.

That wasn't me that said that.

> There seems to be some miscommunication about the actual process being
> proposed.

Possibly - or there's some confusion about what you're proposing. I
certainly have found your suggestions difficult to follow, and I don't
recall you having said "we do sdist->wheel then fall back to
requesting wheels directly". I also don't see how that's incompatible
with systems that don't resemble distutils, so I suspect that whoever
you quoted (which I think might have been Daniel) understood whatever
you did say to mean something different.

I know Daniel has been involved in the discussion and I don't think
he's raised any such objection about the PEP (and he developed
enscons, so he has direct experience of writing backends).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

>  I *do* care about telling backends we don't want "different
results from those that would be obtained by exporting an sdist
first".

I completely agree with this statement, but I don't believe that it can or
should be accomplished with this parameter. Let me just quote the process
that I proposed:


> Proposed process:
- Frontend copies source tree to temporary directory
- Frontend invokes build-sdist to build an sdist
- Frontend extracts sdist to new temporary directory
- Frontend reloads backend from sdist directory and invokes build-wheel

This process makes the most sense to me because there will be a limited
number of frontends AND the frontends will enforce correctness. I don't
think we should be too cautious about burdening the frontends will more
responsibility, because it will only be implemented once or twice. And to
clarify, the build_sdist is required.

2017-08-24 9:51 GMT-05:00 Paul Moore :

> On 24 August 2017 at 15:26, Thomas Kluyver  wrote:
> > I particularly want to hear from Nick, Daniel, Donald and Paul on their
> > understanding of the build_directory parameter. There were reasons why it
> > got added, and even if those reasons no longer seem so convincing to me,
> > it's only going to be removed again if we can reach a consensus on
> working
> > without it.
>
> OK. I've not had much time recently (RL issues) and so my apologies if
> I've forgotten context here. But my feelings are:
>
> 1. I've been pretty much persuaded that including features in the
> interface to defend against badly written backends is not worthwhile.
> (I still worry that it may be hard to provide good error reporting for
> backend errors, but that's a lesser issue).
> 2. I'm not completely clear on how pip's implementation will work - I
> think the intention is to always build a sdist and build a wheel from
> that, unless the backend reports it can't build a sdist, in which case
> we ask it to build a wheel directly.
> 3. I'm not particularly comfortable with backends saying that they can
> only produce a wheel and not a sdist, but accept that might have to be
> a reality in some cases. I would be *really* bothered by a backend
> that wasn't even willing to accept that sdist->wheel and direct wheel
> builds should be equivalent - but I don't know if that's something
> that is likely to happen (we can of course state that expectation in
> the PEP). The PEP seems to think this is possible, though[1] (see my
> next point).
> 4. The point of all this is that the definition of build_directory
> says "If build_directory is None, the backend may do an 'in place'
> build which modifies the source directory and may produce different
> results from those that would be obtained by exporting an sdist
> first". That's the bit that bothers me a lot - if we lose
> build_directory, we lose the option for pip to request that a backend
> produce a wheel "equivalent to producing an sdist first". And that
> capability is something I think pip should require[2]. I don't
> particularly care who specifies the actual directory (as per your
> comment, I thought someone else wanted this, for caching or something)
> but I *do* care about telling backends we don't want "different
> results from those that would be obtained by exporting an sdist
> first".
>
> Paul
>
> [1] The only example I can think of is flit where the user doesn't
> have git installed. But then I'd expect that a wheel generated
> directly in that case would be the same as if the user installed git
> then did sdist->wheel, so I'd still consider that to be maintaining
> the invariant I care about.
> [2] I'm not likely to be the one implementing pip's PEP 517 support,
> so I could be overridden on this, of course.
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

> 2. I'm not completely clear on how pip's implementation will work - I
think the intention is to always build a sdist and build a wheel from
that, unless the backend reports it can't build a sdist, in which case
we ask it to build a wheel directly.

This was the exact process that I proposed, but was told that:


> I agree that the way you want to do packaging is fundamentally
incompatible
with build systems that do not resemble distutils. Perhaps since this is
distutils sig some here are too used to distutils as the only model of how
packaging might work. Beware that, when fighting monsters, you yourself do
not become a monster... for when you gaze long into the abyss. The abyss
gazes also into you.

There seems to be some miscommunication about the actual process being
proposed.


2017-08-24 10:07 GMT-05:00 Nick Coghlan :

> On 25 August 2017 at 00:51, Paul Moore  wrote:
> > 4. The point of all this is that the definition of build_directory
> > says "If build_directory is None, the backend may do an 'in place'
> > build which modifies the source directory and may produce different
> > results from those that would be obtained by exporting an sdist
> > first". That's the bit that bothers me a lot - if we lose
> > build_directory, we lose the option for pip to request that a backend
> > produce a wheel "equivalent to producing an sdist first". And that
> > capability is something I think pip should require[2]. I don't
> > particularly care who specifies the actual directory (as per your
> > comment, I thought someone else wanted this, for caching or something)
> > but I *do* care about telling backends we don't want "different
> > results from those that would be obtained by exporting an sdist
> > first".
>
> That's a phrasing error in the PEP - ideally, backends should always
> produce consistent results regardless of whether or not the frontend
> builds an sdist first. What that phrasing was attempting to
> acknowledge is that actually ensuring sdist/wheel consistency may have
> some extra requirements (like the git CLI) that may not apply to the
> "just give me a usable wheel, I don't care whether it's publication
> quality or not" case.
>
> The key thing that changed my mind on whether or not we needed to
> include the build_directory parameter from the start was actually the
> NotImplementedError discussion: as soon as we permitted backends to
> raise NotImplementedError for any non-None build_directory setting,
> we'd effectively made supporting the feature optional.
>
> And if supporting the feature's going to be optional anyway, then the
> *whole point* of defining the backend API in Python rather than as a
> CLI was to make it easy for us to add new optional methods later,
> which means we can start without build_directory support, and if we
> find we miss it for some specific reason, then we can add it back as a
> suitably named optional method like "build_wheel_variant" (if the
> use-case is cross-compiling for multiple ABIs without clobbering
> intermediate artifacts), or "build_wheel_with_artifact_cache" (if the
> use case is to allow the frontend to ask that the backend cache
> intermediate values in a particular place).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Nick Coghlan]

On 25 August 2017 at 00:51, Paul Moore  wrote:
> 4. The point of all this is that the definition of build_directory
> says "If build_directory is None, the backend may do an 'in place'
> build which modifies the source directory and may produce different
> results from those that would be obtained by exporting an sdist
> first". That's the bit that bothers me a lot - if we lose
> build_directory, we lose the option for pip to request that a backend
> produce a wheel "equivalent to producing an sdist first". And that
> capability is something I think pip should require[2]. I don't
> particularly care who specifies the actual directory (as per your
> comment, I thought someone else wanted this, for caching or something)
> but I *do* care about telling backends we don't want "different
> results from those that would be obtained by exporting an sdist
> first".

That's a phrasing error in the PEP - ideally, backends should always
produce consistent results regardless of whether or not the frontend
builds an sdist first. What that phrasing was attempting to
acknowledge is that actually ensuring sdist/wheel consistency may have
some extra requirements (like the git CLI) that may not apply to the
"just give me a usable wheel, I don't care whether it's publication
quality or not" case.

The key thing that changed my mind on whether or not we needed to
include the build_directory parameter from the start was actually the
NotImplementedError discussion: as soon as we permitted backends to
raise NotImplementedError for any non-None build_directory setting,
we'd effectively made supporting the feature optional.

And if supporting the feature's going to be optional anyway, then the
*whole point* of defining the backend API in Python rather than as a
CLI was to make it easy for us to add new optional methods later,
which means we can start without build_directory support, and if we
find we miss it for some specific reason, then we can add it back as a
suitably named optional method like "build_wheel_variant" (if the
use-case is cross-compiling for multiple ABIs without clobbering
intermediate artifacts), or "build_wheel_with_artifact_cache" (if the
use case is to allow the frontend to ask that the backend cache
intermediate values in a particular place).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Paul Moore]

On 24 August 2017 at 15:26, Thomas Kluyver  wrote:
> I particularly want to hear from Nick, Daniel, Donald and Paul on their
> understanding of the build_directory parameter. There were reasons why it
> got added, and even if those reasons no longer seem so convincing to me,
> it's only going to be removed again if we can reach a consensus on working
> without it.

OK. I've not had much time recently (RL issues) and so my apologies if
I've forgotten context here. But my feelings are:

1. I've been pretty much persuaded that including features in the
interface to defend against badly written backends is not worthwhile.
(I still worry that it may be hard to provide good error reporting for
backend errors, but that's a lesser issue).
2. I'm not completely clear on how pip's implementation will work - I
think the intention is to always build a sdist and build a wheel from
that, unless the backend reports it can't build a sdist, in which case
we ask it to build a wheel directly.
3. I'm not particularly comfortable with backends saying that they can
only produce a wheel and not a sdist, but accept that might have to be
a reality in some cases. I would be *really* bothered by a backend
that wasn't even willing to accept that sdist->wheel and direct wheel
builds should be equivalent - but I don't know if that's something
that is likely to happen (we can of course state that expectation in
the PEP). The PEP seems to think this is possible, though[1] (see my
next point).
4. The point of all this is that the definition of build_directory
says "If build_directory is None, the backend may do an 'in place'
build which modifies the source directory and may produce different
results from those that would be obtained by exporting an sdist
first". That's the bit that bothers me a lot - if we lose
build_directory, we lose the option for pip to request that a backend
produce a wheel "equivalent to producing an sdist first". And that
capability is something I think pip should require[2]. I don't
particularly care who specifies the actual directory (as per your
comment, I thought someone else wanted this, for caching or something)
but I *do* care about telling backends we don't want "different
results from those that would be obtained by exporting an sdist
first".

Paul

[1] The only example I can think of is flit where the user doesn't
have git installed. But then I'd expect that a wheel generated
directly in that case would be the same as if the user installed git
then did sdist->wheel, so I'd still consider that to be maintaining
the invariant I care about.
[2] I'm not likely to be the one implementing pip's PEP 517 support,
so I could be overridden on this, of course.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

Thomas,

I did not receive the last message.

On Aug 24, 2017 9:31 AM, "Thomas Kluyver"  wrote:

> On Thu, Aug 24, 2017, at 03:29 PM, xoviat wrote:
>
> I understand that, but what I disagree with is modifying build_wheel to
> make up for a lack of consensus on editable installs.
>
>
> The reasoning for the build_directory parameter was not based around
> editable installs. This is the confusion I was referring to about the
> multiple meanings of 'out-of-tree builds'.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Thomas Kluyver]

On Thu, Aug 24, 2017, at 03:29 PM, xoviat wrote:
> I understand that, but what I disagree with is modifying build_wheel
> to make up for a lack of consensus on editable installs.
The reasoning for the build_directory parameter was not based around
editable installs. This is the confusion I was referring to about the
multiple meanings of 'out-of-tree builds'.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

I understand that, but what I disagree with is modifying build_wheel to
make up for a lack of consensus on editable installs.

On Aug 24, 2017 9:27 AM, "Thomas Kluyver"  wrote:

> On Thu, Aug 24, 2017, at 03:05 PM, xoviat wrote:
>
> During development, you want to use an editable install, which does modify
> the source tree.
>
>
> I think it was before you joined the discussion, but editable installs
> came up before and have deliberately been left out of PEP 517 because of
> the complexities. I am very keen that we do not go back into that
> discussion.
>
> I particularly want to hear from Nick, Daniel, Donald and Paul on their
> understanding of the build_directory parameter. There were reasons why it
> got added, and even if those reasons no longer seem so convincing to me,
> it's only going to be removed again if we can reach a consensus on working
> without it.
>
> Thomas
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Thomas Kluyver]

On Thu, Aug 24, 2017, at 03:05 PM, xoviat wrote:
>  During development, you want to use an editable install, which does
>  modify the source tree.
I think it was before you joined the discussion, but editable installs
came up before and have deliberately been left out of PEP 517 because of
the complexities. I am very keen that we do not go back into that
discussion.
I particularly want to hear from Nick, Daniel, Donald and Paul on their
understanding of the build_directory parameter. There were reasons why
it got added, and even if those reasons no longer seem so convincing to
me, it's only going to be removed again if we can reach a consensus on
working without it.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[xoviat]

WRT the return value, I don't really care. But I am strongly in favor of
removing the build_dir option as I think that the entire premise is wrong.
During development, you want to use an editable install, which does modify
the source tree. And if you're just installing, then you don't care about
cache. And there is also nothing preventing pip from caching the temporary
directory, which may contain build artifacts.

On Aug 24, 2017 8:41 AM, "xoviat"  wrote:

> -- Forwarded message --
> From: "xoviat" 
> Date: Aug 24, 2017 8:39 AM
> Subject: Re: [Distutils] PEP 517 again
> To: "Thomas Kluyver" 
> Cc:
>
> That's actually the general argument against exceptions and why golang
> doesn't have them. I have not seen notimplemented used in the wild ever
> though.
>
> Also, regarding the build_dir argument, it won't be used in pip. The pip
> devs may have said that but it's too much work to support now and most of
> them seem to have a shortage of time. Pip will be copying to a temporary
> directory for a long time.
>
> On Aug 24, 2017 8:28 AM, "Thomas Kluyver"  wrote:
>
>> On Thu, Aug 24, 2017, at 02:21 PM, xoviat wrote:
>>
>> I mean is this golang or Python? In Python, you raise notimplementederror.
>>
>>
>> But there's a NotImplemented singleton in Python as well. The argument
>> for using a return value is that the hook code has to deliberately return
>> that, whereas a NotImplementedError could bubble up from some internal
>> call, in which case it should really be registered as an error.
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Fwd: Re: PEP 517 again

[Leonardo Rochael Almeida]

On 24 August 2017 at 10:41, xoviat  wrote:

> On Aug 24, 2017 8:28 AM, "Thomas Kluyver"  wrote:
>
>> On Thu, Aug 24, 2017, at 02:21 PM, xoviat wrote:
>>
>> I mean is this golang or Python? In Python, you raise notimplementederror.
>>
>>
>> But there's a NotImplemented singleton in Python as well. The argument
>> for using a return value is that the hook code has to deliberately return
>> that, whereas a NotImplementedError could bubble up from some internal
>> call, in which case it should really be registered as an error.
>>
>
>
> That's actually the general argument against exceptions and why golang
> doesn't have them. I have not seen notimplemented used in the wild ever
> though.
>

https://github.com/python/cpython/blob/6f0eb93183519024cb360162bdd81b9faec97ba6/Lib/fractions.py#L382

So, yes, it's used in the wild, by stdlib itself no less.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig