[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Nick Coghlan]

On Tue., 2 Oct. 2018, 4:05 pm Chris Jerdonek, 
wrote:

>
> In general, I think it's better to be more open about long-term strategies
> within an organization and who is working on what and toward what goals,
> etc. That way people can make better decisions about what to work on and
> how to spend their time.
>

Yeah, I'd agree with that, and the fact I wasn't making the time to better
facilitate that kind of information sharing is one of the reasons it seemed
like a good idea to step back from the way I had been doing things and
instead try to encourage more direct communication about what folks are
working on.

Cheers,
Nick.



>>>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/RJTHCVHG6IGSDLMALIPRKZNQ7UTBDZNV/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Chris Jerdonek]

On Mon, Oct 1, 2018 at 8:46 PM Nick Coghlan  wrote:

> On Wed., 26 Sep. 2018, 2:40 pm Chris Jerdonek, 
> wrote:
>
>> My advice to you (and which I've been following myself) would be for
>> you to break changes into small, tightly-focused PR's, and to have
>> only one outstanding at a time (or perhaps two if non-overlapping).
>> The main reason is that there aren't too many people reviewing and
>> merging changes (mainly just Pradyun in my case), so I don't want to
>> overwhelm them. I've been trying to follow the maxim "go slow to go
>> fast." Also, plan to expect / be okay with a fair of amount of
>> preliminary refactoring PR's before splitting something out might even
>> be possible. I.e. you will probably need a lot of "set up," especially
>> if it's a gnarlier area of the code base.
>>
>
> Note that the difficulty of doing this is a key part of why my own
> recommendation to the folks working on refactoring pipenv's internals was
> to handle the API design process reasonably independently of pip (at least
> initially): the point of providing a public Python level API is to address
> the cases that pip itself doesn't address, and while refactoring pip to use
> the same common set of libraries is important to reducing the long term
> collective maintenance burden, it's unlikely to be the quickest way of
> addressing the near term ecosystem level API need for projects like
> pip-tools, pipenv, and poetry.
>
> So if anyone's irritated by that approach, best to focus your irritation
> on me for providing that advice, not on the folks doing the work of
> breaking out pipenv's internals as reusable libraries that can hopefully
> eventually be shared with other projects (including pip) as their APIs
> stabilise. When they say that's the advice they received, it's an accurate
> statement, it's just that by participating directly in distutils-sig rather
> than proxying through me, they're now also hearing from folks that would
> have favoured a different approach :)
>

In general, I think it's better to be more open about long-term strategies
within an organization and who is working on what and toward what goals,
etc. That way people can make better decisions about what to work on and
how to spend their time.

For example, if I had known that the pipenv developers were working on
reusable libraries that were intended to later be used by pip, maybe I
would have chosen to help them instead of working on pip. This in turn
could have helped address the community's near-term needs more quickly, or
at least ensure that my own work is in concert with theirs (e.g. by
preparing for the longer term "convergence," as the piece you linked to
calls it). It's not a good feeling when people feel like they've been kept
in the dark. As I said, it can lead to duplicated work and the feeling that
one has been wasting their time by working on something that is later going
to be swapped out or replaced.

--Chris



> Cheers,
> Nick.
>
> P.S. At a larger meta-level, I find it's worth keeping in mind that there
> will be always be ebbs and flows of convergence and divergence in any large
> scale organisational system, and I really like
> http://theagilepirate.net/archives/1392 as a write-up of that in a
> corporate context.
>
> In an open source community context, I think the same phenomenon shows up
> as cycles of competition (different approaches to superficially common
> problems with no clear winner amongst them) and consolidation (improved
> understanding of the problem space resulting in clearer articulation of the
> design trade-offs involved, and the emergence of "obvious" default choices
> at particular points in the problem space).
>
>
>
>
>> --Chris
>>
>>
>> >
>> > Dan Ryan // pipenv maintainer
>> > gh: @techalchemy
>> >
>> >> On Sep 25, 2018, at 5:47 PM, Chris Jerdonek 
>> wrote:
>> >>
>> >>> On Tue, Sep 25, 2018 at 6:46 AM, Dan Ryan  wrote:
>> >>> As far as the pipenv roadmap, we are rewriting a good chunk of it but
>> it essentially the atomic things we do or would like to do with regards to
>> shared functionality at a high level:
>> >>>
>> >>> - parse user input/requirements
>> >>> - retrieve package metadata
>> >>> - resolve dependencies (not in pip right now obviously) — may involve
>> building wheels/sdists
>> >>> - download / build packages
>> >>> - install packages (into a virtualenv sometimes)
>> >>> - uninstall packages (from a virtualenv sometimes)
>> >>> - list packages etc (in a virtualenv sometimes)
>> >>>
>> >>> We know how these are accomplished in pip and we know how we are
>> handling them both in pipenv and in our alternative implementations, which
>> I believe still have some dependency on pip. For us the roadmap is
>> basically to figure out
>> >>>
>> >>> - where do the abstractions live
>> >>> - how do we start
>> >>> - ???
>> >>
>> >> Thanks, Dan. I think Donald answered this in an email much earlier in
>> >> this thread -- the one he sent on Sept. 19 beginning:
>> >>
>> >> """
>> >> My general 

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Dan Ryan]

Thanks Chris, these are excellent suggestions. Can you provide a reference to a 
PR you’re working or have worked on so we can jumpstart the process on our end?

Dan Ryan // pipenv maintainer
gh: @techalchemy

> On Sep 25, 2018, at 5:47 PM, Chris Jerdonek  wrote:
> 
>> On Tue, Sep 25, 2018 at 6:46 AM, Dan Ryan  wrote:
>> As far as the pipenv roadmap, we are rewriting a good chunk of it but it 
>> essentially the atomic things we do or would like to do with regards to 
>> shared functionality at a high level:
>> 
>> - parse user input/requirements
>> - retrieve package metadata
>> - resolve dependencies (not in pip right now obviously) — may involve 
>> building wheels/sdists
>> - download / build packages
>> - install packages (into a virtualenv sometimes)
>> - uninstall packages (from a virtualenv sometimes)
>> - list packages etc (in a virtualenv sometimes)
>> 
>> We know how these are accomplished in pip and we know how we are handling 
>> them both in pipenv and in our alternative implementations, which I believe 
>> still have some dependency on pip. For us the roadmap is basically to figure 
>> out
>> 
>> - where do the abstractions live
>> - how do we start
>> - ???
> 
> Thanks, Dan. I think Donald answered this in an email much earlier in
> this thread -- the one he sent on Sept. 19 beginning:
> 
> """
> My general recommendation if you want a Python
> implementation/interface for something pip does, is:
> - Open an issue on the pip repository to document your intent and to
> make sure that there is nobody there who is against having that
> functionality split out. ...
> ...
> """
> 
> In addition, to ensure you're taking the approach of splitting out
> parts of pip instead of creating yet another implementation separate
> from pip (leading to duplicated work as I said), I would recommend
> also doing the following:
> 
> File PR's in pip to refactor out various functionality that makes
> sense into stand-alone bits with their own tests (e.g. unit tests,
> etc). This would live in pip but be disentangled from the rest of pip
> from a code dependency perspective. It could even be in a separate
> "future libraries" subdirectory if the pip committers are okay with
> it. (Optional: since you're already vendoring pip, you could perhaps
> use this factored out code at your own risk, like you're already doing
> with some of pip's code base anyway.)
> 
> The above is something I started doing at a small scale -- not to
> break it out into a library, but for its own sake because it makes it
> easier to maintain and work on pip as a code base, fixing bugs, etc.
> 
> Some of the nice things about this approach are that--
> 
> 1. It can be done even before the steps that Donald outlined (like I'm
> doing), e.g. so that you don't need to wait for a PEP to be finished.
> And it could even help in the later creation of such PEPs.
> 2. By doing so, as a side effect of the PR process, it automatically
> causes communication with pip maintainers and works towards the
> "splitting out parts of pip" goal -- all without duplicating any work
> because both pip and pipenv developers are aware and collaborating on
> these changes.
> 3. It is incremental as you go, ensuring that it is compatible with
> pip and that pip is using it, etc.
> 4. It improves pip in the process (as well as pipenv, if you're still
> vendoring in some form). In other words, the work is useful from the
> beginning, whether or not it's ever broken out into a separately
> maintained library.
> 5. It is forward progress towards the goal of a separate library.
> 
> --Chris
> 
> 
> 
>> 
>> Dan Ryan // pipenv maintainer
>> gh: @techalchemy
>> 
 On Sep 25, 2018, at 8:11 AM, Paul Moore  wrote:
 
> On Tue, 25 Sep 2018 at 12:53, Chris Jerdonek  
> wrote:
> 
> On Tue, Sep 25, 2018 at 3:47 AM, Tzu-ping Chung  
> wrote:
> We are using pip internals for things pip wasn’t implemented for. 
> Specifically,
> Pipenv uses pip’s package-fetching functions to implement its 
> platform-agnostic
> resolver. pip does not have this, so there’s no functional overlap here. 
> Those
> utilities are used to build something that doesn’t exist in pip, so 
> there’s no
> duplicated efforts.
> 
> My recent focus on making sense of packaging implementations and 
> splitting out
> parts of pip is exactly to prevent potential duplicated efforts. If we 
> can’t use pip
> internals, let’s make things we want to use *not* internal!
 
 I don't see this practice of "splitting out parts of pip" actually
 being followed so far though, and I want to tell you how it's leading
 to duplicated efforts right now -- even for the PR's I happen to be
 working on today.
>>> 
>>> [...]
>>> 
 So by creating implementations of functions separate from pip instead
 of splitting them out of pip, it's leading to duplicated work for
 people working on pip's code base. A process of 

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Chris Jerdonek]

On Tue, Sep 25, 2018 at 6:30 PM, Dan Ryan  wrote:
> Thanks Chris, these are excellent suggestions. Can you provide a reference to 
> a PR you’re working or have worked on so we can jumpstart the process on our 
> end?


Hi Dan,

The PR's I've been submitting and working on aren't *primarily* of the
"split functionality out into stand-alone bits" variety. But there are
some. Two such recent PR's of mine that fit that category are--

1) one from July (merged) to split out and add
split_auth_from_netloc() to misc.py:
https://github.com/pypa/pip/pull/5657

2) the one I mentioned in an email yesterday and that I just submitted
(not yet merged), which splits out a function I called
make_vcs_requirement_url() and adds it to misc.py:
https://github.com/pypa/pip/pull/5810

My PR's have mostly been limited to the VCS part of the code:
refactoring, fixing bugs, enhancements, etc.

My advice to you (and which I've been following myself) would be for
you to break changes into small, tightly-focused PR's, and to have
only one outstanding at a time (or perhaps two if non-overlapping).
The main reason is that there aren't too many people reviewing and
merging changes (mainly just Pradyun in my case), so I don't want to
overwhelm them. I've been trying to follow the maxim "go slow to go
fast." Also, plan to expect / be okay with a fair of amount of
preliminary refactoring PR's before splitting something out might even
be possible. I.e. you will probably need a lot of "set up," especially
if it's a gnarlier area of the code base.

--Chris


>
> Dan Ryan // pipenv maintainer
> gh: @techalchemy
>
>> On Sep 25, 2018, at 5:47 PM, Chris Jerdonek  wrote:
>>
>>> On Tue, Sep 25, 2018 at 6:46 AM, Dan Ryan  wrote:
>>> As far as the pipenv roadmap, we are rewriting a good chunk of it but it 
>>> essentially the atomic things we do or would like to do with regards to 
>>> shared functionality at a high level:
>>>
>>> - parse user input/requirements
>>> - retrieve package metadata
>>> - resolve dependencies (not in pip right now obviously) — may involve 
>>> building wheels/sdists
>>> - download / build packages
>>> - install packages (into a virtualenv sometimes)
>>> - uninstall packages (from a virtualenv sometimes)
>>> - list packages etc (in a virtualenv sometimes)
>>>
>>> We know how these are accomplished in pip and we know how we are handling 
>>> them both in pipenv and in our alternative implementations, which I believe 
>>> still have some dependency on pip. For us the roadmap is basically to 
>>> figure out
>>>
>>> - where do the abstractions live
>>> - how do we start
>>> - ???
>>
>> Thanks, Dan. I think Donald answered this in an email much earlier in
>> this thread -- the one he sent on Sept. 19 beginning:
>>
>> """
>> My general recommendation if you want a Python
>> implementation/interface for something pip does, is:
>> - Open an issue on the pip repository to document your intent and to
>> make sure that there is nobody there who is against having that
>> functionality split out. ...
>> ...
>> """
>>
>> In addition, to ensure you're taking the approach of splitting out
>> parts of pip instead of creating yet another implementation separate
>> from pip (leading to duplicated work as I said), I would recommend
>> also doing the following:
>>
>> File PR's in pip to refactor out various functionality that makes
>> sense into stand-alone bits with their own tests (e.g. unit tests,
>> etc). This would live in pip but be disentangled from the rest of pip
>> from a code dependency perspective. It could even be in a separate
>> "future libraries" subdirectory if the pip committers are okay with
>> it. (Optional: since you're already vendoring pip, you could perhaps
>> use this factored out code at your own risk, like you're already doing
>> with some of pip's code base anyway.)
>>
>> The above is something I started doing at a small scale -- not to
>> break it out into a library, but for its own sake because it makes it
>> easier to maintain and work on pip as a code base, fixing bugs, etc.
>>
>> Some of the nice things about this approach are that--
>>
>> 1. It can be done even before the steps that Donald outlined (like I'm
>> doing), e.g. so that you don't need to wait for a PEP to be finished.
>> And it could even help in the later creation of such PEPs.
>> 2. By doing so, as a side effect of the PR process, it automatically
>> causes communication with pip maintainers and works towards the
>> "splitting out parts of pip" goal -- all without duplicating any work
>> because both pip and pipenv developers are aware and collaborating on
>> these changes.
>> 3. It is incremental as you go, ensuring that it is compatible with
>> pip and that pip is using it, etc.
>> 4. It improves pip in the process (as well as pipenv, if you're still
>> vendoring in some form). In other words, the work is useful from the
>> beginning, whether or not it's ever broken out into a separately
>> maintained library.
>> 5. It is forward progress 

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Paul Moore]

On Tue, 25 Sep 2018 at 22:09, Dan Ryan  wrote:

> To Chris’ broader point it is definitely duplicated effort and I am in full 
> agreement which is why I want to establish which code should be extracted and 
> generalized and where it should be maintained. But as Paul mentioned there 
> also is no PyPA strategy at play here. We aren’t trying to move into pip’s 
> space and vice versa AFAIU but I’m also not too sure how much trying to point 
> a finger at us here is helpful. You find it frustrating that we are working 
> in isolation, I find it frustrating that I keep being pushed to go away and 
> write libraries whenever I mention sharing functionality, which to me sounds 
> like you want me to work in isolation. Obviously there is some communication 
> breakdown— At the end of the day I just want to give users a consistent 
> experience. Hopefully this is a goal others share.

Yeah, miscommunication. It's not useful to dwell on mistakes made in
the past, so thanks for simply stating your position, and moving
forward. In the interests of doing the same I'll just say that I've
always *intended* the message to be "if you need pip's internal
functionality, pull it out into a library and write an API for it",
not that people should reimplement what we'd done, like you and Chris
say, that's wasted effort. If that didn't come across properly, that's
my mistake.

Donald said much the same, but gave more detail about the best way to
go about that process, and Chris has added further suggestions in the
message he just sent, all of which I agree with.

Paul
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/2OOZNN5U62QFULVMCD5R7HSV5MODSQXM/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Chris Jerdonek]

On Tue, Sep 25, 2018 at 6:46 AM, Dan Ryan  wrote:
> As far as the pipenv roadmap, we are rewriting a good chunk of it but it 
> essentially the atomic things we do or would like to do with regards to 
> shared functionality at a high level:
>
> - parse user input/requirements
> - retrieve package metadata
> - resolve dependencies (not in pip right now obviously) — may involve 
> building wheels/sdists
> - download / build packages
> - install packages (into a virtualenv sometimes)
> - uninstall packages (from a virtualenv sometimes)
> - list packages etc (in a virtualenv sometimes)
>
> We know how these are accomplished in pip and we know how we are handling 
> them both in pipenv and in our alternative implementations, which I believe 
> still have some dependency on pip. For us the roadmap is basically to figure 
> out
>
> - where do the abstractions live
> - how do we start
> - ???

Thanks, Dan. I think Donald answered this in an email much earlier in
this thread -- the one he sent on Sept. 19 beginning:

"""
My general recommendation if you want a Python
implementation/interface for something pip does, is:
- Open an issue on the pip repository to document your intent and to
make sure that there is nobody there who is against having that
functionality split out. ...
...
"""

In addition, to ensure you're taking the approach of splitting out
parts of pip instead of creating yet another implementation separate
from pip (leading to duplicated work as I said), I would recommend
also doing the following:

File PR's in pip to refactor out various functionality that makes
sense into stand-alone bits with their own tests (e.g. unit tests,
etc). This would live in pip but be disentangled from the rest of pip
from a code dependency perspective. It could even be in a separate
"future libraries" subdirectory if the pip committers are okay with
it. (Optional: since you're already vendoring pip, you could perhaps
use this factored out code at your own risk, like you're already doing
with some of pip's code base anyway.)

The above is something I started doing at a small scale -- not to
break it out into a library, but for its own sake because it makes it
easier to maintain and work on pip as a code base, fixing bugs, etc.

Some of the nice things about this approach are that--

1. It can be done even before the steps that Donald outlined (like I'm
doing), e.g. so that you don't need to wait for a PEP to be finished.
And it could even help in the later creation of such PEPs.
2. By doing so, as a side effect of the PR process, it automatically
causes communication with pip maintainers and works towards the
"splitting out parts of pip" goal -- all without duplicating any work
because both pip and pipenv developers are aware and collaborating on
these changes.
3. It is incremental as you go, ensuring that it is compatible with
pip and that pip is using it, etc.
4. It improves pip in the process (as well as pipenv, if you're still
vendoring in some form). In other words, the work is useful from the
beginning, whether or not it's ever broken out into a separately
maintained library.
5. It is forward progress towards the goal of a separate library.

--Chris



>
> Dan Ryan // pipenv maintainer
> gh: @techalchemy
>
>> On Sep 25, 2018, at 8:11 AM, Paul Moore  wrote:
>>
>>> On Tue, 25 Sep 2018 at 12:53, Chris Jerdonek  
>>> wrote:
>>>
 On Tue, Sep 25, 2018 at 3:47 AM, Tzu-ping Chung  wrote:
 We are using pip internals for things pip wasn’t implemented for. 
 Specifically,
 Pipenv uses pip’s package-fetching functions to implement its 
 platform-agnostic
 resolver. pip does not have this, so there’s no functional overlap here. 
 Those
 utilities are used to build something that doesn’t exist in pip, so 
 there’s no
 duplicated efforts.

 My recent focus on making sense of packaging implementations and splitting 
 out
 parts of pip is exactly to prevent potential duplicated efforts. If we 
 can’t use pip
 internals, let’s make things we want to use *not* internal!
>>>
>>> I don't see this practice of "splitting out parts of pip" actually
>>> being followed so far though, and I want to tell you how it's leading
>>> to duplicated efforts right now -- even for the PR's I happen to be
>>> working on today.
>>
>> [...]
>>
>>> So by creating implementations of functions separate from pip instead
>>> of splitting them out of pip, it's leading to duplicated work for
>>> people working on pip's code base. A process of splitting something
>>> out of pip would I think involve a process more like what I'm
>>> following -- namely to refactor functionality in pip *first* so it can
>>> be used separately, and then pull that out so you only have one
>>> implementation at any point in time. Otherwise, you wind up creating
>>> two versions of similar functionality that not only need to be created
>>> twice, but also later reconciled if the plan is merge them back
>>> together.
>>
>> 

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Dan Ryan]

FYI pipenv have been vendoring pip since pretty early in the projects 
existence, since before 10 was released. At no point was I in touch to say that 
was a major issue though because we use the api precisely as you describe, *at 
our own risk*. Every time I’ve been in touch it’s been to ask how to go about 
starting the conversation around sharing elements of a codebase, to which the 
answer has universally been ‘don’t use our internals, go write a library if you 
want’. So as a result we’ve mostly gone away and kept pip vendored as we 
started some libraries. 

The internal changes didn’t do much as far as I’m concerned, and as we knew the 
risks and had a vendored copy we were able to upgrade on our own clock. We use 
this copy only to fetch things from the index for generating lockfiles. Also 
there was significant publicity saying that people shouldn’t use internals and 
it seemed like it should honestly not be on you to worry about — that’s the 
whole point of vendoring it. I don’t think any of us saw a point in trying to 
force that problem on pip given the internals changes. And pipenv was not a 
part of the pypa at the start of the project, so we certainly _were_ working in 
isolation at that time. 

To Chris’ broader point it is definitely duplicated effort and I am in full 
agreement which is why I want to establish which code should be extracted and 
generalized and where it should be maintained. But as Paul mentioned there also 
is no PyPA strategy at play here. We aren’t trying to move into pip’s space and 
vice versa AFAIU but I’m also not too sure how much trying to point a finger at 
us here is helpful. You find it frustrating that we are working in isolation, I 
find it frustrating that I keep being pushed to go away and write libraries 
whenever I mention sharing functionality, which to me sounds like you want me 
to work in isolation. Obviously there is some communication breakdown— At the 
end of the day I just want to give users a consistent experience. Hopefully 
this is a goal others share. 

As far as the pipenv roadmap, we are rewriting a good chunk of it but it 
essentially the atomic things we do or would like to do with regards to shared 
functionality at a high level:

- parse user input/requirements
- retrieve package metadata
- resolve dependencies (not in pip right now obviously) — may involve building 
wheels/sdists
- download / build packages
- install packages (into a virtualenv sometimes)
- uninstall packages (from a virtualenv sometimes)
- list packages etc (in a virtualenv sometimes)

We know how these are accomplished in pip and we know how we are handling them 
both in pipenv and in our alternative implementations, which I believe still 
have some dependency on pip. For us the roadmap is basically to figure out

- where do the abstractions live
- how do we start
- ???

Dan Ryan // pipenv maintainer
gh: @techalchemy

> On Sep 25, 2018, at 8:11 AM, Paul Moore  wrote:
> 
>> On Tue, 25 Sep 2018 at 12:53, Chris Jerdonek  
>> wrote:
>> 
>>> On Tue, Sep 25, 2018 at 3:47 AM, Tzu-ping Chung  wrote:
>>> We are using pip internals for things pip wasn’t implemented for. 
>>> Specifically,
>>> Pipenv uses pip’s package-fetching functions to implement its 
>>> platform-agnostic
>>> resolver. pip does not have this, so there’s no functional overlap here. 
>>> Those
>>> utilities are used to build something that doesn’t exist in pip, so there’s 
>>> no
>>> duplicated efforts.
>>> 
>>> My recent focus on making sense of packaging implementations and splitting 
>>> out
>>> parts of pip is exactly to prevent potential duplicated efforts. If we 
>>> can’t use pip
>>> internals, let’s make things we want to use *not* internal!
>> 
>> I don't see this practice of "splitting out parts of pip" actually
>> being followed so far though, and I want to tell you how it's leading
>> to duplicated efforts right now -- even for the PR's I happen to be
>> working on today.
> 
> [...]
> 
>> So by creating implementations of functions separate from pip instead
>> of splitting them out of pip, it's leading to duplicated work for
>> people working on pip's code base. A process of splitting something
>> out of pip would I think involve a process more like what I'm
>> following -- namely to refactor functionality in pip *first* so it can
>> be used separately, and then pull that out so you only have one
>> implementation at any point in time. Otherwise, you wind up creating
>> two versions of similar functionality that not only need to be created
>> twice, but also later reconciled if the plan is merge them back
>> together.
> 
> This is a very good point, and thanks for the concrete example. For
> the record, I'm also a little annoyed that pipenv have been doing all
> this work in isolation. There's been very little communication between
> the pip and pipenv projects, and I think that's contributed to the
> problem. I, for one, wasn't even aware for a long time that they were
> embedding their own 

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Paul Moore]

On Tue, 25 Sep 2018 at 12:53, Chris Jerdonek  wrote:
>
> On Tue, Sep 25, 2018 at 3:47 AM, Tzu-ping Chung  wrote:
> > We are using pip internals for things pip wasn’t implemented for. 
> > Specifically,
> > Pipenv uses pip’s package-fetching functions to implement its 
> > platform-agnostic
> > resolver. pip does not have this, so there’s no functional overlap here. 
> > Those
> > utilities are used to build something that doesn’t exist in pip, so there’s 
> > no
> > duplicated efforts.
> >
> > My recent focus on making sense of packaging implementations and splitting 
> > out
> > parts of pip is exactly to prevent potential duplicated efforts. If we 
> > can’t use pip
> > internals, let’s make things we want to use *not* internal!
>
> I don't see this practice of "splitting out parts of pip" actually
> being followed so far though, and I want to tell you how it's leading
> to duplicated efforts right now -- even for the PR's I happen to be
> working on today.

[...]

> So by creating implementations of functions separate from pip instead
> of splitting them out of pip, it's leading to duplicated work for
> people working on pip's code base. A process of splitting something
> out of pip would I think involve a process more like what I'm
> following -- namely to refactor functionality in pip *first* so it can
> be used separately, and then pull that out so you only have one
> implementation at any point in time. Otherwise, you wind up creating
> two versions of similar functionality that not only need to be created
> twice, but also later reconciled if the plan is merge them back
> together.

This is a very good point, and thanks for the concrete example. For
the record, I'm also a little annoyed that pipenv have been doing all
this work in isolation. There's been very little communication between
the pip and pipenv projects, and I think that's contributed to the
problem. I, for one, wasn't even aware for a long time that they were
embedding their own copy of pip. When we made pip's internals in
private in pip 10, we got essentially no feedback from anyone saying
that it would be a problem, and finding out that it had a sufficiently
major impact on pipenv that they had to embed the old version of pip
*after* the release was (to put it politely) suboptimal :-(

Having said all this, this thread is the start of an attempt to work
better together and I think we should be glad that it's happening and
try to make it work. But I don't think we've got off to a particularly
good start and it will need work. In particular, I'd like to see an
attempt from *both* projects to clarify our goals - I've been speaking
solely for myself so far, it would be good to get the other pip
developers' views and set out a sort of roadmap, and for pipenv to do
the same.

Paul
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/OGXUNDRBLTNPOWLLFBVF45SW4SRGYSPP/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Chris Jerdonek]

On Tue, Sep 25, 2018 at 3:47 AM, Tzu-ping Chung  wrote:
> We are using pip internals for things pip wasn’t implemented for. 
> Specifically,
> Pipenv uses pip’s package-fetching functions to implement its 
> platform-agnostic
> resolver. pip does not have this, so there’s no functional overlap here. Those
> utilities are used to build something that doesn’t exist in pip, so there’s no
> duplicated efforts.
>
> My recent focus on making sense of packaging implementations and splitting out
> parts of pip is exactly to prevent potential duplicated efforts. If we can’t 
> use pip
> internals, let’s make things we want to use *not* internal!

I don't see this practice of "splitting out parts of pip" actually
being followed so far though, and I want to tell you how it's leading
to duplicated efforts right now -- even for the PR's I happen to be
working on today.

Earlier in this thread, Dan said the pipenv maintainers are
reimplementing certain pip functionality rather than splitting it out:
"we basically had to start by calling pip directly, then slowly
reimplement each aspect of the underlying logic using various elements
in distlib/setuptools or rebuilding those." And then he listed seven
libraries they are working on that contain code like this.

Today I happened to file a simple PR to pip to continue DRYing up
pip's VCS-related code base and making it more testable, etc:
https://github.com/pypa/pip/pull/5810
My PR adds a function called `make_vcs_requirement_url()` and
refactors various parts of pip to use it:
https://github.com/pypa/pip/pull/5810/files#diff-c9e9f4633cef78355bd4a581febff330R859

However, when I look at requirementslib (one of the libraries Dan
mentioned), I see they have already reimplemented the function I added
in my PR. In requirementslib, it's called `build_vcs_link()`:
https://github.com/sarugaku/requirementslib/blob/d457054d969fcb5e712c8f74f9ddac182335443b/src/requirementslib/models/utils.py#L79-L95

So instead of splitting it out of pip and modifying pip to use that,
they created a new implementation separate from pip. And now I am
redoing the implementation a second time. Other VCS-related functions
in that same requirementslib file (e.g. "add_ssh_scheme_to_git_uri()")
overlap with VCS code in pip that I've been doing similar work on.

So by creating implementations of functions separate from pip instead
of splitting them out of pip, it's leading to duplicated work for
people working on pip's code base. A process of splitting something
out of pip would I think involve a process more like what I'm
following -- namely to refactor functionality in pip *first* so it can
be used separately, and then pull that out so you only have one
implementation at any point in time. Otherwise, you wind up creating
two versions of similar functionality that not only need to be created
twice, but also later reconciled if the plan is merge them back
together.

--Chris


>
> TP
>
>
>> On 25/9, 2018, at 18:38, Chris Jerdonek  wrote:
>>
>> On Tue, Sep 25, 2018 at 3:21 AM, Nick Coghlan  wrote:
>>> On Tue, 25 Sep 2018 at 19:48, Chris Jerdonek  
>>> wrote:
 What I'm trying to gauge is, if the plan is for pipenv not to depend
 on pip, and pipenv has strictly greater functionality than pip, then
 what purpose will PyPA have in continuing to develop pip in addition
 to pipenv?
>>>
>>> That's not the plan, as all of pip's features for actually
>>> installing/uninstalling packages, and for introspecting the *as built*
>>> package environment, aren't things where pipenv's needs diverge from
>>> pip's.
>>
>> That's not what Tzu-ping said though. In an earlier email, he said,
>> "If we can’t use pip internals, then yes, the plan is to not depend on
>> pip."
>>
>> --Chris
>>
>>
>>>
>>> Where their needs diverge is at the dependency resolver level, as
>>> pipenv needs to be able to generate a lock file for an arbitrary
>>> target environment that may not match the currently running Python
>>> interpreter *without* necessarily installing those packages anywhere
>>> (although it may need to build wheels to get the dependency listing),
>>> whereas pip has the more concrete task of "get theses packages and
>>> their dependencies installed into the currently active environment".
>>>
>>> If it helps, think of pipenv relating to pip in much the same way as
>>> pip-tools (pip-compile/pip-sync) relates to pip, just with Pipfile and
>>> Pipfile.lock instead of requirements.in and requirements.txt.
>>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/ERR2YSW3VWYAQCQ4BHPRWOGTDVICVYUH/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Tzu-ping Chung]

We are using pip internals for things pip wasn’t implemented for. Specifically,
Pipenv uses pip’s package-fetching functions to implement its platform-agnostic
resolver. pip does not have this, so there’s no functional overlap here. Those
utilities are used to build something that doesn’t exist in pip, so there’s no
duplicated efforts.

My recent focus on making sense of packaging implementations and splitting out
parts of pip is exactly to prevent potential duplicated efforts. If we can’t 
use pip
internals, let’s make things we want to use *not* internal!

TP


> On 25/9, 2018, at 18:38, Chris Jerdonek  wrote:
> 
> On Tue, Sep 25, 2018 at 3:21 AM, Nick Coghlan  wrote:
>> On Tue, 25 Sep 2018 at 19:48, Chris Jerdonek  
>> wrote:
>>> What I'm trying to gauge is, if the plan is for pipenv not to depend
>>> on pip, and pipenv has strictly greater functionality than pip, then
>>> what purpose will PyPA have in continuing to develop pip in addition
>>> to pipenv?
>> 
>> That's not the plan, as all of pip's features for actually
>> installing/uninstalling packages, and for introspecting the *as built*
>> package environment, aren't things where pipenv's needs diverge from
>> pip's.
> 
> That's not what Tzu-ping said though. In an earlier email, he said,
> "If we can’t use pip internals, then yes, the plan is to not depend on
> pip."
> 
> --Chris
> 
> 
>> 
>> Where their needs diverge is at the dependency resolver level, as
>> pipenv needs to be able to generate a lock file for an arbitrary
>> target environment that may not match the currently running Python
>> interpreter *without* necessarily installing those packages anywhere
>> (although it may need to build wheels to get the dependency listing),
>> whereas pip has the more concrete task of "get theses packages and
>> their dependencies installed into the currently active environment".
>> 
>> If it helps, think of pipenv relating to pip in much the same way as
>> pip-tools (pip-compile/pip-sync) relates to pip, just with Pipfile and
>> Pipfile.lock instead of requirements.in and requirements.txt.
>> 
>> Cheers,
>> Nick.
>> 
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/LYY4SED3GISTJPNURZKAM45FZMYAAVKF/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Paul Moore]

On Tue, 25 Sep 2018 at 10:46, Chris Jerdonek  wrote:
>
> On Tue, Sep 25, 2018 at 1:25 AM, Tzu-ping Chung  wrote:
> > Pipenv wraps pip usages inside a virtual environment, so pip is always
> > available via “pipenv run pip”,
> > so in a sense Pipenv “supports” everything pip does. But as far as things
> > Pipenv actually has wrapper
> > commands for, it only tries to be pip’s functional superset in “install” and
> > “uninstall”; everything else
> > is out of scope.
>
> I was asking more in terms of useful functionality in the broader
> sense, rather than literal pip commands. Is any of pip's functionality
> in the broader sense out of scope?
>
> For example, I'm guessing that the information in commands like pip
> freeze, hash, list, and show might already be covered in a different
> way by pipenv analogs. For reference, I'm including the list of
> commands outputted by "pip help" below.
>
> What I'm trying to gauge is, if the plan is for pipenv not to depend
> on pip, and pipenv has strictly greater functionality than pip, then
> what purpose will PyPA have in continuing to develop pip in addition
> to pipenv? This is what I was referring to in my earlier email when I
> said it looks like it could "split the user and maintainer base in
> two."
>
> This question is personal to me because I currently spend some time
> improving and contributing to pip. But I may want to reevaluate that
> if another, more active PyPA project is in the process of duplicating
> that work or making pip superfluous.
>
> It seems like duplication of effort within an organization should be
> discouraged when there is limited person power to begin with.

Speaking as a member of the pip development team, who has invested a
lot of time in the development of pip (and still does), I can say with
100% certainty that there's no intention from me that pip is going to
be "made superfluous". As a member of PyPA, I'll say that PyPA is
basically a loose confederation of projects, and there's no real
"overall PyPA plan" to consider here. There's likely a perception
problem here about the role of the PyPA that we should try to address,
but that's somewhat a separate problem.

There's also something of a perception that pipenv and pip are somehow
"competitors". There are a couple of aspects to that:

1. Is pipenv *actually* competing with pip? I don't think so, and I
don't think that pipenv do either, but personally, I'd like to see a
bit more clarity in the pipenv docs of how it addresses a different
use case than pip. There was some unfortunate (again, IMO)
miscommunication earlier on that they need to address, but again, I
think it's just that - miscomunication, not reality.
2. Regardless, is it even a problem for 2 competing projects to be
under the PyPA banner? I don't think it is (see "loose confederation"
above) but again, that's related to clarifying the role of the PyPA.

As far as duplication of effort is concerned, one of the points of
this conversation is to ensure that the work that the pipenv
developers are doing can also be used by pip - that's very much
because we *don't* want pip to get relegated to a legacy role. But
neither pip, nor (to my knowledge) pipenv support use as a library -
so code sharing *has* to be done by identifying useful code and moving
it out into supported libraries. Maybe, in the long term future, we'll
end up with the PyPA managing a set of "packaging interop libraries"
and a number of tools like pip, pipenv and maybe others, that layer a
UI on top of that functionality. But there will still be a pip
project, and it will still be developing and growing.

tl; dr; Pip needs all the help you can offer and that's not going to
change as a result of pipenv. Only you can choose what you're
interested in working on, but don't make that decision based on any
misguided idea that someone's making your work on pip obsolete.

Paul
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/PGYZJKP6CW33HPVJGZYEJSSJC5DB5SH2/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Chris Jerdonek]

On Tue, Sep 25, 2018 at 3:21 AM, Nick Coghlan  wrote:
> On Tue, 25 Sep 2018 at 19:48, Chris Jerdonek  wrote:
>> What I'm trying to gauge is, if the plan is for pipenv not to depend
>> on pip, and pipenv has strictly greater functionality than pip, then
>> what purpose will PyPA have in continuing to develop pip in addition
>> to pipenv?
>
> That's not the plan, as all of pip's features for actually
> installing/uninstalling packages, and for introspecting the *as built*
> package environment, aren't things where pipenv's needs diverge from
> pip's.

That's not what Tzu-ping said though. In an earlier email, he said,
"If we can’t use pip internals, then yes, the plan is to not depend on
pip."

--Chris


>
> Where their needs diverge is at the dependency resolver level, as
> pipenv needs to be able to generate a lock file for an arbitrary
> target environment that may not match the currently running Python
> interpreter *without* necessarily installing those packages anywhere
> (although it may need to build wheels to get the dependency listing),
> whereas pip has the more concrete task of "get theses packages and
> their dependencies installed into the currently active environment".
>
> If it helps, think of pipenv relating to pip in much the same way as
> pip-tools (pip-compile/pip-sync) relates to pip, just with Pipfile and
> Pipfile.lock instead of requirements.in and requirements.txt.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/OITWLEVD2RJOG6N2F7ADB7JPCDNV6HK5/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Nick Coghlan]

On Tue, 25 Sep 2018 at 19:48, Chris Jerdonek  wrote:
> What I'm trying to gauge is, if the plan is for pipenv not to depend
> on pip, and pipenv has strictly greater functionality than pip, then
> what purpose will PyPA have in continuing to develop pip in addition
> to pipenv?

That's not the plan, as all of pip's features for actually
installing/uninstalling packages, and for introspecting the *as built*
package environment, aren't things where pipenv's needs diverge from
pip's.

Where their needs diverge is at the dependency resolver level, as
pipenv needs to be able to generate a lock file for an arbitrary
target environment that may not match the currently running Python
interpreter *without* necessarily installing those packages anywhere
(although it may need to build wheels to get the dependency listing),
whereas pip has the more concrete task of "get theses packages and
their dependencies installed into the currently active environment".

If it helps, think of pipenv relating to pip in much the same way as
pip-tools (pip-compile/pip-sync) relates to pip, just with Pipfile and
Pipfile.lock instead of requirements.in and requirements.txt.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/HJ35YZEWMMZ6F6GN4WY5SCN35OXPE2CL/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Chris Jerdonek]

On Tue, Sep 25, 2018 at 1:25 AM, Tzu-ping Chung  wrote:
> Pipenv wraps pip usages inside a virtual environment, so pip is always
> available via “pipenv run pip”,
> so in a sense Pipenv “supports” everything pip does. But as far as things
> Pipenv actually has wrapper
> commands for, it only tries to be pip’s functional superset in “install” and
> “uninstall”; everything else
> is out of scope.

I was asking more in terms of useful functionality in the broader
sense, rather than literal pip commands. Is any of pip's functionality
in the broader sense out of scope?

For example, I'm guessing that the information in commands like pip
freeze, hash, list, and show might already be covered in a different
way by pipenv analogs. For reference, I'm including the list of
commands outputted by "pip help" below.

What I'm trying to gauge is, if the plan is for pipenv not to depend
on pip, and pipenv has strictly greater functionality than pip, then
what purpose will PyPA have in continuing to develop pip in addition
to pipenv? This is what I was referring to in my earlier email when I
said it looks like it could "split the user and maintainer base in
two."

This question is personal to me because I currently spend some time
improving and contributing to pip. But I may want to reevaluate that
if another, more active PyPA project is in the process of duplicating
that work or making pip superfluous.

It seems like duplication of effort within an organization should be
discouraged when there is limited person power to begin with.

--Chris


$ pip help
...
Commands:
  install - Install packages.
  download - Download packages.
  uninstall - Uninstall packages.
  freeze - Output installed packages in requirements format.
  list - List installed packages.
  show - Show information about installed packages.
  check - Verify installed packages have compatible dependencies.
  config - Manage local and global configuration.
  search - Search PyPI for packages.
  wheel - Build wheels from your requirements.
  hash - Compute hashes of package archives.
  completion - A helper command used for command completion.
  help - Show help for commands.




>
> There are constantly people advocating adding more, or having confusion
> between similarly-named
> commands (e.g. check), but that’s another issue…
>
> TP
>
>
> On 25/9, 2018, at 08:22, Chris Jerdonek  wrote:
>
> On Fri, Sep 21, 2018 at 5:14 AM, Paul Moore  wrote:
>
> If I'm misunderstanding the relationship between pip and pipenv, or
> between pipenv and pipfile, I'm happy to be corrected. But can I
> suggest that the best way to do so would be to amend the project pages
> that are giving me the impressions I have above, and pointing me at
> the corrected versions? That way, we can make sure that any
> misinformation is corrected at source...
>
>
> [This question isn't directed at Paul, even though it's in reply to his
> email.]
>
> Thanks for the good discussion, all. To clarify the overlap in
> functionality between pip and pipenv further, will pipenv be a strict
> superset in terms of what it can do?
>
> Tzu-ping said earlier that users expect pipenv to behave the same as
> pip, so I'm wondering if there are any areas of functionality that pip
> covers that pipenv won't be able to (or that doesn't plan to).
>
> --Chris
>
>
>
> Paul
>
> PS Full disclosure - I've tried to use pipenv in a couple of local
> projects, because of the hype about it being the "great new thing" and
> found it basically of no use for my requirements/workflow. So I may
> have a biased view of either pipenv, or how it's being presented. I'm
> trying to be objective in the above, but my bias may have slipped
> through.
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/SR3FAMLZM646GT3IPYFILD47KMRWOALD/
>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/Y75UQYWNCCPMM77UVQ7NTDMSIBRIO27W/
>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/U26XGX2KFTAFUITIRAFDYJCFWR4UQ455/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Tzu-ping Chung]

Pipenv wraps pip usages inside a virtual environment, so pip is always 
available via “pipenv run pip”,
so in a sense Pipenv “supports” everything pip does. But as far as things 
Pipenv actually has wrapper
commands for, it only tries to be pip’s functional superset in “install” and 
“uninstall”; everything else
is out of scope.

There are constantly people advocating adding more, or having confusion between 
similarly-named
commands (e.g. check), but that’s another issue…

TP


> On 25/9, 2018, at 08:22, Chris Jerdonek  wrote:
> 
> On Fri, Sep 21, 2018 at 5:14 AM, Paul Moore  > wrote:
>> If I'm misunderstanding the relationship between pip and pipenv, or
>> between pipenv and pipfile, I'm happy to be corrected. But can I
>> suggest that the best way to do so would be to amend the project pages
>> that are giving me the impressions I have above, and pointing me at
>> the corrected versions? That way, we can make sure that any
>> misinformation is corrected at source...
> 
> [This question isn't directed at Paul, even though it's in reply to his 
> email.]
> 
> Thanks for the good discussion, all. To clarify the overlap in
> functionality between pip and pipenv further, will pipenv be a strict
> superset in terms of what it can do?
> 
> Tzu-ping said earlier that users expect pipenv to behave the same as
> pip, so I'm wondering if there are any areas of functionality that pip
> covers that pipenv won't be able to (or that doesn't plan to).
> 
> --Chris
> 
> 
>> 
>> Paul
>> 
>> PS Full disclosure - I've tried to use pipenv in a couple of local
>> projects, because of the hype about it being the "great new thing" and
>> found it basically of no use for my requirements/workflow. So I may
>> have a biased view of either pipenv, or how it's being presented. I'm
>> trying to be objective in the above, but my bias may have slipped
>> through.
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
>> Message archived at 
>> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/SR3FAMLZM646GT3IPYFILD47KMRWOALD/
> --
> Distutils-SIG mailing list -- distutils-sig@python.org 
> 
> To unsubscribe send an email to distutils-sig-le...@python.org 
> 
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ 
> 
> Message archived at 
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/Y75UQYWNCCPMM77UVQ7NTDMSIBRIO27W/
>  
> 
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/Q4IZBOMZLZZ7SWQHHHYQDDXNFVVRILEL/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Chris Jerdonek]

On Fri, Sep 21, 2018 at 5:14 AM, Paul Moore  wrote:
> If I'm misunderstanding the relationship between pip and pipenv, or
> between pipenv and pipfile, I'm happy to be corrected. But can I
> suggest that the best way to do so would be to amend the project pages
> that are giving me the impressions I have above, and pointing me at
> the corrected versions? That way, we can make sure that any
> misinformation is corrected at source...

[This question isn't directed at Paul, even though it's in reply to his email.]

Thanks for the good discussion, all. To clarify the overlap in
functionality between pip and pipenv further, will pipenv be a strict
superset in terms of what it can do?

Tzu-ping said earlier that users expect pipenv to behave the same as
pip, so I'm wondering if there are any areas of functionality that pip
covers that pipenv won't be able to (or that doesn't plan to).

--Chris


>
> Paul
>
> PS Full disclosure - I've tried to use pipenv in a couple of local
> projects, because of the hype about it being the "great new thing" and
> found it basically of no use for my requirements/workflow. So I may
> have a biased view of either pipenv, or how it's being presented. I'm
> trying to be objective in the above, but my bias may have slipped
> through.
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at 
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/SR3FAMLZM646GT3IPYFILD47KMRWOALD/
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/Y75UQYWNCCPMM77UVQ7NTDMSIBRIO27W/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Nick Coghlan]

On Fri, 21 Sep 2018 at 23:10, Donald Stufft  wrote:
> So I should probably be explicit that this was basically an idea I had, that 
> Kenneth rolled with and originally tried to implement as a PR to pip but when 
> pip’s code base was too, gnarly to get it in directly, he made pipenv to wrap 
> pip to implement it, and since then pipenv has grown it’s own features and 
> such and is now an independent tool. Maybe it no longer makes sense for 
> Pipfile to be part of pip, maybe the idea is still good, but we need to do it 
> in a different way, I dunno. My thoughts aren’t really fully formulated on it 
> beyond “this is a thing I’d like to do, or at least explore doing” because 
> I’ve been focused on other things.
>
> The original intent *was* to replace requirements.txt, because the 
> requirements.txt format is not super great and is kind of unwieldy to work 
> with. It conflates the idea of a lock file with the idea of the things I 
> actually want to install, which makes it harder to work with than needs to 
> be. It’s fine-ish for simple stuff, since it ultimately ends up being a 
> simple text file, but the more complex you need to go, the worse it gets 
> since everything is implemented as a —flag=whatever option on the line, so 
> lines end up becoming huge and unwieldy.
>
> I don’t know that there’s a good summary available and honestly, I haven’t 
> tracked what has happened to the format since Kenneth started working on it, 
> so possible that my opinion is that the current state isn’t acceptable for 
> inclusion directly in pip! I wouldn’t know currently, the idea is very much 
> just a vague “I wanted this for pip years ago, pipenv actually implemented 
> it, it would be great to fold that back into pip”

https://github.com/pypa/pipfile/issues/108 is an issue I put together
after starting to use pipenv and work with Kenneth, Dan, et al on
pipenv UX design questions.

TL;DR is that I think native Pipfile/Pipfile.lock support at the pip
layer does still make sense, but the UX of it wouldn't look the same
as it does in pipenv (since it would be an opt-in thing to request to
track your changes to the current environment, rather than the default
behaviour, and because pip *wouldn't* gain the ability to generate
`Pipfile.lock` directly, only the ability to derive one from the
current environment as part of `pip freeze`).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/K3ZIR7PPY5OKMDBVLIRDBI4THUUTDJC4/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Dan Ryan]

Pipenv also uses pip as mentioned several times in the thread, and (reiterating 
here) the entire point of the conversation is about how both can work together 
on changes. That is the thrust of the whole discussion. We are actively using 
pip via its internals and pips developers (who _actively develop pip_) would 
like us to an alternate approach. 

The discussion is about how to find one and then contribute it back to pip. 
Nobody is discontinuing work on pip, nobody is splitting from pip, and I would 
prefer if we could refrain from trying to spread this kind of inaccurate 
picture. I know we have had unproductive conversations on the issue tracker, 
please don’t bring them to the mailing list. 

Dan Ryan // pipenv maintainer
gh: @techalchemy

> On Sep 20, 2018, at 2:29 PM, Bert JW Regeer  wrote:
> 
> 
> 
>> On Sep 20, 2018, at 12:11, Tzu-ping Chung  wrote:
>> 
>> 
>>> On 21 Sep 2018, at 02:01, Bert JW Regeer  wrote:
>>> 
>>> 
>>> 
 On Sep 19, 2018, at 23:22, Chris Jerdonek  wrote:
 
 Thus, it's looking like things could be on track to split the user and 
 maintainer base in two, with pip bearing the legacy burden and perhaps not 
 seeing the improvements. Are we okay with that future?
>>> 
>>> This'll be a sad day. pip is still used as an installer by other build 
>>> system where using pipenv is simply not a possibility.
>> 
>> I am not quite sure I understand why you’d think so. pip has been bearing 
>> the legacy burden for years, and if this is the future (not saying it is), 
>> it would more like just another day in the office for pip users, since 
>> nothing is changing.
> 
> pip not seeing any improvements is something I think will be sad. I don't use 
> pipenv, but use poetry which uses pip behind the scenes to do installation. I 
> also use flit. For either of those cases I would think it sad that pipenv 
> splits from pip, and then developers of alternate tooling around building 
> packages (but not installing) don't get new improvements because "pip is 
> legacy".
> 
> pipenv doesn't work in various scenarios, and trying to shoehorn it into 
> those scenarios is just wrong especially since it wasn't designed to do those 
> things.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/APGRL2LOTGHCDP6AI5S7RJXO7TE5RORO/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Bert JW Regeer]

> On Sep 20, 2018, at 12:42, Donald Stufft  wrote:
> 
> 
> 
>> On Sep 20, 2018, at 2:29 PM, Bert JW Regeer > > wrote:
>> 
>> pip not seeing any improvements is something I think will be sad. I don't 
>> use pipenv, but use poetry which uses pip behind the scenes to do 
>> installation. I also use flit. For either of those cases I would think it 
>> sad that pipenv splits from pip, and then developers of alternate tooling 
>> around building packages (but not installing) don't get new improvements 
>> because "pip is legacy".
>> 
>> pipenv doesn't work in various scenarios, and trying to shoehorn it into 
>> those scenarios is just wrong especially since it wasn't designed to do 
>> those things.
> 
> FWIW, I don’t think that anyone is trying to sunset pip.

That's great, however that is not how the previous emails read to me. I'm glad 
that is sorted.--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/UWVZDM3CHNZSJL7M6HREJQ4QKDJK2EKM/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Donald Stufft]

> On Sep 21, 2018, at 8:14 AM, Paul Moore  wrote:
> 
> On Fri, 21 Sep 2018 at 11:41, Nick Coghlan  > wrote:
>> 
>> On Fri, 21 Sep 2018 at 05:47, Donald Stufft  wrote:
>>> On Sep 20, 2018, at 3:35 PM, Paul Moore  wrote:
>>> I don't think anyone's even spoken to the pip maintainers (yet?) about
>>> supporting the pipfile format
>>> 
>>> That comes from me, I initially wrote the Pipfile as a proof of concept / 
>>> sketch of an API for replacing the requirements.txt format, which Kenneth 
>>> took and created pipenv from. At some point I plan on trying to push 
>>> support for those ideas back into pip (not the virtual environment 
>>> management bits though). That’s obviously my personal goal though, and 
>>> doesn’t represent an agreed upon direction for pip.
>> 
>> And it's one where I think there are a couple of different levels of
>> support that are worth considering:
>> 
>> Q. Should pip support installing from Pipfile.lock files as well as
>> requirements.txt files?
>> 
>> A. Once the lock file format stabilises, absolutely, since this is
>> squarely in pip's "component installer" wheelhouse.
>> 
>> Q. Should "pip install" support saving the installed components to a
>> Pipfile, and then regenerating Pipfile.lock?
>> 
>> A. This is far less of a clearcut decision, as managing updates to a
>> file that's intended to be checked in to source control is where I
>> draw the line between "component installer" and
>> "application/environment dependency manager".
> 
> Speaking as a pip developer:
> 
> Where's there a good summary of the pipfile format, the pipfile.lock
> format, and their relationship and contrast with requirements.txt? I
> don't view https://github.com/pypa/pipfile  
> as a "good summary",
> because it explicitly states that pipfile is intended to *replace*
> requirements.txt, and I disagree strongly with that.

So I should probably be explicit that this was basically an idea I had, that 
Kenneth rolled with and originally tried to implement as a PR to pip but when 
pip’s code base was too, gnarly to get it in directly, he made pipenv to wrap 
pip to implement it, and since then pipenv has grown it’s own features and such 
and is now an independent tool. Maybe it no longer makes sense for Pipfile to 
be part of pip, maybe the idea is still good, but we need to do it in a 
different way, I dunno. My thoughts aren’t really fully formulated on it beyond 
“this is a thing I’d like to do, or at least explore doing” because I’ve been 
focused on other things.

The original intent *was* to replace requirements.txt, because the 
requirements.txt format is not super great and is kind of unwieldy to work 
with. It conflates the idea of a lock file with the idea of the things I 
actually want to install, which makes it harder to work with than needs to be. 
It’s fine-ish for simple stuff, since it ultimately ends up being a simple text 
file, but the more complex you need to go, the worse it gets since everything 
is implemented as a —flag=whatever option on the line, so lines end up becoming 
huge and unwieldy.

I don’t know that there’s a good summary available and honestly, I haven’t 
tracked what has happened to the format since Kenneth started working on it, so 
possible that my opinion is that the current state isn’t acceptable for 
inclusion directly in pip! I wouldn’t know currently, the idea is very much 
just a vague “I wanted this for pip years ago, pipenv actually implemented it, 
it would be great to fold that back into pip”

Digging back through my gists, I’m not sure I can find the original Pipfile 
idea, but I did find one of the early incantations of me sketching ideas out — 
https://gist.github.com/dstufft/e61c97ee30192e575140 
, I know I have others in 
my gists somewhere but I got tired of clicking through them.

> 
> Also, pipfile is human-readable, but pipfile.lock isn't. As far as I
> know, pipfile.lock is currently generated solely by pipfile - before
> pip consumes pipfile.lock, I'd like to see that format discussed and
> agreed as a formal interop standard that any tools wanting to pass
> data to pip (for the use case the standard describes) can use. One
> obvious thing I'd like to consider is changing the name to something
> less tool-specific - requirements.lock maybe?

So the name came from me, because at the time it was intended to be a pip 
specific format (though one of my TODOs was asking if the name made sense).

> 
> As far as the pipfile format is concerned, I see that more as pipenv's
> human readable input file that is used to *generate* the lock file,
> and I don't see it as something pip should consume directly, as that
> would mean pip overlapping in functionality with pipenv.
> 
> If I'm misunderstanding the relationship between pip and pipenv, or
> between pipenv and pipfile, I'm happy to be corrected. But can I
> suggest that the best way to do 

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Sumana Harihareswara]

Bert, it sounds like you have a Code of Conduct complaint about a PyPA-
maintained project -- may I formally pass that along to all of the
pipenv maintainers perhttp://www.pypa.io/en/latest/code-of-conduct/ 
for followup and ask for more specific details? You can let me know on-
list or offlist.--
Sumana Harihareswara
Changeset Consulting
s...@changeset.nyc



On Thu, Sep 20, 2018, at 9:34 PM, Bert JW Regeer wrote:
> 
> 
>> On Sep 20, 2018, at 16:30, Dan Ryan  wrote:
>> 
>> Pipenv also uses pip as mentioned several times in the thread, and
>> (reiterating here) the entire point of the conversation is about how
>> both can work together on changes. That is the thrust of the whole
>> discussion. We are actively using pip via its internals and pips
>> developers (who _actively develop pip_) would like us to an alternate
>> approach.>> 
>> The discussion is about how to find one and then contribute it back
>> to pip. Nobody is discontinuing work on pip, nobody is splitting from
>> pip, and I would prefer if we could refrain from trying to spread
>> this kind of inaccurate picture.> 
> Wait, what? How did my apparently misunderstanding of what "it's
> looking like things could be on track to split the user and maintainer
> base in two" and me explaining why I don't think all new innovation
> should go into pipenv suddenly turn into "spread this kind of
> inaccurate picture".> 
>> I know we have had unproductive conversations on the issue tracker,
>> please don’t bring them to the mailing list.> 
> This isn't about you, has absolutely NOTHING to do with you, don't
> make it about you. I am trying to contribute my thoughts back to the
> discussion which only is of peripherally concerned about pipenv, but
> is about the future of pip/package installation, and a comment that
> was made regarding pip becoming "legacy".> 
> You made me feel incredibly unwelcome to pipenv, I will no longer
> actively attempt to contribute back to that community. I have gone out
> of my way to stay away from any PyPA projects because of the actions
> and behaviours you showed on the pipenv tracker, and have actively
> encouraged others to do the same and look at other open source
> projects instead. Let us be crystal clear here, the way you and
> Kenneth have shown your colours on the pipenv issue tracker is a real
> shame and is turning off many potential contributors and good feedback
> to help improve pipenv.> 
> This post, right here, has re-iterated that view.
> 
> Don't contact me again.
> 
>> 
>> Dan Ryan // pipenv maintainer
>> gh: @techalchemy
>> 
>> On Sep 20, 2018, at 2:29 PM, Bert JW Regeer
>>  wrote:>>> 
>>> 
 On Sep 20, 2018, at 12:11, Tzu-ping Chung 
 wrote: 
 
 
 On 21 Sep 2018, at 02:01, Bert JW Regeer  wrote:> 
> 
>> On Sep 19, 2018, at 23:22, Chris Jerdonek
>>  wrote:>> 
>> Thus, it's looking like things could be on track to split the
>> user and maintainer base in two, with pip bearing the legacy
>> burden and perhaps not seeing the improvements. Are we okay with
>> that future?> 
> This'll be a sad day. pip is still used as an installer by other
> build system where using pipenv is simply not a possibility. 
 I am not quite sure I understand why you’d think so. pip has been
 bearing the legacy burden for years, and if this is the future (not
 saying it is), it would more like just another day in the office
 for pip users, since nothing is changing.>>> 
>>> pip not seeing any improvements is something I think will be sad. I
>>> don't use pipenv, but use poetry which uses pip behind the scenes to
>>> do installation. I also use flit. For either of those cases I would
>>> think it sad that pipenv splits from pip, and then developers of
>>> alternate tooling around building packages (but not installing)
>>> don't get new improvements because "pip is legacy".>>> 
>>> pipenv doesn't work in various scenarios, and trying to shoehorn it
>>> into those scenarios is just wrong especially since it wasn't
>>> designed to do those things.> 
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/BC7DTEX7VGBZ5NEAXQH2TRF7EXF3PMH3/
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/KXNRH75RP5USUTYWGQGGKGNT34VQYHZU/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Paul Moore]

On Fri, 21 Sep 2018 at 11:41, Nick Coghlan  wrote:
>
> On Fri, 21 Sep 2018 at 05:47, Donald Stufft  wrote:
> > On Sep 20, 2018, at 3:35 PM, Paul Moore  wrote:
> > I don't think anyone's even spoken to the pip maintainers (yet?) about
> > supporting the pipfile format
> >
> > That comes from me, I initially wrote the Pipfile as a proof of concept / 
> > sketch of an API for replacing the requirements.txt format, which Kenneth 
> > took and created pipenv from. At some point I plan on trying to push 
> > support for those ideas back into pip (not the virtual environment 
> > management bits though). That’s obviously my personal goal though, and 
> > doesn’t represent an agreed upon direction for pip.
>
> And it's one where I think there are a couple of different levels of
> support that are worth considering:
>
> Q. Should pip support installing from Pipfile.lock files as well as
> requirements.txt files?
>
> A. Once the lock file format stabilises, absolutely, since this is
> squarely in pip's "component installer" wheelhouse.
>
> Q. Should "pip install" support saving the installed components to a
> Pipfile, and then regenerating Pipfile.lock?
>
> A. This is far less of a clearcut decision, as managing updates to a
> file that's intended to be checked in to source control is where I
> draw the line between "component installer" and
> "application/environment dependency manager".

Speaking as a pip developer:

Where's there a good summary of the pipfile format, the pipfile.lock
format, and their relationship and contrast with requirements.txt? I
don't view https://github.com/pypa/pipfile as a "good summary",
because it explicitly states that pipfile is intended to *replace*
requirements.txt, and I disagree strongly with that.

Also, pipfile is human-readable, but pipfile.lock isn't. As far as I
know, pipfile.lock is currently generated solely by pipfile - before
pip consumes pipfile.lock, I'd like to see that format discussed and
agreed as a formal interop standard that any tools wanting to pass
data to pip (for the use case the standard describes) can use. One
obvious thing I'd like to consider is changing the name to something
less tool-specific - requirements.lock maybe?

As far as the pipfile format is concerned, I see that more as pipenv's
human readable input file that is used to *generate* the lock file,
and I don't see it as something pip should consume directly, as that
would mean pip overlapping in functionality with pipenv.

If I'm misunderstanding the relationship between pip and pipenv, or
between pipenv and pipfile, I'm happy to be corrected. But can I
suggest that the best way to do so would be to amend the project pages
that are giving me the impressions I have above, and pointing me at
the corrected versions? That way, we can make sure that any
misinformation is corrected at source...

Paul

PS Full disclosure - I've tried to use pipenv in a couple of local
projects, because of the hype about it being the "great new thing" and
found it basically of no use for my requirements/workflow. So I may
have a biased view of either pipenv, or how it's being presented. I'm
trying to be objective in the above, but my bias may have slipped
through.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/SR3FAMLZM646GT3IPYFILD47KMRWOALD/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Nick Coghlan]

On Fri, 21 Sep 2018 at 05:47, Donald Stufft  wrote:
> On Sep 20, 2018, at 3:35 PM, Paul Moore  wrote:
> I don't think anyone's even spoken to the pip maintainers (yet?) about
> supporting the pipfile format
>
> That comes from me, I initially wrote the Pipfile as a proof of concept / 
> sketch of an API for replacing the requirements.txt format, which Kenneth 
> took and created pipenv from. At some point I plan on trying to push support 
> for those ideas back into pip (not the virtual environment management bits 
> though). That’s obviously my personal goal though, and doesn’t represent an 
> agreed upon direction for pip.

And it's one where I think there are a couple of different levels of
support that are worth considering:

Q. Should pip support installing from Pipfile.lock files as well as
requirements.txt files?

A. Once the lock file format stabilises, absolutely, since this is
squarely in pip's "component installer" wheelhouse.

Q. Should "pip install" support saving the installed components to a
Pipfile, and then regenerating Pipfile.lock?

A. This is far less of a clearcut decision, as managing updates to a
file that's intended to be checked in to source control is where I
draw the line between "component installer" and
"application/environment dependency manager".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/XD5DOF75MPWMQVCMT4O2J3X6C4RR47BM/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Dan Ryan]

Sorry that you feel such hostility toward the PyPA, they are certainly not 
responsible for my actions; 

 

> explaining why I don't think all new innovation should go into pipenv

 

Nobody is advocating this position

 

> I am trying to contribute my thoughts back to the discussion which only is of 
> peripherally concerned about pipenv, but is about the future of pip/package 
> installation

 

The thread in question is a direct dialog about how we can work together on 
joint projects at this point, which does include pipenv and pip and several 
other tools; hopefully we will be able to break this down into a coherent list 
of other useful smaller utilities which could be consumed.  

 

Unfortunately this is a mailing list and the discussion is likely to continue 
so there is a high probability that my name will show up in your inbox

 

Dan Ryan

gh: @techalchemy <https://github.com/techalchemy>  // e: d...@danryan.co

 

From: Bert JW Regeer [mailto:xiste...@0x58.com] 
Sent: Thursday, September 20, 2018 9:34 PM
To: Dan Ryan
Cc: Tzu-ping Chung; Chris Jerdonek; Distutils; Donald Stufft
Subject: Re: [Distutils] Distlib vs Packaging (Was: disable building wheel for 
a package)

 

 





On Sep 20, 2018, at 16:30, Dan Ryan  wrote:

 

Pipenv also uses pip as mentioned several times in the thread, and (reiterating 
here) the entire point of the conversation is about how both can work together 
on changes. That is the thrust of the whole discussion. We are actively using 
pip via its internals and pips developers (who _actively develop pip_) would 
like us to an alternate approach. 

 

The discussion is about how to find one and then contribute it back to pip. 
Nobody is discontinuing work on pip, nobody is splitting from pip, and I would 
prefer if we could refrain from trying to spread this kind of inaccurate 
picture. 

 

Wait, what? How did my apparently misunderstanding of what "it's looking like 
things could be on track to split the user and maintainer base in two" and me 
explaining why I don't think all new innovation should go into pipenv suddenly 
turn into "spread this kind of inaccurate picture". 





I know we have had unproductive conversations on the issue tracker, please 
don’t bring them to the mailing list. 

 

This isn't about you, has absolutely NOTHING to do with you, don't make it 
about you. I am trying to contribute my thoughts back to the discussion which 
only is of peripherally concerned about pipenv, but is about the future of 
pip/package installation, and a comment that was made regarding pip becoming 
"legacy".

 

You made me feel incredibly unwelcome to pipenv, I will no longer actively 
attempt to contribute back to that community. I have gone out of my way to stay 
away from any PyPA projects because of the actions and behaviours you showed on 
the pipenv tracker, and have actively encouraged others to do the same and look 
at other open source projects instead. Let us be crystal clear here, the way 
you and Kenneth have shown your colours on the pipenv issue tracker is a real 
shame and is turning off many potential contributors and good feedback to help 
improve pipenv.

 

This post, right here, has re-iterated that view.

 

Don't contact me again.





 

Dan Ryan // pipenv maintainer

gh: @techalchemy


On Sep 20, 2018, at 2:29 PM, Bert JW Regeer  wrote:

 





On Sep 20, 2018, at 12:11, Tzu-ping Chung  wrote:

 

 

On 21 Sep 2018, at 02:01, Bert JW Regeer  wrote:

 





On Sep 19, 2018, at 23:22, Chris Jerdonek  wrote:

 

Thus, it's looking like things could be on track to split the user and 
maintainer base in two, with pip bearing the legacy burden and perhaps not 
seeing the improvements. Are we okay with that future?

 

This'll be a sad day. pip is still used as an installer by other build system 
where using pipenv is simply not a possibility.

 

I am not quite sure I understand why you’d think so. pip has been bearing the 
legacy burden for years, and if this is the future (not saying it is), it would 
more like just another day in the office for pip users, since nothing is 
changing.

 

pip not seeing any improvements is something I think will be sad. I don't use 
pipenv, but use poetry which uses pip behind the scenes to do installation. I 
also use flit. For either of those cases I would think it sad that pipenv 
splits from pip, and then developers of alternate tooling around building 
packages (but not installing) don't get new improvements because "pip is 
legacy".

 

pipenv doesn't work in various scenarios, and trying to shoehorn it into those 
scenarios is just wrong especially since it wasn't designed to do those things.

 

--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.pyt

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Bert JW Regeer]

> On Sep 20, 2018, at 16:30, Dan Ryan  wrote:
> 
> Pipenv also uses pip as mentioned several times in the thread, and 
> (reiterating here) the entire point of the conversation is about how both can 
> work together on changes. That is the thrust of the whole discussion. We are 
> actively using pip via its internals and pips developers (who _actively 
> develop pip_) would like us to an alternate approach. 
> 
> The discussion is about how to find one and then contribute it back to pip. 
> Nobody is discontinuing work on pip, nobody is splitting from pip, and I 
> would prefer if we could refrain from trying to spread this kind of 
> inaccurate picture.

Wait, what? How did my apparently misunderstanding of what "it's looking like 
things could be on track to split the user and maintainer base in two" and me 
explaining why I don't think all new innovation should go into pipenv suddenly 
turn into "spread this kind of inaccurate picture". 

> I know we have had unproductive conversations on the issue tracker, please 
> don’t bring them to the mailing list. 

This isn't about you, has absolutely NOTHING to do with you, don't make it 
about you. I am trying to contribute my thoughts back to the discussion which 
only is of peripherally concerned about pipenv, but is about the future of 
pip/package installation, and a comment that was made regarding pip becoming 
"legacy".

You made me feel incredibly unwelcome to pipenv, I will no longer actively 
attempt to contribute back to that community. I have gone out of my way to stay 
away from any PyPA projects because of the actions and behaviours you showed on 
the pipenv tracker, and have actively encouraged others to do the same and look 
at other open source projects instead. Let us be crystal clear here, the way 
you and Kenneth have shown your colours on the pipenv issue tracker is a real 
shame and is turning off many potential contributors and good feedback to help 
improve pipenv.

This post, right here, has re-iterated that view.

Don't contact me again.

> 
> Dan Ryan // pipenv maintainer
> gh: @techalchemy
> 
> On Sep 20, 2018, at 2:29 PM, Bert JW Regeer  > wrote:
> 
>> 
>> 
>>> On Sep 20, 2018, at 12:11, Tzu-ping Chung >> > wrote:
>>> 
>>> 
>>> On 21 Sep 2018, at 02:01, Bert JW Regeer >> > wrote:
>>> 
 
 
> On Sep 19, 2018, at 23:22, Chris Jerdonek  > wrote:
> 
> Thus, it's looking like things could be on track to split the user and 
> maintainer base in two, with pip bearing the legacy burden and perhaps 
> not seeing the improvements. Are we okay with that future?
 
 This'll be a sad day. pip is still used as an installer by other build 
 system where using pipenv is simply not a possibility.
>>> 
>>> I am not quite sure I understand why you’d think so. pip has been bearing 
>>> the legacy burden for years, and if this is the future (not saying it is), 
>>> it would more like just another day in the office for pip users, since 
>>> nothing is changing.
>> 
>> pip not seeing any improvements is something I think will be sad. I don't 
>> use pipenv, but use poetry which uses pip behind the scenes to do 
>> installation. I also use flit. For either of those cases I would think it 
>> sad that pipenv splits from pip, and then developers of alternate tooling 
>> around building packages (but not installing) don't get new improvements 
>> because "pip is legacy".
>> 
>> pipenv doesn't work in various scenarios, and trying to shoehorn it into 
>> those scenarios is just wrong especially since it wasn't designed to do 
>> those things.

--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/BC7DTEX7VGBZ5NEAXQH2TRF7EXF3PMH3/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Donald Stufft]

> On Sep 20, 2018, at 3:35 PM, Paul Moore  wrote:
> 
> I don't think anyone's even spoken to the pip maintainers (yet?) about
> supporting the pipfile format

That comes from me, I initially wrote the Pipfile as a proof of concept / 
sketch of an API for replacing the requirements.txt format, which Kenneth took 
and created pipenv from. At some point I plan on trying to push support for 
those ideas back into pip (not the virtual environment management bits though). 
That’s obviously my personal goal though, and doesn’t represent an agreed upon 
direction for pip.--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/SOMJOXHJELMRGLP7TJ5AF7UZLBRLMDI3/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Paul Moore]

On Thu, 20 Sep 2018 at 19:52, Michael Merickel  wrote:
>
> I think it's far-fetched to start thinking pip is legacy. Pipfile has had a 
> goal from day 1 to be a format that pip would support. PEP 582 is a path 
> forward here for providing a default location for a virtualenv [2] - it's 
> just that everything moves slower in pip because it supports more use-cases 
> than a tool like pipenv.

I don't think anyone's even spoken to the pip maintainers (yet?) about
supporting the pipfile format. And no-one from the pip team has ever
said that we're retiring pip in favour of pipenv. At one point, I
think there was a lot of rhetoric around pipenv, but IMO it was just
that, rhetoric. I'm not sure where the "everything moves slower in
pip" comment comes from - pip's moving at a fair pace. I've no feel
for how fast pipenv is moving (although for the parts where they use
pip, they are "obviously" going to move faster in some sense, because
they can use all the changes in pip and add their own :-))

> What started out as a reference implementation has definitely taken on a life 
> of its own of course and it's up to PyPA to manage that relationship and 
> offer a good story around the tooling it's building.

As far as I'm concerned, pip and pipenv are different tools,
supporting different use cases. I don't know enough about pipenv to
say much more than that. The "official PyPA position" (if that's a
thing, and if it's what someone is after) is probably at
https://packaging.python.org/ and that document describes pip in the
"Installing Packages" section, and pipenv under "Managing Application
Dependencies". To me, that's a pretty clear distinction.

Paul
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/TXQQUXPT4FRV2YI5TPTO5PCGBF3OAYZ3/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Michael Merickel]

I think it's far-fetched to start thinking pip is legacy. Pipfile has had a
goal from day 1 to be a format that pip would support. PEP 582 is a path
forward here for providing a default location for a virtualenv [2] - it's
just that everything moves slower in pip because it supports more use-cases
than a tool like pipenv. What started out as a reference implementation has
definitely taken on a life of its own of course and it's up to PyPA to
manage that relationship and offer a good story around the tooling it's
building.

[1] https://github.com/pypa/pipfile#pip-integration-eventual
[2] https://www.python.org/dev/peps/pep-0582/

On Thu, Sep 20, 2018 at 1:38 PM Bert JW Regeer  wrote:

>
>
> On Sep 20, 2018, at 12:11, Tzu-ping Chung  wrote:
>
>
> On 21 Sep 2018, at 02:01, Bert JW Regeer  wrote:
>
>
>
> On Sep 19, 2018, at 23:22, Chris Jerdonek 
> wrote:
>
> Thus, it's looking like things could be on track to split the user and
> maintainer base in two, with pip bearing the legacy burden and perhaps not
> seeing the improvements. Are we okay with that future?
>
>
> This'll be a sad day. pip is still used as an installer by other build
> system where using pipenv is simply not a possibility.
>
>
> I am not quite sure I understand why you’d think so. pip has been bearing
> the legacy burden for years, and if this is the future (not saying it is),
> it would more like just another day in the office for pip users, since
> nothing is changing.
>
>
> pip not seeing any improvements is something I think will be sad. I don't
> use pipenv, but use poetry which uses pip behind the scenes to do
> installation. I also use flit. For either of those cases I would think it
> sad that pipenv splits from pip, and then developers of alternate tooling
> around building packages (but not installing) don't get new improvements
> because "pip is legacy".
>
> pipenv doesn't work in various scenarios, and trying to shoehorn it into
> those scenarios is just wrong especially since it wasn't designed to do
> those things.
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/3JKW3DJYVAGENQNQRLEAKKJXQC2QXIZF/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/OSVC5AT4XBBAUGCF7YVHDZ4ZN5R6FRFA/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Donald Stufft]

> On Sep 20, 2018, at 2:29 PM, Bert JW Regeer  wrote:
> 
> pip not seeing any improvements is something I think will be sad. I don't use 
> pipenv, but use poetry which uses pip behind the scenes to do installation. I 
> also use flit. For either of those cases I would think it sad that pipenv 
> splits from pip, and then developers of alternate tooling around building 
> packages (but not installing) don't get new improvements because "pip is 
> legacy".
> 
> pipenv doesn't work in various scenarios, and trying to shoehorn it into 
> those scenarios is just wrong especially since it wasn't designed to do those 
> things.

FWIW, I don’t think that anyone is trying to sunset pip.--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/I4QINGFIVEPMKVOJ4FLWIAO7XUIFNDOQ/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Bert JW Regeer]

> On Sep 20, 2018, at 12:11, Tzu-ping Chung  wrote:
> 
> 
> On 21 Sep 2018, at 02:01, Bert JW Regeer  > wrote:
> 
>> 
>> 
>>> On Sep 19, 2018, at 23:22, Chris Jerdonek >> > wrote:
>>> 
>>> Thus, it's looking like things could be on track to split the user and 
>>> maintainer base in two, with pip bearing the legacy burden and perhaps not 
>>> seeing the improvements. Are we okay with that future?
>> 
>> This'll be a sad day. pip is still used as an installer by other build 
>> system where using pipenv is simply not a possibility.
> 
> I am not quite sure I understand why you’d think so. pip has been bearing the 
> legacy burden for years, and if this is the future (not saying it is), it 
> would more like just another day in the office for pip users, since nothing 
> is changing.

pip not seeing any improvements is something I think will be sad. I don't use 
pipenv, but use poetry which uses pip behind the scenes to do installation. I 
also use flit. For either of those cases I would think it sad that pipenv 
splits from pip, and then developers of alternate tooling around building 
packages (but not installing) don't get new improvements because "pip is 
legacy".

pipenv doesn't work in various scenarios, and trying to shoehorn it into those 
scenarios is just wrong especially since it wasn't designed to do those things.--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/3JKW3DJYVAGENQNQRLEAKKJXQC2QXIZF/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Bert JW Regeer]

> On Sep 19, 2018, at 23:22, Chris Jerdonek  wrote:
> 
> Thus, it's looking like things could be on track to split the user and 
> maintainer base in two, with pip bearing the legacy burden and perhaps not 
> seeing the improvements. Are we okay with that future?

This'll be a sad day. pip is still used as an installer by other build system 
where using pipenv is simply not a possibility.--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/TDB7OETMEAFRVCASVF3GWF527OH7JSK2/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Tzu-ping Chung]

> On 21 Sep 2018, at 02:01, Bert JW Regeer  wrote:
> 
> 
> 
>> On Sep 19, 2018, at 23:22, Chris Jerdonek  wrote:
>> 
>> Thus, it's looking like things could be on track to split the user and 
>> maintainer base in two, with pip bearing the legacy burden and perhaps not 
>> seeing the improvements. Are we okay with that future?
> 
> This'll be a sad day. pip is still used as an installer by other build system 
> where using pipenv is simply not a possibility.

I am not quite sure I understand why you’d think so. pip has been bearing the 
legacy burden for years, and if this is the future (not saying it is), it would 
more like just another day in the office for pip users, since nothing is 
changing.--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/IUGYOCCIOJ7PRKNYEFW5FDEZBXYGOSPM/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Dan Ryan]

I have no issue moving things to the PyPA, the list was more of a ‘which of 
these is useful’ check

Dan Ryan
gh: @techalchemy // e: d...@danryan.co

From: Donald Stufft [mailto:don...@stufft.io] 
Sent: Thursday, September 20, 2018 8:15 AM
To: Dan Ryan
Cc: Tzu-ping Chung; Distutils
Subject: Re: [Distutils] Distlib vs Packaging (Was: disable building wheel for 
a package)




On Sep 20, 2018, at 8:12 AM, Donald Stufft  wrote:

Depending on how vital a particular bit of functionality is to pip, we’re 
likely going to want most libraries that are pulling functionality out of pip 
to live under the PyPA banner, and ideally should be setup in a way that 
existing pip contributors can work on them as well. While conceptually these 
are becoming distinct entities, for end users they’re going to be part of the 
nebulous thing that is pip, and changes and for “core” bits, pip wouldn’t want 
to lose the ability to work on these bits of functionality directly.


Quick clarification— That’s not to suggest that any particular one of these 
libraries need to move under the PyPA banner at this point. Just that as a 
general rule of thumb, stuff that is core to pip, we’re going to want the above 
before we would likely accept a PR that switches pip over to using it (again, 
depending on how “core” it is).
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/C3F5KCJDKGDENGSMIRENEDQHYXFXRRGU/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Donald Stufft]

> On Sep 20, 2018, at 8:12 AM, Donald Stufft  wrote:
> 
> Depending on how vital a particular bit of functionality is to pip, we’re 
> likely going to want most libraries that are pulling functionality out of pip 
> to live under the PyPA banner, and ideally should be setup in a way that 
> existing pip contributors can work on them as well. While conceptually these 
> are becoming distinct entities, for end users they’re going to be part of the 
> nebulous thing that is pip, and changes and for “core” bits, pip wouldn’t 
> want to lose the ability to work on these bits of functionality directly.


Quick clarification— That’s not to suggest that any particular one of these 
libraries need to move under the PyPA banner at this point. Just that as a 
general rule of thumb, stuff that is core to pip, we’re going to want the above 
before we would likely accept a PR that switches pip over to using it (again, 
depending on how “core” it is).--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/QRKLNVOKHDM3KYSPDDOZJHP2COJA4PAQ/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Donald Stufft]

> On Sep 19, 2018, at 11:54 PM, Dan Ryan  wrote:
> 
> Since you mentioned following along, here's what we're working on right now:
> 
> https://github.com/sarugaku/requirementslib 
>  -- abstraction layer for parsing
> and converting various requirements formats
> (pipfile/requirements.txt/command line/InstallRequirement) and moving
> between all of them
> https://github.com/sarugaku/resolvelib 
>  -- directed acyclic graph library for
> handling dependency resolution (not yet being used in pipenv)
> https://github.com/sarugaku/passa  -- 
> dependency resolver/installer/pipfile
> manager (bulk of the logic we have been talking about is in here right now)
> -- I think we will probably split this back out into multiple other smaller
> libraries or something based on the discussion
> https://github.com/sarugaku/plette  -- 
> this is a rewrite of pipfile with some
> additional logic / validation
> https://github.com/sarugaku/shellingham 
>  -- this is a shell detection library
> made up of some tooling we built in pipenv for environment detection
> https://github.com/sarugaku/pythonfinder 
>  -- this is a library for finding
> python (pep 514 compliant) by version and for finding any other executables
> (cross platform)
> https://github.com/sarugaku/virtenv  -- 
> python api for virtualenv creation

There’s been a lot of great discussion here, and I’m going to try to find time 
to go through these, though I wanted to make something explicit that I think 
maybe I was leaving implicit, and I thought should be made explicit:

Depending on how vital a particular bit of functionality is to pip, we’re 
likely going to want most libraries that are pulling functionality out of pip 
to live under the PyPA banner, and ideally should be setup in a way that 
existing pip contributors can work on them as well. While conceptually these 
are becoming distinct entities, for end users they’re going to be part of the 
nebulous thing that is pip, and changes and for “core” bits, pip wouldn’t want 
to lose the ability to work on these bits of functionality directly.

Obviously there is some stuff that isn’t “core” to what pip does, that we’re 
generally fine with being owned in a way that we aren’t part owners of. For 
instance we use requests, CacheControl, etc. The key difference there is that 
these are all things that aren’t really specific to pip’s core functionality 
(even though we may use them in implementing that) and so we don’t need to care 
too much about their implementation one way or another.

We’re also probably going to need/want to figure out some sort of shared 
requirements for things like “when do we drop support for a version of Python 
in xyzlib” and such like that.--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/YYRENORJIQHLDQ3PK64FLXTDDJXSAIP4/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Paul Moore]

On Thu, 20 Sep 2018 at 08:01, Tzu-ping Chung  wrote:
> I’m afraid the new implementation will still need to deal with compatibility 
> issues.
> Users expect Pipenv to work exactly as pip, and get very angry if it does not,
> especially when they see it is under the PyPA organisation on GitHub. The last
> time Pipenv tries to explain it does whatever arbitrary things it does, we get
> labelled as “toxic” (there are other issues in play, but this is IMO the 
> ultimate
> cause). Whether the image is fair or not, I would most definitely want to 
> avoid
> similar incidents from happening again.

IMO, the only way to address that is by defining standards for the
behaviour. Having a standard document to point to that says "this is
what's been agreed in public debate" gives both pipenv and pip a solid
basis to explain why we do what we do. There will likely be corner
cases where the details are implementation dependent, but again, the
fact that the standard doesn't mandate behaviour is the best argument
you're going to get for that. There will always be people that
complain if you're not 100% bug-for-bug compatible with pip, but
that's life.

Obviously, any standard will have to look at pip's behaviour as a
starting point (simply because pip's been around as the only
implementation for so long). But simplifying and cutting out some of
the cruft is part of any standards process, so it's perfectly OK to
mark certain parts of what pip does now as "implementation defined" or
"needs to change".

Also, Dan said:
> Since you mentioned following along, here's what we're working on right now:

The problem here is the same - without some sort of agreement (in the
form of a documented standard[1]) that what those libraries do is "the
right behaviour", it's not clear how pip can switch to using them. And
promises that they "do the same as pip does" are not likely to work,
for precisely the reasons Tzu-ping noted above (there's always someone
that will pick up on any discrepancy, no mater how small).

Paul

PS While I don't have much time for people standing on the sidelines
and telling us what we "should" do, I do think that by putting
projects under the PyPA banner, we assume a responsibility for making
sure we behave consistently, whether we like it or not. Interop
standards documents have been how we've discharged that responsibility
so far, but pipenv has such a strong overlap with pip that it opens up
a lot of areas where we haven't even thought about standards yet.
Managing expectations while we get things in line is not a pleasant
task, but it's one we need to do.

[1] I'm at least as sick of saying "standard" as you are of hearing
it. Take it to mean "everyone's agreed and anyone likely to complain
afterwards has had an opportunity to speak up, and there's a record" -
I'm not wedded to any particular process here.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/IPS7PDROXEY2I353A6MURRSP2KWOIB3N/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Tzu-ping Chung]

Thanks for the advices, they are really helpful. Incidentally (or maybe not? I 
wonder
if there is an underlying pattern here) the two areas I do want to work on 
first are
a) how to find a package, and b) how to choose an artifact for a given package.

I think I’m starting with the package discovery part first and work my way from
there. I’ll create an issue in pypa/pip and try to outline the intention (and 
summarise
this thread), but there’s a couple of things I wish to clarify first:

1. Should dependency link processing be included? Since it is un-deprecated 
now, I
guess the answer is yes?
2. (Maybe not an immediate issue) What formats should I include? Wheels and 
.tar.gz
sdists, of course, what others? Eggs? .zip sdists? Are there other formats?

TP


> On 20/9, 2018, at 02:40, Paul Moore  wrote:
> 
> On Wed, 19 Sep 2018 at 18:52, Donald Stufft  wrote:
>> 
>> On Sep 19, 2018, at 1:14 PM, Tzu-ping Chung  wrote:
>> 
>> I feel the plan is quite solid. This however leaves us (who want a Python 
>> implementation and interface to do what pip does) in an interesting place. 
>> So I can tell there are a couple of principles:
>> 
>> 1. Do not use pip internals
>> 2. pip won’t be using either distlib or setuptools, so they might not match 
>> what pip does, in the long run
>> 
>> Does this leaves us only one option left, to implement a library that 
>> matches what pip does (follows the standards), but is not pip? That feels 
>> quite counter-productive to me, but if it’s what things would be, I’d accept 
>> it.
>> 
>> The next step (for me) in that case would then be to start working on that 
>> library. Since existing behaviours in setuptools and pip (including the part 
>> it uses distlib for) are likely to be standardised, I can rely on distlib 
>> for script creation, setuptools for some miscellaneous things (editable 
>> installs?), and pull (or reimplement) parts out of pip for others. Are there 
>> caveats I should look out?
>> 
>> 
>> My general recommendation if you want a Python implementation/interface for 
>> something pip does, is:
>> 
>> - Open an issue on the pip repository to document your intent and to make 
>> sure that there is nobody there who is against having that functionality 
>> split out. This might also give a chance for people with familiarity in that 
>> API to mention pain points that you can solve in a new API. We can also 
>> probably give you a good sense if the thing you want in a library is 
>> something that probably has multiple things that are dependent on getting 
>> split out first (for instance, if you said you wanted a library for 
>> installing wheels, we’d probably tell you that there is a dependency on PEP 
>> 425 tags, pip locations, maybe other that need resolved first) and also 
>> whether this is something that should have a PEP first or not. Getting some 
>> rough agreement on the plan to split X thing out before you start is overall 
>> a good thing.
>> 
>> - Create or update a PEP if required, and get it into the provisional state.
>> 
>> - Make the library, either as a PR to packaging or as it’s own independent 
>> library. If there are questions that come up while creating that library/PR 
>> that have to do with specific pip behaviors, go back to that original issue 
>> and ask for clarification etc. Ideally at some point you’ll open a PR on pip 
>> that uses the new library (my suggestion is to not bundle the library in the 
>> initial PR, and just import it normally so that the PR diff doesn’t include 
>> the full bundled library until there’s agreement on it). If there’s another 
>> tool (pipenv, whatever) that is looking to use that same functionality, open 
>> a WIP PR there too that switches it to using that. Use feedback and what you 
>> learn from trying to integrate in those libraries to influence back the 
>> design of the API itself.
>> 
>> Creating a PEP and creating the library and the PRs can happen in parallel, 
>> but at least for pip if something deserves a PEP, we’re not going to merge a 
>> PR until that PEP is generally agreed on. However it can be supremely useful 
>> to have them all going at the same time, because you run into things that 
>> you didn’t really notice until you went to actually implement it.
>> 
>> My other big suggestion would be to e careful about how much you bite off at 
>> one time. Pip’s internal code base is not the greatest, so pulling out 
>> smaller chunks at a time rather than trying to start right off pulling out a 
>> big topic is more likely to meet with success. Be cognizant of what the 
>> dependencies are for the feature you want to implement, because if it has 
>> dependencies, you’ll need to pull them out first before you can pull it out 
>> OR you’ll need to design the API to invert those dependencies so they get 
>> passed in instead.
>> 
>> I personally would be happy to at a minimum participate on any issue where 
>> someone was trying to split out some functionality from pip into a 

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Tzu-ping Chung]

lmost) exactly what pip is doing, 
> unless
> we can have people (pip maintainers or otherwise) backing the differences. 
> Whether
> pip uses the new implementation or not wouldn’t change the requirement :(
> 
> 
> TP
> 
> 
>> 
>> --Chris
>> 
>> 
>> Since you mentioned following along, here's what we're working on right now:
>> 
>> https://github.com/sarugaku/requirementslib 
>> <https://github.com/sarugaku/requirementslib> -- abstraction layer for 
>> parsing
>> and converting various requirements formats
>> (pipfile/requirements.txt/command line/InstallRequirement) and moving
>> between all of them
>> https://github.com/sarugaku/resolvelib 
>> <https://github.com/sarugaku/resolvelib> -- directed acyclic graph library 
>> for
>> handling dependency resolution (not yet being used in pipenv)
>> https://github.com/sarugaku/passa <https://github.com/sarugaku/passa> -- 
>> dependency resolver/installer/pipfile
>> manager (bulk of the logic we have been talking about is in here right now)
>> -- I think we will probably split this back out into multiple other smaller
>> libraries or something based on the discussion
>> https://github.com/sarugaku/plette <https://github.com/sarugaku/plette> -- 
>> this is a rewrite of pipfile with some
>> additional logic / validation
>> https://github.com/sarugaku/shellingham 
>> <https://github.com/sarugaku/shellingham> -- this is a shell detection 
>> library
>> made up of some tooling we built in pipenv for environment detection
>> https://github.com/sarugaku/pythonfinder 
>> <https://github.com/sarugaku/pythonfinder> -- this is a library for finding
>> python (pep 514 compliant) by version and for finding any other executables
>> (cross platform)
>> https://github.com/sarugaku/virtenv <https://github.com/sarugaku/virtenv> -- 
>> python api for virtualenv creation
>> 
>> Happy to provide access or take advice as needed on any of those.  Thanks
>> all for the receptiveness and collaboration
>> 
>> Dan Ryan
>> gh: @techalchemy // e: d...@danryan.co <mailto:d...@danryan.co>
>> 
>> From: Donald Stufft [mailto:don...@stufft.io <mailto:don...@stufft.io>] 
>> Sent: Wednesday, September 19, 2018 1:52 PM
>> To: Tzu-ping Chung
>> Cc: Distutils
>> Subject: [Distutils] Re: Distlib vs Packaging (Was: disable building wheel
>> for a package)
>> 
>> My general recommendation if you want a Python implementation/interface for
>> something pip does, is:
>> 
>> - Open an issue on the pip repository to document your intent and to make
>> sure that there is nobody there who is against having that functionality
>> split out. This might also give a chance for people with familiarity in that
>> API to mention pain points that you can solve in a new API. We can also
>> probably give you a good sense if the thing you want in a library is
>> something that probably has multiple things that are dependent on getting
>> split out first (for instance, if you said you wanted a library for
>> installing wheels, we'd probably tell you that there is a dependency on PEP
>> 425 tags, pip locations, maybe other that need resolved first) and also
>> whether this is something that should have a PEP first or not. Getting some
>> rough agreement on the plan to split X thing out before you start is overall
>> a good thing.
>> 
>> - Create or update a PEP if required, and get it into the provisional state.
>> 
>> - Make the library, either as a PR to packaging or as it's own independent
>> library. If there are questions that come up while creating that library/PR
>> that have to do with specific pip behaviors, go back to that original issue
>> and ask for clarification etc. Ideally at some point you'll open a PR on pip
>> that uses the new library (my suggestion is to not bundle the library in the
>> initial PR, and just import it normally so that the PR diff doesn't include
>> the full bundled library until there's agreement on it). If there's another
>> tool (pipenv, whatever) that is looking to use that same functionality, open
>> a WIP PR there too that switches it to using that. Use feedback and what you
>> learn from trying to integrate in those libraries to influence back the
>> design of the API itself.
>> 
>> Creating a PEP and creating the library and the PRs can happen in parallel,
>> but at least for pip if something deserves a PEP, we're not going to merge a
>> PR until that PEP is generally agreed on. However it can be supremely useful
>> to 

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Tzu-ping Chung]

 https://github.com/sarugaku/shellingham 
> <https://github.com/sarugaku/shellingham> -- this is a shell detection library
> made up of some tooling we built in pipenv for environment detection
> https://github.com/sarugaku/pythonfinder 
> <https://github.com/sarugaku/pythonfinder> -- this is a library for finding
> python (pep 514 compliant) by version and for finding any other executables
> (cross platform)
> https://github.com/sarugaku/virtenv <https://github.com/sarugaku/virtenv> -- 
> python api for virtualenv creation
> 
> Happy to provide access or take advice as needed on any of those.  Thanks
> all for the receptiveness and collaboration
> 
> Dan Ryan
> gh: @techalchemy // e: d...@danryan.co <mailto:d...@danryan.co>
> 
> From: Donald Stufft [mailto:don...@stufft.io <mailto:don...@stufft.io>] 
> Sent: Wednesday, September 19, 2018 1:52 PM
> To: Tzu-ping Chung
> Cc: Distutils
> Subject: [Distutils] Re: Distlib vs Packaging (Was: disable building wheel
> for a package)
> 
> My general recommendation if you want a Python implementation/interface for
> something pip does, is:
> 
> - Open an issue on the pip repository to document your intent and to make
> sure that there is nobody there who is against having that functionality
> split out. This might also give a chance for people with familiarity in that
> API to mention pain points that you can solve in a new API. We can also
> probably give you a good sense if the thing you want in a library is
> something that probably has multiple things that are dependent on getting
> split out first (for instance, if you said you wanted a library for
> installing wheels, we'd probably tell you that there is a dependency on PEP
> 425 tags, pip locations, maybe other that need resolved first) and also
> whether this is something that should have a PEP first or not. Getting some
> rough agreement on the plan to split X thing out before you start is overall
> a good thing.
> 
> - Create or update a PEP if required, and get it into the provisional state.
> 
> - Make the library, either as a PR to packaging or as it's own independent
> library. If there are questions that come up while creating that library/PR
> that have to do with specific pip behaviors, go back to that original issue
> and ask for clarification etc. Ideally at some point you'll open a PR on pip
> that uses the new library (my suggestion is to not bundle the library in the
> initial PR, and just import it normally so that the PR diff doesn't include
> the full bundled library until there's agreement on it). If there's another
> tool (pipenv, whatever) that is looking to use that same functionality, open
> a WIP PR there too that switches it to using that. Use feedback and what you
> learn from trying to integrate in those libraries to influence back the
> design of the API itself.
> 
> Creating a PEP and creating the library and the PRs can happen in parallel,
> but at least for pip if something deserves a PEP, we're not going to merge a
> PR until that PEP is generally agreed on. However it can be supremely useful
> to have them all going at the same time, because you run into things that
> you didn't really notice until you went to actually implement it.
> 
> My other big suggestion would be to e careful about how much you bite off at
> one time. Pip's internal code base is not the greatest, so pulling out
> smaller chunks at a time rather than trying to start right off pulling out a
> big topic is more likely to meet with success. Be cognizant of what the
> dependencies are for the feature you want to implement, because if it has
> dependencies, you'll need to pull them out first before you can pull it out
> OR you'll need to design the API to invert those dependencies so they get
> passed in instead.
> 
> I personally would be happy to at a minimum participate on any issue where
> someone was trying to split out some functionality from pip into a re-usable
> library if not follow the develop of that library directly to help guide it
> more closely. My hope for pip is that it ends up being the glue around a
> bunch of these libraries, and that it doesn't implement most of the stuff
> itself anymore.
> --
> Distutils-SIG mailing list -- distutils-sig@python.org 
> <mailto:distutils-sig@python.org>
> To unsubscribe send an email to distutils-sig-le...@python.org 
> <mailto:distutils-sig-le...@python.org>
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ 
> <https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/>
> Message archived at 
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/IQVZVVWX2BLEP6D4WQMKNXZHBF2NZINU/
>  
> <https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/IQVZVVWX2BLEP6D4WQMKNXZHBF2NZINU/>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/CIB4OWV2CWR3RVOPTS55WWL4ISFDGTSC/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Dan Ryan]

ompliant) by version and for finding any other executables
>> (cross platform)
>> https://github.com/sarugaku/virtenv -- python api for virtualenv creation
>> 
>> Happy to provide access or take advice as needed on any of those.  Thanks
>> all for the receptiveness and collaboration
>> 
>> Dan Ryan
>> gh: @techalchemy // e: d...@danryan.co
>> 
>> From: Donald Stufft [mailto:don...@stufft.io] 
>> Sent: Wednesday, September 19, 2018 1:52 PM
>> To: Tzu-ping Chung
>> Cc: Distutils
>> Subject: [Distutils] Re: Distlib vs Packaging (Was: disable building wheel
>> for a package)
>> 
>> My general recommendation if you want a Python implementation/interface for
>> something pip does, is:
>> 
>> - Open an issue on the pip repository to document your intent and to make
>> sure that there is nobody there who is against having that functionality
>> split out. This might also give a chance for people with familiarity in that
>> API to mention pain points that you can solve in a new API. We can also
>> probably give you a good sense if the thing you want in a library is
>> something that probably has multiple things that are dependent on getting
>> split out first (for instance, if you said you wanted a library for
>> installing wheels, we'd probably tell you that there is a dependency on PEP
>> 425 tags, pip locations, maybe other that need resolved first) and also
>> whether this is something that should have a PEP first or not. Getting some
>> rough agreement on the plan to split X thing out before you start is overall
>> a good thing.
>> 
>> - Create or update a PEP if required, and get it into the provisional state.
>> 
>> - Make the library, either as a PR to packaging or as it's own independent
>> library. If there are questions that come up while creating that library/PR
>> that have to do with specific pip behaviors, go back to that original issue
>> and ask for clarification etc. Ideally at some point you'll open a PR on pip
>> that uses the new library (my suggestion is to not bundle the library in the
>> initial PR, and just import it normally so that the PR diff doesn't include
>> the full bundled library until there's agreement on it). If there's another
>> tool (pipenv, whatever) that is looking to use that same functionality, open
>> a WIP PR there too that switches it to using that. Use feedback and what you
>> learn from trying to integrate in those libraries to influence back the
>> design of the API itself.
>> 
>> Creating a PEP and creating the library and the PRs can happen in parallel,
>> but at least for pip if something deserves a PEP, we're not going to merge a
>> PR until that PEP is generally agreed on. However it can be supremely useful
>> to have them all going at the same time, because you run into things that
>> you didn't really notice until you went to actually implement it.
>> 
>> My other big suggestion would be to e careful about how much you bite off at
>> one time. Pip's internal code base is not the greatest, so pulling out
>> smaller chunks at a time rather than trying to start right off pulling out a
>> big topic is more likely to meet with success. Be cognizant of what the
>> dependencies are for the feature you want to implement, because if it has
>> dependencies, you'll need to pull them out first before you can pull it out
>> OR you'll need to design the API to invert those dependencies so they get
>> passed in instead.
>> 
>> I personally would be happy to at a minimum participate on any issue where
>> someone was trying to split out some functionality from pip into a re-usable
>> library if not follow the develop of that library directly to help guide it
>> more closely. My hope for pip is that it ends up being the glue around a
>> bunch of these libraries, and that it doesn't implement most of the stuff
>> itself anymore.
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
>> Message archived at 
>> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/IQVZVVWX2BLEP6D4WQMKNXZHBF2NZINU/
> 
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/TWPILKETHEG3NFLIINFBWC7VTTKQ6BNJ/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Chris Jerdonek]

On Wed, Sep 19, 2018 at 8:54 PM, Dan Ryan  wrote:

> I should clarify that we have already implemented a number of these as
> libraries over the last several months (and I am super familiar with pip's
> internals by now and I'm sure TP is getting there as well). More on this
> below
> ...
> We are super cognizant of that aspect as I am pretty sure we are hitting
> this wall in a full (nearly) pip-free reimplementation of all of the pipenv
> internals from the ground up, including wheel building/installation, but we
> basically had to start by calling pip directly, then slowly reimplement
> each
> aspect of the underlying logic using various elements in distlib/setuptools
> or rebuilding those.
>

Is the hope or game plan then for pipenv not to have to depend on pip? This
is partly what I was trying to learn in my email to this list a month ago (on
Aug. 20, with subject: "pipenv and pip"):
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/2QECNWSHNEW7UBB24M2K5BISYJY7GMZF/

Based on the replies, I wasn't getting that impression at the time (though
I don't remember getting a clear answer), but maybe things have changed
since then.

It should certainly be a lot easier for pipenv to move fast since there is
no legacy base of users to maintain compatibility with. However, I worry
about the fracturing this will cause. In creating these libraries, from the
pip tracker it doesn't look like any effort is going into refactoring pip
to make use of them. This relates to the point I made earlier today about
how there won't be an easy way to cut pip over to using a new library unless
an effort is made from the beginning. Thus, it's looking like things could
be on track to split the user and maintainer base in two, with pip bearing
the legacy burden and perhaps not seeing the improvements. Are we okay with
that future?

--Chris


Since you mentioned following along, here's what we're working on right now:
>
> https://github.com/sarugaku/requirementslib -- abstraction layer for
> parsing
> and converting various requirements formats
> (pipfile/requirements.txt/command line/InstallRequirement) and moving
> between all of them
> https://github.com/sarugaku/resolvelib -- directed acyclic graph library
> for
> handling dependency resolution (not yet being used in pipenv)
> https://github.com/sarugaku/passa -- dependency resolver/installer/pipfile
> manager (bulk of the logic we have been talking about is in here right now)
> -- I think we will probably split this back out into multiple other smaller
> libraries or something based on the discussion
> https://github.com/sarugaku/plette -- this is a rewrite of pipfile with
> some
> additional logic / validation
> https://github.com/sarugaku/shellingham -- this is a shell detection
> library
> made up of some tooling we built in pipenv for environment detection
> https://github.com/sarugaku/pythonfinder -- this is a library for finding
> python (pep 514 compliant) by version and for finding any other executables
> (cross platform)
> https://github.com/sarugaku/virtenv -- python api for virtualenv creation
>
> Happy to provide access or take advice as needed on any of those.  Thanks
> all for the receptiveness and collaboration
>
> Dan Ryan
> gh: @techalchemy // e: d...@danryan.co
>
> From: Donald Stufft [mailto:don...@stufft.io]
> Sent: Wednesday, September 19, 2018 1:52 PM
> To: Tzu-ping Chung
> Cc: Distutils
> Subject: [Distutils] Re: Distlib vs Packaging (Was: disable building wheel
> for a package)
>
> My general recommendation if you want a Python implementation/interface for
> something pip does, is:
>
> - Open an issue on the pip repository to document your intent and to make
> sure that there is nobody there who is against having that functionality
> split out. This might also give a chance for people with familiarity in
> that
> API to mention pain points that you can solve in a new API. We can also
> probably give you a good sense if the thing you want in a library is
> something that probably has multiple things that are dependent on getting
> split out first (for instance, if you said you wanted a library for
> installing wheels, we'd probably tell you that there is a dependency on PEP
> 425 tags, pip locations, maybe other that need resolved first) and also
> whether this is something that should have a PEP first or not. Getting some
> rough agreement on the plan to split X thing out before you start is
> overall
> a good thing.
>
> - Create or update a PEP if required, and get it into the provisional
> state.
>
> - Make the library, either as a PR to packaging or as it's own independent
> library. If there are questions that come up while creating that library/PR
> that have to do with specific pip behaviors, go ba

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Dan Ryan]

I should clarify that we have already implemented a number of these as
libraries over the last several months (and I am super familiar with pip's
internals by now and I'm sure TP is getting there as well). More on this
below

> This might also give a chance for people with familiarity in that API to
mention pain points that you can solve in a new API. We can also probably
give you a good sense if the thing you want in a library is something that
probably has multiple things that are dependent on getting split out first

This would be super helpful, although there is a decent chance we can make
some initial headway on this aspect of it just with the first pass agreement

> Make the library, either as a PR to packaging or as it's own independent
library

Basically the entire InstallRequirement model is the single most imported
item (in my experience) from pip's internals and it is (almost) never used
for installing, but often just for metadata access/normalization/parsing.  I
reimplemented the parsing logic for pipenv  in 'requirementslib' (link
below)

> pulling out smaller chunks at a time rather than trying to start right off
pulling out a big topic is more likely to meet with success. Be cognizant of
what the dependencies are for the feature you want to implement, because if
it has dependencies, you'll need to pull them out first before you can pull
it out OR you'll need to design the API to invert those dependencies so they
get passed in instead

We are super cognizant of that aspect as I am pretty sure we are hitting
this wall in a full (nearly) pip-free reimplementation of all of the pipenv
internals from the ground up, including wheel building/installation, but we
basically had to start by calling pip directly, then slowly reimplement each
aspect of the underlying logic using various elements in distlib/setuptools
or rebuilding those.

Since you mentioned following along, here's what we're working on right now:

https://github.com/sarugaku/requirementslib -- abstraction layer for parsing
and converting various requirements formats
(pipfile/requirements.txt/command line/InstallRequirement) and moving
between all of them
https://github.com/sarugaku/resolvelib -- directed acyclic graph library for
handling dependency resolution (not yet being used in pipenv)
https://github.com/sarugaku/passa -- dependency resolver/installer/pipfile
manager (bulk of the logic we have been talking about is in here right now)
-- I think we will probably split this back out into multiple other smaller
libraries or something based on the discussion
https://github.com/sarugaku/plette -- this is a rewrite of pipfile with some
additional logic / validation
https://github.com/sarugaku/shellingham -- this is a shell detection library
made up of some tooling we built in pipenv for environment detection
https://github.com/sarugaku/pythonfinder -- this is a library for finding
python (pep 514 compliant) by version and for finding any other executables
(cross platform)
https://github.com/sarugaku/virtenv -- python api for virtualenv creation

Happy to provide access or take advice as needed on any of those.  Thanks
all for the receptiveness and collaboration

Dan Ryan
gh: @techalchemy // e: d...@danryan.co

From: Donald Stufft [mailto:don...@stufft.io] 
Sent: Wednesday, September 19, 2018 1:52 PM
To: Tzu-ping Chung
Cc: Distutils
Subject: [Distutils] Re: Distlib vs Packaging (Was: disable building wheel
for a package)

My general recommendation if you want a Python implementation/interface for
something pip does, is:

- Open an issue on the pip repository to document your intent and to make
sure that there is nobody there who is against having that functionality
split out. This might also give a chance for people with familiarity in that
API to mention pain points that you can solve in a new API. We can also
probably give you a good sense if the thing you want in a library is
something that probably has multiple things that are dependent on getting
split out first (for instance, if you said you wanted a library for
installing wheels, we'd probably tell you that there is a dependency on PEP
425 tags, pip locations, maybe other that need resolved first) and also
whether this is something that should have a PEP first or not. Getting some
rough agreement on the plan to split X thing out before you start is overall
a good thing.

- Create or update a PEP if required, and get it into the provisional state.

- Make the library, either as a PR to packaging or as it's own independent
library. If there are questions that come up while creating that library/PR
that have to do with specific pip behaviors, go back to that original issue
and ask for clarification etc. Ideally at some point you'll open a PR on pip
that uses the new library (my suggestion is to not bundle the library in the
initial PR, and just import it normally so that the PR diff doesn't include
the full bundled library until there's agreement on it). If there's ano

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Brett Cannon]

On Wed, 19 Sep 2018 at 11:41 Paul Moore  wrote:

> On Wed, 19 Sep 2018 at 18:52, Donald Stufft  wrote:
> >
> > On Sep 19, 2018, at 1:14 PM, Tzu-ping Chung  wrote:
> >
> > I feel the plan is quite solid. This however leaves us (who want a
> Python implementation and interface to do what pip does) in an interesting
> place. So I can tell there are a couple of principles:
> >
> > 1. Do not use pip internals
> > 2. pip won’t be using either distlib or setuptools, so they might not
> match what pip does, in the long run
> >
> > Does this leaves us only one option left, to implement a library that
> matches what pip does (follows the standards), but is not pip? That feels
> quite counter-productive to me, but if it’s what things would be, I’d
> accept it.
> >
> > The next step (for me) in that case would then be to start working on
> that library. Since existing behaviours in setuptools and pip (including
> the part it uses distlib for) are likely to be standardised, I can rely on
> distlib for script creation, setuptools for some miscellaneous things
> (editable installs?), and pull (or reimplement) parts out of pip for
> others. Are there caveats I should look out?
> >
> >
> > My general recommendation if you want a Python implementation/interface
> for something pip does, is:
> >
> > - Open an issue on the pip repository to document your intent and to
> make sure that there is nobody there who is against having that
> functionality split out. This might also give a chance for people with
> familiarity in that API to mention pain points that you can solve in a new
> API. We can also probably give you a good sense if the thing you want in a
> library is something that probably has multiple things that are dependent
> on getting split out first (for instance, if you said you wanted a library
> for installing wheels, we’d probably tell you that there is a dependency on
> PEP 425 tags, pip locations, maybe other that need resolved first) and also
> whether this is something that should have a PEP first or not. Getting some
> rough agreement on the plan to split X thing out before you start is
> overall a good thing.
> >
> > - Create or update a PEP if required, and get it into the provisional
> state.
> >
> > - Make the library, either as a PR to packaging or as it’s own
> independent library. If there are questions that come up while creating
> that library/PR that have to do with specific pip behaviors, go back to
> that original issue and ask for clarification etc. Ideally at some point
> you’ll open a PR on pip that uses the new library (my suggestion is to not
> bundle the library in the initial PR, and just import it normally so that
> the PR diff doesn’t include the full bundled library until there’s
> agreement on it). If there’s another tool (pipenv, whatever) that is
> looking to use that same functionality, open a WIP PR there too that
> switches it to using that. Use feedback and what you learn from trying to
> integrate in those libraries to influence back the design of the API itself.
> >
> > Creating a PEP and creating the library and the PRs can happen in
> parallel, but at least for pip if something deserves a PEP, we’re not going
> to merge a PR until that PEP is generally agreed on. However it can be
> supremely useful to have them all going at the same time, because you run
> into things that you didn’t really notice until you went to actually
> implement it.
> >
> > My other big suggestion would be to e careful about how much you bite
> off at one time. Pip’s internal code base is not the greatest, so pulling
> out smaller chunks at a time rather than trying to start right off pulling
> out a big topic is more likely to meet with success. Be cognizant of what
> the dependencies are for the feature you want to implement, because if it
> has dependencies, you’ll need to pull them out first before you can pull it
> out OR you’ll need to design the API to invert those dependencies so they
> get passed in instead.
> >
> > I personally would be happy to at a minimum participate on any issue
> where someone was trying to split out some functionality from pip into a
> re-usable library if not follow the develop of that library directly to
> help guide it more closely. My hope for pip is that it ends up being the
> glue around a bunch of these libraries, and that it doesn’t implement most
> of the stuff itself anymore.
>
> I basically agree with everything Donald said, and I'd also be happy
> to support any work along these lines. If you're looking for a place
> to start, I'd strongly recommend some of the foundational areas -
> something like pip.locations (I know there are others who have
> expressed an interest in this PI being exposed), or pep425tags which
> has the advantage of already having a standard, or something at that
> level. Starting with something at the level of the finder or the
> installer is likely to be way too much to start with, even if it feels
> like it would be more directly 

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Paul Moore]

On Wed, 19 Sep 2018 at 18:52, Donald Stufft  wrote:
>
> On Sep 19, 2018, at 1:14 PM, Tzu-ping Chung  wrote:
>
> I feel the plan is quite solid. This however leaves us (who want a Python 
> implementation and interface to do what pip does) in an interesting place. So 
> I can tell there are a couple of principles:
>
> 1. Do not use pip internals
> 2. pip won’t be using either distlib or setuptools, so they might not match 
> what pip does, in the long run
>
> Does this leaves us only one option left, to implement a library that matches 
> what pip does (follows the standards), but is not pip? That feels quite 
> counter-productive to me, but if it’s what things would be, I’d accept it.
>
> The next step (for me) in that case would then be to start working on that 
> library. Since existing behaviours in setuptools and pip (including the part 
> it uses distlib for) are likely to be standardised, I can rely on distlib for 
> script creation, setuptools for some miscellaneous things (editable 
> installs?), and pull (or reimplement) parts out of pip for others. Are there 
> caveats I should look out?
>
>
> My general recommendation if you want a Python implementation/interface for 
> something pip does, is:
>
> - Open an issue on the pip repository to document your intent and to make 
> sure that there is nobody there who is against having that functionality 
> split out. This might also give a chance for people with familiarity in that 
> API to mention pain points that you can solve in a new API. We can also 
> probably give you a good sense if the thing you want in a library is 
> something that probably has multiple things that are dependent on getting 
> split out first (for instance, if you said you wanted a library for 
> installing wheels, we’d probably tell you that there is a dependency on PEP 
> 425 tags, pip locations, maybe other that need resolved first) and also 
> whether this is something that should have a PEP first or not. Getting some 
> rough agreement on the plan to split X thing out before you start is overall 
> a good thing.
>
> - Create or update a PEP if required, and get it into the provisional state.
>
> - Make the library, either as a PR to packaging or as it’s own independent 
> library. If there are questions that come up while creating that library/PR 
> that have to do with specific pip behaviors, go back to that original issue 
> and ask for clarification etc. Ideally at some point you’ll open a PR on pip 
> that uses the new library (my suggestion is to not bundle the library in the 
> initial PR, and just import it normally so that the PR diff doesn’t include 
> the full bundled library until there’s agreement on it). If there’s another 
> tool (pipenv, whatever) that is looking to use that same functionality, open 
> a WIP PR there too that switches it to using that. Use feedback and what you 
> learn from trying to integrate in those libraries to influence back the 
> design of the API itself.
>
> Creating a PEP and creating the library and the PRs can happen in parallel, 
> but at least for pip if something deserves a PEP, we’re not going to merge a 
> PR until that PEP is generally agreed on. However it can be supremely useful 
> to have them all going at the same time, because you run into things that you 
> didn’t really notice until you went to actually implement it.
>
> My other big suggestion would be to e careful about how much you bite off at 
> one time. Pip’s internal code base is not the greatest, so pulling out 
> smaller chunks at a time rather than trying to start right off pulling out a 
> big topic is more likely to meet with success. Be cognizant of what the 
> dependencies are for the feature you want to implement, because if it has 
> dependencies, you’ll need to pull them out first before you can pull it out 
> OR you’ll need to design the API to invert those dependencies so they get 
> passed in instead.
>
> I personally would be happy to at a minimum participate on any issue where 
> someone was trying to split out some functionality from pip into a re-usable 
> library if not follow the develop of that library directly to help guide it 
> more closely. My hope for pip is that it ends up being the glue around a 
> bunch of these libraries, and that it doesn’t implement most of the stuff 
> itself anymore.

I basically agree with everything Donald said, and I'd also be happy
to support any work along these lines. If you're looking for a place
to start, I'd strongly recommend some of the foundational areas -
something like pip.locations (I know there are others who have
expressed an interest in this PI being exposed), or pep425tags which
has the advantage of already having a standard, or something at that
level. Starting with something at the level of the finder or the
installer is likely to be way too much to start with, even if it feels
like it would be more directly useful to you.

Paul
--
Distutils-SIG mailing list -- distutils-sig@python.org
To 

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Chris Jerdonek]

On Wed, Sep 19, 2018 at 10:14 AM, Tzu-ping Chung  wrote:
> I feel the plan is quite solid. This however leaves us (who want a Python
> implementation and interface to do what pip does) in an interesting place.
> So I can tell there are a couple of principles:
>
> 1. Do not use pip internals
> 2. pip won’t be using either distlib or setuptools, so they might not match
> what pip does, in the long run
>
> Does this leaves us only one option left, to implement a library that
> matches what pip does (follows the standards), but is not pip? That feels
> quite counter-productive to me, but if it’s what things would be, I’d accept
> it.
>
> The next step (for me) in that case would then be to start working on that
> library.

From what Donald said, the first step is to standardize some chunk of
functionality into a PEP (for functionality that doesn't already have
a PEP). If you start the library before standardizing via a PEP, it
seems like you're just going to exacerbate the problem by creating yet
another library with its own implementation variations, introduce new
compatibility problems for the future, etc. And a PEP will still need
to be created later, anyways.

For cases where a PEP already exists, I would also encourage you to
add to the library only as you are having pip start to use it.
Otherwise, there won't be an easy way to cut pip over to using it,
without risking major breakage. This process would also help ensure
the implementation matches pip's behavior.

--Chris


> Since existing behaviours in setuptools and pip (including the part
> it uses distlib for) are likely to be standardised, I can rely on distlib
> for script creation, setuptools for some miscellaneous things (editable
> installs?), and pull (or reimplement) parts out of pip for others. Are there
> caveats I should look out?
>
> TP
>
> --
> Tzu-ping Chung (@uranusjr)
> uranu...@gmail.com
> Sent from my iPhone
>
> On 20 Sep 2018, at 00:39, Donald Stufft  wrote:
>
>
> On Sep 19, 2018, at 5:17 AM, Paul Moore  wrote:
>
>
>
> What is the current situation regarding distlib vs packaging and various
> pieces in pip? Many parts of distlib seems to have duplicates in either
> packaging or pip/setuptools internals. I understand this is a historical
> artifact, but what is the plan going forward, and what strategy, if any,
> should a person take if they are to make the attempt of merging, or
> collecting pieces from existing code bases into a workable library?
>
>
> Note: This is my personal view of the history only, Vinay and Donald
> would be better able to give definitive answers
>
>
> I’ve personally always planned on pulling out the last bits of what we do
> use distlib for in pip, and not relying on it any longer.
>
> My general plan for extracting stuff from pip and/or setuptools has always
> been to first standardize in a PEP (if a sufficient one doesn’t already
> exist) anything that makes sense to be as a standard, and then start either
> reimplementing or pulling code out of pip (or setuptools if pip is using
> setuptools). When doing that I had always planned on spending a lot of
> effort ensuring that the behavior matches what pip is already doing (or have
> known, specific divergences).
>
> Now some of this already exists in distlib, but I don’t plan on using it.
> Part of that is because I find it easier to identify things that should be
> standardized but aren’t if I’m not using a big bundle of already implemented
> stuff already (for instance, script support needs to be standardized, but it
> didn’t occur to us at the time because we just used what distlib had). It
> also has a bunch of functionality that exists only in distlib, like
> attempting to use JSON to find packages (at one point there was even a
> locator that implicitly used a non PyPI server, no idea if there still is)
> that I felt made it harder to use the library in a way that didn’t basically
> create a new set of implementation defined semantics. It also copied APIs
> and semantics that I think *shouldn’t* have been (for instance, it has an
> implicitly caching resource API like setuptools does… a generally bad idea
> IMO, whereas using the new importlib.resources is a much saner API).
>
> So in general, there are things that currently only exist in distlib or
> setuptools, but my personal long term plan for pip is that we should get
> solid implementations of those things out of those libraries, but generally
> my mind puts distlib and setuptools in largely the same boat.
>
>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/A42FTYI2Y3TEMEJK7VW25DBVLEAPYYYE/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Donald Stufft]

> On Sep 19, 2018, at 1:14 PM, Tzu-ping Chung  wrote:
> 
> I feel the plan is quite solid. This however leaves us (who want a Python 
> implementation and interface to do what pip does) in an interesting place. So 
> I can tell there are a couple of principles:
> 
> 1. Do not use pip internals
> 2. pip won’t be using either distlib or setuptools, so they might not match 
> what pip does, in the long run
> 
> Does this leaves us only one option left, to implement a library that matches 
> what pip does (follows the standards), but is not pip? That feels quite 
> counter-productive to me, but if it’s what things would be, I’d accept it.
> 
> The next step (for me) in that case would then be to start working on that 
> library. Since existing behaviours in setuptools and pip (including the part 
> it uses distlib for) are likely to be standardised, I can rely on distlib for 
> script creation, setuptools for some miscellaneous things (editable 
> installs?), and pull (or reimplement) parts out of pip for others. Are there 
> caveats I should look out?

My general recommendation if you want a Python implementation/interface for 
something pip does, is:

- Open an issue on the pip repository to document your intent and to make sure 
that there is nobody there who is against having that functionality split out. 
This might also give a chance for people with familiarity in that API to 
mention pain points that you can solve in a new API. We can also probably give 
you a good sense if the thing you want in a library is something that probably 
has multiple things that are dependent on getting split out first (for 
instance, if you said you wanted a library for installing wheels, we’d probably 
tell you that there is a dependency on PEP 425 tags, pip locations, maybe other 
that need resolved first) and also whether this is something that should have a 
PEP first or not. Getting some rough agreement on the plan to split X thing out 
before you start is overall a good thing.

- Create or update a PEP if required, and get it into the provisional state.

- Make the library, either as a PR to packaging or as it’s own independent 
library. If there are questions that come up while creating that library/PR 
that have to do with specific pip behaviors, go back to that original issue and 
ask for clarification etc. Ideally at some point you’ll open a PR on pip that 
uses the new library (my suggestion is to not bundle the library in the initial 
PR, and just import it normally so that the PR diff doesn’t include the full 
bundled library until there’s agreement on it). If there’s another tool 
(pipenv, whatever) that is looking to use that same functionality, open a WIP 
PR there too that switches it to using that. Use feedback and what you learn 
from trying to integrate in those libraries to influence back the design of the 
API itself.

Creating a PEP and creating the library and the PRs can happen in parallel, but 
at least for pip if something deserves a PEP, we’re not going to merge a PR 
until that PEP is generally agreed on. However it can be supremely useful to 
have them all going at the same time, because you run into things that you 
didn’t really notice until you went to actually implement it.

My other big suggestion would be to e careful about how much you bite off at 
one time. Pip’s internal code base is not the greatest, so pulling out smaller 
chunks at a time rather than trying to start right off pulling out a big topic 
is more likely to meet with success. Be cognizant of what the dependencies are 
for the feature you want to implement, because if it has dependencies, you’ll 
need to pull them out first before you can pull it out OR you’ll need to design 
the API to invert those dependencies so they get passed in instead.

I personally would be happy to at a minimum participate on any issue where 
someone was trying to split out some functionality from pip into a re-usable 
library if not follow the develop of that library directly to help guide it 
more closely. My hope for pip is that it ends up being the glue around a bunch 
of these libraries, and that it doesn’t implement most of the stuff itself 
anymore.--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/UX5JCA6JZW5YHJGJ7BYI4GL2CU3S6Y2D/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Tzu-ping Chung]

I feel the plan is quite solid. This however leaves us (who want a Python 
implementation and interface to do what pip does) in an interesting place. So I 
can tell there are a couple of principles:

1. Do not use pip internals
2. pip won’t be using either distlib or setuptools, so they might not match 
what pip does, in the long run

Does this leaves us only one option left, to implement a library that matches 
what pip does (follows the standards), but is not pip? That feels quite 
counter-productive to me, but if it’s what things would be, I’d accept it.

The next step (for me) in that case would then be to start working on that 
library. Since existing behaviours in setuptools and pip (including the part it 
uses distlib for) are likely to be standardised, I can rely on distlib for 
script creation, setuptools for some miscellaneous things (editable installs?), 
and pull (or reimplement) parts out of pip for others. Are there caveats I 
should look out?

TP

--
Tzu-ping Chung (@uranusjr)
uranu...@gmail.com
Sent from my iPhone

> On 20 Sep 2018, at 00:39, Donald Stufft  wrote:
> 
> 
>>> On Sep 19, 2018, at 5:17 AM, Paul Moore  wrote:
>>> 
>>> 
>>> 
>>> What is the current situation regarding distlib vs packaging and various 
>>> pieces in pip? Many parts of distlib seems to have duplicates in either 
>>> packaging or pip/setuptools internals. I understand this is a historical 
>>> artifact, but what is the plan going forward, and what strategy, if any, 
>>> should a person take if they are to make the attempt of merging, or 
>>> collecting pieces from existing code bases into a workable library?
>> 
>> Note: This is my personal view of the history only, Vinay and Donald
>> would be better able to give definitive answers
> 
> I’ve personally always planned on pulling out the last bits of what we do use 
> distlib for in pip, and not relying on it any longer.
> 
> My general plan for extracting stuff from pip and/or setuptools has always 
> been to first standardize in a PEP (if a sufficient one doesn’t already 
> exist) anything that makes sense to be as a standard, and then start either 
> reimplementing or pulling code out of pip (or setuptools if pip is using 
> setuptools). When doing that I had always planned on spending a lot of effort 
> ensuring that the behavior matches what pip is already doing (or have known, 
> specific divergences).
> 
> Now some of this already exists in distlib, but I don’t plan on using it. 
> Part of that is because I find it easier to identify things that should be 
> standardized but aren’t if I’m not using a big bundle of already implemented 
> stuff already (for instance, script support needs to be standardized, but it 
> didn’t occur to us at the time because we just used what distlib had). It 
> also has a bunch of functionality that exists only in distlib, like 
> attempting to use JSON to find packages (at one point there was even a 
> locator that implicitly used a non PyPI server, no idea if there still is) 
> that I felt made it harder to use the library in a way that didn’t basically 
> create a new set of implementation defined semantics. It also copied APIs and 
> semantics that I think *shouldn’t* have been (for instance, it has an 
> implicitly caching resource API like setuptools does… a generally bad idea 
> IMO, whereas using the new importlib.resources is a much saner API).
> 
> So in general, there are things that currently only exist in distlib or 
> setuptools, but my personal long term plan for pip is that we should get 
> solid implementations of those things out of those libraries, but generally 
> my mind puts distlib and setuptools in largely the same boat.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/A42FTYI2Y3TEMEJK7VW25DBVLEAPYYYE/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Donald Stufft]

> On Sep 19, 2018, at 5:17 AM, Paul Moore  wrote:
> 
>> 
>> 
>> What is the current situation regarding distlib vs packaging and various 
>> pieces in pip? Many parts of distlib seems to have duplicates in either 
>> packaging or pip/setuptools internals. I understand this is a historical 
>> artifact, but what is the plan going forward, and what strategy, if any, 
>> should a person take if they are to make the attempt of merging, or 
>> collecting pieces from existing code bases into a workable library?
> 
> Note: This is my personal view of the history only, Vinay and Donald
> would be better able to give definitive answers

I’ve personally always planned on pulling out the last bits of what we do use 
distlib for in pip, and not relying on it any longer.

My general plan for extracting stuff from pip and/or setuptools has always been 
to first standardize in a PEP (if a sufficient one doesn’t already exist) 
anything that makes sense to be as a standard, and then start either 
reimplementing or pulling code out of pip (or setuptools if pip is using 
setuptools). When doing that I had always planned on spending a lot of effort 
ensuring that the behavior matches what pip is already doing (or have known, 
specific divergences).

Now some of this already exists in distlib, but I don’t plan on using it. Part 
of that is because I find it easier to identify things that should be 
standardized but aren’t if I’m not using a big bundle of already implemented 
stuff already (for instance, script support needs to be standardized, but it 
didn’t occur to us at the time because we just used what distlib had). It also 
has a bunch of functionality that exists only in distlib, like attempting to 
use JSON to find packages (at one point there was even a locator that 
implicitly used a non PyPI server, no idea if there still is) that I felt made 
it harder to use the library in a way that didn’t basically create a new set of 
implementation defined semantics. It also copied APIs and semantics that I 
think *shouldn’t* have been (for instance, it has an implicitly caching 
resource API like setuptools does… a generally bad idea IMO, whereas using the 
new importlib.resources is a much saner API).

So in general, there are things that currently only exist in distlib or 
setuptools, but my personal long term plan for pip is that we should get solid 
implementations of those things out of those libraries, but generally my mind 
puts distlib and setuptools in largely the same boat.--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/2DTOA7EA4ZSL5442G33TVCMCNP7ZNGGS/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Paul Moore]

On Wed, 19 Sep 2018 at 11:34, Tzu-ping Chung  wrote:
>
> I have the same experience with Pipenv as Nick’s. I would also guess
> another reason is the lack of knowledge—this is certainly my own before
> I get involved in Pipenv. There is barely any guide on how I should
> implement such a thing, and my developer’s instinct would tell me to look
> at a known implementation, i.e. pip. This also ties back to the problem that
> pip uses distlib internally barely at all—had it used more of it, people might
> be pointed to the right direction.
>
> Migrating Pipenv’s internals from pip instead of distlib is actually the exact
> thing I am thinking about when I raised the question. There are, as mentioned,
> a lot of pieces missing in distlib. For example, distlib knows how to find a
> distribution, and how to install wheels, but not how a non-wheel distribution
> can be turned into a wheel. [1] It also has no functionalities on 
> uninstallation.
> If I’m to glue together a working thing, I would likely need to 
> copy/reimplement
> parts of pip, but where should they live? Do I add yet another layer above
> distlib to include them, or do I try to include them in distlib?

That's probably Vinay's call. IMO, whatever layer there is above
packaging, it should do stuff like this.

> Although distlib provides a nice basis, I feel it is still one layer below 
> what most
> people want to do, e.g. install a thing by name (or URL). But would a 
> three-layer
> design be too much, or should distlib have a high-level API as well?

I don't think a 3-layer approach is sensible, two layers is more than
enough. Maybe not go all the way to "install something by name". Maybe
APIs to

* find what's available in a set of indexes/locations
* check if a compatible wheel is in that list
* make a wheel from a sdist
* install a wheel

It's hard to make something like this sound like anything other than
put "pip into a library and make pip itself a thin shell round that
library" though. At which point we come full circle round to the fact
that the *reason* we don't support a pip API is that the cost of
designing one, restructuring pip to expose (and use!) it, and creating
documentation and tests to make sure that API is stable and properly
supported, is huge. And we don't have the resources.

It's not so much a technical/design or layering issue, it's really a
resourcing and sustainability issue.

> [1]: Also, while I’m having your attention—I’m trying to use the pep517 
> library
> as part of the solution to build an sdist into wheel, but I’m hitting a bug. 
> Could
> you help review my PR? :p https://github.com/pypa/pep517/pull/15

It's not really me that can comment, you probably need Thomas. To
clarify, even though I have commit rights on the pep517 project, I'm
not really a project maintainer there. My interest in it is as a
consumer from pip, which only uses the hook wrapper code. So this PR
is in something I've not used or looked at closely. Pip does its own
build isolation, ironically enough :-(

Paul
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/HGL5BDYSOCPNKI3Q4N3GO4KMO23LGGVX/

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Tzu-ping Chung]

I have the same experience with Pipenv as Nick’s. I would also guess
another reason is the lack of knowledge—this is certainly my own before
I get involved in Pipenv. There is barely any guide on how I should
implement such a thing, and my developer’s instinct would tell me to look
at a known implementation, i.e. pip. This also ties back to the problem that
pip uses distlib internally barely at all—had it used more of it, people might
be pointed to the right direction.

Migrating Pipenv’s internals from pip instead of distlib is actually the exact
thing I am thinking about when I raised the question. There are, as mentioned,
a lot of pieces missing in distlib. For example, distlib knows how to find a
distribution, and how to install wheels, but not how a non-wheel distribution
can be turned into a wheel. [1] It also has no functionalities on 
uninstallation.
If I’m to glue together a working thing, I would likely need to copy/reimplement
parts of pip, but where should they live? Do I add yet another layer above
distlib to include them, or do I try to include them in distlib?

Although distlib provides a nice basis, I feel it is still one layer below what 
most
people want to do, e.g. install a thing by name (or URL). But would a 
three-layer
design be too much, or should distlib have a high-level API as well?


[1]: Also, while I’m having your attention—I’m trying to use the pep517 library
as part of the solution to build an sdist into wheel, but I’m hitting a bug. 
Could
you help review my PR? :p https://github.com/pypa/pep517/pull/15 



TP


> On 19/9, 2018, at 17:17, Paul Moore  wrote:
> 
> On Wed, 19 Sep 2018 at 09:39, Tzu-ping Chung  wrote:
>> Risking thread hijacking, I want to take this chance and ask about one 
>> particular multiple implementation problem I found recently.
> 
> I changed the subject to keep things easier to follow. Hope that's OK.
> 
>> 
>> What is the current situation regarding distlib vs packaging and various 
>> pieces in pip? Many parts of distlib seems to have duplicates in either 
>> packaging or pip/setuptools internals. I understand this is a historical 
>> artifact, but what is the plan going forward, and what strategy, if any, 
>> should a person take if they are to make the attempt of merging, or 
>> collecting pieces from existing code bases into a workable library?
> 
> Note: This is my personal view of the history only, Vinay and Donald
> would be better able to give definitive answers
> 
>> 
>> From what I can tell (very limited), distlib seems to contain a good 
>> baseline design of a library fulfilling the intended purpose, but is 
>> currently missing parts to be fully usable on its own. Would it be a good 
>> idea to extend it with picked parts from pip? Should I contribute directly 
>> to it, or make a (higher level) wrapper around it with those parts? Should I 
>> actually use parts from it, or from other projects (e.g. distlib.version vs 
>> packaging.version, distlib.locator or pip’s PackageFinder)? It would be 
>> extremely helpful if there is a somewhat general, high-level view to the 
>> whole situation.
> 
> Distlib was created as a place to experiment with making a
> library-style interface to various pieces of packaging functionality.
> At the time it was created, there were not many standardised parts of
> the packaging ecosystem, so while it followed the standards where they
> existed, it also implemented a number of pieces of functionality that
> *weren't* backed by standards (obvious examples being the script
> creation stuff and the package finder).
> 
> Packaging, in the other hand, was designed to focus strictly on
> implementations of agreed standards, providing reference APIs for
> projects to use.
> 
> Pip uses both libraries, but as far as I'm aware, we'd use an API from
> packaging in preference to distlib. The only distlib API we use is the
> script maker API. Pretty much everything else in distlib, we already
> had an internal implementation for by the time distlib was written, so
> there was no benefit in changing (in contrast, the benefit in
> switching to packaging is "by design conformance to the relevant
> standards").
> 
> My recommendations would be:
> 
> 1. Use packaging APIs always where they exist, even if a distlib
> equivalent exists.
> 2. Never use pip APIs, they are internal use only (Paul bangs on that
> old drum again :-))
> 3. Consider using distlib APIs for things like the locator API,
> because it's better than writing your own code, but be aware of the
> risks.
> 
> When I say risks here, the things I'd consider are:
> 
> * Distlibs APIs aren't used in many projects, so they are likely less
> well tested (that's a chicken and egg issue, people need to use them
> to make them better).
> * Be aware that they may not behave identically to pip outside of the
> standardised areas.
> * In particular, review the details of which locators you use - the
> default is very different 

[Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

[Paul Moore]

On Wed, 19 Sep 2018 at 10:54, Nick Coghlan  wrote:
> Given that, and assuming Vinay is amenable to the idea, it would be
> nice to revisit the concept of the two layer architecture, with
> packaging as the lower level minimalist strictly standards compliant
> layer, and distlib as the higher level general purpose toolkit that
> brings together various other libraries (including packaging itself)
> under a more comprehensive API.

I'd certainly be OK with delegating more of the "common activities" to
distlib. But it's a *long* way from being simple to do so, and pip
would need to take a lot of care to ensure that doing so didn't result
in behavioural differences. Also, we'd need to be careful of dumping
too much on distlib without making the sustainability problem even
worse - at the moment, as far as I know, Vinay is the sole distlib
developer.

Paul
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/V5JQID7LDAIDZT2FCSIDEEIHS6FFXWAW/