On Tue, Sep 25, 2018 at 6:30 PM, Dan Ryan <d...@danryan.co> wrote: > Thanks Chris, these are excellent suggestions. Can you provide a reference to > a PR you’re working or have worked on so we can jumpstart the process on our > end?
Hi Dan, The PR's I've been submitting and working on aren't *primarily* of the "split functionality out into stand-alone bits" variety. But there are some. Two such recent PR's of mine that fit that category are-- 1) one from July (merged) to split out and add split_auth_from_netloc() to misc.py: https://github.com/pypa/pip/pull/5657 2) the one I mentioned in an email yesterday and that I just submitted (not yet merged), which splits out a function I called make_vcs_requirement_url() and adds it to misc.py: https://github.com/pypa/pip/pull/5810 My PR's have mostly been limited to the VCS part of the code: refactoring, fixing bugs, enhancements, etc. My advice to you (and which I've been following myself) would be for you to break changes into small, tightly-focused PR's, and to have only one outstanding at a time (or perhaps two if non-overlapping). The main reason is that there aren't too many people reviewing and merging changes (mainly just Pradyun in my case), so I don't want to overwhelm them. I've been trying to follow the maxim "go slow to go fast." Also, plan to expect / be okay with a fair of amount of preliminary refactoring PR's before splitting something out might even be possible. I.e. you will probably need a lot of "set up," especially if it's a gnarlier area of the code base. --Chris > > Dan Ryan // pipenv maintainer > gh: @techalchemy > >> On Sep 25, 2018, at 5:47 PM, Chris Jerdonek <chris.jerdo...@gmail.com> wrote: >> >>> On Tue, Sep 25, 2018 at 6:46 AM, Dan Ryan <d...@danryan.co> wrote: >>> As far as the pipenv roadmap, we are rewriting a good chunk of it but it >>> essentially the atomic things we do or would like to do with regards to >>> shared functionality at a high level: >>> >>> - parse user input/requirements >>> - retrieve package metadata >>> - resolve dependencies (not in pip right now obviously) — may involve >>> building wheels/sdists >>> - download / build packages >>> - install packages (into a virtualenv sometimes) >>> - uninstall packages (from a virtualenv sometimes) >>> - list packages etc (in a virtualenv sometimes) >>> >>> We know how these are accomplished in pip and we know how we are handling >>> them both in pipenv and in our alternative implementations, which I believe >>> still have some dependency on pip. For us the roadmap is basically to >>> figure out >>> >>> - where do the abstractions live >>> - how do we start >>> - ??? >> >> Thanks, Dan. I think Donald answered this in an email much earlier in >> this thread -- the one he sent on Sept. 19 beginning: >> >> """ >> My general recommendation if you want a Python >> implementation/interface for something pip does, is: >> - Open an issue on the pip repository to document your intent and to >> make sure that there is nobody there who is against having that >> functionality split out. ... >> ... >> """ >> >> In addition, to ensure you're taking the approach of splitting out >> parts of pip instead of creating yet another implementation separate >> from pip (leading to duplicated work as I said), I would recommend >> also doing the following: >> >> File PR's in pip to refactor out various functionality that makes >> sense into stand-alone bits with their own tests (e.g. unit tests, >> etc). This would live in pip but be disentangled from the rest of pip >> from a code dependency perspective. It could even be in a separate >> "future libraries" subdirectory if the pip committers are okay with >> it. (Optional: since you're already vendoring pip, you could perhaps >> use this factored out code at your own risk, like you're already doing >> with some of pip's code base anyway.) >> >> The above is something I started doing at a small scale -- not to >> break it out into a library, but for its own sake because it makes it >> easier to maintain and work on pip as a code base, fixing bugs, etc. >> >> Some of the nice things about this approach are that-- >> >> 1. It can be done even before the steps that Donald outlined (like I'm >> doing), e.g. so that you don't need to wait for a PEP to be finished. >> And it could even help in the later creation of such PEPs. >> 2. By doing so, as a side effect of the PR process, it automatically >> causes communication with pip maintainers and works towards the >> "splitting out parts of pip" goal -- all without duplicating any work >> because both pip and pipenv developers are aware and collaborating on >> these changes. >> 3. It is incremental as you go, ensuring that it is compatible with >> pip and that pip is using it, etc. >> 4. It improves pip in the process (as well as pipenv, if you're still >> vendoring in some form). In other words, the work is useful from the >> beginning, whether or not it's ever broken out into a separately >> maintained library. >> 5. It is forward progress towards the goal of a separate library. >> >> --Chris >> >> >> >>> >>> Dan Ryan // pipenv maintainer >>> gh: @techalchemy >>> >>>>> On Sep 25, 2018, at 8:11 AM, Paul Moore <p.f.mo...@gmail.com> wrote: >>>>> >>>>>> On Tue, 25 Sep 2018 at 12:53, Chris Jerdonek <chris.jerdo...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> On Tue, Sep 25, 2018 at 3:47 AM, Tzu-ping Chung <uranu...@gmail.com> >>>>>> wrote: >>>>>> We are using pip internals for things pip wasn’t implemented for. >>>>>> Specifically, >>>>>> Pipenv uses pip’s package-fetching functions to implement its >>>>>> platform-agnostic >>>>>> resolver. pip does not have this, so there’s no functional overlap here. >>>>>> Those >>>>>> utilities are used to build something that doesn’t exist in pip, so >>>>>> there’s no >>>>>> duplicated efforts. >>>>>> >>>>>> My recent focus on making sense of packaging implementations and >>>>>> splitting out >>>>>> parts of pip is exactly to prevent potential duplicated efforts. If we >>>>>> can’t use pip >>>>>> internals, let’s make things we want to use *not* internal! >>>>> >>>>> I don't see this practice of "splitting out parts of pip" actually >>>>> being followed so far though, and I want to tell you how it's leading >>>>> to duplicated efforts right now -- even for the PR's I happen to be >>>>> working on today. >>>> >>>> [...] >>>> >>>>> So by creating implementations of functions separate from pip instead >>>>> of splitting them out of pip, it's leading to duplicated work for >>>>> people working on pip's code base. A process of splitting something >>>>> out of pip would I think involve a process more like what I'm >>>>> following -- namely to refactor functionality in pip *first* so it can >>>>> be used separately, and then pull that out so you only have one >>>>> implementation at any point in time. Otherwise, you wind up creating >>>>> two versions of similar functionality that not only need to be created >>>>> twice, but also later reconciled if the plan is merge them back >>>>> together. >>>> >>>> This is a very good point, and thanks for the concrete example. For >>>> the record, I'm also a little annoyed that pipenv have been doing all >>>> this work in isolation. There's been very little communication between >>>> the pip and pipenv projects, and I think that's contributed to the >>>> problem. I, for one, wasn't even aware for a long time that they were >>>> embedding their own copy of pip. When we made pip's internals in >>>> private in pip 10, we got essentially no feedback from anyone saying >>>> that it would be a problem, and finding out that it had a sufficiently >>>> major impact on pipenv that they had to embed the old version of pip >>>> *after* the release was (to put it politely) suboptimal :-( >>>> >>>> Having said all this, this thread is the start of an attempt to work >>>> better together and I think we should be glad that it's happening and >>>> try to make it work. But I don't think we've got off to a particularly >>>> good start and it will need work. In particular, I'd like to see an >>>> attempt from *both* projects to clarify our goals - I've been speaking >>>> solely for myself so far, it would be good to get the other pip >>>> developers' views and set out a sort of roadmap, and for pipenv to do >>>> the same. >>>> >>>> Paul >>>> -- >>>> Distutils-SIG mailing list -- distutils-sig@python.org >>>> To unsubscribe send an email to distutils-sig-le...@python.org >>>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ >>>> Message archived at >>>> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/OGXUNDRBLTNPOWLLFBVF45SW4SRGYSPP/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/JQGH4HPIKWXGEWYEARESL5KC5WYEPR2S/
