Re: [openstack-dev] PBR and Pipfile
Hello, Thank you for this response. It took me quite some times to carefully read it, it was far beyond my expectation! So thanks a lot, lot of thing to understand. > >> There are actually three different relevant use cases here, with some > patterns available to draw from. I'm going to spell them out to just make > sure we're on the same page. > > * Library > * Application > * Suite of Coordinated Applications [...] Can we say packaging a python application for a linux distribution may fall in this "Suite of Coordinate Applications" kind? I really liked your way of describing these differences, I actually started using it as basis for my internal Python courses I give in my company on a regular basis :) As maintainer of a small project (Guake), I am in contact with package maintainers (Debian, arch, now, fedora,...) that relates me these kind of issues (Guake mainly has system python dependencies such as pygtk). They (the maintainers) has similar issues as OpenStack might have, in that they need to sync all dependencies for a given version distribution. Of course they do not use this upper-constraint requirements file as a way of fixing the same dependencies for all apps. > > For Pipfile, I believe we'd want to see is adding support for --constraints > to pipenv install - so that we can update our Pipfile.lock file for each > application in the context of the global constraints file. This can be > simulated today without any support from pipenv directly like this: > > pipenv install > $(pipenv --venv)/bin/pip install -U -c > https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt > -r requirements.txt > pipenv lock the pipenv lock will not lock the dependencies added manually by pip, unfortunately. What you can do it open a bug on pipenv project [1] on my behalf, with your need and assign me on it. I am not sure how this would need to be handled, but this worth notifying the pipenv projects of your needs (never used the -c option of pip install) > There is also works on PEP around pyproject.toml ([4]), which looks quite >> similar to PBR's setup.cfg. What do you think about it? >> > > It's a bit different. There is also a philosophical disagreement about the > use of TOML that's not worth going in to here we agree on TOML > - but from a pbr perspecitve I'd like to minimize use of pyproject.toml to > the bare minimm needed to bootstrap things into pbr's control. In the first > phase I expect to replace our current setup.py boilerplate: > > setuptools.setup( > setup_requires=['pbr'], > pbr=True) > > with: > > setuptool.setup(pbr=True) > > and add pyproject.toml files with: > > [build-system] > requires = ["setuptools", "wheel", "pbr"] [...] That would would indeed way beyond all what I wanted to work on for pbr. Do you have a plan for this support for pbr? > My opinion is this difference in behaviourbetween lib and app has >> technical reasons, but as a community we would gain a lot of unifying both >> workflows. I am using PBR + a few hacks [5], and I am pretty satisfied with >> the overall result. >> > > There are two topics your pbr patch opens up that need to be covered: > > * pbr behavior > * dependencies > > ** pbr behavior ** > > I appreciate what you're saying about unifying the lib and app workflow, > but I think the general pattern across the language communities (javascript > and rust both have similar patterns to Pipefile) is that the two different > options are important. We may just need to have a better document - rust has an excellent > description: > > https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html > There are lot of talks in the pipenv community at this moment, we ended up with best practices for python: - app: Pipfile and Pipfile.lock tracked in git - library: do not track Pipfile*, but use ‘pipenv install -e .’ I am trying to amend it with pbr support for libs (and there are actually several persons that uses pbr). In any case, I think what pbr should do with pipfiles is: > > * If pbr discovers a Pipfile and no Pipfile.lock, it should treat the > content in the packages section of Pipfile as it currently does with > requirements.txt (and how you have done in the current patch) That’s my goal in my patch. If I understand how it works (please tell me if i am wrong), on a distribution package/wheel, pbr looks for requirements.txt and inject as dependencies. The major change on my patch is: - having to vendor the basic toml parser. * If pbr discoves a Pipfile.lock, it should treat the content in > Pipfile.lock as it currently does requirements.txt. This is actually the “normal” behavior of pipenv, and more generally, of the Pipfile parser: prefer the lock file over the pipenv. The ultimate source of truth is the lock file and the Pipfile is only a convinient way of describing the dependencies, at least that is for apps. Then, we either need to: > > * Add support to pipfile install for specifying a
Re: [openstack-dev] PBR and Pipfile
On 04/08/2018 04:10 AM, Gaetan wrote: Hello OpenStack dev community, I am currently working on the support of Pipfile for PBR ([1]), and I also follow actively the work on pipenv, which is now in officially supported by PyPA. Awesome - welcome! This is a fun topic ... There have been recently an intense discussion on the difficulties about Python libraries development, and how to spread good practices [2] on the pipenv community and enhance its documentation. As a user of PBR, and big fan of it, I try to bridge the link between pbr and pipenv (with [1]) but I am interested in getting the feedback of Python developers of OpenStack that may have much more experience using PBR and more generally packaging python libraries than me. Great - I'll comment more on this a little later. The main point is that packaging an application is quite easy or at least understandable by newcomers, using `requirements.txt` or `Pipfile`+ `Pipfile.lock` with pipenv. At least it is easily "teachable". Packaging a library is harder, and require to explain why by default `requirements.txt`(or `Pipfile`) does not work. Some "advanced" documentation exists but it still hard to understand why Python ended up with something complex for libraries ([3]). One needs to ensure `install_requires`declares the dependencies to that pip can find them during transitive dependencies installation (that is, installing the dependencies of a given dependency). PBR helps on this point but some does not want its other features. In general, as you might imagine, pbr has a difference of opinion with the pypa community about requirements.txt and install_requires. I'm going to respond from my POV about how things should work - and how I believe they MUST work for a project such as OpenStack to be able to operate. There are actually three different relevant use cases here, with some patterns available to draw from. I'm going to spell them out to just make sure we're on the same page. * Library * Application * Suite of Coordinated Applications A Library needs to declare the requirements it has along with any relevant ranges. Such as "this library requires 'foo' at at least version 2 but less than version 4". Since it's a library it needs to be able to handle being included in more than one application that may have different sets of requirements, so as a library it should attempt to have as wide a set of acceptable requirements as possible - but it should declare if there are versions of requirements it does not work with. In Pipfile world, this means "commit Pipfile but not Pipfile.lock". In pbr+requirements.txt it means "commit the requirements.txt with ranges and not == declared." An Application isn't included in other things, it's the end point. So declaring a specific set of versions of things that the application is known to work in addition to the logical requirement range is considered a best practice. In Pipfile world, this is "commit both Pipefile and Pipfile.lock". There isn't a direct analog for pbr+requirements.txt, although you could simulate this by executing pip with a -c constraints.txt file. A Suite of Coordinated Applications (like OpenStack) needs to communicate the specific versions the applications have been tested to work with, but they need to be the same so that all of the applications can be deployed side-by-side on the same machine without conflict. In OpenStack we do this by keeping a centrally managed constraints file [1] that our CI system adds to the pip install line when installing any of the OpenStack projects. A person who wants to install OpenStack from pip can also choose to do so using the upper-constraints.txt file and they can know they'll be getting the versions of dependencies we tested with. There is also no direct support for making this easier in pbr. For Pipfile, I believe we'd want to see is adding support for --constraints to pipenv install - so that we can update our Pipfile.lock file for each application in the context of the global constraints file. This can be simulated today without any support from pipenv directly like this: pipenv install $(pipenv --venv)/bin/pip install -U -c https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt -r requirements.txt pipenv lock There is also works on PEP around pyproject.toml ([4]), which looks quite similar to PBR's setup.cfg. What do you think about it? It's a bit different. There is also a philosophical disagreement about the use of TOML that's not worth going in to here - but from a pbr perspecitve I'd like to minimize use of pyproject.toml to the bare minimm needed to bootstrap things into pbr's control. In the first phase I expect to replace our current setup.py boilerplate: setuptools.setup( setup_requires=['pbr'], pbr=True) with: setuptool.setup(pbr=True) and add pyproject.toml files with: [build-system] requires = ["setuptools",
[openstack-dev] PBR and Pipfile
Hello OpenStack dev community, I am currently working on the support of Pipfile for PBR ([1]), and I also follow actively the work on pipenv, which is now in officially supported by PyPA. There have been recently an intense discussion on the difficulties about Python libraries development, and how to spread good practices [2] on the pipenv community and enhance its documentation. As a user of PBR, and big fan of it, I try to bridge the link between pbr and pipenv (with [1]) but I am interested in getting the feedback of Python developers of OpenStack that may have much more experience using PBR and more generally packaging python libraries than me. The main point is that packaging an application is quite easy or at least understandable by newcomers, using `requirements.txt` or `Pipfile`+ `Pipfile.lock` with pipenv. At least it is easily "teachable". Packaging a library is harder, and require to explain why by default `requirements.txt`(or `Pipfile`) does not work. Some "advanced" documentation exists but it still hard to understand why Python ended up with something complex for libraries ([3]). One needs to ensure `install_requires`declares the dependencies to that pip can find them during transitive dependencies installation (that is, installing the dependencies of a given dependency). PBR helps on this point but some does not want its other features. There is also works on PEP around pyproject.toml ([4]), which looks quite similar to PBR's setup.cfg. What do you think about it? My opinion is this difference in behaviour between lib and app has technical reasons, but as a community we would gain a lot of unifying both workflows. I am using PBR + a few hacks [5], and I am pretty satisfied with the overall result. So, in short, I simply start a general thread here to retrieve your general feedback around these points. Thanks for your feedbacks Gaetan [1]: https://review.openstack.org/#/c/524436/ [2]: https://github.com/pypa/pipenv/issues/1911 [3]: https://docs.pipenv.org/advanced/#pipfile-vs-setup-py [4]: https://www.python.org/dev/peps/pep-0518/ [5]: library: - pipenv to maintain Pipfile and Pipfile.lock - Pipfile.lock not tracked (local reproductivity), - pipenv-to-requirements [6] to generate a `requirements.txt` without version freeze, also tracked applications: - pipenv to maintain Pipfile and Pipfile.lock - Pipfile.lock not tracked (global reproductivity), - pipenv-to-requirements [6] to generate a `requirements.txt` and `requirements-dev.txt` with version freeze, both tracked The development done with [1] should allow to get rid of [6]. [6] https://github.com/gsemet/pipenv-to-requirements - Gaetan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
