On 04/08/2018 04:10 AM, Gaetan wrote:
Hello OpenStack dev community,I am currently working on the support of Pipfile for PBR ([1]), and I also follow actively the work on pipenv, which is now in officially supported by PyPA.
Awesome - welcome! This is a fun topic ...
There have been recently an intense discussion on the difficulties about Python libraries development, and how to spread good practices [2] on the pipenv community and enhance its documentation.As a user of PBR, and big fan of it, I try to bridge the link between pbr and pipenv (with [1]) but I am interested in getting the feedback of Python developers of OpenStack that may have much more experience using PBR and more generally packaging python libraries than me.
Great - I'll comment more on this a little later.
The main point is that packaging an application is quite easy or at least understandable by newcomers, using `requirements.txt` or `Pipfile`+ `Pipfile.lock` with pipenv. At least it is easily "teachable". Packaging a library is harder, and require to explain why by default `requirements.txt`(or `Pipfile`) does not work. Some "advanced" documentation exists but it still hard to understand why Python ended up with something complex for libraries ([3]). One needs to ensure `install_requires`declares the dependencies to that pip can find them during transitive dependencies installation (that is, installing the dependencies of a given dependency). PBR helps on this point but some does not want its other features.
In general, as you might imagine, pbr has a difference of opinion with the pypa community about requirements.txt and install_requires. I'm going to respond from my POV about how things should work - and how I believe they MUST work for a project such as OpenStack to be able to operate.
There are actually three different relevant use cases here, with some patterns available to draw from. I'm going to spell them out to just make sure we're on the same page.
* Library * Application * Suite of Coordinated ApplicationsA Library needs to declare the requirements it has along with any relevant ranges. Such as "this library requires 'foo' at at least version 2 but less than version 4". Since it's a library it needs to be able to handle being included in more than one application that may have different sets of requirements, so as a library it should attempt to have as wide a set of acceptable requirements as possible - but it should declare if there are versions of requirements it does not work with. In Pipfile world, this means "commit Pipfile but not Pipfile.lock". In pbr+requirements.txt it means "commit the requirements.txt with ranges and not == declared."
An Application isn't included in other things, it's the end point. So declaring a specific set of versions of things that the application is known to work in addition to the logical requirement range is considered a best practice. In Pipfile world, this is "commit both Pipefile and Pipfile.lock". There isn't a direct analog for pbr+requirements.txt, although you could simulate this by executing pip with a -c constraints.txt file.
A Suite of Coordinated Applications (like OpenStack) needs to communicate the specific versions the applications have been tested to work with, but they need to be the same so that all of the applications can be deployed side-by-side on the same machine without conflict. In OpenStack we do this by keeping a centrally managed constraints file [1] that our CI system adds to the pip install line when installing any of the OpenStack projects. A person who wants to install OpenStack from pip can also choose to do so using the upper-constraints.txt file and they can know they'll be getting the versions of dependencies we tested with. There is also no direct support for making this easier in pbr. For Pipfile, I believe we'd want to see is adding support for --constraints to pipenv install - so that we can update our Pipfile.lock file for each application in the context of the global constraints file. This can be simulated today without any support from pipenv directly like this:
pipenv install$(pipenv --venv)/bin/pip install -U -c https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt -r requirements.txt
pipenv lock
There is also works on PEP around pyproject.toml ([4]), which looks quite similar to PBR's setup.cfg. What do you think about it?
It's a bit different. There is also a philosophical disagreement about the use of TOML that's not worth going in to here - but from a pbr perspecitve I'd like to minimize use of pyproject.toml to the bare minimm needed to bootstrap things into pbr's control. In the first phase I expect to replace our current setup.py boilerplate:
setuptools.setup(
setup_requires=['pbr'],
pbr=True)
with:
setuptool.setup(pbr=True)
and add pyproject.toml files with:
[build-system]
requires = ["setuptools", "wheel", "pbr"]
This will allow us to reasonably have projects declare minimum ranges on
the pbr depend - and can allow us to give pbr dependencies (which is
impossible today due to how setup_requires works) If we made setuptools
and wheel depends of pbr,we could redue the pyproject.toml file to:
[build-system] requires = ["pbr"] but we need to test to make sure that works first.Once pep517 is implemented, we can implement a hook in pbr and add a line to pyproject.toml in all of the projects, something like:
build-backend = "pbr.core:build"Come to think of it, we could go ahead and implement pep517 support in pbr today and go ahead and start having the pbr pyproject.toml file to be:
[build-system] requires = ["pbr"] build-backend = "pbr.core:build"We'll have to keep the setup.py files until such a time as pip has full 517 supported added.
My opinion is this difference in behaviourbetween lib and app has technical reasons, but as a community we would gain a lot of unifying both workflows. I am using PBR + a few hacks [5], and I am pretty satisfied with the overall result.
There are two topics your pbr patch opens up that need to be covered: * pbr behavior * dependencies ** pbr behavior **I appreciate what you're saying about unifying the lib and app workflow, but I think the general pattern across the language communities (javascript and rust both have similar patterns to Pipefile) is that the two different options are important. We may just need to have a better document - rust has an excellent description:
https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html In any case, I think what pbr should do with pipfiles is:* If pbr discovers a Pipfile and no Pipfile.lock, it should treat the content in the packages section of Pipfile as it currently does with requirements.txt (and how you have done in the current patch)
* If pbr discoves a Pipfile.lock, it should treat the content in Pipfile.lock as it currently does requirements.txt. This way if someone commits a Pipfile.lock because they have chosen the application workflow, pbr will behave as they would expect.
Then, we either need to: * Add support to pipfile install for specifying a pip-style constraints file* Add support to pipfile install for specifying a constraints file that is in the format of a Pipfile.lock - but which does the same thing.
* Write a pbr utility subcommand for generating a Pipfile.lock from a Pipfile taking a provided constraints file into account.
We may also want to write a utility for creating a Pipefile and/or lock from a pbr-oriented requirements.txt/test-requirements.txt. (it should use pipfile on the backend of course) that can do the appropriate initial dance.
** dependencies ** The pep518 support in pip10 is really important here. Because ... We should not vendor code into pbr.While vendoring code has been found to be acceptable by other portions of the Python community, it is not acceptable here.
Once pip10 is released next week with pyproject.toml support, as mentioned earlier, we'll be able to start using (a reasonable set) of dependencies as is appropriate. In order to ensure backwards compat, I would recommend we do the following:
* Add toml and pipfile as depends to pbr* Protect their imports with a try/except (for people using old pip which won't install any depends pbr has) * Declare that pbr support for Pipfile will only work with people using pip>=10 and for projects that add a pyproject.toml to their project containing
[build-system] requires = ["pbr"]* If pbr tries to import toml/pipfile and fails, it should fall back to reading requirements.txt (this allows us to keep backwards compat until it's reasonable to expect everyone to be on pip10)
To support that last point, we should write a utility function, let's call it 'pbr lock', with the following behavior:
* If a Pipfile and a Pipfile.lock are present, it runs: pipfile lock -r* If there is no Pipfile.lock, simply read the Pipfile and write the specifiers into requirements.txt in non-pinned format.
This will allow pbr users to maintain their projects in such a way as to be backwards compatible while they start to use Pipfile/Pipefile.lock
We MAY want to consider adding an option flag to setup.cfg, like: [pbr] type = application or [pbr] type = libraryfor declaring to pbr which of Pipfile / Pipfile.lock should pbr pay attention to, regardless of which files might be present. I'm not sure whether that would be better or worse than inferring behavior from the presence of files. Of course, we could have the default behavior if the config setting isn't there to be to infer behavior from presence of files, but have the config setting for people who want to be explicit - and in the docs just don't mention omitting the setting - tell people to choose one or the other. What do you think?
So, in short, I simply start a general thread here to retrieve your general feedback around these points.Thanks for your feedbacks
[1] http://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt
Gaetan [1]: https://review.openstack.org/#/c/524436/ [2]: https://github.com/pypa/pipenv/issues/1911 [3]: https://docs.pipenv.org/advanced/#pipfile-vs-setup-py [4]: https://www.python.org/dev/peps/pep-0518/ [5]: library: - pipenv to maintain Pipfile and Pipfile.lock - Pipfile.lock not tracked (local reproductivity),- pipenv-to-requirements [6] to generate a `requirements.txt` without version freeze, also trackedapplications: - pipenv to maintain Pipfile and Pipfile.lock - Pipfile.lock not tracked (global reproductivity),- pipenv-to-requirements [6] to generate a `requirements.txt` and `requirements-dev.txt` with version freeze, both trackedThe development done with [1] should allow to get rid of [6]. [6] https://github.com/gsemet/pipenv-to-requirements ----- Gaetan __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
