Re: [openstack-dev] [PBR] Patches on PBR
Hello, I have started a few other patches on PBR, can anyone give me some feedback on them? https://review.openstack.org/558181 https://review.openstack.org/561731 https://review.openstack.org/559484 They don't do that much, but they can help dev workflow Thanks a lot ! - Gaetan 2018-04-08 9:39 GMT+02:00 Gaetan : > Hello, > > I have started a few patch on PBR which fail, but I am not sure the > reason, they seem related to something external of my changes: > > - https://review.openstack.org/#/c/559484/6: 'pbr boostrap' command. > Error seems:"testtools.matchers._impl.MismatchError: b'STARTING test > server pbr_testpackage.wsgi' not in b''" > - https://review.openstack.org/#/c/558181/: proposal for update of > sem-ver 3 doc > - https://review.openstack.org/#/c/524436/: Pipfile support (still WIP) > > Can you review them? > Thanks, > > - > Gaetan > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] PBR and Pipfile
of pipenv, and more generally, of the Pipfile parser: prefer the lock file over the pipenv. The ultimate source of truth is the lock file and the Pipfile is only a convinient way of describing the dependencies, at least that is for apps. Then, we either need to: > > * Add support to pipfile install for specifying a pip-style constraints > file Not sure if I’ll be the right dev to handle the constraint feature... * Add support to pipfile install for specifying a constraints file that is > in the format of a Pipfile.lock - but which does the same thing. > > * Write a pbr utility subcommand for generating a Pipfile.lock from a > Pipfile taking a provided constraints file into account. Ok, you mean, as OpenStack will keep the requirements constraint, pbr needs to support it to be compatible. Pipenv is already able to support read from requirement, but it reflect on the Pipfile (that’s probably not wanted) We may also want to write a utility for creating a Pipefile and/or lock > from a pbr-oriented requirements.txt/test-requirements.txt. (it should use > pipfile on the backend of course) that can do the appropriate initial dance. Do you mean a constraint file or any requirements.txt? Pipenv is already able to convert a requirements.txt to Pipfile. ** dependencies ** > > The pep518 support in pip10 is really important here. Because ... > > We should not vendor code into pbr. Oops, I was exactly thinking this would cause problem. I actually kind of vendor 2 deps, “pipfile” parser and “toml parser”. While vendoring code has been found to be acceptable by other portions of > the Python community, it is not acceptable here. There are intense talks on this vendoring issues on pipenv, for example some deps are actually vendored 3 times (request if I remember). So, I agree this is not acceptable, but I need to better understand how pip 10 works. Once pip10 is released next week with pyproject.toml support, as mentioned > earlier, we'll be able to start using (a reasonable set) of dependencies as > is appropriate. In order to ensure backwards compat, I would recommend we > do the following: > > * Add toml and pipfile as depends to pbr > * Protect their imports with a try/except (for people using old pip which > won't install any depends pbr has) > * Declare that pbr support for Pipfile will only work with people using > pip>=10 and for projects that add a pyproject.toml to their project > containing > > [build-system] > requires = ["pbr"] > > * If pbr tries to import toml/pipfile and fails, it should fall back to > reading requirements.txt (this allows us to keep backwards compat until > it's reasonable to expect everyone to be on pip10) Seems way much more what I was expecting for the Pipfile support, but if you (the openstack community and pbr dévelopers) can guide me on this I think I can work on this full pip10+Pipfile support. The thing is, for the moment the current situation works pretty well for us, pbr does a marvelous work on both our libraries (but require to generate the requirement.twt with pipenv-to-requirements). I started using reno and its Sphinx plug-in (see the Guake source code :) ). To support that last point, we should write a utility function, let's call > it 'pbr lock', with the following behavior: > > * If a Pipfile and a Pipfile.lock are present, it runs: > > pipfile lock -r Not sure why it would be needed, if the project uses pipenv the normal way is to use pipenv lock directly to generate Pipfile.lock. I am not sure to understand the need for generating the requirements.txt with pipenv lock -r? Is it to support projects that uses Pipfile but on an old pip (<10?) is there reason not to update pip to the latest version ? * If there is no Pipfile.lock, simply read the Pipfile and write the > specifiers into requirements.txt in non-pinned format. > > This will allow pbr users to maintain their projects in such a way as to > be backwards compatible while they start to use Pipfile/Pipefile.lock > > We MAY want to consider adding an option flag to setup.cfg, like: > > [pbr] > type = application > > or > > [pbr] > type = library > > for declaring to pbr which of Pipfile / Pipfile.lock should pbr pay > attention to, regardless of which files might be present. I really liked this feature, may I add another setting in pbr: allow or disallow v-version. I have added support for version with a “v” or “V” prefix, that works really great with Gitlab’s protected tag regular expression. I would like my project for example to fail if one create a tag on master without the v prefix. It is not mandatory, but when the project will be handed over to new maintainer, they may do the mistake of using a version without v/V and it will work silently. I'm not sure whether t
[openstack-dev] PBR and Pipfile
Hello OpenStack dev community, I am currently working on the support of Pipfile for PBR ([1]), and I also follow actively the work on pipenv, which is now in officially supported by PyPA. There have been recently an intense discussion on the difficulties about Python libraries development, and how to spread good practices [2] on the pipenv community and enhance its documentation. As a user of PBR, and big fan of it, I try to bridge the link between pbr and pipenv (with [1]) but I am interested in getting the feedback of Python developers of OpenStack that may have much more experience using PBR and more generally packaging python libraries than me. The main point is that packaging an application is quite easy or at least understandable by newcomers, using `requirements.txt` or `Pipfile`+ `Pipfile.lock` with pipenv. At least it is easily "teachable". Packaging a library is harder, and require to explain why by default `requirements.txt`(or `Pipfile`) does not work. Some "advanced" documentation exists but it still hard to understand why Python ended up with something complex for libraries ([3]). One needs to ensure `install_requires`declares the dependencies to that pip can find them during transitive dependencies installation (that is, installing the dependencies of a given dependency). PBR helps on this point but some does not want its other features. There is also works on PEP around pyproject.toml ([4]), which looks quite similar to PBR's setup.cfg. What do you think about it? My opinion is this difference in behaviour between lib and app has technical reasons, but as a community we would gain a lot of unifying both workflows. I am using PBR + a few hacks [5], and I am pretty satisfied with the overall result. So, in short, I simply start a general thread here to retrieve your general feedback around these points. Thanks for your feedbacks Gaetan [1]: https://review.openstack.org/#/c/524436/ [2]: https://github.com/pypa/pipenv/issues/1911 [3]: https://docs.pipenv.org/advanced/#pipfile-vs-setup-py [4]: https://www.python.org/dev/peps/pep-0518/ [5]: library: - pipenv to maintain Pipfile and Pipfile.lock - Pipfile.lock not tracked (local reproductivity), - pipenv-to-requirements [6] to generate a `requirements.txt` without version freeze, also tracked applications: - pipenv to maintain Pipfile and Pipfile.lock - Pipfile.lock not tracked (global reproductivity), - pipenv-to-requirements [6] to generate a `requirements.txt` and `requirements-dev.txt` with version freeze, both tracked The development done with [1] should allow to get rid of [6]. [6] https://github.com/gsemet/pipenv-to-requirements - Gaetan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Patches on PBR
Hello, I have started a few patch on PBR which fail, but I am not sure the reason, they seem related to something external of my changes: - https://review.openstack.org/#/c/559484/6: 'pbr boostrap' command. Error seems:"testtools.matchers._impl.MismatchError: b'STARTING test server pbr_testpackage.wsgi' not in b''" - https://review.openstack.org/#/c/558181/: proposal for update of sem-ver 3 doc - https://review.openstack.org/#/c/524436/: Pipfile support (still WIP) Can you review them? Thanks, - Gaetan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [pbr] support v_version
> > > > I guess it somehow didn't used my build of the pbr package when > > running in gitlab pipeline. > > It sounds like the CE pipeline is not building packages in the same way? > Or is using an old version of pbr? > > I guess it was the pbr version from pypi.python.org, not my customized build published on our internal pypi https://books.sonatype.com/nexus-book/3.0/reference/pypi.html > > Do you plan on releasing PBR soon on pypi? > > I have to build myself and push it on our internal nexus pypi, but I > think > > the > > safest way is to wait for an official pbr release on pypi.python.org :) > > Unfortunately we're approaching a significant set of feature freeze > deadlines for this OpenStack development cycle. Because it is very > difficult to "pin" a library like pbr that appears in the setup_requires > list for projects in our CI pipelines, and this next release of pbr > would have quite a few changes, we have decided in our meeting today > to be cautious and delay the next release for a few weeks. > > Have you ever considered using pipenv/pipfile ? I started using it a few months ago it helps a lot with dependency freezing I have scheduled a reminder to tag the pbr release during the first > week of March, after the OpenStack release is safely completed. > > I'm sorry if this causes inconvenience. We're going to work on > ensuring we have more frequent releases so we reduce our chance of > ending up in a similar situation again in the future. > That is not a problem by itself, I still have this self-hosted Pypi repository in the meantime. To react on the IRL log, indeed, the proposal I make in this thread is purely optional, for my need, gitlab handle correctly the protection against unwanted publication. Just hope it will be useful for other project, probably not for OpenStack itself. Thanks a lot for supporting external projects :) Gaetan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [pbr] support v_version
First, thanks a lot for your support and your kindness ! Really appreciate that :) > > Do you know where I need to hack PBR to fix it? > > So 'pbr' correctly parses the prefixed tags, but it's just the output > packages (sdists, wheels) that always unversioned? If so, this sounds > correct. Python packaging, as defined in PEP-440 [1], doesn't use the > 'v' prefixes, so it doesn't really make sense to stick them in here. > Out of curiosity, what's your rationale for modifying the package name? > The package name is not changed actually. With the patch in PBR that has been merged, one could add a tag named "v1.0.0" on mypackage package, and the sdist will generate a distribution package mypackage-0.0.4.tar.gz So I think (hope?) this is still PEP440 compliant. I tried this feature on another software that also uses pbr and there is no problem: v version works great with sdist/bdist/wheel packages. I use it inside a Gitlab CE pipeline on a tag pipeline (CI is triggered when a git tag that follows the "v*" regular expression), and instead of creating a package mypackage-0.0.4-py2.py3-none-any.whl it created mypackage-0.0.3.dev3-py2.py3-none-any.whl. When I retried manually on my development environment, pbr works perfectly again on the same code. I guess it somehow didn't used my build of the pbr package when running in gitlab pipeline. Do you plan on releasing PBR soon on pypi? I have to build myself and push it on our internal nexus pypi, but I think the safest way is to wait for an official pbr release on pypi.python.org :) > Second point, to go to the end of the logic of my change, I would > > like to propose an optional way (in setup.cfg?) to **prevent** any > > tag without the 'v' prefix, ie, where a bare version tag like `1.0.0` > > is not to be considered as a valid version. > > That way, on system such as gitlab or github: > > - repository owners "protect" tags with pattern "v*", ie, all tags > > for release such as "v1.0.0", ... cannot be pushed by anyone but the > > owners/masters > > - other developer can still push other tags for other purpose > > So this could be used to prevent pbr reading the tags, but it won't > stop anyone from creating them in the first place (i.e. "protect" > tags). Yes, I agree this is not really mandatory. Gitlab tag protection should be enough. I am using a "protected environment variables" on gitlab, and indeed, the credentials to push on Pypi are only sent when the pipeline is triggered on such a "protected" branch or "protected tag". So we "protect" only tags starting with a "v*" and only this triggered pipeline can publish to pypi (we use Nexus). This allows other developers to add any tags not started with v (only repository owners can create tags starting with a "v*"). Note this "v*" regular expression is configurable and seem to default/good practice on GitLab CE/EE. > We can do this but it would be a separate feature and, to be > honest, I'd suggest using Git hooks or some form of access control as a > better way to do this (Note: it seems GitLab already supports something > similar [2]). > Yes this is what I actually use :) Thanks In short: pbr v_version seems to work great, just hoping for the official PBR release on pypi.python.org :) Thanks Gaetan Semet __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [pbr] support v_version
Hello I have submitted this patch ([1]) that add support for v_version in PBR. Basically I can tag v1.0.0 instead of 1.0.0 to release 1.0.0. However, after rework it appears PBR does not behaves well, even if the unit tests pass: On tag for instance v1.0.0, the result packages in named `-1.0.0.dev1`. Do you know where I need to hack PBR to fix it? Second point, to go to the end of the logic of my change, I would like to propose an optional way (in setup.cfg?) to **prevent** any tag without the 'v' prefix, ie, where a bare version tag like `1.0.0` is not to be considered as a valid version. That way, on system such as gitlab or github: - repository owners "protect" tags with pattern "v*", ie, all tags for release such as "v1.0.0", ... cannot be pushed by anyone but the owners/masters - other developer can still push other tags for other purpose What do you think about this proposal? [1] https://review.openstack.org/#/c/531161/ Thanks Regards - Gaetan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [reno] questions about reno
Hello Thanks for your answers! - unreleased notes appear in release note with a title such as "0.1.0-2". > > Is it possible to not have any title or use "0.1.0-dev2" pattern like > pbr ? > > I'm not sure why it matters, but if you want to work on that patch I'll > help with reviews. > Ok, I'll see what I can do :) > > > - I guess that all notes should stay in the same folder version after > > versions, and the > > release notes of all versions will keep being automatically generated. > > Don't you think > > it might get difficult to manage all theses files? Is is possible to move > > them in different folder (at least a folder "archives?) > > We've put off doing anything like that until we have a project with > enough notes that we can observe the problems and decide how to fix > them. Have you already reached that point or are you anticipating > problems in the future? > No, just started using reno and just see that this folder might get messy quickly. Maybe I over-think this, I agree with you to observe first > > > - it is possible to generate the NEWS file using reno ? I started trying > > conversion with pandoc but the result are not great. > > How is the NEWS file different from CHANGES.txt that pbr produces? Is it > the format, or the content? > So, I like PBR that generates ChangeLog from the git history, but it has lot of details (maybe too much). So, I was thinking to store in NEWS only the release note As an example, you can look how I plan to use it for Guake: https://github.com/Guake/guake I usually write the NEWS at each release manually ( https://github.com/Guake/guake/blob/master/NEWS), and that's where reno shines in my eyes :) Regards, Gaetan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [reno] questions about reno
Hello I played this week with reno and i really like it, but I have a bunch of question: - unreleased notes appear in release note with a title such as "0.1.0-2". Is it possible to not have any title or use "0.1.0-dev2" pattern like pbr ? - I guess that all notes should stay in the same folder version after versions, and the release notes of all versions will keep being automatically generated. Don't you think it might get difficult to manage all theses files? Is is possible to move them in different folder (at least a folder "archives?) - it is possible to generate the NEWS file using reno ? I started trying conversion with pandoc but the result are not great. If you find some features interesting, I'll be happy to contribute ! - Gaetan / Stibbons __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [pbr] Git submodules support
Hi, > > * But once in the docker build, PBR complains about not getting access to* > > > * upstream GIT. And it is true: the git tree (with dependencies) has been* > > > * checkedout by gitlab and the docker build (running in a gitlab runner) > does*>* not give access to the repository, because it is not required.* > Is the issue here that submodules somehow cause PBR to need access to > upstream git repositories or is the installation process not including the > .git directories for repositories and only including their checked out > state? PBR should work with local git repository checkouts without needing > to talk to any remotes/upstreams. I am pretty sure there is the .git in the submodules, and indeed it only occurs on the submodules, not on the main module that also uses pbr. Maybe a side effect of using git submodules with pbr. But, if you tell me this is not normal for PBR to try to connect to upstream git if all the .git directory (of the submodules) are here, i'll try to debug it and see what's wrong. > *The other solution might be to allow pbr to NOT have git upstream access > *>* in _get_version_from_git. > * > At least without submodules this shouldn't be required. Yes, on the main module, PBR is happy. Only on the submodules > Without submodules you only need access to the local git repository and not > any remotes. I think this behavior is desireable and if we can have that > behavior with submodules as well that is probably a good improvement to make. Roger, I'll debug it. > > *I will open another thread on this ML if*> > * you agree, when my unit tests will pass, but i will not be able to do it* > >* until net year.* > Adding pipfile support seems like an excellent thing to add considering > people are trying to use pipfile and pbr together now. Great, I am happy to ear it. Can you give an eye to my first draft? https://review.openstack.org/#/c/524436/ The main issue i can see it that I had to copy toml parser and pipfile parser from external projects, I don't know if there will be any policy against it in PBR (or licence compatibility issue, ...). >* Also, I maintain a fork a pbr "pbrlgs" ( > *>* https://github.com/Stibbons/pbr/tree/pbrlgs > <https://github.com/Stibbons/pbr/tree/pbrlgs>) that makes PBR happy with git > *>* submodules. > *Are these improvements something that should be upstreamed as well? It is basically HEAD of PBR + my pipfile patch + my git submodules environment variables hack. Of course it is not intended to be kept alive if PBR merges my contributions. But we use it in production now: - pipfile patch allows PBR to install the dependencies of a dependency that uses a Pipfile - env var patch makes pbr happy with submodules in our docker environment. Thanks Best Regards, Gaetan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [pbr] Git submodules support
Hello I recently submitted the following pull request in order to allow using git submodules with PBR modules: https://review.openstack.org/#/c/524421/ To give a little bit of context, in my team, all our python modules are following almos the same pattern: - python 3 only - setup.py/setuptools/distutil stuff completely handled by PBR - dependencies handled by Pipfile (see my first draft to support it with PBR here: https://review.openstack.org/#/c/524436/) - "active dependencies", basically shared modules we may want to update quickly are injected in the project using Git Submodules, and installed with `pipenv install ./deps/mydependency` (equivalent in pip: `pip install -e deps/mydepedency`). - all on gitlab (so using the github/gitlab "pull request"/"mergerequest" mecanism). - Python applications (that used PBR) are deployed in docker This is a very, very cool architecture, allowing us to do change very quickly in depedencies and reference it directly even if it has been merged into the dependency yet. A whole cycle or review might takes a few days, change may be rejected, unit tests might be requested before merge,... So during all this time, in the "normal" way of releasing a change on a dependency (mergerequest + git tag (thanks PBR) + upload to our internal pypi), the change might not be available in the parent module that needs it. So injecting dependencies with git submodules is very handy, even if all dependencies, at the end, once stabilized, will end up in a clean pypi repo. Now, why Pipfile? Because it is also much more convinient to use that a requirements.txt (and if you want to accurately handle the versions, you need a requirements.txt.in + pip-tools to generate requirements.txt). Same for the development requirements. So, pipfile handles all of that for us, and Pipfile will eventually become the defacto standard for dependency declaration, replacing requirements*.txt. Pipfile recently moved to Pypa, becoming a little bit more official: https://github.com/pypa/pipfile. So, back to my first patch (https://review.openstack.org/#/c/524421/). It fixes an issue when trying to use everything all together. In our Gitlab CI, PBR is happy and almost does the right thing, even it does not handle Pipfile directly (see https://review.openstack.org/#/c/524436/ for the limitations). But once in the docker build, PBR complains about not getting access to upstream GIT. And it is true: the git tree (with dependencies) has been checkedout by gitlab and the docker build (running in a gitlab runner) does not give access to the repository, because it is not required. Why PBR does that? Because: - git submodules dependencies does not provided pkg metadata, because they are not distribution package (they are installed from source) - the method that finds the version from the git tree fails with this upstream access error probably because it expects to be able to fetch something from the remote, and the environment does not give this access) So my simple and dirty solution is to use the PBR_VERSION environment variable to skip this check, that is not required on dependencies coming from git submodules since the "version matching" is actually handled by git sumodules. I first only moved PBR_VERSION handling a few lines to avoid affecting other packages that uses PBR: https://review.openstack.org/#/c/524421/1/pbr/packaging.py There are a bit of discussion on my patch, thanks for the reviewers by the way. I see basically 3 solutions for making PBR happy with git submodules and docker, all requiring developer to set an envvar in their Dockerfile: - developers set the environment variable PBR_VERSION to freeze the version of dependencies injected by git submodules. This only requires my first patchset - same but with another variable (PBR_FALLBACK_VERSION ?). - same but with an environment variable with the name of the package in it. if the name can take non alphanumeric character, I can strip them out with a regex for example (see documentation proposal here: https://review.openstack.org/#/c/524421/3/doc/source/user/packagers.rst) The other solution might be to allow pbr to NOT have git upstream access in _get_version_from_git. Sorry for the long context description. Hope this will help you being interested by my patches. PS: about Pipfile support in PBR, I will open another thread on this ML if you agree, when my unit tests will pass, but i will not be able to do it until net year. Also, I maintain a fork a pbr "pbrlgs" ( https://github.com/Stibbons/pbr/tree/pbrlgs) that makes PBR happy with git submodules. I had to in order to use it in our internal build system. Thanks - Gaetan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
