Re: [openstack-dev] [all]deprecating [test-]requirements-PYN.txt
On Thu, Jul 02, 2015 at 09:22:03PM +0100, Dave Walker wrote: On 29 June 2015 at 04:59, Robert Collins robe...@robertcollins.net wrote: Hi, so we're nearly ready to deprecate the python-version-specific requirements files. Once we have infra's requirements cross checking jobs all copacetic again, we should be able to move forward. There isn't a specific spec for this in pbr, and I wanted to get some broad input into the manner of the deprecation. SNIP Slightly offtopic, but I've noticed that some consumers of bandit[0] have been creating requirements-bandit.txt. This is to specify bandit requirements without requiring the whole test-requirements.txt env to be installed, to run what is essentially a linting tool. I'm not sure I like the idea of creating MORE requirements.txt style files as it pollutes the project root namespace and currently has no syncing from global-requirements. I wondered if you had any ideas on how to solve this for bandit usage, and potentially other projects? I would use setuptools extras to do this.[1] PBR has support for this since the 1.0.0 release. [2] I used it on subunit2sql to separate the fairly heavyweight requirements for using the graph command from the rest of the dependencies. [3] -Matt Treinish [0] https://wiki.openstack.org/wiki/Security/Projects/Bandit [1] https://www.python.org/dev/peps/pep-0426/#extras-optional-dependencies [2] http://docs.openstack.org/developer/pbr/#extra-requirements [3] https://review.openstack.org/#/c/184278/ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all]deprecating [test-]requirements-PYN.txt
On 2015-07-02 21:22:03 +0100 (+0100), Dave Walker wrote: [...] I wondered if you had any ideas on how to solve this for bandit usage, and potentially other projects? Brainstorming here, but I wonder if extras_require[1] would be a good solution down the road. I.e., having `tox -e bandit` run something like `pip install .[test_bandit]` in its virtualenv before invoking bandit. [1] URL: https://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all]deprecating [test-]requirements-PYN.txt
On 29 June 2015 at 04:59, Robert Collins robe...@robertcollins.net wrote: Hi, so we're nearly ready to deprecate the python-version-specific requirements files. Once we have infra's requirements cross checking jobs all copacetic again, we should be able to move forward. There isn't a specific spec for this in pbr, and I wanted to get some broad input into the manner of the deprecation. SNIP Slightly offtopic, but I've noticed that some consumers of bandit[0] have been creating requirements-bandit.txt. This is to specify bandit requirements without requiring the whole test-requirements.txt env to be installed, to run what is essentially a linting tool. I'm not sure I like the idea of creating MORE requirements.txt style files as it pollutes the project root namespace and currently has no syncing from global-requirements. I wondered if you had any ideas on how to solve this for bandit usage, and potentially other projects? [0] https://wiki.openstack.org/wiki/Security/Projects/Bandit -- Kind Regards, Dave Walker __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all]deprecating [test-]requirements-PYN.txt
On 3 July 2015 at 08:22, Dave Walker em...@daviey.com wrote: On 29 June 2015 at 04:59, Robert Collins robe...@robertcollins.net wrote: Hi, so we're nearly ready to deprecate the python-version-specific requirements files. Once we have infra's requirements cross checking jobs all copacetic again, we should be able to move forward. There isn't a specific spec for this in pbr, and I wanted to get some broad input into the manner of the deprecation. SNIP Slightly offtopic, but I've noticed that some consumers of bandit[0] have been creating requirements-bandit.txt. This is to specify bandit requirements without requiring the whole test-requirements.txt env to be installed, to run what is essentially a linting tool. Well, this doesn't really make sense to me. making a new virtualenv with tools like that is a few seconds (after the first warm-up time), and the virtualenvs persist. For CI, its not the bottleneck and thus perf is irrelevant. I'm not sure I like the idea of creating MORE requirements.txt style files as it pollutes the project root namespace and currently has no syncing from global-requirements. I wondered if you had any ideas on how to solve this for bandit usage, and potentially other projects? [0] https://wiki.openstack.org/wiki/Security/Projects/Bandit Its feasible to use extras for this, but I wouldn't. requirements-bandit.txt is definitely a problem though as its not synced, not linted, and not cross-checked. So my recommendations are: 1) test-requirements. Thats what its there for. 2) extras if that realy is an issue. Also if doing this help everyone by documenting where the issue is. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all]deprecating [test-]requirements-PYN.txt
On 29 June 2015 at 23:56, Doug Hellmann d...@doughellmann.com wrote: I think we should do three things: - error if universal builds are requested and python versioned requirements files are present. That may break some of the Oslo stable libs, since not all of them were ready for Python 3 last cycle, and certainly not before. Have you done any analysis to find those libs so we can get patches ready preemptively? Doug So if that breaks them, they must have: [wheel] universal = 1 in their setup.cfg *and* they must have [test-]requirements[-pyN].txt rather than just requirements.txt. If they have that, they are broken already: installing them on python3 will install the wrong dependencies. I haven't done a system wide grep for that no. -Rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all]deprecating [test-]requirements-PYN.txt
Excerpts from Robert Collins's message of 2015-06-29 15:59:12 +1200: Hi, so we're nearly ready to deprecate the python-version-specific requirements files. Once we have infra's requirements cross checking jobs all copacetic again, we should be able to move forward. There isn't a specific spec for this in pbr, and I wanted to get some broad input into the manner of the deprecation. As a reminder, for context, we have several bits of context to consider. Firstly, we're aligning with upstream packaging precepts, so we want to remove all non-deployment-specific usage of requirements.txt and similar files. Secondly, the Python version specific files are incompatible with universal wheels, which are desirable because our infrastructure only knows how to build one wheel when a tag is made, and its less redundant downloads for users with multiple python versions. Thirdly, we can't do any backwards incompatible changes in pbr without breaking any existing users of $thing. Because we're a setup_requires, and setuptools can't handle version dependencies of setup_requires. So whatever we do will affect all stable branches immediately, in all gate jobs. I think we should do three things: - error if universal builds are requested and python versioned requirements files are present. That may break some of the Oslo stable libs, since not all of them were ready for Python 3 last cycle, and certainly not before. Have you done any analysis to find those libs so we can get patches ready preemptively? Doug - warn on stdout if versioned requirements files are present - start reflecting the 'test' extra into tests_require in the setup_kwargs The downside of this is that it will warn indefinitely for existing stable branches. But I think that that is tolerable. If its not, we could write a timestamp somewhere and only warn once/day, but I think that that is likely to lead to confusion, not clarity. -Rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all]deprecating [test-]requirements-PYN.txt
Hi, so we're nearly ready to deprecate the python-version-specific requirements files. Once we have infra's requirements cross checking jobs all copacetic again, we should be able to move forward. There isn't a specific spec for this in pbr, and I wanted to get some broad input into the manner of the deprecation. As a reminder, for context, we have several bits of context to consider. Firstly, we're aligning with upstream packaging precepts, so we want to remove all non-deployment-specific usage of requirements.txt and similar files. Secondly, the Python version specific files are incompatible with universal wheels, which are desirable because our infrastructure only knows how to build one wheel when a tag is made, and its less redundant downloads for users with multiple python versions. Thirdly, we can't do any backwards incompatible changes in pbr without breaking any existing users of $thing. Because we're a setup_requires, and setuptools can't handle version dependencies of setup_requires. So whatever we do will affect all stable branches immediately, in all gate jobs. I think we should do three things: - error if universal builds are requested and python versioned requirements files are present. - warn on stdout if versioned requirements files are present - start reflecting the 'test' extra into tests_require in the setup_kwargs The downside of this is that it will warn indefinitely for existing stable branches. But I think that that is tolerable. If its not, we could write a timestamp somewhere and only warn once/day, but I think that that is likely to lead to confusion, not clarity. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
