Re: [openstack-dev] [all][infra] Constraints are ready to be used for tox.ini
On 08/12/2016 01:44 PM, Ihar Hrachyshka wrote:
> Andreas Jaeger wrote:
>
>> On 08/12/2016 12:13 PM, Ihar Hrachyshka wrote:
>>> Jeremy Stanley wrote:
>>>
On 2016-08-11 20:19:51 +0200 (+0200), Ihar Hrachyshka wrote:
> Do I read it right that we can now use constraints for post queue too
> (releasenotes, cover, venv targets)?
Yes, that was the hardest part to get working, but thanks to
tireless efforts on the part of a number of people that has now been
fixed and tested.
>>>
>>> Yay! Kudos to everyone who made it happen!
>>>
>>> I posted a bunch of patches for neutron repos to validate it works:
>>> https://review.openstack.org/#/q/I02b28d3b354c3b175147c5be36eea4dc7e05f2a3,n,z
>>>
>>
>> The real validation is checking the results of post, tag, and release
>> queue...
>>
>> I don't expect surprises in check/gate queue…
>>
>
> Yeah, we will need to land those to check post queue. But at least we
> can validate that constraints are correctly applied for where we execute
> the targets in check queue (f.e. tox -e releasenotes). I checked
> releasenotes target in all neutron repos with those patches, and at
> least os-client-config is now not pulling the latest version, but the
> one in current upper-constraints.txt file, so it’s already something.
glad to hear.
>> One caveat: Constraints cannot be used for jobs running on long-lived
>> nodes unless special care is taken - like I did for translation jobs,
>
> I probably lack some knowledge here. What are those jobs, and how do I
> detect them?
Sorry, that was more for completeness for the rest of the infra team.
The long-lived nodes have special purposes and you're not messing with
them directly,
Andreas
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][infra] Constraints are ready to be used for tox.ini
Andreas Jaeger wrote: On 08/12/2016 12:13 PM, Ihar Hrachyshka wrote: Jeremy Stanley wrote: On 2016-08-11 20:19:51 +0200 (+0200), Ihar Hrachyshka wrote: Do I read it right that we can now use constraints for post queue too (releasenotes, cover, venv targets)? Yes, that was the hardest part to get working, but thanks to tireless efforts on the part of a number of people that has now been fixed and tested. Yay! Kudos to everyone who made it happen! I posted a bunch of patches for neutron repos to validate it works: https://review.openstack.org/#/q/I02b28d3b354c3b175147c5be36eea4dc7e05f2a3,n,z The real validation is checking the results of post, tag, and release queue... I don't expect surprises in check/gate queue… Yeah, we will need to land those to check post queue. But at least we can validate that constraints are correctly applied for where we execute the targets in check queue (f.e. tox -e releasenotes). I checked releasenotes target in all neutron repos with those patches, and at least os-client-config is now not pulling the latest version, but the one in current upper-constraints.txt file, so it’s already something. One caveat: Constraints cannot be used for jobs running on long-lived nodes unless special care is taken - like I did for translation jobs, I probably lack some knowledge here. What are those jobs, and how do I detect them? Ihar __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][infra] Constraints are ready to be used for tox.ini
On 08/12/2016 12:13 PM, Ihar Hrachyshka wrote:
> Jeremy Stanley wrote:
>
>> On 2016-08-11 20:19:51 +0200 (+0200), Ihar Hrachyshka wrote:
>>> Do I read it right that we can now use constraints for post queue too
>>> (releasenotes, cover, venv targets)?
>>
>> Yes, that was the hardest part to get working, but thanks to
>> tireless efforts on the part of a number of people that has now been
>> fixed and tested.
>
> Yay! Kudos to everyone who made it happen!
>
> I posted a bunch of patches for neutron repos to validate it works:
> https://review.openstack.org/#/q/I02b28d3b354c3b175147c5be36eea4dc7e05f2a3,n,z
The real validation is checking the results of post, tag, and release
queue...
I don't expect surprises in check/gate queue...
One caveat: Constraints cannot be used for jobs running on long-lived
nodes unless special care is taken - like I did for translation jobs,
Andreas
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][infra] Constraints are ready to be used for tox.ini
Jeremy Stanley wrote: On 2016-08-11 20:19:51 +0200 (+0200), Ihar Hrachyshka wrote: Do I read it right that we can now use constraints for post queue too (releasenotes, cover, venv targets)? Yes, that was the hardest part to get working, but thanks to tireless efforts on the part of a number of people that has now been fixed and tested. Yay! Kudos to everyone who made it happen! I posted a bunch of patches for neutron repos to validate it works: https://review.openstack.org/#/q/I02b28d3b354c3b175147c5be36eea4dc7e05f2a3,n,z Thanks, Ihar __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][infra] Constraints are ready to be used for tox.ini
Fantastic news Andreas! -Rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][infra] Constraints are ready to be used for tox.ini
> -Original Message-
> From: John Dickinson [mailto:m...@not.mn]
> Sent: Thursday, August 11, 2016 2:09 PM
> To: OpenStack Development Mailing List
> Subject: Re: [openstack-dev] [all][infra] Constraints are ready to be used
> for tox.ini
>
>
>
> On 11 Aug 2016, at 10:03, Andreas Jaeger wrote:
>
> > TL;DR: upper-constraints can be used now for all tox based jobs
> >
> > With any software package, you will need additional packages to run it.
> > Often, there's a tight coupling: The software package will only run with
> > specific other package versions. This dependency information is
> > sometimes found in README files, in code, or in package metadata. If you
> > install the package, you need to figure out the dependency and
> > handle it properly.
> >
> > The Python package installer pip uses a list of requirements to install
> > dependent Python packages. This list not only contains the name of
> > packages but also limits which versions to use, or not to use.
> > In OpenStack we handle these dependencies in a global requirements list
> > and use it for most of the repositories. During initial testing a
> > specific package version is tested but at a later point, another one
> > might be used, or during deployment again another one.
> >
> > To document what was tested, give guidenance for deployment, and help to
> > figure out breakage by upstream projects, the OpenStack requirements
> > projects maintains a set of constraints with packages pinned to specific
> > package versions that are known to be working.
> > These are in the upper-constraints.txt file.
> >
> > Devstack already handles upper-constraints.txt when installing packages
> > and I'm happy to say that tox, the Python testing framework used in
> > OpenStack, can now handle upper-constraints as well everywhere.
> >
> >
> > Constraints for tox based jobs
> > ==
> > To use constraints, change in tox.ini the install command to:
> >
> > install_command = pip install
> > -
> c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requ
> irements/plain/upper-constraints.txt}
> > {opts} {packages}
>
> I've proposed a patch to Swift to do this:
>
> https://review.openstack.org/#/c/354291/
>
> I'd appreciate any advice on it.
[amrith] Ditto for Trove, I've proposed
https://review.openstack.org/#/c/354335/
>
>
> >
> > Caveat
> > ==
> >
> > Note that constraints are used for the installation of each packages, so
> > if you want to install a package from source and have constraints for a
> > specific version in the constraints file, it will not work. This happens
> > with some of the OpenStack python client packages: When they install
> > their dependencies, those might have a dependency on the client package
> > itself. And this then will cause an error since the client package
> > should get installed from source.
> >
> > So, projects need to remove the constraints file for themselves if they
> > run into this. Packages like python-novaclient and python-glanceclient
> > therefore use a wrapper (tools/tox_install.sh) as
> > install command to edit the constraints file first and remove their own
> > project from it.
> >
> > Also, be aware that this only for those jobs that have been enabled for
> > it in the project-config repository. It's done for all the generic tox
> > enabled targets and should be done for all custom tox targets as well.
> > Some repositories are not using constraints like project-config
> > itself, so those jobs are not set up.
> >
> > Constraints for DevStack jobs
> > =
> > Devstack-gate takes care using constraints, there is nothing for a
> > repository to do to honor constraints.
> >
> > Check the devstacklog.txt file, if constraints are in use it will use
> > lines like:
> >
> > Collecting oslo.context===2.7.0 (from -c
> > /opt/stack/new/requirements/upper-constraints.txt (line 204))
> >
> > References
> > ==
> >
> > http://docs.openstack.org/developer/requirements/
> > https://specs.openstack.org/openstack/openstack-
> specs/specs/requirements-management.html
> >
> >
> > Thanks
> > ==
> > As usual in OpenStack, such work is a team work of many people. I'd like
> > to thank especially:
> >
> > *Robert Collins 'lifeless': For writing the initial spec, implementation
> > work, and giving guideance on many of these detai
Re: [openstack-dev] [all][infra] Constraints are ready to be used for tox.ini
On 2016-08-11 20:19:51 +0200 (+0200), Ihar Hrachyshka wrote: > Do I read it right that we can now use constraints for post queue too > (releasenotes, cover, venv targets)? Yes, that was the hardest part to get working, but thanks to tireless efforts on the part of a number of people that has now been fixed and tested. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][infra] Constraints are ready to be used for tox.ini
Do I read it right that we can now use constraints for post queue too
(releasenotes, cover, venv targets)?
Ihar
Andreas Jaeger wrote:
TL;DR: upper-constraints can be used now for all tox based jobs
With any software package, you will need additional packages to run it.
Often, there's a tight coupling: The software package will only run with
specific other package versions. This dependency information is
sometimes found in README files, in code, or in package metadata. If you
install the package, you need to figure out the dependency and
handle it properly.
The Python package installer pip uses a list of requirements to install
dependent Python packages. This list not only contains the name of
packages but also limits which versions to use, or not to use.
In OpenStack we handle these dependencies in a global requirements list
and use it for most of the repositories. During initial testing a
specific package version is tested but at a later point, another one
might be used, or during deployment again another one.
To document what was tested, give guidenance for deployment, and help to
figure out breakage by upstream projects, the OpenStack requirements
projects maintains a set of constraints with packages pinned to specific
package versions that are known to be working.
These are in the upper-constraints.txt file.
Devstack already handles upper-constraints.txt when installing packages
and I'm happy to say that tox, the Python testing framework used in
OpenStack, can now handle upper-constraints as well everywhere.
Constraints for tox based jobs
==
To use constraints, change in tox.ini the install command to:
install_command = pip install
-c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt}
{opts} {packages}
Caveat
==
Note that constraints are used for the installation of each packages, so
if you want to install a package from source and have constraints for a
specific version in the constraints file, it will not work. This happens
with some of the OpenStack python client packages: When they install
their dependencies, those might have a dependency on the client package
itself. And this then will cause an error since the client package
should get installed from source.
So, projects need to remove the constraints file for themselves if they
run into this. Packages like python-novaclient and python-glanceclient
therefore use a wrapper (tools/tox_install.sh) as
install command to edit the constraints file first and remove their own
project from it.
Also, be aware that this only for those jobs that have been enabled for
it in the project-config repository. It's done for all the generic tox
enabled targets and should be done for all custom tox targets as well.
Some repositories are not using constraints like project-config
itself, so those jobs are not set up.
Constraints for DevStack jobs
=
Devstack-gate takes care using constraints, there is nothing for a
repository to do to honor constraints.
Check the devstacklog.txt file, if constraints are in use it will use
lines like:
Collecting oslo.context===2.7.0 (from -c
/opt/stack/new/requirements/upper-constraints.txt (line 204))
References
==
http://docs.openstack.org/developer/requirements/
https://specs.openstack.org/openstack/openstack-specs/specs/requirements-management.html
Thanks
==
As usual in OpenStack, such work is a team work of many people. I'd like
to thank especially:
*Robert Collins 'lifeless': For writing the initial spec, implementation
work, and giving guideance on many of these details.
* Sean Dague: He was bold enough to try using constraints everywhere and
showing us where it failed.
* Sachi King for making zuul-cloner usable in the post queue. This was a
missing part in the last months.
* The OpenStack infra team for many reviews and design discussions -
especially to Jeremy Stanley and Jim Blair.
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][infra] Constraints are ready to be used for tox.ini
On 11 Aug 2016, at 10:03, Andreas Jaeger wrote:
> TL;DR: upper-constraints can be used now for all tox based jobs
>
> With any software package, you will need additional packages to run it.
> Often, there's a tight coupling: The software package will only run with
> specific other package versions. This dependency information is
> sometimes found in README files, in code, or in package metadata. If you
> install the package, you need to figure out the dependency and
> handle it properly.
>
> The Python package installer pip uses a list of requirements to install
> dependent Python packages. This list not only contains the name of
> packages but also limits which versions to use, or not to use.
> In OpenStack we handle these dependencies in a global requirements list
> and use it for most of the repositories. During initial testing a
> specific package version is tested but at a later point, another one
> might be used, or during deployment again another one.
>
> To document what was tested, give guidenance for deployment, and help to
> figure out breakage by upstream projects, the OpenStack requirements
> projects maintains a set of constraints with packages pinned to specific
> package versions that are known to be working.
> These are in the upper-constraints.txt file.
>
> Devstack already handles upper-constraints.txt when installing packages
> and I'm happy to say that tox, the Python testing framework used in
> OpenStack, can now handle upper-constraints as well everywhere.
>
>
> Constraints for tox based jobs
> ==
> To use constraints, change in tox.ini the install command to:
>
> install_command = pip install
> -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt}
> {opts} {packages}
I've proposed a patch to Swift to do this:
https://review.openstack.org/#/c/354291/
I'd appreciate any advice on it.
>
> Caveat
> ==
>
> Note that constraints are used for the installation of each packages, so
> if you want to install a package from source and have constraints for a
> specific version in the constraints file, it will not work. This happens
> with some of the OpenStack python client packages: When they install
> their dependencies, those might have a dependency on the client package
> itself. And this then will cause an error since the client package
> should get installed from source.
>
> So, projects need to remove the constraints file for themselves if they
> run into this. Packages like python-novaclient and python-glanceclient
> therefore use a wrapper (tools/tox_install.sh) as
> install command to edit the constraints file first and remove their own
> project from it.
>
> Also, be aware that this only for those jobs that have been enabled for
> it in the project-config repository. It's done for all the generic tox
> enabled targets and should be done for all custom tox targets as well.
> Some repositories are not using constraints like project-config
> itself, so those jobs are not set up.
>
> Constraints for DevStack jobs
> =
> Devstack-gate takes care using constraints, there is nothing for a
> repository to do to honor constraints.
>
> Check the devstacklog.txt file, if constraints are in use it will use
> lines like:
>
> Collecting oslo.context===2.7.0 (from -c
> /opt/stack/new/requirements/upper-constraints.txt (line 204))
>
> References
> ==
>
> http://docs.openstack.org/developer/requirements/
> https://specs.openstack.org/openstack/openstack-specs/specs/requirements-management.html
>
>
> Thanks
> ==
> As usual in OpenStack, such work is a team work of many people. I'd like
> to thank especially:
>
> *Robert Collins 'lifeless': For writing the initial spec, implementation
> work, and giving guideance on many of these details.
> * Sean Dague: He was bold enough to try using constraints everywhere and
> showing us where it failed.
> * Sachi King for making zuul-cloner usable in the post queue. This was a
> missing part in the last months.
> * The OpenStack infra team for many reviews and design discussions -
> especially to Jeremy Stanley and Jim Blair.
>
> --
> Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
>GF: Felix Imendörffer, Jane Smithard, Graham Norton,
>HRB 21284 (AG Nürnberg)
> GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-req
[openstack-dev] [all][infra] Constraints are ready to be used for tox.ini
TL;DR: upper-constraints can be used now for all tox based jobs
With any software package, you will need additional packages to run it.
Often, there's a tight coupling: The software package will only run with
specific other package versions. This dependency information is
sometimes found in README files, in code, or in package metadata. If you
install the package, you need to figure out the dependency and
handle it properly.
The Python package installer pip uses a list of requirements to install
dependent Python packages. This list not only contains the name of
packages but also limits which versions to use, or not to use.
In OpenStack we handle these dependencies in a global requirements list
and use it for most of the repositories. During initial testing a
specific package version is tested but at a later point, another one
might be used, or during deployment again another one.
To document what was tested, give guidenance for deployment, and help to
figure out breakage by upstream projects, the OpenStack requirements
projects maintains a set of constraints with packages pinned to specific
package versions that are known to be working.
These are in the upper-constraints.txt file.
Devstack already handles upper-constraints.txt when installing packages
and I'm happy to say that tox, the Python testing framework used in
OpenStack, can now handle upper-constraints as well everywhere.
Constraints for tox based jobs
==
To use constraints, change in tox.ini the install command to:
install_command = pip install
-c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt}
{opts} {packages}
Caveat
==
Note that constraints are used for the installation of each packages, so
if you want to install a package from source and have constraints for a
specific version in the constraints file, it will not work. This happens
with some of the OpenStack python client packages: When they install
their dependencies, those might have a dependency on the client package
itself. And this then will cause an error since the client package
should get installed from source.
So, projects need to remove the constraints file for themselves if they
run into this. Packages like python-novaclient and python-glanceclient
therefore use a wrapper (tools/tox_install.sh) as
install command to edit the constraints file first and remove their own
project from it.
Also, be aware that this only for those jobs that have been enabled for
it in the project-config repository. It's done for all the generic tox
enabled targets and should be done for all custom tox targets as well.
Some repositories are not using constraints like project-config
itself, so those jobs are not set up.
Constraints for DevStack jobs
=
Devstack-gate takes care using constraints, there is nothing for a
repository to do to honor constraints.
Check the devstacklog.txt file, if constraints are in use it will use
lines like:
Collecting oslo.context===2.7.0 (from -c
/opt/stack/new/requirements/upper-constraints.txt (line 204))
References
==
http://docs.openstack.org/developer/requirements/
https://specs.openstack.org/openstack/openstack-specs/specs/requirements-management.html
Thanks
==
As usual in OpenStack, such work is a team work of many people. I'd like
to thank especially:
*Robert Collins 'lifeless': For writing the initial spec, implementation
work, and giving guideance on many of these details.
* Sean Dague: He was bold enough to try using constraints everywhere and
showing us where it failed.
* Sachi King for making zuul-cloner usable in the post queue. This was a
missing part in the last months.
* The OpenStack infra team for many reviews and design discussions -
especially to Jeremy Stanley and Jim Blair.
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
