Re: [openstack-dev] [devstack][qa] Changes to devstack LIBS_FROM_GIT
On Thu, Mar 29, 2018 at 5:21 AM, James E. Blair wrote: > Hi, > > I've proposed a change to devstack which slightly alters the > LIBS_FROM_GIT behavior. This shouldn't be a significant change for > those using legacy devstack jobs (but you may want to be aware of it). > It is more significant for new-style devstack jobs. > > The change is at https://review.openstack.org/549252 > > In summary, when this change lands, new-style devstack jobs should no > longer need to set LIBS_FROM_GIT explicitly. Existing legacy jobs > should be unaffected (but there is a change to the verification process > performed by devstack). > > > Currently devstack expects the contents of LIBS_FROM_GIT to be > exclusively a list of python packages which, obviously, should be > installed from git and not pypi. It is used for two purposes: > determining whether an individual package should be installed from git, > and verifying that a package was installed from git. > > In the old devstack-gate system, we prepared many of the common git > repos, whether they were used or not. So LIBS_FROM_GIT was created to > indicate that in some cases devstack should ignore those repos and > install from pypi instead. In other words, its original purpose was > purely as a method of selecting whether a devstack-gate prepared repo > should be used or ignored. > > In Zuul v3, we have a good way to indicate whether a job is going to use > a repo or not -- add it to "required-projects". Considering that, the > LIBS_FROM_GIT variable is redundant. So my patch causes it to be > automatically generated based on the contents of required-projects. > This means that job authors don't need to list every required repository > twice. > > However, a naïve implementation of that runs afoul of the second use of > LIBS_FROM_GIT -- verifying that python packages are installed from git. > > This usage was added later, after a typographical error ("-" vs "_" in a > python package name) in a constraints file caused us not to install a > package from git. Now devstack verifies that every package in > LIBS_FROM_GIT is installed. However, Zuul doesn't know that devstack, > tempest, and other packages aren't installed. So adding them > automatically to LIBS_FROM_GIT will cause devstack to fail. > > My change modifies this verification to only check that packages > mentioned in LIBS_FROM_GIT that devstack tried to install were actually > installed. I realize that stated as such this sounds tautological, > however, this check is still valid -- it would have caught the original > error that prompted the check in the first case. > > What the revised check will no longer handle is a typo in a legacy job. > If someone enters a typo into LIBS_FROM_GIT, it will no longer fail. > However, I think the risk is worthwhile -- particularly since it is in > service of a system which eliminates the opportunity to introduce such > an error in the first place. > > To see the result in action, take a look at this change which, in only a > few lines, implements what was a significantly more complex undertaking > in Zuul v2: > > https://review.openstack.org/548331 > > Finally, a note on the automatic generation of LIBS_FROM_GIT -- if, for > some reason, you require a new-style devstack job to manually set > LIBS_FROM_GIT, that will still work. Simply define the variable as > normal, and the module which generates the devstack config will bypass > automatic generation if the variable is already set. +1, thanks Jim. idea looks good to me as long as it still works for non-zuulv3 users. ll check the patch. -gmann > > -Jim > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack][qa] Changes to devstack LIBS_FROM_GIT
> > Neither local nor third-party CI use should be affected. There's no > change in behavior based on current usage patterns. Only the caveat > that if you introduce an error into LIBS_FROM_GIT (e.g., a misspelled or > non-existent package name), it will not automatically be caught. > > -Jim Perfect, thanks Jim. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack][qa] Changes to devstack LIBS_FROM_GIT
Sean McGinnis writes: > On Wed, Mar 28, 2018 at 07:37:19PM -0400, Doug Hellmann wrote: >> Excerpts from corvus's message of 2018-03-28 13:21:38 -0700: >> > Hi, >> > >> > I've proposed a change to devstack which slightly alters the >> > LIBS_FROM_GIT behavior. This shouldn't be a significant change for >> > those using legacy devstack jobs (but you may want to be aware of it). >> > It is more significant for new-style devstack jobs. >> > >> > -snip- >> > >> >> How does this apply to uses of devstack outside of zuul, such as in a >> local development environment? >> >> Doug >> > > This is my question too. I know in Cinder there are a lot of third party CI > systems that do not use zuul. If they are impacted in any way by changes to > devstack, we will need to make sure they are all aware of those changes (and > have an alternative method for them to get the same functionality). Neither local nor third-party CI use should be affected. There's no change in behavior based on current usage patterns. Only the caveat that if you introduce an error into LIBS_FROM_GIT (e.g., a misspelled or non-existent package name), it will not automatically be caught. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack][qa] Changes to devstack LIBS_FROM_GIT
On Wed, Mar 28, 2018 at 07:37:19PM -0400, Doug Hellmann wrote: > Excerpts from corvus's message of 2018-03-28 13:21:38 -0700: > > Hi, > > > > I've proposed a change to devstack which slightly alters the > > LIBS_FROM_GIT behavior. This shouldn't be a significant change for > > those using legacy devstack jobs (but you may want to be aware of it). > > It is more significant for new-style devstack jobs. > > > > -snip- > > > > How does this apply to uses of devstack outside of zuul, such as in a > local development environment? > > Doug > This is my question too. I know in Cinder there are a lot of third party CI systems that do not use zuul. If they are impacted in any way by changes to devstack, we will need to make sure they are all aware of those changes (and have an alternative method for them to get the same functionality). Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack][qa] Changes to devstack LIBS_FROM_GIT
Excerpts from corvus's message of 2018-03-28 13:21:38 -0700: > Hi, > > I've proposed a change to devstack which slightly alters the > LIBS_FROM_GIT behavior. This shouldn't be a significant change for > those using legacy devstack jobs (but you may want to be aware of it). > It is more significant for new-style devstack jobs. > > The change is at https://review.openstack.org/549252 > > In summary, when this change lands, new-style devstack jobs should no > longer need to set LIBS_FROM_GIT explicitly. Existing legacy jobs > should be unaffected (but there is a change to the verification process > performed by devstack). > > > Currently devstack expects the contents of LIBS_FROM_GIT to be > exclusively a list of python packages which, obviously, should be > installed from git and not pypi. It is used for two purposes: > determining whether an individual package should be installed from git, > and verifying that a package was installed from git. > > In the old devstack-gate system, we prepared many of the common git > repos, whether they were used or not. So LIBS_FROM_GIT was created to > indicate that in some cases devstack should ignore those repos and > install from pypi instead. In other words, its original purpose was > purely as a method of selecting whether a devstack-gate prepared repo > should be used or ignored. > > In Zuul v3, we have a good way to indicate whether a job is going to use > a repo or not -- add it to "required-projects". Considering that, the > LIBS_FROM_GIT variable is redundant. So my patch causes it to be > automatically generated based on the contents of required-projects. > This means that job authors don't need to list every required repository > twice. > > However, a naïve implementation of that runs afoul of the second use of > LIBS_FROM_GIT -- verifying that python packages are installed from git. > > This usage was added later, after a typographical error ("-" vs "_" in a > python package name) in a constraints file caused us not to install a > package from git. Now devstack verifies that every package in > LIBS_FROM_GIT is installed. However, Zuul doesn't know that devstack, > tempest, and other packages aren't installed. So adding them > automatically to LIBS_FROM_GIT will cause devstack to fail. > > My change modifies this verification to only check that packages > mentioned in LIBS_FROM_GIT that devstack tried to install were actually > installed. I realize that stated as such this sounds tautological, > however, this check is still valid -- it would have caught the original > error that prompted the check in the first case. > > What the revised check will no longer handle is a typo in a legacy job. > If someone enters a typo into LIBS_FROM_GIT, it will no longer fail. > However, I think the risk is worthwhile -- particularly since it is in > service of a system which eliminates the opportunity to introduce such > an error in the first place. > > To see the result in action, take a look at this change which, in only a > few lines, implements what was a significantly more complex undertaking > in Zuul v2: > > https://review.openstack.org/548331 > > Finally, a note on the automatic generation of LIBS_FROM_GIT -- if, for > some reason, you require a new-style devstack job to manually set > LIBS_FROM_GIT, that will still work. Simply define the variable as > normal, and the module which generates the devstack config will bypass > automatic generation if the variable is already set. > > -Jim > How does this apply to uses of devstack outside of zuul, such as in a local development environment? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [devstack][qa] Changes to devstack LIBS_FROM_GIT
Hi, I've proposed a change to devstack which slightly alters the LIBS_FROM_GIT behavior. This shouldn't be a significant change for those using legacy devstack jobs (but you may want to be aware of it). It is more significant for new-style devstack jobs. The change is at https://review.openstack.org/549252 In summary, when this change lands, new-style devstack jobs should no longer need to set LIBS_FROM_GIT explicitly. Existing legacy jobs should be unaffected (but there is a change to the verification process performed by devstack). Currently devstack expects the contents of LIBS_FROM_GIT to be exclusively a list of python packages which, obviously, should be installed from git and not pypi. It is used for two purposes: determining whether an individual package should be installed from git, and verifying that a package was installed from git. In the old devstack-gate system, we prepared many of the common git repos, whether they were used or not. So LIBS_FROM_GIT was created to indicate that in some cases devstack should ignore those repos and install from pypi instead. In other words, its original purpose was purely as a method of selecting whether a devstack-gate prepared repo should be used or ignored. In Zuul v3, we have a good way to indicate whether a job is going to use a repo or not -- add it to "required-projects". Considering that, the LIBS_FROM_GIT variable is redundant. So my patch causes it to be automatically generated based on the contents of required-projects. This means that job authors don't need to list every required repository twice. However, a naïve implementation of that runs afoul of the second use of LIBS_FROM_GIT -- verifying that python packages are installed from git. This usage was added later, after a typographical error ("-" vs "_" in a python package name) in a constraints file caused us not to install a package from git. Now devstack verifies that every package in LIBS_FROM_GIT is installed. However, Zuul doesn't know that devstack, tempest, and other packages aren't installed. So adding them automatically to LIBS_FROM_GIT will cause devstack to fail. My change modifies this verification to only check that packages mentioned in LIBS_FROM_GIT that devstack tried to install were actually installed. I realize that stated as such this sounds tautological, however, this check is still valid -- it would have caught the original error that prompted the check in the first case. What the revised check will no longer handle is a typo in a legacy job. If someone enters a typo into LIBS_FROM_GIT, it will no longer fail. However, I think the risk is worthwhile -- particularly since it is in service of a system which eliminates the opportunity to introduce such an error in the first place. To see the result in action, take a look at this change which, in only a few lines, implements what was a significantly more complex undertaking in Zuul v2: https://review.openstack.org/548331 Finally, a note on the automatic generation of LIBS_FROM_GIT -- if, for some reason, you require a new-style devstack job to manually set LIBS_FROM_GIT, that will still work. Simply define the variable as normal, and the module which generates the devstack config will bypass automatic generation if the variable is already set. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev