Re: [openstack-dev] [devstack][qa] Changes to devstack LIBS_FROM_GIT

[Ghanshyam Mann]

On Thu, Mar 29, 2018 at 5:21 AM, James E. Blair  wrote:
> Hi,
>
> I've proposed a change to devstack which slightly alters the
> LIBS_FROM_GIT behavior.  This shouldn't be a significant change for
> those using legacy devstack jobs (but you may want to be aware of it).
> It is more significant for new-style devstack jobs.
>
> The change is at https://review.openstack.org/549252
>
> In summary, when this change lands, new-style devstack jobs should no
> longer need to set LIBS_FROM_GIT explicitly.  Existing legacy jobs
> should be unaffected (but there is a change to the verification process
> performed by devstack).
>
>
> Currently devstack expects the contents of LIBS_FROM_GIT to be
> exclusively a list of python packages which, obviously, should be
> installed from git and not pypi.  It is used for two purposes:
> determining whether an individual package should be installed from git,
> and verifying that a package was installed from git.
>
> In the old devstack-gate system, we prepared many of the common git
> repos, whether they were used or not.  So LIBS_FROM_GIT was created to
> indicate that in some cases devstack should ignore those repos and
> install from pypi instead.  In other words, its original purpose was
> purely as a method of selecting whether a devstack-gate prepared repo
> should be used or ignored.
>
> In Zuul v3, we have a good way to indicate whether a job is going to use
> a repo or not -- add it to "required-projects".  Considering that, the
> LIBS_FROM_GIT variable is redundant.  So my patch causes it to be
> automatically generated based on the contents of required-projects.
> This means that job authors don't need to list every required repository
> twice.
>
> However, a naïve implementation of that runs afoul of the second use of
> LIBS_FROM_GIT -- verifying that python packages are installed from git.
>
> This usage was added later, after a typographical error ("-" vs "_" in a
> python package name) in a constraints file caused us not to install a
> package from git.  Now devstack verifies that every package in
> LIBS_FROM_GIT is installed.  However, Zuul doesn't know that devstack,
> tempest, and other packages aren't installed.  So adding them
> automatically to LIBS_FROM_GIT will cause devstack to fail.
>
> My change modifies this verification to only check that packages
> mentioned in LIBS_FROM_GIT that devstack tried to install were actually
> installed.  I realize that stated as such this sounds tautological,
> however, this check is still valid -- it would have caught the original
> error that prompted the check in the first case.
>
> What the revised check will no longer handle is a typo in a legacy job.
> If someone enters a typo into LIBS_FROM_GIT, it will no longer fail.
> However, I think the risk is worthwhile -- particularly since it is in
> service of a system which eliminates the opportunity to introduce such
> an error in the first place.
>
> To see the result in action, take a look at this change which, in only a
> few lines, implements what was a significantly more complex undertaking
> in Zuul v2:
>
> https://review.openstack.org/548331
>
> Finally, a note on the automatic generation of LIBS_FROM_GIT -- if, for
> some reason, you require a new-style devstack job to manually set
> LIBS_FROM_GIT, that will still work.  Simply define the variable as
> normal, and the module which generates the devstack config will bypass
> automatic generation if the variable is already set.

+1, thanks Jim. idea looks good to me as long as it still works for
non-zuulv3 users. ll check the patch.

-gmann

>
> -Jim
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack][qa] Changes to devstack LIBS_FROM_GIT

[Sean McGinnis]

> 
> Neither local nor third-party CI use should be affected.  There's no
> change in behavior based on current usage patterns.  Only the caveat
> that if you introduce an error into LIBS_FROM_GIT (e.g., a misspelled or
> non-existent package name), it will not automatically be caught.
> 
> -Jim

Perfect, thanks Jim.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack][qa] Changes to devstack LIBS_FROM_GIT

[James E. Blair]

Sean McGinnis  writes:

> On Wed, Mar 28, 2018 at 07:37:19PM -0400, Doug Hellmann wrote:
>> Excerpts from corvus's message of 2018-03-28 13:21:38 -0700:
>> > Hi,
>> > 
>> > I've proposed a change to devstack which slightly alters the
>> > LIBS_FROM_GIT behavior.  This shouldn't be a significant change for
>> > those using legacy devstack jobs (but you may want to be aware of it).
>> > It is more significant for new-style devstack jobs.
>> > 
>> > -snip-
>> > 
>> 
>> How does this apply to uses of devstack outside of zuul, such as in a
>> local development environment?
>> 
>> Doug
>> 
>
> This is my question too. I know in Cinder there are a lot of third party CI
> systems that do not use zuul. If they are impacted in any way by changes to
> devstack, we will need to make sure they are all aware of those changes (and
> have an alternative method for them to get the same functionality).

Neither local nor third-party CI use should be affected.  There's no
change in behavior based on current usage patterns.  Only the caveat
that if you introduce an error into LIBS_FROM_GIT (e.g., a misspelled or
non-existent package name), it will not automatically be caught.

-Jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack][qa] Changes to devstack LIBS_FROM_GIT

[Sean McGinnis]

On Wed, Mar 28, 2018 at 07:37:19PM -0400, Doug Hellmann wrote:
> Excerpts from corvus's message of 2018-03-28 13:21:38 -0700:
> > Hi,
> > 
> > I've proposed a change to devstack which slightly alters the
> > LIBS_FROM_GIT behavior.  This shouldn't be a significant change for
> > those using legacy devstack jobs (but you may want to be aware of it).
> > It is more significant for new-style devstack jobs.
> > 
> > -snip-
> > 
> 
> How does this apply to uses of devstack outside of zuul, such as in a
> local development environment?
> 
> Doug
> 

This is my question too. I know in Cinder there are a lot of third party CI
systems that do not use zuul. If they are impacted in any way by changes to
devstack, we will need to make sure they are all aware of those changes (and
have an alternative method for them to get the same functionality).

Sean

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack][qa] Changes to devstack LIBS_FROM_GIT

[Doug Hellmann]

Excerpts from corvus's message of 2018-03-28 13:21:38 -0700:
> Hi,
> 
> I've proposed a change to devstack which slightly alters the
> LIBS_FROM_GIT behavior.  This shouldn't be a significant change for
> those using legacy devstack jobs (but you may want to be aware of it).
> It is more significant for new-style devstack jobs.
> 
> The change is at https://review.openstack.org/549252
> 
> In summary, when this change lands, new-style devstack jobs should no
> longer need to set LIBS_FROM_GIT explicitly.  Existing legacy jobs
> should be unaffected (but there is a change to the verification process
> performed by devstack).
> 
> 
> Currently devstack expects the contents of LIBS_FROM_GIT to be
> exclusively a list of python packages which, obviously, should be
> installed from git and not pypi.  It is used for two purposes:
> determining whether an individual package should be installed from git,
> and verifying that a package was installed from git.
> 
> In the old devstack-gate system, we prepared many of the common git
> repos, whether they were used or not.  So LIBS_FROM_GIT was created to
> indicate that in some cases devstack should ignore those repos and
> install from pypi instead.  In other words, its original purpose was
> purely as a method of selecting whether a devstack-gate prepared repo
> should be used or ignored.
> 
> In Zuul v3, we have a good way to indicate whether a job is going to use
> a repo or not -- add it to "required-projects".  Considering that, the
> LIBS_FROM_GIT variable is redundant.  So my patch causes it to be
> automatically generated based on the contents of required-projects.
> This means that job authors don't need to list every required repository
> twice.
> 
> However, a naïve implementation of that runs afoul of the second use of
> LIBS_FROM_GIT -- verifying that python packages are installed from git.
> 
> This usage was added later, after a typographical error ("-" vs "_" in a
> python package name) in a constraints file caused us not to install a
> package from git.  Now devstack verifies that every package in
> LIBS_FROM_GIT is installed.  However, Zuul doesn't know that devstack,
> tempest, and other packages aren't installed.  So adding them
> automatically to LIBS_FROM_GIT will cause devstack to fail.
> 
> My change modifies this verification to only check that packages
> mentioned in LIBS_FROM_GIT that devstack tried to install were actually
> installed.  I realize that stated as such this sounds tautological,
> however, this check is still valid -- it would have caught the original
> error that prompted the check in the first case.
> 
> What the revised check will no longer handle is a typo in a legacy job.
> If someone enters a typo into LIBS_FROM_GIT, it will no longer fail.
> However, I think the risk is worthwhile -- particularly since it is in
> service of a system which eliminates the opportunity to introduce such
> an error in the first place.
> 
> To see the result in action, take a look at this change which, in only a
> few lines, implements what was a significantly more complex undertaking
> in Zuul v2:
> 
> https://review.openstack.org/548331
> 
> Finally, a note on the automatic generation of LIBS_FROM_GIT -- if, for
> some reason, you require a new-style devstack job to manually set
> LIBS_FROM_GIT, that will still work.  Simply define the variable as
> normal, and the module which generates the devstack config will bypass
> automatic generation if the variable is already set.
> 
> -Jim
> 

How does this apply to uses of devstack outside of zuul, such as in a
local development environment?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [devstack][qa] Changes to devstack LIBS_FROM_GIT

[James E. Blair]

Hi,

I've proposed a change to devstack which slightly alters the
LIBS_FROM_GIT behavior.  This shouldn't be a significant change for
those using legacy devstack jobs (but you may want to be aware of it).
It is more significant for new-style devstack jobs.

The change is at https://review.openstack.org/549252

In summary, when this change lands, new-style devstack jobs should no
longer need to set LIBS_FROM_GIT explicitly.  Existing legacy jobs
should be unaffected (but there is a change to the verification process
performed by devstack).


Currently devstack expects the contents of LIBS_FROM_GIT to be
exclusively a list of python packages which, obviously, should be
installed from git and not pypi.  It is used for two purposes:
determining whether an individual package should be installed from git,
and verifying that a package was installed from git.

In the old devstack-gate system, we prepared many of the common git
repos, whether they were used or not.  So LIBS_FROM_GIT was created to
indicate that in some cases devstack should ignore those repos and
install from pypi instead.  In other words, its original purpose was
purely as a method of selecting whether a devstack-gate prepared repo
should be used or ignored.

In Zuul v3, we have a good way to indicate whether a job is going to use
a repo or not -- add it to "required-projects".  Considering that, the
LIBS_FROM_GIT variable is redundant.  So my patch causes it to be
automatically generated based on the contents of required-projects.
This means that job authors don't need to list every required repository
twice.

However, a naïve implementation of that runs afoul of the second use of
LIBS_FROM_GIT -- verifying that python packages are installed from git.

This usage was added later, after a typographical error ("-" vs "_" in a
python package name) in a constraints file caused us not to install a
package from git.  Now devstack verifies that every package in
LIBS_FROM_GIT is installed.  However, Zuul doesn't know that devstack,
tempest, and other packages aren't installed.  So adding them
automatically to LIBS_FROM_GIT will cause devstack to fail.

My change modifies this verification to only check that packages
mentioned in LIBS_FROM_GIT that devstack tried to install were actually
installed.  I realize that stated as such this sounds tautological,
however, this check is still valid -- it would have caught the original
error that prompted the check in the first case.

What the revised check will no longer handle is a typo in a legacy job.
If someone enters a typo into LIBS_FROM_GIT, it will no longer fail.
However, I think the risk is worthwhile -- particularly since it is in
service of a system which eliminates the opportunity to introduce such
an error in the first place.

To see the result in action, take a look at this change which, in only a
few lines, implements what was a significantly more complex undertaking
in Zuul v2:

https://review.openstack.org/548331

Finally, a note on the automatic generation of LIBS_FROM_GIT -- if, for
some reason, you require a new-style devstack job to manually set
LIBS_FROM_GIT, that will still work.  Simply define the variable as
normal, and the module which generates the devstack config will bypass
automatic generation if the variable is already set.

-Jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev