Re: [openstack-dev] [puppet] use zuul-cloner when running rspec

[Sofer Athlan-Guyot]

Emilien Macchi  writes:

> Background
> ==
>
> Current rspec tests are tested with modules mentioned in .fixtures.yaml
> file of each module.
>
> * the file is not consistent across all modules
> * it hardcodes module names & versions

IMHO, this alone justify it.

> * this way does not allow to use "Depend-On" feature, that would allow
> to test cross-modules patches
>
> Proposal
> 
>
> * Like we do in beaker & integration jobs, use zuul-cloner to clone
> modules in our CI jobs.
> * Use r10k to prepare fixtures modules.
> * Use Puppetfile hosted by openstack/puppet-openstack-integration
>
> In that way:
> * we will have modules name + versions testing consistency across all
> modules
> * the same Puppetfile would be used by unit/beaker/integration testing.
> * the patch that pass tests on your laptop would pass tests in upstream CI
> * if you don't have zuul-cloner on your laptop, don't worry it will use
> git clone. Though you won't have Depends-On feature working on your
> laptop (technically not possible).
> * Though your patch will support Depends-On in OpenStack Infra for unit
> tests. If you submit a patch in puppet-openstacklib that drop something
> wrong, you can send a patch in puppet-nova that will test it, and unit
> tests will fail.

+1

>
> Drawbacks
> =
> * cloning from .fixtures.yaml takes ~ 10 seconds
> * using r10k + zuul-clone takes ~50 seconds (more modules to clone).
>
> I think 40 seconds is something accept regarding the benefit.

Especially if one is using this workflow : 
 1. rake spec_prep and then:
- rake spec_standalone;
- rake spec_standalone;
- rake spec_standalone;
- ...

So it's a one time 40 seconds.

>
> Next steps
> ==
>
> * PoC in puppet-nova: https://review.openstack.org/#/c/226830/
> * Patch openstack/puppet-modulesync-config to be consistent across all
> our modules.
>
> Bonus
> =
> we might need (asap) a canary job for puppet-openstack-integration
> repository, that would run tests on a puppet-* module (since we're using
> install_modules.sh & Puppetfile files in puppet-* modules).
> Nothing has been done yet for this work.
>
>
> Thoughts?

-- 
Sofer Athlan-Guyot

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] use zuul-cloner when running rspec

[Alex Schultz]

On Wed, Sep 23, 2015 at 4:56 PM, Emilien Macchi  wrote:
> Background
> ==
>
> Current rspec tests are tested with modules mentioned in .fixtures.yaml
> file of each module.
>
> * the file is not consistent across all modules
> * it hardcodes module names & versions
> * this way does not allow to use "Depend-On" feature, that would allow
> to test cross-modules patches
>
> Proposal
> 
>
> * Like we do in beaker & integration jobs, use zuul-cloner to clone
> modules in our CI jobs.
> * Use r10k to prepare fixtures modules.
> * Use Puppetfile hosted by openstack/puppet-openstack-integration
>
> In that way:
> * we will have modules name + versions testing consistency across all
> modules
> * the same Puppetfile would be used by unit/beaker/integration testing.
> * the patch that pass tests on your laptop would pass tests in upstream CI
> * if you don't have zuul-cloner on your laptop, don't worry it will use
> git clone. Though you won't have Depends-On feature working on your
> laptop (technically not possible).
> * Though your patch will support Depends-On in OpenStack Infra for unit
> tests. If you submit a patch in puppet-openstacklib that drop something
> wrong, you can send a patch in puppet-nova that will test it, and unit
> tests will fail.
>
> Drawbacks
> =
> * cloning from .fixtures.yaml takes ~ 10 seconds
> * using r10k + zuul-clone takes ~50 seconds (more modules to clone).
>
> I think 40 seconds is something accept regarding the benefit.
>

As someone who consumes these modules downstream and has our own CI
setup to run the rspec items, this ties it too closely to the
openstack infrastructure. If we replace the .fixtures.yml with
zuul-cloner, it assumes I always want the openstack version of the
modules. This is not necessarily true. I like being able to replace
items within fixtures.yml when doing dev work. For example If i want
to test upgrading another module not related to openstack, like
inifile, how does that work with the proposed solution?  This is also
moving away from general puppet module conventions for testing. My
preference would be that this be a different task and we have both
.fixtures.yml (for general use/development) and the zuul method of
cloning (for CI).  You have to also think about this from a consumer
standpoint and this is adding an external dependency on the OpenStack
infrastructure for anyone trying to run rspec or trying to consume the
published versions from the forge.  Would I be able to run these tests
in an offline mode with this change? With the .fixures.yml it's a
minor edit to switch to local versions. Is the same true for the
zuul-cloner version?

>
> Next steps
> ==
>
> * PoC in puppet-nova: https://review.openstack.org/#/c/226830/
> * Patch openstack/puppet-modulesync-config to be consistent across all
> our modules.
>
> Bonus
> =
> we might need (asap) a canary job for puppet-openstack-integration
> repository, that would run tests on a puppet-* module (since we're using
> install_modules.sh & Puppetfile files in puppet-* modules).
> Nothing has been done yet for this work.
>
>
> Thoughts?
> --
> Emilien Macchi
>
>

I think we need this functionality, I just don't think it's a
replacement for the .fixures.yml.

Thanks,
-Alex

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] use zuul-cloner when running rspec

[Emilien Macchi]

On 09/24/2015 02:19 PM, Alex Schultz wrote:
> On Thu, Sep 24, 2015 at 11:54 AM, Emilien Macchi  wrote:
>>
>>
>> On 09/24/2015 10:14 AM, Alex Schultz wrote:
>>> On Wed, Sep 23, 2015 at 4:56 PM, Emilien Macchi  wrote:
 Background
 ==

 Current rspec tests are tested with modules mentioned in .fixtures.yaml
 file of each module.

 * the file is not consistent across all modules
 * it hardcodes module names & versions
 * this way does not allow to use "Depend-On" feature, that would allow
 to test cross-modules patches

 Proposal
 

 * Like we do in beaker & integration jobs, use zuul-cloner to clone
 modules in our CI jobs.
 * Use r10k to prepare fixtures modules.
 * Use Puppetfile hosted by openstack/puppet-openstack-integration

 In that way:
 * we will have modules name + versions testing consistency across all
 modules
 * the same Puppetfile would be used by unit/beaker/integration testing.
 * the patch that pass tests on your laptop would pass tests in upstream CI
 * if you don't have zuul-cloner on your laptop, don't worry it will use
 git clone. Though you won't have Depends-On feature working on your
 laptop (technically not possible).
 * Though your patch will support Depends-On in OpenStack Infra for unit
 tests. If you submit a patch in puppet-openstacklib that drop something
 wrong, you can send a patch in puppet-nova that will test it, and unit
 tests will fail.

 Drawbacks
 =
 * cloning from .fixtures.yaml takes ~ 10 seconds
 * using r10k + zuul-clone takes ~50 seconds (more modules to clone).

 I think 40 seconds is something accept regarding the benefit.

>>>
>>> As someone who consumes these modules downstream and has our own CI
>>> setup to run the rspec items, this ties it too closely to the
>>> openstack infrastructure. If we replace the .fixtures.yml with
>>> zuul-cloner, it assumes I always want the openstack version of the
>>> modules. This is not necessarily true. I like being able to replace
>>> items within fixtures.yml when doing dev work. For example If i want
>>> to test upgrading another module not related to openstack, like
>>> inifile, how does that work with the proposed solution?  This is also
>>> moving away from general puppet module conventions for testing. My
>>> preference would be that this be a different task and we have both
>>> .fixtures.yml (for general use/development) and the zuul method of
>>> cloning (for CI).  You have to also think about this from a consumer
>>> standpoint and this is adding an external dependency on the OpenStack
>>> infrastructure for anyone trying to run rspec or trying to consume the
>>> published versions from the forge.  Would I be able to run these tests
>>> in an offline mode with this change? With the .fixures.yml it's a
>>> minor edit to switch to local versions. Is the same true for the
>>> zuul-cloner version?
>>
>> What you did before:
>> * Edit .fixtures.yaml and put the version you like.
>>
>> What you would do this the current proposal:
>> * Edit openstack/puppet-openstack-integration/Puppetfile and put the
>> version you like.
>>
> 
> So I have to edit a file in another module to test changes in
> puppet-neutron, puppet-nova, etc? With the zuul-cloner version, for
> local testing what does that workflow look like?

If you need to test your code with cross-project dependencies, having
current .fixtures.yaml or the proposal won't change anything regarding
that, you'll still have to trick the YAML file that define the modules
name/versions.

> 
>> What you're suggesting has a huge downside:
>> People will still use fixtures by default and not test what is actually
>> tested by our CI.
>> A few people will know about the specific Rake task so a few people will
>> test exactly what upstream does. That will cause frustration to the most
>> of people who will see tests failing in our CI and not on their laptop.
>> I'm not sure we want that.
> 
> You're right that the specific rake task may not be ideal. But that
> was one option, another option could be use fixtures first then
> replace with zuul-cloner provided versions but provide me the ability
> to turn of the zuul cloner part? I'm just saying as it is today, this
> change adds more complexity and hard ties into the OpenStack
> infrastructure with non-trival work arounds. I would love to solve the
> Depends-On issue, but I don't think that should include a deviation
> from generally accepted testing practices of puppet modules.

I agree it's not best practice in Puppet but I don't see that as an huge
blocker. Our Puppet modules are Approved by Puppetlabs and respect most
of best practices AFIK. Is that fixctures thing a big deal?
I would like to hear from *cough*Hunner/Cody*cough* Puppetlabs about that.
Another proposal is welcome though, please go ahead.

>>

Re: [openstack-dev] [puppet] use zuul-cloner when running rspec

[Alex Schultz]

On Thu, Sep 24, 2015 at 1:58 PM, Emilien Macchi  wrote:
>
>
> On 09/24/2015 02:19 PM, Alex Schultz wrote:
>> On Thu, Sep 24, 2015 at 11:54 AM, Emilien Macchi  wrote:
>>>
>>>
>>> On 09/24/2015 10:14 AM, Alex Schultz wrote:
 On Wed, Sep 23, 2015 at 4:56 PM, Emilien Macchi  wrote:
> Background
> ==
>
> Current rspec tests are tested with modules mentioned in .fixtures.yaml
> file of each module.
>
> * the file is not consistent across all modules
> * it hardcodes module names & versions
> * this way does not allow to use "Depend-On" feature, that would allow
> to test cross-modules patches
>
> Proposal
> 
>
> * Like we do in beaker & integration jobs, use zuul-cloner to clone
> modules in our CI jobs.
> * Use r10k to prepare fixtures modules.
> * Use Puppetfile hosted by openstack/puppet-openstack-integration
>
> In that way:
> * we will have modules name + versions testing consistency across all
> modules
> * the same Puppetfile would be used by unit/beaker/integration testing.
> * the patch that pass tests on your laptop would pass tests in upstream CI
> * if you don't have zuul-cloner on your laptop, don't worry it will use
> git clone. Though you won't have Depends-On feature working on your
> laptop (technically not possible).
> * Though your patch will support Depends-On in OpenStack Infra for unit
> tests. If you submit a patch in puppet-openstacklib that drop something
> wrong, you can send a patch in puppet-nova that will test it, and unit
> tests will fail.
>
> Drawbacks
> =
> * cloning from .fixtures.yaml takes ~ 10 seconds
> * using r10k + zuul-clone takes ~50 seconds (more modules to clone).
>
> I think 40 seconds is something accept regarding the benefit.
>

 As someone who consumes these modules downstream and has our own CI
 setup to run the rspec items, this ties it too closely to the
 openstack infrastructure. If we replace the .fixtures.yml with
 zuul-cloner, it assumes I always want the openstack version of the
 modules. This is not necessarily true. I like being able to replace
 items within fixtures.yml when doing dev work. For example If i want
 to test upgrading another module not related to openstack, like
 inifile, how does that work with the proposed solution?  This is also
 moving away from general puppet module conventions for testing. My
 preference would be that this be a different task and we have both
 .fixtures.yml (for general use/development) and the zuul method of
 cloning (for CI).  You have to also think about this from a consumer
 standpoint and this is adding an external dependency on the OpenStack
 infrastructure for anyone trying to run rspec or trying to consume the
 published versions from the forge.  Would I be able to run these tests
 in an offline mode with this change? With the .fixures.yml it's a
 minor edit to switch to local versions. Is the same true for the
 zuul-cloner version?
>>>
>>> What you did before:
>>> * Edit .fixtures.yaml and put the version you like.
>>>
>>> What you would do this the current proposal:
>>> * Edit openstack/puppet-openstack-integration/Puppetfile and put the
>>> version you like.
>>>
>>
>> So I have to edit a file in another module to test changes in
>> puppet-neutron, puppet-nova, etc? With the zuul-cloner version, for
>> local testing what does that workflow look like?
>
> If you need to test your code with cross-project dependencies, having
> current .fixtures.yaml or the proposal won't change anything regarding
> that, you'll still have to trick the YAML file that define the modules
> name/versions.
>
>>
>>> What you're suggesting has a huge downside:
>>> People will still use fixtures by default and not test what is actually
>>> tested by our CI.
>>> A few people will know about the specific Rake task so a few people will
>>> test exactly what upstream does. That will cause frustration to the most
>>> of people who will see tests failing in our CI and not on their laptop.
>>> I'm not sure we want that.
>>
>> You're right that the specific rake task may not be ideal. But that
>> was one option, another option could be use fixtures first then
>> replace with zuul-cloner provided versions but provide me the ability
>> to turn of the zuul cloner part? I'm just saying as it is today, this
>> change adds more complexity and hard ties into the OpenStack
>> infrastructure with non-trival work arounds. I would love to solve the
>> Depends-On issue, but I don't think that should include a deviation
>> from generally accepted testing practices of puppet modules.
>
> I agree it's not best practice in Puppet but I don't see that as an huge
> blocker. Our Puppet modules are Approved by Puppetlabs and respect most
> of best 

Re: [openstack-dev] [puppet] use zuul-cloner when running rspec

[Alex Schultz]

On Thu, Sep 24, 2015 at 11:54 AM, Emilien Macchi  wrote:
>
>
> On 09/24/2015 10:14 AM, Alex Schultz wrote:
>> On Wed, Sep 23, 2015 at 4:56 PM, Emilien Macchi  wrote:
>>> Background
>>> ==
>>>
>>> Current rspec tests are tested with modules mentioned in .fixtures.yaml
>>> file of each module.
>>>
>>> * the file is not consistent across all modules
>>> * it hardcodes module names & versions
>>> * this way does not allow to use "Depend-On" feature, that would allow
>>> to test cross-modules patches
>>>
>>> Proposal
>>> 
>>>
>>> * Like we do in beaker & integration jobs, use zuul-cloner to clone
>>> modules in our CI jobs.
>>> * Use r10k to prepare fixtures modules.
>>> * Use Puppetfile hosted by openstack/puppet-openstack-integration
>>>
>>> In that way:
>>> * we will have modules name + versions testing consistency across all
>>> modules
>>> * the same Puppetfile would be used by unit/beaker/integration testing.
>>> * the patch that pass tests on your laptop would pass tests in upstream CI
>>> * if you don't have zuul-cloner on your laptop, don't worry it will use
>>> git clone. Though you won't have Depends-On feature working on your
>>> laptop (technically not possible).
>>> * Though your patch will support Depends-On in OpenStack Infra for unit
>>> tests. If you submit a patch in puppet-openstacklib that drop something
>>> wrong, you can send a patch in puppet-nova that will test it, and unit
>>> tests will fail.
>>>
>>> Drawbacks
>>> =
>>> * cloning from .fixtures.yaml takes ~ 10 seconds
>>> * using r10k + zuul-clone takes ~50 seconds (more modules to clone).
>>>
>>> I think 40 seconds is something accept regarding the benefit.
>>>
>>
>> As someone who consumes these modules downstream and has our own CI
>> setup to run the rspec items, this ties it too closely to the
>> openstack infrastructure. If we replace the .fixtures.yml with
>> zuul-cloner, it assumes I always want the openstack version of the
>> modules. This is not necessarily true. I like being able to replace
>> items within fixtures.yml when doing dev work. For example If i want
>> to test upgrading another module not related to openstack, like
>> inifile, how does that work with the proposed solution?  This is also
>> moving away from general puppet module conventions for testing. My
>> preference would be that this be a different task and we have both
>> .fixtures.yml (for general use/development) and the zuul method of
>> cloning (for CI).  You have to also think about this from a consumer
>> standpoint and this is adding an external dependency on the OpenStack
>> infrastructure for anyone trying to run rspec or trying to consume the
>> published versions from the forge.  Would I be able to run these tests
>> in an offline mode with this change? With the .fixures.yml it's a
>> minor edit to switch to local versions. Is the same true for the
>> zuul-cloner version?
>
> What you did before:
> * Edit .fixtures.yaml and put the version you like.
>
> What you would do this the current proposal:
> * Edit openstack/puppet-openstack-integration/Puppetfile and put the
> version you like.
>

So I have to edit a file in another module to test changes in
puppet-neutron, puppet-nova, etc? With the zuul-cloner version, for
local testing what does that workflow look like?

> What you're suggesting has a huge downside:
> People will still use fixtures by default and not test what is actually
> tested by our CI.
> A few people will know about the specific Rake task so a few people will
> test exactly what upstream does. That will cause frustration to the most
> of people who will see tests failing in our CI and not on their laptop.
> I'm not sure we want that.

You're right that the specific rake task may not be ideal. But that
was one option, another option could be use fixtures first then
replace with zuul-cloner provided versions but provide me the ability
to turn of the zuul cloner part? I'm just saying as it is today, this
change adds more complexity and hard ties into the OpenStack
infrastructure with non-trival work arounds. I would love to solve the
Depends-On issue, but I don't think that should include a deviation
from generally accepted testing practices of puppet modules.

>
> I think more than most of people that run tests on their laptops want to
> see them passing in upstream CI.
> The few people that want to trick versions & modules, will have to run
> Rake, trick the Puppetfile and run Rake again. It's not a big deal and
> I'm sure this few people can deal with that.
>

So for me the zuul-cloner task seems more of a CI specific job that
solves the Depends-On issues we currently have. Much like the beaker
and acceptance tests that's not something I run locally. I usually run
the local rspec tests first before shipping off to CI to see how that
plays out but I would manage the .fixtures.yml if necessary to test
cross module dependancies.  I don't expect to replicate an entire 

Re: [openstack-dev] [puppet] use zuul-cloner when running rspec

[Clark Boylan]

On Thu, Sep 24, 2015, at 12:39 PM, Alex Schultz wrote:
> On Thu, Sep 24, 2015 at 1:58 PM, Emilien Macchi 
> wrote:
> >
> >
> > On 09/24/2015 02:19 PM, Alex Schultz wrote:
> >> On Thu, Sep 24, 2015 at 11:54 AM, Emilien Macchi  
> >> wrote:
> >>>
> >>>
> >>> On 09/24/2015 10:14 AM, Alex Schultz wrote:
>  On Wed, Sep 23, 2015 at 4:56 PM, Emilien Macchi  
>  wrote:
> > Background
> > ==
> >
> > Current rspec tests are tested with modules mentioned in .fixtures.yaml
> > file of each module.
> >
> > * the file is not consistent across all modules
> > * it hardcodes module names & versions
> > * this way does not allow to use "Depend-On" feature, that would allow
> > to test cross-modules patches
> >
> > Proposal
> > 
> >
> > * Like we do in beaker & integration jobs, use zuul-cloner to clone
> > modules in our CI jobs.
> > * Use r10k to prepare fixtures modules.
> > * Use Puppetfile hosted by openstack/puppet-openstack-integration
> >
> > In that way:
> > * we will have modules name + versions testing consistency across all
> > modules
> > * the same Puppetfile would be used by unit/beaker/integration testing.
> > * the patch that pass tests on your laptop would pass tests in upstream 
> > CI
> > * if you don't have zuul-cloner on your laptop, don't worry it will use
> > git clone. Though you won't have Depends-On feature working on your
> > laptop (technically not possible).
> > * Though your patch will support Depends-On in OpenStack Infra for unit
> > tests. If you submit a patch in puppet-openstacklib that drop something
> > wrong, you can send a patch in puppet-nova that will test it, and unit
> > tests will fail.
> >
> > Drawbacks
> > =
> > * cloning from .fixtures.yaml takes ~ 10 seconds
> > * using r10k + zuul-clone takes ~50 seconds (more modules to clone).
> >
> > I think 40 seconds is something accept regarding the benefit.
> >
> 
>  As someone who consumes these modules downstream and has our own CI
>  setup to run the rspec items, this ties it too closely to the
>  openstack infrastructure. If we replace the .fixtures.yml with
>  zuul-cloner, it assumes I always want the openstack version of the
>  modules. This is not necessarily true. I like being able to replace
>  items within fixtures.yml when doing dev work. For example If i want
>  to test upgrading another module not related to openstack, like
>  inifile, how does that work with the proposed solution?  This is also
>  moving away from general puppet module conventions for testing. My
>  preference would be that this be a different task and we have both
>  .fixtures.yml (for general use/development) and the zuul method of
>  cloning (for CI).  You have to also think about this from a consumer
>  standpoint and this is adding an external dependency on the OpenStack
>  infrastructure for anyone trying to run rspec or trying to consume the
>  published versions from the forge.  Would I be able to run these tests
>  in an offline mode with this change? With the .fixures.yml it's a
>  minor edit to switch to local versions. Is the same true for the
>  zuul-cloner version?
> >>>
> >>> What you did before:
> >>> * Edit .fixtures.yaml and put the version you like.
> >>>
> >>> What you would do this the current proposal:
> >>> * Edit openstack/puppet-openstack-integration/Puppetfile and put the
> >>> version you like.
> >>>
> >>
> >> So I have to edit a file in another module to test changes in
> >> puppet-neutron, puppet-nova, etc? With the zuul-cloner version, for
> >> local testing what does that workflow look like?
> >
> > If you need to test your code with cross-project dependencies, having
> > current .fixtures.yaml or the proposal won't change anything regarding
> > that, you'll still have to trick the YAML file that define the modules
> > name/versions.
> >
> >>
> >>> What you're suggesting has a huge downside:
> >>> People will still use fixtures by default and not test what is actually
> >>> tested by our CI.
> >>> A few people will know about the specific Rake task so a few people will
> >>> test exactly what upstream does. That will cause frustration to the most
> >>> of people who will see tests failing in our CI and not on their laptop.
> >>> I'm not sure we want that.
> >>
> >> You're right that the specific rake task may not be ideal. But that
> >> was one option, another option could be use fixtures first then
> >> replace with zuul-cloner provided versions but provide me the ability
> >> to turn of the zuul cloner part? I'm just saying as it is today, this
> >> change adds more complexity and hard ties into the OpenStack
> >> infrastructure with non-trival work arounds. I would love to solve the
> >> Depends-On 

Re: [openstack-dev] [puppet] use zuul-cloner when running rspec

[Jeremy Stanley]

On 2015-09-24 14:39:49 -0500 (-0500), Alex Schultz wrote:
[...]
> Being able to run tests without internet connectivity is important to
> some people so I want to make sure that can continue without having to
> break the process mid-cycle to try and inject a workaround. It would
> better if we could have a workaround upfront. For example make a
> Puppetfile location an environment variable and if not defined pull
> down the puppet-openstack-integration one?  I wish there was a better
> dependency resolution method that just pulling everything down from
> the internets.  I just know that doesn't work everywhere.

To build on Clark's response, THIS is basically why we use tools
like zuul-cloner. In our CI we're attempting to minimize or even
eliminate network use during tests, and so zuul-cloner leverages
local caches and is sufficiently flexible to obtain the repositories
in question from anywhere you want (which could also just be to
always use your locally cached/modified copy and never hit the
network at all). Pass it whatever Git repository URLs you want,
including file:///some/thing.git if that's your preference.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] use zuul-cloner when running rspec

[Emilien Macchi]

On 09/24/2015 05:16 PM, Jeremy Stanley wrote:
> On 2015-09-24 14:39:49 -0500 (-0500), Alex Schultz wrote:
> [...]
>> Being able to run tests without internet connectivity is important to
>> some people so I want to make sure that can continue without having to
>> break the process mid-cycle to try and inject a workaround. It would
>> better if we could have a workaround upfront. For example make a
>> Puppetfile location an environment variable and if not defined pull
>> down the puppet-openstack-integration one?  I wish there was a better
>> dependency resolution method that just pulling everything down from
>> the internets.  I just know that doesn't work everywhere.
> 
> To build on Clark's response, THIS is basically why we use tools
> like zuul-cloner. In our CI we're attempting to minimize or even
> eliminate network use during tests, and so zuul-cloner leverages
> local caches and is sufficiently flexible to obtain the repositories
> in question from anywhere you want (which could also just be to
> always use your locally cached/modified copy and never hit the
> network at all). Pass it whatever Git repository URLs you want,
> including file:///some/thing.git if that's your preference.
> 

So we had a discussion on #puppet-openstack channel, and Alex's main
concern was people should be able to continue to be able to edit a file
(it was .fixtures.yaml, it will be a Puppetfile now) to run tests
against custom dependencies (modules + version can be whatever you like).

It has been addressed in the last patchset:
https://review.openstack.org/#/c/226830/21..22/Rakefile,cm

So in your care, you'll have to:

1/ Build your Puppetfile that contains your custom deps (instead of
editing the .fixtures.yaml)
2/ Export PUPPETFILE=/path/Puppetfile
3/ Run `rake rspec` like before.

This solution should make happy everyone:

* Default usage (on my laptop) will test the same things as Puppet
OpenStack CI
* Allow to use Depends-On in OpenStack CI
* Allow to customize the dependencies for downstream CI by creating a
Puppetfile and exporting its path.

Deal?
-- 
Emilien Macchi



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] use zuul-cloner when running rspec

[Emilien Macchi]

On 09/24/2015 10:14 AM, Alex Schultz wrote:
> On Wed, Sep 23, 2015 at 4:56 PM, Emilien Macchi  wrote:
>> Background
>> ==
>>
>> Current rspec tests are tested with modules mentioned in .fixtures.yaml
>> file of each module.
>>
>> * the file is not consistent across all modules
>> * it hardcodes module names & versions
>> * this way does not allow to use "Depend-On" feature, that would allow
>> to test cross-modules patches
>>
>> Proposal
>> 
>>
>> * Like we do in beaker & integration jobs, use zuul-cloner to clone
>> modules in our CI jobs.
>> * Use r10k to prepare fixtures modules.
>> * Use Puppetfile hosted by openstack/puppet-openstack-integration
>>
>> In that way:
>> * we will have modules name + versions testing consistency across all
>> modules
>> * the same Puppetfile would be used by unit/beaker/integration testing.
>> * the patch that pass tests on your laptop would pass tests in upstream CI
>> * if you don't have zuul-cloner on your laptop, don't worry it will use
>> git clone. Though you won't have Depends-On feature working on your
>> laptop (technically not possible).
>> * Though your patch will support Depends-On in OpenStack Infra for unit
>> tests. If you submit a patch in puppet-openstacklib that drop something
>> wrong, you can send a patch in puppet-nova that will test it, and unit
>> tests will fail.
>>
>> Drawbacks
>> =
>> * cloning from .fixtures.yaml takes ~ 10 seconds
>> * using r10k + zuul-clone takes ~50 seconds (more modules to clone).
>>
>> I think 40 seconds is something accept regarding the benefit.
>>
> 
> As someone who consumes these modules downstream and has our own CI
> setup to run the rspec items, this ties it too closely to the
> openstack infrastructure. If we replace the .fixtures.yml with
> zuul-cloner, it assumes I always want the openstack version of the
> modules. This is not necessarily true. I like being able to replace
> items within fixtures.yml when doing dev work. For example If i want
> to test upgrading another module not related to openstack, like
> inifile, how does that work with the proposed solution?  This is also
> moving away from general puppet module conventions for testing. My
> preference would be that this be a different task and we have both
> .fixtures.yml (for general use/development) and the zuul method of
> cloning (for CI).  You have to also think about this from a consumer
> standpoint and this is adding an external dependency on the OpenStack
> infrastructure for anyone trying to run rspec or trying to consume the
> published versions from the forge.  Would I be able to run these tests
> in an offline mode with this change? With the .fixures.yml it's a
> minor edit to switch to local versions. Is the same true for the
> zuul-cloner version?

What you did before:
* Edit .fixtures.yaml and put the version you like.

What you would do this the current proposal:
* Edit openstack/puppet-openstack-integration/Puppetfile and put the
version you like.

What you're suggesting has a huge downside:
People will still use fixtures by default and not test what is actually
tested by our CI.
A few people will know about the specific Rake task so a few people will
test exactly what upstream does. That will cause frustration to the most
of people who will see tests failing in our CI and not on their laptop.
I'm not sure we want that.

I think more than most of people that run tests on their laptops want to
see them passing in upstream CI.
The few people that want to trick versions & modules, will have to run
Rake, trick the Puppetfile and run Rake again. It's not a big deal and
I'm sure this few people can deal with that.

>>
>> Next steps
>> ==
>>
>> * PoC in puppet-nova: https://review.openstack.org/#/c/226830/
>> * Patch openstack/puppet-modulesync-config to be consistent across all
>> our modules.
>>
>> Bonus
>> =
>> we might need (asap) a canary job for puppet-openstack-integration
>> repository, that would run tests on a puppet-* module (since we're using
>> install_modules.sh & Puppetfile files in puppet-* modules).
>> Nothing has been done yet for this work.
>>
>>
>> Thoughts?
>> --
>> Emilien Macchi
>>
>>
> 
> I think we need this functionality, I just don't think it's a
> replacement for the .fixures.yml.
> 
> Thanks,
> -Alex
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Emilien Macchi



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] use zuul-cloner when running rspec

[Emilien Macchi]

Background
==

Current rspec tests are tested with modules mentioned in .fixtures.yaml
file of each module.

* the file is not consistent across all modules
* it hardcodes module names & versions
* this way does not allow to use "Depend-On" feature, that would allow
to test cross-modules patches

Proposal


* Like we do in beaker & integration jobs, use zuul-cloner to clone
modules in our CI jobs.
* Use r10k to prepare fixtures modules.
* Use Puppetfile hosted by openstack/puppet-openstack-integration

In that way:
* we will have modules name + versions testing consistency across all
modules
* the same Puppetfile would be used by unit/beaker/integration testing.
* the patch that pass tests on your laptop would pass tests in upstream CI
* if you don't have zuul-cloner on your laptop, don't worry it will use
git clone. Though you won't have Depends-On feature working on your
laptop (technically not possible).
* Though your patch will support Depends-On in OpenStack Infra for unit
tests. If you submit a patch in puppet-openstacklib that drop something
wrong, you can send a patch in puppet-nova that will test it, and unit
tests will fail.

Drawbacks
=
* cloning from .fixtures.yaml takes ~ 10 seconds
* using r10k + zuul-clone takes ~50 seconds (more modules to clone).

I think 40 seconds is something accept regarding the benefit.


Next steps
==

* PoC in puppet-nova: https://review.openstack.org/#/c/226830/
* Patch openstack/puppet-modulesync-config to be consistent across all
our modules.

Bonus
=
we might need (asap) a canary job for puppet-openstack-integration
repository, that would run tests on a puppet-* module (since we're using
install_modules.sh & Puppetfile files in puppet-* modules).
Nothing has been done yet for this work.


Thoughts?
-- 
Emilien Macchi



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev