Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[Emilien Macchi]

On Wed, Sep 7, 2016 at 9:18 AM, Dmitry Tantsur  wrote:
> On 09/01/2016 05:48 PM, Emilien Macchi wrote:
>>
>> On Thu, Aug 25, 2016 at 9:16 AM, Steven Hardy  wrote:
>>>
>>> On Wed, Aug 17, 2016 at 07:20:59AM -0400, James Slagle wrote:

 On Wed, Aug 17, 2016 at 4:04 AM, Dmitry Tantsur 
 wrote:
>
> However, the current gate system allows to run jobs based on files
> affected.
> So we can also run a scenario covering ironic on THT check/gate if
> puppet/services/*ironic* is affected, but not in the other cases. This
> won't
> cover all potential failures, but it would be of great help anyway. It
> should also run in experimental pipeline, so that it can be triggered
> on any
> patch.
>
> This is in addition to periodic jobs you're proposing, not replacing
> them.
> WDYT?


 Using the files affected to trigger a scenario test that uses the
 affected composable service sounds like a really good idea to me.
>>>
>>>
>>> +1 I think this sounds like a really good idea.
>>>
>>> Now that we're doing almost all per-service configuration in the
>>> respective
>>> templates and puppet profiles, this should be much easier to implement I
>>> think so definitely +1 on giving it a go.
>>>
>>
>> So I would like to give an update about this.
>> The work has been done to have the structure in place.
>> I first used experimental pipeline to test the new jobs but they are
>> now in check pipeline, as non-voting.
>>
>> I created a multinode jobs CI matrix here:
>> https://github.com/openstack-infra/tripleo-ci#service-testing-matrix
>> I encourage everyone to have a quick look at it.
>>
>> What is happening now?
>>
>> - If you submit a patch in THT (in puppet/services/ceilometer*) it
>> will trigger scenario001 job (example with Telemetry). Same thing for
>> Puppet profiles in puppet-tripleo. Note: with Zuul v3 we'll be able to
>> make things more granular and specific to projects and files. We're
>> looking for having it!
>> - Since multinode-nonha initially created by slagle is now a bit
>> redundant with scenarios, I'll make it lighter to test basic compute
>> services with the less services as possible.
>> - I'll continue to extend the scenarios complexity and start testing
>> different backends (ie, ceph/rbd on scenario001 with Telemetry, etc),
>> like we're doing in Puppet CI:
>> https://github.com/openstack/puppet-openstack-integration#description
>> - For now, we're using pingtest to test that services actually work. I
>> guess it's good for now, but we also might want to consider Tempest
>> sometimes. I know Tempest already runs on periodic jobs, but should we
>> also consider it in those multinode jobs? The feedback would be
>> valuable though but we would have to make tempest configuration
>> composable, depending on the services we actually run (maybe using
>> discovery, etc).
>>
>> Any help and feedback on this topic is highly welcome,
>>
>
> It's pretty cool, unfortunately putting nova-compute on controllers is
> incompatible with the approach we currently take for ironic... so we either
> need a separate scenario or another approach here.

Yes, if you read the end of my blog post, you'll see that we need to
investigate adding one more node for the scenario where Ironic would
run... That's something we will investigate and probably implement
soon.

>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[Dmitry Tantsur]

On 09/01/2016 05:48 PM, Emilien Macchi wrote:

On Thu, Aug 25, 2016 at 9:16 AM, Steven Hardy  wrote:

On Wed, Aug 17, 2016 at 07:20:59AM -0400, James Slagle wrote:

On Wed, Aug 17, 2016 at 4:04 AM, Dmitry Tantsur  wrote:

However, the current gate system allows to run jobs based on files affected.
So we can also run a scenario covering ironic on THT check/gate if
puppet/services/*ironic* is affected, but not in the other cases. This won't
cover all potential failures, but it would be of great help anyway. It
should also run in experimental pipeline, so that it can be triggered on any
patch.

This is in addition to periodic jobs you're proposing, not replacing them.
WDYT?


Using the files affected to trigger a scenario test that uses the
affected composable service sounds like a really good idea to me.


+1 I think this sounds like a really good idea.

Now that we're doing almost all per-service configuration in the respective
templates and puppet profiles, this should be much easier to implement I
think so definitely +1 on giving it a go.



So I would like to give an update about this.
The work has been done to have the structure in place.
I first used experimental pipeline to test the new jobs but they are
now in check pipeline, as non-voting.

I created a multinode jobs CI matrix here:
https://github.com/openstack-infra/tripleo-ci#service-testing-matrix
I encourage everyone to have a quick look at it.

What is happening now?

- If you submit a patch in THT (in puppet/services/ceilometer*) it
will trigger scenario001 job (example with Telemetry). Same thing for
Puppet profiles in puppet-tripleo. Note: with Zuul v3 we'll be able to
make things more granular and specific to projects and files. We're
looking for having it!
- Since multinode-nonha initially created by slagle is now a bit
redundant with scenarios, I'll make it lighter to test basic compute
services with the less services as possible.
- I'll continue to extend the scenarios complexity and start testing
different backends (ie, ceph/rbd on scenario001 with Telemetry, etc),
like we're doing in Puppet CI:
https://github.com/openstack/puppet-openstack-integration#description
- For now, we're using pingtest to test that services actually work. I
guess it's good for now, but we also might want to consider Tempest
sometimes. I know Tempest already runs on periodic jobs, but should we
also consider it in those multinode jobs? The feedback would be
valuable though but we would have to make tempest configuration
composable, depending on the services we actually run (maybe using
discovery, etc).

Any help and feedback on this topic is highly welcome,



It's pretty cool, unfortunately putting nova-compute on controllers is 
incompatible with the approach we currently take for ironic... so we 
either need a separate scenario or another approach here.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[Emilien Macchi]

On Thu, Sep 1, 2016 at 11:48 AM, Emilien Macchi  wrote:
> On Thu, Aug 25, 2016 at 9:16 AM, Steven Hardy  wrote:
>> On Wed, Aug 17, 2016 at 07:20:59AM -0400, James Slagle wrote:
>>> On Wed, Aug 17, 2016 at 4:04 AM, Dmitry Tantsur  wrote:
>>> > However, the current gate system allows to run jobs based on files 
>>> > affected.
>>> > So we can also run a scenario covering ironic on THT check/gate if
>>> > puppet/services/*ironic* is affected, but not in the other cases. This 
>>> > won't
>>> > cover all potential failures, but it would be of great help anyway. It
>>> > should also run in experimental pipeline, so that it can be triggered on 
>>> > any
>>> > patch.
>>> >
>>> > This is in addition to periodic jobs you're proposing, not replacing them.
>>> > WDYT?
>>>
>>> Using the files affected to trigger a scenario test that uses the
>>> affected composable service sounds like a really good idea to me.
>>
>> +1 I think this sounds like a really good idea.
>>
>> Now that we're doing almost all per-service configuration in the respective
>> templates and puppet profiles, this should be much easier to implement I
>> think so definitely +1 on giving it a go.
>>
>
> So I would like to give an update about this.
> The work has been done to have the structure in place.
> I first used experimental pipeline to test the new jobs but they are
> now in check pipeline, as non-voting.
>
> I created a multinode jobs CI matrix here:
> https://github.com/openstack-infra/tripleo-ci#service-testing-matrix
> I encourage everyone to have a quick look at it.
>
> What is happening now?
>
> - If you submit a patch in THT (in puppet/services/ceilometer*) it
> will trigger scenario001 job (example with Telemetry). Same thing for
> Puppet profiles in puppet-tripleo. Note: with Zuul v3 we'll be able to
> make things more granular and specific to projects and files. We're
> looking for having it!
> - Since multinode-nonha initially created by slagle is now a bit
> redundant with scenarios, I'll make it lighter to test basic compute
> services with the less services as possible.
> - I'll continue to extend the scenarios complexity and start testing
> different backends (ie, ceph/rbd on scenario001 with Telemetry, etc),
> like we're doing in Puppet CI:
> https://github.com/openstack/puppet-openstack-integration#description
> - For now, we're using pingtest to test that services actually work. I
> guess it's good for now, but we also might want to consider Tempest
> sometimes. I know Tempest already runs on periodic jobs, but should we
> also consider it in those multinode jobs? The feedback would be
> valuable though but we would have to make tempest configuration
> composable, depending on the services we actually run (maybe using
> discovery, etc).
>
> Any help and feedback on this topic is highly welcome,
> --
> Emilien Macchi

I wrote a blog post so people can understand how it works:
http://my1.fr/blog/scaling-up-tripleo-ci-coverage-with-scenarios/

Let me know if I need to go more in details, any feedback is welcome as usual.
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[Emilien Macchi]

On Thu, Aug 25, 2016 at 9:16 AM, Steven Hardy  wrote:
> On Wed, Aug 17, 2016 at 07:20:59AM -0400, James Slagle wrote:
>> On Wed, Aug 17, 2016 at 4:04 AM, Dmitry Tantsur  wrote:
>> > However, the current gate system allows to run jobs based on files 
>> > affected.
>> > So we can also run a scenario covering ironic on THT check/gate if
>> > puppet/services/*ironic* is affected, but not in the other cases. This 
>> > won't
>> > cover all potential failures, but it would be of great help anyway. It
>> > should also run in experimental pipeline, so that it can be triggered on 
>> > any
>> > patch.
>> >
>> > This is in addition to periodic jobs you're proposing, not replacing them.
>> > WDYT?
>>
>> Using the files affected to trigger a scenario test that uses the
>> affected composable service sounds like a really good idea to me.
>
> +1 I think this sounds like a really good idea.
>
> Now that we're doing almost all per-service configuration in the respective
> templates and puppet profiles, this should be much easier to implement I
> think so definitely +1 on giving it a go.
>

So I would like to give an update about this.
The work has been done to have the structure in place.
I first used experimental pipeline to test the new jobs but they are
now in check pipeline, as non-voting.

I created a multinode jobs CI matrix here:
https://github.com/openstack-infra/tripleo-ci#service-testing-matrix
I encourage everyone to have a quick look at it.

What is happening now?

- If you submit a patch in THT (in puppet/services/ceilometer*) it
will trigger scenario001 job (example with Telemetry). Same thing for
Puppet profiles in puppet-tripleo. Note: with Zuul v3 we'll be able to
make things more granular and specific to projects and files. We're
looking for having it!
- Since multinode-nonha initially created by slagle is now a bit
redundant with scenarios, I'll make it lighter to test basic compute
services with the less services as possible.
- I'll continue to extend the scenarios complexity and start testing
different backends (ie, ceph/rbd on scenario001 with Telemetry, etc),
like we're doing in Puppet CI:
https://github.com/openstack/puppet-openstack-integration#description
- For now, we're using pingtest to test that services actually work. I
guess it's good for now, but we also might want to consider Tempest
sometimes. I know Tempest already runs on periodic jobs, but should we
also consider it in those multinode jobs? The feedback would be
valuable though but we would have to make tempest configuration
composable, depending on the services we actually run (maybe using
discovery, etc).

Any help and feedback on this topic is highly welcome,
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[Steven Hardy]

On Wed, Aug 17, 2016 at 07:20:59AM -0400, James Slagle wrote:
> On Wed, Aug 17, 2016 at 4:04 AM, Dmitry Tantsur  wrote:
> > However, the current gate system allows to run jobs based on files affected.
> > So we can also run a scenario covering ironic on THT check/gate if
> > puppet/services/*ironic* is affected, but not in the other cases. This won't
> > cover all potential failures, but it would be of great help anyway. It
> > should also run in experimental pipeline, so that it can be triggered on any
> > patch.
> >
> > This is in addition to periodic jobs you're proposing, not replacing them.
> > WDYT?
> 
> Using the files affected to trigger a scenario test that uses the
> affected composable service sounds like a really good idea to me.

+1 I think this sounds like a really good idea.

Now that we're doing almost all per-service configuration in the respective
templates and puppet profiles, this should be much easier to implement I
think so definitely +1 on giving it a go.

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[Dmitry Tantsur]

Hi!

Looks great! Ironic currently requires a few manual steps, I wonder how 
we do them, but I guess we can figure out later.


On 08/24/2016 08:39 PM, Emilien Macchi wrote:

Ok I have  PoC ready for review and feedback:

- First iteration of scenario001 job in TripleO CI:
https://review.openstack.org/#/c/360039
I checked, and the job is not triggered if we don't touch Sahara files directly.

- Patch in THT that tries to modify Sahara files:
https://review.openstack.org/#/c/360040
I checked, and when running "check experimental", the job is triggered
because we modify puppet/services/sahara-base.yaml.

So the mechanism is in place (experimental status now) but ready for review.
Please give any feedback.

Once we have this mechanism in place, we'll be able to add more
services coverage, and run the jobs in a smart way thanks to Zuul.

Thanks,

On Wed, Aug 17, 2016 at 3:52 PM, Emilien Macchi  wrote:

On Wed, Aug 17, 2016 at 7:20 AM, James Slagle  wrote:

On Wed, Aug 17, 2016 at 4:04 AM, Dmitry Tantsur  wrote:

However, the current gate system allows to run jobs based on files affected.
So we can also run a scenario covering ironic on THT check/gate if
puppet/services/*ironic* is affected, but not in the other cases. This won't
cover all potential failures, but it would be of great help anyway. It
should also run in experimental pipeline, so that it can be triggered on any
patch.

This is in addition to periodic jobs you're proposing, not replacing them.
WDYT?


Using the files affected to trigger a scenario test that uses the
affected composable service sounds like a really good idea to me.



I have a PoC, everything is explained in commit message:
https://review.openstack.org/#/c/356675/

Please review it and give feedback !



--
-- James Slagle
--

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Emilien Macchi







__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[Emilien Macchi]

Ok I have  PoC ready for review and feedback:

- First iteration of scenario001 job in TripleO CI:
https://review.openstack.org/#/c/360039
I checked, and the job is not triggered if we don't touch Sahara files directly.

- Patch in THT that tries to modify Sahara files:
https://review.openstack.org/#/c/360040
I checked, and when running "check experimental", the job is triggered
because we modify puppet/services/sahara-base.yaml.

So the mechanism is in place (experimental status now) but ready for review.
Please give any feedback.

Once we have this mechanism in place, we'll be able to add more
services coverage, and run the jobs in a smart way thanks to Zuul.

Thanks,

On Wed, Aug 17, 2016 at 3:52 PM, Emilien Macchi  wrote:
> On Wed, Aug 17, 2016 at 7:20 AM, James Slagle  wrote:
>> On Wed, Aug 17, 2016 at 4:04 AM, Dmitry Tantsur  wrote:
>>> However, the current gate system allows to run jobs based on files affected.
>>> So we can also run a scenario covering ironic on THT check/gate if
>>> puppet/services/*ironic* is affected, but not in the other cases. This won't
>>> cover all potential failures, but it would be of great help anyway. It
>>> should also run in experimental pipeline, so that it can be triggered on any
>>> patch.
>>>
>>> This is in addition to periodic jobs you're proposing, not replacing them.
>>> WDYT?
>>
>> Using the files affected to trigger a scenario test that uses the
>> affected composable service sounds like a really good idea to me.
>>
>
> I have a PoC, everything is explained in commit message:
> https://review.openstack.org/#/c/356675/
>
> Please review it and give feedback !
>
>>
>> --
>> -- James Slagle
>> --
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Emilien Macchi



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[Emilien Macchi]

On Wed, Aug 17, 2016 at 7:20 AM, James Slagle  wrote:
> On Wed, Aug 17, 2016 at 4:04 AM, Dmitry Tantsur  wrote:
>> However, the current gate system allows to run jobs based on files affected.
>> So we can also run a scenario covering ironic on THT check/gate if
>> puppet/services/*ironic* is affected, but not in the other cases. This won't
>> cover all potential failures, but it would be of great help anyway. It
>> should also run in experimental pipeline, so that it can be triggered on any
>> patch.
>>
>> This is in addition to periodic jobs you're proposing, not replacing them.
>> WDYT?
>
> Using the files affected to trigger a scenario test that uses the
> affected composable service sounds like a really good idea to me.
>

I have a PoC, everything is explained in commit message:
https://review.openstack.org/#/c/356675/

Please review it and give feedback !

>
> --
> -- James Slagle
> --
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[James Slagle]

On Wed, Aug 17, 2016 at 4:04 AM, Dmitry Tantsur  wrote:
> However, the current gate system allows to run jobs based on files affected.
> So we can also run a scenario covering ironic on THT check/gate if
> puppet/services/*ironic* is affected, but not in the other cases. This won't
> cover all potential failures, but it would be of great help anyway. It
> should also run in experimental pipeline, so that it can be triggered on any
> patch.
>
> This is in addition to periodic jobs you're proposing, not replacing them.
> WDYT?

Using the files affected to trigger a scenario test that uses the
affected composable service sounds like a really good idea to me.



-- 
-- James Slagle
--

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[Dmitry Tantsur]

On 08/17/2016 03:21 AM, Emilien Macchi wrote:

On Tue, Aug 16, 2016 at 4:49 PM, James Slagle  wrote:

On Mon, Aug 15, 2016 at 4:54 AM, Dmitry Tantsur  wrote:

Hi everyone, happy Monday :)

I'd like to start the discussion about CI-testing the optional composable
services in the CI (I'm primarily interested in Ironic, but I know there are
a lot more).





The reason being is that there are already many different services
that can break TripleO, and I'd rather focus on improving the testing
of the actual deployment framework itself, instead of testing the
"whole world" on every patch. We only have so much capacity. For
example, I'd rather see us testing updates or upgrades on each patch
instead of all the services.


I don't suggest we test Ironic, I suggest we test the composable 
services installing Ironic, and then just do a sanity-check. Without 
such check an actual failure can go unnoticed, I've already faced with that.




That being said, if you wanted to add a job that covered Ironic, I'd
at least start with adding a job in the experimental queue. If it
proves to be stable, we can always consider moving it to the check
queue.



TripleO CI is having the same problem as Puppet CI had some time ago,
when we wanted to test more and more.

We solved this problem with scenarios.
Everything is documented here:
https://github.com/openstack/puppet-openstack-integration#description

We're testing all scenarios for all commits in
puppet-openstack-integration (equivalent to tripleo-ci) but only some
scenarios for each Puppet module.
Ex: OpenStack Sahara is deployed in scenario003, so a patch in
puppet-sahara will only run scenario003 job. We do that in Zuul
layout.

Because it would be complicated to do the same with TripleO Heat
Templates, we could run the same kind of job in periodic pipeline.
Though for Puppet jobs, we could run the tripleo scenarios in the
check/gate, as we do with the current multinode job.

So here's a summary of my proposal (and I volunteer to actually work
on it, like I did for Puppet CI):
* Call current jobs "compute-kit" which are current nonha/ha (ovb and
multinode) jobs deploying the basic services (undercloud +
Nova/Neutron/Glance/Keystone/Heat/Cinder/Swift and common services,
apache, mysql, etc).
* Create TripleO envs deploying different scenarios (we could start by
scenario001 deploying "compute-kit" + ceph (rbd backend for Nova /
Glance / Cinder). It's an example, feel free to propose something
else. The envs would live in tripleo-ci repo among others ci envs.
* Switch puppet-ceph zuul layout to stop running ovb ha job and run
the scenario001 job in check/gate (with the experimental pipeline
transition, as usual).
* Run scenario001 job in check/gate of tripleo-ci among other jobs.
* Run scenario001 job in periodic pipeline for puppet-tripleo and
tripleo-heat-templates.

Any feedback is welcome, but I think this would be a good start of
scaling our CI jobs coverage.



I'm not particularly fond of using *only* periodic jobs. I don't think 
it alone solves the problems I've raised, because:
1. Periodic jobs do not give an immediate feedback of whether we break 
something in a THT patch.

2. Periodic jobs do not help tracking which exactly patch was breaking.
3. Periodic jobs results are slightly hidden, so a failure in a 
non-priority service (like Ironic) will probably stay unnoticed for 
quite a while.


However, the current gate system allows to run jobs based on files 
affected. So we can also run a scenario covering ironic on THT 
check/gate if puppet/services/*ironic* is affected, but not in the other 
cases. This won't cover all potential failures, but it would be of great 
help anyway. It should also run in experimental pipeline, so that it can 
be triggered on any patch.


This is in addition to periodic jobs you're proposing, not replacing 
them. WDYT?


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[Emilien Macchi]

On Tue, Aug 16, 2016 at 4:49 PM, James Slagle  wrote:
> On Mon, Aug 15, 2016 at 4:54 AM, Dmitry Tantsur  wrote:
>> Hi everyone, happy Monday :)
>>
>> I'd like to start the discussion about CI-testing the optional composable
>> services in the CI (I'm primarily interested in Ironic, but I know there are
>> a lot more).
>>
>> Currently every time we change something in an optional service, we have to
>> create a DO-NOT-MERGE patch making the service in question not optional.
>> This approach has several problems:
>>
>> 1. It's not usually done for global refactorings.
>>
>> 2. The current CI does not have any specific tests to check that the
>> services in question actually works at all (e.g. in my experience the CI was
>> green even though nova-compute could not reach ironic).
>>
>> 3. If something breaks, it's hard to track the problem down to a specific
>> patch, as there is no history of gate runs.
>>
>> 4. It does not test the environment files we provide for enabling the
>> service.
>>
>> So, are there any plans to start covering optional services? Maybe at least
>> a non-HA job with all environment files included? It would be cool to also
>> somehow provide additional checks though. Or, in case of ironic, to disable
>> the regular nova compute, so that the ping test runs on an ironic instance.
>
> There are no immediate plans. Although I think the CI testing matrix
> is always open for discussion.
>
> I'm a little skeptical we will be able to deploy all services within
> the job timeout. And if we are, such a job seems better suited as a
> periodic job than in the check queue.
>
> The reason being is that there are already many different services
> that can break TripleO, and I'd rather focus on improving the testing
> of the actual deployment framework itself, instead of testing the
> "whole world" on every patch. We only have so much capacity. For
> example, I'd rather see us testing updates or upgrades on each patch
> instead of all the services.
>
> That being said, if you wanted to add a job that covered Ironic, I'd
> at least start with adding a job in the experimental queue. If it
> proves to be stable, we can always consider moving it to the check
> queue.
>

TripleO CI is having the same problem as Puppet CI had some time ago,
when we wanted to test more and more.

We solved this problem with scenarios.
Everything is documented here:
https://github.com/openstack/puppet-openstack-integration#description

We're testing all scenarios for all commits in
puppet-openstack-integration (equivalent to tripleo-ci) but only some
scenarios for each Puppet module.
Ex: OpenStack Sahara is deployed in scenario003, so a patch in
puppet-sahara will only run scenario003 job. We do that in Zuul
layout.

Because it would be complicated to do the same with TripleO Heat
Templates, we could run the same kind of job in periodic pipeline.
Though for Puppet jobs, we could run the tripleo scenarios in the
check/gate, as we do with the current multinode job.

So here's a summary of my proposal (and I volunteer to actually work
on it, like I did for Puppet CI):
* Call current jobs "compute-kit" which are current nonha/ha (ovb and
multinode) jobs deploying the basic services (undercloud +
Nova/Neutron/Glance/Keystone/Heat/Cinder/Swift and common services,
apache, mysql, etc).
* Create TripleO envs deploying different scenarios (we could start by
scenario001 deploying "compute-kit" + ceph (rbd backend for Nova /
Glance / Cinder). It's an example, feel free to propose something
else. The envs would live in tripleo-ci repo among others ci envs.
* Switch puppet-ceph zuul layout to stop running ovb ha job and run
the scenario001 job in check/gate (with the experimental pipeline
transition, as usual).
* Run scenario001 job in check/gate of tripleo-ci among other jobs.
* Run scenario001 job in periodic pipeline for puppet-tripleo and
tripleo-heat-templates.

Any feedback is welcome, but I think this would be a good start of
scaling our CI jobs coverage.
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[James Slagle]

On Mon, Aug 15, 2016 at 4:54 AM, Dmitry Tantsur  wrote:
> Hi everyone, happy Monday :)
>
> I'd like to start the discussion about CI-testing the optional composable
> services in the CI (I'm primarily interested in Ironic, but I know there are
> a lot more).
>
> Currently every time we change something in an optional service, we have to
> create a DO-NOT-MERGE patch making the service in question not optional.
> This approach has several problems:
>
> 1. It's not usually done for global refactorings.
>
> 2. The current CI does not have any specific tests to check that the
> services in question actually works at all (e.g. in my experience the CI was
> green even though nova-compute could not reach ironic).
>
> 3. If something breaks, it's hard to track the problem down to a specific
> patch, as there is no history of gate runs.
>
> 4. It does not test the environment files we provide for enabling the
> service.
>
> So, are there any plans to start covering optional services? Maybe at least
> a non-HA job with all environment files included? It would be cool to also
> somehow provide additional checks though. Or, in case of ironic, to disable
> the regular nova compute, so that the ping test runs on an ironic instance.

There are no immediate plans. Although I think the CI testing matrix
is always open for discussion.

I'm a little skeptical we will be able to deploy all services within
the job timeout. And if we are, such a job seems better suited as a
periodic job than in the check queue.

The reason being is that there are already many different services
that can break TripleO, and I'd rather focus on improving the testing
of the actual deployment framework itself, instead of testing the
"whole world" on every patch. We only have so much capacity. For
example, I'd rather see us testing updates or upgrades on each patch
instead of all the services.

That being said, if you wanted to add a job that covered Ironic, I'd
at least start with adding a job in the experimental queue. If it
proves to be stable, we can always consider moving it to the check
queue.

-- 
-- James Slagle
--

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Testing optional composable services in the CI

[Giulio Fidente]

On 08/15/2016 10:54 AM, Dmitry Tantsur wrote:

Hi everyone, happy Monday :)

I'd like to start the discussion about CI-testing the optional
composable services in the CI (I'm primarily interested in Ironic, but I
know there are a lot more).


thanks for bringing this up, with "pluggability" comes responsibility it 
seems


there is also a conflicting (yet valid) interest in keeping the number 
of services deployed in the overcloud to a minimum to avoid even longer 
CI run times



So, are there any plans to start covering optional services? Maybe at
least a non-HA job with all environment files included? It would be cool
to also somehow provide additional checks though. Or, in case of ironic,
to disable the regular nova compute, so that the ping test runs on an
ironic instance.


it isn't really a case of HA vs non-HA, with the newer HA architecture 
we're only managing via pcmk those openstack services which need to be 
(including recent additions like manila-share or cinder-backup) and 
these should be tested in the HA scenario which IMHO at this point could 
become a default


it looks to me that a scenario in the experimental queue deploying a 
"full" overcloud could work?


there is a similar requirement for testing 'competing' services, like 
swift and ceph/rgw which we're about to merge ... but applies to other 
things, like the neutron plugins

--
Giulio Fidente
GPG KEY: 08D733BA | IRC: gfidente

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] Testing optional composable services in the CI

[Dmitry Tantsur]

Hi everyone, happy Monday :)

I'd like to start the discussion about CI-testing the optional 
composable services in the CI (I'm primarily interested in Ironic, but I 
know there are a lot more).


Currently every time we change something in an optional service, we have 
to create a DO-NOT-MERGE patch making the service in question not 
optional. This approach has several problems:


1. It's not usually done for global refactorings.

2. The current CI does not have any specific tests to check that the 
services in question actually works at all (e.g. in my experience the CI 
was green even though nova-compute could not reach ironic).


3. If something breaks, it's hard to track the problem down to a 
specific patch, as there is no history of gate runs.


4. It does not test the environment files we provide for enabling the 
service.


So, are there any plans to start covering optional services? Maybe at 
least a non-HA job with all environment files included? It would be cool 
to also somehow provide additional checks though. Or, in case of ironic, 
to disable the regular nova compute, so that the ping test runs on an 
ironic instance.


WDYT?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev