Re: [openstack-dev] [tripleo] CI scenarios design - how to add more services
Giving few updates here: - we implemented option 1.a), which means that we moved the tripleo CI scenarios environments and pingtests into tripleo-heat-template. - we created tripleo-scenarioXXX-puppet jobs that run on some modules. Some Example: - puppet-gnocchi now runs tripleo-scenario001, that deploy TripleO with Telemetry services - if you submit a patch in puppet-tripleo that touch the gnocchi profiles, tripleo-scenario001 will also run - if you submit a patch in THT that touch the gnocchi composable service, tripleo-scenario001 will also run - if you add a new service in TripleO during Pike and test it in a scenario, TripleO CI scenarios for Ocata will continue to work as we now use THT to store the CI environments and we don't backport features. Future: - investigate if we could run the scenarios outside tripleo CI (example: run tripleo-scenario001 in Gnocchi upstream CI, beside other devstack jobs) - keep increasing coverage of use-cases: more services, ssl, ipv6, more plugins, etc. - investigate how we could run multinode scenarios by using tripleo-quickstart. Any feedback and help on this topic is welcome as usual. Don't hesitate to contribute, add your own service, or propose scenario improvements, it's highly welcome! Thanks, On Mon, Nov 28, 2016 at 3:35 PM, John Trowbridge wrote: > > > On 11/22/2016 09:02 PM, Emilien Macchi wrote: >> == Context >> >> In Newton we added new multinode jobs called "scenarios". >> The challenged we tried to solve was "how to test the maximum of >> services without overloading the nodes that run tests". >> >> Each scenarios deploys a set of services, which allows us to >> horizontally scale the number of scenarios to increase the service >> testing coverage. >> See the result here: >> https://github.com/openstack-infra/tripleo-ci#service-testing-matrix >> >> To implement this model, we took example of Puppet OpenStack CI: >> https://github.com/openstack/puppet-openstack-integration#description >> We even tried to keep consistent the services/scenarios relations, so >> it's consistent and easier to maintain. >> >> Everything was fine until we had to add new services during Ocata cycles. >> Because tripleo-ci repository is not branched, adding Barbican service >> in the TripleO environment for scenario002 would break Newton CI jobs. >> During my vacations, the team created a new scenario, scenario004, >> that deploys Barbican and that is only run for Ocata jobs. >> I don't think we should proceed this way, and let me explain why. >> >> == Problem >> >> How to scale the number of services that we test without increasing >> the number of scenarios and therefore the complexity of maintaining >> them on long-term. >> >> >> == Solutions >> >> The list is not exhaustive, feel free to add more. >> >> 1) Re-use experience from Puppet OpenStack CI and have environments >> that are in a branched repository. >> environments. >> In Puppet OpenStack CI, the repository that deploys environments >> (puppet-openstack-integration) is branched. So if puppet-barbican is >> ready to be tested in Ocata, we'll patch >> puppet-openstack-integration/master to start testing it and it won't >> break stable jobs. >> Like this, we were successfully able to maintain a fair number of >> scenarios and keep our coverage increasing over each cycle. >> >> I see 2 sub-options here: >> >> a) Move CI environments and pingtest into >> tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo >> is branched and we could add a README to explain these files are used >> in CI and we don't guarantee they would work outside TripleO CI tools. > > I also like this solution the best. It has the added benefit of being > able to review the CI for a new service in the same patch (or patch > chain) that adds the new service. We already have the low-memory > environment in tht, which while not CI specific, is definitely a CI > requirement. > >> b) Branch tripleo-ci repository. Personally I don't like this solution >> because a lot of patches in this repo are not related to OpenStack >> versions, which means we would need to backport most of the things >> from master. >> >> 2) Introduce branch-based scenario tests - >> https://review.openstack.org/#/c/396008/ >> It duplicates a lot of code and it's imho not really effective, though >> this solution would correctly works. >> >> 3) Introduce a new scenario each time we have new services (like we >> did with scenario004). >> By adding new scenarios at each release because we test new services >> is imho the wrong choice because: >> a) it adds complexity in our we're going to maintain these scenarios. >> b) it consumes more CI resources that we would need when some patches >> have to run all scenarios jobs. >> >> >> So I gave my opinion on the solutions, discussion is now open and my >> hope is that we find a consensus soon, so we can make progress in our >> testing coverage. >> Thanks, >> > >
Re: [openstack-dev] [tripleo] CI scenarios design - how to add more services
On 11/22/2016 09:02 PM, Emilien Macchi wrote: > == Context > > In Newton we added new multinode jobs called "scenarios". > The challenged we tried to solve was "how to test the maximum of > services without overloading the nodes that run tests". > > Each scenarios deploys a set of services, which allows us to > horizontally scale the number of scenarios to increase the service > testing coverage. > See the result here: > https://github.com/openstack-infra/tripleo-ci#service-testing-matrix > > To implement this model, we took example of Puppet OpenStack CI: > https://github.com/openstack/puppet-openstack-integration#description > We even tried to keep consistent the services/scenarios relations, so > it's consistent and easier to maintain. > > Everything was fine until we had to add new services during Ocata cycles. > Because tripleo-ci repository is not branched, adding Barbican service > in the TripleO environment for scenario002 would break Newton CI jobs. > During my vacations, the team created a new scenario, scenario004, > that deploys Barbican and that is only run for Ocata jobs. > I don't think we should proceed this way, and let me explain why. > > == Problem > > How to scale the number of services that we test without increasing > the number of scenarios and therefore the complexity of maintaining > them on long-term. > > > == Solutions > > The list is not exhaustive, feel free to add more. > > 1) Re-use experience from Puppet OpenStack CI and have environments > that are in a branched repository. > environments. > In Puppet OpenStack CI, the repository that deploys environments > (puppet-openstack-integration) is branched. So if puppet-barbican is > ready to be tested in Ocata, we'll patch > puppet-openstack-integration/master to start testing it and it won't > break stable jobs. > Like this, we were successfully able to maintain a fair number of > scenarios and keep our coverage increasing over each cycle. > > I see 2 sub-options here: > > a) Move CI environments and pingtest into > tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo > is branched and we could add a README to explain these files are used > in CI and we don't guarantee they would work outside TripleO CI tools. I also like this solution the best. It has the added benefit of being able to review the CI for a new service in the same patch (or patch chain) that adds the new service. We already have the low-memory environment in tht, which while not CI specific, is definitely a CI requirement. > b) Branch tripleo-ci repository. Personally I don't like this solution > because a lot of patches in this repo are not related to OpenStack > versions, which means we would need to backport most of the things > from master. > > 2) Introduce branch-based scenario tests - > https://review.openstack.org/#/c/396008/ > It duplicates a lot of code and it's imho not really effective, though > this solution would correctly works. > > 3) Introduce a new scenario each time we have new services (like we > did with scenario004). > By adding new scenarios at each release because we test new services > is imho the wrong choice because: > a) it adds complexity in our we're going to maintain these scenarios. > b) it consumes more CI resources that we would need when some patches > have to run all scenarios jobs. > > > So I gave my opinion on the solutions, discussion is now open and my > hope is that we find a consensus soon, so we can make progress in our > testing coverage. > Thanks, > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] CI scenarios design - how to add more services
On Fri, Nov 25, 2016 at 7:22 AM, Gabriele Cerami wrote: > On 22 Nov, Emilien Macchi wrote: >> 1) Re-use experience from Puppet OpenStack CI and have environments >> that are in a branched repository. >> a) Move CI environments and pingtest into >> tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo >> is branched and we could add a README to explain these files are used >> in CI and we don't guarantee they would work outside TripleO CI tools. > > I would consider it a big step for tht repo, to include test specific > code in it. *Upstream CI specific code* even > >> b) Branch tripleo-ci repository. Personally I don't like this solution >> because a lot of patches in this repo are not related to OpenStack >> versions, which means we would need to backport most of the things >> from master. > > I agree, we should be able to branch only the options and configuration, > not the entire process. Process is constantly changing and it would be > a pain to backport all this changes. > >> 2) Introduce branch-based scenario tests - >> https://review.openstack.org/#/c/396008/ >> It duplicates a lot of code and it's imho not really effective, though >> this solution would correctly works. > > I don't understand why you consider this code duplication and don't > consider branching code duplication too. > With branches you're duplicating even if you don't see all the > duplicates at once. Consider that with this method it could be easier to > "backport" a scenario modification, since you can do everything with a > single commit. > I would prefer a different format tough > templates/$BRANCH/scenario-001.yaml > test-environments/$BRANCH/scenario-001.yaml With branches, we might see the "duplication" as a simple backport. Remember, we don't backport features, so once we cut a release, we should not backport any change in our scenarios environments, since we don't had features (again). Which means, once we implement the CI templates in THT, we shouldn't backport anything but bugfixes in stable branches. Scenarios will evolve as we add features in master (ex: barbican service, etc). That said, 2) would also work. It's just a lot of files and lines of code in tripleo-ci, that could live in a branched repo (THT), which is why I took the simplest solution with 1)a). >> 3) Introduce a new scenario each time we have new services (like we >> did with scenario004). > > This doesn't scale well, we would have to be able to select which > scenario to run anyway > > > I have one last consideration to make. > With the transition to quickstart, we're effectively moving the process > part of tripleo-ci to another repo. > After the transition we could consider tripleo-ci more options and > configuration that process, and the we could start branching it, without > all the drawbacks we considered here. > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Emilien Macchi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] CI scenarios design - how to add more services
On 22 Nov, Emilien Macchi wrote: > 1) Re-use experience from Puppet OpenStack CI and have environments > that are in a branched repository. > a) Move CI environments and pingtest into > tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo > is branched and we could add a README to explain these files are used > in CI and we don't guarantee they would work outside TripleO CI tools. I would consider it a big step for tht repo, to include test specific code in it. *Upstream CI specific code* even > b) Branch tripleo-ci repository. Personally I don't like this solution > because a lot of patches in this repo are not related to OpenStack > versions, which means we would need to backport most of the things > from master. I agree, we should be able to branch only the options and configuration, not the entire process. Process is constantly changing and it would be a pain to backport all this changes. > 2) Introduce branch-based scenario tests - > https://review.openstack.org/#/c/396008/ > It duplicates a lot of code and it's imho not really effective, though > this solution would correctly works. I don't understand why you consider this code duplication and don't consider branching code duplication too. With branches you're duplicating even if you don't see all the duplicates at once. Consider that with this method it could be easier to "backport" a scenario modification, since you can do everything with a single commit. I would prefer a different format tough templates/$BRANCH/scenario-001.yaml test-environments/$BRANCH/scenario-001.yaml > 3) Introduce a new scenario each time we have new services (like we > did with scenario004). This doesn't scale well, we would have to be able to select which scenario to run anyway I have one last consideration to make. With the transition to quickstart, we're effectively moving the process part of tripleo-ci to another repo. After the transition we could consider tripleo-ci more options and configuration that process, and the we could start branching it, without all the drawbacks we considered here. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] CI scenarios design - how to add more services
On Thu, Nov 24, 2016 at 11:08 AM, Juan Antonio Osorio wrote: > I don't have a strong opinion about any option, as long as we have something > in place I'm happy. > > But regarding option 1.A: what would be done for newton once these templates > are moved to t-h-t. Would they be backported? What about mitaka? Newton: yes: we'll have to backport https://review.openstack.org/#/c/402119/ and that's it! Mitaka: we never supported and ran scenarios, so no problem here. I started a first iteration: https://review.openstack.org/#/q/topic:tripleo-ci/scenarios Feel free to review and give any feedback. Thanks, > > On 24 Nov 2016 17:55, "Carlos Camacho Gonzalez" wrote: >> >> I think would be cool to go with option, +1 to 1.A >> >> IMHO, >> - Easier to read. >> - Easier to maintain. >> - We don't make backports, instead we guarantee backwards compatibility. >> - We'll re-use experience from Puppet OpenStack CI. >> >> On Wed, Nov 23, 2016 at 10:13 PM, Giulio Fidente >> wrote: >> > hi Emilien, >> > >> > thanks for putting some thought into this. We have a similar problem to >> > test >> > RGW which was only added in Newton. >> > >> > >> > On 11/23/2016 03:02 AM, Emilien Macchi wrote: >> >> >> >> == Context >> >> >> >> In Newton we added new multinode jobs called "scenarios". >> >> The challenged we tried to solve was "how to test the maximum of >> >> services without overloading the nodes that run tests". >> >> >> >> Each scenarios deploys a set of services, which allows us to >> >> horizontally scale the number of scenarios to increase the service >> >> testing coverage. >> >> See the result here: >> >> https://github.com/openstack-infra/tripleo-ci#service-testing-matrix >> >> >> >> To implement this model, we took example of Puppet OpenStack CI: >> >> https://github.com/openstack/puppet-openstack-integration#description >> >> We even tried to keep consistent the services/scenarios relations, so >> >> it's consistent and easier to maintain. >> >> >> >> Everything was fine until we had to add new services during Ocata >> >> cycles. >> >> Because tripleo-ci repository is not branched, adding Barbican service >> >> in the TripleO environment for scenario002 would break Newton CI jobs. >> >> During my vacations, the team created a new scenario, scenario004, >> >> that deploys Barbican and that is only run for Ocata jobs. >> >> I don't think we should proceed this way, and let me explain why. >> >> >> >> == Problem >> >> >> >> How to scale the number of services that we test without increasing >> >> the number of scenarios and therefore the complexity of maintaining >> >> them on long-term. >> >> >> >> >> >> == Solutions >> >> >> >> The list is not exhaustive, feel free to add more. >> >> >> >> 1) Re-use experience from Puppet OpenStack CI and have environments >> >> that are in a branched repository. >> >> environments. >> >> In Puppet OpenStack CI, the repository that deploys environments >> >> (puppet-openstack-integration) is branched. So if puppet-barbican is >> >> ready to be tested in Ocata, we'll patch >> >> puppet-openstack-integration/master to start testing it and it won't >> >> break stable jobs. >> >> Like this, we were successfully able to maintain a fair number of >> >> scenarios and keep our coverage increasing over each cycle. >> >> >> >> I see 2 sub-options here: >> >> >> >> a) Move CI environments and pingtest into >> >> tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo >> >> is branched and we could add a README to explain these files are used >> >> in CI and we don't guarantee they would work outside TripleO CI tools. >> >> b) Branch tripleo-ci repository. Personally I don't like this solution >> >> because a lot of patches in this repo are not related to OpenStack >> >> versions, which means we would need to backport most of the things >> >> from master. >> > >> > >> > I'd +1 this idea. It sounds like we could make the scenarios generic >> > enough >> > to be usable also outside CI? Maybe they can serve as samples? >> > -- >> > Giulio Fidente >> > GPG KEY: 08D733BA >> > >> > >> > >> > __ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Emilien Macchi
Re: [openstack-dev] [tripleo] CI scenarios design - how to add more services
I don't have a strong opinion about any option, as long as we have something in place I'm happy. But regarding option 1.A: what would be done for newton once these templates are moved to t-h-t. Would they be backported? What about mitaka? On 24 Nov 2016 17:55, "Carlos Camacho Gonzalez" wrote: > I think would be cool to go with option, +1 to 1.A > > IMHO, > - Easier to read. > - Easier to maintain. > - We don't make backports, instead we guarantee backwards compatibility. > - We'll re-use experience from Puppet OpenStack CI. > > On Wed, Nov 23, 2016 at 10:13 PM, Giulio Fidente > wrote: > > hi Emilien, > > > > thanks for putting some thought into this. We have a similar problem to > test > > RGW which was only added in Newton. > > > > > > On 11/23/2016 03:02 AM, Emilien Macchi wrote: > >> > >> == Context > >> > >> In Newton we added new multinode jobs called "scenarios". > >> The challenged we tried to solve was "how to test the maximum of > >> services without overloading the nodes that run tests". > >> > >> Each scenarios deploys a set of services, which allows us to > >> horizontally scale the number of scenarios to increase the service > >> testing coverage. > >> See the result here: > >> https://github.com/openstack-infra/tripleo-ci#service-testing-matrix > >> > >> To implement this model, we took example of Puppet OpenStack CI: > >> https://github.com/openstack/puppet-openstack-integration#description > >> We even tried to keep consistent the services/scenarios relations, so > >> it's consistent and easier to maintain. > >> > >> Everything was fine until we had to add new services during Ocata > cycles. > >> Because tripleo-ci repository is not branched, adding Barbican service > >> in the TripleO environment for scenario002 would break Newton CI jobs. > >> During my vacations, the team created a new scenario, scenario004, > >> that deploys Barbican and that is only run for Ocata jobs. > >> I don't think we should proceed this way, and let me explain why. > >> > >> == Problem > >> > >> How to scale the number of services that we test without increasing > >> the number of scenarios and therefore the complexity of maintaining > >> them on long-term. > >> > >> > >> == Solutions > >> > >> The list is not exhaustive, feel free to add more. > >> > >> 1) Re-use experience from Puppet OpenStack CI and have environments > >> that are in a branched repository. > >> environments. > >> In Puppet OpenStack CI, the repository that deploys environments > >> (puppet-openstack-integration) is branched. So if puppet-barbican is > >> ready to be tested in Ocata, we'll patch > >> puppet-openstack-integration/master to start testing it and it won't > >> break stable jobs. > >> Like this, we were successfully able to maintain a fair number of > >> scenarios and keep our coverage increasing over each cycle. > >> > >> I see 2 sub-options here: > >> > >> a) Move CI environments and pingtest into > >> tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo > >> is branched and we could add a README to explain these files are used > >> in CI and we don't guarantee they would work outside TripleO CI tools. > >> b) Branch tripleo-ci repository. Personally I don't like this solution > >> because a lot of patches in this repo are not related to OpenStack > >> versions, which means we would need to backport most of the things > >> from master. > > > > > > I'd +1 this idea. It sounds like we could make the scenarios generic > enough > > to be usable also outside CI? Maybe they can serve as samples? > > -- > > Giulio Fidente > > GPG KEY: 08D733BA > > > > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: > unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] CI scenarios design - how to add more services
I think would be cool to go with option, +1 to 1.A IMHO, - Easier to read. - Easier to maintain. - We don't make backports, instead we guarantee backwards compatibility. - We'll re-use experience from Puppet OpenStack CI. On Wed, Nov 23, 2016 at 10:13 PM, Giulio Fidente wrote: > hi Emilien, > > thanks for putting some thought into this. We have a similar problem to test > RGW which was only added in Newton. > > > On 11/23/2016 03:02 AM, Emilien Macchi wrote: >> >> == Context >> >> In Newton we added new multinode jobs called "scenarios". >> The challenged we tried to solve was "how to test the maximum of >> services without overloading the nodes that run tests". >> >> Each scenarios deploys a set of services, which allows us to >> horizontally scale the number of scenarios to increase the service >> testing coverage. >> See the result here: >> https://github.com/openstack-infra/tripleo-ci#service-testing-matrix >> >> To implement this model, we took example of Puppet OpenStack CI: >> https://github.com/openstack/puppet-openstack-integration#description >> We even tried to keep consistent the services/scenarios relations, so >> it's consistent and easier to maintain. >> >> Everything was fine until we had to add new services during Ocata cycles. >> Because tripleo-ci repository is not branched, adding Barbican service >> in the TripleO environment for scenario002 would break Newton CI jobs. >> During my vacations, the team created a new scenario, scenario004, >> that deploys Barbican and that is only run for Ocata jobs. >> I don't think we should proceed this way, and let me explain why. >> >> == Problem >> >> How to scale the number of services that we test without increasing >> the number of scenarios and therefore the complexity of maintaining >> them on long-term. >> >> >> == Solutions >> >> The list is not exhaustive, feel free to add more. >> >> 1) Re-use experience from Puppet OpenStack CI and have environments >> that are in a branched repository. >> environments. >> In Puppet OpenStack CI, the repository that deploys environments >> (puppet-openstack-integration) is branched. So if puppet-barbican is >> ready to be tested in Ocata, we'll patch >> puppet-openstack-integration/master to start testing it and it won't >> break stable jobs. >> Like this, we were successfully able to maintain a fair number of >> scenarios and keep our coverage increasing over each cycle. >> >> I see 2 sub-options here: >> >> a) Move CI environments and pingtest into >> tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo >> is branched and we could add a README to explain these files are used >> in CI and we don't guarantee they would work outside TripleO CI tools. >> b) Branch tripleo-ci repository. Personally I don't like this solution >> because a lot of patches in this repo are not related to OpenStack >> versions, which means we would need to backport most of the things >> from master. > > > I'd +1 this idea. It sounds like we could make the scenarios generic enough > to be usable also outside CI? Maybe they can serve as samples? > -- > Giulio Fidente > GPG KEY: 08D733BA > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] CI scenarios design - how to add more services
hi Emilien, thanks for putting some thought into this. We have a similar problem to test RGW which was only added in Newton. On 11/23/2016 03:02 AM, Emilien Macchi wrote: == Context In Newton we added new multinode jobs called "scenarios". The challenged we tried to solve was "how to test the maximum of services without overloading the nodes that run tests". Each scenarios deploys a set of services, which allows us to horizontally scale the number of scenarios to increase the service testing coverage. See the result here: https://github.com/openstack-infra/tripleo-ci#service-testing-matrix To implement this model, we took example of Puppet OpenStack CI: https://github.com/openstack/puppet-openstack-integration#description We even tried to keep consistent the services/scenarios relations, so it's consistent and easier to maintain. Everything was fine until we had to add new services during Ocata cycles. Because tripleo-ci repository is not branched, adding Barbican service in the TripleO environment for scenario002 would break Newton CI jobs. During my vacations, the team created a new scenario, scenario004, that deploys Barbican and that is only run for Ocata jobs. I don't think we should proceed this way, and let me explain why. == Problem How to scale the number of services that we test without increasing the number of scenarios and therefore the complexity of maintaining them on long-term. == Solutions The list is not exhaustive, feel free to add more. 1) Re-use experience from Puppet OpenStack CI and have environments that are in a branched repository. environments. In Puppet OpenStack CI, the repository that deploys environments (puppet-openstack-integration) is branched. So if puppet-barbican is ready to be tested in Ocata, we'll patch puppet-openstack-integration/master to start testing it and it won't break stable jobs. Like this, we were successfully able to maintain a fair number of scenarios and keep our coverage increasing over each cycle. I see 2 sub-options here: a) Move CI environments and pingtest into tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo is branched and we could add a README to explain these files are used in CI and we don't guarantee they would work outside TripleO CI tools. b) Branch tripleo-ci repository. Personally I don't like this solution because a lot of patches in this repo are not related to OpenStack versions, which means we would need to backport most of the things from master. I'd +1 this idea. It sounds like we could make the scenarios generic enough to be usable also outside CI? Maybe they can serve as samples? -- Giulio Fidente GPG KEY: 08D733BA __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tripleo] CI scenarios design - how to add more services
== Context In Newton we added new multinode jobs called "scenarios". The challenged we tried to solve was "how to test the maximum of services without overloading the nodes that run tests". Each scenarios deploys a set of services, which allows us to horizontally scale the number of scenarios to increase the service testing coverage. See the result here: https://github.com/openstack-infra/tripleo-ci#service-testing-matrix To implement this model, we took example of Puppet OpenStack CI: https://github.com/openstack/puppet-openstack-integration#description We even tried to keep consistent the services/scenarios relations, so it's consistent and easier to maintain. Everything was fine until we had to add new services during Ocata cycles. Because tripleo-ci repository is not branched, adding Barbican service in the TripleO environment for scenario002 would break Newton CI jobs. During my vacations, the team created a new scenario, scenario004, that deploys Barbican and that is only run for Ocata jobs. I don't think we should proceed this way, and let me explain why. == Problem How to scale the number of services that we test without increasing the number of scenarios and therefore the complexity of maintaining them on long-term. == Solutions The list is not exhaustive, feel free to add more. 1) Re-use experience from Puppet OpenStack CI and have environments that are in a branched repository. environments. In Puppet OpenStack CI, the repository that deploys environments (puppet-openstack-integration) is branched. So if puppet-barbican is ready to be tested in Ocata, we'll patch puppet-openstack-integration/master to start testing it and it won't break stable jobs. Like this, we were successfully able to maintain a fair number of scenarios and keep our coverage increasing over each cycle. I see 2 sub-options here: a) Move CI environments and pingtest into tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo is branched and we could add a README to explain these files are used in CI and we don't guarantee they would work outside TripleO CI tools. b) Branch tripleo-ci repository. Personally I don't like this solution because a lot of patches in this repo are not related to OpenStack versions, which means we would need to backport most of the things from master. 2) Introduce branch-based scenario tests - https://review.openstack.org/#/c/396008/ It duplicates a lot of code and it's imho not really effective, though this solution would correctly works. 3) Introduce a new scenario each time we have new services (like we did with scenario004). By adding new scenarios at each release because we test new services is imho the wrong choice because: a) it adds complexity in our we're going to maintain these scenarios. b) it consumes more CI resources that we would need when some patches have to run all scenarios jobs. So I gave my opinion on the solutions, discussion is now open and my hope is that we find a consensus soon, so we can make progress in our testing coverage. Thanks, -- Emilien Macchi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev