I think would be cool to go with option, +1 to 1.A IMHO, - Easier to read. - Easier to maintain. - We don't make backports, instead we guarantee backwards compatibility. - We'll re-use experience from Puppet OpenStack CI.
On Wed, Nov 23, 2016 at 10:13 PM, Giulio Fidente <gfide...@redhat.com> wrote: > hi Emilien, > > thanks for putting some thought into this. We have a similar problem to test > RGW which was only added in Newton. > > > On 11/23/2016 03:02 AM, Emilien Macchi wrote: >> >> == Context >> >> In Newton we added new multinode jobs called "scenarios". >> The challenged we tried to solve was "how to test the maximum of >> services without overloading the nodes that run tests". >> >> Each scenarios deploys a set of services, which allows us to >> horizontally scale the number of scenarios to increase the service >> testing coverage. >> See the result here: >> https://github.com/openstack-infra/tripleo-ci#service-testing-matrix >> >> To implement this model, we took example of Puppet OpenStack CI: >> https://github.com/openstack/puppet-openstack-integration#description >> We even tried to keep consistent the services/scenarios relations, so >> it's consistent and easier to maintain. >> >> Everything was fine until we had to add new services during Ocata cycles. >> Because tripleo-ci repository is not branched, adding Barbican service >> in the TripleO environment for scenario002 would break Newton CI jobs. >> During my vacations, the team created a new scenario, scenario004, >> that deploys Barbican and that is only run for Ocata jobs. >> I don't think we should proceed this way, and let me explain why. >> >> == Problem >> >> How to scale the number of services that we test without increasing >> the number of scenarios and therefore the complexity of maintaining >> them on long-term. >> >> >> == Solutions >> >> The list is not exhaustive, feel free to add more. >> >> 1) Re-use experience from Puppet OpenStack CI and have environments >> that are in a branched repository. >> environments. >> In Puppet OpenStack CI, the repository that deploys environments >> (puppet-openstack-integration) is branched. So if puppet-barbican is >> ready to be tested in Ocata, we'll patch >> puppet-openstack-integration/master to start testing it and it won't >> break stable jobs. >> Like this, we were successfully able to maintain a fair number of >> scenarios and keep our coverage increasing over each cycle. >> >> I see 2 sub-options here: >> >> a) Move CI environments and pingtest into >> tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo >> is branched and we could add a README to explain these files are used >> in CI and we don't guarantee they would work outside TripleO CI tools. >> b) Branch tripleo-ci repository. Personally I don't like this solution >> because a lot of patches in this repo are not related to OpenStack >> versions, which means we would need to backport most of the things >> from master. > > > I'd +1 this idea. It sounds like we could make the scenarios generic enough > to be usable also outside CI? Maybe they can serve as samples? > -- > Giulio Fidente > GPG KEY: 08D733BA > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev