On 11/22/2016 09:02 PM, Emilien Macchi wrote: > == Context > > In Newton we added new multinode jobs called "scenarios". > The challenged we tried to solve was "how to test the maximum of > services without overloading the nodes that run tests". > > Each scenarios deploys a set of services, which allows us to > horizontally scale the number of scenarios to increase the service > testing coverage. > See the result here: > https://github.com/openstack-infra/tripleo-ci#service-testing-matrix > > To implement this model, we took example of Puppet OpenStack CI: > https://github.com/openstack/puppet-openstack-integration#description > We even tried to keep consistent the services/scenarios relations, so > it's consistent and easier to maintain. > > Everything was fine until we had to add new services during Ocata cycles. > Because tripleo-ci repository is not branched, adding Barbican service > in the TripleO environment for scenario002 would break Newton CI jobs. > During my vacations, the team created a new scenario, scenario004, > that deploys Barbican and that is only run for Ocata jobs. > I don't think we should proceed this way, and let me explain why. > > == Problem > > How to scale the number of services that we test without increasing > the number of scenarios and therefore the complexity of maintaining > them on long-term. > > > == Solutions > > The list is not exhaustive, feel free to add more. > > 1) Re-use experience from Puppet OpenStack CI and have environments > that are in a branched repository. > environments. > In Puppet OpenStack CI, the repository that deploys environments > (puppet-openstack-integration) is branched. So if puppet-barbican is > ready to be tested in Ocata, we'll patch > puppet-openstack-integration/master to start testing it and it won't > break stable jobs. > Like this, we were successfully able to maintain a fair number of > scenarios and keep our coverage increasing over each cycle. > > I see 2 sub-options here: > > a) Move CI environments and pingtest into > tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo > is branched and we could add a README to explain these files are used > in CI and we don't guarantee they would work outside TripleO CI tools.
I also like this solution the best. It has the added benefit of being able to review the CI for a new service in the same patch (or patch chain) that adds the new service. We already have the low-memory environment in tht, which while not CI specific, is definitely a CI requirement. > b) Branch tripleo-ci repository. Personally I don't like this solution > because a lot of patches in this repo are not related to OpenStack > versions, which means we would need to backport most of the things > from master. > > 2) Introduce branch-based scenario tests - > https://review.openstack.org/#/c/396008/ > It duplicates a lot of code and it's imho not really effective, though > this solution would correctly works. > > 3) Introduce a new scenario each time we have new services (like we > did with scenario004). > By adding new scenarios at each release because we test new services > is imho the wrong choice because: > a) it adds complexity in our we're going to maintain these scenarios. > b) it consumes more CI resources that we would need when some patches > have to run all scenarios jobs. > > > So I gave my opinion on the solutions, discussion is now open and my > hope is that we find a consensus soon, so we can make progress in our > testing coverage. > Thanks, > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
