Hi, In last weeks QA meeting, there was some talk about making improvements around devstack-gate and tempest.
As I understand it, one of the problems is that things are changing in other projects that are causing tests in the full (non-smoke) tempest suite to fail, which prevents changes being merged to tempest. This is unique to tempest because of the asymmetrical gate currently in place, where tempest is gated on the full suite but other projects are gated only on the smoke tests. There was talk about solving this by adding a second, non-gating, run of tempest to run those jobs. I'd like to keep the number of devstack/tempest runs low, because of the heavy resource consumption -- especially when you consider that we've committed to doing cross-version testing, which will start to exponentially increase the number of devstack runs we do. Here's what I'd like to propose instead: 1) Make the tempest gate symmetrical -- cross-project gating works best when all the projects gate on the same tests so that one project can't break another. 2) Add more tests to the set of tests that get run in the devstack/tempest gate (currently the smoke tests). It sounds like several tests in the full suite really are breaking because of changes in other projects -- those should probably be considered for promotion. 3) Modify devstack-gate to run tempest a second time (on the same host, after completing the smoke tests), to run the rest of the suite, but ignore the results of that for gating purposes. This way the results will be available for every change, without affecting gating or consuming more virtual machines. In other words, have devstack gate: run devstack run smoke tests save return code run full suite - smoke tests exit with saved return code How does that sound? -Jim -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
