On Wed, Apr 5, 2017 at 1:49 PM, Emilien Macchi <[email protected]> wrote: [...]
> == Solutions > > 1) Switch pingtest to Tempest run on some specific tests, with feature > parity of what we had with pingtest. > For example, we could imagine to run the scenarios that deploys VM and > boot from volume. It would test the same thing as pingtest (details > can be discussed here). > Each scenario would run more tests depending on the service that they > run (scenario001 is telemetry, so it would run some tempest tests for > Ceilometer, Aodh, Gnocchi, etc). > We should work at making the tempest run as short as possible, and the > close as possible from what we have with a pingtest. [...] 4 months later :-) We enabled Tempest on the following jobs: gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container gate-tripleo-ci-centos-7-scenario003-multinode-oooq-container gate-tripleo-ci-centos-7-scenario004-multinode-oooq-container gate-tripleo-ci-centos-7-scenario001-multinode-oooq gate-tripleo-ci-centos-7-scenario002-multinode-oooq gate-tripleo-ci-centos-7-scenario003-multinode-oooq gate-tripleo-ci-centos-7-scenario004-multinode-oooq gate-tripleo-ci-centos-7-nonha-multinode-oooq gate-tripleo-ci-centos-7-containers-multinode It has a feature parity with what pingtest did. For example, scenario001 (focused on Telemetry) test boot from volume and the whole autoscsaling scenario from Telemetry services, so we can test end-to-end that our users can deploy autocsaling apps using OpenStack. We run Tempest on stable/pike and master, and still run pingtest on stable/newton and stable/ocata (no change is planned for stable branches). No change has been planned yet for OVB jobs but we might want to think about it later. You can go on http://status.openstack.org/openstack-health/#/job/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container and see stats as well. We also watched the amount of time spent on pingtest versus tempest and we knew tempest would take more time. It's the case and we've noticed the average is ~2 min more for now. It's acceptable I guess for now. Anyway, we hope this effort was useful and we can make progress to the next steps: - notify teams when some tests fail - increase test coverage depending on which services are deployed -- Emilien Macchi __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
