Boris Stoyanov created CLOUDSTACK-10017:
-------------------------------------------
Summary: Add quick-test attribute to cherry picked tests in the
smoke tests suite
Key: CLOUDSTACK-10017
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10017
Project: CloudStack
Issue Type: Improvement
Security Level: Public (Anyone can view this level - this is the default.)
Components: marvin
Reporter: Boris Stoyanov
As a first step we can work with the test attributes and introduce a
“quick-test” tag. This tag would be used on tests that are vital and really
high level, like for example CRUD operations and some basic functionality like
deploy a VM. The idea behind that is to execute this for less then an hour and
get quick feedback on your changes. Based on this feedback you’ll be able to
take decision whether to continue using this build for testing or not.
Currently smoketests run for about a day. And surely we can skip some of the
areas since they are not touched. For example if you did a change in the VR,
there’s no need to run all the Account tests on this PR, but focus more on the
tests that are relative to the change. Using the same approach we can divide
all the tests into categories, for example: vm-deployment, virtual-router,
accounts, storage, infrastructure, projects, events and so on… Having this
categories we would have more focused test execution for our PRs and it’ll make
it more faster.
For example if I submit a PR for a change in the virtual router. As a first
step I would execute the “quick-test” label, this would give me early feedback
that there’s generally nothing wrong with my changes and most of the CloudStack
feature are working (<1h in time). Then I would run the tests with the label
“virtual-router” which would drill down the directory and fetch me the tests
related to the VR only and execute them.
This approach could save about half the time of test execution or even more…
Nevertheless these changes would bring no change in the way people are using
the tests right now and they will be able to execute them as before. It’s more
like a on demand asking for a group of tests to be executed.
As next steps we could think of a way to utilize the labels in GitHub more and
link them with the test categories and specific hypervisors. After that we
could look for a way to automate the test runs with BlueOrangutan with the
given labels.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
