Re: [DISCUSS] Extending Marvin Smoketest Suite

[Boris Stoyanov]

Hi all, 

Furthermore I think there’s a way to slice through the tests and pick up those 
you need more efficiently. 

As a first step we can work with the test attributes and introduce a 
“quick-test” tag. This tag would be used on tests that are vital and really 
high level, like for example CRUD operations and some basic functionality like 
deploy a VM. The idea behind that is to execute this for less then an hour and 
get quick feedback on your changes. Based on this feedback you’ll be able to 
take decision whether to continue using this build for testing or not. 

Currently smoketests run for about a day. And surely we can skip some of the 
areas since they are not touched. For example if you did a change in the VR, 
there’s no need to run all the Account tests on this PR, but focus more on the 
tests that are relative to the change. Using the same approach we can divide 
all the tests into categories, for example: vm-deployment, virtual-router, 
accounts, storage, infrastructure, projects, events and so on… Having this 
categories we would have more focused test execution for our PRs and it’ll make 
it more faster. 

For example if I submit a PR for a change in the virtual router. As a first 
step I would execute the “quick-test” label, this would give me early feedback 
that there’s generally nothing wrong with my changes and most of the CloudStack 
feature are working (<1h in time). Then I would run the tests with the label 
“virtual-router” which would drill down the directory and fetch me the tests 
related to the VR only and execute them. 
This approach could save about half the time of test execution or even more… 

Nevertheless these changes would bring no change in the way people are using 
the tests right now and they will be able to execute them as before. It’s more 
like a on demand asking for a group of tests to be executed. 

As next steps we could think of a way to utilize the labels in GitHub more and 
link them with the test categories and specific hypervisors. After that we 
could look for a way to automate the test runs with BlueOrangutan with the 
given labels. 

Thanks,
Boris Stoyanov
 


boris.stoya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

> On Jul 10, 2017, at 10:14 AM, Boris Stoyanov  
> wrote:
> 
> Thanks Will, 
> 
> There’s big set of projects tests in the component suite, I’ll add that in 
> the the task and review what tests would be good to go in the smoke test 
> suite. Please shout if you explicitly need something added. 
> 
> Thanks,
> Bobby.
> 
> boris.stoya...@shapeblue.com 
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
> 
> 
> 
>> On Jul 7, 2017, at 2:12 PM, Will Stevens  wrote:
>> 
>> I would like to see better testing of 'projects' as well, it is almost
>> entirely untested.
>> 
>> *Will Stevens*
>> CTO
>> 
>> 
>> 
>> On Fri, Jul 7, 2017 at 7:05 AM, Boris Stoyanov >> wrote:
>> 
>>> Hi all,
>>> 
>>> I’ve started up this improvement, which is to extend Marvin smoketests
>>> with some more tests. I’ve indicated a few tests from the component tests
>>> that I think would be really good to have into smoketests, like for example
>>> Usage tests ( we don’t have any currently).
>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9989
>>> 
>>> 
>>> Here’s a list of items that I think are suitable candidates:
>>> test_accounts.py - There are no tests for accounts/users in the current
>>> suite
>>> test_host_maintenance.py - We have received reports for multiple failures
>>> around moving VMs away from a Host set for Maintenance.
>>> test_regions.py - there are no tests for this in the current suite
>>> test_usage.py - No tests and this is quite vital to public and private
>>> cloud providers, CloudStack users rely on this usage metrics and we should
>>> be testing this on a daily basis.
>>> 
>>> In the JIRA ticket I’ve added up more details against each group of tests
>>> since there are some with problems at this stage.
>>> 
>>> Can you guys please review and add more areas that you would like to be
>>> smoke tested as part of each PR in the community repo?
>>> 
>>> Thanks
>>> Boris Stoyanov
>>> 
>>> boris.stoya...@shapeblue.com
>>> www.shapeblue.com
>>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> @shapeblue
>>> 
>>> 
>>> 
>>> 
> 

Re: [DISCUSS] Extending Marvin Smoketest Suite

[Boris Stoyanov]

Thanks Will, 

There’s big set of projects tests in the component suite, I’ll add that in the 
the task and review what tests would be good to go in the smoke test suite. 
Please shout if you explicitly need something added. 

Thanks,
Bobby.

boris.stoya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

> On Jul 7, 2017, at 2:12 PM, Will Stevens  wrote:
> 
> I would like to see better testing of 'projects' as well, it is almost
> entirely untested.
> 
> *Will Stevens*
> CTO
> 
> 
> 
> On Fri, Jul 7, 2017 at 7:05 AM, Boris Stoyanov > wrote:
> 
>> Hi all,
>> 
>> I’ve started up this improvement, which is to extend Marvin smoketests
>> with some more tests. I’ve indicated a few tests from the component tests
>> that I think would be really good to have into smoketests, like for example
>> Usage tests ( we don’t have any currently).
>> https://issues.apache.org/jira/browse/CLOUDSTACK-9989
>> 
>> 
>> Here’s a list of items that I think are suitable candidates:
>> test_accounts.py - There are no tests for accounts/users in the current
>> suite
>> test_host_maintenance.py - We have received reports for multiple failures
>> around moving VMs away from a Host set for Maintenance.
>> test_regions.py - there are no tests for this in the current suite
>> test_usage.py - No tests and this is quite vital to public and private
>> cloud providers, CloudStack users rely on this usage metrics and we should
>> be testing this on a daily basis.
>> 
>> In the JIRA ticket I’ve added up more details against each group of tests
>> since there are some with problems at this stage.
>> 
>> Can you guys please review and add more areas that you would like to be
>> smoke tested as part of each PR in the community repo?
>> 
>> Thanks
>> Boris Stoyanov
>> 
>> boris.stoya...@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>> 
>> 
>> 
>> 

Re: [DISCUSS] Extending Marvin Smoketest Suite

[Will Stevens]

I would like to see better testing of 'projects' as well, it is almost
entirely untested.

*Will Stevens*
CTO



On Fri, Jul 7, 2017 at 7:05 AM, Boris Stoyanov  wrote:

> Hi all,
>
> I’ve started up this improvement, which is to extend Marvin smoketests
> with some more tests. I’ve indicated a few tests from the component tests
> that I think would be really good to have into smoketests, like for example
> Usage tests ( we don’t have any currently).
> https://issues.apache.org/jira/browse/CLOUDSTACK-9989
>
>
> Here’s a list of items that I think are suitable candidates:
> test_accounts.py - There are no tests for accounts/users in the current
> suite
> test_host_maintenance.py - We have received reports for multiple failures
> around moving VMs away from a Host set for Maintenance.
> test_regions.py - there are no tests for this in the current suite
> test_usage.py - No tests and this is quite vital to public and private
> cloud providers, CloudStack users rely on this usage metrics and we should
> be testing this on a daily basis.
>
> In the JIRA ticket I’ve added up more details against each group of tests
> since there are some with problems at this stage.
>
> Can you guys please review and add more areas that you would like to be
> smoke tested as part of each PR in the community repo?
>
> Thanks
> Boris Stoyanov
>
> boris.stoya...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>

[DISCUSS] Extending Marvin Smoketest Suite

[Boris Stoyanov]

Hi all,

I’ve started up this improvement, which is to extend Marvin smoketests with 
some more tests. I’ve indicated a few tests from the component tests that I 
think would be really good to have into smoketests, like for example Usage 
tests ( we don’t have any currently).
https://issues.apache.org/jira/browse/CLOUDSTACK-9989


Here’s a list of items that I think are suitable candidates:
test_accounts.py - There are no tests for accounts/users in the current suite
test_host_maintenance.py - We have received reports for multiple failures 
around moving VMs away from a Host set for Maintenance.
test_regions.py - there are no tests for this in the current suite
test_usage.py - No tests and this is quite vital to public and private cloud 
providers, CloudStack users rely on this usage metrics and we should be testing 
this on a daily basis.

In the JIRA ticket I’ve added up more details against each group of tests since 
there are some with problems at this stage.

Can you guys please review and add more areas that you would like to be smoke 
tested as part of each PR in the community repo?

Thanks
Boris Stoyanov

boris.stoya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue