Re: [openstack-dev] [PTG][QA] Queens PTG - QA Summary

[Andrea Frittoli]

On Tue, Sep 19, 2017 at 6:51 PM Andrea Frittoli 
wrote:

> Dear Bats,
>
> thanks everyone for participating to the Queens PTG - in person and
> remotely - and for making it a successful and enjoyable event.
>
> Here's a summary of what we discussed and achieved during the week.
> This report is far from complete - please complement it with QA related
> stories I may have missed.
>
> Unfortunately I ran out of bat stickers, I hope to have some more for the
> next time we meet :)
>
> Pike Retrospective
> -
> We experimented this time with running a retrospective [0].
> The input for the retrospective was almost exclusively from members of the
> QA team, which means that either the retrospective was poorly advertised by
> me, or that we did a nice job for the community during the QA cycle. I hope
> the latter :)
>
> The outcome of the retrospective was mostly 'yay' for tasks completed, and
> 'next cycle' for things we did not have time to work on.
> A few extra things that came up were:
> - bug triage and fix: we may need a bug czar and more automation to stay
> on top of the bug queue
> - elastic recheck categorisation: we may need an e-r czar to ensure we
> don't leave gate failures and races uncategorised and accumulating
> - meetings are mostly attended in the APAC time zone and very seldom
> attended by non-qa folks. We will consider shortening / reducing them and
> complementing them with QA office hours
>
> Monitoring the Gate
> --
> At the beginning of the Pike cycle we had a number of stability issues in
> the gate.
> To prevent issues from accumulating over time we discussed a few ideas
> about monitoring the status of the gate and providing more feedback to
> reviewers to help catch patches that may introduce issues [1].
> There are a few things that can be done relatively easily (or that we
> already have) in terms of data collection: job duration and failure rates,
> aggregated dstat data, resources created by tests.
> We miss OpenStack Health contributors to create new views for this data.
> Links from gerrit and zuul dashboard into OpenStack Health would help
> making the data discoverable.
>
> Tempest
> 
> A large chunk of the patches required to mark test.py as a stable
> interface for plugins was merged during the summit.
> It was good to have them merged during PTG so it was easier to fix
> resulting issues in Tempest plugins - at least Manila and Sahara needed a
> small patch.
> There are two patch series left [2][3] which should have very little
> impact on plugins - we'll try to merge them as soon as possible to avoid
> disruptions later in the Queens cycle.
>
> We worked with a few project teams on the goal to split Tempest plugins to
> a dedicated repo.
> The step by step process [4] linked to the goal includes an example
> section - we hope to get more and more examples in there from team who
> already went through the process.
>
> Devstack
> 
> We discussed about devstack runtime. The time to setup an OpenStack cloud
> using devstack seems to have increased over time [5][6] - it would be good
> to investigate why and see if there is time we can save in the gate and on
> each developer laptop :) Between August and September the average runtime
> in the gate increased by about 200s.
>
> Upgrade Testing
> --
> Rolling upgrade testing via grenade is important for project that seek
> obtaining the support rolling upgrade tag [7].
> While the scope of Grenade is rather fixed, it should be possible to
> support ordering (or relative ordering) in project updates.
>
> Policy Testing
> --
> We discussed what's next for the Queens cycle - support for multi-policy
> testing is the largest chunk of work planned for now.
>
> stestr
> ---
> The migration in ostestr to use stestr internally happened shortly before
> the PTG [8] and we worked through the PTG to amend any deviation in
> behaviour that this may have caused.
> Next in the todo list is to run stestr natively in Tempest, bypassing
> ostestr completely.
> The plan is for this to lead the way for projects to gradually remove the
> dependency to ostestr completely.
>
> HA Testing
> ---
> We talked quite a bit about HA and non-functional testing in general.
> Non-functional testing is not a good fit for gate testing, since it's not
> as reliable as functional / integration testing and it often produces
> results which needs to be interpreted by a human being.
> It also has strong dependencies to the deployment tooling and
> architecture. Until now most of OpenStack non-functional testing has been
> done by vendors and operators using downstream tools.
>
> SamP and guatamdivgi presented to the QA team a proposal for an Ansible
> based framework for HA testing [9]. Plugins will allow to seamlessly port
> different test scenario against different cloud architecture, thus
> rendering the framework 

[openstack-dev] [PTG][QA] Queens PTG - QA Summary

[Andrea Frittoli]

Dear Bats,

thanks everyone for participating to the Queens PTG - in person and
remotely - and for making it a successful and enjoyable event.

Here's a summary of what we discussed and achieved during the week.
This report is far from complete - please complement it with QA related
stories I may have missed.

Unfortunately I ran out of bat stickers, I hope to have some more for the
next time we meet :)

Pike Retrospective
-
We experimented this time with running a retrospective [0].
The input for the retrospective was almost exclusively from members of the
QA team, which means that either the retrospective was poorly advertised by
me, or that we did a nice job for the community during the QA cycle. I hope
the latter :)

The outcome of the retrospective was mostly 'yay' for tasks completed, and
'next cycle' for things we did not have time to work on.
A few extra things that came up were:
- bug triage and fix: we may need a bug czar and more automation to stay on
top of the bug queue
- elastic recheck categorisation: we may need an e-r czar to ensure we
don't leave gate failures and races uncategorised and accumulating
- meetings are mostly attended in the APAC time zone and very seldom
attended by non-qa folks. We will consider shortening / reducing them and
complementing them with QA office hours

Monitoring the Gate
--
At the beginning of the Pike cycle we had a number of stability issues in
the gate.
To prevent issues from accumulating over time we discussed a few ideas
about monitoring the status of the gate and providing more feedback to
reviewers to help catch patches that may introduce issues [1].
There are a few things that can be done relatively easily (or that we
already have) in terms of data collection: job duration and failure rates,
aggregated dstat data, resources created by tests.
We miss OpenStack Health contributors to create new views for this data.
Links from gerrit and zuul dashboard into OpenStack Health would help
making the data discoverable.

Tempest

A large chunk of the patches required to mark test.py as a stable interface
for plugins was merged during the summit.
It was good to have them merged during PTG so it was easier to fix
resulting issues in Tempest plugins - at least Manila and Sahara needed a
small patch.
There are two patch series left [2][3] which should have very little impact
on plugins - we'll try to merge them as soon as possible to avoid
disruptions later in the Queens cycle.

We worked with a few project teams on the goal to split Tempest plugins to
a dedicated repo.
The step by step process [4] linked to the goal includes an example section
- we hope to get more and more examples in there from team who already went
through the process.

Devstack

We discussed about devstack runtime. The time to setup an OpenStack cloud
using devstack seems to have increased over time [5][6] - it would be good
to investigate why and see if there is time we can save in the gate and on
each developer laptop :) Between August and September the average runtime
in the gate increased by about 200s.

Upgrade Testing
--
Rolling upgrade testing via grenade is important for project that seek
obtaining the support rolling upgrade tag [7].
While the scope of Grenade is rather fixed, it should be possible to
support ordering (or relative ordering) in project updates.

Policy Testing
--
We discussed what's next for the Queens cycle - support for multi-policy
testing is the largest chunk of work planned for now.

stestr
---
The migration in ostestr to use stestr internally happened shortly before
the PTG [8] and we worked through the PTG to amend any deviation in
behaviour that this may have caused.
Next in the todo list is to run stestr natively in Tempest, bypassing
ostestr completely.
The plan is for this to lead the way for projects to gradually remove the
dependency to ostestr completely.

HA Testing
---
We talked quite a bit about HA and non-functional testing in general.
Non-functional testing is not a good fit for gate testing, since it's not
as reliable as functional / integration testing and it often produces
results which needs to be interpreted by a human being.
It also has strong dependencies to the deployment tooling and architecture.
Until now most of OpenStack non-functional testing has been done by vendors
and operators using downstream tools.

SamP and guatamdivgi presented to the QA team a proposal for an Ansible
based framework for HA testing [9]. Plugins will allow to seamlessly port
different test scenario against different cloud architecture, thus
rendering the framework of interest as a general testing tool for
OpenStack. The same concept can be extended to non-functional testing in
general.

It's not clear yet if any of this could run as part of OpenStack CI.  We
hope to see a PoC after a couple of months in the Queens cycle.
The NFV ecosystem