Re: [openstack-dev] [tripleo] CI Squads’ Sprint 12 Summary: libvirt-reproducer, python-tempestconf

2018-05-09 Thread Bogdan Dobrelya 

On 5/9/18 4:24 AM, Matt Young wrote:

Greetings,

The TripleO squads for CI and Tempest have just completed Sprint 12.  
The following is a summary of activities during this sprint.   Details 
on our team structure can be found in the spec [1].


---

# Sprint 12 Epic (CI): Libvirt Reproducer

* Epic Card: https://trello.com/c/JEGLSVh6/51-reproduce-ci-jobs-with-libvirt
* Tasks: http://ow.ly/O1vZ30jTSc3

"Allow developers to reproduce a multinode CI job on a bare metal host 
using libvirt"
"Enable the same workflows used in upstream CI / reproducer using 
libvirt instead of OVB as the provisioning mechanism"


The CI Squad prototyped, designed, and implemented new functionality for 
our CI reproducer.   “Reproducers” are scripts generated by each CI job 
that allow the job/test to be recreated.  These are useful to both CI 
team members when investigating failures, as well as developers creating 
failures with the intent to iteratively debug and/or fix issues.  Prior 
to this sprint, the reproducer scripts supported reproduction of 
upstream CI jobs using OVB, typically on RDO Cloud.  This sprint we 
extended this capability to support reproduction of jobs in libvirt.


This work was done for a few reasons:

* (short term) enable the team to work on upgrades and other CI team 
tasks more efficiently by mitigating recurring RDO Cloud infrastructure 
issues.  This was the primary motivator for doing this work at this time.
* (mid-longer term) enhance / enable iterative workflows such as THT 
development, debugging deployment scenarios, etc.  Snapshots in 
particular have proven quite useful.  As we look towards a future with a 
viable single-node deployment capability, libvirt has clear benefits for 
common developer scenarios.


Thank you for that, a really cool feature for tripleo development!



It is expected that further iteration and refinement of this initial 
implementation will be required before the tripleo-ci team is able to 
support this broadly.  What we’ve done works as designed.  While we 
welcome folks to explore, please note that we are not announcing a 
supported libvirt reproducer meant for use outside the tripleo-ci team 
at this time.  We expect some degree of change, and have a number of 
RFE’s resulting from our testing as well as documentation patches that 
we’re iterating on.


That said, we think it’s really cool, works well in its current form, 
and are optimistic about its future.


## We did the following (CI):

* Add support to the reproducer script [2,3] generated by CI to enable 
libvirt.

* Basic snapshot create/restore [4] capability.
* Tested Scenarios: featureset 3 (UC idem), 10 (multinode containers), 
37 (min OC + minor update).  See sprint cards for details.
* 14-18 RFE’s identified as part of testing for future work 
http://ow.ly/J2u830jTSLG


---

# Sprint 12 Epic (Tempest):

* Epic Card: https://trello.com/c/ifIYQsxs/75-sprint-12-undercloud-tempest
* Tasks: http://ow.ly/GGvc30jTSfV

“Run tempest on undercloud by using containerized and packaged tempest”
“Complete work items carried from sprint 11 or another side work going on.”

## We did the following (Tempest):

* Create tripleo-ci jobs that run containerized tempest on all stable 
branches.
* Create documentation for configuring and running tempest using 
containerized tempest on UC @tripleo.org , and blog 
posts. [5,6,7]

* Run certification tests via new Jenkins job using ansible role [8]
* Refactor validate-tempest CI role for UC and containers

---

# Ruck and Rover

Each sprint two of the team members assume the roles of Ruck and Rover 
(each for half of the sprint).


* Ruck is responsible to monitoring the CI, checking for failures, 
opening bugs, participate on meetings, and this is your focal point to 
any CI issues.
* Rover is responsible to work on these bugs, fix problems and the rest 
of the team are focused on the sprint. For more information about our 
structure, check [1]


## Ruck & Rover (Sprint 12), Etherpad [9,10]:

* Quique Llorente(quiquell)
* Gabriele Cerami (panda)

A few notable issues where substantial time was spent were:

1767099 
periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master vxlan 
tunnel fails randomly

1758899 reproducer-quickstart.sh building wrong gating package.
1767343 gate tripleo-ci-centos-7-containers-multinode fails to update 
packages in cron container
1762351 
periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload 
is timeout Depends on https://bugzilla.redhat.com/show_bug.cgi?id=1565179

1766873 quickstart on ovb doesn't yield a deployment
1767049 Error during test discovery : 'must specify exactly one of host 
or intercept' Depends on https://bugzilla.redhat.com/show_bug.cgi?id=1434385
1767076 Creating pingtest_sack fails: Failed to schedule instances: 
NoValidHost_Remote: No valid host was found

1763634 devmode.sh --ovb fails to deploy overcloud
1765680 Incorrect branch used for not gated tripleo-upgrade repo

[openstack-dev] [tripleo] CI Squads’ Sprint 12 Summary: libvirt-reproducer, python-tempestconf

2018-05-08 Thread Matt Young 
Greetings,

The TripleO squads for CI and Tempest have just completed Sprint 12.  The
following is a summary of activities during this sprint.   Details on our
team structure can be found in the spec [1].

---

# Sprint 12 Epic (CI): Libvirt Reproducer

* Epic Card: https://trello.com/c/JEGLSVh6/51-reproduce-ci-jobs-with-libvirt
* Tasks: http://ow.ly/O1vZ30jTSc3

"Allow developers to reproduce a multinode CI job on a bare metal host
using libvirt"
"Enable the same workflows used in upstream CI / reproducer using libvirt
instead of OVB as the provisioning mechanism"

The CI Squad prototyped, designed, and implemented new functionality for
our CI reproducer.   “Reproducers” are scripts generated by each CI job
that allow the job/test to be recreated.  These are useful to both CI team
members when investigating failures, as well as developers creating
failures with the intent to iteratively debug and/or fix issues.  Prior to
this sprint, the reproducer scripts supported reproduction of upstream CI
jobs using OVB, typically on RDO Cloud.  This sprint we extended this
capability to support reproduction of jobs in libvirt.

This work was done for a few reasons:

* (short term) enable the team to work on upgrades and other CI team tasks
more efficiently by mitigating recurring RDO Cloud infrastructure issues.
This was the primary motivator for doing this work at this time.
* (mid-longer term) enhance / enable iterative workflows such as THT
development, debugging deployment scenarios, etc.  Snapshots in particular
have proven quite useful.  As we look towards a future with a viable
single-node deployment capability, libvirt has clear benefits for common
developer scenarios.

It is expected that further iteration and refinement of this initial
implementation will be required before the tripleo-ci team is able to
support this broadly.  What we’ve done works as designed.  While we welcome
folks to explore, please note that we are not announcing a supported
libvirt reproducer meant for use outside the tripleo-ci team at this time.
We expect some degree of change, and have a number of RFE’s resulting from
our testing as well as documentation patches that we’re iterating on.

That said, we think it’s really cool, works well in its current form, and
are optimistic about its future.

## We did the following (CI):

* Add support to the reproducer script [2,3] generated by CI to enable
libvirt.
* Basic snapshot create/restore [4] capability.
* Tested Scenarios: featureset 3 (UC idem), 10 (multinode containers), 37
(min OC + minor update).  See sprint cards for details.
* 14-18 RFE’s identified as part of testing for future work
http://ow.ly/J2u830jTSLG

---

# Sprint 12 Epic (Tempest):

* Epic Card: https://trello.com/c/ifIYQsxs/75-sprint-12-undercloud-tempest
* Tasks: http://ow.ly/GGvc30jTSfV

“Run tempest on undercloud by using containerized and packaged tempest”
“Complete work items carried from sprint 11 or another side work going on.”

## We did the following (Tempest):

* Create tripleo-ci jobs that run containerized tempest on all stable
branches.
* Create documentation for configuring and running tempest using
containerized tempest on UC @tripleo.org, and blog posts. [5,6,7]
* Run certification tests via new Jenkins job using ansible role [8]
* Refactor validate-tempest CI role for UC and containers

---

# Ruck and Rover

Each sprint two of the team members assume the roles of Ruck and Rover
(each for half of the sprint).

* Ruck is responsible to monitoring the CI, checking for failures, opening
bugs, participate on meetings, and this is your focal point to any CI
issues.
* Rover is responsible to work on these bugs, fix problems and the rest of
the team are focused on the sprint. For more information about our
structure, check [1]

## Ruck & Rover (Sprint 12), Etherpad [9,10]:

* Quique Llorente(quiquell)
* Gabriele Cerami (panda)

A few notable issues where substantial time was spent were:

1767099 periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master
vxlan tunnel fails randomly
1758899 reproducer-quickstart.sh building wrong gating package.
1767343 gate tripleo-ci-centos-7-containers-multinode fails to update
packages in cron container
1762351
periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload is
timeout Depends on https://bugzilla.redhat.com/show_bug.cgi?id=1565179
1766873 quickstart on ovb doesn't yield a deployment
1767049 Error during test discovery : 'must specify exactly one of host or
intercept' Depends on https://bugzilla.redhat.com/show_bug.cgi?id=1434385
1767076 Creating pingtest_sack fails: Failed to schedule instances:
NoValidHost_Remote: No valid host was found
1763634 devmode.sh --ovb fails to deploy overcloud
1765680 Incorrect branch used for not gated tripleo-upgrade repo

If you have any questions and/or suggestions, please contact us in #oooq or
#tripleo

Thanks,

Matt


tq: https://github.com/openstack/tripleo-quickstart
tqe: