Re: [openstack-dev] [os-ansible-deployment] Periodic job in infra to test upgrades?

[Sean M. Collins]

Jesse Pretorius wrote:
> Hi Sean,
> 
> Great to see you taking the initiative on this.
> 
> I think the starting point we’d have to work from with the way the builds are 
> executed now would be to have the upgrade job execute in a periodic pipeline 
> that has a longer timeout. While it would be ideal to do on-commit tests it’s 
> just untenable right now as it would severely slow down the workflow.

OK - I pushed the following patch to project-config to reflect this
change

https://review.openstack.org/419517

It depends on https://review.openstack.org/418521 - I will work on
cleaning it up and fixing the errors (it might just need a recheck?)
-- 
Sean M. Collins

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-ansible-deployment] Periodic job in infra to test upgrades?

[Jesse Pretorius]

Hi Sean,

Great to see you taking the initiative on this.

I think the starting point we’d have to work from with the way the builds are 
executed now would be to have the upgrade job execute in a periodic pipeline 
that has a longer timeout. While it would be ideal to do on-commit tests it’s 
just untenable right now as it would severely slow down the workflow.

In terms of using a ‘cached’ or pre-built container rootfs, pre-build wheels 
and venvs, etc this is something that we at Rackspace are working on already 
and are hoping to share some experience with at the Pike PTG. The initial work 
is a bit of a hack job, but we’re hoping to learn enough to share and are 
hoping to collaborate to put together something that can be used by a broader 
community and contributed to by a broader community.

Right now, however, I think we need some sort of upgrade tests on a regular 
basis more than we need to implement an alternative emans of deploying. Our 
choices are therefore, as far as I can see, to have this done in External CI or 
to use OpenStack-Infra’s periodic jobs. Once we have something in place we can 
work towards improving the execution speed (using whatever means available) to 
get to the point where we can usefully execute the jobs on-commit.

J
IRC: odyssey4me

On 1/11/17, 11:03 PM, "Sean M. Collins"  wrote:

OK - with https://review.openstack.org/#/c/418521/ we have at least a
working POC of what we can do.

The issue is that we're running into the Zuul timeout.

Depending on how quickly the AIO is built, we can get to the point where
we run the upgrade script[2].

However in some runs we don't get to the end of the AIO build[3].

So, the question is, how do we proceed? I'm not a real LXC expert but if
we could somehow cache stable builds of the LXC containers, so that
bootstrapping the AIO just means downloading and launching them, so that
we can use the majority of the Zuul runtime to execute the upgrade
script, that'd be great.

I know he have diskimage builder that does something sort of like this,
maybe we can do something similar for the LXC containers?


[1]: 
http://logs.openstack.org/21/418521/7/experimental/gate-openstack-ansible-openstack-ansible-upgrade-ubuntu-xenial-nv/6704087/console.html#_2017-01-11_05_13_16_114022
[2]: 
http://logs.openstack.org/21/418521/7/experimental/gate-openstack-ansible-openstack-ansible-upgrade-ubuntu-xenial-nv/6704087/console.html#_2017-01-11_05_13_24_895056
[3]: 
http://logs.openstack.org/21/418521/8/experimental/gate-openstack-ansible-openstack-ansible-upgrade-ubuntu-xenial-nv/ac09458/console.html#_2017-01-11_21_13_55_572404
--
Sean M. Collins

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-ansible-deployment] Periodic job in infra to test upgrades?

[Sean M. Collins]

OK - with https://review.openstack.org/#/c/418521/ we have at least a
working POC of what we can do.

The issue is that we're running into the Zuul timeout.

Depending on how quickly the AIO is built, we can get to the point where
we run the upgrade script[2].

However in some runs we don't get to the end of the AIO build[3].

So, the question is, how do we proceed? I'm not a real LXC expert but if
we could somehow cache stable builds of the LXC containers, so that
bootstrapping the AIO just means downloading and launching them, so that
we can use the majority of the Zuul runtime to execute the upgrade
script, that'd be great.

I know he have diskimage builder that does something sort of like this,
maybe we can do something similar for the LXC containers?


[1]: 
http://logs.openstack.org/21/418521/7/experimental/gate-openstack-ansible-openstack-ansible-upgrade-ubuntu-xenial-nv/6704087/console.html#_2017-01-11_05_13_16_114022
[2]: 
http://logs.openstack.org/21/418521/7/experimental/gate-openstack-ansible-openstack-ansible-upgrade-ubuntu-xenial-nv/6704087/console.html#_2017-01-11_05_13_24_895056
[3]: 
http://logs.openstack.org/21/418521/8/experimental/gate-openstack-ansible-openstack-ansible-upgrade-ubuntu-xenial-nv/ac09458/console.html#_2017-01-11_21_13_55_572404
-- 
Sean M. Collins

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-ansible-deployment] Periodic job in infra to test upgrades?

[Sean M. Collins]

Andy McCrae wrote:
> The work we can do now, which if you're able to help with would be really
> great, is:
> 
> Plan how we execute the actual job - at the moment the aio is run through a
> script (scripts/gate-check-commit.sh) - the upgrade test would have to hook
> into this (using the SCENARIO=upgrade) var, or we'd need to look into
> changing this method up entirely (role gates all use tox for example).
> 
> If the current method is sufficient we need to set the job up. Once we have
> a working scenario (regardless of time taken for the test), we can look
> into ways we can ensure this gets run, and what options we have.


Excellent. Thanks for all the info. I'm going to start poking around at
the gate-check-commit script and see if I can build up an AIO node, then
do the upgrade.

-- 
Sean M. Collins

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-ansible-deployment] Periodic job in infra to test upgrades?

[Andy McCrae]

>
>
> OK - is there any way that I can assist?
>
> What about the challenges I discussed in my initial mail related to
> long AIO build times etc..?
>
>
I think there are some options, but the idea we're going with at the moment
is to look into setting this up as a periodical - we're at about 1hr per
full AIO build, so i don't think we'll be able to (reliably) keep the gate
to under 1hr30 if we include an upgrade scenario - unless we have some
ideas around how we build the initial AIO build - that could cut down the
time.

That's mostly conjecture at this point though - since we don't have a test
scenario setup.

The work we can do now, which if you're able to help with would be really
great, is:

Plan how we execute the actual job - at the moment the aio is run through a
script (scripts/gate-check-commit.sh) - the upgrade test would have to hook
into this (using the SCENARIO=upgrade) var, or we'd need to look into
changing this method up entirely (role gates all use tox for example).

If the current method is sufficient we need to set the job up. Once we have
a working scenario (regardless of time taken for the test), we can look
into ways we can ensure this gets run, and what options we have.

Andy
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-ansible-deployment] Periodic job in infra to test upgrades?

[Sean M. Collins]

Andy McCrae wrote:
> Completely agree, the end goal is to have the full end-to-end testing in
> the openstack-ansible repository, along with the individual repo tests.
> We have an experimental job setup in
> project-config:
> (gate-openstack-ansible-openstack-ansible-upgrade-ubuntu-xenial-nv)
> although no concrete work has gone into adding this to the repo - so that
> one is pending.
> 
> The idea would be to run the upgrade scripts against an AIO built with
> stable/newton, and run tests after the upgrade - I'm hoping to see that
> land this cycle.


OK - is there any way that I can assist?

What about the challenges I discussed in my initial mail related to
long AIO build times etc..?


-- 
Sean M. Collins

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-ansible-deployment] Periodic job in infra to test upgrades?

[Andy McCrae]

>
>
> Ah OK I see now.
>
> I guess what I'm driving at, is how do we get automated coverage for the
> full end-to-end upgrade, so that issues[1] that I uncovered during manual
> testing start getting uncovered by automation?
>
> [1]: https://review.openstack.org/#/c/400847/


Completely agree, the end goal is to have the full end-to-end testing in
the openstack-ansible repository, along with the individual repo tests.
We have an experimental job setup in
project-config:
(gate-openstack-ansible-openstack-ansible-upgrade-ubuntu-xenial-nv)
although no concrete work has gone into adding this to the repo - so that
one is pending.

The idea would be to run the upgrade scripts against an AIO built with
stable/newton, and run tests after the upgrade - I'm hoping to see that
land this cycle.

Andy
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-ansible-deployment] Periodic job in infra to test upgrades?

[Sean M. Collins]

Andy McCrae wrote:
> Hi Sean,
> 
> 
> > OK - is this a job in Zuul ? Where can I find more information?
> >
> The test is essentially:
> 
> Deploy infra for testing
> Deploy other required services
> Deploy "stable/newton" version of project_name
> Run "master" version of project_name
> Run tests for project_name.
> 
> I'm not too sure what further detail you're looking for - but the tests are
> run from the "test.yml" play in the "tests" directory of each repository -
> and for upgrades we added a "_upgrade" boolean var that is
> set when the upgrade job is run via tox - so feel free to take a look there.


Ah OK I see now.

I guess what I'm driving at, is how do we get automated coverage for the
full end-to-end upgrade, so that issues[1] that I uncovered during manual
testing start getting uncovered by automation?

[1]: https://review.openstack.org/#/c/400847/


-- 
Sean M. Collins

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-ansible-deployment] Periodic job in infra to test upgrades?

[Andy McCrae]

Hi Sean,


> OK - is this a job in Zuul ? Where can I find more information?
>

They are Zuul jobs, and are currently non-voting (but should be passing -
we're waiting to ensure the tests are reliably passing before moving to
voting).

The test is essentially:

Deploy infra for testing
Deploy other required services
Deploy "stable/newton" version of project_name
Run "master" version of project_name
Run tests for project_name.

I'm not too sure what further detail you're looking for - but the tests are
run from the "test.yml" play in the "tests" directory of each repository -
and for upgrades we added a "_upgrade" boolean var that is
set when the upgrade job is run via tox - so feel free to take a look there.

Andy
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-ansible-deployment] Periodic job in infra to test upgrades?

[Sean M. Collins]

Andy McCrae wrote:
> Hi Sean,
> 
> I assume you're talking about OpenStack-Ansible (was OS-A-D before moving
> to the OpenStack namespace), if not, feel free to ignore my response!

Right.

> This is actually something we're working on - we have in role testing for
> some key OpenStack services (Nova, Neutron, Swift, Keystone, Cinder,
> Glance), and upgrade testing for the rabbitmq_server and galera_server
> roles.
> 
> This is a first pass effort, the testing right now deploys the Newton
> version of the service and then upgrades it and runs a functional test.
> This isn't perfect but should help ensure individual services can upgrade
> successfully - and can be built upon.

OK - is this a job in Zuul ? Where can I find more information?

> 
> The next step is to add an integrated build upgrade test, and we have plans
> to get that done (it's been an action item in our meetings for a few weeks,
> but hasn't landed just yet). If you have any questions or would like to get
> involved feel free to stop by and discuss in the #openstack-ansible channel
> on freenode.
> 

Thanks


-- 
Sean M. Collins

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-ansible-deployment] Periodic job in infra to test upgrades?

[Andy McCrae]

Hi Sean,

I assume you're talking about OpenStack-Ansible (was OS-A-D before moving
to the OpenStack namespace), if not, feel free to ignore my response!

This is actually something we're working on - we have in role testing for
some key OpenStack services (Nova, Neutron, Swift, Keystone, Cinder,
Glance), and upgrade testing for the rabbitmq_server and galera_server
roles.

This is a first pass effort, the testing right now deploys the Newton
version of the service and then upgrades it and runs a functional test.
This isn't perfect but should help ensure individual services can upgrade
successfully - and can be built upon.

The next step is to add an integrated build upgrade test, and we have plans
to get that done (it's been an action item in our meetings for a few weeks,
but hasn't landed just yet). If you have any questions or would like to get
involved feel free to stop by and discuss in the #openstack-ansible channel
on freenode.

As an aside, we have some upgrade documentation here:
http://docs.openstack.org/developer/openstack-ansible/upgrade-guide/index.html
which may be useful, but it's due for an overhaul in the near future.

Thanks,
Andy
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [os-ansible-deployment] Periodic job in infra to test upgrades?

[Sean M. Collins]

Hi,

A couple weeks ago I did some manual testing of upgrades from Liberty ->
Mitaka and Mitaka -> Newton using OSAD and a AIO node.

Whenever I hit an issue, I'd report it.

I'd be great if we could automate this work. The only challenge I can
think of is the fact that building up an AIO node, installing the base
version, then doing the upgrade takes quite a bit of time. I don't know
if periodic jobs have the same time constraints as jobs in the check and
gate queue though.

At least on my local machine, I sped up the process by snapshotting the
AIO box (disk and running memory) in VBox so that I could re-run the upgrade
quickly. Worst case perhaps set up a 3rd party CI system that could use
that trick? Just throwing ideas out there.

Thoughts?

-- 
Sean M. Collins

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev