Re: [openstack-dev] [tripleo] Updates on containerized undercloud

2018-03-25 Thread Emilien Macchi 
This is an update on what has been achieved the last month with the regard
of Containerized Undercloud efforts in TripleO:

## CI
- Running OVB (ovs-ha, fs001) with a containerized undercloud: it finally
works, with some workarounds, all work in progress.
Results can be seen here:
https://logs.rdoproject.org/56/542556/79/openstack-check/
gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/
Ze1390e6e0df54a88836d75316da4b206/console.txt.gz#_2018-03-24_06_07_23_171
List of workarounds/blockers:
* we need a new release of python-openstackclient that includes
https://review.openstack.org/#/c/553374/ and therefore we need
https://review.openstack.org/#/c/553026/
* container workflow to be finished (sbaker is on it) (in the meantime
we're loading envs in quickstart).
* masquerading workaround: https://review.openstack.org/#/c/553620 (long
term solution will be https://review.openstack.org/#/c/553427/ but still
WIP)
Once we clear the workarounds/blockers and have a clean / stable
deployment, we'll switch featureset001 (ovb-ha) to deploy a containerized
undercloud. The target was end of rocky-m1 and we still aim for it.

- Running an CI job that test upgrades from a non containerized undercloud
on Queens to a containerized undercloud on Rocky. Work is in progress and
can be monitored here: https://review.openstack.org/#/c/553633/
Special thanks to the Upgrade squads who help a lot on that front!

## Upgrades support
We said we would provide a way to upgrade a non containerized undercloud to
a containerized undercloud by rocky-m1 and we still aim for it.
This is a demo of an upgrade from Queens (non containerized) to Rocky
(containerized): https://youtu.be/5gLKL3YkC2c
We'll wait a bit for feedback from the demo and start documenting. Note
that most of the workflow remains the same as before (we still use
openstack undercloud upgrade command).
We'll also continue to push efforts to have this workflow tested by the CI
job in progress.

## Other items
- TripleO UI has been containerized.
- Routed networks is still in progress by Harald (we probably aim for
rocky-m2 now).
- We're investigating some way to validate than an upgrade to a
containerized undercloud worked fine (with Ansible?). More to come.
- Containerization of Tempest so we can run Tempest against a containerized
undercloud and also investigate how we could switch CI scenarios to be
deployed on one node.
- Port the TLS by default done in instack-undercloud.

Any feedback or help on testing is very welcome.
All efforts can be seen here: https://trello.com/b/nmGSNPoQ/containerized-
undercloud

Thanks everyone who helped in these efforts so far!

On Sun, Feb 18, 2018 at 10:18 AM, Emilien Macchi  wrote:

> This is an update on what has been achieved this week with the regard of
> Containerized Undercloud efforts in TripleO:
>
> TL;DR: really good efforts have been made and we can now deploy a full
> (multinode) overcloud in CI. OVB testing in progress and lot of remaining
> items!
>
> ## Bugfixes
> docker-registry: add missing firewall rules - https://review.openstack.
> org/#/c/545185/
> mistral-executor: mount /var/lib/mistral - https://review.openstack.
> org/#/c/545143/
> docker: configure group/user for deployment_user -
> https://review.openstack.org/#/c/544761/ + dependencies
> Fix PublicVirtualFixedIPs in envs - https://review.openstack.
> org/#/c/544744/
> Align zaqar max_messages_post_size with undercloud -
> https://review.openstack.org/#/c/544756/
> undercloud_post: fix subnet name - https://review.openstack.
> org/#/c/544587/
>
> ## CI
> We manage to run a containerized overcloud deployed by a containerized
> undercloud in CI, results can be seen here: https://review.
> openstack.org/#/c/542906/
> The job is running on featureser010 now (for testing purpose) but as James
> mentioned in the review, we won't switch this job to run a containerized
> undercloud. Note there is no impact on the job runtime.
> We'll need to properly deprecate the non-containerized undercloud first
> but we'll need to find a CI job that we can use for gating, so we avoid
> regression during the cycle.
> Now we're working on deploying featureset001 (ovb-ha), with TLS, net-iso,
> Ironic/Nova/Neutron (baremetal bits) from a containerized undercloud:
> https://review.openstack.org/#/c/542556/
> It's not working yet but we're working toward the blockers as they come
> during testing.
>
> # TLS Support
> All patches that were in progress have been merged, and now under testing
> in ovb-ha + containerized u/c (see above).
>
> # UI Support
> Work is still in progress, patches are ready for review, but some one them
> don't pass pep8 yet. We'll hopefully fix it soon.
>
> # Other items
> routed ctlplane networking: Harald is currently making progress on the
> items, some patches are ready for review.
> Create temp copy of tripleo-heat-templates before processing them: Bogdan
> is working on https://review.openstack.org/#/c/542875 - the patch is
> under

Re: [openstack-dev] [tripleo] Updates on containerized undercloud

2018-02-18 Thread Emilien Macchi 
This is an update on what has been achieved this week with the regard of
Containerized Undercloud efforts in TripleO:

TL;DR: really good efforts have been made and we can now deploy a full
(multinode) overcloud in CI. OVB testing in progress and lot of remaining
items!

## Bugfixes
docker-registry: add missing firewall rules -
https://review.openstack.org/#/c/545185/
mistral-executor: mount /var/lib/mistral -
https://review.openstack.org/#/c/545143/
docker: configure group/user for deployment_user -
https://review.openstack.org/#/c/544761/ + dependencies
Fix PublicVirtualFixedIPs in envs - https://review.openstack.org/#/c/544744/
Align zaqar max_messages_post_size with undercloud -
https://review.openstack.org/#/c/544756/
undercloud_post: fix subnet name - https://review.openstack.org/#/c/544587/

## CI
We manage to run a containerized overcloud deployed by a containerized
undercloud in CI, results can be seen here:
https://review.openstack.org/#/c/542906/
The job is running on featureser010 now (for testing purpose) but as James
mentioned in the review, we won't switch this job to run a containerized
undercloud. Note there is no impact on the job runtime.
We'll need to properly deprecate the non-containerized undercloud first but
we'll need to find a CI job that we can use for gating, so we avoid
regression during the cycle.
Now we're working on deploying featureset001 (ovb-ha), with TLS, net-iso,
Ironic/Nova/Neutron (baremetal bits) from a containerized undercloud:
https://review.openstack.org/#/c/542556/
It's not working yet but we're working toward the blockers as they come
during testing.

# TLS Support
All patches that were in progress have been merged, and now under testing
in ovb-ha + containerized u/c (see above).

# UI Support
Work is still in progress, patches are ready for review, but some one them
don't pass pep8 yet. We'll hopefully fix it soon.

# Other items
routed ctlplane networking: Harald is currently making progress on the
items, some patches are ready for review.
Create temp copy of tripleo-heat-templates before processing them: Bogdan
is working on https://review.openstack.org/#/c/542875 - the patch is under
review!
Upgrades: no work has been started so far but we'll probably discuss about
this topic during the PTG.

As usual please comment or add anything that I missed.
Thanks all for your help/reviews/efforts so far,

Emilien


On Tue, Feb 13, 2018 at 6:41 AM, Emilien Macchi  wrote:

>
>
> On Tue, Feb 13, 2018 at 2:40 AM, Harald Jensås  wrote:
>
>> On Fri, 2018-02-09 at 14:39 -0800, Emilien Macchi wrote:
>> > On Fri, Feb 9, 2018 at 2:30 PM, James Slagle 
>> > wrote:
>> > [...]
>> >
>> > > You may want to add an item for the routed ctlplane work that
>> > > landed
>> > > at the end of Queens. Afaik, that will need to be supported with
>> > > the
>> > > containerized undercloud.
>> >
>> > Done: https://trello.com/c/kFtIkto1/17-routed-ctlplane-networking
>> >
>>
>> Tanks Emilien,
>>
>>
>> I added several work items to the Trello card, and a few patches. Still
>> WiP.
>>
>> Do we have any CI that use containerized undercloud with actual Ironic
>> deployement? Or are they all using deployed-server?
>>
>> E.g do we have anything actually testing this type of change?
>>https://review.openstack.org/#/c/543582
>>
>> I belive that would have to be an ovb job with containerized undercloud?
>>
>
> I'm working on it since last week: https://trello.com/c/
> uLqbHTip/13-switch-other-jobs-to-run-a-containerized-undercloud
> But currently trying to make things stable again, we introduce regressions
> and this is high prio now.
> --
> Emilien Macchi
>



-- 
Emilien Macchi
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Updates on containerized undercloud

2018-02-13 Thread Emilien Macchi 
On Tue, Feb 13, 2018 at 2:40 AM, Harald Jensås  wrote:

> On Fri, 2018-02-09 at 14:39 -0800, Emilien Macchi wrote:
> > On Fri, Feb 9, 2018 at 2:30 PM, James Slagle 
> > wrote:
> > [...]
> >
> > > You may want to add an item for the routed ctlplane work that
> > > landed
> > > at the end of Queens. Afaik, that will need to be supported with
> > > the
> > > containerized undercloud.
> >
> > Done: https://trello.com/c/kFtIkto1/17-routed-ctlplane-networking
> >
>
> Tanks Emilien,
>
>
> I added several work items to the Trello card, and a few patches. Still
> WiP.
>
> Do we have any CI that use containerized undercloud with actual Ironic
> deployement? Or are they all using deployed-server?
>
> E.g do we have anything actually testing this type of change?
>https://review.openstack.org/#/c/543582
>
> I belive that would have to be an ovb job with containerized undercloud?
>

I'm working on it since last week:
https://trello.com/c/uLqbHTip/13-switch-other-jobs-to-run-a-containerized-undercloud
But currently trying to make things stable again, we introduce regressions
and this is high prio now.
-- 
Emilien Macchi
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Updates on containerized undercloud

2018-02-13 Thread Harald Jensås 
On Fri, 2018-02-09 at 14:39 -0800, Emilien Macchi wrote:
> On Fri, Feb 9, 2018 at 2:30 PM, James Slagle 
> wrote:
> [...]
> 
> > You may want to add an item for the routed ctlplane work that
> > landed
> > at the end of Queens. Afaik, that will need to be supported with
> > the
> > containerized undercloud.
> 
> Done: https://trello.com/c/kFtIkto1/17-routed-ctlplane-networking
> 

Tanks Emilien,


I added several work items to the Trello card, and a few patches. Still
WiP.

Do we have any CI that use containerized undercloud with actual Ironic
deployement? Or are they all using deployed-server?

E.g do we have anything actually testing this type of change?
   https://review.openstack.org/#/c/543582

I belive that would have to be an ovb job with containerized undercloud?

-- 
| Harald Jensås
| hjensas:irc

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Updates on containerized undercloud

2018-02-09 Thread Emilien Macchi 
On Fri, Feb 9, 2018 at 2:30 PM, James Slagle  wrote:
[...]

You may want to add an item for the routed ctlplane work that landed
> at the end of Queens. Afaik, that will need to be supported with the
> containerized undercloud.
>

Done: https://trello.com/c/kFtIkto1/17-routed-ctlplane-networking

Thanks,
-- 
Emilien Macchi
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Updates on containerized undercloud

2018-02-09 Thread James Slagle 
On Fri, Feb 9, 2018 at 5:02 PM, Emilien Macchi  wrote:
> Quite a lot of progress has been made over the last months (and days), so I
> found useful to share an update on where we are with the efforts on
> containerized undercloud.
>
> ## CI efforts
>
> - tripleo-ci-centos-7-undercloud-containers job has been reworked to use the
> "undercloud install" interface. Job is green and will probably start voting
> during the next days. See https://review.openstack.org/#/c/517445/ and
> https://review.openstack.org/#/c/517444/.
> - we're looking at switching some other CI jobs (maybe one to start) to
> deploy a containerized undercloud, and then deploy an overcloud (probably
> featureset010). We have a few blockers but we're working on it. It's mostly
> the overcloud_prep that fails:
> http://logs.openstack.org/06/542906/1/check/tripleo-ci-centos-7-containers-multinode/0b18c49/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz#_2018-02-09_17_07_27
> - because of that effort, we're also taking an opportunity to refactor
> tripleo-quickstart-extras roles to be more "standard" for both undercloud
> and overcloud (example, renaming overcloud-prep-containers to
> prep-containers, etc). We'll need help from TripleO CI squad probably.
> - Chandan will help us to run validate-tempest in
> tripleo-ci-centos-7-undercloud-containers so we can have some actual testing
> for this job, since no overcloud is deployed.
>
> ## Feature parity
>
> - TLS work is ongoing.
> - TripleO UI containerization is ongoing.
> - Nova join support is targeted for Rocky
> - Upgrade workflow is under investigation. We'll work on re-using the
> upgrade_tasks in THT to upgrade a non-containerized undercloud (Queens) to a
> containerized undercloud (Rocky) like we did between Ocata and Pike with the
> upgrade_tasks. We'll actually re-use the same code but will have to change
> the undercloud upgrade workflow in tripleoclient.

You may want to add an item for the routed ctlplane work that landed
at the end of Queens. Afaik, that will need to be supported with the
containerized undercloud.

-- 
-- James Slagle
--

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Updates on containerized undercloud

2018-02-09 Thread Emilien Macchi 
On Fri, Feb 9, 2018 at 2:02 PM, Emilien Macchi  wrote:

> Quite a lot of progress has been made over the last months (and days), so
> I found useful to share an update on where we are with the efforts on
> containerized undercloud.
>
> ## CI efforts
>
> - tripleo-ci-centos-7-undercloud-containers job has been reworked to use
> the "undercloud install" interface. Job is green and will probably start
> voting during the next days. See https://review.openstack.org/#/c/517445/
> and https://review.openstack.org/#/c/517444/.
> - we're looking at switching some other CI jobs (maybe one to start) to
> deploy a containerized undercloud, and then deploy an overcloud (probably
> featureset010). We have a few blockers but we're working on it. It's mostly
> the overcloud_prep that fails:
> http://logs.openstack.org/06/542906/1/check/tripleo-ci-
> centos-7-containers-multinode/0b18c49/logs/undercloud/home/
> zuul/overcloud_prep_containers.log.txt.gz#_2018-02-09_17_07_27
> - because of that effort, we're also taking an opportunity to refactor
> tripleo-quickstart-extras roles to be more "standard" for both undercloud
> and overcloud (example, renaming overcloud-prep-containers to
> prep-containers, etc). We'll need help from TripleO CI squad probably.
> - Chandan will help us to run validate-tempest in 
> tripleo-ci-centos-7-undercloud-containers
> so we can have some actual testing for this job, since no overcloud is
> deployed.
>
> ## Feature parity
>
> - TLS work is ongoing.
> - TripleO UI containerization is ongoing.
> - Nova join support is targeted for Rocky
> - Upgrade workflow is under investigation. We'll work on re-using the
> upgrade_tasks in THT to upgrade a non-containerized undercloud (Queens) to
> a containerized undercloud (Rocky) like we did between Ocata and Pike with
> the upgrade_tasks. We'll actually re-use the same code but will have to
> change the undercloud upgrade workflow in tripleoclient.
>
> That highlights the current efforts, if you have any question, need more
> specific or just any feedback, please go ahead.
> At the PTG, we'll discuss about some technical details and hope to move
> forward with this nice feature during Rocky cycle.
>
> Thanks,
> --
> Emilien Macchi
>

I forgot to mention, but people working on this topic have been using
Trello to collaborate:
https://trello.com/b/nmGSNPoQ/containerized-undercloud

To keep things in the open, here's the link and anyone is free to
participate.

Thanks,
-- 
Emilien Macchi
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] Updates on containerized undercloud

2018-02-09 Thread Emilien Macchi 
Quite a lot of progress has been made over the last months (and days), so I
found useful to share an update on where we are with the efforts on
containerized undercloud.

## CI efforts

- tripleo-ci-centos-7-undercloud-containers job has been reworked to use
the "undercloud install" interface. Job is green and will probably start
voting during the next days. See https://review.openstack.org/#/c/517445/
and https://review.openstack.org/#/c/517444/.
- we're looking at switching some other CI jobs (maybe one to start) to
deploy a containerized undercloud, and then deploy an overcloud (probably
featureset010). We have a few blockers but we're working on it. It's mostly
the overcloud_prep that fails:
http://logs.openstack.org/06/542906/1/check/tripleo-ci-centos-7-containers-multinode/0b18c49/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz#_2018-02-09_17_07_27
- because of that effort, we're also taking an opportunity to refactor
tripleo-quickstart-extras roles to be more "standard" for both undercloud
and overcloud (example, renaming overcloud-prep-containers to
prep-containers, etc). We'll need help from TripleO CI squad probably.
- Chandan will help us to run validate-tempest in
tripleo-ci-centos-7-undercloud-containers so we can have some actual
testing for this job, since no overcloud is deployed.

## Feature parity

- TLS work is ongoing.
- TripleO UI containerization is ongoing.
- Nova join support is targeted for Rocky
- Upgrade workflow is under investigation. We'll work on re-using the
upgrade_tasks in THT to upgrade a non-containerized undercloud (Queens) to
a containerized undercloud (Rocky) like we did between Ocata and Pike with
the upgrade_tasks. We'll actually re-use the same code but will have to
change the undercloud upgrade workflow in tripleoclient.

That highlights the current efforts, if you have any question, need more
specific or just any feedback, please go ahead.
At the PTG, we'll discuss about some technical details and hope to move
forward with this nice feature during Rocky cycle.

Thanks,
-- 
Emilien Macchi
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev