Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Tony Breeds]

On Sun, Apr 15, 2018 at 07:24:58PM -0700, Emilien Macchi wrote:

> This patch: https://review.openstack.org/#/c/561377 is deploying Docker and
> Docker Registry v2 *before* containers deployment in the docker_steps.
> It's using the external_deploy_tasks interface that runs right after the
> host_prep_tasks, so still before starting containers.
> 
> It's using the Ansible role that was prototyped on Friday, please take a
> look and raise any concern.
> Now I would like to investigate how we can run container workflows between
> the deployment and docker and containers deployments.

This looks pretty good to me and if I understand correctly as it's
creating a v2 registry then we'll get manifest list images (for
multi-arch) by default which is a massive win for me.

Thanks Emilien
> -- 
> Emilien Macchi

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Bogdan Dobrelya]

On 4/20/18 8:56 PM, Emilien Macchi wrote:
So the role has proven to be useful and we're now sure that we need it 
to deploy a container registry before any container in the deployment, 
which means we can't use the puppet code anymore for this service.


I propose that we move the role to OpenStack:
https://review.openstack.org/#/c/563197/
https://review.openstack.org/#/c/563198/
https://review.openstack.org/#/c/563200/

So we can properly peer review and gate the new role.

In the meantime, we continue to work on the new workflow.
Thanks,

On Sun, Apr 15, 2018 at 7:24 PM, Emilien Macchi > wrote:


On Fri, Apr 13, 2018 at 5:58 PM, Emilien Macchi mailto:emil...@redhat.com>> wrote:


A bit of progress today, I prototyped an Ansible role for that
purpose:
https://github.com/EmilienM/ansible-role-container-registry


The next step is, I'm going to investigate if we can deploy
Docker and Docker Distribution (to deploy the registry v2) via
the existing composable services in THT by
using external_deploy_tasks maybe (or another interface).
The idea is really to have the registry ready before actually
deploying the undercloud containers, so we can modify them in
the middle (run container-check in our case).


This patch: https://review.openstack.org/#/c/561377
 is deploying Docker and
Docker Registry v2 *before* containers deployment in the docker_steps.
It's using the external_deploy_tasks interface that runs right after
the host_prep_tasks, so still before starting containers.

It's using the Ansible role that was prototyped on Friday, please
take a look and raise any concern.


I have only a question if we could reuse something instead if already 
had been solved in projects like Kolla? Otherwise it's LGTM.



Now I would like to investigate how we can run container workflows
between the deployment and docker and containers deployments.
-- 
Emilien Macchi





--
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Emilien Macchi]

So the role has proven to be useful and we're now sure that we need it to
deploy a container registry before any container in the deployment, which
means we can't use the puppet code anymore for this service.

I propose that we move the role to OpenStack:
https://review.openstack.org/#/c/563197/
https://review.openstack.org/#/c/563198/
https://review.openstack.org/#/c/563200/

So we can properly peer review and gate the new role.

In the meantime, we continue to work on the new workflow.
Thanks,

On Sun, Apr 15, 2018 at 7:24 PM, Emilien Macchi  wrote:

> On Fri, Apr 13, 2018 at 5:58 PM, Emilien Macchi 
> wrote:
>
>>
>> A bit of progress today, I prototyped an Ansible role for that purpose:
>> https://github.com/EmilienM/ansible-role-container-registry
>>
>> The next step is, I'm going to investigate if we can deploy Docker and
>> Docker Distribution (to deploy the registry v2) via the existing composable
>> services in THT by using external_deploy_tasks maybe (or another interface).
>> The idea is really to have the registry ready before actually deploying
>> the undercloud containers, so we can modify them in the middle (run
>> container-check in our case).
>>
>
> This patch: https://review.openstack.org/#/c/561377 is deploying Docker
> and Docker Registry v2 *before* containers deployment in the docker_steps.
> It's using the external_deploy_tasks interface that runs right after the
> host_prep_tasks, so still before starting containers.
>
> It's using the Ansible role that was prototyped on Friday, please take a
> look and raise any concern.
> Now I would like to investigate how we can run container workflows between
> the deployment and docker and containers deployments.
> --
> Emilien Macchi
>



-- 
Emilien Macchi
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Emilien Macchi]

On Fri, Apr 13, 2018 at 5:58 PM, Emilien Macchi  wrote:

>
> A bit of progress today, I prototyped an Ansible role for that purpose:
> https://github.com/EmilienM/ansible-role-container-registry
>
> The next step is, I'm going to investigate if we can deploy Docker and
> Docker Distribution (to deploy the registry v2) via the existing composable
> services in THT by using external_deploy_tasks maybe (or another interface).
> The idea is really to have the registry ready before actually deploying
> the undercloud containers, so we can modify them in the middle (run
> container-check in our case).
>

This patch: https://review.openstack.org/#/c/561377 is deploying Docker and
Docker Registry v2 *before* containers deployment in the docker_steps.
It's using the external_deploy_tasks interface that runs right after the
host_prep_tasks, so still before starting containers.

It's using the Ansible role that was prototyped on Friday, please take a
look and raise any concern.
Now I would like to investigate how we can run container workflows between
the deployment and docker and containers deployments.
-- 
Emilien Macchi
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Emilien Macchi]

On Wed, Apr 11, 2018 at 3:38 PM, Steve Baker  wrote:
>
> - If agreed, we'll create a new Ansible role called 
> ansible-role-container-registry
> that for now will deploy exactly what we have in TripleO, without extra
> feature.
>
> +1
>

A bit of progress today, I prototyped an Ansible role for that purpose:
https://github.com/EmilienM/ansible-role-container-registry

The next step is, I'm going to investigate if we can deploy Docker and
Docker Distribution (to deploy the registry v2) via the existing composable
services in THT by using external_deploy_tasks maybe (or another interface).
The idea is really to have the registry ready before actually deploying the
undercloud containers, so we can modify them in the middle (run
container-check in our case).
-- 
Emilien Macchi
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Emilien Macchi]

On Thu, Apr 12, 2018 at 1:16 AM, Bogdan Dobrelya 
wrote:
[...]

> Deploy own registry as part of UC deployment or use external. For
>> instance, for production use I would like to have a cluster of 3-5
>> registries with HAProxy in front to speed up 1k nodes deployments.
>>
>
> Note that this implies HA undercloud as well. Although, given that HA
> undercloud is goodness indeed, I would *not* invest time into reliable
> container registry deployment architecture for undercloud as we'll have it
> for free, once kubernetes/openshift control plane for openstack becomes
> adopted. There is a very strong notion of build pipelines, reliable
> containers registries et al.


Right. HA undercloud is out of context now.
-- 
Emilien Macchi
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Emilien Macchi]

On Thu, Apr 12, 2018 at 1:08 AM, Sergii Golovatiuk 
wrote:
[...]

> > One of the great outcomes of this blueprint is that in Rocky, the
> operator
> > won't have to run all the "openstack overcloud container" commands to
> > prepare the container registry and upload the containers. Indeed, it'll
> be
> > driven by Heat and Mistral mostly.
>
> I am trying to think as operator and it's very similar to 'openstack
> container' which is swift. So it might be confusing I guess.


"openstack overcloud container" was already in Pike, Queens for your
information.

[...]

> > - We need a tool to update containers from this registry and *before*
> > deploying them. We already have this tool in place in our CI for the
> > overcloud (see [3] and [4]). Now we need a similar thing for the
> undercloud.
>
> I would use external registry in this case. Quay.io might be a good
> choice to have rock solid simplicity. It might not be good for CI as
> requires very strong connectivity but it should be sufficient for
> developers.
>

No. We'll use docker-distribution for now, and will consider more support
in the future but what we want right now is parity.

[...]
-- 
Emilien Macchi
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Bogdan Dobrelya]

On 4/12/18 12:38 AM, Steve Baker wrote:



On 11/04/18 12:50, Emilien Macchi wrote:

Greetings,

Steve Baker and I had a quick chat today about the work that is being 
done around containers workflow in Rocky cycle.


If you're not familiar with the topic, I suggest to first read the 
blueprint to understand the context here:

https://blueprints.launchpad.net/tripleo/+spec/container-prepare-workflow

One of the great outcomes of this blueprint is that in Rocky, the 
operator won't have to run all the "openstack overcloud container" 
commands to prepare the container registry and upload the containers. 
Indeed, it'll be driven by Heat and Mistral mostly.
But today our discussion extended on 2 uses-cases that we're going to 
explore and find how we can address them:
1) I'm a developer and want to deploy a containerized undercloud with 
customized containers (more or less related to the all-in-one 
discussions on another thread [1]).
2) I'm submitting a patch in tripleo-common (let's say a workflow) and 
need my patch to be tested when the undercloud is containerized (see 
[2] for an excellent example).


I'm fairly sure the only use cases for this will be developer or CI 
based. I think we need to be strongly encouraging image modifications 
for production deployments to go through some kind of image building 
pipeline. See Next Steps below for the implications of this.



Both cases would require additional things:
- The container registry needs to be deployed *before* actually 
installing the undercloud.
- We need a tool to update containers from this registry and *before* 
deploying them. We already have this tool in place in our CI for the 
overcloud (see [3] and [4]). Now we need a similar thing for the 
undercloud.


One problem I see is that we use roles and environment files to filter 
the images to be pulled/modified/uploaded. Now we would need to assemble 
a list of undercloud *and* overcloud environments, and build some kind 
of aggregate role data for both. This would need to happen before the 
undercloud is even deployed, which is quite a different order from what 
quickstart does currently.


Either that or we do no image filtering and just process every image 
regardless of whether it will be used.




Next steps:
- Agree that we need to deploy the container-registry before the 
undercloud.
- If agreed, we'll create a new Ansible role called 
ansible-role-container-registry that for now will deploy exactly what 
we have in TripleO, without extra feature.

+1
- Drive the playbook runtime from tripleoclient to bootstrap the 
container registry (which of course could be disabled in undercloud.conf).
tripleoclient could switch to using this role instead of puppet-tripleo 
to install the registry, however since the only use-cases we have are 
dev/CI driven I wonder if quickstart/infrared can just invoke the role 
when required, before tripleoclient is involved.


- Create another Ansible role that would re-use container-check tool 
but the idea is to provide a role to modify containers when needed, 
and we could also control it from tripleoclient. The role would be 
using the ContainerImagePrepare parameter, which Steve is working on 
right now.


Since the use cases are all upstream CI/dev I do wonder if we should 
just have a dedicated container-check 
 role inside 
tripleo-quickstart-extras which can continue to use the script[3] or 
whatever. Keeping the logic in quickstart will remove the temptation to 
use it instead of a proper image build pipeline for production deployments.


+1 to put it in quickstart-extras to "hide" it from the production use 
cases.




Alternatively it could still be a standalone role which quickstart 
invokes, just to accommodate development workflows which don't use 
quickstart.



Feedback is welcome, thanks.

[1] All-In-One thread: 
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128900.html
[2] Bug report when undercloud is containeirzed 
https://bugs.launchpad.net/tripleo/+bug/1762422
[3] Tool to update containers if needed: 
https://github.com/imain/container-check
[4] Container-check running in TripleO CI: 
https://review.openstack.org/#/c/558885/ and 
https://review.openstack.org/#/c/529399/

--
Emilien Macchi


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Developmen

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Bogdan Dobrelya]

On 4/12/18 12:38 AM, Steve Baker wrote:



On 11/04/18 12:50, Emilien Macchi wrote:

Greetings,

Steve Baker and I had a quick chat today about the work that is being 
done around containers workflow in Rocky cycle.


If you're not familiar with the topic, I suggest to first read the 
blueprint to understand the context here:

https://blueprints.launchpad.net/tripleo/+spec/container-prepare-workflow

One of the great outcomes of this blueprint is that in Rocky, the 
operator won't have to run all the "openstack overcloud container" 
commands to prepare the container registry and upload the containers. 
Indeed, it'll be driven by Heat and Mistral mostly.
But today our discussion extended on 2 uses-cases that we're going to 
explore and find how we can address them:
1) I'm a developer and want to deploy a containerized undercloud with 
customized containers (more or less related to the all-in-one 
discussions on another thread [1]).
2) I'm submitting a patch in tripleo-common (let's say a workflow) and 
need my patch to be tested when the undercloud is containerized (see 
[2] for an excellent example).


I'm fairly sure the only use cases for this will be developer or CI 
based. I think we need to be strongly encouraging image modifications 
for production deployments to go through some kind of image building 
pipeline. See Next Steps below for the implications of this.



Both cases would require additional things:
- The container registry needs to be deployed *before* actually 
installing the undercloud.
- We need a tool to update containers from this registry and *before* 
deploying them. We already have this tool in place in our CI for the 
overcloud (see [3] and [4]). Now we need a similar thing for the 
undercloud.


One problem I see is that we use roles and environment files to filter 
the images to be pulled/modified/uploaded. Now we would need to assemble 
a list of undercloud *and* overcloud environments, and build some kind 
of aggregate role data for both. This would need to happen before the 
undercloud is even deployed, which is quite a different order from what 
quickstart does currently.


Either that or we do no image filtering and just process every image 
regardless of whether it will be used.




Next steps:
- Agree that we need to deploy the container-registry before the 
undercloud.
- If agreed, we'll create a new Ansible role called 
ansible-role-container-registry that for now will deploy exactly what 
we have in TripleO, without extra feature.

+1
- Drive the playbook runtime from tripleoclient to bootstrap the 
container registry (which of course could be disabled in undercloud.conf).
tripleoclient could switch to using this role instead of puppet-tripleo 
to install the registry, however since the only use-cases we have are 
dev/CI driven I wonder if quickstart/infrared can just invoke the role 
when required, before tripleoclient is involved.


Please let's do that for tripleoclient and only make quickstart and 
other tools to invoke commands. We should keep being close to what users 
would do, which is only issuing client commands.




- Create another Ansible role that would re-use container-check tool 
but the idea is to provide a role to modify containers when needed, 
and we could also control it from tripleoclient. The role would be 
using the ContainerImagePrepare parameter, which Steve is working on 
right now.


Since the use cases are all upstream CI/dev I do wonder if we should 
just have a dedicated container-check 
 role inside 
tripleo-quickstart-extras which can continue to use the script[3] or 
whatever. Keeping the logic in quickstart will remove the temptation to 
use it instead of a proper image build pipeline for production deployments.


Alternatively it could still be a standalone role which quickstart 
invokes, just to accommodate development workflows which don't use 
quickstart.



Feedback is welcome, thanks.

[1] All-In-One thread: 
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128900.html
[2] Bug report when undercloud is containeirzed 
https://bugs.launchpad.net/tripleo/+bug/1762422
[3] Tool to update containers if needed: 
https://github.com/imain/container-check
[4] Container-check running in TripleO CI: 
https://review.openstack.org/#/c/558885/ and 
https://review.openstack.org/#/c/529399/

--
Emilien Macchi


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Best regards,
Bogdan Dobrely

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Bogdan Dobrelya]

On 4/12/18 10:08 AM, Sergii Golovatiuk wrote:

Hi,

Thank you very much for brining up this topic.

On Wed, Apr 11, 2018 at 2:50 AM, Emilien Macchi  wrote:

Greetings,

Steve Baker and I had a quick chat today about the work that is being done
around containers workflow in Rocky cycle.

If you're not familiar with the topic, I suggest to first read the blueprint
to understand the context here:
https://blueprints.launchpad.net/tripleo/+spec/container-prepare-workflow

One of the great outcomes of this blueprint is that in Rocky, the operator
won't have to run all the "openstack overcloud container" commands to
prepare the container registry and upload the containers. Indeed, it'll be
driven by Heat and Mistral mostly.


I am trying to think as operator and it's very similar to 'openstack
container' which is swift. So it might be confusing I guess.



But today our discussion extended on 2 uses-cases that we're going to
explore and find how we can address them:
1) I'm a developer and want to deploy a containerized undercloud with
customized containers (more or less related to the all-in-one discussions on
another thread [1]).
2) I'm submitting a patch in tripleo-common (let's say a workflow) and need
my patch to be tested when the undercloud is containerized (see [2] for an
excellent example).


That's very nice initiative.


Both cases would require additional things:
- The container registry needs to be deployed *before* actually installing
the undercloud.
- We need a tool to update containers from this registry and *before*
deploying them. We already have this tool in place in our CI for the
overcloud (see [3] and [4]). Now we need a similar thing for the undercloud.


I would use external registry in this case. Quay.io might be a good
choice to have rock solid simplicity. It might not be good for CI as
requires very strong connectivity but it should be sufficient for
developers.


Next steps:
- Agree that we need to deploy the container-registry before the undercloud.
- If agreed, we'll create a new Ansible role called
ansible-role-container-registry that for now will deploy exactly what we
have in TripleO, without extra feature.


Deploy own registry as part of UC deployment or use external. For
instance, for production use I would like to have a cluster of 3-5
registries with HAProxy in front to speed up 1k nodes deployments.


Note that this implies HA undercloud as well. Although, given that HA 
undercloud is goodness indeed, I would *not* invest time into reliable 
container registry deployment architecture for undercloud as we'll have 
it for free, once kubernetes/openshift control plane for openstack 
becomes adopted. There is a very strong notion of build pipelines, 
reliable containers registries et al.





- Drive the playbook runtime from tripleoclient to bootstrap the container
registry (which of course could be disabled in undercloud.conf).
- Create another Ansible role that would re-use container-check tool but the
idea is to provide a role to modify containers when needed, and we could
also control it from tripleoclient. The role would be using the
ContainerImagePrepare parameter, which Steve is working on right now.

Feedback is welcome, thanks.

[1] All-In-One thread:
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128900.html
[2] Bug report when undercloud is containeirzed
https://bugs.launchpad.net/tripleo/+bug/1762422
[3] Tool to update containers if needed:
https://github.com/imain/container-check
[4] Container-check running in TripleO CI:
https://review.openstack.org/#/c/558885/ and
https://review.openstack.org/#/c/529399/
--
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev








--
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Bogdan Dobrelya]

On 4/12/18 12:38 AM, Steve Baker wrote:



On 11/04/18 12:50, Emilien Macchi wrote:

Greetings,

Steve Baker and I had a quick chat today about the work that is being 
done around containers workflow in Rocky cycle.


If you're not familiar with the topic, I suggest to first read the 
blueprint to understand the context here:

https://blueprints.launchpad.net/tripleo/+spec/container-prepare-workflow

One of the great outcomes of this blueprint is that in Rocky, the 
operator won't have to run all the "openstack overcloud container" 
commands to prepare the container registry and upload the containers. 
Indeed, it'll be driven by Heat and Mistral mostly.
But today our discussion extended on 2 uses-cases that we're going to 
explore and find how we can address them:
1) I'm a developer and want to deploy a containerized undercloud with 
customized containers (more or less related to the all-in-one 
discussions on another thread [1]).
2) I'm submitting a patch in tripleo-common (let's say a workflow) and 
need my patch to be tested when the undercloud is containerized (see 
[2] for an excellent example).


I'm fairly sure the only use cases for this will be developer or CI 
based. I think we need to be strongly encouraging image modifications 
for production deployments to go through some kind of image building 
pipeline. See Next Steps below for the implications of this.


Yes, this. I would love to see container-check tool improving CI and dev
experience and would not happy to see it as a blessed part of the 
product architecture. Containers should be immutable and nothing should 
be mutated runtime, like updating packages et al.





Both cases would require additional things:
- The container registry needs to be deployed *before* actually 
installing the undercloud.
- We need a tool to update containers from this registry and *before* 
deploying them. We already have this tool in place in our CI for the 
overcloud (see [3] and [4]). Now we need a similar thing for the 
undercloud.


One problem I see is that we use roles and environment files to filter 
the images to be pulled/modified/uploaded. Now we would need to assemble 
a list of undercloud *and* overcloud environments, and build some kind 
of aggregate role data for both. This would need to happen before the 
undercloud is even deployed, which is quite a different order from what 
quickstart does currently.


Either that or we do no image filtering and just process every image 
regardless of whether it will be used.




Next steps:
- Agree that we need to deploy the container-registry before the 
undercloud.
- If agreed, we'll create a new Ansible role called 
ansible-role-container-registry that for now will deploy exactly what 
we have in TripleO, without extra feature.

+1
- Drive the playbook runtime from tripleoclient to bootstrap the 
container registry (which of course could be disabled in undercloud.conf).
tripleoclient could switch to using this role instead of puppet-tripleo 
to install the registry, however since the only use-cases we have are 
dev/CI driven I wonder if quickstart/infrared can just invoke the role 
when required, before tripleoclient is involved.


- Create another Ansible role that would re-use container-check tool 
but the idea is to provide a role to modify containers when needed, 
and we could also control it from tripleoclient. The role would be 
using the ContainerImagePrepare parameter, which Steve is working on 
right now.


Since the use cases are all upstream CI/dev I do wonder if we should 
just have a dedicated container-check 
 role inside 
tripleo-quickstart-extras which can continue to use the script[3] or 
whatever. Keeping the logic in quickstart will remove the temptation to 
use it instead of a proper image build pipeline for production deployments.


Alternatively it could still be a standalone role which quickstart 
invokes, just to accommodate development workflows which don't use 
quickstart.



Feedback is welcome, thanks.

[1] All-In-One thread: 
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128900.html
[2] Bug report when undercloud is containeirzed 
https://bugs.launchpad.net/tripleo/+bug/1762422
[3] Tool to update containers if needed: 
https://github.com/imain/container-check
[4] Container-check running in TripleO CI: 
https://review.openstack.org/#/c/558885/ and 
https://review.openstack.org/#/c/529399/

--
Emilien Macchi


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Sergii Golovatiuk]

Hi,

Thank you very much for brining up this topic.

On Wed, Apr 11, 2018 at 2:50 AM, Emilien Macchi  wrote:
> Greetings,
>
> Steve Baker and I had a quick chat today about the work that is being done
> around containers workflow in Rocky cycle.
>
> If you're not familiar with the topic, I suggest to first read the blueprint
> to understand the context here:
> https://blueprints.launchpad.net/tripleo/+spec/container-prepare-workflow
>
> One of the great outcomes of this blueprint is that in Rocky, the operator
> won't have to run all the "openstack overcloud container" commands to
> prepare the container registry and upload the containers. Indeed, it'll be
> driven by Heat and Mistral mostly.

I am trying to think as operator and it's very similar to 'openstack
container' which is swift. So it might be confusing I guess.

>
> But today our discussion extended on 2 uses-cases that we're going to
> explore and find how we can address them:
> 1) I'm a developer and want to deploy a containerized undercloud with
> customized containers (more or less related to the all-in-one discussions on
> another thread [1]).
> 2) I'm submitting a patch in tripleo-common (let's say a workflow) and need
> my patch to be tested when the undercloud is containerized (see [2] for an
> excellent example).

That's very nice initiative.

> Both cases would require additional things:
> - The container registry needs to be deployed *before* actually installing
> the undercloud.
> - We need a tool to update containers from this registry and *before*
> deploying them. We already have this tool in place in our CI for the
> overcloud (see [3] and [4]). Now we need a similar thing for the undercloud.

I would use external registry in this case. Quay.io might be a good
choice to have rock solid simplicity. It might not be good for CI as
requires very strong connectivity but it should be sufficient for
developers.

> Next steps:
> - Agree that we need to deploy the container-registry before the undercloud.
> - If agreed, we'll create a new Ansible role called
> ansible-role-container-registry that for now will deploy exactly what we
> have in TripleO, without extra feature.

Deploy own registry as part of UC deployment or use external. For
instance, for production use I would like to have a cluster of 3-5
registries with HAProxy in front to speed up 1k nodes deployments.

> - Drive the playbook runtime from tripleoclient to bootstrap the container
> registry (which of course could be disabled in undercloud.conf).
> - Create another Ansible role that would re-use container-check tool but the
> idea is to provide a role to modify containers when needed, and we could
> also control it from tripleoclient. The role would be using the
> ContainerImagePrepare parameter, which Steve is working on right now.
>
> Feedback is welcome, thanks.
>
> [1] All-In-One thread:
> http://lists.openstack.org/pipermail/openstack-dev/2018-March/128900.html
> [2] Bug report when undercloud is containeirzed
> https://bugs.launchpad.net/tripleo/+bug/1762422
> [3] Tool to update containers if needed:
> https://github.com/imain/container-check
> [4] Container-check running in TripleO CI:
> https://review.openstack.org/#/c/558885/ and
> https://review.openstack.org/#/c/529399/
> --
> Emilien Macchi
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Best Regards,
Sergii Golovatiuk

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Steve Baker]

On 11/04/18 12:50, Emilien Macchi wrote:

Greetings,

Steve Baker and I had a quick chat today about the work that is being 
done around containers workflow in Rocky cycle.


If you're not familiar with the topic, I suggest to first read the 
blueprint to understand the context here:

https://blueprints.launchpad.net/tripleo/+spec/container-prepare-workflow

One of the great outcomes of this blueprint is that in Rocky, the 
operator won't have to run all the "openstack overcloud container" 
commands to prepare the container registry and upload the containers. 
Indeed, it'll be driven by Heat and Mistral mostly.
But today our discussion extended on 2 uses-cases that we're going to 
explore and find how we can address them:
1) I'm a developer and want to deploy a containerized undercloud with 
customized containers (more or less related to the all-in-one 
discussions on another thread [1]).
2) I'm submitting a patch in tripleo-common (let's say a workflow) and 
need my patch to be tested when the undercloud is containerized (see 
[2] for an excellent example).


I'm fairly sure the only use cases for this will be developer or CI 
based. I think we need to be strongly encouraging image modifications 
for production deployments to go through some kind of image building 
pipeline. See Next Steps below for the implications of this.



Both cases would require additional things:
- The container registry needs to be deployed *before* actually 
installing the undercloud.
- We need a tool to update containers from this registry and *before* 
deploying them. We already have this tool in place in our CI for the 
overcloud (see [3] and [4]). Now we need a similar thing for the 
undercloud.


One problem I see is that we use roles and environment files to filter 
the images to be pulled/modified/uploaded. Now we would need to assemble 
a list of undercloud *and* overcloud environments, and build some kind 
of aggregate role data for both. This would need to happen before the 
undercloud is even deployed, which is quite a different order from what 
quickstart does currently.


Either that or we do no image filtering and just process every image 
regardless of whether it will be used.




Next steps:
- Agree that we need to deploy the container-registry before the 
undercloud.
- If agreed, we'll create a new Ansible role called 
ansible-role-container-registry that for now will deploy exactly what 
we have in TripleO, without extra feature.

+1
- Drive the playbook runtime from tripleoclient to bootstrap the 
container registry (which of course could be disabled in undercloud.conf).
tripleoclient could switch to using this role instead of puppet-tripleo 
to install the registry, however since the only use-cases we have are 
dev/CI driven I wonder if quickstart/infrared can just invoke the role 
when required, before tripleoclient is involved.


- Create another Ansible role that would re-use container-check tool 
but the idea is to provide a role to modify containers when needed, 
and we could also control it from tripleoclient. The role would be 
using the ContainerImagePrepare parameter, which Steve is working on 
right now.


Since the use cases are all upstream CI/dev I do wonder if we should 
just have a dedicated container-check 
 role inside 
tripleo-quickstart-extras which can continue to use the script[3] or 
whatever. Keeping the logic in quickstart will remove the temptation to 
use it instead of a proper image build pipeline for production deployments.


Alternatively it could still be a standalone role which quickstart 
invokes, just to accommodate development workflows which don't use 
quickstart.



Feedback is welcome, thanks.

[1] All-In-One thread: 
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128900.html
[2] Bug report when undercloud is containeirzed 
https://bugs.launchpad.net/tripleo/+bug/1762422
[3] Tool to update containers if needed: 
https://github.com/imain/container-check
[4] Container-check running in TripleO CI: 
https://review.openstack.org/#/c/558885/ and 
https://review.openstack.org/#/c/529399/

--
Emilien Macchi


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Steve Baker]

On 12/04/18 00:58, Wesley Hayutin wrote:



On Tue, 10 Apr 2018 at 20:51 Emilien Macchi > wrote:


Greetings,

Steve Baker and I had a quick chat today about the work that is
being done around containers workflow in Rocky cycle.

If you're not familiar with the topic, I suggest to first read the
blueprint to understand the context here:
https://blueprints.launchpad.net/tripleo/+spec/container-prepare-workflow

One of the great outcomes of this blueprint is that in Rocky, the
operator won't have to run all the "openstack overcloud container"
commands to prepare the container registry and upload the
containers. Indeed, it'll be driven by Heat and Mistral mostly.
But today our discussion extended on 2 uses-cases that we're going
to explore and find how we can address them:
1) I'm a developer and want to deploy a containerized undercloud
with customized containers (more or less related to the all-in-one
discussions on another thread [1]).
2) I'm submitting a patch in tripleo-common (let's say a workflow)
and need my patch to be tested when the undercloud is
containerized (see [2] for an excellent example).

Both cases would require additional things:
- The container registry needs to be deployed *before* actually
installing the undercloud.
- We need a tool to update containers from this registry and
*before* deploying them. We already have this tool in place in our
CI for the overcloud (see [3] and [4]). Now we need a similar
thing for the undercloud.

Next steps:
- Agree that we need to deploy the container-registry before the
undercloud.
- If agreed, we'll create a new Ansible role called
ansible-role-container-registry that for now will deploy exactly
what we have in TripleO, without extra feature.
- Drive the playbook runtime from tripleoclient to bootstrap the
container registry (which of course could be disabled in
undercloud.conf).
- Create another Ansible role that would re-use container-check
tool but the idea is to provide a role to modify containers when
needed, and we could also control it from tripleoclient. The role
would be using the ContainerImagePrepare parameter, which Steve is
working on right now.


This all looks really good Emilien, thanks for sending it out.
Regarding the update of containers, we would just want to be 100% sure 
that we can control which yum repositories are in play for the 
update.  Maybe it will be done by the user prior to running the 
command, or maybe with some flags to what ever command Steve is 
working on.


Is it enough to retain the existing container-check 
 behavior of just mounting in 
the undercloud's /etc/yum.repos.d?


FYI.. we've noticed in CI that when the base os updates ( not baseos) 
are included you tend to fail on at least on package download on one 
of the 50+ containers due to infra/network.  In CI we only enable 
baseos, dlrn updates and the dependency change [1]


It would be interesting to see what speed/reliability change there would 
be if the concurrency of container-check 
 was disabled and the 
undercloud's /var/cache/yum was mounted in to each container to avoid 
duplicate package download.



Thanks

[1] 
https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L104-L109



Feedback is welcome, thanks.

[1] All-In-One thread:
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128900.html
[2] Bug report when undercloud is containeirzed
https://bugs.launchpad.net/tripleo/+bug/1762422
[3] Tool to update containers if needed:
https://github.com/imain/container-check
[4] Container-check running in TripleO CI:
https://review.openstack.org/#/c/558885/ and
https://review.openstack.org/#/c/529399/
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Ben Nemec]

On 04/11/2018 07:58 AM, Wesley Hayutin wrote:



On Tue, 10 Apr 2018 at 20:51 Emilien Macchi > wrote:


Greetings,

Steve Baker and I had a quick chat today about the work that is
being done around containers workflow in Rocky cycle.

If you're not familiar with the topic, I suggest to first read the
blueprint to understand the context here:
https://blueprints.launchpad.net/tripleo/+spec/container-prepare-workflow

One of the great outcomes of this blueprint is that in Rocky, the
operator won't have to run all the "openstack overcloud container"
commands to prepare the container registry and upload the
containers. Indeed, it'll be driven by Heat and Mistral mostly.
But today our discussion extended on 2 uses-cases that we're going
to explore and find how we can address them:
1) I'm a developer and want to deploy a containerized undercloud
with customized containers (more or less related to the all-in-one
discussions on another thread [1]).
2) I'm submitting a patch in tripleo-common (let's say a workflow)
and need my patch to be tested when the undercloud is containerized
(see [2] for an excellent example).

Both cases would require additional things:
- The container registry needs to be deployed *before* actually
installing the undercloud.
- We need a tool to update containers from this registry and
*before* deploying them. We already have this tool in place in our
CI for the overcloud (see [3] and [4]). Now we need a similar thing
for the undercloud.

Next steps:
- Agree that we need to deploy the container-registry before the
undercloud.
- If agreed, we'll create a new Ansible role called
ansible-role-container-registry that for now will deploy exactly
what we have in TripleO, without extra feature.
- Drive the playbook runtime from tripleoclient to bootstrap the
container registry (which of course could be disabled in
undercloud.conf).
- Create another Ansible role that would re-use container-check tool
but the idea is to provide a role to modify containers when needed,
and we could also control it from tripleoclient. The role would be
using the ContainerImagePrepare parameter, which Steve is working on
right now.


This all looks really good Emilien, thanks for sending it out.
Regarding the update of containers, we would just want to be 100% sure 
that we can control which yum repositories are in play for the update.  
Maybe it will be done by the user prior to running the command, or maybe 
with some flags to what ever command Steve is working on.
FYI.. we've noticed in CI that when the base os updates ( not baseos) 
are included you tend to fail on at least on package download on one of 
the 50+ containers due to infra/network.  In CI we only enable baseos, 
dlrn updates and the dependency change [1]


I will note that this was the sort of use case the -o parameter to 
tripleo-repos was intended to handle.  It can write the configured repos 
to an arbitrary location that we could then mount into the containers so 
the update repos are independent from the underlying system.


https://github.com/openstack/tripleo-repos/blob/8961edcd2d9dd1f2c50d3da51f4129daaad85ab0/tripleo_repos/main.py#L88



Thanks

[1] 
https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L104-L109



Feedback is welcome, thanks.

[1] All-In-One thread:
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128900.html
[2] Bug report when undercloud is containeirzed
https://bugs.launchpad.net/tripleo/+bug/1762422
[3] Tool to update containers if needed:
https://github.com/imain/container-check
[4] Container-check running in TripleO CI:
https://review.openstack.org/#/c/558885/ and
https://review.openstack.org/#/c/529399/
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Wesley Hayutin]

On Tue, 10 Apr 2018 at 20:51 Emilien Macchi  wrote:

> Greetings,
>
> Steve Baker and I had a quick chat today about the work that is being done
> around containers workflow in Rocky cycle.
>
> If you're not familiar with the topic, I suggest to first read the
> blueprint to understand the context here:
> https://blueprints.launchpad.net/tripleo/+spec/container-prepare-workflow
>
> One of the great outcomes of this blueprint is that in Rocky, the operator
> won't have to run all the "openstack overcloud container" commands to
> prepare the container registry and upload the containers. Indeed, it'll be
> driven by Heat and Mistral mostly.
>
> But today our discussion extended on 2 uses-cases that we're going to
> explore and find how we can address them:
> 1) I'm a developer and want to deploy a containerized undercloud with
> customized containers (more or less related to the all-in-one discussions
> on another thread [1]).
> 2) I'm submitting a patch in tripleo-common (let's say a workflow) and
> need my patch to be tested when the undercloud is containerized (see [2]
> for an excellent example).
>
> Both cases would require additional things:
> - The container registry needs to be deployed *before* actually installing
> the undercloud.
> - We need a tool to update containers from this registry and *before*
> deploying them. We already have this tool in place in our CI for the
> overcloud (see [3] and [4]). Now we need a similar thing for the undercloud.
>
> Next steps:
> - Agree that we need to deploy the container-registry before the
> undercloud.
> - If agreed, we'll create a new Ansible role called
> ansible-role-container-registry that for now will deploy exactly what we
> have in TripleO, without extra feature.
> - Drive the playbook runtime from tripleoclient to bootstrap the container
> registry (which of course could be disabled in undercloud.conf).
> - Create another Ansible role that would re-use container-check tool but
> the idea is to provide a role to modify containers when needed, and we
> could also control it from tripleoclient. The role would be using
> the ContainerImagePrepare parameter, which Steve is working on right now.
>

This all looks really good Emilien, thanks for sending it out.
Regarding the update of containers, we would just want to be 100% sure that
we can control which yum repositories are in play for the update.  Maybe it
will be done by the user prior to running the command, or maybe with some
flags to what ever command Steve is working on.
FYI.. we've noticed in CI that when the base os updates ( not baseos) are
included you tend to fail on at least on package download on one of the 50+
containers due to infra/network.  In CI we only enable baseos, dlrn updates
and the dependency change [1]

Thanks

[1]
https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L104-L109


>
> Feedback is welcome, thanks.
>
> [1] All-In-One thread:
> http://lists.openstack.org/pipermail/openstack-dev/2018-March/128900.html
> [2] Bug report when undercloud is containeirzed
> https://bugs.launchpad.net/tripleo/+bug/1762422
> [3] Tool to update containers if needed:
> https://github.com/imain/container-check
> [4] Container-check running in TripleO CI:
> https://review.openstack.org/#/c/558885/ and
> https://review.openstack.org/#/c/529399/
> --
> Emilien Macchi
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] roadmap on containers workflow

[Emilien Macchi]

Greetings,

Steve Baker and I had a quick chat today about the work that is being done
around containers workflow in Rocky cycle.

If you're not familiar with the topic, I suggest to first read the
blueprint to understand the context here:
https://blueprints.launchpad.net/tripleo/+spec/container-prepare-workflow

One of the great outcomes of this blueprint is that in Rocky, the operator
won't have to run all the "openstack overcloud container" commands to
prepare the container registry and upload the containers. Indeed, it'll be
driven by Heat and Mistral mostly.

But today our discussion extended on 2 uses-cases that we're going to
explore and find how we can address them:
1) I'm a developer and want to deploy a containerized undercloud with
customized containers (more or less related to the all-in-one discussions
on another thread [1]).
2) I'm submitting a patch in tripleo-common (let's say a workflow) and need
my patch to be tested when the undercloud is containerized (see [2] for an
excellent example).

Both cases would require additional things:
- The container registry needs to be deployed *before* actually installing
the undercloud.
- We need a tool to update containers from this registry and *before*
deploying them. We already have this tool in place in our CI for the
overcloud (see [3] and [4]). Now we need a similar thing for the undercloud.

Next steps:
- Agree that we need to deploy the container-registry before the undercloud.
- If agreed, we'll create a new Ansible role called
ansible-role-container-registry that for now will deploy exactly what we
have in TripleO, without extra feature.
- Drive the playbook runtime from tripleoclient to bootstrap the container
registry (which of course could be disabled in undercloud.conf).
- Create another Ansible role that would re-use container-check tool but
the idea is to provide a role to modify containers when needed, and we
could also control it from tripleoclient. The role would be using
the ContainerImagePrepare parameter, which Steve is working on right now.

Feedback is welcome, thanks.

[1] All-In-One thread:
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128900.html
[2] Bug report when undercloud is containeirzed
https://bugs.launchpad.net/tripleo/+bug/1762422
[3] Tool to update containers if needed:
https://github.com/imain/container-check
[4] Container-check running in TripleO CI:
https://review.openstack.org/#/c/558885/ and
https://review.openstack.org/#/c/529399/
-- 
Emilien Macchi
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev