Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Flavio Percoco]

On 17/07/17 14:05 +0200, Flavio Percoco wrote:

Thanks for all the feedback so far. This is one of the things I appreciate the
most about this community, Open conversations, honest feedback and will to
collaborate.

I'm top-posting to announce that we'll have a joint meeting with the Kolla team
on Wednesday at 16:00 UTC. I know it's not an ideal time for many (it's not for
me) but I do want to have a live discussion with the rest of the Kolla team.

Some questions about the meeting:

* How much time can we allocate?
* Can we prepare an agenda rather than just discussing "TripleO is thinking of
using Ansible and not kolla-kubernetes"? (I'm happy to come up with such
agenda)

One last point. I'm not interested in conversations around competition,
re-invention, etc. I think I speak for the entire TripleO team when I say that
this is not about "winning" in this space but rather seeing how/if we can
collaborate and how/if it makes sense to keep exploring the path described in
the email below.


Hey y'all,

Sorry for not having sent this earlier but, Life Happened (TM).

In preparation for the meeting today, I took the time to collect some thoughts
on the topic so that we can, hopefully, have a more focused and constructive
conversation.

Please, find my thoughts on this etherpad and feel free to comment on it. I've
disabled color so please, tag your comments with your nickname.

https://etherpad.openstack.org/p/tripleo-ptg-queens-kubernetes

Flavio

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Sanjay Upadhyay]

On Tue, Jul 18, 2017 at 12:21 PM, Flavio Percoco  wrote:

> On 17/07/17 16:48 -0400, Ryan Hallisey wrote:
>
>> One other thing to mention. Maybe folks can speed up writing these
>> playbooks by using kolla-ansible's playbooks as a shell. Here's an
>> example: [1] Take lines 1-16 and replace it with helm install mariadb
>> or
>> kubectl create -f mariabd-pod.yaml and set inventory to localhost.
>> Just a thought.
>>
>> There may be some other playbooks out there I don' know about that you
>> can use, but that could at least get some of the collaboration started
>> so folks don't have to start from scratch.
>>
>> [1] - https://github.com/openstack/kolla-ansible/blob/afdd11b9a22e
>> cca70962a4637d89ad50b7ded2e5/ansible/roles/mariadb/tasks/start.yml#L1-L16
>>
>
> +1 This is why I think there's still room for collaboration and we can
> re-use
> several of the existing things. I don't think everything would have to be
> written from scratch.
>
>

Not sure if this is an important criteria, however, are we also looking at
using OCI, so that users can perhaps choose between Container platforms?

On the upgrade path we are already using ansible playbooks. How hard would
it be to do a major migrate from current tripleo to a new standard,
whatever is chosen.

One more thing I needed to emphasize is that probably *only* tripleo as it
is now, addresses datacenter/telco specific network paradigm ie what is
done via os-net-config (bonding,sriov,dpdk). We would want to have that
addressed, or at least have a plan in whatever path we chose.

regards
/sanjay


> Flavio
>
> --
> @flaper87
> Flavio Percoco
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Flavio Percoco]

On 17/07/17 16:48 -0400, Ryan Hallisey wrote:

One other thing to mention. Maybe folks can speed up writing these
playbooks by using kolla-ansible's playbooks as a shell. Here's an
example: [1] Take lines 1-16 and replace it with helm install mariadb
or
kubectl create -f mariabd-pod.yaml and set inventory to localhost.
Just a thought.

There may be some other playbooks out there I don' know about that you
can use, but that could at least get some of the collaboration started
so folks don't have to start from scratch.

[1] - 
https://github.com/openstack/kolla-ansible/blob/afdd11b9a22ecca70962a4637d89ad50b7ded2e5/ansible/roles/mariadb/tasks/start.yml#L1-L16


+1 This is why I think there's still room for collaboration and we can re-use
several of the existing things. I don't think everything would have to be
written from scratch.

Flavio

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Fox, Kevin M]

I re-read this and maybe you mean, some containers will live only outside of 
k8s and some will live in k8s, not that you want to to support not having k8s 
at all with the same code base? That would be a much easier thing, and agree 
ansible would be very good at that.

Thanks,
Kevin

From: Fox, Kevin M
Sent: Monday, July 17, 2017 4:45 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack 
services on Kubernetes

I think if you try to go down the Kubernetes & !Kubernetes path, you'll end up 
re-implementing pretty much all of Kubernetes, or you will use Kubernetes just 
like !Kubernetes and gain very little benefit from it.

Thanks,
Kevin

From: Flavio Percoco [fla...@redhat.com]
Sent: Monday, July 17, 2017 8:12 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack 
services on Kubernetes

On 17/07/17 09:47 -0400, James Slagle wrote:
>On Mon, Jul 17, 2017 at 8:05 AM, Flavio Percoco <fla...@redhat.com> wrote:
>> Thanks for all the feedback so far. This is one of the things I appreciate
>> the
>> most about this community, Open conversations, honest feedback and will to
>> collaborate.
>>
>> I'm top-posting to announce that we'll have a joint meeting with the Kolla
>> team
>> on Wednesday at 16:00 UTC. I know it's not an ideal time for many (it's not
>> for
>> me) but I do want to have a live discussion with the rest of the Kolla team.
>>
>> Some questions about the meeting:
>>
>> * How much time can we allocate?
>> * Can we prepare an agenda rather than just discussing "TripleO is thinking
>> of
>>  using Ansible and not kolla-kubernetes"? (I'm happy to come up with such
>>  agenda)
>
>It may help to prepare some high level requirements around what we
>need out of a solution. For the ansible discussion I started this
>etherpad:
>
>https://etherpad.openstack.org/p/tripleo-ptg-queens-ansible
>
>How we use Ansible and what we want to use it for, is related to this
>discussion around Helm. Although, it's not the exact same discussion,
>so if you wanted to start a new etherpad more specific to
>tripleo/kubernetes that may be good as well.
>
>One thing I think is important in this discussion is that we should be
>thinking about deploying containers on both Kubernetes and
>!Kubernetes. That is one of the reasons I like the ansible approach,
>in that I think it could address both cases with a common interface
>and API. I don't think we should necessarily choose a solution that
>requires to deploy on Kubernetes. Because then we are stuck with that
>choice. It'd be really nice to just "docker run" sometimes for
>dev/test. I don't know if Helm has that abstraction or not, I'm just
>trying to capture the requirement.

Yes!

Thanks for pointing this out as this is one of the reasons why I was proposing
ansible as our common interface w/o any extra layer.

I'll probably start a new etherpad for this as I would prefer not to distract
the rest of the TripleO + ansible discussion. At the end, if ansible ends up
being the tool we pick, I'll make sure to update your etherpad.

Flavio

>If you consider the parallel with Heat in this regard, we are
>currently "stuck" deploying on OpenStack (undercloud with Heat). We've
>had to work an a lot of complimentary features to add the flexibility
>to TripleO that are a result of having to use OpenStack (OVB,
>split-stack).
>
>That's exactly why we are starting a discussion around using Ansible,
>and is one of the fundamental changes that operators have been
>requesting in TripleO.
>
>--
>-- James Slagle
>--
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
@flaper87
Flavio Percoco

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Fox, Kevin M]

I think thats a good question without an easy answer. I think TripleO's own 
struggle with orchestration has shown that its maybe one of the hardest pieces. 
There are a lot of orchestration tools out there. Each has its 
strengths/weaknesses. I  personally can't really pick what the best one is for 
this sort of thing. I've been trying to stay neutral, and let the low level 
kolla-kubernetes components be easily sharable between all the projects that 
already have chosen an orchestration strategy. I think the real answer is 
probably that the best orchestration tool for the job depends entirely on the 
deployment tool. So, TripleO's answer might be different then say, something 
Ubuntu does.

Kolla-kubernetes has implemented reference orchestration a few different ways 
now. We deploy the gates using pure shell. Its not the prettiest way but works 
reliably now. (I would not recommend users do this)

We have a document for manual orchestration.  (slow and very manual, but you 
get to see all the pieces, which can be instructive)

We have helm based orchestration that bundles several microservice charts into 
service charts and deploys similarly to openstack-helm. We built them to test 
the waters of this approach and they do work, but I have doubts they could be 
made robust enough to handle things like live rolling upgrades of OpenStack. It 
may be robust enough to do upgrades that require downtimes. I think it also may 
be hard to debug if the upgrade fails half way through. I admit I could totally 
be wrong though.

Theres also been a couple of ansible based orchestrators that have been 
proposed. They seem to work well, and I think they would be much easier to 
extend to do a live rolling OpenStack upgrade. I'd very much like to see an 
Ansible one finished and kick the tires with it. I do think having both some 
folks in Kolla-Kubernetes and folks in TripleO independently implement k8s 
deployment with it shows there is a lot of potential in that form of 
orchestration and that there's even more room for collaboration between the two 
projects.

Thanks,
Kevin

From: Bogdan Dobrelya [bdobr...@redhat.com]
Sent: Monday, July 17, 2017 1:10 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack 
services on Kubernetes

On 14.07.2017 22:55, Fox, Kevin M wrote:
> Part of the confusion I think is in the different ways helm can be used.
>
> Helm can be used to orchestrate the deployment of a whole service (ex, nova). 
> "launch these 3 k8s objects, template out this config file, run this job to 
> init the db, or this job to upgrade the db, etc", all as a single unit.
>
> It can also be used purely for its templating ability.
>
> So, "render this single k8s object using these values".
>
> This is one of the main differences between openstack-helm and 
> kolla-kubernetes.
>
> Openstack-helm has charts only for orchestrating the deployment of whole 
> openstack services.
>
> Kolla-kubernetes has taken a different track though. While it does use helm 
> for its golang templater, it has taken a microservices approach to be 
> shareable with other tools. So, each openstack process (nova-api, 
> neutron-server, neutron-openvswitch-agent), etc, has its own chart and can be 
> independently configured/placed as needed by an external orchestration 
> system. Kolla-Kubernetes microservice charts are to Kubernetes what 
> Kolla-Containers are to Docker. Reusable building blocks of known tested 
> functionality and assemblable anyway the orchestration system/user feels is 
> in their best interest.

A great summary!
As TripleO Pike docker-based containers architecture rely on
Kolla-Containers bits a lot, which is run-time kolla config/bootstrap
and build-time images overrides, it seems reasonable to continue
following that path by relying on Kolla-Kubernetes microservice Helm
charts for Kubernetes based architecture. Isn't it?

The remaining question is though, if Kolla-kubernetes doesn't consume
the Openstack-helm's opinionated "orchestration of the deployment of
whole openstack services", which tools to use then to fill the advanced
data parameterization gaps like "happens before/after" relationships and
data dependencies/ordering?

>
> This is why I think kolla-kubernetes would be a good fit for TripleO, as you 
> can replace a single component at a time, however you want, using the config 
> files you already have and upgrade the system a piece at a time from non 
> container to containered. It doesn't have to happen all at once, even within 
> a single service, or within a single TripleO release. The orchestration of it 
> is totally up to you, and can be tailored very precisely to deal with the 
> particulars of the upgrade strategy needed by TripleO's existin

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Steven Dake]

On Mon, Jul 17, 2017 at 10:13 AM, Emilien Macchi  wrote:

> On Mon, Jul 17, 2017 at 5:32 AM, Flavio Percoco  wrote:
> > On 14/07/17 08:08 -0700, Emilien Macchi wrote:
> >>
> >> On Fri, Jul 14, 2017 at 2:17 AM, Flavio Percoco 
> wrote:
> >>>
> >>>
> >>> Greetings,
> >>>
> >>> As some of you know, I've been working on the second phase of TripleO's
> >>> containerization effort. This phase if about migrating the docker based
> >>> deployment onto Kubernetes.
> >>>
> >>> These phase requires work on several areas: Kubernetes deployment,
> >>> OpenStack
> >>> deployment on Kubernetes, configuration management, etc. While I've
> been
> >>> diving
> >>> into all of these areas, this email is about the second point,
> OpenStack
> >>> deployment on Kubernetes.
> >>>
> >>> There are several tools we could use for this task. kolla-kubernetes,
> >>> openstack-helm, ansible roles, among others. I've looked into these
> tools
> >>> and
> >>> I've come to the conclusion that TripleO would be better of by having
> >>> ansible
> >>> roles that would allow for deploying OpenStack services on Kubernetes.
> >>>
> >>> The existing solutions in the OpenStack community require using Helm.
> >>> While
> >>> I
> >>> like Helm and both, kolla-kubernetes and openstack-helm OpenStack
> >>> projects,
> >>> I
> >>> believe using any of them would add an extra layer of complexity to
> >>> TripleO,
> >>> which is something the team has been fighting for years years -
> >>> especially
> >>> now
> >>> that the snowball is being chopped off.
> >>>
> >>> Adopting any of the existing projects in the OpenStack communty would
> >>> require
> >>> TripleO to also write the logic to manage those projects. For example,
> in
> >>> the
> >>> case of openstack-helm, the TripleO team would have to write either
> >>> ansible
> >>> roles or heat templates to manage - install, remove, upgrade - the
> charts
> >>> (I'm
> >>> happy to discuss this point further but I'm keepping it at a high-level
> >>> on
> >>> purpose for the sake of not writing a 10k-words-long email).
> >>>
> >>> James Slagle sent an email[0], a couple of days ago, to form TripleO
> >>> plans
> >>> around ansible. One take-away from this thread is that TripleO is
> >>> adopting
> >>> ansible more and more, which is great and it fits perfectly with the
> >>> conclusion
> >>> I reached.
> >>>
> >>> Now, what this work means is that we would have to write an ansible
> role
> >>> for
> >>> each service that will deploy the service on a Kubernetes cluster.
> >>> Ideally
> >>> these
> >>> roles will also generate the configuration files (removing the need of
> >>> puppet
> >>> entirely) and they would manage the lifecycle. The roles would be
> >>> isolated
> >>> and
> >>> this will reduce the need of TripleO Heat templates. Doing this would
> >>> give
> >>> TripleO full control on the deployment process too.
> >>>
> >>> In addition, we could also write Ansible Playbook Bundles to contain
> >>> these
> >>> roles
> >>> and run them using the existing docker-cmd implementation that is
> coming
> >>> out
> >>> in
> >>> Pike (you can find a PoC/example of this in this repo[1]).
> >>>
> >>> Now, I do realize the amount of work this implies and that this is my
> >>> opinion/conclusion. I'm sending this email out to kick-off the
> discussion
> >>> and
> >>> gather thoughts and opinions from the rest of the community.
> >>>
> >>> Finally, what I really like about writing pure ansible roles is that
> >>> ansible
> >>> is
> >>> a known, powerfull, tool that has been adopted by many operators
> already.
> >>> It'll
> >>> provide the flexibility needed and, if structured correctly, it'll
> allow
> >>> for
> >>> operators (and other teams) to just use the parts they need/want
> without
> >>> depending on the full-stack. I like the idea of being able to separate
> >>> concerns
> >>> in the deployment workflow and the idea of making it simple for users
> of
> >>> TripleO
> >>> to do the same at runtime. Unfortunately, going down this road means
> that
> >>> my
> >>> hope of creating a field where we could collaborate even more with
> other
> >>> deployment tools will be a bit limited but I'm confident the result
> would
> >>> also
> >>> be useful for others and that we all will benefit from it... My hopes
> >>> might
> >>> be a
> >>> bit naive *shrugs*
> >>
> >>
> >> Of course I'm biased since I've been (a little) involved in that work
> >> but I like the idea of :
> >>
> >> - Moving forward with our containerization. docker-cmd will help us
> >> for sure for this transition (I insist on the fact TripleO is a
> >> product that you can upgrade and we try to make it smooth for our
> >> operators), so we can't just trash everything and switch to a new
> >> tool. I think the approach that we're taking is great and made of baby
> >> steps where we try to solve different problems.
> >> - Using more Ansible - the right way - when it makes sense : with the
> 

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Fox, Kevin M]

I think if you try to go down the Kubernetes & !Kubernetes path, you'll end up 
re-implementing pretty much all of Kubernetes, or you will use Kubernetes just 
like !Kubernetes and gain very little benefit from it.

Thanks,
Kevin

From: Flavio Percoco [fla...@redhat.com]
Sent: Monday, July 17, 2017 8:12 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack 
services on Kubernetes

On 17/07/17 09:47 -0400, James Slagle wrote:
>On Mon, Jul 17, 2017 at 8:05 AM, Flavio Percoco <fla...@redhat.com> wrote:
>> Thanks for all the feedback so far. This is one of the things I appreciate
>> the
>> most about this community, Open conversations, honest feedback and will to
>> collaborate.
>>
>> I'm top-posting to announce that we'll have a joint meeting with the Kolla
>> team
>> on Wednesday at 16:00 UTC. I know it's not an ideal time for many (it's not
>> for
>> me) but I do want to have a live discussion with the rest of the Kolla team.
>>
>> Some questions about the meeting:
>>
>> * How much time can we allocate?
>> * Can we prepare an agenda rather than just discussing "TripleO is thinking
>> of
>>  using Ansible and not kolla-kubernetes"? (I'm happy to come up with such
>>  agenda)
>
>It may help to prepare some high level requirements around what we
>need out of a solution. For the ansible discussion I started this
>etherpad:
>
>https://etherpad.openstack.org/p/tripleo-ptg-queens-ansible
>
>How we use Ansible and what we want to use it for, is related to this
>discussion around Helm. Although, it's not the exact same discussion,
>so if you wanted to start a new etherpad more specific to
>tripleo/kubernetes that may be good as well.
>
>One thing I think is important in this discussion is that we should be
>thinking about deploying containers on both Kubernetes and
>!Kubernetes. That is one of the reasons I like the ansible approach,
>in that I think it could address both cases with a common interface
>and API. I don't think we should necessarily choose a solution that
>requires to deploy on Kubernetes. Because then we are stuck with that
>choice. It'd be really nice to just "docker run" sometimes for
>dev/test. I don't know if Helm has that abstraction or not, I'm just
>trying to capture the requirement.

Yes!

Thanks for pointing this out as this is one of the reasons why I was proposing
ansible as our common interface w/o any extra layer.

I'll probably start a new etherpad for this as I would prefer not to distract
the rest of the TripleO + ansible discussion. At the end, if ansible ends up
being the tool we pick, I'll make sure to update your etherpad.

Flavio

>If you consider the parallel with Heat in this regard, we are
>currently "stuck" deploying on OpenStack (undercloud with Heat). We've
>had to work an a lot of complimentary features to add the flexibility
>to TripleO that are a result of having to use OpenStack (OVB,
>split-stack).
>
>That's exactly why we are starting a discussion around using Ansible,
>and is one of the fundamental changes that operators have been
>requesting in TripleO.
>
>--
>-- James Slagle
>--
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
@flaper87
Flavio Percoco

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Fox, Kevin M]

We do support some upstream charts but we started mariadb/rabbit before some of 
the upstream charts were written, so we duplicate a little bit of functionality 
at the moment. You can mix and match though. If an upstream chart doesn't work 
with kolla-kubernetes, I consider that a bug we should fix. Likewise, you 
should be able to run noncontainerized stuff mixed in too. If it doesn't work, 
its likewise a bug. You should be able to run kolla-kubernetes with a baremetal 
db.

Some known working stuff: prometheus/grafana upstream charts start collecting 
data from the containers as soon as they are launched.
I have also tested a bit with the upstream fluent-bit chart and have a ps in 
the works to make it work much better.

Thanks,
Kevin

From: Emilien Macchi [emil...@redhat.com]
Sent: Monday, July 17, 2017 10:13 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack 
services on Kubernetes

On Mon, Jul 17, 2017 at 5:32 AM, Flavio Percoco <fla...@redhat.com> wrote:
> On 14/07/17 08:08 -0700, Emilien Macchi wrote:
>>
>> On Fri, Jul 14, 2017 at 2:17 AM, Flavio Percoco <fla...@redhat.com> wrote:
>>>
>>>
>>> Greetings,
>>>
>>> As some of you know, I've been working on the second phase of TripleO's
>>> containerization effort. This phase if about migrating the docker based
>>> deployment onto Kubernetes.
>>>
>>> These phase requires work on several areas: Kubernetes deployment,
>>> OpenStack
>>> deployment on Kubernetes, configuration management, etc. While I've been
>>> diving
>>> into all of these areas, this email is about the second point, OpenStack
>>> deployment on Kubernetes.
>>>
>>> There are several tools we could use for this task. kolla-kubernetes,
>>> openstack-helm, ansible roles, among others. I've looked into these tools
>>> and
>>> I've come to the conclusion that TripleO would be better of by having
>>> ansible
>>> roles that would allow for deploying OpenStack services on Kubernetes.
>>>
>>> The existing solutions in the OpenStack community require using Helm.
>>> While
>>> I
>>> like Helm and both, kolla-kubernetes and openstack-helm OpenStack
>>> projects,
>>> I
>>> believe using any of them would add an extra layer of complexity to
>>> TripleO,
>>> which is something the team has been fighting for years years -
>>> especially
>>> now
>>> that the snowball is being chopped off.
>>>
>>> Adopting any of the existing projects in the OpenStack communty would
>>> require
>>> TripleO to also write the logic to manage those projects. For example, in
>>> the
>>> case of openstack-helm, the TripleO team would have to write either
>>> ansible
>>> roles or heat templates to manage - install, remove, upgrade - the charts
>>> (I'm
>>> happy to discuss this point further but I'm keepping it at a high-level
>>> on
>>> purpose for the sake of not writing a 10k-words-long email).
>>>
>>> James Slagle sent an email[0], a couple of days ago, to form TripleO
>>> plans
>>> around ansible. One take-away from this thread is that TripleO is
>>> adopting
>>> ansible more and more, which is great and it fits perfectly with the
>>> conclusion
>>> I reached.
>>>
>>> Now, what this work means is that we would have to write an ansible role
>>> for
>>> each service that will deploy the service on a Kubernetes cluster.
>>> Ideally
>>> these
>>> roles will also generate the configuration files (removing the need of
>>> puppet
>>> entirely) and they would manage the lifecycle. The roles would be
>>> isolated
>>> and
>>> this will reduce the need of TripleO Heat templates. Doing this would
>>> give
>>> TripleO full control on the deployment process too.
>>>
>>> In addition, we could also write Ansible Playbook Bundles to contain
>>> these
>>> roles
>>> and run them using the existing docker-cmd implementation that is coming
>>> out
>>> in
>>> Pike (you can find a PoC/example of this in this repo[1]).
>>>
>>> Now, I do realize the amount of work this implies and that this is my
>>> opinion/conclusion. I'm sending this email out to kick-off the discussion
>>> and
>>> gather thoughts and opinions from the rest of the community.
>>>
>>> Fin

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Ryan Hallisey]

> I think this at some point might be single biggest benefit of using
> helm on the long run - leverage infrastructure charts that aren't
> openstack-centric. Things like etcd are already written and
> potentially we can just help support them.

I think the tools that are being discussion here are both very good
(helm & ansible), but I have a slightly different opinion about how
Helm should be used.

Helm is a *package manager*. It's scope is for simple applications
that need to bundle resources.  It's great at saving me time on doing
simple recipes like: kubectl create -f  and kubectl create -f
 over and over again. But, beyond a single app with a few
resources, Helm is going to struggle on it's own. Meaning, either Helm
would have to change or another tool would have to fill the gaps.

If Helm wants to change, it becomes less differentiated from what
Ansible already does. It's niche as a simple app package manager will
evaporate and Ansible already owns the orchestration space. Therefore,
I think long term Helm as an orchestration tool doesn't make sense
because it's limited to Kubernetes and Ansible adoption is wide
spread.

That doesn't mean that Helm is useless.  In fact, I think the Helm
charts are great when used as simple standalone recipes. However, for
a complex app like OpenStack, I think you need something like Ansible
to provide the orchestration. Underneath, you can use whatever you
want to create the Kubernetes resources. In the end, the difference
will be `helm create mariadb` vs `kubectl create -f mariadb-pod.yaml`.
Both solutions will work, but the Helm work seems to be much farther
along.

One other thing to mention. Maybe folks can speed up writing these
playbooks by using kolla-ansible's playbooks as a shell. Here's an
example: [1] Take lines 1-16 and replace it with helm install mariadb
or
kubectl create -f mariabd-pod.yaml and set inventory to localhost.
Just a thought.

There may be some other playbooks out there I don' know about that you
can use, but that could at least get some of the collaboration started
so folks don't have to start from scratch.

[1] - 
https://github.com/openstack/kolla-ansible/blob/afdd11b9a22ecca70962a4637d89ad50b7ded2e5/ansible/roles/mariadb/tasks/start.yml#L1-L16

Sincerely,
Ryan

On Mon, Jul 17, 2017 at 1:37 PM, Michał Jastrzębski  wrote:
> On 17 July 2017 at 10:13, Emilien Macchi  wrote:
>> On Mon, Jul 17, 2017 at 5:32 AM, Flavio Percoco  wrote:
>>> On 14/07/17 08:08 -0700, Emilien Macchi wrote:

 On Fri, Jul 14, 2017 at 2:17 AM, Flavio Percoco  wrote:
>
>
> Greetings,
>
> As some of you know, I've been working on the second phase of TripleO's
> containerization effort. This phase if about migrating the docker based
> deployment onto Kubernetes.
>
> These phase requires work on several areas: Kubernetes deployment,
> OpenStack
> deployment on Kubernetes, configuration management, etc. While I've been
> diving
> into all of these areas, this email is about the second point, OpenStack
> deployment on Kubernetes.
>
> There are several tools we could use for this task. kolla-kubernetes,
> openstack-helm, ansible roles, among others. I've looked into these tools
> and
> I've come to the conclusion that TripleO would be better of by having
> ansible
> roles that would allow for deploying OpenStack services on Kubernetes.
>
> The existing solutions in the OpenStack community require using Helm.
> While
> I
> like Helm and both, kolla-kubernetes and openstack-helm OpenStack
> projects,
> I
> believe using any of them would add an extra layer of complexity to
> TripleO,
> which is something the team has been fighting for years years -
> especially
> now
> that the snowball is being chopped off.
>
> Adopting any of the existing projects in the OpenStack communty would
> require
> TripleO to also write the logic to manage those projects. For example, in
> the
> case of openstack-helm, the TripleO team would have to write either
> ansible
> roles or heat templates to manage - install, remove, upgrade - the charts
> (I'm
> happy to discuss this point further but I'm keepping it at a high-level
> on
> purpose for the sake of not writing a 10k-words-long email).
>
> James Slagle sent an email[0], a couple of days ago, to form TripleO
> plans
> around ansible. One take-away from this thread is that TripleO is
> adopting
> ansible more and more, which is great and it fits perfectly with the
> conclusion
> I reached.
>
> Now, what this work means is that we would have to write an ansible role
> for
> each service that will deploy the service on a Kubernetes cluster.
> Ideally
> these
> roles will also generate the configuration files (removing the 

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Michał Jastrzębski]

On 17 July 2017 at 10:13, Emilien Macchi  wrote:
> On Mon, Jul 17, 2017 at 5:32 AM, Flavio Percoco  wrote:
>> On 14/07/17 08:08 -0700, Emilien Macchi wrote:
>>>
>>> On Fri, Jul 14, 2017 at 2:17 AM, Flavio Percoco  wrote:


 Greetings,

 As some of you know, I've been working on the second phase of TripleO's
 containerization effort. This phase if about migrating the docker based
 deployment onto Kubernetes.

 These phase requires work on several areas: Kubernetes deployment,
 OpenStack
 deployment on Kubernetes, configuration management, etc. While I've been
 diving
 into all of these areas, this email is about the second point, OpenStack
 deployment on Kubernetes.

 There are several tools we could use for this task. kolla-kubernetes,
 openstack-helm, ansible roles, among others. I've looked into these tools
 and
 I've come to the conclusion that TripleO would be better of by having
 ansible
 roles that would allow for deploying OpenStack services on Kubernetes.

 The existing solutions in the OpenStack community require using Helm.
 While
 I
 like Helm and both, kolla-kubernetes and openstack-helm OpenStack
 projects,
 I
 believe using any of them would add an extra layer of complexity to
 TripleO,
 which is something the team has been fighting for years years -
 especially
 now
 that the snowball is being chopped off.

 Adopting any of the existing projects in the OpenStack communty would
 require
 TripleO to also write the logic to manage those projects. For example, in
 the
 case of openstack-helm, the TripleO team would have to write either
 ansible
 roles or heat templates to manage - install, remove, upgrade - the charts
 (I'm
 happy to discuss this point further but I'm keepping it at a high-level
 on
 purpose for the sake of not writing a 10k-words-long email).

 James Slagle sent an email[0], a couple of days ago, to form TripleO
 plans
 around ansible. One take-away from this thread is that TripleO is
 adopting
 ansible more and more, which is great and it fits perfectly with the
 conclusion
 I reached.

 Now, what this work means is that we would have to write an ansible role
 for
 each service that will deploy the service on a Kubernetes cluster.
 Ideally
 these
 roles will also generate the configuration files (removing the need of
 puppet
 entirely) and they would manage the lifecycle. The roles would be
 isolated
 and
 this will reduce the need of TripleO Heat templates. Doing this would
 give
 TripleO full control on the deployment process too.

 In addition, we could also write Ansible Playbook Bundles to contain
 these
 roles
 and run them using the existing docker-cmd implementation that is coming
 out
 in
 Pike (you can find a PoC/example of this in this repo[1]).

 Now, I do realize the amount of work this implies and that this is my
 opinion/conclusion. I'm sending this email out to kick-off the discussion
 and
 gather thoughts and opinions from the rest of the community.

 Finally, what I really like about writing pure ansible roles is that
 ansible
 is
 a known, powerfull, tool that has been adopted by many operators already.
 It'll
 provide the flexibility needed and, if structured correctly, it'll allow
 for
 operators (and other teams) to just use the parts they need/want without
 depending on the full-stack. I like the idea of being able to separate
 concerns
 in the deployment workflow and the idea of making it simple for users of
 TripleO
 to do the same at runtime. Unfortunately, going down this road means that
 my
 hope of creating a field where we could collaborate even more with other
 deployment tools will be a bit limited but I'm confident the result would
 also
 be useful for others and that we all will benefit from it... My hopes
 might
 be a
 bit naive *shrugs*
>>>
>>>
>>> Of course I'm biased since I've been (a little) involved in that work
>>> but I like the idea of :
>>>
>>> - Moving forward with our containerization. docker-cmd will help us
>>> for sure for this transition (I insist on the fact TripleO is a
>>> product that you can upgrade and we try to make it smooth for our
>>> operators), so we can't just trash everything and switch to a new
>>> tool. I think the approach that we're taking is great and made of baby
>>> steps where we try to solve different problems.
>>> - Using more Ansible - the right way - when it makes sense : with the
>>> TripleO containerization, we only use Puppet for Configuration
>>> Management, managing a few resources but not for orchestration (or not
>>> all the 

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Emilien Macchi]

On Mon, Jul 17, 2017 at 5:32 AM, Flavio Percoco  wrote:
> On 14/07/17 08:08 -0700, Emilien Macchi wrote:
>>
>> On Fri, Jul 14, 2017 at 2:17 AM, Flavio Percoco  wrote:
>>>
>>>
>>> Greetings,
>>>
>>> As some of you know, I've been working on the second phase of TripleO's
>>> containerization effort. This phase if about migrating the docker based
>>> deployment onto Kubernetes.
>>>
>>> These phase requires work on several areas: Kubernetes deployment,
>>> OpenStack
>>> deployment on Kubernetes, configuration management, etc. While I've been
>>> diving
>>> into all of these areas, this email is about the second point, OpenStack
>>> deployment on Kubernetes.
>>>
>>> There are several tools we could use for this task. kolla-kubernetes,
>>> openstack-helm, ansible roles, among others. I've looked into these tools
>>> and
>>> I've come to the conclusion that TripleO would be better of by having
>>> ansible
>>> roles that would allow for deploying OpenStack services on Kubernetes.
>>>
>>> The existing solutions in the OpenStack community require using Helm.
>>> While
>>> I
>>> like Helm and both, kolla-kubernetes and openstack-helm OpenStack
>>> projects,
>>> I
>>> believe using any of them would add an extra layer of complexity to
>>> TripleO,
>>> which is something the team has been fighting for years years -
>>> especially
>>> now
>>> that the snowball is being chopped off.
>>>
>>> Adopting any of the existing projects in the OpenStack communty would
>>> require
>>> TripleO to also write the logic to manage those projects. For example, in
>>> the
>>> case of openstack-helm, the TripleO team would have to write either
>>> ansible
>>> roles or heat templates to manage - install, remove, upgrade - the charts
>>> (I'm
>>> happy to discuss this point further but I'm keepping it at a high-level
>>> on
>>> purpose for the sake of not writing a 10k-words-long email).
>>>
>>> James Slagle sent an email[0], a couple of days ago, to form TripleO
>>> plans
>>> around ansible. One take-away from this thread is that TripleO is
>>> adopting
>>> ansible more and more, which is great and it fits perfectly with the
>>> conclusion
>>> I reached.
>>>
>>> Now, what this work means is that we would have to write an ansible role
>>> for
>>> each service that will deploy the service on a Kubernetes cluster.
>>> Ideally
>>> these
>>> roles will also generate the configuration files (removing the need of
>>> puppet
>>> entirely) and they would manage the lifecycle. The roles would be
>>> isolated
>>> and
>>> this will reduce the need of TripleO Heat templates. Doing this would
>>> give
>>> TripleO full control on the deployment process too.
>>>
>>> In addition, we could also write Ansible Playbook Bundles to contain
>>> these
>>> roles
>>> and run them using the existing docker-cmd implementation that is coming
>>> out
>>> in
>>> Pike (you can find a PoC/example of this in this repo[1]).
>>>
>>> Now, I do realize the amount of work this implies and that this is my
>>> opinion/conclusion. I'm sending this email out to kick-off the discussion
>>> and
>>> gather thoughts and opinions from the rest of the community.
>>>
>>> Finally, what I really like about writing pure ansible roles is that
>>> ansible
>>> is
>>> a known, powerfull, tool that has been adopted by many operators already.
>>> It'll
>>> provide the flexibility needed and, if structured correctly, it'll allow
>>> for
>>> operators (and other teams) to just use the parts they need/want without
>>> depending on the full-stack. I like the idea of being able to separate
>>> concerns
>>> in the deployment workflow and the idea of making it simple for users of
>>> TripleO
>>> to do the same at runtime. Unfortunately, going down this road means that
>>> my
>>> hope of creating a field where we could collaborate even more with other
>>> deployment tools will be a bit limited but I'm confident the result would
>>> also
>>> be useful for others and that we all will benefit from it... My hopes
>>> might
>>> be a
>>> bit naive *shrugs*
>>
>>
>> Of course I'm biased since I've been (a little) involved in that work
>> but I like the idea of :
>>
>> - Moving forward with our containerization. docker-cmd will help us
>> for sure for this transition (I insist on the fact TripleO is a
>> product that you can upgrade and we try to make it smooth for our
>> operators), so we can't just trash everything and switch to a new
>> tool. I think the approach that we're taking is great and made of baby
>> steps where we try to solve different problems.
>> - Using more Ansible - the right way - when it makes sense : with the
>> TripleO containerization, we only use Puppet for Configuration
>> Management, managing a few resources but not for orchestration (or not
>> all the features that Puppet provide) and for Data Binding (Hiera). To
>> me, it doesn't make sense for us to keep investing much in Puppet
>> modules if we go k8s & Ansible. That said, see the next point.

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Flavio Percoco]

On 17/07/17 09:47 -0400, James Slagle wrote:

On Mon, Jul 17, 2017 at 8:05 AM, Flavio Percoco  wrote:

Thanks for all the feedback so far. This is one of the things I appreciate
the
most about this community, Open conversations, honest feedback and will to
collaborate.

I'm top-posting to announce that we'll have a joint meeting with the Kolla
team
on Wednesday at 16:00 UTC. I know it's not an ideal time for many (it's not
for
me) but I do want to have a live discussion with the rest of the Kolla team.

Some questions about the meeting:

* How much time can we allocate?
* Can we prepare an agenda rather than just discussing "TripleO is thinking
of
 using Ansible and not kolla-kubernetes"? (I'm happy to come up with such
 agenda)


It may help to prepare some high level requirements around what we
need out of a solution. For the ansible discussion I started this
etherpad:

https://etherpad.openstack.org/p/tripleo-ptg-queens-ansible

How we use Ansible and what we want to use it for, is related to this
discussion around Helm. Although, it's not the exact same discussion,
so if you wanted to start a new etherpad more specific to
tripleo/kubernetes that may be good as well.

One thing I think is important in this discussion is that we should be
thinking about deploying containers on both Kubernetes and
!Kubernetes. That is one of the reasons I like the ansible approach,
in that I think it could address both cases with a common interface
and API. I don't think we should necessarily choose a solution that
requires to deploy on Kubernetes. Because then we are stuck with that
choice. It'd be really nice to just "docker run" sometimes for
dev/test. I don't know if Helm has that abstraction or not, I'm just
trying to capture the requirement.


Yes!

Thanks for pointing this out as this is one of the reasons why I was proposing
ansible as our common interface w/o any extra layer.

I'll probably start a new etherpad for this as I would prefer not to distract
the rest of the TripleO + ansible discussion. At the end, if ansible ends up
being the tool we pick, I'll make sure to update your etherpad.

Flavio


If you consider the parallel with Heat in this regard, we are
currently "stuck" deploying on OpenStack (undercloud with Heat). We've
had to work an a lot of complimentary features to add the flexibility
to TripleO that are a result of having to use OpenStack (OVB,
split-stack).

That's exactly why we are starting a discussion around using Ansible,
and is one of the fundamental changes that operators have been
requesting in TripleO.

--
-- James Slagle
--

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[James Slagle]

On Mon, Jul 17, 2017 at 8:05 AM, Flavio Percoco  wrote:
> Thanks for all the feedback so far. This is one of the things I appreciate
> the
> most about this community, Open conversations, honest feedback and will to
> collaborate.
>
> I'm top-posting to announce that we'll have a joint meeting with the Kolla
> team
> on Wednesday at 16:00 UTC. I know it's not an ideal time for many (it's not
> for
> me) but I do want to have a live discussion with the rest of the Kolla team.
>
> Some questions about the meeting:
>
> * How much time can we allocate?
> * Can we prepare an agenda rather than just discussing "TripleO is thinking
> of
>  using Ansible and not kolla-kubernetes"? (I'm happy to come up with such
>  agenda)

It may help to prepare some high level requirements around what we
need out of a solution. For the ansible discussion I started this
etherpad:

https://etherpad.openstack.org/p/tripleo-ptg-queens-ansible

How we use Ansible and what we want to use it for, is related to this
discussion around Helm. Although, it's not the exact same discussion,
so if you wanted to start a new etherpad more specific to
tripleo/kubernetes that may be good as well.

One thing I think is important in this discussion is that we should be
thinking about deploying containers on both Kubernetes and
!Kubernetes. That is one of the reasons I like the ansible approach,
in that I think it could address both cases with a common interface
and API. I don't think we should necessarily choose a solution that
requires to deploy on Kubernetes. Because then we are stuck with that
choice. It'd be really nice to just "docker run" sometimes for
dev/test. I don't know if Helm has that abstraction or not, I'm just
trying to capture the requirement.

If you consider the parallel with Heat in this regard, we are
currently "stuck" deploying on OpenStack (undercloud with Heat). We've
had to work an a lot of complimentary features to add the flexibility
to TripleO that are a result of having to use OpenStack (OVB,
split-stack).

That's exactly why we are starting a discussion around using Ansible,
and is one of the fundamental changes that operators have been
requesting in TripleO.

-- 
-- James Slagle
--

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Flavio Percoco]

On 14/07/17 08:08 -0700, Emilien Macchi wrote:

On Fri, Jul 14, 2017 at 2:17 AM, Flavio Percoco  wrote:


Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment, OpenStack
deployment on Kubernetes, configuration management, etc. While I've been
diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these tools
and
I've come to the conclusion that TripleO would be better of by having
ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm. While
I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects,
I
believe using any of them would add an extra layer of complexity to TripleO,
which is something the team has been fighting for years years - especially
now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would
require
TripleO to also write the logic to manage those projects. For example, in
the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the charts
(I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the
conclusion
I reached.

Now, what this work means is that we would have to write an ansible role for
each service that will deploy the service on a Kubernetes cluster. Ideally
these
roles will also generate the configuration files (removing the need of
puppet
entirely) and they would manage the lifecycle. The roles would be isolated
and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain these
roles
and run them using the existing docker-cmd implementation that is coming out
in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the discussion
and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that ansible
is
a known, powerfull, tool that has been adopted by many operators already.
It'll
provide the flexibility needed and, if structured correctly, it'll allow for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate
concerns
in the deployment workflow and the idea of making it simple for users of
TripleO
to do the same at runtime. Unfortunately, going down this road means that my
hope of creating a field where we could collaborate even more with other
deployment tools will be a bit limited but I'm confident the result would
also
be useful for others and that we all will benefit from it... My hopes might
be a
bit naive *shrugs*


Of course I'm biased since I've been (a little) involved in that work
but I like the idea of :

- Moving forward with our containerization. docker-cmd will help us
for sure for this transition (I insist on the fact TripleO is a
product that you can upgrade and we try to make it smooth for our
operators), so we can't just trash everything and switch to a new
tool. I think the approach that we're taking is great and made of baby
steps where we try to solve different problems.
- Using more Ansible - the right way - when it makes sense : with the
TripleO containerization, we only use Puppet for Configuration
Management, managing a few resources but not for orchestration (or not
all the features that Puppet provide) and for Data Binding (Hiera). To
me, it doesn't make sense for us to keep investing much in Puppet
modules if we go k8s & Ansible. That said, see the next point.
- Having a transition path between TripleO with Puppet and TripleO
with apbs and have some sort of binding between previous hieradata
generated by TripleO & a similar data binding within Ansible playbooks
would help. I saw your PoC Flavio, I found it great and I think we
should make 
https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/hiera.yaml
optional when running apbs, and allow to provide another format (more
Ansiblish) to let folks not 

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Flavio Percoco]

Thanks for all the feedback so far. This is one of the things I appreciate the
most about this community, Open conversations, honest feedback and will to
collaborate.

I'm top-posting to announce that we'll have a joint meeting with the Kolla team
on Wednesday at 16:00 UTC. I know it's not an ideal time for many (it's not for
me) but I do want to have a live discussion with the rest of the Kolla team.

Some questions about the meeting:

* How much time can we allocate?
* Can we prepare an agenda rather than just discussing "TripleO is thinking of
 using Ansible and not kolla-kubernetes"? (I'm happy to come up with such
 agenda)

One last point. I'm not interested in conversations around competition,
re-invention, etc. I think I speak for the entire TripleO team when I say that
this is not about "winning" in this space but rather seeing how/if we can
collaborate and how/if it makes sense to keep exploring the path described in
the email below.

Flavio

On 14/07/17 11:17 +0200, Flavio Percoco wrote:


Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment, OpenStack
deployment on Kubernetes, configuration management, etc. While I've been diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these tools and
I've come to the conclusion that TripleO would be better of by having ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm. While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects, I
believe using any of them would add an extra layer of complexity to TripleO,
which is something the team has been fighting for years years - especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would require
TripleO to also write the logic to manage those projects. For example, in the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the charts (I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the conclusion
I reached.

Now, what this work means is that we would have to write an ansible role for
each service that will deploy the service on a Kubernetes cluster. Ideally these
roles will also generate the configuration files (removing the need of puppet
entirely) and they would manage the lifecycle. The roles would be isolated and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain these roles
and run them using the existing docker-cmd implementation that is coming out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the discussion and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that ansible is
a known, powerfull, tool that has been adopted by many operators already. It'll
provide the flexibility needed and, if structured correctly, it'll allow for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate concerns
in the deployment workflow and the idea of making it simple for users of TripleO
to do the same at runtime. Unfortunately, going down this road means that my
hope of creating a field where we could collaborate even more with other
deployment tools will be a bit limited but I'm confident the result would also
be useful for others and that we all will benefit from it... My hopes might be a
bit naive *shrugs*

Flavio

[0] http://lists.openstack.org/pipermail/openstack-dev/2017-July/119405.html
[1] https://github.com/tripleo-apb/tripleo-apbs

--
@flaper87
Flavio Percoco




--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Jiří Stránský]

On 14.7.2017 23:00, Ben Nemec wrote:



On 07/14/2017 11:43 AM, Joshua Harlow wrote:

Out of curiosity, since I keep on hearing/reading all the tripleo
discussions on how tripleo folks are apparently thinking/doing?
redesigning the whole thing to use ansible + mistral + heat, or ansible
+ kubernetes or ansible + mistral + heat + ansible (a second time!) or ...

Seeing all those kinds of questions and suggestions around what should
be used and why and how (and even this thread) makes me really wonder
who actually uses tripleo and can afford/understand such kinds of changes?

Does anyone?

If there are  is there going to be an upgrade
path for there existing 'cloud/s' to whatever this solution is?

What operator(s) has the ability to do such a massive shift at this
point in time? Who are these 'mystical' operators?

All this has really peaked my curiosity because I am personally trying
to do that shift (not exactly the same solution...) and I know it is a
massive undertaking (that will take quite a while to get right) even for
a simple operator with limited needs out of openstack (ie godaddy); so I
don't really understand how the generic solution for all existing
tripleo operators can even work...


This is a valid point.  Up until now the answer has been that we
abstracted most of the ugliness of major changes behind either Heat or
tripleoclient.  If we end up essentially dropping those two in favor of
some other method of driving deployments it's going to be a lot harder
to migrate.  And I could be wrong, but I'm pretty sure it _is_ important
to our users to have an in-place upgrade path (see the first bullet
point in [1]).

New, shiny technology is great and all, but we do need to remember that
we have a lot of users out there already depending on the old,
not-so-shiny bits too.  They're not going to be happy if we leave them
hanging.


Exactly. Reuse is nice to have, while some sort of an upgrade path is a 
must have. We should be aware of this when selecting tools for Kubernetes.


Jirka



1: http://lists.openstack.org/pipermail/openstack-dev/2017-June/119063.html



Flavio Percoco wrote:


Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment,
OpenStack
deployment on Kubernetes, configuration management, etc. While I've been
diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these
tools and
I've come to the conclusion that TripleO would be better of by having
ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm.
While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack
projects, I
believe using any of them would add an extra layer of complexity to
TripleO,
which is something the team has been fighting for years years -
especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would
require
TripleO to also write the logic to manage those projects. For example,
in the
case of openstack-helm, the TripleO team would have to write either
ansible
roles or heat templates to manage - install, remove, upgrade - the
charts (I'm
happy to discuss this point further but I'm keepping it at a
high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO
plans
around ansible. One take-away from this thread is that TripleO is
adopting
ansible more and more, which is great and it fits perfectly with the
conclusion
I reached.

Now, what this work means is that we would have to write an ansible role
for
each service that will deploy the service on a Kubernetes cluster.
Ideally these
roles will also generate the configuration files (removing the need of
puppet
entirely) and they would manage the lifecycle. The roles would be
isolated and
this will reduce the need of TripleO Heat templates. Doing this would
give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain
these roles
and run them using the existing docker-cmd implementation that is coming
out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the
discussion and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that
ansible is
a known, powerfull, tool that has been adopted by many operators
already. It'll
provide the flexibility needed and, if 

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Bogdan Dobrelya]

On 14.07.2017 22:55, Fox, Kevin M wrote:
> Part of the confusion I think is in the different ways helm can be used.
> 
> Helm can be used to orchestrate the deployment of a whole service (ex, nova). 
> "launch these 3 k8s objects, template out this config file, run this job to 
> init the db, or this job to upgrade the db, etc", all as a single unit.
> 
> It can also be used purely for its templating ability.
> 
> So, "render this single k8s object using these values".
> 
> This is one of the main differences between openstack-helm and 
> kolla-kubernetes.
> 
> Openstack-helm has charts only for orchestrating the deployment of whole 
> openstack services.
> 
> Kolla-kubernetes has taken a different track though. While it does use helm 
> for its golang templater, it has taken a microservices approach to be 
> shareable with other tools. So, each openstack process (nova-api, 
> neutron-server, neutron-openvswitch-agent), etc, has its own chart and can be 
> independently configured/placed as needed by an external orchestration 
> system. Kolla-Kubernetes microservice charts are to Kubernetes what 
> Kolla-Containers are to Docker. Reusable building blocks of known tested 
> functionality and assemblable anyway the orchestration system/user feels is 
> in their best interest.

A great summary!
As TripleO Pike docker-based containers architecture rely on
Kolla-Containers bits a lot, which is run-time kolla config/bootstrap
and build-time images overrides, it seems reasonable to continue
following that path by relying on Kolla-Kubernetes microservice Helm
charts for Kubernetes based architecture. Isn't it?

The remaining question is though, if Kolla-kubernetes doesn't consume
the Openstack-helm's opinionated "orchestration of the deployment of
whole openstack services", which tools to use then to fill the advanced
data parameterization gaps like "happens before/after" relationships and
data dependencies/ordering?

> 
> This is why I think kolla-kubernetes would be a good fit for TripleO, as you 
> can replace a single component at a time, however you want, using the config 
> files you already have and upgrade the system a piece at a time from non 
> container to containered. It doesn't have to happen all at once, even within 
> a single service, or within a single TripleO release. The orchestration of it 
> is totally up to you, and can be tailored very precisely to deal with the 
> particulars of the upgrade strategy needed by TripleO's existing deployments.
> 
> Does that help to alleviate some of the confusion?
> 
> Thanks,
> Kevin


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Ben Nemec]

On 07/14/2017 11:43 AM, Joshua Harlow wrote:
Out of curiosity, since I keep on hearing/reading all the tripleo 
discussions on how tripleo folks are apparently thinking/doing? 
redesigning the whole thing to use ansible + mistral + heat, or ansible 
+ kubernetes or ansible + mistral + heat + ansible (a second time!) or ...


Seeing all those kinds of questions and suggestions around what should 
be used and why and how (and even this thread) makes me really wonder 
who actually uses tripleo and can afford/understand such kinds of changes?


Does anyone?

If there are  is there going to be an upgrade 
path for there existing 'cloud/s' to whatever this solution is?


What operator(s) has the ability to do such a massive shift at this 
point in time? Who are these 'mystical' operators?


All this has really peaked my curiosity because I am personally trying 
to do that shift (not exactly the same solution...) and I know it is a 
massive undertaking (that will take quite a while to get right) even for 
a simple operator with limited needs out of openstack (ie godaddy); so I 
don't really understand how the generic solution for all existing 
tripleo operators can even work...


This is a valid point.  Up until now the answer has been that we 
abstracted most of the ugliness of major changes behind either Heat or 
tripleoclient.  If we end up essentially dropping those two in favor of 
some other method of driving deployments it's going to be a lot harder 
to migrate.  And I could be wrong, but I'm pretty sure it _is_ important 
to our users to have an in-place upgrade path (see the first bullet 
point in [1]).


New, shiny technology is great and all, but we do need to remember that 
we have a lot of users out there already depending on the old, 
not-so-shiny bits too.  They're not going to be happy if we leave them 
hanging.


1: http://lists.openstack.org/pipermail/openstack-dev/2017-June/119063.html



Flavio Percoco wrote:


Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment,
OpenStack
deployment on Kubernetes, configuration management, etc. While I've been
diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these
tools and
I've come to the conclusion that TripleO would be better of by having
ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm.
While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack
projects, I
believe using any of them would add an extra layer of complexity to
TripleO,
which is something the team has been fighting for years years -
especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would
require
TripleO to also write the logic to manage those projects. For example,
in the
case of openstack-helm, the TripleO team would have to write either 
ansible

roles or heat templates to manage - install, remove, upgrade - the
charts (I'm
happy to discuss this point further but I'm keepping it at a 
high-level on

purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO 
plans
around ansible. One take-away from this thread is that TripleO is 
adopting

ansible more and more, which is great and it fits perfectly with the
conclusion
I reached.

Now, what this work means is that we would have to write an ansible role
for
each service that will deploy the service on a Kubernetes cluster.
Ideally these
roles will also generate the configuration files (removing the need of
puppet
entirely) and they would manage the lifecycle. The roles would be
isolated and
this will reduce the need of TripleO Heat templates. Doing this would 
give

TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain
these roles
and run them using the existing docker-cmd implementation that is coming
out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the
discussion and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that
ansible is
a known, powerfull, tool that has been adopted by many operators
already. It'll
provide the flexibility needed and, if structured correctly, it'll allow
for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to 

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Fox, Kevin M]

Part of the confusion I think is in the different ways helm can be used.

Helm can be used to orchestrate the deployment of a whole service (ex, nova). 
"launch these 3 k8s objects, template out this config file, run this job to 
init the db, or this job to upgrade the db, etc", all as a single unit.

It can also be used purely for its templating ability.

So, "render this single k8s object using these values".

This is one of the main differences between openstack-helm and kolla-kubernetes.

Openstack-helm has charts only for orchestrating the deployment of whole 
openstack services.

Kolla-kubernetes has taken a different track though. While it does use helm for 
its golang templater, it has taken a microservices approach to be shareable 
with other tools. So, each openstack process (nova-api, neutron-server, 
neutron-openvswitch-agent), etc, has its own chart and can be independently 
configured/placed as needed by an external orchestration system. 
Kolla-Kubernetes microservice charts are to Kubernetes what Kolla-Containers 
are to Docker. Reusable building blocks of known tested functionality and 
assemblable anyway the orchestration system/user feels is in their best 
interest.

This is why I think kolla-kubernetes would be a good fit for TripleO, as you 
can replace a single component at a time, however you want, using the config 
files you already have and upgrade the system a piece at a time from non 
container to containered. It doesn't have to happen all at once, even within a 
single service, or within a single TripleO release. The orchestration of it is 
totally up to you, and can be tailored very precisely to deal with the 
particulars of the upgrade strategy needed by TripleO's existing deployments.

Does that help to alleviate some of the confusion?

Thanks,
Kevin

From: James Slagle [james.sla...@gmail.com]
Sent: Friday, July 14, 2017 10:26 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack 
services on Kubernetes

On Fri, Jul 14, 2017 at 12:16 PM, Fox, Kevin M <kevin@pnnl.gov> wrote:
> https://xkcd.com/927/

That's cute, but we aren't really trying to have competing standards.
It's not really about competition between tools.

> I don't think adopting helm as a dependency adds more complexity then writing 
> more new k8s object deployment tooling?

That depends, and will likely end up containing a fair amount of
subjectivity. What we're trying to do is explore choices around
tooling.

>
> There are efforts to make it easy to deploy kolla-kubernetes microservice 
> charts using ansible for orchestration in kolla-kubernetes. See:
> https://review.openstack.org/#/c/473588/
> What kolla-kubernetes brings to the table is a tested/shared base k8s object 
> layer. Orchestration is done by ansible via TripleO, and the solutions 
> already found/debugged to how to deploy OpenStack in containers on Kubernetes 
> can be reused/shared.

That's good, and we'd like to reuse existing code and patterns. I
admit to not being super famliliar with kolla-kubernetes. Are there
reusable components without having to also use Helm?

> See for example:
> https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/main.yaml

Pretty sure that was just a POC/example.

>
> I don't see much by way of dealing with fernet token rotation. That was a 
> tricky bit of code to get to work, but kolla-kubernetes has a solution to it. 
> You can get it by: helm install kolla/keystone-fernet-rotate-job.
>
> We designed this layer to be shareable so we all can contribute to the 
> commons rather then having every project reimplement their own and have to 
> chase bugs across all the implementations. The deployment projects will be 
> stronger together if we can share as much as possible.
>
> Please reconsider. I'd be happy to talk with you more if you want.

Just to frame the conversation with a bit more context, I'm sure there
are many individual features/bugs/special handling that TripleO and
Kolla both do today that the other does not.

TripleO had about a 95% solution for deploying OpenStack when
kolla-ansible did not exist and was started from scratch. But, kolla
made a choice based around tooling, which I contend is perfectly valid
given that we are creating deployment tools. Part of the individual
value in each deployment project is the underlying tooling itself.

I think what TripleO is trying to do here is not immediately jump to a
solution that uses Helm and explore what alternatives exist. Even if
the project chooses not to use Helm I still see room for collaboration
on code beneath the Helm/whatever layer.

--
-- James Slagle
--

_

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[James Slagle]

On Fri, Jul 14, 2017 at 3:38 PM, Steven Dake  wrote:
>
>
> On Fri, Jul 14, 2017 at 10:26 AM, James Slagle 
> wrote:
>>
> James,
>
>>
>> Just to frame the conversation with a bit more context, I'm sure there
>> are many individual features/bugs/special handling that TripleO and
>> Kolla both do today that the other does not.
>>
>
> I think what you are saying in a nutshell is that TripleO and Kolla compete.

No. That is not what I'm saying. In fact I said:

 It's not really about competition between tools.

I'm not sure how you thought that meant I was saying that the two tools compete.

Some may consider that to be the case (that they compete), but that is
more a personal frame of reference. I don't think that either project
is trying to "win" the deployment battle. Or there even is a battle.
If that were the case, it would be very difficult to work together, as
we do effectively quite a bit today already.

>> TripleO had about a 95% solution for deploying OpenStack when
>> kolla-ansible did not exist and was started from scratch. But, kolla
>> made a choice based around tooling, which I contend is perfectly valid
>> given that we are creating deployment tools. Part of the individual
>> value in each deployment project is the underlying tooling itself.
>>
>
> I think what you are saying here is Kolla chose to compete on tooling.  I
> haven't really given it a lot of thought; I'd say all are technical choices
> made with Kolla had mostly to do with selecting wisely from the technical
> ecosystem.

No. What I'm saying is exactly what I wrote. Please don't read or
project anything else onto it about "competition".

Again, I don't think that is all that relevant or healthy to the
conversation (hence why I dismissed the comic: it's a farce of the
actual situation).

I see it more as differentiation instead of competition.  Especially
since we are talking about open source projects. There are advantages
and disadvantages to every tool choice, including Heat vs Ansible.
What I said was that "kolla made a choice based around tooling". And
that is a valid thing to do and creates individual value to that
project that differentiates it from TripleO.

>> I think what TripleO is trying to do here is not immediately jump to a
>> solution that uses Helm and explore what alternatives exist. Even if
>> the project chooses not to use Helm I still see room for collaboration
>> on code beneath the Helm/whatever layer.
>>
>
> I believe it wise that you don't jump to any conclusion or solution that
> does or doesn't use Helm.  I'd encourage you to understand how Kubernetes
> works before making such technical choices.

Exactly. Which is why "just use kolla-kubernetes" is not a silver
bullet to this discussion.

> All that said, there is clearly value in working together rather than apart.
> To me, that is more important then the technical choices you are presented
> with.

-- 
-- James Slagle
--

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Steven Dake]

On Fri, Jul 14, 2017 at 10:26 AM, James Slagle 
wrote:

> On Fri, Jul 14, 2017 at 12:16 PM, Fox, Kevin M  wrote:
> > https://xkcd.com/927/
>
> That's cute, but we aren't really trying to have competing standards.
> It's not really about competition between tools.
>
> > I don't think adopting helm as a dependency adds more complexity then
> writing more new k8s object deployment tooling?
>
> That depends, and will likely end up containing a fair amount of
> subjectivity. What we're trying to do is explore choices around
> tooling.
>
> >
> > There are efforts to make it easy to deploy kolla-kubernetes
> microservice charts using ansible for orchestration in kolla-kubernetes.
> See:
> > https://review.openstack.org/#/c/473588/
> > What kolla-kubernetes brings to the table is a tested/shared base k8s
> object layer. Orchestration is done by ansible via TripleO, and the
> solutions already found/debugged to how to deploy OpenStack in containers
> on Kubernetes can be reused/shared.
>
> That's good, and we'd like to reuse existing code and patterns. I
> admit to not being super famliliar with kolla-kubernetes. Are there
> reusable components without having to also use Helm?
>
> > See for example:
> > https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/
> 331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-
> apb/tasks/main.yaml
>
> Pretty sure that was just a POC/example.
>
> >
> > I don't see much by way of dealing with fernet token rotation. That was
> a tricky bit of code to get to work, but kolla-kubernetes has a solution to
> it. You can get it by: helm install kolla/keystone-fernet-rotate-job.
> >
> > We designed this layer to be shareable so we all can contribute to the
> commons rather then having every project reimplement their own and have to
> chase bugs across all the implementations. The deployment projects will be
> stronger together if we can share as much as possible.
> >
> > Please reconsider. I'd be happy to talk with you more if you want.
>
> James,


> Just to frame the conversation with a bit more context, I'm sure there
> are many individual features/bugs/special handling that TripleO and
> Kolla both do today that the other does not.
>
>
I think what you are saying in a nutshell is that TripleO and Kolla compete.


> TripleO had about a 95% solution for deploying OpenStack when
> kolla-ansible did not exist and was started from scratch. But, kolla
> made a choice based around tooling, which I contend is perfectly valid
> given that we are creating deployment tools. Part of the individual
> value in each deployment project is the underlying tooling itself.
>
>
I think what you are saying here is Kolla chose to compete on tooling.  I
haven't really given it a lot of thought; I'd say all are technical choices
made with Kolla had mostly to do with selecting wisely from the technical
ecosystem.


> I think what TripleO is trying to do here is not immediately jump to a
> solution that uses Helm and explore what alternatives exist. Even if
> the project chooses not to use Helm I still see room for collaboration
> on code beneath the Helm/whatever layer.
>
>
I believe it wise that you don't jump to any conclusion or solution that
does or doesn't use Helm.  I'd encourage you to understand how Kubernetes
works before making such technical choices.

All that said, there is clearly value in working together rather than
apart.  To me, that is more important then the technical choices you are
presented with.

Regards
-steve


> --
> -- James Slagle
> --
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Juan Antonio Osorio]

I actually like the idea of moving to kolla-kubernetes. I guess there would
be a bunch of work towards giving folks an upgrade path and reaching
feature parity; but this would happen anyway eurgh the switch to
kubernetes.  And this would have the added value of merging two
communities, thus more devs and folks testing :D . I like it!

On 14 Jul 2017 18:56, "Michał Jastrzębski"  wrote:

Guys you just described Kolla-Kubernetes pretty much... how about
we join effort and work towards this goal together?

On 14 July 2017 at 08:43, Flavio Percoco  wrote:
> On 14/07/17 17:26 +0200, Bogdan Dobrelya wrote:
>>
>> On 14.07.2017 11:17, Flavio Percoco wrote:
>>>
>>>
>>> Greetings,
>>>
>>> As some of you know, I've been working on the second phase of TripleO's
>>> containerization effort. This phase if about migrating the docker based
>>> deployment onto Kubernetes.
>>>
>>> These phase requires work on several areas: Kubernetes deployment,
>>> OpenStack
>>> deployment on Kubernetes, configuration management, etc. While I've been
>>> diving
>>> into all of these areas, this email is about the second point, OpenStack
>>> deployment on Kubernetes.
>>>
>>> There are several tools we could use for this task. kolla-kubernetes,
>>> openstack-helm, ansible roles, among others. I've looked into these
>>> tools and
>>> I've come to the conclusion that TripleO would be better of by having
>>> ansible
>>> roles that would allow for deploying OpenStack services on Kubernetes.
>>>
>>> The existing solutions in the OpenStack community require using Helm.
>>> While I
>>> like Helm and both, kolla-kubernetes and openstack-helm OpenStack
>>> projects, I
>>> believe using any of them would add an extra layer of complexity to
>>> TripleO,
>>
>>
>> It's hard to estimate that complexity w/o having a PoC of such an
>> integration. We should come up with a final choice once we have it done.
>>
>> My vote would go for investing engineering resources into solutions that
>> have problems already solved, even by the price of added complexity (but
>> that sort of depends...). Added complexity may be compensated with
>> removed complexity (like those client -> Mistral -> Heat -> Mistral ->
>> Ansible manipulations discussed in the mail thread mentioned below [0])
>
>
> I agree it's hard to estimate but you gotta draw the line somewhere. I
> actually
> spent time on this and here's a small PoC of ansible+mariadb+helm. I wrote
> the
> pyhelm lib (took some code from the openstack-helm folks) and I wrote the
> ansible helm module myself. I'd say I've spent enough time on this
research.
>
> I don't think getting a full PoC working is worth it as that will require
> way
> more work for not much value since we can anticipate some of the
> complexities
> already.
>
> As far as the complexity comment goes, I disagree with you. I don't think
> you're
> evaluating the amount of complexity that there *IS* already in TripleO and
> how
> adding more complexity (layers, states, services) would make things worse
> for
> not much extra value.
>
> By all means, I might be wrong here so, do let me know if you're seeing
> something I'm not.
> Flavio
> --
> @flaper87
> Flavio Percoco
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[James Slagle]

On Fri, Jul 14, 2017 at 12:16 PM, Fox, Kevin M  wrote:
> https://xkcd.com/927/

That's cute, but we aren't really trying to have competing standards.
It's not really about competition between tools.

> I don't think adopting helm as a dependency adds more complexity then writing 
> more new k8s object deployment tooling?

That depends, and will likely end up containing a fair amount of
subjectivity. What we're trying to do is explore choices around
tooling.

>
> There are efforts to make it easy to deploy kolla-kubernetes microservice 
> charts using ansible for orchestration in kolla-kubernetes. See:
> https://review.openstack.org/#/c/473588/
> What kolla-kubernetes brings to the table is a tested/shared base k8s object 
> layer. Orchestration is done by ansible via TripleO, and the solutions 
> already found/debugged to how to deploy OpenStack in containers on Kubernetes 
> can be reused/shared.

That's good, and we'd like to reuse existing code and patterns. I
admit to not being super famliliar with kolla-kubernetes. Are there
reusable components without having to also use Helm?

> See for example:
> https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/main.yaml

Pretty sure that was just a POC/example.

>
> I don't see much by way of dealing with fernet token rotation. That was a 
> tricky bit of code to get to work, but kolla-kubernetes has a solution to it. 
> You can get it by: helm install kolla/keystone-fernet-rotate-job.
>
> We designed this layer to be shareable so we all can contribute to the 
> commons rather then having every project reimplement their own and have to 
> chase bugs across all the implementations. The deployment projects will be 
> stronger together if we can share as much as possible.
>
> Please reconsider. I'd be happy to talk with you more if you want.

Just to frame the conversation with a bit more context, I'm sure there
are many individual features/bugs/special handling that TripleO and
Kolla both do today that the other does not.

TripleO had about a 95% solution for deploying OpenStack when
kolla-ansible did not exist and was started from scratch. But, kolla
made a choice based around tooling, which I contend is perfectly valid
given that we are creating deployment tools. Part of the individual
value in each deployment project is the underlying tooling itself.

I think what TripleO is trying to do here is not immediately jump to a
solution that uses Helm and explore what alternatives exist. Even if
the project chooses not to use Helm I still see room for collaboration
on code beneath the Helm/whatever layer.

-- 
-- James Slagle
--

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Clint Byrum]

Excerpts from Bogdan Dobrelya's message of 2017-07-14 18:14:42 +0200:
> On 14.07.2017 17:55, Michał Jastrzębski wrote:
> > Guys you just described Kolla-Kubernetes pretty much... how about
> > we join effort and work towards this goal together?
> 
> That's exactly that I'd like we all to do.
> 

Agreed, and ...

> > 
> > On 14 July 2017 at 08:43, Flavio Percoco  wrote:
> >> On 14/07/17 17:26 +0200, Bogdan Dobrelya wrote:
> >>>
> >>> On 14.07.2017 11:17, Flavio Percoco wrote:
> 
> 
>  Greetings,
> 
>  As some of you know, I've been working on the second phase of TripleO's
>  containerization effort. This phase if about migrating the docker based
>  deployment onto Kubernetes.
> 
>  These phase requires work on several areas: Kubernetes deployment,
>  OpenStack
>  deployment on Kubernetes, configuration management, etc. While I've been
>  diving
>  into all of these areas, this email is about the second point, OpenStack
>  deployment on Kubernetes.
> 
>  There are several tools we could use for this task. kolla-kubernetes,
>  openstack-helm, ansible roles, among others. I've looked into these
>  tools and
>  I've come to the conclusion that TripleO would be better of by having
>  ansible
>  roles that would allow for deploying OpenStack services on Kubernetes.
> 
>  The existing solutions in the OpenStack community require using Helm.
>  While I
>  like Helm and both, kolla-kubernetes and openstack-helm OpenStack
>  projects, I
>  believe using any of them would add an extra layer of complexity to
>  TripleO,
> >>>
> >>>
> >>> It's hard to estimate that complexity w/o having a PoC of such an
> >>> integration. We should come up with a final choice once we have it done.
> >>>
> >>> My vote would go for investing engineering resources into solutions that
> >>> have problems already solved, even by the price of added complexity (but
> >>> that sort of depends...). Added complexity may be compensated with
> >>> removed complexity (like those client -> Mistral -> Heat -> Mistral ->
> >>> Ansible manipulations discussed in the mail thread mentioned below [0])
> >>
> >>
> >> I agree it's hard to estimate but you gotta draw the line somewhere. I
> >> actually
> >> spent time on this and here's a small PoC of ansible+mariadb+helm. I wrote
> >> the
> >> pyhelm lib (took some code from the openstack-helm folks) and I wrote the
> >> ansible helm module myself. I'd say I've spent enough time on this 
> >> research.
> >>
> >> I don't think getting a full PoC working is worth it as that will require
> >> way
> >> more work for not much value since we can anticipate some of the
> >> complexities
> >> already.
> >>
> >> As far as the complexity comment goes, I disagree with you. I don't think
> >> you're
> >> evaluating the amount of complexity that there *IS* already in TripleO and
> >> how
> >> adding more complexity (layers, states, services) would make things worse
> >> for
> >> not much extra value.
> >>
> >> By all means, I might be wrong here so, do let me know if you're seeing
> >> something I'm not.
> 
> My point was to "trade" complexity described in the "Forming our plans
> around Ansible​" ML thread:
> 
> (3) Mistral calling Heat calling Mistral calling Ansible
> 
> to just
> 
> (3') something calls kolla-kubernetes/openstack-helm, via some wrapper
> composition overlay (which creates complexity), or the like
> 
> While the latter might add complexity like the way you (Flavio) have
> described, the former would remove *another* type of complexity, and the
> result might worth the efforts.
>

The two options seem to be

a. Bootstrap helm and charts and then use openstack-helm/kolla-kubernetes
b. Bootstrap (something) and then use newly minted ansible to manipulate
   kubernetes.

(a) seems like less net complexity, as bootstrapping code is usually
able to be more naive. The new ansible will have to be at least as good
as openstack-helm and kolla-kubernetes, and still needs bootstraps of
its own.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Dmitry Tantsur]

On 07/14/2017 06:16 PM, Fox, Kevin M wrote:

https://xkcd.com/927/

I don't think adopting helm as a dependency adds more complexity then writing 
more new k8s object deployment tooling?


I don't know much about the containerization work, and I don't have a big say in 
TripleO, but that's the question I have as well. If we are going now to rewrite 
ansible modules for everything (including MariaDB per Emilien's comment), this 
may require too much effort.


Think of TripleO contributors, who are not on tripleo-core (the group which 
probably contains 99% of people understanding TripleO well). Writing heat 
templates is already not fun, but at least people got used to it more or less. 
Now we will need to rewrite a lot of puppet into a lot of ansible, and a lot of 
yaml into... mmm.. more ansible? If we go down this way, let's at least make 
sure we're not inventing a bicycle.




There are efforts to make it easy to deploy kolla-kubernetes microservice 
charts using ansible for orchestration in kolla-kubernetes. See:
https://review.openstack.org/#/c/473588/
What kolla-kubernetes brings to the table is a tested/shared base k8s object 
layer. Orchestration is done by ansible via TripleO, and the solutions already 
found/debugged to how to deploy OpenStack in containers on Kubernetes can be 
reused/shared.

See for example:
https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/main.yaml

I don't see much by way of dealing with fernet token rotation. That was a 
tricky bit of code to get to work, but kolla-kubernetes has a solution to it. 
You can get it by: helm install kolla/keystone-fernet-rotate-job.

We designed this layer to be shareable so we all can contribute to the commons 
rather then having every project reimplement their own and have to chase bugs 
across all the implementations. The deployment projects will be stronger 
together if we can share as much as possible.


+++



Please reconsider. I'd be happy to talk with you more if you want.

Thanks,
Kevin

From: Flavio Percoco [fla...@redhat.com]
Sent: Friday, July 14, 2017 2:17 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack 
services on Kubernetes

Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment, OpenStack
deployment on Kubernetes, configuration management, etc. While I've been diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these tools and
I've come to the conclusion that TripleO would be better of by having ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm. While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects, I
believe using any of them would add an extra layer of complexity to TripleO,
which is something the team has been fighting for years years - especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would require
TripleO to also write the logic to manage those projects. For example, in the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the charts (I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the conclusion
I reached.

Now, what this work means is that we would have to write an ansible role for
each service that will deploy the service on a Kubernetes cluster. Ideally these
roles will also generate the configuration files (removing the need of puppet
entirely) and they would manage the lifecycle. The roles would be isolated and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain these roles
and run them using the existing docker-cmd implementation that is coming out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the discussion and
gather thoughts and opinions from the rest of the community

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Flavio Percoco]

First and foremost I just realized that I forgot to tag kolla and openstack-helm
in the subject so, I apologize. I'm glad the subject was catchy enough to get
your attention.

Just want to raise here what I just mentioned on IRC:

It's late in EU so I shouldn't be here right now but, I do want to point out
that, as usual, I asked for feedback and clarifications from everyone in this
thread.

I'm not trying to re-invent the wheel. What's in my original email is my
conclusion based on a research I did across the different tools there are. I
can, of course, be wrong and I'd like you all to help us by providing feedback.

I'm not expecting sales pitches but I'd love to have a more technical discussion
on how we can, hopefully, make this work.

On 14/07/17 16:16 +, Fox, Kevin M wrote:

https://xkcd.com/927/

I don't think adopting helm as a dependency adds more complexity then writing 
more new k8s object deployment tooling?

There are efforts to make it easy to deploy kolla-kubernetes microservice 
charts using ansible for orchestration in kolla-kubernetes. See:
https://review.openstack.org/#/c/473588/
What kolla-kubernetes brings to the table is a tested/shared base k8s object 
layer. Orchestration is done by ansible via TripleO, and the solutions already 
found/debugged to how to deploy OpenStack in containers on Kubernetes can be 
reused/shared.

See for example:
https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/main.yaml

I don't see much by way of dealing with fernet token rotation. That was a 
tricky bit of code to get to work, but kolla-kubernetes has a solution to it. 
You can get it by: helm install kolla/keystone-fernet-rotate-job.


It's just a PoC, don't take the implementation as definitive.


We designed this layer to be shareable so we all can contribute to the commons 
rather then having every project reimplement their own and have to chase bugs 
across all the implementations. The deployment projects will be stronger 
together if we can share as much as possible.

Please reconsider. I'd be happy to talk with you more if you want.


Let's talk, that's the whole point of this thread.
Flavio

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Joshua Harlow]

Out of curiosity, since I keep on hearing/reading all the tripleo 
discussions on how tripleo folks are apparently thinking/doing? 
redesigning the whole thing to use ansible + mistral + heat, or ansible 
+ kubernetes or ansible + mistral + heat + ansible (a second time!) or ...


Seeing all those kinds of questions and suggestions around what should 
be used and why and how (and even this thread) makes me really wonder 
who actually uses tripleo and can afford/understand such kinds of changes?


Does anyone?

If there are  is there going to be an upgrade 
path for there existing 'cloud/s' to whatever this solution is?


What operator(s) has the ability to do such a massive shift at this 
point in time? Who are these 'mystical' operators?


All this has really peaked my curiosity because I am personally trying 
to do that shift (not exactly the same solution...) and I know it is a 
massive undertaking (that will take quite a while to get right) even for 
a simple operator with limited needs out of openstack (ie godaddy); so I 
don't really understand how the generic solution for all existing 
tripleo operators can even work...


Flavio Percoco wrote:


Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment,
OpenStack
deployment on Kubernetes, configuration management, etc. While I've been
diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these
tools and
I've come to the conclusion that TripleO would be better of by having
ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm.
While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack
projects, I
believe using any of them would add an extra layer of complexity to
TripleO,
which is something the team has been fighting for years years -
especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would
require
TripleO to also write the logic to manage those projects. For example,
in the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the
charts (I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the
conclusion
I reached.

Now, what this work means is that we would have to write an ansible role
for
each service that will deploy the service on a Kubernetes cluster.
Ideally these
roles will also generate the configuration files (removing the need of
puppet
entirely) and they would manage the lifecycle. The roles would be
isolated and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain
these roles
and run them using the existing docker-cmd implementation that is coming
out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the
discussion and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that
ansible is
a known, powerfull, tool that has been adopted by many operators
already. It'll
provide the flexibility needed and, if structured correctly, it'll allow
for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate
concerns
in the deployment workflow and the idea of making it simple for users of
TripleO
to do the same at runtime. Unfortunately, going down this road means
that my
hope of creating a field where we could collaborate even more with other
deployment tools will be a bit limited but I'm confident the result
would also
be useful for others and that we all will benefit from it... My hopes
might be a
bit naive *shrugs*

Flavio

[0]
http://lists.openstack.org/pipermail/openstack-dev/2017-July/119405.html
[1] https://github.com/tripleo-apb/tripleo-apbs

--
@flaper87
Flavio Percoco

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Bogdan Dobrelya]

On 14.07.2017 18:16, Fox, Kevin M wrote:
> https://xkcd.com/927/
> 
> I don't think adopting helm as a dependency adds more complexity then writing 
> more new k8s object deployment tooling?
> 
> There are efforts to make it easy to deploy kolla-kubernetes microservice 
> charts using ansible for orchestration in kolla-kubernetes. See:
> https://review.openstack.org/#/c/473588/
> What kolla-kubernetes brings to the table is a tested/shared base k8s object 
> layer. Orchestration is done by ansible via TripleO, and the solutions 
> already found/debugged to how to deploy OpenStack in containers on Kubernetes 
> can be reused/shared.
> 
> See for example:
> https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/main.yaml
> 
> I don't see much by way of dealing with fernet token rotation. That was a 
> tricky bit of code to get to work, but kolla-kubernetes has a solution to it. 
> You can get it by: helm install kolla/keystone-fernet-rotate-job.
> 
> We designed this layer to be shareable so we all can contribute to the 
> commons rather then having every project reimplement their own and have to 
> chase bugs across all the implementations. The deployment projects will be 
> stronger together if we can share as much as possible.

Thank you Kevin, this ^^ expresses my thoughts better than I could ever say.

> 
> Please reconsider. I'd be happy to talk with you more if you want.
> 
> Thanks,
> Kevin


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Fox, Kevin M]

https://xkcd.com/927/

I don't think adopting helm as a dependency adds more complexity then writing 
more new k8s object deployment tooling?

There are efforts to make it easy to deploy kolla-kubernetes microservice 
charts using ansible for orchestration in kolla-kubernetes. See:
https://review.openstack.org/#/c/473588/
What kolla-kubernetes brings to the table is a tested/shared base k8s object 
layer. Orchestration is done by ansible via TripleO, and the solutions already 
found/debugged to how to deploy OpenStack in containers on Kubernetes can be 
reused/shared.

See for example:
https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/main.yaml

I don't see much by way of dealing with fernet token rotation. That was a 
tricky bit of code to get to work, but kolla-kubernetes has a solution to it. 
You can get it by: helm install kolla/keystone-fernet-rotate-job.

We designed this layer to be shareable so we all can contribute to the commons 
rather then having every project reimplement their own and have to chase bugs 
across all the implementations. The deployment projects will be stronger 
together if we can share as much as possible.

Please reconsider. I'd be happy to talk with you more if you want.

Thanks,
Kevin

From: Flavio Percoco [fla...@redhat.com]
Sent: Friday, July 14, 2017 2:17 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack 
services on Kubernetes

Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment, OpenStack
deployment on Kubernetes, configuration management, etc. While I've been diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these tools and
I've come to the conclusion that TripleO would be better of by having ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm. While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects, I
believe using any of them would add an extra layer of complexity to TripleO,
which is something the team has been fighting for years years - especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would require
TripleO to also write the logic to manage those projects. For example, in the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the charts (I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the conclusion
I reached.

Now, what this work means is that we would have to write an ansible role for
each service that will deploy the service on a Kubernetes cluster. Ideally these
roles will also generate the configuration files (removing the need of puppet
entirely) and they would manage the lifecycle. The roles would be isolated and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain these roles
and run them using the existing docker-cmd implementation that is coming out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the discussion and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that ansible is
a known, powerfull, tool that has been adopted by many operators already. It'll
provide the flexibility needed and, if structured correctly, it'll allow for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate concerns
in the deployment workflow and the idea of making it simple for users of TripleO
to do the same at runtime. Unfortunately, going down this road means that my
hope of creating a field where we could collaborate even more with other
deployment tools will be a bit limited but I'm confident the result would also
be useful for others and that we all will benefit from it... My

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Bogdan Dobrelya]

On 14.07.2017 17:55, Michał Jastrzębski wrote:
> Guys you just described Kolla-Kubernetes pretty much... how about
> we join effort and work towards this goal together?

That's exactly that I'd like we all to do.

> 
> On 14 July 2017 at 08:43, Flavio Percoco  wrote:
>> On 14/07/17 17:26 +0200, Bogdan Dobrelya wrote:
>>>
>>> On 14.07.2017 11:17, Flavio Percoco wrote:


 Greetings,

 As some of you know, I've been working on the second phase of TripleO's
 containerization effort. This phase if about migrating the docker based
 deployment onto Kubernetes.

 These phase requires work on several areas: Kubernetes deployment,
 OpenStack
 deployment on Kubernetes, configuration management, etc. While I've been
 diving
 into all of these areas, this email is about the second point, OpenStack
 deployment on Kubernetes.

 There are several tools we could use for this task. kolla-kubernetes,
 openstack-helm, ansible roles, among others. I've looked into these
 tools and
 I've come to the conclusion that TripleO would be better of by having
 ansible
 roles that would allow for deploying OpenStack services on Kubernetes.

 The existing solutions in the OpenStack community require using Helm.
 While I
 like Helm and both, kolla-kubernetes and openstack-helm OpenStack
 projects, I
 believe using any of them would add an extra layer of complexity to
 TripleO,
>>>
>>>
>>> It's hard to estimate that complexity w/o having a PoC of such an
>>> integration. We should come up with a final choice once we have it done.
>>>
>>> My vote would go for investing engineering resources into solutions that
>>> have problems already solved, even by the price of added complexity (but
>>> that sort of depends...). Added complexity may be compensated with
>>> removed complexity (like those client -> Mistral -> Heat -> Mistral ->
>>> Ansible manipulations discussed in the mail thread mentioned below [0])
>>
>>
>> I agree it's hard to estimate but you gotta draw the line somewhere. I
>> actually
>> spent time on this and here's a small PoC of ansible+mariadb+helm. I wrote
>> the
>> pyhelm lib (took some code from the openstack-helm folks) and I wrote the
>> ansible helm module myself. I'd say I've spent enough time on this research.
>>
>> I don't think getting a full PoC working is worth it as that will require
>> way
>> more work for not much value since we can anticipate some of the
>> complexities
>> already.
>>
>> As far as the complexity comment goes, I disagree with you. I don't think
>> you're
>> evaluating the amount of complexity that there *IS* already in TripleO and
>> how
>> adding more complexity (layers, states, services) would make things worse
>> for
>> not much extra value.
>>
>> By all means, I might be wrong here so, do let me know if you're seeing
>> something I'm not.

My point was to "trade" complexity described in the "Forming our plans
around Ansible​" ML thread:

(3) Mistral calling Heat calling Mistral calling Ansible

to just

(3') something calls kolla-kubernetes/openstack-helm, via some wrapper
composition overlay (which creates complexity), or the like

While the latter might add complexity like the way you (Flavio) have
described, the former would remove *another* type of complexity, and the
result might worth the efforts.

>> Flavio
>> --
>> @flaper87
>> Flavio Percoco
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Michał Jastrzębski]

Guys you just described Kolla-Kubernetes pretty much... how about
we join effort and work towards this goal together?

On 14 July 2017 at 08:43, Flavio Percoco  wrote:
> On 14/07/17 17:26 +0200, Bogdan Dobrelya wrote:
>>
>> On 14.07.2017 11:17, Flavio Percoco wrote:
>>>
>>>
>>> Greetings,
>>>
>>> As some of you know, I've been working on the second phase of TripleO's
>>> containerization effort. This phase if about migrating the docker based
>>> deployment onto Kubernetes.
>>>
>>> These phase requires work on several areas: Kubernetes deployment,
>>> OpenStack
>>> deployment on Kubernetes, configuration management, etc. While I've been
>>> diving
>>> into all of these areas, this email is about the second point, OpenStack
>>> deployment on Kubernetes.
>>>
>>> There are several tools we could use for this task. kolla-kubernetes,
>>> openstack-helm, ansible roles, among others. I've looked into these
>>> tools and
>>> I've come to the conclusion that TripleO would be better of by having
>>> ansible
>>> roles that would allow for deploying OpenStack services on Kubernetes.
>>>
>>> The existing solutions in the OpenStack community require using Helm.
>>> While I
>>> like Helm and both, kolla-kubernetes and openstack-helm OpenStack
>>> projects, I
>>> believe using any of them would add an extra layer of complexity to
>>> TripleO,
>>
>>
>> It's hard to estimate that complexity w/o having a PoC of such an
>> integration. We should come up with a final choice once we have it done.
>>
>> My vote would go for investing engineering resources into solutions that
>> have problems already solved, even by the price of added complexity (but
>> that sort of depends...). Added complexity may be compensated with
>> removed complexity (like those client -> Mistral -> Heat -> Mistral ->
>> Ansible manipulations discussed in the mail thread mentioned below [0])
>
>
> I agree it's hard to estimate but you gotta draw the line somewhere. I
> actually
> spent time on this and here's a small PoC of ansible+mariadb+helm. I wrote
> the
> pyhelm lib (took some code from the openstack-helm folks) and I wrote the
> ansible helm module myself. I'd say I've spent enough time on this research.
>
> I don't think getting a full PoC working is worth it as that will require
> way
> more work for not much value since we can anticipate some of the
> complexities
> already.
>
> As far as the complexity comment goes, I disagree with you. I don't think
> you're
> evaluating the amount of complexity that there *IS* already in TripleO and
> how
> adding more complexity (layers, states, services) would make things worse
> for
> not much extra value.
>
> By all means, I might be wrong here so, do let me know if you're seeing
> something I'm not.
> Flavio
> --
> @flaper87
> Flavio Percoco
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Jiří Stránský]

On 14.7.2017 11:17, Flavio Percoco wrote:


Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment, OpenStack
deployment on Kubernetes, configuration management, etc. While I've been diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these tools and
I've come to the conclusion that TripleO would be better of by having ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm. While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects, I
believe using any of them would add an extra layer of complexity to TripleO,
which is something the team has been fighting for years years - especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would require
TripleO to also write the logic to manage those projects. For example, in the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the charts (I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the conclusion
I reached.

Now, what this work means is that we would have to write an ansible role for
each service that will deploy the service on a Kubernetes cluster. Ideally these
roles will also generate the configuration files (removing the need of puppet
entirely) and they would manage the lifecycle. The roles would be isolated and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain these roles
and run them using the existing docker-cmd implementation that is coming out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the discussion and
gather thoughts and opinions from the rest of the community.


I agree this is a direction we should explore further. This would give 
us the option to tailor things exactly as we need -- good for keeping 
our balance in having interfaces as stable as possible, while still 
making enough development progress. And we'd keep our ability to make 
important changes (e.g. bugfixes) without delays.


We'll have to write more code ourselves, but it's possible that if we 
picked up an existing tool, we'd have to spend that time (if not more) 
elsewhere. Migrating existing non-kubernetized TripleO deployments to 
kubernetized is going to be pretty difficult even if we do what you 
suggested. I imagine that if we also had to fit into some pre-existing 
external deployment/management interfaces, while trying to keep ours 
stable or make just iterative changes, it might turn out to be a surreal 
effort. We will have to design things with migration from "legacy 
TripleO" in mind, or make later amendments here and there solely for 
this purpose. Such design and patches would probably not be a good fit 
for non-tripleo projects.


What i recall from our old PoC [2], defining the resources and init 
containers etc. will probably not be the most difficult task, and 
furthermore we can largely draw inspiration from our current 
containerized solution too. I think the more challenging things might be 
e.g. config generation with Ansible, and how major upgrades and rolling 
updates will be done (how all this ties into the APB way of 
provisioning/deprovisioning). And of course how to fulfill the 
expectations that TripleO has set around network isolation and HA :)


I'm eager to give the latest code a try myself :) Thanks for working on 
this, it looks like there's been great progress lately!


Jirka



Finally, what I really like about writing pure ansible roles is that ansible is
a known, powerfull, tool that has been adopted by many operators already. It'll
provide the flexibility needed and, if structured correctly, it'll allow for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate concerns
in the deployment workflow and the idea of making it simple for users of TripleO
to do the same at runtime. Unfortunately, going down this road means 

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Flavio Percoco]

On 14/07/17 17:26 +0200, Bogdan Dobrelya wrote:

On 14.07.2017 11:17, Flavio Percoco wrote:


Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment,
OpenStack
deployment on Kubernetes, configuration management, etc. While I've been
diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these
tools and
I've come to the conclusion that TripleO would be better of by having
ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm.
While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack
projects, I
believe using any of them would add an extra layer of complexity to
TripleO,


It's hard to estimate that complexity w/o having a PoC of such an
integration. We should come up with a final choice once we have it done.

My vote would go for investing engineering resources into solutions that
have problems already solved, even by the price of added complexity (but
that sort of depends...). Added complexity may be compensated with
removed complexity (like those client -> Mistral -> Heat -> Mistral ->
Ansible manipulations discussed in the mail thread mentioned below [0])


I agree it's hard to estimate but you gotta draw the line somewhere. I actually
spent time on this and here's a small PoC of ansible+mariadb+helm. I wrote the
pyhelm lib (took some code from the openstack-helm folks) and I wrote the
ansible helm module myself. I'd say I've spent enough time on this research.

I don't think getting a full PoC working is worth it as that will require way
more work for not much value since we can anticipate some of the complexities
already.

As far as the complexity comment goes, I disagree with you. I don't think you're
evaluating the amount of complexity that there *IS* already in TripleO and how
adding more complexity (layers, states, services) would make things worse for
not much extra value.

By all means, I might be wrong here so, do let me know if you're seeing
something I'm not.
Flavio
--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Bogdan Dobrelya]

On 14.07.2017 11:17, Flavio Percoco wrote:
> 
> Greetings,
> 
> As some of you know, I've been working on the second phase of TripleO's
> containerization effort. This phase if about migrating the docker based
> deployment onto Kubernetes.
> 
> These phase requires work on several areas: Kubernetes deployment,
> OpenStack
> deployment on Kubernetes, configuration management, etc. While I've been
> diving
> into all of these areas, this email is about the second point, OpenStack
> deployment on Kubernetes.
> 
> There are several tools we could use for this task. kolla-kubernetes,
> openstack-helm, ansible roles, among others. I've looked into these
> tools and
> I've come to the conclusion that TripleO would be better of by having
> ansible
> roles that would allow for deploying OpenStack services on Kubernetes.
> 
> The existing solutions in the OpenStack community require using Helm.
> While I
> like Helm and both, kolla-kubernetes and openstack-helm OpenStack
> projects, I
> believe using any of them would add an extra layer of complexity to
> TripleO,

It's hard to estimate that complexity w/o having a PoC of such an
integration. We should come up with a final choice once we have it done.

My vote would go for investing engineering resources into solutions that
have problems already solved, even by the price of added complexity (but
that sort of depends...). Added complexity may be compensated with
removed complexity (like those client -> Mistral -> Heat -> Mistral ->
Ansible manipulations discussed in the mail thread mentioned below [0])

> which is something the team has been fighting for years years -
> especially now
> that the snowball is being chopped off.
> 
> Adopting any of the existing projects in the OpenStack communty would
> require
> TripleO to also write the logic to manage those projects. For example,
> in the
> case of openstack-helm, the TripleO team would have to write either ansible
> roles or heat templates to manage - install, remove, upgrade - the
> charts (I'm
> happy to discuss this point further but I'm keepping it at a high-level on
> purpose for the sake of not writing a 10k-words-long email).
> 
> James Slagle sent an email[0], a couple of days ago, to form TripleO plans
> around ansible. One take-away from this thread is that TripleO is adopting
> ansible more and more, which is great and it fits perfectly with the
> conclusion
> I reached.
> 
> Now, what this work means is that we would have to write an ansible role
> for
> each service that will deploy the service on a Kubernetes cluster.
> Ideally these
> roles will also generate the configuration files (removing the need of
> puppet
> entirely) and they would manage the lifecycle. The roles would be
> isolated and
> this will reduce the need of TripleO Heat templates. Doing this would give
> TripleO full control on the deployment process too.
> 
> In addition, we could also write Ansible Playbook Bundles to contain
> these roles
> and run them using the existing docker-cmd implementation that is coming
> out in
> Pike (you can find a PoC/example of this in this repo[1]).
> 
> Now, I do realize the amount of work this implies and that this is my
> opinion/conclusion. I'm sending this email out to kick-off the
> discussion and
> gather thoughts and opinions from the rest of the community.
> 
> Finally, what I really like about writing pure ansible roles is that
> ansible is
> a known, powerfull, tool that has been adopted by many operators
> already. It'll
> provide the flexibility needed and, if structured correctly, it'll allow
> for
> operators (and other teams) to just use the parts they need/want without
> depending on the full-stack. I like the idea of being able to separate
> concerns
> in the deployment workflow and the idea of making it simple for users of
> TripleO
> to do the same at runtime. Unfortunately, going down this road means
> that my
> hope of creating a field where we could collaborate even more with other
> deployment tools will be a bit limited but I'm confident the result
> would also
> be useful for others and that we all will benefit from it... My hopes
> might be a
> bit naive *shrugs*
> 
> Flavio
> 
> [0]
> http://lists.openstack.org/pipermail/openstack-dev/2017-July/119405.html
> [1] https://github.com/tripleo-apb/tripleo-apbs
> 
> -- 
> @flaper87
> Flavio Percoco
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Emilien Macchi]

On Fri, Jul 14, 2017 at 2:17 AM, Flavio Percoco  wrote:
>
> Greetings,
>
> As some of you know, I've been working on the second phase of TripleO's
> containerization effort. This phase if about migrating the docker based
> deployment onto Kubernetes.
>
> These phase requires work on several areas: Kubernetes deployment, OpenStack
> deployment on Kubernetes, configuration management, etc. While I've been
> diving
> into all of these areas, this email is about the second point, OpenStack
> deployment on Kubernetes.
>
> There are several tools we could use for this task. kolla-kubernetes,
> openstack-helm, ansible roles, among others. I've looked into these tools
> and
> I've come to the conclusion that TripleO would be better of by having
> ansible
> roles that would allow for deploying OpenStack services on Kubernetes.
>
> The existing solutions in the OpenStack community require using Helm. While
> I
> like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects,
> I
> believe using any of them would add an extra layer of complexity to TripleO,
> which is something the team has been fighting for years years - especially
> now
> that the snowball is being chopped off.
>
> Adopting any of the existing projects in the OpenStack communty would
> require
> TripleO to also write the logic to manage those projects. For example, in
> the
> case of openstack-helm, the TripleO team would have to write either ansible
> roles or heat templates to manage - install, remove, upgrade - the charts
> (I'm
> happy to discuss this point further but I'm keepping it at a high-level on
> purpose for the sake of not writing a 10k-words-long email).
>
> James Slagle sent an email[0], a couple of days ago, to form TripleO plans
> around ansible. One take-away from this thread is that TripleO is adopting
> ansible more and more, which is great and it fits perfectly with the
> conclusion
> I reached.
>
> Now, what this work means is that we would have to write an ansible role for
> each service that will deploy the service on a Kubernetes cluster. Ideally
> these
> roles will also generate the configuration files (removing the need of
> puppet
> entirely) and they would manage the lifecycle. The roles would be isolated
> and
> this will reduce the need of TripleO Heat templates. Doing this would give
> TripleO full control on the deployment process too.
>
> In addition, we could also write Ansible Playbook Bundles to contain these
> roles
> and run them using the existing docker-cmd implementation that is coming out
> in
> Pike (you can find a PoC/example of this in this repo[1]).
>
> Now, I do realize the amount of work this implies and that this is my
> opinion/conclusion. I'm sending this email out to kick-off the discussion
> and
> gather thoughts and opinions from the rest of the community.
>
> Finally, what I really like about writing pure ansible roles is that ansible
> is
> a known, powerfull, tool that has been adopted by many operators already.
> It'll
> provide the flexibility needed and, if structured correctly, it'll allow for
> operators (and other teams) to just use the parts they need/want without
> depending on the full-stack. I like the idea of being able to separate
> concerns
> in the deployment workflow and the idea of making it simple for users of
> TripleO
> to do the same at runtime. Unfortunately, going down this road means that my
> hope of creating a field where we could collaborate even more with other
> deployment tools will be a bit limited but I'm confident the result would
> also
> be useful for others and that we all will benefit from it... My hopes might
> be a
> bit naive *shrugs*

Of course I'm biased since I've been (a little) involved in that work
but I like the idea of :

- Moving forward with our containerization. docker-cmd will help us
for sure for this transition (I insist on the fact TripleO is a
product that you can upgrade and we try to make it smooth for our
operators), so we can't just trash everything and switch to a new
tool. I think the approach that we're taking is great and made of baby
steps where we try to solve different problems.
- Using more Ansible - the right way - when it makes sense : with the
TripleO containerization, we only use Puppet for Configuration
Management, managing a few resources but not for orchestration (or not
all the features that Puppet provide) and for Data Binding (Hiera). To
me, it doesn't make sense for us to keep investing much in Puppet
modules if we go k8s & Ansible. That said, see the next point.
- Having a transition path between TripleO with Puppet and TripleO
with apbs and have some sort of binding between previous hieradata
generated by TripleO & a similar data binding within Ansible playbooks
would help. I saw your PoC Flavio, I found it great and I think we
should make 

[openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Flavio Percoco]

Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment, OpenStack
deployment on Kubernetes, configuration management, etc. While I've been diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these tools and
I've come to the conclusion that TripleO would be better of by having ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm. While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects, I
believe using any of them would add an extra layer of complexity to TripleO,
which is something the team has been fighting for years years - especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would require
TripleO to also write the logic to manage those projects. For example, in the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the charts (I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the conclusion
I reached.

Now, what this work means is that we would have to write an ansible role for
each service that will deploy the service on a Kubernetes cluster. Ideally these
roles will also generate the configuration files (removing the need of puppet
entirely) and they would manage the lifecycle. The roles would be isolated and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain these roles
and run them using the existing docker-cmd implementation that is coming out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the discussion and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that ansible is
a known, powerfull, tool that has been adopted by many operators already. It'll
provide the flexibility needed and, if structured correctly, it'll allow for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate concerns
in the deployment workflow and the idea of making it simple for users of TripleO
to do the same at runtime. Unfortunately, going down this road means that my
hope of creating a field where we could collaborate even more with other
deployment tools will be a bit limited but I'm confident the result would also
be useful for others and that we all will benefit from it... My hopes might be a
bit naive *shrugs*

Flavio

[0] http://lists.openstack.org/pipermail/openstack-dev/2017-July/119405.html
[1] https://github.com/tripleo-apb/tripleo-apbs

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev