Re: [openstack-dev] [tripleo] Deployment plan management efforts sync up

[Dougal Matthews]

On 2 February 2017 at 11:56, Jiri Tomasek  wrote:

> Hello all,
>
> there has been several ongoing efforts in TripleO regarding Deployment
> Plans management and Deployment configuration itself. A lot of this work is
> done to satisfy certain individual requirements but I think some further
> discussion needs to happen to make sure the solutions we create are
> effective for all parts of TripleO.
>
> There are several goals / features we currently aim for:
> - Define and manage custom Roles (GUI/CLI)
> - Define and manage networks (CLI/GUI)
> - Import/Export Deployment plans so it is possible to reuse them or use
> them as a reference/starting point
>
> Currently the Deployment plan stored in Swift consist of:
> - tripleo heat templates
> - roles_data.yaml - meta file used as an input to define roles (and assign
> networks to roles [1])
> - network_data.yaml - meta file used as input to define networks [2]
> - capabilities-map.yaml - meta file to describe THT environment files
> capabilities
> - mistral environment - json structure in Mistral which we use as a
> backend store accessible by mistral actions and workflows (tripleo-common)
>
> Currently, only possibility to configure roles and networks is by creating
> or updating the plan with changed meta files. We need to create Mistral
> actions to handle manipulating roles and networks so GUI (also CLI) can
> retrieve current roles/networks configuration and update it, which in turn
> will regenerate related templates. Now, the question arises: Do we want to
> use roles_data.yaml in Swift container as a storage for this information? I
> thought we agreed on using Mistral environment to store plan related data.
> (See here for additional context [3] )
>

We did agree that, I do wonder if we should revisit that decision. Nobody
else uses Mistral as a permanent, long term store. It wasn't designed for
that purpose and I am not aware of anyone else using it like this. It also
doesn't really give us any advantages (in reality, Mistral environments
have far less features than Swift). Accessing Swift is just probably just
as easy if not easier. So I think it might be worth considering moving
everything to Swift and keeping everything in one place.

This means that on plan creation, we use roles_data.yaml (and
> network_data.yaml etc.) to populate Mistral environment and generate
> templates using this data. roles_data.yaml (and others) then need to be
> discarded because from this point on, the data will be updated in Mistral
> environment through tripleo-common actions. roles_data.yaml is therefore
> used just as a default which is used when plan is created (or updated).
>

I think one of the mistakes we have made is to never fully model the data,
we don't have any code in tripleo-common that abstracts the storage and
defines how everything is stored. Instead we have a collection of actions
that understand how the structure should look in different places. This
means that as we have added new features, they all do everything a bit
differently and I can see it becoming hard to maintain and update.


Now, plan export comes into play [4]. We want to be able to pull down the
> plan and deploy it using CLI directly (which creates/updates plan as far as
> during deploy command), We want to be able to reuse the plan in other
> deployments or use it as a reference architecture for subsequent
> deployments. This means we don't only want to download the contents of
> Swift container, but also configuration stored in Mistral environment.
>

If we did put everything in Swift, this bit would be much easier.


>
> So Plan export action pulls down files from Swift container, adds meta
> files: roles_data.yaml, network_data.yaml... which it populates by looking
> at appropriate keys in Mistral environment + plan_environment.yaml which
> includes remaining data from Mistral environment such as parameter values,
> environments etc. All of those are then returned in a single tarball.
> Question to consider is whether to split all this data into separate files
> or keep it in single one.
>
> Plan Import [5] then allows to provide plan_environment.yaml to enable
> populating parameter values and environments selection during plan creation.
>
>
> Alternative solution is to store meta files (roles_data.yaml,
> network_data.yaml...) in Swift container and use those files as a data
> store which IMHO does not comply with decision to use Mistral environment
> as a plan data store. In that case we should probably get rid of using
> Mistral environment altogether and use plan_environment.yaml file in Swift
> container to store data which we currently store in Mistral environment. I
> am quite convinced that there have been good reasons to not to do this and
> use Mistral environment.
>

I am not convinced. Can you remember any of these reasons? I can't?


>
>
> [1] https://review.openstack.org/#/c/409920/
> [2] https://review.openstack.org/#/c/409921/
> [3] 

Re: [openstack-dev] [tripleo] Deployment plan management efforts sync up

[Jiri Tomasek]

On 2.2.2017 13:57, Ana Krivokapic wrote:



On Thu, Feb 2, 2017 at 1:46 PM, Emilien Macchi > wrote:


On Thu, Feb 2, 2017 at 6:56 AM, Jiri Tomasek > wrote:
> Hello all,
>
> there has been several ongoing efforts in TripleO regarding
Deployment Plans
> management and Deployment configuration itself. A lot of this
work is done
> to satisfy certain individual requirements but I think some further
> discussion needs to happen to make sure the solutions we create are
> effective for all parts of TripleO.
>
> There are several goals / features we currently aim for:
> - Define and manage custom Roles (GUI/CLI)
> - Define and manage networks (CLI/GUI)
> - Import/Export Deployment plans so it is possible to reuse them
or use them
> as a reference/starting point
>
> Currently the Deployment plan stored in Swift consist of:
> - tripleo heat templates
> - roles_data.yaml - meta file used as an input to define roles
(and assign
> networks to roles [1])
> - network_data.yaml - meta file used as input to define networks [2]
> - capabilities-map.yaml - meta file to describe THT environment
files
> capabilities
> - mistral environment - json structure in Mistral which we use
as a backend
> store accessible by mistral actions and workflows (tripleo-common)
>
> Currently, only possibility to configure roles and networks is
by creating
> or updating the plan with changed meta files. We need to create
Mistral
> actions to handle manipulating roles and networks so GUI (also
CLI) can
> retrieve current roles/networks configuration and update it,
which in turn
> will regenerate related templates. Now, the question arises: Do
we want to
> use roles_data.yaml in Swift container as a storage for this
information? I
> thought we agreed on using Mistral environment to store plan
related data.
> (See here for additional context [3] )
>
> This means that on plan creation, we use roles_data.yaml (and
> network_data.yaml etc.) to populate Mistral environment and generate
> templates using this data. roles_data.yaml (and others) then
need to be
> discarded because from this point on, the data will be updated
in Mistral
> environment through tripleo-common actions. roles_data.yaml is
therefore
> used just as a default which is used when plan is created (or
updated).
>
> Now, plan export comes into play [4]. We want to be able to pull
down the
> plan and deploy it using CLI directly (which creates/updates
plan as far as
> during deploy command), We want to be able to reuse the plan in
other
> deployments or use it as a reference architecture for subsequent
> deployments. This means we don't only want to download the
contents of Swift
> container, but also configuration stored in Mistral environment.
>
> So Plan export action pulls down files from Swift container,
adds meta
> files: roles_data.yaml, network_data.yaml... which it populates
by looking
> at appropriate keys in Mistral environment +
plan_environment.yaml which
> includes remaining data from Mistral environment such as
parameter values,
> environments etc. All of those are then returned in a single
tarball.
> Question to consider is whether to split all this data into
separate files
> or keep it in single one.
>
> Plan Import [5] then allows to provide plan_environment.yaml to
enable
> populating parameter values and environments selection during
plan creation.

For which cycle would you target this blueprint?
We need to update it accordingly:

https://blueprints.launchpad.net/tripleo/+spec/enhance-plan-creation-with-plan-environment-json




I think it should be targeted for Pike 1.

+1




>
> Alternative solution is to store meta files (roles_data.yaml,
> network_data.yaml...) in Swift container and use those files as
a data store
> which IMHO does not comply with decision to use Mistral
environment as a
> plan data store. In that case we should probably get rid of
using Mistral
> environment altogether and use plan_environment.yaml file in
Swift container
> to store data which we currently store in Mistral environment. I
am quite
> convinced that there have been good reasons to not to do this
and use
> Mistral environment.
>
>
> [1] https://review.openstack.org/#/c/409920/

> [2] https://review.openstack.org/#/c/409921/

> [3] 

Re: [openstack-dev] [tripleo] Deployment plan management efforts sync up

[Ana Krivokapic]

On Thu, Feb 2, 2017 at 1:46 PM, Emilien Macchi  wrote:

> On Thu, Feb 2, 2017 at 6:56 AM, Jiri Tomasek  wrote:
> > Hello all,
> >
> > there has been several ongoing efforts in TripleO regarding Deployment
> Plans
> > management and Deployment configuration itself. A lot of this work is
> done
> > to satisfy certain individual requirements but I think some further
> > discussion needs to happen to make sure the solutions we create are
> > effective for all parts of TripleO.
> >
> > There are several goals / features we currently aim for:
> > - Define and manage custom Roles (GUI/CLI)
> > - Define and manage networks (CLI/GUI)
> > - Import/Export Deployment plans so it is possible to reuse them or use
> them
> > as a reference/starting point
> >
> > Currently the Deployment plan stored in Swift consist of:
> > - tripleo heat templates
> > - roles_data.yaml - meta file used as an input to define roles (and
> assign
> > networks to roles [1])
> > - network_data.yaml - meta file used as input to define networks [2]
> > - capabilities-map.yaml - meta file to describe THT environment files
> > capabilities
> > - mistral environment - json structure in Mistral which we use as a
> backend
> > store accessible by mistral actions and workflows (tripleo-common)
> >
> > Currently, only possibility to configure roles and networks is by
> creating
> > or updating the plan with changed meta files. We need to create Mistral
> > actions to handle manipulating roles and networks so GUI (also CLI) can
> > retrieve current roles/networks configuration and update it, which in
> turn
> > will regenerate related templates. Now, the question arises: Do we want
> to
> > use roles_data.yaml in Swift container as a storage for this
> information? I
> > thought we agreed on using Mistral environment to store plan related
> data.
> > (See here for additional context [3] )
> >
> > This means that on plan creation, we use roles_data.yaml (and
> > network_data.yaml etc.) to populate Mistral environment and generate
> > templates using this data. roles_data.yaml (and others) then need to be
> > discarded because from this point on, the data will be updated in Mistral
> > environment through tripleo-common actions. roles_data.yaml is therefore
> > used just as a default which is used when plan is created (or updated).
> >
> > Now, plan export comes into play [4]. We want to be able to pull down the
> > plan and deploy it using CLI directly (which creates/updates plan as far
> as
> > during deploy command), We want to be able to reuse the plan in other
> > deployments or use it as a reference architecture for subsequent
> > deployments. This means we don't only want to download the contents of
> Swift
> > container, but also configuration stored in Mistral environment.
> >
> > So Plan export action pulls down files from Swift container, adds meta
> > files: roles_data.yaml, network_data.yaml... which it populates by
> looking
> > at appropriate keys in Mistral environment + plan_environment.yaml which
> > includes remaining data from Mistral environment such as parameter
> values,
> > environments etc. All of those are then returned in a single tarball.
> > Question to consider is whether to split all this data into separate
> files
> > or keep it in single one.
> >
> > Plan Import [5] then allows to provide plan_environment.yaml to enable
> > populating parameter values and environments selection during plan
> creation.
>
> For which cycle would you target this blueprint?
> We need to update it accordingly:
> https://blueprints.launchpad.net/tripleo/+spec/enhance-
> plan-creation-with-plan-environment-json


I think it should be targeted for Pike 1.


>
>
> >
> > Alternative solution is to store meta files (roles_data.yaml,
> > network_data.yaml...) in Swift container and use those files as a data
> store
> > which IMHO does not comply with decision to use Mistral environment as a
> > plan data store. In that case we should probably get rid of using Mistral
> > environment altogether and use plan_environment.yaml file in Swift
> container
> > to store data which we currently store in Mistral environment. I am quite
> > convinced that there have been good reasons to not to do this and use
> > Mistral environment.
> >
> >
> > [1] https://review.openstack.org/#/c/409920/
> > [2] https://review.openstack.org/#/c/409921/
> > [3] https://review.openstack.org/#/c/409921/
> > [4]
> > http://specs.openstack.org/openstack/tripleo-specs/specs/
> ocata/gui-plan-import-export.html#problem-description
> > [5]
> > https://blueprints.launchpad.net/tripleo/+spec/enhance-
> plan-creation-with-plan-environment-json
> >
> >
> > -- Jirka
>
> Sounds good to me! Thanks for sharing it.
> --
> Emilien Macchi
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [tripleo] Deployment plan management efforts sync up

[Emilien Macchi]

On Thu, Feb 2, 2017 at 6:56 AM, Jiri Tomasek  wrote:
> Hello all,
>
> there has been several ongoing efforts in TripleO regarding Deployment Plans
> management and Deployment configuration itself. A lot of this work is done
> to satisfy certain individual requirements but I think some further
> discussion needs to happen to make sure the solutions we create are
> effective for all parts of TripleO.
>
> There are several goals / features we currently aim for:
> - Define and manage custom Roles (GUI/CLI)
> - Define and manage networks (CLI/GUI)
> - Import/Export Deployment plans so it is possible to reuse them or use them
> as a reference/starting point
>
> Currently the Deployment plan stored in Swift consist of:
> - tripleo heat templates
> - roles_data.yaml - meta file used as an input to define roles (and assign
> networks to roles [1])
> - network_data.yaml - meta file used as input to define networks [2]
> - capabilities-map.yaml - meta file to describe THT environment files
> capabilities
> - mistral environment - json structure in Mistral which we use as a backend
> store accessible by mistral actions and workflows (tripleo-common)
>
> Currently, only possibility to configure roles and networks is by creating
> or updating the plan with changed meta files. We need to create Mistral
> actions to handle manipulating roles and networks so GUI (also CLI) can
> retrieve current roles/networks configuration and update it, which in turn
> will regenerate related templates. Now, the question arises: Do we want to
> use roles_data.yaml in Swift container as a storage for this information? I
> thought we agreed on using Mistral environment to store plan related data.
> (See here for additional context [3] )
>
> This means that on plan creation, we use roles_data.yaml (and
> network_data.yaml etc.) to populate Mistral environment and generate
> templates using this data. roles_data.yaml (and others) then need to be
> discarded because from this point on, the data will be updated in Mistral
> environment through tripleo-common actions. roles_data.yaml is therefore
> used just as a default which is used when plan is created (or updated).
>
> Now, plan export comes into play [4]. We want to be able to pull down the
> plan and deploy it using CLI directly (which creates/updates plan as far as
> during deploy command), We want to be able to reuse the plan in other
> deployments or use it as a reference architecture for subsequent
> deployments. This means we don't only want to download the contents of Swift
> container, but also configuration stored in Mistral environment.
>
> So Plan export action pulls down files from Swift container, adds meta
> files: roles_data.yaml, network_data.yaml... which it populates by looking
> at appropriate keys in Mistral environment + plan_environment.yaml which
> includes remaining data from Mistral environment such as parameter values,
> environments etc. All of those are then returned in a single tarball.
> Question to consider is whether to split all this data into separate files
> or keep it in single one.
>
> Plan Import [5] then allows to provide plan_environment.yaml to enable
> populating parameter values and environments selection during plan creation.

For which cycle would you target this blueprint?
We need to update it accordingly:
https://blueprints.launchpad.net/tripleo/+spec/enhance-plan-creation-with-plan-environment-json

>
> Alternative solution is to store meta files (roles_data.yaml,
> network_data.yaml...) in Swift container and use those files as a data store
> which IMHO does not comply with decision to use Mistral environment as a
> plan data store. In that case we should probably get rid of using Mistral
> environment altogether and use plan_environment.yaml file in Swift container
> to store data which we currently store in Mistral environment. I am quite
> convinced that there have been good reasons to not to do this and use
> Mistral environment.
>
>
> [1] https://review.openstack.org/#/c/409920/
> [2] https://review.openstack.org/#/c/409921/
> [3] https://review.openstack.org/#/c/409921/
> [4]
> http://specs.openstack.org/openstack/tripleo-specs/specs/ocata/gui-plan-import-export.html#problem-description
> [5]
> https://blueprints.launchpad.net/tripleo/+spec/enhance-plan-creation-with-plan-environment-json
>
>
> -- Jirka

Sounds good to me! Thanks for sharing it.
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev