Re: [openstack-dev] [TripleO] Next steps for pre-deployment workflows (e.g derive parameters)

[Saravanan KR]

As discussed in IRC, I have collated all the important discussions to
the etherpad (gdoc was not publicly shareable).

https://etherpad.openstack.org/p/tripleo-derive-parameters-v2

Lets continue discussion on the etherpad to finalize.

Regards,
Saravanan KR

On Thu, Nov 9, 2017 at 11:05 AM, Saravanan KR  wrote:
> On Thu, Nov 9, 2017 at 2:57 AM, Jiri Tomasek  wrote:
>>
>>
>> On Wed, Nov 8, 2017 at 6:09 AM, Steven Hardy  wrote:
>>>
>>> Hi all,
>>>
>>> Today I had a productive hallway discussion with jtomasek and
>>> stevebaker re $subject, so I wanted to elaborate here for the benefit
>>> of those folks not present.  Hopefully we can get feedback on the
>>> ideas and see if it makes sense to continue and work on some patches:
>>>
>>> The problem under discussion is how do we run pre-deployment workflows
>>> (such as those integrated recently to calculate derived parameters,
>>> and in future perhaps also those which download container images etc),
>>> and in particular how do we make these discoverable via the UI
>>> (including any input parameters).
>>>
>>> The idea we came up with has two parts:
>>>
>>> 1. Add a new optional section to roles_data for services that require
>>> pre-deploy workflows
>>>
>>> E.g something like this:
>>>
>>>  pre_deploy_workflows:
>>> - derive_params:
>>>   workflow_name:
>>> tripleo.derive_params_formulas.v1.dpdk_derive_params
>>>   inputs:
>>>   ...
>>>
>>> This would allow us to associate a specific mistral workflow with a
>>> given service template, and also work around the fact that currently
>>> mistral inputs don't have any schema (only key/value input) as we
>>> could encode the required type and any constraints in the inputs block
>>> (clearly this could be removed in future should typed parameters
>>> become available in mistral).
>>>
>>> 2. Add a new workflow that calculates the enabled services and returns
>>> all pre_deploy_workflows
>>>
>>> This would take all enabled environments, then use heat to validate
>>> the configuration and return the merged resource registry (which will
>>> require https://review.openstack.org/#/c/509760/), then we would
>>> iterate over all enabled services in the registry and extract a given
>>> roles_data key (e.g pre_deploy_workflows)
>>>
>>> The result of the workflow would be a list of all pre_deploy_workflows
>>> for all enabled services, which the UI could then use to run the
>>> workflows as part of the pre-deploy process.
>>
>>
>> As I think about this more, we may find out that matching a service to
>> workflow is not enough as workflow may require several services (together
>> defining a feature) So maybe doing it in separate file would help. E.g.
>>
>> pre-deploy-workflows.yaml
>> - name: my.workflow
>>   services: a, b, c, d
>>
>> Maybe there is a better way, maybe this is not even needed. I am not sure.
>> What do you think?
>
> Currently, HCI derive parameters workflow is invoked if a role has
> both NovaCompute and CephOSD services enabled.
>
>>
>>
>> What I really like about this proposal is that it provides a standard way to
>> configure deployment features and provides clear means to add additional
>> such configurations.
>>
>> The resulting deployment configuration steps in GUI would look following:
>>
>> 1/ Hardware (reg. nodes, introspect etc)
>>
>> 2/ High level deployment configuration (basically selecting additional
>> environment files)
>>
>> 3/ Roles management (Roles selection, roles -> nodes assignment, roles
>> configuration - setting roles_data properties)
>>
>> 4/ Network configuration -  network configuration wizard: (I'll describe
>> this in separate email)
>>
>> 5/ Deployment Features configuration (This proposal) - a list of features to
>> configure, the list is nicely generated from information provided in
>> previous steps, user has all the information to configure those features at
>> hand and can go through these step by step.
>
> Agreed on the UI workflow.
>
> For DPDK and SR-IOV, there are common host specific parameters to be
> derived. It has been added as a separate host-specific parameters
> workflow. And both DPDK and SR-IOV workflow execution should follow
> host-specific workflow.
> In case of DPDK and HCI in same role, it is expected that DPDK
> workflow is executed before HCI. And the service configuration should
> provide this order to UI.
> I am not able to realize how this information will be provided and
> processed in UI with user. Do you have a UI wire frame for this
> workflow?
>
>>
>> 6/ Advanced deployment config - a view providing a way to review
>> Environment/Roles/Services parameters, search and tweak them if needed.
>>
>> 7/ Deploy.
>>
>> I believe these steps should cover anything we should need to do for
>> deployment configuration.
>>
>> -- Jirka
>>
>>
>>>
>>>
>>> If this makes sense I can go ahead and push some patches so we can
>>> iterate on the 

Re: [openstack-dev] [TripleO] Next steps for pre-deployment workflows (e.g derive parameters)

[Saravanan KR]

On Thu, Nov 9, 2017 at 2:57 AM, Jiri Tomasek  wrote:
>
>
> On Wed, Nov 8, 2017 at 6:09 AM, Steven Hardy  wrote:
>>
>> Hi all,
>>
>> Today I had a productive hallway discussion with jtomasek and
>> stevebaker re $subject, so I wanted to elaborate here for the benefit
>> of those folks not present.  Hopefully we can get feedback on the
>> ideas and see if it makes sense to continue and work on some patches:
>>
>> The problem under discussion is how do we run pre-deployment workflows
>> (such as those integrated recently to calculate derived parameters,
>> and in future perhaps also those which download container images etc),
>> and in particular how do we make these discoverable via the UI
>> (including any input parameters).
>>
>> The idea we came up with has two parts:
>>
>> 1. Add a new optional section to roles_data for services that require
>> pre-deploy workflows
>>
>> E.g something like this:
>>
>>  pre_deploy_workflows:
>> - derive_params:
>>   workflow_name:
>> tripleo.derive_params_formulas.v1.dpdk_derive_params
>>   inputs:
>>   ...
>>
>> This would allow us to associate a specific mistral workflow with a
>> given service template, and also work around the fact that currently
>> mistral inputs don't have any schema (only key/value input) as we
>> could encode the required type and any constraints in the inputs block
>> (clearly this could be removed in future should typed parameters
>> become available in mistral).
>>
>> 2. Add a new workflow that calculates the enabled services and returns
>> all pre_deploy_workflows
>>
>> This would take all enabled environments, then use heat to validate
>> the configuration and return the merged resource registry (which will
>> require https://review.openstack.org/#/c/509760/), then we would
>> iterate over all enabled services in the registry and extract a given
>> roles_data key (e.g pre_deploy_workflows)
>>
>> The result of the workflow would be a list of all pre_deploy_workflows
>> for all enabled services, which the UI could then use to run the
>> workflows as part of the pre-deploy process.
>
>
> As I think about this more, we may find out that matching a service to
> workflow is not enough as workflow may require several services (together
> defining a feature) So maybe doing it in separate file would help. E.g.
>
> pre-deploy-workflows.yaml
> - name: my.workflow
>   services: a, b, c, d
>
> Maybe there is a better way, maybe this is not even needed. I am not sure.
> What do you think?

Currently, HCI derive parameters workflow is invoked if a role has
both NovaCompute and CephOSD services enabled.

>
>
> What I really like about this proposal is that it provides a standard way to
> configure deployment features and provides clear means to add additional
> such configurations.
>
> The resulting deployment configuration steps in GUI would look following:
>
> 1/ Hardware (reg. nodes, introspect etc)
>
> 2/ High level deployment configuration (basically selecting additional
> environment files)
>
> 3/ Roles management (Roles selection, roles -> nodes assignment, roles
> configuration - setting roles_data properties)
>
> 4/ Network configuration -  network configuration wizard: (I'll describe
> this in separate email)
>
> 5/ Deployment Features configuration (This proposal) - a list of features to
> configure, the list is nicely generated from information provided in
> previous steps, user has all the information to configure those features at
> hand and can go through these step by step.

Agreed on the UI workflow.

For DPDK and SR-IOV, there are common host specific parameters to be
derived. It has been added as a separate host-specific parameters
workflow. And both DPDK and SR-IOV workflow execution should follow
host-specific workflow.
In case of DPDK and HCI in same role, it is expected that DPDK
workflow is executed before HCI. And the service configuration should
provide this order to UI.
I am not able to realize how this information will be provided and
processed in UI with user. Do you have a UI wire frame for this
workflow?

>
> 6/ Advanced deployment config - a view providing a way to review
> Environment/Roles/Services parameters, search and tweak them if needed.
>
> 7/ Deploy.
>
> I believe these steps should cover anything we should need to do for
> deployment configuration.
>
> -- Jirka
>
>
>>
>>
>> If this makes sense I can go ahead and push some patches so we can
>> iterate on the implementation?
Agreed.

Regards,
Saravanan KR

>>
>> Thanks,
>>
>> Steve
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development 

Re: [openstack-dev] [TripleO] Next steps for pre-deployment workflows (e.g derive parameters)

[Steven Hardy]

On Wed, Nov 8, 2017 at 10:55 PM, James Slagle  wrote:
> On Wed, Nov 8, 2017 at 7:16 AM, James Slagle  wrote:
>> On Wed, Nov 8, 2017 at 12:09 AM, Steven Hardy  wrote:
>>> Hi all,
>>>
>>> Today I had a productive hallway discussion with jtomasek and
>>> stevebaker re $subject, so I wanted to elaborate here for the benefit
>>> of those folks not present.  Hopefully we can get feedback on the
>>> ideas and see if it makes sense to continue and work on some patches:
>>>
>>> The problem under discussion is how do we run pre-deployment workflows
>>> (such as those integrated recently to calculate derived parameters,
>>> and in future perhaps also those which download container images etc),
>>> and in particular how do we make these discoverable via the UI
>>> (including any input parameters).
>
> After chatting with jtomasek on irc, I wanted to clarify that the part
> of this proposal I'm hesitant about.
>
> Specifically, it's adding an interface for any service to specify a
> mistral workflow, that in theory could do anything, as part of a
> pre_deploy interface "contract". If we do offer such an interface, I
> think it ought to be driven via ansible tasks/plays that are available
> as Heat stack outputs to match the config-download pattern.

Thanks for the feedback, yes I agree if we can do this with pure
ansible that would be great, but we'll have to take a closer look at
the existing implementation, e.g as mentioned by Saravanan there is an
existing integration with mistral workflows, which we'll either have
to maintain or migrate away from.

> Perhaps for just deriving parameters, having a way to specify
> workflows for the UI is Ok. It's more of the generic interface I'm not
> so keen on. As it relates to your example of downloading container
> images, it seems we could have a generic ansible task to do that, that
> could then be executed with Mistral for API purposes instead of
> specifying the Mistral workflow directly in the templates/roles_data.

Yeah good point, and also the point about CI moving towards undercloud
deploy is a good one - if we can work out a way to do this via ansible
(even if that means ansible running a mistral workflow as a
transitional step?) that would certainly be easier.

Hopefully we can chat more about this on IRC next week and prototype
the ansible approach to see how it could work.

Thanks!

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Next steps for pre-deployment workflows (e.g derive parameters)

[James Slagle]

On Wed, Nov 8, 2017 at 7:16 AM, James Slagle  wrote:
> On Wed, Nov 8, 2017 at 12:09 AM, Steven Hardy  wrote:
>> Hi all,
>>
>> Today I had a productive hallway discussion with jtomasek and
>> stevebaker re $subject, so I wanted to elaborate here for the benefit
>> of those folks not present.  Hopefully we can get feedback on the
>> ideas and see if it makes sense to continue and work on some patches:
>>
>> The problem under discussion is how do we run pre-deployment workflows
>> (such as those integrated recently to calculate derived parameters,
>> and in future perhaps also those which download container images etc),
>> and in particular how do we make these discoverable via the UI
>> (including any input parameters).

After chatting with jtomasek on irc, I wanted to clarify that the part
of this proposal I'm hesitant about.

Specifically, it's adding an interface for any service to specify a
mistral workflow, that in theory could do anything, as part of a
pre_deploy interface "contract". If we do offer such an interface, I
think it ought to be driven via ansible tasks/plays that are available
as Heat stack outputs to match the config-download pattern.

Perhaps for just deriving parameters, having a way to specify
workflows for the UI is Ok. It's more of the generic interface I'm not
so keen on. As it relates to your example of downloading container
images, it seems we could have a generic ansible task to do that, that
could then be executed with Mistral for API purposes instead of
specifying the Mistral workflow directly in the templates/roles_data.

-- 
-- James Slagle
--

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Next steps for pre-deployment workflows (e.g derive parameters)

[Jiri Tomasek]

On Wed, Nov 8, 2017 at 6:09 AM, Steven Hardy  wrote:

> Hi all,
>
> Today I had a productive hallway discussion with jtomasek and
> stevebaker re $subject, so I wanted to elaborate here for the benefit
> of those folks not present.  Hopefully we can get feedback on the
> ideas and see if it makes sense to continue and work on some patches:
>
> The problem under discussion is how do we run pre-deployment workflows
> (such as those integrated recently to calculate derived parameters,
> and in future perhaps also those which download container images etc),
> and in particular how do we make these discoverable via the UI
> (including any input parameters).
>
> The idea we came up with has two parts:
>
> 1. Add a new optional section to roles_data for services that require
> pre-deploy workflows
>
> E.g something like this:
>
>  pre_deploy_workflows:
> - derive_params:
>   workflow_name:
> tripleo.derive_params_formulas.v1.dpdk_derive_params
>   inputs:
>   ...
>
> This would allow us to associate a specific mistral workflow with a
> given service template, and also work around the fact that currently
> mistral inputs don't have any schema (only key/value input) as we
> could encode the required type and any constraints in the inputs block
> (clearly this could be removed in future should typed parameters
> become available in mistral).
>
> 2. Add a new workflow that calculates the enabled services and returns
> all pre_deploy_workflows
>
> This would take all enabled environments, then use heat to validate
> the configuration and return the merged resource registry (which will
> require https://review.openstack.org/#/c/509760/), then we would
> iterate over all enabled services in the registry and extract a given
> roles_data key (e.g pre_deploy_workflows)
>
> The result of the workflow would be a list of all pre_deploy_workflows
> for all enabled services, which the UI could then use to run the
> workflows as part of the pre-deploy process.
>

As I think about this more, we may find out that matching a service to
workflow is not enough as workflow may require several services (together
defining a feature) So maybe doing it in separate file would help. E.g.

pre-deploy-workflows.yaml
- name: my.workflow
  services: a, b, c, d

Maybe there is a better way, maybe this is not even needed. I am not sure.
What do you think?


What I really like about this proposal is that it provides a standard way
to configure deployment features and provides clear means to add additional
such configurations.

The resulting deployment configuration steps in GUI would look following:

1/ Hardware (reg. nodes, introspect etc)

2/ High level deployment configuration (basically selecting additional
environment files)

3/ Roles management (Roles selection, roles -> nodes assignment, roles
configuration - setting roles_data properties)

4/ Network configuration -  network configuration wizard: (I'll describe
this in separate email)

5/ Deployment Features configuration (This proposal) - a list of features
to configure, the list is nicely generated from information provided in
previous steps, user has all the information to configure those features at
hand and can go through these step by step.

6/ Advanced deployment config - a view providing a way to review
Environment/Roles/Services parameters, search and tweak them if needed.

7/ Deploy.

I believe these steps should cover anything we should need to do for
deployment configuration.

-- Jirka



>
> If this makes sense I can go ahead and push some patches so we can
> iterate on the implementation?
>
> Thanks,
>
> Steve
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Next steps for pre-deployment workflows (e.g derive parameters)

[Jiri Tomasek]

On Wed, Nov 8, 2017 at 11:09 AM, Saravanan KR  wrote:

> Thanks Steven for the update.
>
> Current CLI flow:
> --
> * User need to add -p parameter for the overcloud deploy command with
> workflows to be invoked [1]
> * Plan will update updated to the swift container
> * Derived parameters workflow is initiated
> - For each role
> * Get the introspection data of first node assigned to the role
> * Find the list features based on the services or parameters
> * If dpdk present, run dpdk formulas workflow
> * if sriov is present, run sriov formulas workfow (under
> development)
> * if sriov or dpdk is present, run host formulas workflow
> * if hci present, run hci formulas workflow
>
> Here the order of the formulas workflow invocation is important. For
> example,  in Compute-DPDK-HCI role, HCI formulas should exclude the
> CPUs allocated for DPDK PMD threads, while calculating cpu allocation
> ratio.
>
> I am trying to understand the proposed changes. Is it for assisting UI
> only or changing the existing CLI flow too? If the idea is to invoke
> the individual formulas workflow, it will not be possible with
> existing implementation, need to be re-worked. We need to introduce
> order for formulas workflow and direct fetching and merging of derived
> parameters in plan.
>

So there are several problems we're trying to solve with this proposal. In
general is to provide feature based workflows which will configure these
features, as well as provide means to get current configuration of these
features and provide sensible information about the input for these
workflows.

I think one of the main problem of current implementation is that user is
not able to get any information about input required to provide to run
derivation workflows. That information is purely documentation based and
also involves tweaking deployment-plan which I am convinced is not a good
way to provide the input.

So what we're proposing is to bring up a mechanism of mapping the
derivation workflows to services (roles/environments) so as Steven
described we're able to identify which workflows are possible to run and
provide extensive input definition so user can see what he is configuring
and why (input type, description, label).

This also means that there is several parameter derivation workflows rather
than just one and the input for the workflow is the actual input passed to
mistral (no plan-environment.yaml changes involved). Using this whole
approach means that for each such identified feature, we can provide -
Input details, general feature description (mistral workflow description)
and current configuration (by reaching to mistral workflow execution if
that was run before)

If as you're saying certain workflows depend on each other those should
probably be in one workflow. On the other hand, I think it is not goo
approach to try to put all the parameter derivation workflows into single
workflow.


-- Jirka


>
> As per earlier discussion jtomasek, to invoke derived parameters
> workflow (existing) for a plan, UI requires following information:
> * Whether derived parameters should be invoked for this deployment
> (based on roles and enabled services)
> * If yes, list of parameters, its types, and its default values (and
> choices if present), are required
>
> Did I miss anything?
>
> Regards,
> Saravanan KR
>
> [1] https://github.com/openstack/tripleo-heat-templates/blob/
> master/plan-samples/plan-environment-derived-params.yaml
>
> On Wed, Nov 8, 2017 at 2:39 PM, Bogdan Dobrelya 
> wrote:
> > On 11/8/17 6:09 AM, Steven Hardy wrote:
> >>
> >> Hi all,
> >>
> >> Today I had a productive hallway discussion with jtomasek and
> >> stevebaker re $subject, so I wanted to elaborate here for the benefit
> >> of those folks not present.  Hopefully we can get feedback on the
> >> ideas and see if it makes sense to continue and work on some patches:
> >>
> >> The problem under discussion is how do we run pre-deployment workflows
> >> (such as those integrated recently to calculate derived parameters,
> >> and in future perhaps also those which download container images etc),
> >> and in particular how do we make these discoverable via the UI
> >> (including any input parameters).
> >>
> >> The idea we came up with has two parts:
> >>
> >> 1. Add a new optional section to roles_data for services that require
> >> pre-deploy workflows
> >>
> >> E.g something like this:
> >>
> >>   pre_deploy_workflows:
> >>  - derive_params:
> >>workflow_name:
> >> tripleo.derive_params_formulas.v1.dpdk_derive_params
> >>inputs:
> >>...
> >>
> >> This would allow us to associate a specific mistral workflow with a
> >> given service template, and also work around the fact that currently
> >> mistral inputs don't have any schema (only key/value input) as we
> >> could encode the required 

Re: [openstack-dev] [TripleO] Next steps for pre-deployment workflows (e.g derive parameters)

[James Slagle]

On Wed, Nov 8, 2017 at 12:09 AM, Steven Hardy  wrote:
> Hi all,
>
> Today I had a productive hallway discussion with jtomasek and
> stevebaker re $subject, so I wanted to elaborate here for the benefit
> of those folks not present.  Hopefully we can get feedback on the
> ideas and see if it makes sense to continue and work on some patches:
>
> The problem under discussion is how do we run pre-deployment workflows
> (such as those integrated recently to calculate derived parameters,
> and in future perhaps also those which download container images etc),
> and in particular how do we make these discoverable via the UI
> (including any input parameters).
>
> The idea we came up with has two parts:
>
> 1. Add a new optional section to roles_data for services that require
> pre-deploy workflows
>
> E.g something like this:
>
>  pre_deploy_workflows:
> - derive_params:
>   workflow_name:
> tripleo.derive_params_formulas.v1.dpdk_derive_params
>   inputs:
>   ...
>
> This would allow us to associate a specific mistral workflow with a
> given service template, and also work around the fact that currently
> mistral inputs don't have any schema (only key/value input) as we
> could encode the required type and any constraints in the inputs block
> (clearly this could be removed in future should typed parameters
> become available in mistral).
>
> 2. Add a new workflow that calculates the enabled services and returns
> all pre_deploy_workflows
>
> This would take all enabled environments, then use heat to validate
> the configuration and return the merged resource registry (which will
> require https://review.openstack.org/#/c/509760/), then we would
> iterate over all enabled services in the registry and extract a given
> roles_data key (e.g pre_deploy_workflows)
>
> The result of the workflow would be a list of all pre_deploy_workflows
> for all enabled services, which the UI could then use to run the
> workflows as part of the pre-deploy process.
>
> If this makes sense I can go ahead and push some patches so we can
> iterate on the implementation?

Instead of using mistral to express the pre_deploy workflows could we
just use pure ansible tasks/playbooks? We already use pure ansible for
upgrade and deploy, it seems disjointed to then require Mistral for
pre-deploy.

I think we should continue to move in the direction that the output
from Heat is ansible, which can then be driven via either Mistral or
ad-hoc via ansible-playbook. That pattern is more aligned with the
existing work around config-download as well. So, we would have a
Mistral workflow that automates what you describe above, *except* for
the detail that pre_deploy_workflows becomes either pre_deploy_tasks
or pre_deploy_playbooks, and the output of the Heat validation step is
pure ansible.

From the UI's perspective, whether the output from Heat is pure
ansible or mistral workflows should be an implementation detail hidden
behind a higher level mistral workflow that automates the full
"generation" of all pre_deploy stuff.

Also keep in mind Emilien's proposal to switch all scenario testing to
be driven via undercloud deploy. In that environment, there is no
Mistral, and everything is driven via config-download. What you're
proposing (unless I misunderstand) creates a tight coupling between
the Heat/Mistral ordering and requires one to use Mistral as part of
the deployment. Right now, we have no such requirement, and are able
to drive a deployment with only an ephemeral Heat and
ansible-playbook, which is the basis of how undercloud deploy works.

-- 
-- James Slagle
--

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Next steps for pre-deployment workflows (e.g derive parameters)

[Saravanan KR]

Thanks Steven for the update.

Current CLI flow:
--
* User need to add -p parameter for the overcloud deploy command with
workflows to be invoked [1]
* Plan will update updated to the swift container
* Derived parameters workflow is initiated
- For each role
* Get the introspection data of first node assigned to the role
* Find the list features based on the services or parameters
* If dpdk present, run dpdk formulas workflow
* if sriov is present, run sriov formulas workfow (under development)
* if sriov or dpdk is present, run host formulas workflow
* if hci present, run hci formulas workflow

Here the order of the formulas workflow invocation is important. For
example,  in Compute-DPDK-HCI role, HCI formulas should exclude the
CPUs allocated for DPDK PMD threads, while calculating cpu allocation
ratio.

I am trying to understand the proposed changes. Is it for assisting UI
only or changing the existing CLI flow too? If the idea is to invoke
the individual formulas workflow, it will not be possible with
existing implementation, need to be re-worked. We need to introduce
order for formulas workflow and direct fetching and merging of derived
parameters in plan.

As per earlier discussion jtomasek, to invoke derived parameters
workflow (existing) for a plan, UI requires following information:
* Whether derived parameters should be invoked for this deployment
(based on roles and enabled services)
* If yes, list of parameters, its types, and its default values (and
choices if present), are required

Did I miss anything?

Regards,
Saravanan KR

[1] 
https://github.com/openstack/tripleo-heat-templates/blob/master/plan-samples/plan-environment-derived-params.yaml

On Wed, Nov 8, 2017 at 2:39 PM, Bogdan Dobrelya  wrote:
> On 11/8/17 6:09 AM, Steven Hardy wrote:
>>
>> Hi all,
>>
>> Today I had a productive hallway discussion with jtomasek and
>> stevebaker re $subject, so I wanted to elaborate here for the benefit
>> of those folks not present.  Hopefully we can get feedback on the
>> ideas and see if it makes sense to continue and work on some patches:
>>
>> The problem under discussion is how do we run pre-deployment workflows
>> (such as those integrated recently to calculate derived parameters,
>> and in future perhaps also those which download container images etc),
>> and in particular how do we make these discoverable via the UI
>> (including any input parameters).
>>
>> The idea we came up with has two parts:
>>
>> 1. Add a new optional section to roles_data for services that require
>> pre-deploy workflows
>>
>> E.g something like this:
>>
>>   pre_deploy_workflows:
>>  - derive_params:
>>workflow_name:
>> tripleo.derive_params_formulas.v1.dpdk_derive_params
>>inputs:
>>...
>>
>> This would allow us to associate a specific mistral workflow with a
>> given service template, and also work around the fact that currently
>> mistral inputs don't have any schema (only key/value input) as we
>> could encode the required type and any constraints in the inputs block
>> (clearly this could be removed in future should typed parameters
>> become available in mistral).
>>
>> 2. Add a new workflow that calculates the enabled services and returns
>> all pre_deploy_workflows
>>
>> This would take all enabled environments, then use heat to validate
>> the configuration and return the merged resource registry (which will
>> require https://review.openstack.org/#/c/509760/), then we would
>> iterate over all enabled services in the registry and extract a given
>> roles_data key (e.g pre_deploy_workflows)
>>
>> The result of the workflow would be a list of all pre_deploy_workflows
>> for all enabled services, which the UI could then use to run the
>> workflows as part of the pre-deploy process.
>>
>> If this makes sense I can go ahead and push some patches so we can
>> iterate on the implementation?
>
>
> I apologise for a generic/non-techy comment: it would be nice to keep
> required workflows near the services' definition templates, to keep it as
> much self-contained as possible. IIUC, that's covered by #1.
> For future steps, I'd like to see all of the "bulk processing" to sit in
> those templates as well.
>
>>
>> Thanks,
>>
>> Steve
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> --
> Best regards,
> Bogdan Dobrelya,
> Irc #bogdando
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Next steps for pre-deployment workflows (e.g derive parameters)

[Bogdan Dobrelya]

On 11/8/17 6:09 AM, Steven Hardy wrote:

Hi all,

Today I had a productive hallway discussion with jtomasek and
stevebaker re $subject, so I wanted to elaborate here for the benefit
of those folks not present.  Hopefully we can get feedback on the
ideas and see if it makes sense to continue and work on some patches:

The problem under discussion is how do we run pre-deployment workflows
(such as those integrated recently to calculate derived parameters,
and in future perhaps also those which download container images etc),
and in particular how do we make these discoverable via the UI
(including any input parameters).

The idea we came up with has two parts:

1. Add a new optional section to roles_data for services that require
pre-deploy workflows

E.g something like this:

  pre_deploy_workflows:
 - derive_params:
   workflow_name:
tripleo.derive_params_formulas.v1.dpdk_derive_params
   inputs:
   ...

This would allow us to associate a specific mistral workflow with a
given service template, and also work around the fact that currently
mistral inputs don't have any schema (only key/value input) as we
could encode the required type and any constraints in the inputs block
(clearly this could be removed in future should typed parameters
become available in mistral).

2. Add a new workflow that calculates the enabled services and returns
all pre_deploy_workflows

This would take all enabled environments, then use heat to validate
the configuration and return the merged resource registry (which will
require https://review.openstack.org/#/c/509760/), then we would
iterate over all enabled services in the registry and extract a given
roles_data key (e.g pre_deploy_workflows)

The result of the workflow would be a list of all pre_deploy_workflows
for all enabled services, which the UI could then use to run the
workflows as part of the pre-deploy process.

If this makes sense I can go ahead and push some patches so we can
iterate on the implementation?


I apologise for a generic/non-techy comment: it would be nice to keep 
required workflows near the services' definition templates, to keep it 
as much self-contained as possible. IIUC, that's covered by #1.
For future steps, I'd like to see all of the "bulk processing" to sit in 
those templates as well.




Thanks,

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] Next steps for pre-deployment workflows (e.g derive parameters)

[Steven Hardy]

Hi all,

Today I had a productive hallway discussion with jtomasek and
stevebaker re $subject, so I wanted to elaborate here for the benefit
of those folks not present.  Hopefully we can get feedback on the
ideas and see if it makes sense to continue and work on some patches:

The problem under discussion is how do we run pre-deployment workflows
(such as those integrated recently to calculate derived parameters,
and in future perhaps also those which download container images etc),
and in particular how do we make these discoverable via the UI
(including any input parameters).

The idea we came up with has two parts:

1. Add a new optional section to roles_data for services that require
pre-deploy workflows

E.g something like this:

 pre_deploy_workflows:
- derive_params:
  workflow_name:
tripleo.derive_params_formulas.v1.dpdk_derive_params
  inputs:
  ...

This would allow us to associate a specific mistral workflow with a
given service template, and also work around the fact that currently
mistral inputs don't have any schema (only key/value input) as we
could encode the required type and any constraints in the inputs block
(clearly this could be removed in future should typed parameters
become available in mistral).

2. Add a new workflow that calculates the enabled services and returns
all pre_deploy_workflows

This would take all enabled environments, then use heat to validate
the configuration and return the merged resource registry (which will
require https://review.openstack.org/#/c/509760/), then we would
iterate over all enabled services in the registry and extract a given
roles_data key (e.g pre_deploy_workflows)

The result of the workflow would be a list of all pre_deploy_workflows
for all enabled services, which the UI could then use to run the
workflows as part of the pre-deploy process.

If this makes sense I can go ahead and push some patches so we can
iterate on the implementation?

Thanks,

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev