Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates

[Zane Bitter]

Better late than never...

On 16/12/16 08:57, Steven Hardy wrote:


I know we've previously tried to steer execute/run type actions to signal
driven interfaces (and I myself have opposed these kinds of resources in
the past, to be honest).  However, I can't currently see a better way to
handle this requirement, and given it may be pretty easy to fix (refactor
handle_signal and add a boolean to each handle_foo function), I think this
discussion warrants revisiting.


I think we need to get away from thinking of this as "I just need to run 
a script in the middle of Heat's workflow, let me do it!". That's verb 
thinking and that's the reason I -1'd 
https://review.openstack.org/267770 originally.


We need to think of it as "I have some external thing that I need to 
manage, but I can't teach Heat about it because it means nothing to my 
cloud operator". That's noun thinking and something we can actually work 
with.


For that reason I started talking to a few folks a while back about 
implementing an equivalent of CloudFormation's "CustomResource"[1]. 
Basically, it sends a message to an SNS topic (c.f. Zaqar notification) 
and waits for a success/failure response. We have the technology now to 
trigger Mistral workflows from Zaqar messages.[2] It's clear that this 
is both conceptually sound and solves a number of problems with the best 
available solution, which is triggering workflows via Zaqar 
notifications from user hooks:

 - There's no way to fail a particular resource.
 - The Zaqar queue has to be passed in to the environment when creating 
the stack. This means that the queue has to be created outside of the stack.

 - All stacks in the tree share the same queue for hook notifications.

One reason that CloudFormation implements CustomResource as a message to 
a topic is that it allows integration with third-party service 
providers. That's a valid use case, but I think there's an equally or 
more valid use case for integrating with stuff in your own tenant that 
is just under the end user's control rather than the operators. For that 
case, it makes sense to cut out the Zaqar middleman and just trigger the 
Mistral execution directly.


So I am on board with a Mistral execution resource, BUT:

 * I don't think we should call it a Mistral Execution. I think we 
should give it a name that indicates it's managing some external thing 
by executing Mistral workflows.
 * We need to learn the lesson of SoftwareComponent and include _all_ 
the different phases of the lifecycle in the _same resource_. The user 
should not have to assemble the different phases (Create/Update/Delete) 
of their external resource using discrete workflows, like they do with 
SoftwareConfigs. It's too much to ask them to understand how to build 
the dependency graphs correctly - it took me over a year to work it 
out[3] and that was my whole job.
 * We should also learn the lesson of 
https://launchpad.net/bugs/1595040 and implement a mechanism to allow 
the template author to select (in advance) between update-in-place and 
update-replace for any given update from the get-go.


I also left similar comments on https://review.openstack.org/#/c/420664/ 
but I wanted to go into the background in more depth here.


We may _also_ want to implement something like CustomResource in the 
future, but I agree that this is the higher priority.


cheers,
Zane.

[1] 
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
[2] 
http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Zaqar::MistralTrigger
[3] 
https://review.openstack.org/gitweb?p=openstack/heat.git;a=commitdiff;h=8b732241df6007c3005b14413ee1fe047eb4d108


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates

[Thomas Herve]

On Fri, Dec 16, 2016 at 2:57 PM, Steven Hardy  wrote:
> On Fri, Dec 16, 2016 at 02:03:10PM +0100, Thomas Herve wrote:
>> On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente  wrote:
>> > I was wondering if it would make sense to have a property for the existing
>> > Workflow resource to let the user decide if the workflow should *also* be
>> > triggered on CREATE/UPDATE? And if it would make sense to block the 
>> > Workflow
>> > resource until the execution result is returned in that case?
>>
>> I'm not super in favor of that. It's conflicting 2 different concepts here.
>
> Well, they're already conflated in the mistral workflow resource, because
> we allow creating an execution by sending a signal:
>
> https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/mistral/workflow.py#L586

Right, I mentioned that elsewhere. But it doesn't change the resource
interface, for better or worse.

> That satisfies the requirement for asynchronous (signal/alarm driven)
> workflow execution, but we need the synchronous version that can be fully
> integrated into the heat stack lifecycle (without external dependencies on
> alarms etc).
>
> I know we've previously tried to steer execute/run type actions to signal
> driven interfaces (and I myself have opposed these kinds of resources in
> the past, to be honest).  However, I can't currently see a better way to
> handle this requirement, and given it may be pretty easy to fix (refactor
> handle_signal and add a boolean to each handle_foo function), I think this
> discussion warrants revisiting.

It's unclear what changed since the discussion happened, just that you
have a use case without another approach possible?

>> > Alternatively, would an ex-novo Execution resource make more sense?
>>
>> We had some discussions here: https://review.openstack.org/267770.
>> Executing things as part of a template is a tricky proposition. I
>> think we'd like it to be more akin to software deployments, where it
>> runs on actions. We also were talking about doing something like AWS
>> CustomResource in Heat, which may look like WorkflowExecution (at
>> least one part of it).
>
> Yeah I think whichever approach we take, a list of actions similar to
> SoftwareDeployment makes sense, then you can elect to run a specific
> workflow at any state transition in the lifecycle.

Among the various solutions, I think that's the one I like the best
for now. It doesn't touch the workflow resource interface, and it
seems to fit relatively naturally (an API to call, a state to check,
etc).

>> > Or are there different ideas, approaches to the problem?
>>
>> If you could define the event outside of Heat (in your example,
>> publish something when a swift node is available), then you could use
>> Zaqar to trigger your workflow. If you want Heat to block that won't
>> do it though.
>
> Yeah that doesn't solve our use-case, we want to run a workflow during an
> overcloud stack create, wait for the result, then continue (or fail).

Noted.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates

[Steven Hardy]

On Fri, Dec 16, 2016 at 02:03:10PM +0100, Thomas Herve wrote:
> On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente  wrote:
> > hi,
> >
> > we're trying to address in TripleO a couple of use cases for which we'd like
> > to trigger a Mistral workflow from a Heat template.
> >
> > One example where this would be useful is the creation of the Swift rings,
> > which need some data related to the Heat stack (like the list of Swift nodes
> > and their disks), so it can't be executed in advance, yet it provides data
> > which is needed to complete successfully the deployment of the overcloud.
> >
> > Currently we can create a workflow from Heat, but we can't trigger its
> > execution and also we can't block Heat on the result of the execution.
> 
> You can trigger it out of band with resource signal. But you're right,
> Heat won't wait for the result.
> 
> > I was wondering if it would make sense to have a property for the existing
> > Workflow resource to let the user decide if the workflow should *also* be
> > triggered on CREATE/UPDATE? And if it would make sense to block the Workflow
> > resource until the execution result is returned in that case?
> 
> I'm not super in favor of that. It's conflicting 2 different concepts here.

Well, they're already conflated in the mistral workflow resource, because
we allow creating an execution by sending a signal:

https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/mistral/workflow.py#L586

That satisfies the requirement for asynchronous (signal/alarm driven)
workflow execution, but we need the synchronous version that can be fully
integrated into the heat stack lifecycle (without external dependencies on
alarms etc).

I know we've previously tried to steer execute/run type actions to signal
driven interfaces (and I myself have opposed these kinds of resources in
the past, to be honest).  However, I can't currently see a better way to
handle this requirement, and given it may be pretty easy to fix (refactor
handle_signal and add a boolean to each handle_foo function), I think this
discussion warrants revisiting.

> > Alternatively, would an ex-novo Execution resource make more sense?
> 
> We had some discussions here: https://review.openstack.org/267770.
> Executing things as part of a template is a tricky proposition. I
> think we'd like it to be more akin to software deployments, where it
> runs on actions. We also were talking about doing something like AWS
> CustomResource in Heat, which may look like WorkflowExecution (at
> least one part of it).

Yeah I think whichever approach we take, a list of actions similar to
SoftwareDeployment makes sense, then you can elect to run a specific
workflow at any state transition in the lifecycle.

> > Or are there different ideas, approaches to the problem?
> 
> If you could define the event outside of Heat (in your example,
> publish something when a swift node is available), then you could use
> Zaqar to trigger your workflow. If you want Heat to block that won't
> do it though.

Yeah that doesn't solve our use-case, we want to run a workflow during an
overcloud stack create, wait for the result, then continue (or fail).

Thanks!

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates

[Thomas Herve]

On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente  wrote:
> hi,
>
> we're trying to address in TripleO a couple of use cases for which we'd like
> to trigger a Mistral workflow from a Heat template.
>
> One example where this would be useful is the creation of the Swift rings,
> which need some data related to the Heat stack (like the list of Swift nodes
> and their disks), so it can't be executed in advance, yet it provides data
> which is needed to complete successfully the deployment of the overcloud.
>
> Currently we can create a workflow from Heat, but we can't trigger its
> execution and also we can't block Heat on the result of the execution.

You can trigger it out of band with resource signal. But you're right,
Heat won't wait for the result.

> I was wondering if it would make sense to have a property for the existing
> Workflow resource to let the user decide if the workflow should *also* be
> triggered on CREATE/UPDATE? And if it would make sense to block the Workflow
> resource until the execution result is returned in that case?

I'm not super in favor of that. It's conflicting 2 different concepts here.

> Alternatively, would an ex-novo Execution resource make more sense?

We had some discussions here: https://review.openstack.org/267770.
Executing things as part of a template is a tricky proposition. I
think we'd like it to be more akin to software deployments, where it
runs on actions. We also were talking about doing something like AWS
CustomResource in Heat, which may look like WorkflowExecution (at
least one part of it).

> Or are there different ideas, approaches to the problem?

If you could define the event outside of Heat (in your example,
publish something when a swift node is available), then you could use
Zaqar to trigger your workflow. If you want Heat to block that won't
do it though.


-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates

[Giulio Fidente]

On 12/16/2016 01:56 PM, Christian Schwede wrote:

we're trying to address in TripleO a couple of use cases for which we'd
like to trigger a Mistral workflow from a Heat template.

One example where this would be useful is the creation of the Swift
rings, which need some data related to the Heat stack (like the list of
Swift nodes and their disks), so it can't be executed in advance, yet it
provides data which is needed to complete successfully the deployment of
the overcloud.

Currently we can create a workflow from Heat, but we can't trigger its
execution and also we can't block Heat on the result of the execution.

I was wondering if it would make sense to have a property for the
existing Workflow resource to let the user decide if the workflow should
*also* be triggered on CREATE/UPDATE? And if it would make sense to
block the Workflow resource until the execution result is returned in
that case?


I think it needs to be triggered a bit later actually? For the Swift use
case it needs to be executed after all instances are created (but
preferably before starting any Puppet actions on the nodes), not when
the CREATE/UPDATE itself actually starts.


yep, I was referring to the workflow resource CREATE/UPDATE action

we have complete control in Heat about when the workflow resource itself 
should be created



Alternatively, would an ex-novo Execution resource make more sense?

Or are there different ideas, approaches to the problem?


As a workaround for now I'm using the signal URL and trigger it in a
shell script on the nodes (the shell script is running anyways to fetch
and validate the rings). To avoid multiple parallel workflow executions
triggered by a dozen nodes I set a flag in the Mistral environment;
further actions will immediately return then.

I'd prefer a different and cleaner approach like you proposed but for me
that's working well for the moment.


ack
--
Giulio Fidente
GPG KEY: 08D733BA

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates

[Christian Schwede]

> we're trying to address in TripleO a couple of use cases for which we'd
> like to trigger a Mistral workflow from a Heat template.
> 
> One example where this would be useful is the creation of the Swift
> rings, which need some data related to the Heat stack (like the list of
> Swift nodes and their disks), so it can't be executed in advance, yet it
> provides data which is needed to complete successfully the deployment of
> the overcloud.
> 
> Currently we can create a workflow from Heat, but we can't trigger its
> execution and also we can't block Heat on the result of the execution.
> 
> I was wondering if it would make sense to have a property for the
> existing Workflow resource to let the user decide if the workflow should
> *also* be triggered on CREATE/UPDATE? And if it would make sense to
> block the Workflow resource until the execution result is returned in
> that case?

I think it needs to be triggered a bit later actually? For the Swift use
case it needs to be executed after all instances are created (but
preferably before starting any Puppet actions on the nodes), not when
the CREATE/UPDATE itself actually starts.

> Alternatively, would an ex-novo Execution resource make more sense?
> 
> Or are there different ideas, approaches to the problem?

As a workaround for now I'm using the signal URL and trigger it in a
shell script on the nodes (the shell script is running anyways to fetch
and validate the rings). To avoid multiple parallel workflow executions
triggered by a dozen nodes I set a flag in the Mistral environment;
further actions will immediately return then.

I'd prefer a different and cleaner approach like you proposed but for me
that's working well for the moment.

-- Christian


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat][TripleO] How to run mistral workflows via templates

[Giulio Fidente]

hi,

we're trying to address in TripleO a couple of use cases for which we'd 
like to trigger a Mistral workflow from a Heat template.


One example where this would be useful is the creation of the Swift 
rings, which need some data related to the Heat stack (like the list of 
Swift nodes and their disks), so it can't be executed in advance, yet it 
provides data which is needed to complete successfully the deployment of 
the overcloud.


Currently we can create a workflow from Heat, but we can't trigger its 
execution and also we can't block Heat on the result of the execution.


I was wondering if it would make sense to have a property for the 
existing Workflow resource to let the user decide if the workflow should 
*also* be triggered on CREATE/UPDATE? And if it would make sense to 
block the Workflow resource until the execution result is returned in 
that case?


Alternatively, would an ex-novo Execution resource make more sense?

Or are there different ideas, approaches to the problem?

--
Giulio Fidente
GPG KEY: 08D733BA

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev