Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates
Better late than never... On 16/12/16 08:57, Steven Hardy wrote: I know we've previously tried to steer execute/run type actions to signal driven interfaces (and I myself have opposed these kinds of resources in the past, to be honest). However, I can't currently see a better way to handle this requirement, and given it may be pretty easy to fix (refactor handle_signal and add a boolean to each handle_foo function), I think this discussion warrants revisiting. I think we need to get away from thinking of this as "I just need to run a script in the middle of Heat's workflow, let me do it!". That's verb thinking and that's the reason I -1'd https://review.openstack.org/267770 originally. We need to think of it as "I have some external thing that I need to manage, but I can't teach Heat about it because it means nothing to my cloud operator". That's noun thinking and something we can actually work with. For that reason I started talking to a few folks a while back about implementing an equivalent of CloudFormation's "CustomResource"[1]. Basically, it sends a message to an SNS topic (c.f. Zaqar notification) and waits for a success/failure response. We have the technology now to trigger Mistral workflows from Zaqar messages.[2] It's clear that this is both conceptually sound and solves a number of problems with the best available solution, which is triggering workflows via Zaqar notifications from user hooks: - There's no way to fail a particular resource. - The Zaqar queue has to be passed in to the environment when creating the stack. This means that the queue has to be created outside of the stack. - All stacks in the tree share the same queue for hook notifications. One reason that CloudFormation implements CustomResource as a message to a topic is that it allows integration with third-party service providers. That's a valid use case, but I think there's an equally or more valid use case for integrating with stuff in your own tenant that is just under the end user's control rather than the operators. For that case, it makes sense to cut out the Zaqar middleman and just trigger the Mistral execution directly. So I am on board with a Mistral execution resource, BUT: * I don't think we should call it a Mistral Execution. I think we should give it a name that indicates it's managing some external thing by executing Mistral workflows. * We need to learn the lesson of SoftwareComponent and include _all_ the different phases of the lifecycle in the _same resource_. The user should not have to assemble the different phases (Create/Update/Delete) of their external resource using discrete workflows, like they do with SoftwareConfigs. It's too much to ask them to understand how to build the dependency graphs correctly - it took me over a year to work it out[3] and that was my whole job. * We should also learn the lesson of https://launchpad.net/bugs/1595040 and implement a mechanism to allow the template author to select (in advance) between update-in-place and update-replace for any given update from the get-go. I also left similar comments on https://review.openstack.org/#/c/420664/ but I wanted to go into the background in more depth here. We may _also_ want to implement something like CustomResource in the future, but I agree that this is the higher priority. cheers, Zane. [1] http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html [2] http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Zaqar::MistralTrigger [3] https://review.openstack.org/gitweb?p=openstack/heat.git;a=commitdiff;h=8b732241df6007c3005b14413ee1fe047eb4d108 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates
On Fri, Dec 16, 2016 at 04:02:51PM +0100, Thomas Herve wrote: > On Fri, Dec 16, 2016 at 2:57 PM, Steven Hardy wrote: > > On Fri, Dec 16, 2016 at 02:03:10PM +0100, Thomas Herve wrote: > >> On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente > >> wrote: > >> > I was wondering if it would make sense to have a property for the > >> > existing > >> > Workflow resource to let the user decide if the workflow should *also* be > >> > triggered on CREATE/UPDATE? And if it would make sense to block the > >> > Workflow > >> > resource until the execution result is returned in that case? > >> > >> I'm not super in favor of that. It's conflicting 2 different concepts here. > > > > Well, they're already conflated in the mistral workflow resource, because > > we allow creating an execution by sending a signal: > > > > https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/mistral/workflow.py#L586 > > Right, I mentioned that elsewhere. But it doesn't change the resource > interface, for better or worse. > > > That satisfies the requirement for asynchronous (signal/alarm driven) > > workflow execution, but we need the synchronous version that can be fully > > integrated into the heat stack lifecycle (without external dependencies on > > alarms etc). > > > > I know we've previously tried to steer execute/run type actions to signal > > driven interfaces (and I myself have opposed these kinds of resources in > > the past, to be honest). However, I can't currently see a better way to > > handle this requirement, and given it may be pretty easy to fix (refactor > > handle_signal and add a boolean to each handle_foo function), I think this > > discussion warrants revisiting. > > It's unclear what changed since the discussion happened, just that you > have a use case without another approach possible? Well, honestly yes - my perspective has changed somewhat as I've had recent first-hand exposure to several problems which would be very nicely solved by the additional Execution resource (or interface) under discussion. Also, it's unclear if the hinted-at future model involving zaqar notifications actually solves this (or at least not in it's current form), so I'm looking at this from a pragmatic standpoint and trying to find a relatively low-impact way to just solve the problem and move forward. E.g in https://review.openstack.org/#/c/267770/ the execution resource was rejected based partly on the expectation of a future Zaqar driven model satisfying the same requirement, which was futher described in this summit talk: https://www.openstack.org/videos/video/building-self-healing-applications-with-aodh-zaqar-and-mistral While the patterns discussed there are definitely compelling, they still don't address the requirement of wanting to just run some mistral workflow as part of the heat stack lifecycle? AFAICT it's describing an independent (of Heat) system which can take actions in an event/alarm driven manner e.g the interaction between telemetry services, Zaqar and Mistral are configured by, but essentially asynchronous to Heat - all great, but doesn't solve our problem? > >> > Alternatively, would an ex-novo Execution resource make more sense? > >> > >> We had some discussions here: https://review.openstack.org/267770. > >> Executing things as part of a template is a tricky proposition. I > >> think we'd like it to be more akin to software deployments, where it > >> runs on actions. We also were talking about doing something like AWS > >> CustomResource in Heat, which may look like WorkflowExecution (at > >> least one part of it). > > > > Yeah I think whichever approach we take, a list of actions similar to > > SoftwareDeployment makes sense, then you can elect to run a specific > > workflow at any state transition in the lifecycle. > > Among the various solutions, I think that's the one I like the best > for now. It doesn't touch the workflow resource interface, and it > seems to fit relatively naturally (an API to call, a state to check, > etc). Yeah, honestly I think it's a simple step which some folks will find useful, and it's likely to be low-imact from a heat perspective (folks can continue to prefer the Aodh/Zaqar approach if it suits their use-case better). As always, open to suggestions of a better way to approach this, but atm I'm feeling like just going ahead with the WorkflowExecution resource (ensuring it can handle all lifecycle actions, e.g just like SoftwareDeployment/SoftwareCompoinent which is quite conceptually similar, as noted in 267770 ...) is the most workable option. Thanks! Steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates
On Fri, Dec 16, 2016 at 2:57 PM, Steven Hardy wrote: > On Fri, Dec 16, 2016 at 02:03:10PM +0100, Thomas Herve wrote: >> On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente wrote: >> > I was wondering if it would make sense to have a property for the existing >> > Workflow resource to let the user decide if the workflow should *also* be >> > triggered on CREATE/UPDATE? And if it would make sense to block the >> > Workflow >> > resource until the execution result is returned in that case? >> >> I'm not super in favor of that. It's conflicting 2 different concepts here. > > Well, they're already conflated in the mistral workflow resource, because > we allow creating an execution by sending a signal: > > https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/mistral/workflow.py#L586 Right, I mentioned that elsewhere. But it doesn't change the resource interface, for better or worse. > That satisfies the requirement for asynchronous (signal/alarm driven) > workflow execution, but we need the synchronous version that can be fully > integrated into the heat stack lifecycle (without external dependencies on > alarms etc). > > I know we've previously tried to steer execute/run type actions to signal > driven interfaces (and I myself have opposed these kinds of resources in > the past, to be honest). However, I can't currently see a better way to > handle this requirement, and given it may be pretty easy to fix (refactor > handle_signal and add a boolean to each handle_foo function), I think this > discussion warrants revisiting. It's unclear what changed since the discussion happened, just that you have a use case without another approach possible? >> > Alternatively, would an ex-novo Execution resource make more sense? >> >> We had some discussions here: https://review.openstack.org/267770. >> Executing things as part of a template is a tricky proposition. I >> think we'd like it to be more akin to software deployments, where it >> runs on actions. We also were talking about doing something like AWS >> CustomResource in Heat, which may look like WorkflowExecution (at >> least one part of it). > > Yeah I think whichever approach we take, a list of actions similar to > SoftwareDeployment makes sense, then you can elect to run a specific > workflow at any state transition in the lifecycle. Among the various solutions, I think that's the one I like the best for now. It doesn't touch the workflow resource interface, and it seems to fit relatively naturally (an API to call, a state to check, etc). >> > Or are there different ideas, approaches to the problem? >> >> If you could define the event outside of Heat (in your example, >> publish something when a swift node is available), then you could use >> Zaqar to trigger your workflow. If you want Heat to block that won't >> do it though. > > Yeah that doesn't solve our use-case, we want to run a workflow during an > overcloud stack create, wait for the result, then continue (or fail). Noted. -- Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates
On Fri, Dec 16, 2016 at 02:03:10PM +0100, Thomas Herve wrote: > On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente wrote: > > hi, > > > > we're trying to address in TripleO a couple of use cases for which we'd like > > to trigger a Mistral workflow from a Heat template. > > > > One example where this would be useful is the creation of the Swift rings, > > which need some data related to the Heat stack (like the list of Swift nodes > > and their disks), so it can't be executed in advance, yet it provides data > > which is needed to complete successfully the deployment of the overcloud. > > > > Currently we can create a workflow from Heat, but we can't trigger its > > execution and also we can't block Heat on the result of the execution. > > You can trigger it out of band with resource signal. But you're right, > Heat won't wait for the result. > > > I was wondering if it would make sense to have a property for the existing > > Workflow resource to let the user decide if the workflow should *also* be > > triggered on CREATE/UPDATE? And if it would make sense to block the Workflow > > resource until the execution result is returned in that case? > > I'm not super in favor of that. It's conflicting 2 different concepts here. Well, they're already conflated in the mistral workflow resource, because we allow creating an execution by sending a signal: https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/mistral/workflow.py#L586 That satisfies the requirement for asynchronous (signal/alarm driven) workflow execution, but we need the synchronous version that can be fully integrated into the heat stack lifecycle (without external dependencies on alarms etc). I know we've previously tried to steer execute/run type actions to signal driven interfaces (and I myself have opposed these kinds of resources in the past, to be honest). However, I can't currently see a better way to handle this requirement, and given it may be pretty easy to fix (refactor handle_signal and add a boolean to each handle_foo function), I think this discussion warrants revisiting. > > Alternatively, would an ex-novo Execution resource make more sense? > > We had some discussions here: https://review.openstack.org/267770. > Executing things as part of a template is a tricky proposition. I > think we'd like it to be more akin to software deployments, where it > runs on actions. We also were talking about doing something like AWS > CustomResource in Heat, which may look like WorkflowExecution (at > least one part of it). Yeah I think whichever approach we take, a list of actions similar to SoftwareDeployment makes sense, then you can elect to run a specific workflow at any state transition in the lifecycle. > > Or are there different ideas, approaches to the problem? > > If you could define the event outside of Heat (in your example, > publish something when a swift node is available), then you could use > Zaqar to trigger your workflow. If you want Heat to block that won't > do it though. Yeah that doesn't solve our use-case, we want to run a workflow during an overcloud stack create, wait for the result, then continue (or fail). Thanks! Steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates
On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente wrote: > hi, > > we're trying to address in TripleO a couple of use cases for which we'd like > to trigger a Mistral workflow from a Heat template. > > One example where this would be useful is the creation of the Swift rings, > which need some data related to the Heat stack (like the list of Swift nodes > and their disks), so it can't be executed in advance, yet it provides data > which is needed to complete successfully the deployment of the overcloud. > > Currently we can create a workflow from Heat, but we can't trigger its > execution and also we can't block Heat on the result of the execution. You can trigger it out of band with resource signal. But you're right, Heat won't wait for the result. > I was wondering if it would make sense to have a property for the existing > Workflow resource to let the user decide if the workflow should *also* be > triggered on CREATE/UPDATE? And if it would make sense to block the Workflow > resource until the execution result is returned in that case? I'm not super in favor of that. It's conflicting 2 different concepts here. > Alternatively, would an ex-novo Execution resource make more sense? We had some discussions here: https://review.openstack.org/267770. Executing things as part of a template is a tricky proposition. I think we'd like it to be more akin to software deployments, where it runs on actions. We also were talking about doing something like AWS CustomResource in Heat, which may look like WorkflowExecution (at least one part of it). > Or are there different ideas, approaches to the problem? If you could define the event outside of Heat (in your example, publish something when a swift node is available), then you could use Zaqar to trigger your workflow. If you want Heat to block that won't do it though. -- Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates
On 12/16/2016 01:56 PM, Christian Schwede wrote: we're trying to address in TripleO a couple of use cases for which we'd like to trigger a Mistral workflow from a Heat template. One example where this would be useful is the creation of the Swift rings, which need some data related to the Heat stack (like the list of Swift nodes and their disks), so it can't be executed in advance, yet it provides data which is needed to complete successfully the deployment of the overcloud. Currently we can create a workflow from Heat, but we can't trigger its execution and also we can't block Heat on the result of the execution. I was wondering if it would make sense to have a property for the existing Workflow resource to let the user decide if the workflow should *also* be triggered on CREATE/UPDATE? And if it would make sense to block the Workflow resource until the execution result is returned in that case? I think it needs to be triggered a bit later actually? For the Swift use case it needs to be executed after all instances are created (but preferably before starting any Puppet actions on the nodes), not when the CREATE/UPDATE itself actually starts. yep, I was referring to the workflow resource CREATE/UPDATE action we have complete control in Heat about when the workflow resource itself should be created Alternatively, would an ex-novo Execution resource make more sense? Or are there different ideas, approaches to the problem? As a workaround for now I'm using the signal URL and trigger it in a shell script on the nodes (the shell script is running anyways to fetch and validate the rings). To avoid multiple parallel workflow executions triggered by a dozen nodes I set a flag in the Mistral environment; further actions will immediately return then. I'd prefer a different and cleaner approach like you proposed but for me that's working well for the moment. ack -- Giulio Fidente GPG KEY: 08D733BA __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates
> we're trying to address in TripleO a couple of use cases for which we'd > like to trigger a Mistral workflow from a Heat template. > > One example where this would be useful is the creation of the Swift > rings, which need some data related to the Heat stack (like the list of > Swift nodes and their disks), so it can't be executed in advance, yet it > provides data which is needed to complete successfully the deployment of > the overcloud. > > Currently we can create a workflow from Heat, but we can't trigger its > execution and also we can't block Heat on the result of the execution. > > I was wondering if it would make sense to have a property for the > existing Workflow resource to let the user decide if the workflow should > *also* be triggered on CREATE/UPDATE? And if it would make sense to > block the Workflow resource until the execution result is returned in > that case? I think it needs to be triggered a bit later actually? For the Swift use case it needs to be executed after all instances are created (but preferably before starting any Puppet actions on the nodes), not when the CREATE/UPDATE itself actually starts. > Alternatively, would an ex-novo Execution resource make more sense? > > Or are there different ideas, approaches to the problem? As a workaround for now I'm using the signal URL and trigger it in a shell script on the nodes (the shell script is running anyways to fetch and validate the rings). To avoid multiple parallel workflow executions triggered by a dozen nodes I set a flag in the Mistral environment; further actions will immediately return then. I'd prefer a different and cleaner approach like you proposed but for me that's working well for the moment. -- Christian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Heat][TripleO] How to run mistral workflows via templates
hi, we're trying to address in TripleO a couple of use cases for which we'd like to trigger a Mistral workflow from a Heat template. One example where this would be useful is the creation of the Swift rings, which need some data related to the Heat stack (like the list of Swift nodes and their disks), so it can't be executed in advance, yet it provides data which is needed to complete successfully the deployment of the overcloud. Currently we can create a workflow from Heat, but we can't trigger its execution and also we can't block Heat on the result of the execution. I was wondering if it would make sense to have a property for the existing Workflow resource to let the user decide if the workflow should *also* be triggered on CREATE/UPDATE? And if it would make sense to block the Workflow resource until the execution result is returned in that case? Alternatively, would an ex-novo Execution resource make more sense? Or are there different ideas, approaches to the problem? -- Giulio Fidente GPG KEY: 08D733BA __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev