Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
On 18/10/13 20:24, John Davidge -X (jodavidg - AAP3 INC at Cisco) wrote: It looks like this discussion involves many of the issues faced when developing the Curvature Donabe frameworks, which were presented at the Portland Summit - slides and video here: http://www.openstack.org/summit/portland-2013/session-videos/presentation/i nteractive-visual-orchestration-with-curvature-and-donabe Much of the work on the Donabe side revolved around defining a simple JSON-based API for describing the sorts of virtual application templates being discussed. All of the code for both Curvature and Donabe has recently been made open source and is available here: http://ciscosystems.github.io/curvature/ http://ciscosystems.github.io/donabe/ Hey John, Congrats on getting this stuff Open-Sourced BTW (I know it's been out for a while now). Can you be more specific about the parts that are relevant to this discussion? I'd be interested to know how Donabe handles configuring the software on Nova servers for a start. It looks like some of the ground covered by these projects can be helpful to this discussion. Yep, it would be great to get input from any folks in the community who have experience with this problem. cheers, Zane. John Davidge jodav...@cisco.com -- Forwarded message -- From: Thomas Spatzier thomas.spatz...@de.ibm.com Date: Wed, Oct 9, 2013 at 12:40 AM Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows To: OpenStack Development Mailing List openstack-dev@lists.openstack.org Excerpts from Clint Byrum's message From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org, Date: 09.10.2013 03:54 Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Excerpts from Stan Lagun's message of 2013-10-08 13:53:45 -0700: Hello, That is why it is necessary to have some central coordination service which would handle deployment workflow and perform specific actions (create VMs and other OpenStack resources, do something on that VM) on each stage according to that workflow. We think that Heat is the best place for such service. I'm not so sure. Heat is part of the Orchestration program, not workflow. I agree. HOT so far was thought to be a format for describing templates in a structural, declaritive way. Adding workflows would stretch it quite a bit. Maybe we should see what aspects make sense to be added to HOT, and then how to do workflow like orchestration in a layer on top. Our idea is to extend HOT DSL by adding workflow definition capabilities as an explicit list of resources, components¹ states and actions. States may depend on each other so that you can reach state X only after you¹ve reached states Y and Z that the X depends on. The goal is from initial state to reach some final state ³Deployed². We also would like to add some mechanisms to HOT for declaratively doing software component orchestration in Heat, e.g. saying that one component depends on another one, or needs input from another one once it has been deployed etc. (I BTW started to write a wiki page, which is admittedly far from complete, but I would be happy to work on it with interested folks - https://wiki.openstack.org/wiki/Heat/Software-Configuration-Provider). However, we must be careful not to make such features too complicated so nobody will be able to use it any more. That said, I believe we could make HOT cover some levels of complexity, but not all. And then maybe workflow based orchestration on top is needed. Orchestration is not workflow, and HOT is an orchestration templating language, not a workflow language. Extending it would just complect two very different (though certainly related) tasks. I think the appropriate thing to do is actually to join up with the TaskFlow project and consider building it into a workflow service or tools (it is just a library right now). There is such state graph for each of our deployment entities (service, VMs, other things). There is also an action that must be performed on each state. Heat does its own translation of the orchestration template into a workflow right now, but we have already discussed using TaskFlow to break up the orchestration graph into distributable jobs. As we get more sophisticated on updates (rolling/canary for instance) we'll need to be able to reason about the process without having to glue all the pieces together. We propose to extend HOT DSL with workflow definition capabilities where you can describe step by step instruction to install service and properly handle errors on each step. We already have an experience in implementation of the DSL, workflow description and processing mechanism for complex deployments and believe we¹ll all benefit by re-using this experience and existing code, having properly discussed and agreed on abstraction layers and distribution of responsibilities between OS
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
It looks like this discussion involves many of the issues faced when developing the Curvature Donabe frameworks, which were presented at the Portland Summit - slides and video here: http://www.openstack.org/summit/portland-2013/session-videos/presentation/i nteractive-visual-orchestration-with-curvature-and-donabe Much of the work on the Donabe side revolved around defining a simple JSON-based API for describing the sorts of virtual application templates being discussed. All of the code for both Curvature and Donabe has recently been made open source and is available here: http://ciscosystems.github.io/curvature/ http://ciscosystems.github.io/donabe/ It looks like some of the ground covered by these projects can be helpful to this discussion. John Davidge jodav...@cisco.com -- Forwarded message -- From: Thomas Spatzier thomas.spatz...@de.ibm.com Date: Wed, Oct 9, 2013 at 12:40 AM Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows To: OpenStack Development Mailing List openstack-dev@lists.openstack.org Excerpts from Clint Byrum's message From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org, Date: 09.10.2013 03:54 Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Excerpts from Stan Lagun's message of 2013-10-08 13:53:45 -0700: Hello, That is why it is necessary to have some central coordination service which would handle deployment workflow and perform specific actions (create VMs and other OpenStack resources, do something on that VM) on each stage according to that workflow. We think that Heat is the best place for such service. I'm not so sure. Heat is part of the Orchestration program, not workflow. I agree. HOT so far was thought to be a format for describing templates in a structural, declaritive way. Adding workflows would stretch it quite a bit. Maybe we should see what aspects make sense to be added to HOT, and then how to do workflow like orchestration in a layer on top. Our idea is to extend HOT DSL by adding workflow definition capabilities as an explicit list of resources, components¹ states and actions. States may depend on each other so that you can reach state X only after you¹ve reached states Y and Z that the X depends on. The goal is from initial state to reach some final state ³Deployed². We also would like to add some mechanisms to HOT for declaratively doing software component orchestration in Heat, e.g. saying that one component depends on another one, or needs input from another one once it has been deployed etc. (I BTW started to write a wiki page, which is admittedly far from complete, but I would be happy to work on it with interested folks - https://wiki.openstack.org/wiki/Heat/Software-Configuration-Provider). However, we must be careful not to make such features too complicated so nobody will be able to use it any more. That said, I believe we could make HOT cover some levels of complexity, but not all. And then maybe workflow based orchestration on top is needed. Orchestration is not workflow, and HOT is an orchestration templating language, not a workflow language. Extending it would just complect two very different (though certainly related) tasks. I think the appropriate thing to do is actually to join up with the TaskFlow project and consider building it into a workflow service or tools (it is just a library right now). There is such state graph for each of our deployment entities (service, VMs, other things). There is also an action that must be performed on each state. Heat does its own translation of the orchestration template into a workflow right now, but we have already discussed using TaskFlow to break up the orchestration graph into distributable jobs. As we get more sophisticated on updates (rolling/canary for instance) we'll need to be able to reason about the process without having to glue all the pieces together. We propose to extend HOT DSL with workflow definition capabilities where you can describe step by step instruction to install service and properly handle errors on each step. We already have an experience in implementation of the DSL, workflow description and processing mechanism for complex deployments and believe we¹ll all benefit by re-using this experience and existing code, having properly discussed and agreed on abstraction layers and distribution of responsibilities between OS components. There is an idea of implementing part of workflow processing mechanism as a part of Convection proposal, which would allow other OS projects to benefit by using this. We would like to discuss if such design could become a part of future Heat version as well as other possible contributions from Murano team. Thanks really for thinking this through. Windows servers are not unique in this regard. Puppet and chef are pretty decent
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Steven Dake sd...@redhat.com wrote on 11.10.2013 21:02:38: From: Steven Dake sd...@redhat.com To: OpenStack Development Mailing List openstack-dev@lists.openstack.org, Date: 11.10.2013 21:04 Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows On 10/11/2013 11:55 AM, Lakshminaraya Renganarayana wrote: Clint Byrum cl...@fewbar.com wrote on 10/11/2013 12:40:19 PM: From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Date: 10/11/2013 12:43 PM Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows 3. Ability to return arbitrary (JSON-compatible) data structure from config application and use attributes of that structure as an input for other configs Note that I'd like to see more use cases specified for this ability. The random string generator that Steve Baker has put up should handle most cases where you just need passwords. Generated key sharing might best be deferred to something like Barbican which does a lot more than Heat to try and keep your secrets safe. I had seen a deployment scenario that needed more than random string generator. It was during the deployment of a system that has clustered application servers, i.e., a cluster of application server nodes + a cluster manager node. The deployment progresses by all the VMs (cluster-manager and cluster-nodes) starting concurrently. Then the cluster-nodes wait for the cluster-manager to send them data (xml) to configure themselves. The cluster-manager after reading its own config file, generates config-data for each cluster-node and sends it to them. Is the config data per cluster node unique to each node? If not: I think Lakshmi's example (IBM WebSphere, right?) talks about a case where the per cluster member info is unique per member, so the one fits all approach does not work. In addition, I think there is a constraint that members must join one by one and cannot join concurrently. Change deployment to following model: 1. deploy cluster-manager as a resource with a waitcondition - passing the data using the cfn-signal -d to send the xml blob 2. have cluster nodes wait on wait condition in #1, using data from the cfn-signal If so, join the config data sent in cfn-signal and break it apart by the various cluster nodes in #2 Thanks, LN ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Hi Angus, Thanks for detailed reply. I have a few comments that I have written below in the context. Angus Salkeld asalk...@redhat.com wrote on 10/13/2013 06:40:01 PM: - INPUTS: all the attributes that are consumed/used/read by that resource (currently, we have Ref, GetAttrs that can give this implicitly) - OUTPUTS: all the attributes that are produced/written by that resource (I do not know if this write-set is currently well-defined for a resource. I think some of them are implicitly defined by Heat on particular resource types.) - Global name-space and data-space : all the values produced and consumed (INPUTS/OUTPUTS) are described using a names that are fully qualified (XXX.stack_name.resource_name.property_name). The data values associated with these names are stored in a global data-space. Reads are blocking, i.e., reading a value will block the execution resource/thread until the value is available. Writes are non-blocking, i.e., any thread can write a value and the write will succeed immediately. I don't believe this would give us any new behaviour. I believe that in today's Heat, wait-conditions and signals are the only mechanism for synchronization during software configuration. The proposed mechanism would provide a higher level synchronization based on blocking-reads. For example, if one is using Chef for software configuration, then the recipes can use the proposed mechanism to wait for the all the node[][] attributes they require before starting the recipe execution. And, Heat can actually analyze and reason about deadlock properties of such a synchronization. On the other hand, if the recipe were using wait-conditions how would Heat reason about deadlock properties of it? The ability to define resources at arbitrary levels of granularity together with the explicit specification of INPUTS/OUTPUTS allows us to reap the benefits G1 and G2 outlined above. Note that the ability to reason about the inputs/outputs of each resource and the induced dependencies will also allow Heat to detect dead-locks via dependence cycles (benefit G3). This is already done today in Heat for Refs, GetAttr on base-resources, but the proposal is to extend the same to arbitrary attributes for any resource. How are TemplateResources and NestedStacks any different? To my knowledge this is aleady the case. The blocking-read and non-blocking writes further structures the specification to avoid deadlocks and race conditions (benefit G3). Have you experienced deadlocks with heat? I have never seen this... Heat as it is today does not tackle the problem of synchronization during software configuration and hence the problems I see cannot be attributed to Heat and can only be attributed to the scripts / recipes that do the software configuration. However, if we envision Heat to provide some support for software configuration I can easily imagine cases where it is impossible for Heat to analyze/reason with wait-conditions and hence leading to deadlocks. Wait-conditions and signals are equal to Timed-Semaphores in their power and expressivity and these are known for their problems with deadlocks. To me what is missing to better support complex software configuration is : - better integrating with existing configuration tools (puppet, chef, salt, ansible, etc). (resource types) One question is, whether in this integration the synchronization is completely left to the configuration tools or Heat would be involved in it. If it is left to configuration tools, say chef, then the question is how does the iterative convergence style execution of chef interfere with the schedule order that Heat determines for a template. On the other hand, if Heat provides the mechanism for synchronization, then the question is whether wait-conditions and signals are the right abstractions for them. What are your thoughts on this? Thanks, LN ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
write a Blueprint / proposal with more details and use-cases. We already have this functionality. Whenever you use a Ref or Attr from another resource we become dependant on it. This feels like replacing an existing system with a completely different one with very little benefit. To me what is missing to better support complex software configuration is : - better integrating with existing configuration tools (puppet, chef, salt, ansible, etc). (resource types) - integration with a ssh key store (Barbican) so the tools above can do non-userdata related configuration -Angus Thanks, LN Clint Byrum cl...@fewbar.com wrote on 10/11/2013 12:40:19 PM: From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Date: 10/11/2013 12:43 PM Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Excerpts from Stan Lagun's message of 2013-10-11 07:22:37 -0700: Hello, Thanks Angus, Clint, I've got your design. It seems that Murano can built on top of that. With service metadata knowledge Murano can generate HOT templates with set of interdependent configs. Here is what will be needed: 1. Ability to implement support for custom software configuration tool (type: OS::SoftwareConfig::MuranoAgent) These are really syntactic sugar, but you can implement them as resource plugins for users who want Murano resources. In the absence of this, just putting things in the free form metadata area of the resource that the MuranoAgent can interpret would suffice. 2. Ability to provide arbitrary input values for the config We already have that, there's a free-form json document called metadata attached to every resource. Or maybe I missed what you mean here. The new capability that is in the works that will make that better is to have multiple reusable metadata blocks referenced on one instance. 3. Ability to return arbitrary (JSON-compatible) data structure from config application and use attributes of that structure as an input for other configs Note that I'd like to see more use cases specified for this ability. The random string generator that Steve Baker has put up should handle most cases where you just need passwords. Generated key sharing might best be deferred to something like Barbican which does a lot more than Heat to try and keep your secrets safe. 4. Ability to provide config body that is an input to Murano Agent of arbitrary size Isn't this the same as 2? 5. Work well with large graph of configs with a lot of dependencies. Independent configs on different VMs should be applied in parallel. Yes this does look good. For dependent configs, the exsiting wait condition can be used. Does it confirm to your plans? I think it confirms that we're heading toward consensus on where to draw the software config vs. infrastructure orchestration line. That is very exciting. :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Hello,
Thanks Angus, Clint, I've got your design.
It seems that Murano can built on top of that. With service metadata
knowledge Murano can generate HOT templates with set of interdependent
configs.
Here is what will be needed:
1. Ability to implement support for custom software configuration tool
(type: OS::SoftwareConfig::MuranoAgent)
2. Ability to provide arbitrary input values for the config
3. Ability to return arbitrary (JSON-compatible) data structure from config
application and use attributes of that structure as an input for other
configs
4. Ability to provide config body that is an input to Murano Agent of
arbitrary size
5. Work well with large graph of configs with a lot of dependencies.
Independent configs on different VMs should be applied in parallel.
Does it confirm to your plans?
On Fri, Oct 11, 2013 at 3:47 AM, Clint Byrum cl...@fewbar.com wrote:
Excerpts from Angus Salkeld's message of 2013-10-10 15:27:48 -0700:
On 10/10/13 11:59 +0400, Stan Lagun wrote:
This rises number of questions:
1. What about conditional dependencies? Like config3 depends on config1
AND
config2 OR config3.
We have the AND, but not an OR. To depend on two resources you just
have 2 references to the 2 resources.
AND is concrete. OR is not. I don't actually think it is useful for what
Heat is intended to do. This is not not packaging, this is deploying.
For deploying, Heat needs to know _what to do_, not what is possible.
2. How do I pass values between configs? For example config1 requires
value
from user input and config2 needs an output value obtained from applying
config1
{Fn::GetAtt: [config2, the_name_of_the_attribute]}
This is a little bit misleading. Heat does not have any good ways to
get a value obtained from applying config1. The data attribute of
the WaitCondition is the only way I know, and it is really unwieldy,
as it can basically only dump a json string of all of the things each
signaler has fed back in.
That said, I wonder how many of the value obtained from applying config1
would be satisfied by the recently proposed random string generation
resource. Most of the time what people want to communicate back is just
auth details. If we push auth details in from Heat to both sides, that
alleviates all of my current use cases for this type of feature.
3. How would you do error handling? For example config3 on server3
requires
config1 to be applied on server1 and config2 on server2. Suppose that
there
was an error while applying config2 (and config1 succeeded). How do I
specify reaction for that? Maybe I need then to try to apply config4 to
server2 and continue or maybe just roll everything back
We currently have no on_error but it is not out of scope. The
current action is either to rollback the stack or leave it in the
failed state (depending on what you choose).
Right, I can definitely see more actions being added as we identify the
commonly desired options.
4. How these config dependencies play with nested stacks and resources
like
LoadBalancer that create such stacks? How do I specify that myConfig
depends on HA proxy being configured if that config was declared in
nested
stack that is generated by resource's Python code and is not declared
in my
HOT template?
It is normally based on the actual data/variable that you are
dependant on.
loadbalancer: depends on autoscaling instance_list
(actually in the loadbalancer config would be a GetAtt: [scalegroup,
InstanceList])
Then if you want to depend on that config you could depend on an
attribute of that resource that changes on reconfigure.
config1:
type: OS::SoftwareConfig::Ssh
properties:
script: {GetAtt: [scalegroup, InstanceList]}
hosted_on: loadbalancer
...
config2:
type: OS::SoftwareConfig::Ssh
properties:
script: {GetAtt: [config1, ConfigAppliedCount]}
hosted_on: somewhere_else
...
I am sure we could come up with some better syntax for this. But
the logic seems easily possible to me.
As far as nested stacks go: you just need an output to be useable
externally - basically design your API.
5. The solution is not generic. For example I want to write HOT template
for my custom load-balancer and a scalable web-servers group. Load
balancer
config depends on all configs of web-servers. But web-servers are
created
dynamically (autoscaling). That means dependency graph needs to be also
dynamically modified. But if you explicitly list config names instead of
something like depends on all configs of web-farm X you have no way to
describe such rule. In other words we need generic dependency, not just
dependency on particular config
Why won't just depending on the scaling group be enough? if it needs
to be updated it will update all within the group before progressing
to the dependants.
In the example, loadbalancer doesn't
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Excerpts from Stan Lagun's message of 2013-10-11 07:22:37 -0700: Hello, Thanks Angus, Clint, I've got your design. It seems that Murano can built on top of that. With service metadata knowledge Murano can generate HOT templates with set of interdependent configs. Here is what will be needed: 1. Ability to implement support for custom software configuration tool (type: OS::SoftwareConfig::MuranoAgent) These are really syntactic sugar, but you can implement them as resource plugins for users who want Murano resources. In the absence of this, just putting things in the free form metadata area of the resource that the MuranoAgent can interpret would suffice. 2. Ability to provide arbitrary input values for the config We already have that, there's a free-form json document called metadata attached to every resource. Or maybe I missed what you mean here. The new capability that is in the works that will make that better is to have multiple reusable metadata blocks referenced on one instance. 3. Ability to return arbitrary (JSON-compatible) data structure from config application and use attributes of that structure as an input for other configs Note that I'd like to see more use cases specified for this ability. The random string generator that Steve Baker has put up should handle most cases where you just need passwords. Generated key sharing might best be deferred to something like Barbican which does a lot more than Heat to try and keep your secrets safe. 4. Ability to provide config body that is an input to Murano Agent of arbitrary size Isn't this the same as 2? 5. Work well with large graph of configs with a lot of dependencies. Independent configs on different VMs should be applied in parallel. Yes this does look good. For dependent configs, the exsiting wait condition can be used. Does it confirm to your plans? I think it confirms that we're heading toward consensus on where to draw the software config vs. infrastructure orchestration line. That is very exciting. :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
On 10/9/2013 12:55 PM, Joshua Harlow wrote: Your example sounds a lot like what taskflow is build for doing. https://github.com/stackforge/taskflow/blob/master/taskflow/examples/calculate_in_parallel.py is a decent example. In that one, tasks are created and input/output dependencies are specified (provides, rebind, and the execute function arguments itself). This is combined into the taskflow concept of a flow, one of those flows types is a dependency graph. Using a parallel engine (similar in concept to a heat engine) we can run all non-dependent tasks in parallel. An example that I just created that shows this (and shows it running) that closer matches your example. Program (this will work against the current taskflow codebase): http://paste.openstack.org/show/48156/ I think that there is a major difference between building a set of virtual servers (what Heat does) and performing a specific task on a set of servers (what taskflow is designed for). Taskflow is currently planned to have a more complex and robust state machine that what Heat plans. This is natural given that simplicity has a relatively higher value for deployment, while efficiency of execution matters more for a task that will be performed repeatedly. However, if a simpler state model is needed, keep in mind that a simpler interface can always be translated into the more complete interface with a shim layer. You cannot build a more flexible solution on top of a simple solution. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Excellent discussion on various issues around orchestration and coordination -- thanks to you all, in particular to Clint, Angus, Stan, Thomas, Joshua, Zane, Steve ... After reading the discussions, I am finding the following themes emerging (please feel free to correct/add): 1. Most of the building blocks needed for effective coordination and orchestration are already in Heat/HOT. 2. Heat would like to view software configuration as a resource (type) with related providers + plugins 3. There is scope for communication/synchronization mechanisms that would complement the wait-conditions and signals I would like to propose a simple abstraction that would complement the current wait-conditions and signals. My proposal is based our experience with supporting such an abstraction on our DSL and also on an extension of Heat. In a nut-shell, this abstraction is a global data space (visible across resources, stacks) from which resources can read and write their inputs / outputs PLUS the semantics that reads will block until the read values are available and writes are non-blocking. We used ZooKeeper to implement this global data space and the blocking-read/non-blocking-writes semantics. But, these could be implemented using several other mechanisms and I believe the techniques currently used by Heat for meta-data service can be used here. I would like to make clear that I am not proposing a replacement for wait-conditions and signals. I am hoping that wait-conditions and signals would be used by power-users (concurrent/distributed programming experts) and the proposed abstraction would be used by folks (like me) who do not want to reason about concurrency and related problems. Also, the proposed global data-space with blocking reads and non-blocking writes is not a new idea (google tuple-spaces, linda) and it has been proven in other domains such as coordination languages to improve the level of abstraction and productivity. The benefits of the proposed abstraction are: G1. Support finer granularity of dependences G2. Allow Heat to reason/analyze about these dependences so that it can order resource creations/management G3. Avoid classic synchronization problems such as dead-locks and race conditions G4 *Conjecture* : Capture most of the coordination use cases (including those required for software configuration / orchestration). Here is more detailed description: Let us say that we can use either pre-defined or custom resource types to define resources at arbitrary levels of granularity. This can be easily supported and I guess is already possible in current version of Heat/HOT. Given this, the proposed abstraction has two parts: (1) an interface style specification a resource's inputs and outputs and (2) a global name/data space. The interface specification which would capture - INPUTS: all the attributes that are consumed/used/read by that resource (currently, we have Ref, GetAttrs that can give this implicitly) - OUTPUTS: all the attributes that are produced/written by that resource (I do not know if this write-set is currently well-defined for a resource. I think some of them are implicitly defined by Heat on particular resource types.) - Global name-space and data-space : all the values produced and consumed (INPUTS/OUTPUTS) are described using a names that are fully qualified (XXX.stack_name.resource_name.property_name). The data values associated with these names are stored in a global data-space. Reads are blocking, i.e., reading a value will block the execution resource/thread until the value is available. Writes are non-blocking, i.e., any thread can write a value and the write will succeed immediately. The ability to define resources at arbitrary levels of granularity together with the explicit specification of INPUTS/OUTPUTS allows us to reap the benefits G1 and G2 outlined above. Note that the ability to reason about the inputs/outputs of each resource and the induced dependencies will also allow Heat to detect dead-locks via dependence cycles (benefit G3). This is already done today in Heat for Refs, GetAttr on base-resources, but the proposal is to extend the same to arbitrary attributes for any resource. The blocking-read and non-blocking writes further structures the specification to avoid deadlocks and race conditions (benefit G3). As for G4, the conjecture, I can only give as evidence our experience with using our DSL with the proposed abstraction to deploy a few reasonably large applications :-) I would like to know your comments and suggestions. Also, if there is interest I can write a Blueprint / proposal with more details and use-cases. Thanks, LN Clint Byrum cl...@fewbar.com wrote on 10/11/2013 12:40:19 PM: From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Date: 10/11/2013 12:43 PM Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Excerpts from Stan Lagun's message of 2013-10-11 07:22:37
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Clint Byrum cl...@fewbar.com wrote on 10/11/2013 12:40:19 PM: From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Date: 10/11/2013 12:43 PM Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows 3. Ability to return arbitrary (JSON-compatible) data structure from config application and use attributes of that structure as an input for other configs Note that I'd like to see more use cases specified for this ability. The random string generator that Steve Baker has put up should handle most cases where you just need passwords. Generated key sharing might best be deferred to something like Barbican which does a lot more than Heat to try and keep your secrets safe. I had seen a deployment scenario that needed more than random string generator. It was during the deployment of a system that has clustered application servers, i.e., a cluster of application server nodes + a cluster manager node. The deployment progresses by all the VMs (cluster-manager and cluster-nodes) starting concurrently. Then the cluster-nodes wait for the cluster-manager to send them data (xml) to configure themselves. The cluster-manager after reading its own config file, generates config-data for each cluster-node and sends it to them. Thanks, LN___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
On 10/11/2013 11:55 AM, Lakshminaraya Renganarayana wrote: Clint Byrum cl...@fewbar.com wrote on 10/11/2013 12:40:19 PM: From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Date: 10/11/2013 12:43 PM Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows 3. Ability to return arbitrary (JSON-compatible) data structure from config application and use attributes of that structure as an input for other configs Note that I'd like to see more use cases specified for this ability. The random string generator that Steve Baker has put up should handle most cases where you just need passwords. Generated key sharing might best be deferred to something like Barbican which does a lot more than Heat to try and keep your secrets safe. I had seen a deployment scenario that needed more than random string generator. It was during the deployment of a system that has clustered application servers, i.e., a cluster of application server nodes + a cluster manager node. The deployment progresses by all the VMs (cluster-manager and cluster-nodes) starting concurrently. Then the cluster-nodes wait for the cluster-manager to send them data (xml) to configure themselves. The cluster-manager after reading its own config file, generates config-data for each cluster-node and sends it to them. Is the config data per cluster node unique to each node? If not: Change deployment to following model: 1. deploy cluster-manager as a resource with a waitcondition - passing the data using the cfn-signal -d to send the xml blob 2. have cluster nodes wait on wait condition in #1, using data from the cfn-signal If so, join the config data sent in cfn-signal and break it apart by the various cluster nodes in #2 Thanks, LN ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
On Fri, Oct 11, 2013 at 8:40 PM, Clint Byrum cl...@fewbar.com wrote: 2. Ability to provide arbitrary input values for the config We already have that, there's a free-form json document called metadata attached to every resource. Or maybe I missed what you mean here. The new capability that is in the works that will make that better is to have multiple reusable metadata blocks referenced on one instance. Yes, that's what I meant - metadata attached to configs rather than instances. 3. Ability to return arbitrary (JSON-compatible) data structure from config application and use attributes of that structure as an input for other configs Note that I'd like to see more use cases specified for this ability. The random string generator that Steve Baker has put up should handle most cases where you just need passwords. Generated key sharing might best be deferred to something like Barbican which does a lot more than Heat to try and keep your secrets safe. Murano's execution plans that are sent to Murano Agent are similar to Python functions in that they have input and output. The output may be a script exit code, captured stdout/stderr, value returned from PowerShell/Python function etc. Although it is rare case where output from one execution plan is required as an input for another plan but it happens. For example execution plan 1 did created network interface on VM with DHCP enabled and execution plan (that may be executed on another machine) requires IP address obtained on that interface. In this case IP address would be the returned value 4. Ability to provide config body that is an input to Murano Agent of arbitrary size Isn't this the same as 2? Not exactly the same, but config body can be an attribute of config's metadata with a special reserved name I think it confirms that we're heading toward consensus on where to draw the software config vs. infrastructure orchestration line. That is very exciting. :) This is indeed very promising. If Murano can do all the orchestration via HOT templates and Heat doesn't deal with service metadata that is out of scope of HOT we can eliminate most of the overlapping between projects and actually complement each other. The only thing I concerned about in this context is https://wiki.openstack.org/wiki/Heat/DSL2 and https://wiki.openstack.org/wiki/Heat/Open_API as this is very similar to what Murano does -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
This rises number of questions: 1. What about conditional dependencies? Like config3 depends on config1 AND config2 OR config3. 2. How do I pass values between configs? For example config1 requires value from user input and config2 needs an output value obtained from applying config1 3. How would you do error handling? For example config3 on server3 requires config1 to be applied on server1 and config2 on server2. Suppose that there was an error while applying config2 (and config1 succeeded). How do I specify reaction for that? Maybe I need then to try to apply config4 to server2 and continue or maybe just roll everything back 4. How these config dependencies play with nested stacks and resources like LoadBalancer that create such stacks? How do I specify that myConfig depends on HA proxy being configured if that config was declared in nested stack that is generated by resource's Python code and is not declared in my HOT template? 5. The solution is not generic. For example I want to write HOT template for my custom load-balancer and a scalable web-servers group. Load balancer config depends on all configs of web-servers. But web-servers are created dynamically (autoscaling). That means dependency graph needs to be also dynamically modified. But if you explicitly list config names instead of something like depends on all configs of web-farm X you have no way to describe such rule. In other words we need generic dependency, not just dependency on particular config 6. What would you do on STACK UPDATE that modifies the dependency graph? The notation of configs and there On Thu, Oct 10, 2013 at 4:25 AM, Angus Salkeld asalk...@redhat.com wrote: On 09/10/13 19:31 +0100, Steven Hardy wrote: On Wed, Oct 09, 2013 at 06:59:22PM +0200, Alex Rudenko wrote: Hi everyone, I've read this thread and I'd like to share some thoughts. In my opinion, workflows (which run on VMs) can be integrated with heat templates as follows: 1. workflow definitions should be defined separately and processed by stand-alone workflow engines (chef, puppet etc). I agree, and I think this is the direction we're headed with the software-config blueprints - essentially we should end up with some new Heat *resources* which encapsulate software configuration. Exactly. I think we need a software-configuration-aas sub-project that knows how to take puppet/chef/salt/... config and deploy it. Then Heat just has Resources for these (OS::SoftwareConfig::Puppet). We should even move our WaitConditions and Metadata over to that yet-to-be-made service so that Heat is totally clean of software config. How would this solve ordering issues: resources: config1: type: OS::SoftwareConfig::Puppet hosted_on: server1 ... config2: type: OS::SoftwareConfig::Puppet hosted_on: server1 depends_on: config3 ... config3: type: OS::SoftwareConfig::Puppet hosted_on: server2 depends_on: config1 ... server1: type: OS::Nova::Server ... server2: type: OS::Nova::Server ... Heat knows all about ordering: It starts the resources: server1, server2 config1 config3 config2 There is the normal contract in the client: we post the config to software-config-service and we wait for the state == ACTIVE (when the config is applied) before progressing to a resource that is dependant on it. -Angus IMO there is some confusion around the scope of HOT, we should not be adding functionality to it which already exists in established config management tools IMO, instead we should focus on better integration with exisitng tools at the resource level, and identifying template interfaces which require more flexibility (for example serialization primitives) 2. the HOT resources should reference workflows which they require, specifying a type of workflow and the way to access a workflow definition. The workflow definition might be provided along with HOT. So again, I think this acatually has very little to do with HOT. The *Heat* resources may define software configuration, or possibly some sort of workflow, which is acted upon by $thing which is not Heat. So in the example provided by the OP, maybe you'd have a Murano resource, which knows how to define the input to the Murano API, which might trigger workflow type actions to happen in the Murano service. 3. Heat should treat the orchestration templates as transactions (i.e. Heat should be able to rollback in two cases: 1) if something goes wrong during processing of an orchestration workflow 2) when a stand-alone workflow engine reports an error during processing of a workflow associated with a resource) So we already have the capability for resources to recieve signals, which would allow (2) in the asynchronous case. But it seems to me that this is still a serialization problem, ie a synchronous case, therefore (2) is just part of (1). E.g - Heat stack
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Hi all, Lakshminaraya Renganarayana lren...@us.ibm.com wrote on 10.10.2013 01:34:41: From: Lakshminaraya Renganarayana lren...@us.ibm.com To: Joshua Harlow harlo...@yahoo-inc.com, Cc: OpenStack Development Mailing List openstack-dev@lists.openstack.org Date: 10.10.2013 01:37 Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Hi Joshua, I agree that there is an element of taskflow in what I described. But, I am aiming for something much more lightweight which can be naturally blended with HOT constructs and Heat engine. To be a bit more specific, Heat already has dependencies and coordination mechanisms. So, I am aiming for may be just one additional construct in Heat/HOT and some logic in Heat that would support coordination. First of all, the use case you presented in your earlier mail is really good and illustrative. And I agree that there should be constructs in HOT to declare those kinds of dependencies. So how to define this in HOT is one work item. How this gets implemented is another item, and yes, maybe this is something that Heat can delegate to taskflow. Because if taskflow has those capabilities, why re-implement it. Thanks, LN _ Lakshminarayanan Renganarayana Research Staff Member IBM T.J. Watson Research Center http://researcher.ibm.com/person/us-lrengan Joshua Harlow harlo...@yahoo-inc.com wrote on 10/09/2013 03:55:00 PM: From: Joshua Harlow harlo...@yahoo-inc.com To: OpenStack Development Mailing List openstack- d...@lists.openstack.org, Lakshminaraya Renganarayana/Watson/IBM@IBMUS Date: 10/09/2013 03:55 PM Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Your example sounds a lot like what taskflow is build for doing. https://github.com/stackforge/taskflow/blob/master/taskflow/ examples/calculate_in_parallel.py is a decent example. In that one, tasks are created and input/output dependencies are specified (provides, rebind, and the execute function arguments itself). This is combined into the taskflow concept of a flow, one of those flows types is a dependency graph. Using a parallel engine (similar in concept to a heat engine) we can run all non-dependent tasks in parallel. An example that I just created that shows this (and shows it running) that closer matches your example. Program (this will work against the current taskflow codebase): http://paste.openstack.org/show/48156/ Output @ http://paste.openstack.org/show/48157/ -Josh From: Lakshminaraya Renganarayana lren...@us.ibm.com Reply-To: OpenStack Development Mailing List openstack- d...@lists.openstack.org Date: Wednesday, October 9, 2013 11:31 AM To: OpenStack Development Mailing List openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Steven Hardy sha...@redhat.com wrote on 10/09/2013 05:24:38 AM: So as has already been mentioned, Heat defines an internal workflow, based on the declarative model defined in the template. The model should define dependencies, and Heat should convert those dependencies into a workflow internally. IMO if the user also needs to describe a workflow explicitly in the template, then we've probably failed to provide the right template interfaces for describing depenendencies. I agree with Steven here, models should define the dependencies and Heat should realize/enforce them. An important design issue is granularity at which dependencies are defined and enforced. I am aware of the wait-condition and signal constructs in Heat, but I find them a bit low-level as they are prone to the classic dead-lock and race condition problems. I would like to have higher level constructs that support finer-granularity dependences which are needed for software orchestration. Reading through the variousdisucssion on this topic in this mailing list, I see that many would like to have such higher level constructs for coordination. In our experience with software orchestration using our own DSL and also with some extensions to Heat, we found that the granularity of VMs or Resources to be too coarse for defining dependencies for software orchestration. For example, consider a two VM app, with VMs vmA, vmB, and a set of software components (ai's and bi's) to be installed on them: vmA = base-vmA + a1 + a2 + a3 vmB = base-vmB + b1 + b2 + b3 let us say that software component b1 of vmB, requires a config value produced by software component a1 of vmA. How to declaratively model this dependence? Clearly, modeling a dependence between just base-vmA and base-vmB is not enough. However, defining a dependence between the whole of vmA and vmB is too coarse. It would be ideal to be able to define a dependence at the granularity of software
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
On 10/10/13 11:59 +0400, Stan Lagun wrote:
This rises number of questions:
1. What about conditional dependencies? Like config3 depends on config1 AND
config2 OR config3.
We have the AND, but not an OR. To depend on two resources you just
have 2 references to the 2 resources.
2. How do I pass values between configs? For example config1 requires value
from user input and config2 needs an output value obtained from applying
config1
{Fn::GetAtt: [config2, the_name_of_the_attribute]}
3. How would you do error handling? For example config3 on server3 requires
config1 to be applied on server1 and config2 on server2. Suppose that there
was an error while applying config2 (and config1 succeeded). How do I
specify reaction for that? Maybe I need then to try to apply config4 to
server2 and continue or maybe just roll everything back
We currently have no on_error but it is not out of scope. The
current action is either to rollback the stack or leave it in the
failed state (depending on what you choose).
4. How these config dependencies play with nested stacks and resources like
LoadBalancer that create such stacks? How do I specify that myConfig
depends on HA proxy being configured if that config was declared in nested
stack that is generated by resource's Python code and is not declared in my
HOT template?
It is normally based on the actual data/variable that you are
dependant on.
loadbalancer: depends on autoscaling instance_list
(actually in the loadbalancer config would be a GetAtt: [scalegroup,
InstanceList])
Then if you want to depend on that config you could depend on an
attribute of that resource that changes on reconfigure.
config1:
type: OS::SoftwareConfig::Ssh
properties:
script: {GetAtt: [scalegroup, InstanceList]}
hosted_on: loadbalancer
...
config2:
type: OS::SoftwareConfig::Ssh
properties:
script: {GetAtt: [config1, ConfigAppliedCount]}
hosted_on: somewhere_else
...
I am sure we could come up with some better syntax for this. But
the logic seems easily possible to me.
As far as nested stacks go: you just need an output to be useable
externally - basically design your API.
5. The solution is not generic. For example I want to write HOT template
for my custom load-balancer and a scalable web-servers group. Load balancer
config depends on all configs of web-servers. But web-servers are created
dynamically (autoscaling). That means dependency graph needs to be also
dynamically modified. But if you explicitly list config names instead of
something like depends on all configs of web-farm X you have no way to
describe such rule. In other words we need generic dependency, not just
dependency on particular config
Why won't just depending on the scaling group be enough? if it needs
to be updated it will update all within the group before progressing
to the dependants.
6. What would you do on STACK UPDATE that modifies the dependency graph?
The notation of configs and there
What we normally do go through the resources, see what can be updated:
- without replacement
- needs deleting
- is new
- requires updating
Each resource type can define what will require replacing or not.
I think we can achieve what you want with some small improvements to
the HOT format and with some new resource types - IMHO.
-Angus
On Thu, Oct 10, 2013 at 4:25 AM, Angus Salkeld asalk...@redhat.com wrote:
On 09/10/13 19:31 +0100, Steven Hardy wrote:
On Wed, Oct 09, 2013 at 06:59:22PM +0200, Alex Rudenko wrote:
Hi everyone,
I've read this thread and I'd like to share some thoughts. In my opinion,
workflows (which run on VMs) can be integrated with heat templates as
follows:
1. workflow definitions should be defined separately and processed by
stand-alone workflow engines (chef, puppet etc).
I agree, and I think this is the direction we're headed with the
software-config blueprints - essentially we should end up with some new
Heat *resources* which encapsulate software configuration.
Exactly.
I think we need a software-configuration-aas sub-project that knows
how to take puppet/chef/salt/... config and deploy it. Then Heat just
has Resources for these (OS::SoftwareConfig::Puppet).
We should even move our WaitConditions and Metadata over to that
yet-to-be-made service so that Heat is totally clean of software config.
How would this solve ordering issues:
resources:
config1:
type: OS::SoftwareConfig::Puppet
hosted_on: server1
...
config2:
type: OS::SoftwareConfig::Puppet
hosted_on: server1
depends_on: config3
...
config3:
type: OS::SoftwareConfig::Puppet
hosted_on: server2
depends_on: config1
...
server1:
type: OS::Nova::Server
...
server2:
type: OS::Nova::Server
...
Heat knows all about ordering:
It starts the resources:
server1, server2
config1
config3
config2
There is the normal contract in the client:
we post the config to software-config-service
and we wait for the state == ACTIVE (when the
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Excerpts from Angus Salkeld's message of 2013-10-10 15:27:48 -0700:
On 10/10/13 11:59 +0400, Stan Lagun wrote:
This rises number of questions:
1. What about conditional dependencies? Like config3 depends on config1 AND
config2 OR config3.
We have the AND, but not an OR. To depend on two resources you just
have 2 references to the 2 resources.
AND is concrete. OR is not. I don't actually think it is useful for what
Heat is intended to do. This is not not packaging, this is deploying.
For deploying, Heat needs to know _what to do_, not what is possible.
2. How do I pass values between configs? For example config1 requires value
from user input and config2 needs an output value obtained from applying
config1
{Fn::GetAtt: [config2, the_name_of_the_attribute]}
This is a little bit misleading. Heat does not have any good ways to
get a value obtained from applying config1. The data attribute of
the WaitCondition is the only way I know, and it is really unwieldy,
as it can basically only dump a json string of all of the things each
signaler has fed back in.
That said, I wonder how many of the value obtained from applying config1
would be satisfied by the recently proposed random string generation
resource. Most of the time what people want to communicate back is just
auth details. If we push auth details in from Heat to both sides, that
alleviates all of my current use cases for this type of feature.
3. How would you do error handling? For example config3 on server3 requires
config1 to be applied on server1 and config2 on server2. Suppose that there
was an error while applying config2 (and config1 succeeded). How do I
specify reaction for that? Maybe I need then to try to apply config4 to
server2 and continue or maybe just roll everything back
We currently have no on_error but it is not out of scope. The
current action is either to rollback the stack or leave it in the
failed state (depending on what you choose).
Right, I can definitely see more actions being added as we identify the
commonly desired options.
4. How these config dependencies play with nested stacks and resources like
LoadBalancer that create such stacks? How do I specify that myConfig
depends on HA proxy being configured if that config was declared in nested
stack that is generated by resource's Python code and is not declared in my
HOT template?
It is normally based on the actual data/variable that you are
dependant on.
loadbalancer: depends on autoscaling instance_list
(actually in the loadbalancer config would be a GetAtt: [scalegroup,
InstanceList])
Then if you want to depend on that config you could depend on an
attribute of that resource that changes on reconfigure.
config1:
type: OS::SoftwareConfig::Ssh
properties:
script: {GetAtt: [scalegroup, InstanceList]}
hosted_on: loadbalancer
...
config2:
type: OS::SoftwareConfig::Ssh
properties:
script: {GetAtt: [config1, ConfigAppliedCount]}
hosted_on: somewhere_else
...
I am sure we could come up with some better syntax for this. But
the logic seems easily possible to me.
As far as nested stacks go: you just need an output to be useable
externally - basically design your API.
5. The solution is not generic. For example I want to write HOT template
for my custom load-balancer and a scalable web-servers group. Load balancer
config depends on all configs of web-servers. But web-servers are created
dynamically (autoscaling). That means dependency graph needs to be also
dynamically modified. But if you explicitly list config names instead of
something like depends on all configs of web-farm X you have no way to
describe such rule. In other words we need generic dependency, not just
dependency on particular config
Why won't just depending on the scaling group be enough? if it needs
to be updated it will update all within the group before progressing
to the dependants.
In the example, loadbalancer doesn't have to depend on all of the nodes
being configured. Why would it? It gets a signal when the list changes,
but it can be created as soon as the _group_ is created.
Anyway, no dependency is needed. Your LB has health checks, you feed
them in, and when the webservers are configured, they pass, and it sends
traffic there.
6. What would you do on STACK UPDATE that modifies the dependency graph?
The notation of configs and there
What we normally do go through the resources, see what can be updated:
- without replacement
- needs deleting
- is new
- requires updating
Each resource type can define what will require replacing or not.
I think we can achieve what you want with some small improvements to
the HOT format and with some new resource types - IMHO.
Agree with Angus here. I think we're closer to your use case than you
think. :)
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Excerpts from Clint Byrum's message From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org, Date: 09.10.2013 03:54 Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Excerpts from Stan Lagun's message of 2013-10-08 13:53:45 -0700: Hello, That is why it is necessary to have some central coordination service which would handle deployment workflow and perform specific actions (create VMs and other OpenStack resources, do something on that VM) on each stage according to that workflow. We think that Heat is the best place for such service. I'm not so sure. Heat is part of the Orchestration program, not workflow. I agree. HOT so far was thought to be a format for describing templates in a structural, declaritive way. Adding workflows would stretch it quite a bit. Maybe we should see what aspects make sense to be added to HOT, and then how to do workflow like orchestration in a layer on top. Our idea is to extend HOT DSL by adding workflow definition capabilities as an explicit list of resources, components’ states and actions. States may depend on each other so that you can reach state X only after you’ve reached states Y and Z that the X depends on. The goal is from initial state to reach some final state “Deployed”. We also would like to add some mechanisms to HOT for declaratively doing software component orchestration in Heat, e.g. saying that one component depends on another one, or needs input from another one once it has been deployed etc. (I BTW started to write a wiki page, which is admittedly far from complete, but I would be happy to work on it with interested folks - https://wiki.openstack.org/wiki/Heat/Software-Configuration-Provider). However, we must be careful not to make such features too complicated so nobody will be able to use it any more. That said, I believe we could make HOT cover some levels of complexity, but not all. And then maybe workflow based orchestration on top is needed. Orchestration is not workflow, and HOT is an orchestration templating language, not a workflow language. Extending it would just complect two very different (though certainly related) tasks. I think the appropriate thing to do is actually to join up with the TaskFlow project and consider building it into a workflow service or tools (it is just a library right now). There is such state graph for each of our deployment entities (service, VMs, other things). There is also an action that must be performed on each state. Heat does its own translation of the orchestration template into a workflow right now, but we have already discussed using TaskFlow to break up the orchestration graph into distributable jobs. As we get more sophisticated on updates (rolling/canary for instance) we'll need to be able to reason about the process without having to glue all the pieces together. We propose to extend HOT DSL with workflow definition capabilities where you can describe step by step instruction to install service and properly handle errors on each step. We already have an experience in implementation of the DSL, workflow description and processing mechanism for complex deployments and believe we’ll all benefit by re-using this experience and existing code, having properly discussed and agreed on abstraction layers and distribution of responsibilities between OS components. There is an idea of implementing part of workflow processing mechanism as a part of Convection proposal, which would allow other OS projects to benefit by using this. We would like to discuss if such design could become a part of future Heat version as well as other possible contributions from Murano team. Thanks really for thinking this through. Windows servers are not unique in this regard. Puppet and chef are pretty decent at expressing single-node workflows but they are awkward for deferring control and resuming on other nodes, so I do think there is a need for a general purpose distributed workflow definition tool. I'm not, however, convinced that extending HOT would yield a better experience for users. I'd prefer to see HOT have a well defined interface for where to defer to external workflows. Wait conditions are actually decent at that, and I'm sure with a little more thought we can make them more comfortable to use. Good discussion to have, i.e. what extensions we would need in HOT for making HOT alone more capable, and what we would need to hook it up with other orchestration like workflows. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
On Wed, Oct 09, 2013 at 12:53:45AM +0400, Stan Lagun wrote: Hello, I’m one of the engineer working on Murano project. Recently we started a discussion about Murano and Heat Software orchestration and I want to continue this discussion with more technical details. Thanks, we're certainly interested in Murano, and are keen to discuss your roadmap, and where requirements and integration opportunities exist. In our project we do deployment of complex multi-instance Windows services. Those services usually require specific multi-VM orchestration that is currently impossible or at least not that easy to achieve with Heat. As you are currently doing HOT software orchestration design we would like to participate in HOT Software orchestration design and contribute into it, so that the Heat could address use-cases which we believe are very common. For example here is how deployment of a SQL Server cluster goes: 1. Allocate Windows VMs for SQL Server cluster Heat can already do this, you just define either OS::Nova::Server or AWS::EC2::Instance resource in your template, or possibly a group of instances via OS::Heat::InstanceGroup if the configuration is the same for all VMs 2. Enable secondary IP address from user input on all SQL Windows instances So this again is already possible, via several resource types, AWS::EC2::NetworkInterface, AWS::EC2::EIP, OS::Neutron::Port etc.. I suggest using the Neutron resources where possible, if you don't care about Clouformation portability. 3. Install SQL Server prerequisites on each node So Heat is already able to do this, via a couple of methods, for Linux VMs, so we just need the in-instance agent support for windows (cloud-init, optionally combined with agents like cfn-init from heat-cfntools) Can you clarify what you're using for in-instance agents currently, cloudbase-init, and/or some bespoke tools? 4. Choose a master node and install Failover Cluster on it 5. Configure all nodes so that they know which one of them is the master I'm not sure what's involved in these steps, but it seems like there are serialization requirements, which can be handled via WaitConditions. One thing I think we do need to look at is ways to enable expression of serialization requirements via HOT, which don't require use of the AWS-compatible WaitCondition resources. So I think we already have the required functionality, we just need to build out better native interfaces to it. Configure all nodes so that they know which one of them is the master 6. Install SQL Server on all nodes 7. Initialize AlwaysOn on all nodes except for the master 8. Initialize Primary replica 9. Initialize secondary replicas All of the steps must take place in appropriate order depending on the state of other nodes. Some steps require an output from previous steps and all of them require some input parameters. SQL Server requires an Active Directory service in order to use Failover mechanism and installation of Active Directory with primary and secondary controllers is a complex workflow of its own. So all of this seems possible right now using WaitConditions, but as mentioned above we should look at ways to provide a better and more flexible native interface to similar functionality. That is why it is necessary to have some central coordination service which would handle deployment workflow and perform specific actions (create VMs and other OpenStack resources, do something on that VM) on each stage according to that workflow. We think that Heat is the best place for such service. Yep, we already do coordination of VM deployment and other openstack resources, by managing implicit and explicit dependencies between those resources. Our idea is to extend HOT DSL by adding workflow definition capabilities as an explicit list of resources, components’ states and actions. States may depend on each other so that you can reach state X only after you’ve reached states Y and Z that the X depends on. The goal is from initial state to reach some final state “Deployed”. IMHO there isn't a real need to provide explicit control of the workflow implied by the resource dependencies for the sort of use-case you describe. What I think is needed is simply a better native interface to serialization primitives/resources. There is such state graph for each of our deployment entities (service, VMs, other things). There is also an action that must be performed on each state. For example states graph from example above would look like this: The goal is to reach Service_Done state which depends on VM1_Done and VM2_Done states and so on from initial Service_Start state. We propose to extend HOT DSL with workflow definition capabilities where you can describe step by step instruction to install service and properly handle errors on each step. So as has already been mentioned, Heat defines
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Thanks, we're certainly interested in Murano, and are keen to discuss your roadmap, and where requirements and integration opportunities exist Glad to here it. The same is true from Murano side. On sample SQL workflow: that was just an example. I didn't want to bother you with a SQL Server deployment details as it's really not that important. What I've tried to say is that deployments consists of many steps, the steps vary depending on instance's role, user input and so on. The step on one machine often requires some other machine already be in some state or some output from a deployment step happened elsewhere. I do understand that it is doable using Heat alone. Actually we do use Heat for some parts of workflow. We do not talk to Nova or Neutron directly. The special use case of Murano is that there is no HOT template author. Heat is more an administrator's tool who knows how to write HOT templates and wants to deal with low-level configuration aspects. But Murano is quite different. In Murano the developers of workflows/scripts/metadata/etc are not end-users. The user is not doing any sort of programming. He is given a UI dashboard where he can compose desired environment from available building blocks (services). Services may depend on each other and UI guides him how to fulfill all requirements. User also conigures service's (VM's etc.) settings. The number of instances in SQL Server cluster and which one of them is going to be the master are such settings. Because we do not know in advance all the instances and resources that would be required for the services that user has chosen and the deployment process is strongly depends on user input we cannot just have some hardcoded HOT template. So what we do is we dynamically generate HOT templates my parameterizing and merging several simpler templates together. Then we use our orchestration engine to send commands to Murano Agents on VMs to perform deployment steps in correct order with all needed input. Probably we could do it without orchestration but then we would need to dynamically generate all that WaitConditions and waiting/signaling scripts etc. - something that would be error-prone and hard to manage at large scales. So we do believe that some external state orchestration would be very very helpful for complex software services we deal with. Although Murano currently has such engine it is far from perfect. As we thought on cleaner and more explicit approach for state orchestration we came to a vision how to implement it on top of task-orchestration engine like TaskFlow. And then we came up with idea that we can go even farther and implement TaskFlow-as-a-Service service with its own REST API etc that could handle an abstract task orchestration while leaving all deployment-related out of the scope of such service. That opens many additional opportunities for integration that would not be available if we just use TaskFlow library in-process. But we do believe that it would be better for all of us if it would be Heat and not Murano who provides such orchestration capabilities. And yes, I'm completely agree with you that it should not be a part of HOT templates but something external to it. To my understanding external orchestration needs to be capable of 1. Process some object (JSON) model describing what services/resources/etc. are need to be deployed 2. During orchestration invoke some HOT templates (create nested stacks?) with passing required attributes from that object model as an inputs for the HOT template 3. Be able to send commands (bash scripts, puppet manifests, chef recipes etc) to VM in correct order with a parameters taken from object model and HOT outputs. See https://wiki.openstack.org/wiki/Murano/UnifiedAgent on how it is in Murano We are currently communicating with TaskFlow team on possible contributions to Convection implementation and would be glad to participate in software orchestration part on the Heat side. I'm not pretend that I know all the answers or at least our design is good but in Murano we gained much experience in software orchestration that might be useful for the Heat team and we would definitely like to share our ideas. I also believe that now is the time for that as on summit it may be too late because all the principal decisions are already made. On Wed, Oct 9, 2013 at 1:24 PM, Steven Hardy sha...@redhat.com wrote: On Wed, Oct 09, 2013 at 12:53:45AM +0400, Stan Lagun wrote: Hello, I’m one of the engineer working on Murano project. Recently we started a discussion about Murano and Heat Software orchestration and I want to continue this discussion with more technical details. Thanks, we're certainly interested in Murano, and are keen to discuss your roadmap, and where requirements and integration opportunities exist. In our project we do deployment of complex multi-instance Windows services. Those services usually require specific multi-VM orchestration that is currently
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Hi everyone, I've read this thread and I'd like to share some thoughts. In my opinion, workflows (which run on VMs) can be integrated with heat templates as follows: 1. workflow definitions should be defined separately and processed by stand-alone workflow engines (chef, puppet etc). 2. the HOT resources should reference workflows which they require, specifying a type of workflow and the way to access a workflow definition. The workflow definition might be provided along with HOT. 3. Heat should treat the orchestration templates as transactions (i.e. Heat should be able to rollback in two cases: 1) if something goes wrong during processing of an orchestration workflow 2) when a stand-alone workflow engine reports an error during processing of a workflow associated with a resource) 4. Heat should expose an API which enables basic communication between running workflows. Additionally, Heat should provide an API to workflows that allows workflows to specify whether they completed successfully or not. The reference to these APIs should be passed to the workflow engine that is responsible for executing workflows on VMs. Pros of each point: 1 2 - keeps Heat simple and gives a possibility to choose the best workflows and engines among available ones. 3 - adds some kind of all-or-nothing semantics improving the control and awareness of what's going on inside VMs. 4 - allows workflow synchronization and communication through Heat API. Provides the error reporting mechanism for workflows. If a workflow does not need this functionality, it can ignore it. Cons: - Changes to existing workflows making them aware of Heat existence are required. These thoughts might show some gaps in my understanding of how Heat works, but I would like to share them anyway. Best regards, Oleksii Rudenko On Wed, Oct 9, 2013 at 5:37 PM, Georgy Okrokvertskhov gokrokvertsk...@mirantis.com wrote: Hi, In addition I want to add couple words about flexibility and debugging capabilities. I believe it is quite important for HOT template engine to control all aspects of deployment process execution including software components. Right now I believe Heat lack of control of what is going on the VM side. In my opinion, HOT template user should be able to define what steps are necessary to deploy complex environment and more important, he should be able to provide a hints to the engine how to deal with errors during deployment. Centralized orchestration sees the whole picture of the environment status while scripts on VM usually have quite limited view. Workflows specification can have on_error actions and centralized orchestration will be capable to make smart decision on how to handle errors during deployment. I think we need to have a design discussion about the architecture of this centralized orchestration. From the one side, this orchestration should have the whole information about environment state and as Heat has full exposure to the environment it sounds reasonable to have such orchestration as a part of Heat. At the other side, HOT template should be quite simple to be useful, so additional workflows concepts might overload DSL syntax and additional independent orchestration level also sounds quite reasonable and this is what we have now as a Murano project. It will be nice to have some initial live discussion before the summit as not all developers will be on summit. What do you think about Google hangout session at the end of this week or on the next week? Thanks Gosha On Wed, Oct 9, 2013 at 7:52 AM, Stan Lagun sla...@mirantis.com wrote: Thanks, we're certainly interested in Murano, and are keen to discuss your roadmap, and where requirements and integration opportunities exist Glad to here it. The same is true from Murano side. On sample SQL workflow: that was just an example. I didn't want to bother you with a SQL Server deployment details as it's really not that important. What I've tried to say is that deployments consists of many steps, the steps vary depending on instance's role, user input and so on. The step on one machine often requires some other machine already be in some state or some output from a deployment step happened elsewhere. I do understand that it is doable using Heat alone. Actually we do use Heat for some parts of workflow. We do not talk to Nova or Neutron directly. The special use case of Murano is that there is no HOT template author. Heat is more an administrator's tool who knows how to write HOT templates and wants to deal with low-level configuration aspects. But Murano is quite different. In Murano the developers of workflows/scripts/metadata/etc are not end-users. The user is not doing any sort of programming. He is given a UI dashboard where he can compose desired environment from available building blocks (services). Services may depend on each other and UI guides him how to fulfill
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
I favor separation of concerns. I think (4), at least, has got nothing to do with infrastructure orchestration, the primary concern of today's heat engine. I advocate (4), but as separate functionality. Regards, Mike Alex Rudenko alexei.rude...@gmail.com wrote on 10/09/2013 12:59:22 PM: From: Alex Rudenko alexei.rude...@gmail.com To: OpenStack Development Mailing List openstack-dev@lists.openstack.org, Date: 10/09/2013 01:03 PM Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Hi everyone, I've read this thread and I'd like to share some thoughts. In my opinion, workflows (which run on VMs) can be integrated with heat templates as follows: 1. workflow definitions should be defined separately and processed by stand-alone workflow engines (chef, puppet etc). 2. the HOT resources should reference workflows which they require, specifying a type of workflow and the way to access a workflow definition. The workflow definition might be provided along with HOT. 3. Heat should treat the orchestration templates as transactions (i.e. Heat should be able to rollback in two cases: 1) if something goes wrong during processing of an orchestration workflow 2) when a stand-alone workflow engine reports an error during processing of a workflow associated with a resource) 4. Heat should expose an API which enables basic communication between running workflows. Additionally, Heat should provide an API to workflows that allows workflows to specify whether they completed successfully or not. The reference to these APIs should be passed to the workflow engine that is responsible for executing workflows on VMs. Pros of each point: 1 2 - keeps Heat simple and gives a possibility to choose the best workflows and engines among available ones. 3 - adds some kind of all-or-nothing semantics improving the control and awareness of what's going on inside VMs. 4 - allows workflow synchronization and communication through Heat API. Provides the error reporting mechanism for workflows. If a workflow does not need this functionality, it can ignore it. Cons: - Changes to existing workflows making them aware of Heat existence are required. These thoughts might show some gaps in my understanding of how Heat works, but I would like to share them anyway. Best regards, Oleksii Rudenko ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Excerpts from Georgy Okrokvertskhov's message of 2013-10-09 08:37:36 -0700:
Hi,
In addition I want to add couple words about flexibility and debugging
capabilities. I believe it is quite important for HOT template engine to
control all aspects of deployment process execution including software
components. Right now I believe Heat lack of control of what is going on
the VM side. In my opinion, HOT template user should be able to define
what steps are necessary to deploy complex environment and more important,
he should be able to provide a hints to the engine how to deal with errors
during deployment. Centralized orchestration sees the whole picture of the
environment status while scripts on VM usually have quite limited view.
Workflows specification can have on_error actions and centralized
orchestration will be capable to make smart decision on how to handle
errors during deployment.
What you have described above is some of what I'd like to see in HOT.
It is an evolution beyond the limitations of the waitcondition that
keeps things simple. Basically, orchestration providing a hook point
at which a portion of the workflow is deferred to some other tool. The
tool signals back when it is done, or has an error. We have that now,
but now, the errors just halt the process. We definitely need a way to
say something like this:
on_error_code: {default: {rebuild_resources: [ Instance1, Loadbalancer1]}}
The OS::Heat::HARestarter was sort of an attempt at some of this.
I take issue with a tool that wants to control everything. That may be
the easy way out, however I believe that it will lead to a very large,
very complex tool that users will be suspicious of.
For me, I'd like to know where my orchestration ends, and my software
configuration, installation, and state management all begin. The interface
can be such that Heat doesn't have to be omniscient and omnipotent.
It just has to help simplify the task of orchestration and get out of
workflow/config/installation/etc.'s way.
I think we need to have a design discussion about the architecture of this
centralized orchestration. From the one side, this orchestration should
have the whole information about environment state and as Heat has full
exposure to the environment it sounds reasonable to have such orchestration
as a part of Heat. At the other side, HOT template should be quite simple
to be useful, so additional workflows concepts might overload DSL syntax
and additional independent orchestration level also sounds quite reasonable
and this is what we have now as a Murano project.
It sounds like you have learned a lot on this journey and it would
definitely be valuable to collaborate with you so that we can make sure
Heat accommodates the use cases you have uncovered.
It will be nice to have some initial live discussion before the summit as
not all developers will be on summit. What do you think about Google
hangout session at the end of this week or on the next week?
I find that summit sessions are most useful when we are at a point where
there are just a few decision points to get through. If we come with too
much already done, then group-think will take over and less-well-formed
ideas will get squelched. If we come without clearly defined decisions
to make, then we'll bike-shed for the full 40 minutes.
So, given that, I think a brief pre-summit discussion is a good idea to
help us figure out where we may have conflicting views and then we can
come ready to hash those out in a high bandwidth environment.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Steven Hardy sha...@redhat.com wrote on 10/09/2013 05:24:38 AM: So as has already been mentioned, Heat defines an internal workflow, based on the declarative model defined in the template. The model should define dependencies, and Heat should convert those dependencies into a workflow internally. IMO if the user also needs to describe a workflow explicitly in the template, then we've probably failed to provide the right template interfaces for describing depenendencies. I agree with Steven here, models should define the dependencies and Heat should realize/enforce them. An important design issue is granularity at which dependencies are defined and enforced. I am aware of the wait-condition and signal constructs in Heat, but I find them a bit low-level as they are prone to the classic dead-lock and race condition problems. I would like to have higher level constructs that support finer-granularity dependences which are needed for software orchestration. Reading through the various disucssion on this topic in this mailing list, I see that many would like to have such higher level constructs for coordination. In our experience with software orchestration using our own DSL and also with some extensions to Heat, we found that the granularity of VMs or Resources to be too coarse for defining dependencies for software orchestration. For example, consider a two VM app, with VMs vmA, vmB, and a set of software components (ai's and bi's) to be installed on them: vmA = base-vmA + a1 + a2 + a3 vmB = base-vmB + b1 + b2 + b3 let us say that software component b1 of vmB, requires a config value produced by software component a1 of vmA. How to declaratively model this dependence? Clearly, modeling a dependence between just base-vmA and base-vmB is not enough. However, defining a dependence between the whole of vmA and vmB is too coarse. It would be ideal to be able to define a dependence at the granularity of software components, i.e., vmB.b1 depends on vmA.a1. Of course, it would also be good to capture what value is passed between vmB.b1 and vmA.a1, so that the communication can be facilitated by the orchestration engine. We found that such finer granular modeling of the dependencies provides two valuable benefits: 1. Faster total (resources + software setup) deployment time. For the example described above, a coarse-granularity dependence enforcer would start the deployment of base-vmB after vmA + a1 + a2 + a3 is completed, but a fine-granularity dependence enforcer would start base-vmA and base-vmB concurrently, and then suspend the execution of vmB.b1 until vmA.a1 is complete and then let the rest of deployment proceed concurrently, resulting in a faster completion. 2. More flexible dependencies. For example, mutual dependencies between resources, which can be satisfied when orchestrated at a finer granularity. Using the example described above, fine-granularity would allow vmB.b1 depends_on vmA.a1 and also vmA.a3 depends_on vmB.b2, but coarse-granularity model would flag this as a cyclic dependence. There are two aspects that needs support: 1. Heat/HOT template level constructs to support declarative expression of such fine-granularity dependencies and the values communicated / passed for the dependence. 2. Support from Heat engine / analyzer in supporting the runtime ordering, coordination between resources, and also the communication of the values. What are your thoughts?___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Hi Lakshminaraya, Thank you for bringing your use case and your thought here. That is exactly tried to achieve in Murano project. There are important aspects you highlighted. Sometime resource model is two high level to describe deployment process. If you start to use more granular approach to have defined steps of deployment you will finish with workflow approach where you have fine control of deployment process but description will be quite complex. I think the HOT approach is to provide a simple way do describe you deployment which consists of solid bricks (resources). If you are using standard resources you can easily create a simple HOT template for your deployment. If you need some custom resource you basically have two options - create new resource class and hide all complexity inside the code or use some workflows language to describe all steps required. The first approach is currently supported by Heat. We have an experience of creating new custom resources for orchestration deployment to specific IT infrastructure with specific hardware and software. Right now we are trying to figure out the possibility of adding workflows to HOT. It looks like adding workflows language directly might harm HOT simplicity by overloaded DSL syntax and structures. I actually see the value in Steve's idea to have specific resource or resource set to call workflows execution on external engine. In this case HOT template will be still pretty simple as all workflow details will be hidden, but still manageable without code writing. Thanks Gosha On Wed, Oct 9, 2013 at 11:31 AM, Lakshminaraya Renganarayana lren...@us.ibm.com wrote: Steven Hardy sha...@redhat.com wrote on 10/09/2013 05:24:38 AM: So as has already been mentioned, Heat defines an internal workflow, based on the declarative model defined in the template. The model should define dependencies, and Heat should convert those dependencies into a workflow internally. IMO if the user also needs to describe a workflow explicitly in the template, then we've probably failed to provide the right template interfaces for describing depenendencies. I agree with Steven here, models should define the dependencies and Heat should realize/enforce them. An important design issue is granularity at which dependencies are defined and enforced. I am aware of the wait-condition and signal constructs in Heat, but I find them a bit low-level as they are prone to the classic dead-lock and race condition problems. I would like to have higher level constructs that support finer-granularity dependences which are needed for software orchestration. Reading through the various disucssion on this topic in this mailing list, I see that many would like to have such higher level constructs for coordination. In our experience with software orchestration using our own DSL and also with some extensions to Heat, we found that the granularity of VMs or Resources to be too coarse for defining dependencies for software orchestration. For example, consider a two VM app, with VMs vmA, vmB, and a set of software components (ai's and bi's) to be installed on them: vmA = base-vmA + a1 + a2 + a3 vmB = base-vmB + b1 + b2 + b3 let us say that software component b1 of vmB, requires a config value produced by software component a1 of vmA. How to declaratively model this dependence? Clearly, modeling a dependence between just base-vmA and base-vmB is not enough. However, defining a dependence between the whole of vmA and vmB is too coarse. It would be ideal to be able to define a dependence at the granularity of software components, i.e., vmB.b1 depends on vmA.a1. Of course, it would also be good to capture what value is passed between vmB.b1 and vmA.a1, so that the communication can be facilitated by the orchestration engine. We found that such finer granular modeling of the dependencies provides two valuable benefits: 1. Faster total (resources + software setup) deployment time. For the example described above, a coarse-granularity dependence enforcer would start the deployment of base-vmB after vmA + a1 + a2 + a3 is completed, but a fine-granularity dependence enforcer would start base-vmA and base-vmB concurrently, and then suspend the execution of vmB.b1 until vmA.a1 is complete and then let the rest of deployment proceed concurrently, resulting in a faster completion. 2. More flexible dependencies. For example, mutual dependencies between resources, which can be satisfied when orchestrated at a finer granularity. Using the example described above, fine-granularity would allow vmB.b1 depends_on vmA.a1 and also vmA.a3 depends_on vmB.b2, but coarse-granularity model would flag this as a cyclic dependence. There are two aspects that needs support: 1. Heat/HOT template level constructs to support declarative expression of such fine-granularity dependencies and the values communicated / passed for the
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Your example sounds a lot like what taskflow is build for doing. https://github.com/stackforge/taskflow/blob/master/taskflow/examples/calculate_in_parallel.py is a decent example. In that one, tasks are created and input/output dependencies are specified (provides, rebind, and the execute function arguments itself). This is combined into the taskflow concept of a flow, one of those flows types is a dependency graph. Using a parallel engine (similar in concept to a heat engine) we can run all non-dependent tasks in parallel. An example that I just created that shows this (and shows it running) that closer matches your example. Program (this will work against the current taskflow codebase): http://paste.openstack.org/show/48156/ Output @ http://paste.openstack.org/show/48157/ -Josh From: Lakshminaraya Renganarayana lren...@us.ibm.commailto:lren...@us.ibm.com Reply-To: OpenStack Development Mailing List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Wednesday, October 9, 2013 11:31 AM To: OpenStack Development Mailing List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Steven Hardy sha...@redhat.commailto:sha...@redhat.com wrote on 10/09/2013 05:24:38 AM: So as has already been mentioned, Heat defines an internal workflow, based on the declarative model defined in the template. The model should define dependencies, and Heat should convert those dependencies into a workflow internally. IMO if the user also needs to describe a workflow explicitly in the template, then we've probably failed to provide the right template interfaces for describing depenendencies. I agree with Steven here, models should define the dependencies and Heat should realize/enforce them. An important design issue is granularity at which dependencies are defined and enforced. I am aware of the wait-condition and signal constructs in Heat, but I find them a bit low-level as they are prone to the classic dead-lock and race condition problems. I would like to have higher level constructs that support finer-granularity dependences which are needed for software orchestration. Reading through the various disucssion on this topic in this mailing list, I see that many would like to have such higher level constructs for coordination. In our experience with software orchestration using our own DSL and also with some extensions to Heat, we found that the granularity of VMs or Resources to be too coarse for defining dependencies for software orchestration. For example, consider a two VM app, with VMs vmA, vmB, and a set of software components (ai's and bi's) to be installed on them: vmA = base-vmA + a1 + a2 + a3 vmB = base-vmB + b1 + b2 + b3 let us say that software component b1 of vmB, requires a config value produced by software component a1 of vmA. How to declaratively model this dependence? Clearly, modeling a dependence between just base-vmA and base-vmB is not enough. However, defining a dependence between the whole of vmA and vmB is too coarse. It would be ideal to be able to define a dependence at the granularity of software components, i.e., vmB.b1 depends on vmA.a1. Of course, it would also be good to capture what value is passed between vmB.b1 and vmA.a1, so that the communication can be facilitated by the orchestration engine. We found that such finer granular modeling of the dependencies provides two valuable benefits: 1. Faster total (resources + software setup) deployment time. For the example described above, a coarse-granularity dependence enforcer would start the deployment of base-vmB after vmA + a1 + a2 + a3 is completed, but a fine-granularity dependence enforcer would start base-vmA and base-vmB concurrently, and then suspend the execution of vmB.b1 until vmA.a1 is complete and then let the rest of deployment proceed concurrently, resulting in a faster completion. 2. More flexible dependencies. For example, mutual dependencies between resources, which can be satisfied when orchestrated at a finer granularity. Using the example described above, fine-granularity would allow vmB.b1 depends_on vmA.a1 and also vmA.a3 depends_on vmB.b2, but coarse-granularity model would flag this as a cyclic dependence. There are two aspects that needs support: 1. Heat/HOT template level constructs to support declarative expression of such fine-granularity dependencies and the values communicated / passed for the dependence. 2. Support from Heat engine / analyzer in supporting the runtime ordering, coordination between resources, and also the communication of the values. What are your thoughts? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
It seems to me that something is missing in our discussion. If something depends on something else there must be a definition of that something. It is clear that it is not the case that one instance depends on another but one application depends on another application. But there is no such thing as application (service, whatever) in HOT templates. Only low-level resources. And even resources cannot be grouped to some application scope because typical HOT template has resources that are shared between several applications (network, security groups etc.). It also possible to have several applications sharing a single VM instance. That brings us to a conclusion that applications and resources cannot be mixed in the same template on the same level of abstraction. Now suppose we did somehow established the dependency between two applications. But this dependency is out of scope of particular HOT template. Thats because HOT template says what user whishes to install. But a dependency between applications is an attribute of applications themselves, not the particular deployment. For example WordPress requires database. It always does. Not that it requires it within this particular template but a a universal rule. In Murano we call it data vs. metadata separation. If there is a metadata that says WordPress requires DB then you not just only don't have to repeat it in each template but you cannot even ask a system to deploy WordPress without DB. So the question is maybe we need to think about applications/services and their metadata before going into workflow orchestration? Otherwise the whole orchestration would be reinvented time and time again with each new HOT template. What are your thoughts on this? On Wed, Oct 9, 2013 at 11:37 PM, Georgy Okrokvertskhov gokrokvertsk...@mirantis.com wrote: Hi Lakshminaraya, Thank you for bringing your use case and your thought here. That is exactly tried to achieve in Murano project. There are important aspects you highlighted. Sometime resource model is two high level to describe deployment process. If you start to use more granular approach to have defined steps of deployment you will finish with workflow approach where you have fine control of deployment process but description will be quite complex. I think the HOT approach is to provide a simple way do describe you deployment which consists of solid bricks (resources). If you are using standard resources you can easily create a simple HOT template for your deployment. If you need some custom resource you basically have two options - create new resource class and hide all complexity inside the code or use some workflows language to describe all steps required. The first approach is currently supported by Heat. We have an experience of creating new custom resources for orchestration deployment to specific IT infrastructure with specific hardware and software. Right now we are trying to figure out the possibility of adding workflows to HOT. It looks like adding workflows language directly might harm HOT simplicity by overloaded DSL syntax and structures. I actually see the value in Steve's idea to have specific resource or resource set to call workflows execution on external engine. In this case HOT template will be still pretty simple as all workflow details will be hidden, but still manageable without code writing. Thanks Gosha On Wed, Oct 9, 2013 at 11:31 AM, Lakshminaraya Renganarayana lren...@us.ibm.com wrote: Steven Hardy sha...@redhat.com wrote on 10/09/2013 05:24:38 AM: So as has already been mentioned, Heat defines an internal workflow, based on the declarative model defined in the template. The model should define dependencies, and Heat should convert those dependencies into a workflow internally. IMO if the user also needs to describe a workflow explicitly in the template, then we've probably failed to provide the right template interfaces for describing depenendencies. I agree with Steven here, models should define the dependencies and Heat should realize/enforce them. An important design issue is granularity at which dependencies are defined and enforced. I am aware of the wait-condition and signal constructs in Heat, but I find them a bit low-level as they are prone to the classic dead-lock and race condition problems. I would like to have higher level constructs that support finer-granularity dependences which are needed for software orchestration. Reading through the various disucssion on this topic in this mailing list, I see that many would like to have such higher level constructs for coordination. In our experience with software orchestration using our own DSL and also with some extensions to Heat, we found that the granularity of VMs or Resources to be too coarse for defining dependencies for software orchestration. For example, consider a two VM app, with VMs vmA, vmB, and a set of software components (ai's and
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
On 10/9/2013 12:55 PM, Joshua Harlow wrote: Your example sounds a lot like what taskflow is build for doing. I'm not that familiar with Heat, so I wanted to bounce this off of you before doing a public foot-in-mouth on the mailing list. Is the real issue here the difference between *building* a set of servers (Heat) versus performing a specific *task* on a set of servers that was already built (taskflow)? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Georgy Okrokvertskhov gokrokvertsk...@mirantis.com wrote on 10/09/2013 03:37:01 PM: From: Georgy Okrokvertskhov gokrokvertsk...@mirantis.com To: OpenStack Development Mailing List openstack-dev@lists.openstack.org Date: 10/09/2013 03:41 PM Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Thank you for bringing your use case and your thought here. That is exactly tried to achieve in Murano project. There are important aspects you highlighted. Sometime resource model is two high level to describe deployment process. If you start to use more granular approach to have defined steps of deployment you will finish with workflow approach where you have fine control of deployment process but description will be quite complex. IMHO workflow approaches tend to be heavy-weight. So, I am hoping for more light-weight data-flow constructs and mechanisms that can help with the coordination scenarios I have outlined. Data-flow constructs and mechanisms have had lots of success in other domains, and I wondering why can't we (the Heat community) leverage the related theory and tools! Thanks, LN ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Thanks for the clarification. I'm not sure how much of heat has to change to support what you are aiming for. Maybe heat should use taskflow ;) From: Lakshminaraya Renganarayana lren...@us.ibm.commailto:lren...@us.ibm.com Date: Wednesday, October 9, 2013 4:34 PM To: Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com Cc: OpenStack Development Mailing List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Hi Joshua, I agree that there is an element of taskflow in what I described. But, I am aiming for something much more lightweight which can be naturally blended with HOT constructs and Heat engine. To be a bit more specific, Heat already has dependencies and coordination mechanisms. So, I am aiming for may be just one additional construct in Heat/HOT and some logic in Heat that would support coordination. Thanks, LN _ Lakshminarayanan Renganarayana Research Staff Member IBM T.J. Watson Research Center http://researcher.ibm.com/person/us-lrengan Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote on 10/09/2013 03:55:00 PM: From: Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com To: OpenStack Development Mailing List openstack- d...@lists.openstack.orgmailto:d...@lists.openstack.org, Lakshminaraya Renganarayana/Watson/IBM@IBMUS Date: 10/09/2013 03:55 PM Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Your example sounds a lot like what taskflow is build for doing. https://github.com/stackforge/taskflow/blob/master/taskflow/ examples/calculate_in_parallel.py is a decent example. In that one, tasks are created and input/output dependencies are specified (provides, rebind, and the execute function arguments itself). This is combined into the taskflow concept of a flow, one of those flows types is a dependency graph. Using a parallel engine (similar in concept to a heat engine) we can run all non-dependent tasks in parallel. An example that I just created that shows this (and shows it running) that closer matches your example. Program (this will work against the current taskflow codebase): http://paste.openstack.org/show/48156/ Output @ http://paste.openstack.org/show/48157/ -Josh From: Lakshminaraya Renganarayana lren...@us.ibm.commailto:lren...@us.ibm.com Reply-To: OpenStack Development Mailing List openstack- d...@lists.openstack.orgmailto:d...@lists.openstack.org Date: Wednesday, October 9, 2013 11:31 AM To: OpenStack Development Mailing List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Steven Hardy sha...@redhat.commailto:sha...@redhat.com wrote on 10/09/2013 05:24:38 AM: So as has already been mentioned, Heat defines an internal workflow, based on the declarative model defined in the template. The model should define dependencies, and Heat should convert those dependencies into a workflow internally. IMO if the user also needs to describe a workflow explicitly in the template, then we've probably failed to provide the right template interfaces for describing depenendencies. I agree with Steven here, models should define the dependencies and Heat should realize/enforce them. An important design issue is granularity at which dependencies are defined and enforced. I am aware of the wait-condition and signal constructs in Heat, but I find them a bit low-level as they are prone to the classic dead-lock and race condition problems. I would like to have higher level constructs that support finer-granularity dependences which are needed for software orchestration. Reading through the various disucssion on this topic in this mailing list, I see that many would like to have such higher level constructs for coordination. In our experience with software orchestration using our own DSL and also with some extensions to Heat, we found that the granularity of VMs or Resources to be too coarse for defining dependencies for software orchestration. For example, consider a two VM app, with VMs vmA, vmB, and a set of software components (ai's and bi's) to be installed on them: vmA = base-vmA + a1 + a2 + a3 vmB = base-vmB + b1 + b2 + b3 let us say that software component b1 of vmB, requires a config value produced by software component a1 of vmA. How to declaratively model this dependence? Clearly, modeling a dependence between just base-vmA and base-vmB is not enough. However, defining a dependence between the whole of vmA and vmB is too coarse. It would be ideal to be able to define a dependence at the granularity of software components, i.e., vmB.b1 depends on vmA.a1. Of course, it would also be good to capture what value is passed between
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Stan Lagun sla...@mirantis.com wrote on 10/09/2013 04:07:33 PM: It seems to me that something is missing in our discussion. If something depends on something else there must be a definition of that something. It is clear that it is not the case that one instance depends on another but one application depends on another application. But there is no such thing as application (service, whatever) in HOT templates. Only low-level resources. And even resources cannot be grouped to some application scope because typical HOT template has resources that are shared between several applications (network, security groups etc.). It also possible to have several applications sharing a single VM instance. That brings us to a conclusion that applications and resources cannot be mixed in the same template on the same level of abstraction. Good point on the levels of abstraction. Now suppose we did somehow established the dependency between two applications. But this dependency is out of scope of particular HOT template. Thats because HOT template says what user whishes to install. But a dependency between applications is an attribute of applications themselves, not the particular deployment. For example WordPress requires database. It always does. Not that it requires it within this particular template but a a universal rule. In Murano we call it data vs. metadata separation. If there is a metadata that says WordPress requires DB then you not just only don't have to repeat it in each template but you cannot even ask a system to deploy WordPress without DB. I think the kind of dependency you have outlined above is more of software component requirements of an application. These kind of semantic dependencies are important and are probably outside the scope of Heat. The kind of dependencies I referred to are of the nature of data-flow between software components: for example, a tomcat application server needs (and hence, depends on) the DB's username/password to set up its configuration. How do we model such a data-flow dependence and how to we facilitate the communication of such values from the DB to the tomcat component? IMHO, such questions are related to Heat. So the question is maybe we need to think about applications/ services and their metadata before going into workflow orchestration? Otherwise the whole orchestration would be reinvented time and time again with each new HOT template. What are your thoughts on this? I find your separation of metadata vs. data useful. In my opinion, the kind of metadata you are trying to capture would be best modeled by a DSL that sits on top of HOT/Heat. Thanks, LN On Wed, Oct 9, 2013 at 11:37 PM, Georgy Okrokvertskhov gokrokvertsk...@mirantis.com wrote: Hi Lakshminaraya, Thank you for bringing your use case and your thought here. That is exactly tried to achieve in Murano project. There are important aspects you highlighted. Sometime resource model is two high level to describe deployment process. If you start to use more granular approach to have defined steps of deployment you will finish with workflow approach where you have fine control of deployment process but description will be quite complex. I think the HOT approach is to provide a simple way do describe you deployment which consists of solid bricks (resources). If you are using standard resources you can easily create a simple HOT template for your deployment. If you need some custom resource you basically have two options - create new resource class and hide all complexity inside the code or use some workflows language to describe all steps required. The first approach is currently supported by Heat. We have an experience of creating new custom resources for orchestration deployment to specific IT infrastructure with specific hardware and software. Right now we are trying to figure out the possibility of adding workflows to HOT. It looks like adding workflows language directly might harm HOT simplicity by overloaded DSL syntax and structures. I actually see the value in Steve's idea to have specific resource or resource set to call workflows execution on external engine. In this case HOT template will be still pretty simple as all workflow details will be hidden, but still manageable without code writing. Thanks Gosha On Wed, Oct 9, 2013 at 11:31 AM, Lakshminaraya Renganarayana lren...@us.ibm.com wrote: Steven Hardy sha...@redhat.com wrote on 10/09/2013 05:24:38 AM: So as has already been mentioned, Heat defines an internal workflow, based on the declarative model defined in the template. The model should define dependencies, and Heat should convert those dependencies into a workflow internally. IMO if the user also needs to describe a workflow explicitly in the template, then we've probably failed to provide the right template interfaces for describing depenendencies. I agree with Steven here, models should define
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
On 09/10/13 19:31 +0100, Steven Hardy wrote: On Wed, Oct 09, 2013 at 06:59:22PM +0200, Alex Rudenko wrote: Hi everyone, I've read this thread and I'd like to share some thoughts. In my opinion, workflows (which run on VMs) can be integrated with heat templates as follows: 1. workflow definitions should be defined separately and processed by stand-alone workflow engines (chef, puppet etc). I agree, and I think this is the direction we're headed with the software-config blueprints - essentially we should end up with some new Heat *resources* which encapsulate software configuration. Exactly. I think we need a software-configuration-aas sub-project that knows how to take puppet/chef/salt/... config and deploy it. Then Heat just has Resources for these (OS::SoftwareConfig::Puppet). We should even move our WaitConditions and Metadata over to that yet-to-be-made service so that Heat is totally clean of software config. How would this solve ordering issues: resources: config1: type: OS::SoftwareConfig::Puppet hosted_on: server1 ... config2: type: OS::SoftwareConfig::Puppet hosted_on: server1 depends_on: config3 ... config3: type: OS::SoftwareConfig::Puppet hosted_on: server2 depends_on: config1 ... server1: type: OS::Nova::Server ... server2: type: OS::Nova::Server ... Heat knows all about ordering: It starts the resources: server1, server2 config1 config3 config2 There is the normal contract in the client: we post the config to software-config-service and we wait for the state == ACTIVE (when the config is applied) before progressing to a resource that is dependant on it. -Angus IMO there is some confusion around the scope of HOT, we should not be adding functionality to it which already exists in established config management tools IMO, instead we should focus on better integration with exisitng tools at the resource level, and identifying template interfaces which require more flexibility (for example serialization primitives) 2. the HOT resources should reference workflows which they require, specifying a type of workflow and the way to access a workflow definition. The workflow definition might be provided along with HOT. So again, I think this acatually has very little to do with HOT. The *Heat* resources may define software configuration, or possibly some sort of workflow, which is acted upon by $thing which is not Heat. So in the example provided by the OP, maybe you'd have a Murano resource, which knows how to define the input to the Murano API, which might trigger workflow type actions to happen in the Murano service. 3. Heat should treat the orchestration templates as transactions (i.e. Heat should be able to rollback in two cases: 1) if something goes wrong during processing of an orchestration workflow 2) when a stand-alone workflow engine reports an error during processing of a workflow associated with a resource) So we already have the capability for resources to recieve signals, which would allow (2) in the asynchronous case. But it seems to me that this is still a serialization problem, ie a synchronous case, therefore (2) is just part of (1). E.g - Heat stack create starts - Murano resource created (CREATE IN_PROGRESS state) - Murano workdlow stuff happens, signals Heat with success/failure - Murano resource transitions to either COMPLETE or FAILED state - If a FAILED state happened, e.g on update, we can roll back to the previous stack definition (this is already possible in Heat) 4. Heat should expose an API which enables basic communication between running workflows. Additionally, Heat should provide an API to workflows that allows workflows to specify whether they completed successfully or not. The reference to these APIs should be passed to the workflow engine that is responsible for executing workflows on VMs. I personally don't think this is in scope for Heat. We already have an API which exposes the status of stacks and resources. Exposing some different API which describes a workflow implemented by a specific subset of resource types makes no sense to me. Pros of each point: 1 2 - keeps Heat simple and gives a possibility to choose the best workflows and engines among available ones. 3 - adds some kind of all-or-nothing semantics improving the control and awareness of what's going on inside VMs. 4 - allows workflow synchronization and communication through Heat API. Provides the error reporting mechanism for workflows. If a workflow does not need this functionality, it can ignore it. IMHO (4) is very much a step too far, and is not well aligned with the current interfaces provided by Heat. I'm really keen to further discuss the use-cases here, but if possible, it would be helpful if folks can describe their requirements in less abstract terms, and with reference to our existing interfaces and template model. These thoughts might show some
[openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Hello, I’m one of the engineer working on Murano project. Recently we started a discussion about Murano and Heat Software orchestration and I want to continue this discussion with more technical details. In our project we do deployment of complex multi-instance Windows services. Those services usually require specific multi-VM orchestration that is currently impossible or at least not that easy to achieve with Heat. As you are currently doing HOT software orchestration design we would like to participate in HOT Software orchestration design and contribute into it, so that the Heat could address use-cases which we believe are very common. For example here is how deployment of a SQL Server cluster goes: 1. Allocate Windows VMs for SQL Server cluster 2. Enable secondary IP address from user input on all SQL Windows instances 3. Install SQL Server prerequisites on each node 4. Choose a master node and install Failover Cluster on it 5. Configure all nodes so that they know which one of them is the master 6. Install SQL Server on all nodes 7. Initialize AlwaysOn on all nodes except for the master 8. Initialize Primary replica 9. Initialize secondary replicas All of the steps must take place in appropriate order depending on the state of other nodes. Some steps require an output from previous steps and all of them require some input parameters. SQL Server requires an Active Directory service in order to use Failover mechanism and installation of Active Directory with primary and secondary controllers is a complex workflow of its own. That is why it is necessary to have some central coordination service which would handle deployment workflow and perform specific actions (create VMs and other OpenStack resources, do something on that VM) on each stage according to that workflow. We think that Heat is the best place for such service. Our idea is to extend HOT DSL by adding workflow definition capabilities as an explicit list of resources, components’ states and actions. States may depend on each other so that you can reach state X only after you’ve reached states Y and Z that the X depends on. The goal is from initial state to reach some final state “Deployed”. There is such state graph for each of our deployment entities (service, VMs, other things). There is also an action that must be performed on each state. For example states graph from example above would look like this: The goal is to reach Service_Done state which depends on VM1_Done and VM2_Done states and so on from initial Service_Start state. We propose to extend HOT DSL with workflow definition capabilities where you can describe step by step instruction to install service and properly handle errors on each step. We already have an experience in implementation of the DSL, workflow description and processing mechanism for complex deployments and believe we’ll all benefit by re-using this experience and existing code, having properly discussed and agreed on abstraction layers and distribution of responsibilities between OS components. There is an idea of implementing part of workflow processing mechanism as a part of Convection proposal, which would allow other OS projects to benefit by using this. We would like to discuss if such design could become a part of future Heat version as well as other possible contributions from Murano team. Regards, Stan Lagun -- Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
On 09/10/13 00:53 +0400, Stan Lagun wrote: Hello, I’m one of the engineer working on Murano project. Recently we started a discussion about Murano and Heat Software orchestration and I want to continue this discussion with more technical details. I hope you are going to be at summit, as I expect this to an important session we have there: Related summit sessions: http://summit.openstack.org/cfp/details/82 http://summit.openstack.org/cfp/details/121 http://summit.openstack.org/cfp/details/78 Related blueprints: https://blueprints.launchpad.net/heat/+spec/software-configuration-provider https://blueprints.launchpad.net/heat/+spec/hot-software-config-deps https://blueprints.launchpad.net/heat/+spec/hot-software-config https://blueprints.launchpad.net/heat/+spec/windows-instances Excuse me if you are well aware of these. -Angus In our project we do deployment of complex multi-instance Windows services. Those services usually require specific multi-VM orchestration that is currently impossible or at least not that easy to achieve with Heat. As you are currently doing HOT software orchestration design we would like to participate in HOT Software orchestration design and contribute into it, so that the Heat could address use-cases which we believe are very common. For example here is how deployment of a SQL Server cluster goes: 1. Allocate Windows VMs for SQL Server cluster 2. Enable secondary IP address from user input on all SQL Windows instances 3. Install SQL Server prerequisites on each node 4. Choose a master node and install Failover Cluster on it 5. Configure all nodes so that they know which one of them is the master 6. Install SQL Server on all nodes 7. Initialize AlwaysOn on all nodes except for the master 8. Initialize Primary replica 9. Initialize secondary replicas All of the steps must take place in appropriate order depending on the state of other nodes. Some steps require an output from previous steps and all of them require some input parameters. SQL Server requires an Active Directory service in order to use Failover mechanism and installation of Active Directory with primary and secondary controllers is a complex workflow of its own. That is why it is necessary to have some central coordination service which would handle deployment workflow and perform specific actions (create VMs and other OpenStack resources, do something on that VM) on each stage according to that workflow. We think that Heat is the best place for such service. Our idea is to extend HOT DSL by adding workflow definition capabilities as an explicit list of resources, components’ states and actions. States may depend on each other so that you can reach state X only after you’ve reached states Y and Z that the X depends on. The goal is from initial state to reach some final state “Deployed”. There is such state graph for each of our deployment entities (service, VMs, other things). There is also an action that must be performed on each state. For example states graph from example above would look like this: The goal is to reach Service_Done state which depends on VM1_Done and VM2_Done states and so on from initial Service_Start state. We propose to extend HOT DSL with workflow definition capabilities where you can describe step by step instruction to install service and properly handle errors on each step. We already have an experience in implementation of the DSL, workflow description and processing mechanism for complex deployments and believe we’ll all benefit by re-using this experience and existing code, having properly discussed and agreed on abstraction layers and distribution of responsibilities between OS components. There is an idea of implementing part of workflow processing mechanism as a part of Convection proposal, which would allow other OS projects to benefit by using this. We would like to discuss if such design could become a part of future Heat version as well as other possible contributions from Murano team. I am really happy that you want to get involved and this sounds like it functionally matches quite well to the blueprints at the top. -Angus Regards, Stan Lagun -- Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Hi Angus, We will have representatives from our Team . Alex Tivelkov and I will be on summit. We definitely will participate in design sessions for these hot topics. Before the summit we will work in etherpads to add necessary technical information to have a solid background for discussions. We already checked BPs and we think we can add more details to them. As I see, not all BP have etherpad links published on launchpad. Should we create them and attach to BP's whiteboards? Do you have any internal design sessions before summit? It would be nice if we can participate in them too. Thanks Gosha On Tue, Oct 8, 2013 at 2:46 PM, Angus Salkeld asalk...@redhat.com wrote: On 09/10/13 00:53 +0400, Stan Lagun wrote: Hello, I’m one of the engineer working on Murano project. Recently we started a discussion about Murano and Heat Software orchestration and I want to continue this discussion with more technical details. I hope you are going to be at summit, as I expect this to an important session we have there: Related summit sessions: http://summit.openstack.org/**cfp/details/82http://summit.openstack.org/cfp/details/82 http://summit.openstack.org/**cfp/details/121http://summit.openstack.org/cfp/details/121 http://summit.openstack.org/**cfp/details/78http://summit.openstack.org/cfp/details/78 Related blueprints: https://blueprints.launchpad.**net/heat/+spec/software-** configuration-providerhttps://blueprints.launchpad.net/heat/+spec/software-configuration-provider https://blueprints.launchpad.**net/heat/+spec/hot-software-**config-depshttps://blueprints.launchpad.net/heat/+spec/hot-software-config-deps https://blueprints.launchpad.**net/heat/+spec/hot-software-**confighttps://blueprints.launchpad.net/heat/+spec/hot-software-config https://blueprints.launchpad.**net/heat/+spec/windows-**instanceshttps://blueprints.launchpad.net/heat/+spec/windows-instances Excuse me if you are well aware of these. -Angus In our project we do deployment of complex multi-instance Windows services. Those services usually require specific multi-VM orchestration that is currently impossible or at least not that easy to achieve with Heat. As you are currently doing HOT software orchestration design we would like to participate in HOT Software orchestration design and contribute into it, so that the Heat could address use-cases which we believe are very common. For example here is how deployment of a SQL Server cluster goes: 1. Allocate Windows VMs for SQL Server cluster 2. Enable secondary IP address from user input on all SQL Windows instances 3. Install SQL Server prerequisites on each node 4. Choose a master node and install Failover Cluster on it 5. Configure all nodes so that they know which one of them is the master 6. Install SQL Server on all nodes 7. Initialize AlwaysOn on all nodes except for the master 8. Initialize Primary replica 9. Initialize secondary replicas All of the steps must take place in appropriate order depending on the state of other nodes. Some steps require an output from previous steps and all of them require some input parameters. SQL Server requires an Active Directory service in order to use Failover mechanism and installation of Active Directory with primary and secondary controllers is a complex workflow of its own. That is why it is necessary to have some central coordination service which would handle deployment workflow and perform specific actions (create VMs and other OpenStack resources, do something on that VM) on each stage according to that workflow. We think that Heat is the best place for such service. Our idea is to extend HOT DSL by adding workflow definition capabilities as an explicit list of resources, components’ states and actions. States may depend on each other so that you can reach state X only after you’ve reached states Y and Z that the X depends on. The goal is from initial state to reach some final state “Deployed”. There is such state graph for each of our deployment entities (service, VMs, other things). There is also an action that must be performed on each state. For example states graph from example above would look like this: The goal is to reach Service_Done state which depends on VM1_Done and VM2_Done states and so on from initial Service_Start state. We propose to extend HOT DSL with workflow definition capabilities where you can describe step by step instruction to install service and properly handle errors on each step. We already have an experience in implementation of the DSL, workflow description and processing mechanism for complex deployments and believe we’ll all benefit by re-using this experience and existing code, having properly discussed and agreed on abstraction layers and distribution of responsibilities between OS components. There is an idea of implementing part of workflow processing
Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows
Excerpts from Stan Lagun's message of 2013-10-08 13:53:45 -0700: Hello, I’m one of the engineer working on Murano project. Recently we started a discussion about Murano and Heat Software orchestration and I want to continue this discussion with more technical details. In our project we do deployment of complex multi-instance Windows services. Those services usually require specific multi-VM orchestration that is currently impossible or at least not that easy to achieve with Heat. As you are currently doing HOT software orchestration design we would like to participate in HOT Software orchestration design and contribute into it, so that the Heat could address use-cases which we believe are very common. For example here is how deployment of a SQL Server cluster goes: 1. Allocate Windows VMs for SQL Server cluster 2. Heat does this quite well. :) Enable secondary IP address from user input on all SQL Windows instances 3. I don't understand what that means. Install SQL Server prerequisites on each node 4. Automating software installation is a very common aspect of most configuration tools. Choose a master node and install Failover Cluster on it 5. Leader election should be built in to services that take advantage of it. External leader election is really hard without all of the primitives built into the service. Otherwise you have to resort to things like STONITH to avoid split brain. If Heat tries to do this, it will just result in a complicated design with very few happy users. Configure all nodes so that they know which one of them is the master 6. Anyway, Heat can do this quite well. The template author can just choose one, and flag it as the master in the resource's Metadata. Point all others at that one using Ref's to it. Configuring nodes should be something an automated configuration tool does well. It is pretty straight forward to have a tool fetch the Metadata of a resource via the Heat API and feed it to a configuration tool. Or you can even teach your config tool to read the Metadata. Install SQL Server on all nodes 7. Same as before - any automated software installation tool should suffice. Initialize AlwaysOn on all nodes except for the master 8. I don't know what that means. Initialize Primary replica 9. Initialize secondary replicas I also don't know what that entails, but I would suspect that it is the kind of thing you do via the SQL server API once everything is done. For that, you just need to poll a waitcondition and when all of the nodes have signaled the waitcondition, run this. All of the steps must take place in appropriate order depending on the state of other nodes. Some steps require an output from previous steps and all of them require some input parameters. SQL Server requires an Active Directory service in order to use Failover mechanism and installation of Active Directory with primary and secondary controllers is a complex workflow of its own. That is why it is necessary to have some central coordination service which would handle deployment workflow and perform specific actions (create VMs and other OpenStack resources, do something on that VM) on each stage according to that workflow. We think that Heat is the best place for such service. I'm not so sure. Heat is part of the Orchestration program, not workflow. Our idea is to extend HOT DSL by adding workflow definition capabilities as an explicit list of resources, components’ states and actions. States may depend on each other so that you can reach state X only after you’ve reached states Y and Z that the X depends on. The goal is from initial state to reach some final state “Deployed”. Orchestration is not workflow, and HOT is an orchestration templating language, not a workflow language. Extending it would just complect two very different (though certainly related) tasks. I think the appropriate thing to do is actually to join up with the TaskFlow project and consider building it into a workflow service or tools (it is just a library right now). There is such state graph for each of our deployment entities (service, VMs, other things). There is also an action that must be performed on each state. Heat does its own translation of the orchestration template into a workflow right now, but we have already discussed using TaskFlow to break up the orchestration graph into distributable jobs. As we get more sophisticated on updates (rolling/canary for instance) we'll need to be able to reason about the process without having to glue all the pieces together. We propose to extend HOT DSL with workflow definition capabilities where you can describe step by step instruction to install service and properly handle errors on each step. We already have an experience in implementation of the DSL, workflow description and processing mechanism for complex deployments and
