Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
On Wed, Mar 26, 2014 at 11:25 AM, Keith Bray keith.b...@rackspace.comwrote: On 3/25/14 11:55 AM, Ruslan Kamaldinov rkamaldi...@mirantis.com wrote: * Murano DSL will focus on: a. UI rendering One of the primary reasons I am opposed to using a different DSL/project to accomplish this is that the person authoring the HOT template is usually the system architect, and this is the same person who has the technical knowledge to know what technologies you can swap in/out and still have that system/component work, so they are also the person who can/should define the rules of what component building blocks can and can't work together. There has been an overwhelmingly strong preference from the system architects/DevOps/ApplicationExperts I [1] have talked to for the ability to have control over those rules directly within the HOT file or immediately along-side the HOT file but feed the whole set of files to a single API endpoint. I'm not advocating that this extra stuff be part of Heat Engine (I understand the desire to keep the orchestration engine clean)... But from a barrier to adoption point-of-view, the extra effort for the HOT author to learn another DSL and use yet another system (or even have to write multiple files) should not be underestimated. These people are not OpenStack developers, they are DevOps folks and Application Experts. This is why the Htr[2] project was proposed and threads were started to add extra data to HOT template that Heat engine could essentially ignore, but would make defining UI rendering and component connectivity easy for the HOT author. I think that this is a wrong way to go. First of all there is an issue with separation of concerns as you will have one super-template which will describe the whole world. UI part is only one of the use cases, but when Murano will need some extra parameter will it go to HOT? When Solum will need to define application build sequence to make an app binary from source will this also go to HOT? I think the reason for such talks was the fact that Heat engine accepts only a single file as an input. Which is completely understandable as Heat engine is designed to process a template which describes a set of resources and their relations. While we talk with our customers who want use Murano they are happy to have multiple files keep each one small and simple. In Murano UI definition is a separate file and application developer can have multiple UI file definitions to quickly change look and feel without changing the deploymen template part of application at all. Heat supports template nesting, this is no obvious how the UI for this case will be rendered as this nesting will be processed by Heat engine and final set of resources will be produced by engine. I don't see a big difference in learning between one huge DSL which covers the all possible aspects and a set of small simpler DSLs focused on each specific area. Having one big DSL is worse as the user can construct some complex structures mixing the DSL functions from different are. It will be a nightmare to create an engine which will validate and process such template. I'm all for contributions to OpenStack, so I encourage the Murano team to continue doing its thing if they find it adds value to themselves or others. However, I'd like to see the Orchestration program support the surrounding things the users of the Heat engine want/need from their cloud system instead of having those needs met by separate projects seeking incubation. There are technical ways to keep the core engine clean while having the Orchestration Program API Service move up the stack in terms of cloud user experience. I just think this is a conceptually wrong. The whole idea of OpenStack to have a clean set of API\components focused on specific functionality. Cloud users want to have VMs with attached volumes and networks but is does not mean that it should be single service in OpenStack. This is a power of OpenStack which shows that the proper separation of concern and having multiple services is a right way which allow one to move forward fast making the development process for each service very effective due to narrow scope and functionality focus. I am glad to hear that you want to have something higher up to stack that the current available functionality. I think this support our observation of a huge demand for such higher level functionality in OpenStack. At the same time I am against the proposed way of doing that by extending the Orchestration program mission moving it to upper levels. Having clean layers responsible for particular area is a benefit and strong side of OpenStack from its global architecture view point. b. HOT generation c. Setup other services (like put Mistral tasks to Mistral and bind them with events) Speaking about new DSL for Murano. We're speaking about Application Lifecycle Management. There are a lot of existing tools -
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
On Thu, Mar 27, 2014 at 7:42 PM, Georgy Okrokvertskhov gokrokvertsk...@mirantis.com wrote: Given that I don't see the huge overlap here with Murano functionality as even if Solum stated that as a part of solution Heat template will be generated it does not necessarily mean that Solum itself will do this. From what is listed on the Solum page, in Solum sense - ALM is a way how the application build from source promoted between different CI\CD environments Dev, QA, Stage, Production. Solum can use other service to do this keeping its own focus on the target area. Specifically to the environments - Solum can use Murano environments which for Murano is just a logical unity of multiple applications. Solum can add CI\CD specific stuff on top of it keeping using Murano API for the environment management under the hood. Again, this is a typical OpenStack approach to have different services integrated to achieve the larger goal, keeping services itself very focused. Folks, I'd like to call for a cross-project work group to identify approaches for application description and management in the OpenStack cloud. As this thread shows there are several parties involved - Heat, Mistral, Murano, Solum (did I miss anyone?), there is no clear vision among us where and how we should describe things on top of Heat. We could spend another couple of months in debates, but I feel that focused group of dedicated people (i.e 2 from each project) would progress much faster and will be much more productive. What I'd suggest to expect from this joint group: * Identify how different aspects of applications and their lifecycle can be described and how they can coexist in OpenStack * Define a multi-layered structure to keep each layer with clear focus and set of responsibilities * End goal of the work for this group will be a document with a clear vision of covering areas higher up to Heat stack and how OpenStack should address that. This vision is not clear now for TC and that is the reason they say that it is to big step which Murano did * Agree on further direction * Come to ATL summit, agree again and drink beer Focused group would require additional communication channels: * Calls (Google Hangouts for instance) * Additional IRC meetings * Group work on design documents From Murano project I'd like to propose the following participants: * Stan Lagun (sla...@mirantis.com) * Ruslan Kamaldinov (rkamaldi...@mirantis.com) Do colleagues from Heat, Solum and Mistral feel the same way and would like to support this movement and delegate their participants to this working group? Is this idea viable? -- Ruslan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
On Mar 27, 2014, at 11:27 AM, Ruslan Kamaldinov rkamaldi...@mirantis.com wrote: On Thu, Mar 27, 2014 at 7:42 PM, Georgy Okrokvertskhov gokrokvertsk...@mirantis.com wrote: Given that I don't see the huge overlap here with Murano functionality as even if Solum stated that as a part of solution Heat template will be generated it does not necessarily mean that Solum itself will do this. From what is listed on the Solum page, in Solum sense - ALM is a way how the application build from source promoted between different CI\CD environments Dev, QA, Stage, Production. Solum can use other service to do this keeping its own focus on the target area. Specifically to the environments - Solum can use Murano environments which for Murano is just a logical unity of multiple applications. Solum can add CI\CD specific stuff on top of it keeping using Murano API for the environment management under the hood. Again, this is a typical OpenStack approach to have different services integrated to achieve the larger goal, keeping services itself very focused. Folks, I'd like to call for a cross-project work group to identify approaches for application description and management in the OpenStack cloud. As this thread shows there are several parties involved - Heat, Mistral, Murano, Solum (did I miss anyone?), there is no clear vision among us where and how we should describe things on top of Heat. We could spend another couple of months in debates, but I feel that focused group of dedicated people (i.e 2 from each project) would progress much faster and will be much more productive. What I'd suggest to expect from this joint group: * Identify how different aspects of applications and their lifecycle can be described and how they can coexist in OpenStack * Define a multi-layered structure to keep each layer with clear focus and set of responsibilities * End goal of the work for this group will be a document with a clear vision of covering areas higher up to Heat stack and how OpenStack should address that. This vision is not clear now for TC and that is the reason they say that it is to big step which Murano did * Agree on further direction * Come to ATL summit, agree again and drink beer Focused group would require additional communication channels: * Calls (Google Hangouts for instance) * Additional IRC meetings * Group work on design documents From Murano project I'd like to propose the following participants: * Stan Lagun (sla...@mirantis.com) * Ruslan Kamaldinov (rkamaldi...@mirantis.com) Do colleagues from Heat, Solum and Mistral feel the same way and would like to support this movement and delegate their participants to this working group? Is this idea viable? I will participate on behalf of Solum: Adrian Otto adrian.o...@rackspace.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Keith is my co-worker. I deeply respect his opinion, and agree with his perspective with respect to devops users. That's exactly the persona that OpenStack appeals to today. However, devops is not the only perspective to consider. OpenStack has not yet crossed the barrier into attracting Application Developers en-masse. The Application Developer persona has a different perspective, and would prefer not to use a DSL at all when they are building applications based on common design patterns. One such example is the three-tier-web application. Solum intends to address these use patterns using sensible selectable defaults such that most application developers do not need to use a DSL at all to run apps on OpenStack. They instead use parameters to select well understood and well documented patterns. We will generate HOT files as outputs and feed them into Heat for orchestration. For the smaller minority of application developers who do want a DSL to describe their app topology, we can offer them a choice: 1) Use the Heat DSL, and describe it in terms of infra resources. 2) Use an application-centric DSL that does not directly pertain to the resources in the Heat DSL. In cases where #2 is used, #1 will probably also be used as a complimentary input. There are reasons for having other DSL options that allow modeling of things that are not infrastructure resources. We would be fools to think that HOT is the only home for all that. HOT is about orchestration, not universal entity modeling and management. Devops users will naturally select HOT, not any alternate DSL. With that said, Solum aims to use HOT to the fullest extent. We may also offer to add features to it. Some things still do not fit there. Rather than debating the technical merits of a new DSL, and how it could be accomplished by tweaking existing projects, it would be wise for us to ask (and listen) carefully about WHY the alternate approach is desired. Some of it can certainly be addressed by HOT, and should. Some of it has no business in the orchestration system at all. Let's not quickly dismiss alternate approaches in cases where they do not overlap, or where the style and approach are essentially the same. Example: We have numerous programming languages today. Each one exists for a reason. Understanding those reasons and selecting the right tool for the job is a key to success as a computer scientist. I look forward to further discussions with the Heat team, and other StackForge projects to work to find more common ground and identify those areas where we should splinter off to innovate. Based on my in-person discussions with Georgy from Mirantis this week, I am convinced that they do intend to use Heat to the extent practical in Murano. I am continuing to keep an open mind about the desire to have other DSL systems that work on different planes and for different reasons than Heat. Adrian On Mar 26, 2014, at 1:27 PM, Keith Bray keith.b...@rackspace.com wrote: On 3/25/14 11:55 AM, Ruslan Kamaldinov rkamaldi...@mirantis.com wrote: * Murano DSL will focus on: a. UI rendering One of the primary reasons I am opposed to using a different DSL/project to accomplish this is that the person authoring the HOT template is usually the system architect, and this is the same person who has the technical knowledge to know what technologies you can swap in/out and still have that system/component work, so they are also the person who can/should define the rules of what component building blocks can and can't work together. There has been an overwhelmingly strong preference from the system architects/DevOps/ApplicationExperts I [1] have talked to for the ability to have control over those rules directly within the HOT file or immediately along-side the HOT file but feed the whole set of files to a single API endpoint. I'm not advocating that this extra stuff be part of Heat Engine (I understand the desire to keep the orchestration engine clean)... But from a barrier to adoption point-of-view, the extra effort for the HOT author to learn another DSL and use yet another system (or even have to write multiple files) should not be underestimated. These people are not OpenStack developers, they are DevOps folks and Application Experts. This is why the Htr[2] project was proposed and threads were started to add extra data to HOT template that Heat engine could essentially ignore, but would make defining UI rendering and component connectivity easy for the HOT author. I'm all for contributions to OpenStack, so I encourage the Murano team to continue doing its thing if they find it adds value to themselves or others. However, I'd like to see the Orchestration program support the surrounding things the users of the Heat engine want/need from their cloud system instead of having those needs met by separate projects seeking incubation. There are technical ways to keep the core engine clean
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Hi Thomas, Can you share some documentation of what you're doing right now with TOSCA-compliant layer? We would like to join to this effort. Thanks, Dmitry On Wed, Mar 26, 2014 at 10:38 AM, Thomas Spatzier thomas.spatz...@de.ibm.com wrote: Excerpt from Zane Bitter's message on 26/03/2014 02:26:42: From: Zane Bitter zbit...@redhat.com To: openstack-dev@lists.openstack.org Date: 26/03/2014 02:27 Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? snip Cloud administrators are usually technical guys that are capable of learning HOT and writing YAML templates. They know exact configuration of their cloud (what services are available, what is the version of OpenStack cloud is running) and generally understands how OpenStack works. They also know about software they intent to install. If such guy wants to install Drupal he knows exactly that he needs HOT template describing Fedora VM with Apache + PHP + MySQL + Drupal itself. It is not a problem for him to write such HOT template. I'm aware that TOSCA has these types of constraints, and in fact I suggested to the TOSCA TC that maybe this is where we should draw the line between Heat and some TOSCA-compatible service: HOT should be a concrete description of exactly what you're going to get, whereas some other service (in this case Murano) would act as the constraints solver. e.g. something like an image name would not be hardcoded in a Murano template, you have some constraints about which operating system and what versions should be allowed, and it would pick one and pass it to Heat. So I am interested in this approach. I can just support Zane's statements above. We are working on exactly those issues in the TOSCA YAML definition, so it would be ideal to just collaborate on this. As Zane said, there currently is a thinking that some TOSCA-compliant layer could be a (maybe thin) layer above Heat that resolves a more abstract (thus more portable) template into something concrete, executable. We have started developing code (early versions are on stackforge already) to find out the details. The worst outcome here would be to end up with something that was equivalent to TOSCA but not easily translatable to the TOSCA Simple Profile YAML format (currently a Working Draft). Where 'easily translatable' preferably means 'by just changing some names'. I can't comment on whether this is the case as things stand. The TOSCA Simple Profile in YAML is a working draft at the moment, so we are pretty much open for any input. So let's see to get the right folks together and get it right. Since the Murano folks have indicated before that they are evaluating the option to join the OASIS TC, I am optimistic that we can get the streams together. Having implementation work going on here in this community in parallel to the standards work, and both streams inspiring each other, will be fun :-) Regards, Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Hi Dimitry, the current working draft for the simplified profile in YAML is available at [1]. Note that this is still work in progress, but should already give a good impression of where we want to go. And as I said, we are open for input. The stackforge project [2] that Sahdev from our team created is in its final setup phase (gerrit review still has to be set up), as far as I understood it. My information is that parser code for TOSCA YAML according to the current working draft is going to get in by end of the week. This code is currently maintained in Sahdev's own github repo [3]. Sahdev (IRC spzala) would be the best contact for the moment when it comes to detail questions on the code. [1] https://www.oasis-open.org/committees/document.php?document_id=52571wg_abbrev=tosca [2] https://github.com/stackforge/heat-translator [3] https://github.com/spzala/heat-translator Regards, Thomas From: Dmitry mey...@gmail.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: 26/03/2014 11:17 Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? Hi Thomas, Can you share some documentation of what you're doing right now with TOSCA-compliant layer? We would like to join to this effort. Thanks, Dmitry On Wed, Mar 26, 2014 at 10:38 AM, Thomas Spatzier thomas.spatz...@de.ibm.com wrote: Excerpt from Zane Bitter's message on 26/03/2014 02:26:42: From: Zane Bitter zbit...@redhat.com To: openstack-dev@lists.openstack.org Date: 26/03/2014 02:27 Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? snip Cloud administrators are usually technical guys that are capable of learning HOT and writing YAML templates. They know exact configuration of their cloud (what services are available, what is the version of OpenStack cloud is running) and generally understands how OpenStack works. They also know about software they intent to install. If such guy wants to install Drupal he knows exactly that he needs HOT template describing Fedora VM with Apache + PHP + MySQL + Drupal itself. It is not a problem for him to write such HOT template. I'm aware that TOSCA has these types of constraints, and in fact I suggested to the TOSCA TC that maybe this is where we should draw the line between Heat and some TOSCA-compatible service: HOT should be a concrete description of exactly what you're going to get, whereas some other service (in this case Murano) would act as the constraints solver. e.g. something like an image name would not be hardcoded in a Murano template, you have some constraints about which operating system and what versions should be allowed, and it would pick one and pass it to Heat. So I am interested in this approach. I can just support Zane's statements above. We are working on exactly those issues in the TOSCA YAML definition, so it would be ideal to just collaborate on this. As Zane said, there currently is a thinking that some TOSCA-compliant layer could be a (maybe thin) layer above Heat that resolves a more abstract (thus more portable) template into something concrete, executable. We have started developing code (early versions are on stackforge already) to find out the details. The worst outcome here would be to end up with something that was equivalent to TOSCA but not easily translatable to the TOSCA Simple Profile YAML format (currently a Working Draft). Where 'easily translatable' preferably means 'by just changing some names'. I can't comment on whether this is the case as things stand. The TOSCA Simple Profile in YAML is a working draft at the moment, so we are pretty much open for any input. So let's see to get the right folks together and get it right. Since the Murano folks have indicated before that they are evaluating the option to join the OASIS TC, I am optimistic that we can get the streams together. Having implementation work going on here in this community in parallel to the standards work, and both streams inspiring each other, will be fun :-) Regards, Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Thank you very much for the info! On Wed, Mar 26, 2014 at 6:07 PM, Thomas Spatzier thomas.spatz...@de.ibm.com wrote: Hi Dimitry, the current working draft for the simplified profile in YAML is available at [1]. Note that this is still work in progress, but should already give a good impression of where we want to go. And as I said, we are open for input. The stackforge project [2] that Sahdev from our team created is in its final setup phase (gerrit review still has to be set up), as far as I understood it. My information is that parser code for TOSCA YAML according to the current working draft is going to get in by end of the week. This code is currently maintained in Sahdev's own github repo [3]. Sahdev (IRC spzala) would be the best contact for the moment when it comes to detail questions on the code. [1] https://www.oasis-open.org/committees/document.php?document_id=52571wg_abbrev=tosca [2] https://github.com/stackforge/heat-translator [3] https://github.com/spzala/heat-translator Regards, Thomas From: Dmitry mey...@gmail.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: 26/03/2014 11:17 Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? Hi Thomas, Can you share some documentation of what you're doing right now with TOSCA-compliant layer? We would like to join to this effort. Thanks, Dmitry On Wed, Mar 26, 2014 at 10:38 AM, Thomas Spatzier thomas.spatz...@de.ibm.com wrote: Excerpt from Zane Bitter's message on 26/03/2014 02:26:42: From: Zane Bitter zbit...@redhat.com To: openstack-dev@lists.openstack.org Date: 26/03/2014 02:27 Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? snip Cloud administrators are usually technical guys that are capable of learning HOT and writing YAML templates. They know exact configuration of their cloud (what services are available, what is the version of OpenStack cloud is running) and generally understands how OpenStack works. They also know about software they intent to install. If such guy wants to install Drupal he knows exactly that he needs HOT template describing Fedora VM with Apache + PHP + MySQL + Drupal itself. It is not a problem for him to write such HOT template. I'm aware that TOSCA has these types of constraints, and in fact I suggested to the TOSCA TC that maybe this is where we should draw the line between Heat and some TOSCA-compatible service: HOT should be a concrete description of exactly what you're going to get, whereas some other service (in this case Murano) would act as the constraints solver. e.g. something like an image name would not be hardcoded in a Murano template, you have some constraints about which operating system and what versions should be allowed, and it would pick one and pass it to Heat. So I am interested in this approach. I can just support Zane's statements above. We are working on exactly those issues in the TOSCA YAML definition, so it would be ideal to just collaborate on this. As Zane said, there currently is a thinking that some TOSCA-compliant layer could be a (maybe thin) layer above Heat that resolves a more abstract (thus more portable) template into something concrete, executable. We have started developing code (early versions are on stackforge already) to find out the details. The worst outcome here would be to end up with something that was equivalent to TOSCA but not easily translatable to the TOSCA Simple Profile YAML format (currently a Working Draft). Where 'easily translatable' preferably means 'by just changing some names'. I can't comment on whether this is the case as things stand. The TOSCA Simple Profile in YAML is a working draft at the moment, so we are pretty much open for any input. So let's see to get the right folks together and get it right. Since the Murano folks have indicated before that they are evaluating the option to join the OASIS TC, I am optimistic that we can get the streams together. Having implementation work going on here in this community in parallel to the standards work, and both streams inspiring each other, will be fun :-) Regards, Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
On 3/25/14 11:55 AM, Ruslan Kamaldinov rkamaldi...@mirantis.com wrote: * Murano DSL will focus on: a. UI rendering One of the primary reasons I am opposed to using a different DSL/project to accomplish this is that the person authoring the HOT template is usually the system architect, and this is the same person who has the technical knowledge to know what technologies you can swap in/out and still have that system/component work, so they are also the person who can/should define the rules of what component building blocks can and can't work together. There has been an overwhelmingly strong preference from the system architects/DevOps/ApplicationExperts I [1] have talked to for the ability to have control over those rules directly within the HOT file or immediately along-side the HOT file but feed the whole set of files to a single API endpoint. I'm not advocating that this extra stuff be part of Heat Engine (I understand the desire to keep the orchestration engine clean)... But from a barrier to adoption point-of-view, the extra effort for the HOT author to learn another DSL and use yet another system (or even have to write multiple files) should not be underestimated. These people are not OpenStack developers, they are DevOps folks and Application Experts. This is why the Htr[2] project was proposed and threads were started to add extra data to HOT template that Heat engine could essentially ignore, but would make defining UI rendering and component connectivity easy for the HOT author. I'm all for contributions to OpenStack, so I encourage the Murano team to continue doing its thing if they find it adds value to themselves or others. However, I'd like to see the Orchestration program support the surrounding things the users of the Heat engine want/need from their cloud system instead of having those needs met by separate projects seeking incubation. There are technical ways to keep the core engine clean while having the Orchestration Program API Service move up the stack in terms of cloud user experience. b. HOT generation c. Setup other services (like put Mistral tasks to Mistral and bind them with events) Speaking about new DSL for Murano. We're speaking about Application Lifecycle Management. There are a lot of existing tools - Heat/HOT, Python, etc, but none of them was designed with ALM in mind as a goal. Solum[3] is specifically designed for ALM and purpose built for OpenStack... It has declared that it will generate HOT templates and setup other services, including putting together or executing supplied workflow definition (using Mistral if applicable). Like Murano, Solum is also not an OpenStack incubated project, but it has been designed with community collaboration (based on shared pain across multiple contributors) with the ALM goal in mind from the very beginning. -Keith [1] I regularly speak with DevOps, Application Specialists, and cloud customers, specifically about Orchestration and Heat.. HOT is somewhat simple enough for the most technical of them (DevOps App Specialists) to grasp and have interest in adopting, but their is strong push back from the folks I talk to about having to learn one more thing... Since Heat adopters are exactly the same people who have the knowledge to define the overall system capabilities including component connectivity and how UI should be rendered, I'd like to keep it simple for them. The more we can do to have the Orchestration service look/feel like one thing (even if it's Engine + Other things under the hood), or reuse other OpenStack core components (e.g. Glance) the better for adoption. [2] https://wiki.openstack.org/wiki/Heat/htr [3] http://solum.io ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Hi Thomas, I think we went to the second loop of the discussion about generic language concepts. Murano does not use a new language for the sole purpose of having parameters, constraints and polymorphism. These are generic concepts which are common for different languages, so keeping arguing about these generic concepts is just like a holy war like Python vs. C. Keeping these arguments is just like to say that we don't need Python as functions and parameters already exists in C which is used under the hood in Python. Yes Murano DSL have some generic concepts similar to HOT. I think this is a benefit as user will see the familiar syntax constructions and it will be a lower threshold for him to start using Murano DSL. In a simplified view Murano uses DSL for application definition to solve several particular problems: a) control UI rendering of Application Catalog b) control HOT template generation These aspects are not covered in HOT and probably should not be covered. I don't like an idea of expressing HOT template generation in HOT as it sounds like a creation another Lisp like language :-) I don't think that your statement that most of the people in the community are against new DSL is a right summary. There are some disagreements how it should look like and what are the goals. You will be probably surprised but we are not the first who use DSL for HOT templates generation. Here is an e-mail thread right about Ruby based DSL used in IBM for the same purpose: http://lists.openstack.org/pipermail/openstack-dev/2014-February/026606.html The term Orchestration is quite generic. Saying that orchestration should be Heat job sounds like a well know Henry Ford's phrase You can have any colour as long as it's black.. I think this is again a lack of understanding of the difference between Orchestration program and Heat project. There are many aspects of Orchestration and OpenStack has the Orchestration program for the projects which are focused on some aspects of orchestration. Heat is one of the project inside Orchestration program but it does not mean that Heat should cover everything. That is why we discussed in this thread how workflows aspects should be aligned and how they should be placed into this Orchestration program. Thanks Georgy On Mon, Mar 24, 2014 at 8:28 AM, Dmitry mey...@gmail.com wrote: MuranoPL supposed to provide a solution for the real needs to manage services in the centralized manner and to allow cloud provider customers to create their own services. The application catalog similar to AppDirect (www.appdirect.com) natively supported by OpenStack is a huge step forward. Think about Amazon which provides different services for the different needs: Amazon Cloud Formation, Amazon OpsWorks and Amazon Beanstalk. Following the similar logic (which is fully makes sense for me), OpenStack should provide resource reservation and orchestration (Heat and Climate), Application Catalog (Murano) and PaaS (Solum). Every project can live in harmony with other and contribute for the cloud service provider service completeness. This is my opinion and i would happy to use Murano in our internal solution. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 On Mon, Mar 24, 2014 at 5:13 AM, Thomas Herve thomas.he...@enovance.comwrote: Hi Stan, Comments inline. Zane, I appreciate your explanations on Heat/HOT. This really makes sense. I didn't mean to say that MuranoPL is better for Heat. Actually HOT is good for Heat's mission. I completely acknowledge it. I've tried to avoid comparison between languages and I'm sorry if it felt that way. This is not productive as I don't offer you to replace HOT with MuranoPL (although I believe that certain elements of MuranoPL syntax can be contributed to HOT and be valuable addition there). Also people tend to protect what they have developed and invested into and to be fair this is what we did in this thread to great extent. What I'm trying to achieve is that you and the rest of Heat team understand why it was designed the way it is. I don't feel that Murano can become full-fledged member of OpenStack ecosystem without a bless from Heat team. And it would be even better if we agree on certain design, join our efforts and contribute to each other for sake of Orchestration program. Note that I feel that most people outside of the Murano project are against the idea of using a DSL. You should feel that it could block the integration in OpenStack. I'm sorry for long mail texts written in not-so-good English and appreciate you patience reading and answering them. Having said that let me step backward and explain our design
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
What I can say is that I'm not convinced. The only use-case for a DSL would be if you have to upload user-written code, but what you mentioned is a Web interface, where the user doesn't use the DSL, and the cloud provider is the developer. There is no reason in this case to have a secure environment for the code. I didn't say that. There are at least 2 different roles application developers/publishers and application users. Application developer is not necessary cloud provider. The whole point of AppCatalog is to support scenario when anyone can create and package some application and that package can be uploaded by user alone. Think Apple AppStore or Google Play. Some cloud providers may configure ACLs so that user be allowed to consume applications they decided while others may permit to upload applications to some configurable scope (e.g. apps that would be visible to all cloud users, to particular tenant or be private to the user). We also think to have some of peer relations so that it would be possible to have application upload in one catalog to become automatically available in all connected catalogs. This is similar to how Linux software repos work - AppCatalog is repo, Murano package is what DEB/RPMs are to repo and DSL is what DEB/RPMs manifests are to packages. Just that is run on cloud and designed to handle complex multi-node apps as well as trivial ones in which case this may be narrowed to actual installation of DEB/RPM I'm glad that you bring packages up. This is a really good example of why you don't need a new programming language. Packages uses whatever technology they prefer to handle their scripting needs. They then have an declarative interface which hides the imperative parts behind. You trust OpenStack developers with their code, you trust package developers with their code, why not trust catalog developers? -- Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Hi Thomas, I think we went to the second loop of the discussion about generic language concepts. Murano does not use a new language for the sole purpose of having parameters, constraints and polymorphism. These are generic concepts which are common for different languages, so keeping arguing about these generic concepts is just like a holy war like Python vs. C. Keeping these arguments is just like to say that we don't need Python as functions and parameters already exists in C which is used under the hood in Python. Yes Murano DSL have some generic concepts similar to HOT. I think this is a benefit as user will see the familiar syntax constructions and it will be a lower threshold for him to start using Murano DSL. In a simplified view Murano uses DSL for application definition to solve several particular problems: a) control UI rendering of Application Catalog b) control HOT template generation These aspects are not covered in HOT and probably should not be covered. I don't like an idea of expressing HOT template generation in HOT as it sounds like a creation another Lisp like language :-) I'm not saying that HOT will cover all your needs. I think it will cover a really good portion. And I'm saying that for the remaining part, you can use an existing language and not create a new one. I don't think that your statement that most of the people in the community are against new DSL is a right summary. There are some disagreements how it should look like and what are the goals. You will be probably surprised but we are not the first who use DSL for HOT templates generation. Here is an e-mail thread right about Ruby based DSL used in IBM for the same purpose: http://lists.openstack.org/pipermail/openstack-dev/2014-February/026606.html The term Orchestration is quite generic. Saying that orchestration should be Heat job sounds like a well know Henry Ford's phrase You can have any colour as long as it's black.. That worked okay for him :). I think this is again a lack of understanding of the difference between Orchestration program and Heat project. There are many aspects of Orchestration and OpenStack has the Orchestration program for the projects which are focused on some aspects of orchestration. Heat is one of the project inside Orchestration program but it does not mean that Heat should cover everything. That is why we discussed in this thread how workflows aspects should be aligned and how they should be placed into this Orchestration program. Well, today Heat is the one and only program in the Orchestration program. If and when you have orchestration needs not covered, we are there to make sure Heat is not the best place to handle them. The answer will probably not Heat forever, but we need good use cases to delegate those needs to another project. -- Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
On Tue, Mar 25, 2014 at 3:32 AM, Thomas Herve thomas.he...@enovance.comwrote: Hi Thomas, I think we went to the second loop of the discussion about generic language concepts. Murano does not use a new language for the sole purpose of having parameters, constraints and polymorphism. These are generic concepts which are common for different languages, so keeping arguing about these generic concepts is just like a holy war like Python vs. C. Keeping these arguments is just like to say that we don't need Python as functions and parameters already exists in C which is used under the hood in Python. Yes Murano DSL have some generic concepts similar to HOT. I think this is a benefit as user will see the familiar syntax constructions and it will be a lower threshold for him to start using Murano DSL. In a simplified view Murano uses DSL for application definition to solve several particular problems: a) control UI rendering of Application Catalog b) control HOT template generation These aspects are not covered in HOT and probably should not be covered. I don't like an idea of expressing HOT template generation in HOT as it sounds like a creation another Lisp like language :-) I'm not saying that HOT will cover all your needs. I think it will cover a really good portion. And I'm saying that for the remaining part, you can use an existing language and not create a new one. As a user can't run arbitrary python code in openstack we used Python language to create a new API for the remaining parts. This API service accepts a yaml based description of what should be done. There is no intention to create a new generic programming language. We used OpenStack approach and created a service for specific functions around Application Catalog features. Due to dynamic nature of applications we had to add a bit of dynamics to the service input just because of the same reason why Heat uses templates. I don't think that your statement that most of the people in the community are against new DSL is a right summary. There are some disagreements how it should look like and what are the goals. You will be probably surprised but we are not the first who use DSL for HOT templates generation. Here is an e-mail thread right about Ruby based DSL used in IBM for the same purpose: http://lists.openstack.org/pipermail/openstack-dev/2014-February/026606.html The term Orchestration is quite generic. Saying that orchestration should be Heat job sounds like a well know Henry Ford's phrase You can have any colour as long as it's black.. That worked okay for him :). Not really. The world acknowledged his inventions and new approaches. Other manufacturers adopted his ideas and moved forward, providing more variety, while Ford stuck with his model-T, which was very successful though. The history shows that variety won the battle over single approach and now we have different colors, shapes, engines :-) I think this is again a lack of understanding of the difference between Orchestration program and Heat project. There are many aspects of Orchestration and OpenStack has the Orchestration program for the projects which are focused on some aspects of orchestration. Heat is one of the project inside Orchestration program but it does not mean that Heat should cover everything. That is why we discussed in this thread how workflows aspects should be aligned and how they should be placed into this Orchestration program. Well, today Heat is the one and only program in the Orchestration program. If and when you have orchestration needs not covered, we are there to make sure Heat is not the best place to handle them. The answer will probably not Heat forever, but we need good use cases to delegate those needs to another project. That is exactly the reason why we have these discussions :-) We have the use cases for new functionality and we are trying to find a place for it. -- Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
On Tue, Mar 25, 2014 at 2:27 PM, Thomas Herve thomas.he...@enovance.comwrote: What I can say is that I'm not convinced. The only use-case for a DSL would be if you have to upload user-written code, but what you mentioned is a Web interface, where the user doesn't use the DSL, and the cloud provider is the developer. There is no reason in this case to have a secure environment for the code. I didn't say that. There are at least 2 different roles application developers/publishers and application users. Application developer is not necessary cloud provider. The whole point of AppCatalog is to support scenario when anyone can create and package some application and that package can be uploaded by user alone. Think Apple AppStore or Google Play. Some cloud providers may configure ACLs so that user be allowed to consume applications they decided while others may permit to upload applications to some configurable scope (e.g. apps that would be visible to all cloud users, to particular tenant or be private to the user). We also think to have some of peer relations so that it would be possible to have application upload in one catalog to become automatically available in all connected catalogs. This is similar to how Linux software repos work - AppCatalog is repo, Murano package is what DEB/RPMs are to repo and DSL is what DEB/RPMs manifests are to packages. Just that is run on cloud and designed to handle complex multi-node apps as well as trivial ones in which case this may be narrowed to actual installation of DEB/RPM I'm glad that you bring packages up. This is a really good example of why you don't need a new programming language. Packages uses whatever technology they prefer to handle their scripting needs. They then have an declarative interface which hides the imperative parts behind. The same is true for Murano. MuranoPL is not used to express what should be deployed. In Murano there is object model that describes view of the word. It serves for the same purpose as HOT in Heat but it is simpler because it says just what need to be deployed but not how it should be accomplished as this information is already contained in application definitions. There is REST API to edit/submit Object Model which is again has nothing to do with MuranoPL. UI dashboard talks to AppCatalog to see what applications/classes are available and AppCatalog also knows what are the properties of those classes. This is needed for UI so that it can ask user for appropriate input. This is similar to how Horizon asks user to input parameters that are declared in HOT template. But all the imperative stuff is hidden inside Murano packages and is not used for anything outside Murano engine. MuranoPL is not a replacement for scripting languages. You still use bash/puppet/PowerShell/whatever you like for actual deployment. No MuranoPL code is executed on VM side. So the analogy with RPMs manifests is valid. You trust OpenStack developers with their code, you trust package developers with their code, why not trust catalog developers? They do trust catalog developers (hopefully). But catalog developers have nothing to do with catalog contents. Anyone can create and upload application to App Catalog the same way how anyone can upload his application to Google Play. The fact that I trust Google doesn't mean that I trust all applications in Google Play. The fact that I trust catalog developers doesn't mean that I (as a cloud operator) is going to allow execution of untrusted code in that catalog. Unless that code is sandboxed by design. Similar to that I can navigate to any web site google points me out and let it execute any JavaScript it wants unless it it JavaScript and not browser plugin or desktop application. -- Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Murano folks, I guess we should stop our debates with the Heat team about MuranoPL. What we should do instead is to carefully read this thread couple of times and collect and process all the feedback we've got. Steve and Zane did a very good job helping us to find a way to align with Heat and Orhcestration program. Let me outline the most important (imho) quotes: # START quotes Steven Dake: I see no issue with HOT remaining simple and tidy focused entirely on orchestration (taking a desired state and converting that into reality) with some other imperative language layered on top to handle workflow and ALM. I believe this separation of concerns is best for OpenStack and should be the preferred development path. Zane Bitter: I do think we should implement the hooks I mentioned at the start of this thread to allow tighter integration between Heat and a workflow engine (i.e. Mistral). So building a system on top of Heat is good. Building it on top of Mistral as well would also be good, and that was part of the feedback from the TC. To me, building on top means building on top of the languages (which users will have to invest a lot of work in learning) as well, rather than having a completely different language and only using the underlying implementation(s). To me that implies that Murano should be a relatively thin wrapper that ties together HOT and Mistral's DSL. Steve Dake: --- I don't think HOT can do these things and I don't think we want HOT to do these things. I am ok with that, since I don't see the pushback on having two languages for two different things in OpenStack. I got from gokrove on iRC today that the rationale for the pushback was the TC wanted Murano folks to explore how to integrate better with Heat and possibly the orchestration program. I don't see HOT as a place where there is an opportunity for scope expansion. I see instead Murano creating HOT blobs and feeding them to Heat. Zane Bitter: Because there have to be some base types that you can use as building blocks. In the case of Heat, those base types are the set of things that you can create by talking to OpenStack APIs authenticated with the user's token. In the case of Mistral, I would expect it to be the set of actions that you can take by talking to OpenStack APIs authenticated with the user's token. And in the case of Murano, I would expect it to be the union of those two. Everything is a combination of existing resources, because the set of existing resources is the set of things which the operator provides as-a-Service. The set of things that the operator provides as a service plus the set of things that you can implement yourself on your own server (virtual or not) covers the entire universe of things. What you appear to be suggesting is that OpenStack must provide *Everything*-as-a-Service by allowing users to write their own services and have the operator execute them as-a-Service. This would be a breathtakingly ambitious undertaking, and I don't mean that in a good way. When the TC said Murano is slightly too far up the stack at this point to meet the measured progression of openstack as a whole requirement, IMO one of the major things they meant was that you're inventing your own workflow thing, leading to duplication of effort between this and Workflow as a Service. (And Mistral folks are in turn doing the same thing by not using the same workflow library, taskflow, as the rest of OpenStack.) Candidly, I'm surprised that this is not the #1 thing on your priority list because IMO it's the #1 thing that will delay getting the project incubated. # END quotes Also I should quote what Georgy Okrokvertkhov said: Having this aligned I see a Murano package as an archive with all necessary definitions and resources and Murano service will just properly pass them to related services like Heat and Mistral. I think then Murano DSL will be much more simple and probably will be closer to declarative format with some specific data operations. Summary: I would like to propose further path for Murano evolution where it evolves as an OpenStack project, aligned with others and developed in agreement between all the interested sides. Here is the plan: * Step forward and implement hooks (for workflow customization) in Heat. We will register a blueprint, discuss it with the Heat team and implement it * Use our cross-project session on ATL summit to set clear goals and expectations * Build pilot in Murano which: a. Uses new HOT Software components to do VM side software deployments b. Uses Mistral DSL to describe workflow. It'll require more focused discussion with Mistral team c. Bundles all that into an application package using new Murano DSL d. Allows users to combine applications in a single environenment * Continuously align with the Heat team * Continuously re-evaluate current state of pilot
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
On 21/03/14 18:58, Stan Lagun wrote: Zane, I appreciate your explanations on Heat/HOT. This really makes sense. I didn't mean to say that MuranoPL is better for Heat. Actually HOT is good for Heat's mission. I completely acknowledge it. I've tried to avoid comparison between languages and I'm sorry if it felt that way. This is not productive as I don't offer you to replace No problem, I didn't interpret it that way. I just wanted to use it as an opportunity to document a bit more information about Heat works that may be helpful to you. HOT with MuranoPL (although I believe that certain elements of MuranoPL syntax can be contributed to HOT and be valuable addition there). Also people tend to protect what they have developed and invested into and to be fair this is what we did in this thread to great extent. Yes, we are all human and this is an unavoidable part of life :) What I'm trying to achieve is that you and the rest of Heat team understand why it was designed the way it is. I don't feel that Murano can become full-fledged member of OpenStack ecosystem without a bless from Heat team. And it would be even better if we agree on certain design, join our efforts and contribute to each other for sake of Orchestration program. Thanks. To be clear, this is what I am looking for. At the beginning of this thread I proposed a very simple possible implementation, and I'm wanting folks to tell me what is missing from that model that would justify a more complicated approach. I'm sorry for long mail texts written in not-so-good English and appreciate you patience reading and answering them. Having said that let me step backward and explain our design decisions. Cloud administrators are usually technical guys that are capable of learning HOT and writing YAML templates. They know exact configuration of their cloud (what services are available, what is the version of OpenStack cloud is running) and generally understands how OpenStack works. They also know about software they intent to install. If such guy wants to install Drupal he knows exactly that he needs HOT template describing Fedora VM with Apache + PHP + MySQL + Drupal itself. It is not a problem for him to write such HOT template. I'm aware that TOSCA has these types of constraints, and in fact I suggested to the TOSCA TC that maybe this is where we should draw the line between Heat and some TOSCA-compatible service: HOT should be a concrete description of exactly what you're going to get, whereas some other service (in this case Murano) would act as the constraints solver. e.g. something like an image name would not be hardcoded in a Murano template, you have some constraints about which operating system and what versions should be allowed, and it would pick one and pass it to Heat. So I am interested in this approach. The worst outcome here would be to end up with something that was equivalent to TOSCA but not easily translatable to the TOSCA Simple Profile YAML format (currently a Working Draft). Where 'easily translatable' preferably means 'by just changing some names'. I can't comment on whether this is the case as things stand. Note that such template would be designed for very particular configuration. There are hundreds of combinations that may be used to install that Drupal - use RHEL/Windows/etc instead of Fedora, use ngnix/IIS/etc instead of Apache, use FastCGI instead of mod_php, PostgreSQL instead of MySQL. You may choose to have all software on single VM or have one VM for database and another for Drupal. There are also constraints to those combinations. For example you cannot have Fedora + IIS on the same VM. You cannot have Apache and Drupal on different VMs. So the HOT template represent fixed combination of those software components. HOT may have input parameters like username or dbImageName but the overall structure of template is fixed. You cannot have template that choses whether to use Windows or Linux based on parameter value. As Thomas mentioned, CloudFormation now has conditionals. If we were to add this feature in HOT (which seems likely), you would actually be able to do that. http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html You cannot write HOT that accepts number of instances it allowed to create and then decide what would be installed on each of them. This is just not needed for Heat users. The path we've gone down in Heat is to introduce software component resources so that the definition of the software component to install is separated from the definition of the server it runs on, and may even be in a separate template file. So you can write a template that launches a single server with (e.g.) both Wordpress and MySQL installed, or you can have a separate template (though with the conditional stuff above it could theoretically be the same template even) that launches two servers with Wordpress on one and MySQL on the
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Hi Stan, Comments inline. Zane, I appreciate your explanations on Heat/HOT. This really makes sense. I didn't mean to say that MuranoPL is better for Heat. Actually HOT is good for Heat's mission. I completely acknowledge it. I've tried to avoid comparison between languages and I'm sorry if it felt that way. This is not productive as I don't offer you to replace HOT with MuranoPL (although I believe that certain elements of MuranoPL syntax can be contributed to HOT and be valuable addition there). Also people tend to protect what they have developed and invested into and to be fair this is what we did in this thread to great extent. What I'm trying to achieve is that you and the rest of Heat team understand why it was designed the way it is. I don't feel that Murano can become full-fledged member of OpenStack ecosystem without a bless from Heat team. And it would be even better if we agree on certain design, join our efforts and contribute to each other for sake of Orchestration program. Note that I feel that most people outside of the Murano project are against the idea of using a DSL. You should feel that it could block the integration in OpenStack. I'm sorry for long mail texts written in not-so-good English and appreciate you patience reading and answering them. Having said that let me step backward and explain our design decisions. Cloud administrators are usually technical guys that are capable of learning HOT and writing YAML templates. They know exact configuration of their cloud (what services are available, what is the version of OpenStack cloud is running) and generally understands how OpenStack works. They also know about software they intent to install. If such guy wants to install Drupal he knows exactly that he needs HOT template describing Fedora VM with Apache + PHP + MySQL + Drupal itself. It is not a problem for him to write such HOT template. Note that such template would be designed for very particular configuration. There are hundreds of combinations that may be used to install that Drupal - use RHEL/Windows/etc instead of Fedora, use ngnix/IIS/etc instead of Apache, use FastCGI instead of mod_php, PostgreSQL instead of MySQL. You may choose to have all software on single VM or have one VM for database and another for Drupal. There are also constraints to those combinations. For example you cannot have Fedora + IIS on the same VM. You cannot have Apache and Drupal on different VMs. So the HOT template represent fixed combination of those software components. HOT may have input parameters like username or dbImageName but the overall structure of template is fixed. You cannot have template that choses whether to use Windows or Linux based on parameter value. You cannot write HOT that accepts number of instances it allowed to create and then decide what would be installed on each of them. This is just not needed for Heat users. With Murano the picture looks the opposite. Typical Murano user is a guy who bought account from cloud hosting vendor (cloud operator) and want to run some software in the cloud. He may not even be aware that it is OpenStack. He knows nothing about programming in general and Heat in particular. He doesn't want to write YAMLs. He may not know how exactly Drupal is installed and what components it consists of. So that's where I want to make a first stop. If your primary user is not a developer, there is no reason to introduce a DSL for security reasons. The provider can trust the code he writes, and there is no need to create a dedicated language. So what he does is he goes to his cloud (Murano) dashboard, browses through application catalog, finds Drupal and drags it onto his environment board (think like Visio-style designer). He can stop at this point, click deploy button and the system will deploy Drupal. In another words the system (or maybe better to say cloud operator or application developer) decides what set of components is going to be installed (like 1 Fedora VM for MySQL and 1 CentOS VM for Apache-PHP-Drupal). But user may decide he wants to customize his environment. He digs down and sees that Drupal requires database instance and the default is MySQL. He clicks on a button to see what are other options available for that role. In Heat HOT developer is the user. But in Murano those are completely different roles. There are developers that write application definitions (that is DSL code) and there are end users who compose environments from those applications (components). Application developers may have nothing to do with particular cloud their application deployed on. As for Drupal application the developer knows that Drupal can be run with MySQL or PostgreSQL. But there may be many compatible implementations of those DBMSes - Galera MySQL, TroveMySQL, MMM MySQL etc. So to get a list of what components can be placed in a database role Murano needs to look at
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Hi, So that's where I want to make a first stop. If your primary user is not a developer, there is no reason to introduce a DSL for security reasons. The provider can trust the code he writes, and there is no need to create a dedicated language. I thinks this need to be clarified. The provider does not write code: the provider just hosts the cloud, acting as a moderator. The code is written by another category of end-users, called Application Publishers. This category is untrusted - that's the nature of Application Catalog: anybody can upload everything. The publishers (who write DSL) should not be confused with users (who define object models using it). These are different roles. -- Regards, Alexander Tivelkov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
On Mon, Mar 24, 2014 at 4:13 PM, Thomas Herve thomas.he...@enovance.comwrote: What I can say is that I'm not convinced. The only use-case for a DSL would be if you have to upload user-written code, but what you mentioned is a Web interface, where the user doesn't use the DSL, and the cloud provider is the developer. There is no reason in this case to have a secure environment for the code. I didn't say that. There are at least 2 different roles application developers/publishers and application users. Application developer is not necessary cloud provider. The whole point of AppCatalog is to support scenario when anyone can create and package some application and that package can be uploaded by user alone. Think Apple AppStore or Google Play. Some cloud providers may configure ACLs so that user be allowed to consume applications they decided while others may permit to upload applications to some configurable scope (e.g. apps that would be visible to all cloud users, to particular tenant or be private to the user). We also think to have some of peer relations so that it would be possible to have application upload in one catalog to become automatically available in all connected catalogs. This is similar to how Linux software repos work - AppCatalog is repo, Murano package is what DEB/RPMs are to repo and DSL is what DEB/RPMs manifests are to packages. Just that is run on cloud and designed to handle complex multi-node apps as well as trivial ones in which case this may be narrowed to actual installation of DEB/RPM -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
I completely agree with Georgy, but you raised some questions about Heat
that I want to answer in the interests of spreading knowledge about how
Heat works. A heavily-snipped response follows...
On 21/03/14 05:11, Stan Lagun wrote:
3. Despite HOT being more secure on the surface it is not necessary so
in reality. There is a Python class behind each entry in resources
section of HOT template. That Python code is run with root privileges
and not guaranteed to be safe. People make mistakes, forget to validate
parameters, make incorrect assumptions etc. Even if the code is proven
to be secure every single commit can introduce security breach. And no
testing system can detect this.
Quite right, I should acknowledge that it would be crazy to assume that
HOT is secure just because it is not a programming language, and I do
not make that assumption. (Indeed, YAML itself has been the subject of
many security problems, though afaik not in the safe mode that we use in
Heat.) Thanks for pointing out that I was not clear.
The operator can install whatever plugins they want.
They do but that is a bad solution. The reason is that plugins can
introduce additional resource types but they cannot modify existing
code. Most of the time cloud operators need to customize existing
resources' logic for their needs rather then rewriting it from scratch.
And they want their changes to be opaque to end-users. Imagine that
cloud operator need thats to get permission from his proprietary quota
management system for each VM spawned. If he would create custom
MyInstance resource type end-users could bypass it by using standard
Instance resource rather than custom one. Patching existing Python code
is not good in that then operator need to maintain his private fork of
the Heat and have troubles with CD, upgrades to newer versions etc.
It's not as bad as you think. All of the things you mentioned were
explicit design goals of the plugin system. If you install a plug-in
resource with the same type as a built-in resource then it replaces the
built-in one. And of course you can inherit from the existing plugin to
customise it.
So in this example, the operator would create a plugin like this:
from heat.engine.resources import server
from my.package import do_my_proprietary_quota_thing
class MyServer(server.Server):
def handle_create(self):
do_my_proprietary_quota_thing()
return super(MyServer, self).handle_create()
def resource_mapping():
return {'OS::Nova::Server': MyServer}
and drop it in /usr/lib/heat. As you can see, this is a simple
customisation (10 lines of code), completely opaque to end users
(OS::Nova::Server is replaced), and highly unlikely to be broken by any
changes in Heat (we consider the public APIs of
heat.engine.resource.Resource as a contract with existing plugins that
we can't break, at least without a lot of notice).
(I'm ignoring here that if this is needed for _every_ server, it makes
no sense to do it in Heat, unless you don't expose the Nova API to users
at all.)
Besides plugin system is not secure because plugins run with the
privileges of Heat engine and while I may trust Heat developers
(community) but not necessary trust 3rd party proprietary plugin.
I'm not sure who 'I' means in this context? As an end-user, you have no
way of auditing what code your cloud provider is running in general.
What
if he wants that auto-scaling would be based on input from his
existing
Nagios infrastructure rather then Ceilometer?
This is supported already in autoscaling. Ceilometer just hits a URL
for an alarm, but you don't have to configure it this way. Anything
can hit the URL.
And this is a good example for our general approach - we provide a
way that works using built-in OpenStack services and a hook that
allows you to customise it with your own service, running on your
own machine (whether that be an actual machine or an OpenStack
Compute server). What we *don't* do is provide a way to upload your
own code that we then execute for you as some sort of secondary
Compute service.
1. Anything can hit the URL but it is auto-scaling resource that creates
Ceilometer alarms. And what should I do to make it create Nagios alarms
for example?
That's incorrect, autoscaling doesn't create any alarms. You create an
alarm explicitly using the Ceilometer API, or using an
OS::Ceilometer::Alarm resource in Heat. Or not, if you want to use some
other source for alarms. You connect them together by getting the
alarm_url attribute from the autoscaling policy resource and passing it
to the Ceilometer alarm, but you could also allow any alarm source you
care to use to hit that URL.
[Ceilometer] [Heat]
Metrics --- Alarm - - - - - Policy --- Scaling Group
^
(webhook)
A second option is that you can
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Zane, I appreciate your explanations on Heat/HOT. This really makes sense. I didn't mean to say that MuranoPL is better for Heat. Actually HOT is good for Heat's mission. I completely acknowledge it. I've tried to avoid comparison between languages and I'm sorry if it felt that way. This is not productive as I don't offer you to replace HOT with MuranoPL (although I believe that certain elements of MuranoPL syntax can be contributed to HOT and be valuable addition there). Also people tend to protect what they have developed and invested into and to be fair this is what we did in this thread to great extent. What I'm trying to achieve is that you and the rest of Heat team understand why it was designed the way it is. I don't feel that Murano can become full-fledged member of OpenStack ecosystem without a bless from Heat team. And it would be even better if we agree on certain design, join our efforts and contribute to each other for sake of Orchestration program. I'm sorry for long mail texts written in not-so-good English and appreciate you patience reading and answering them. Having said that let me step backward and explain our design decisions. Cloud administrators are usually technical guys that are capable of learning HOT and writing YAML templates. They know exact configuration of their cloud (what services are available, what is the version of OpenStack cloud is running) and generally understands how OpenStack works. They also know about software they intent to install. If such guy wants to install Drupal he knows exactly that he needs HOT template describing Fedora VM with Apache + PHP + MySQL + Drupal itself. It is not a problem for him to write such HOT template. Note that such template would be designed for very particular configuration. There are hundreds of combinations that may be used to install that Drupal - use RHEL/Windows/etc instead of Fedora, use ngnix/IIS/etc instead of Apache, use FastCGI instead of mod_php, PostgreSQL instead of MySQL. You may choose to have all software on single VM or have one VM for database and another for Drupal. There are also constraints to those combinations. For example you cannot have Fedora + IIS on the same VM. You cannot have Apache and Drupal on different VMs. So the HOT template represent fixed combination of those software components. HOT may have input parameters like username or dbImageName but the overall structure of template is fixed. You cannot have template that choses whether to use Windows or Linux based on parameter value. You cannot write HOT that accepts number of instances it allowed to create and then decide what would be installed on each of them. This is just not needed for Heat users. With Murano the picture looks the opposite. Typical Murano user is a guy who bought account from cloud hosting vendor (cloud operator) and want to run some software in the cloud. He may not even be aware that it is OpenStack. He knows nothing about programming in general and Heat in particular. He doesn't want to write YAMLs. He may not know how exactly Drupal is installed and what components it consists of. So what he does is he goes to his cloud (Murano) dashboard, browses through application catalog, finds Drupal and drags it onto his environment board (think like Visio-style designer). He can stop at this point, click deploy button and the system will deploy Drupal. In another words the system (or maybe better to say cloud operator or application developer) decides what set of components is going to be installed (like 1 Fedora VM for MySQL and 1 CentOS VM for Apache-PHP-Drupal). But user may decide he wants to customize his environment. He digs down and sees that Drupal requires database instance and the default is MySQL. He clicks on a button to see what are other options available for that role. In Heat HOT developer is the user. But in Murano those are completely different roles. There are developers that write application definitions (that is DSL code) and there are end users who compose environments from those applications (components). Application developers may have nothing to do with particular cloud their application deployed on. As for Drupal application the developer knows that Drupal can be run with MySQL or PostgreSQL. But there may be many compatible implementations of those DBMSes - Galera MySQL, TroveMySQL, MMM MySQL etc. So to get a list of what components can be placed in a database role Murano needs to look at all applications in Application Catalog and find which of them are compatible with MySQL and PostgreSQL so that user could choose what implementation is better suits his needs (trade performance for high-availability etc.). User can go deeper and to decide that he wants that MySQL instance (this can be 1 or more VMs depending on implementation) to be shared between Drupal and another application in that environment (say WordPress). He can go even deeper to VM level and decide that he wants to have WordPress,
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
On 19/03/14 17:13, Stan Lagun wrote: On Wed, Mar 19, 2014 at 9:18 PM, Zane Bitter zbit...@redhat.com mailto:zbit...@redhat.com wrote: On 19/03/14 05:00, Stan Lagun wrote: Steven, Agree with your opinion on HOT expansion. I see that inclusion of imperative workflows and ALM would require major Heat redesign and probably would be impossible without loosing compatibility with previous HOT syntax. It would blur Heat mission, confuse current users and rise a lot of questions what should and what should not be in Heat. Thats why we chose to built a system on top of Heat rather then expending HOT. +1, I agree (as we have discussed before) that it would be a mistake to shoehorn workflow stuff into Heat. I do think we should implement the hooks I mentioned at the start of this thread to allow tighter integration between Heat and a workflow engine (i.e. Mistral). So building a system on top of Heat is good. Building it on top of Mistral as well would also be good, and that was part of the feedback from the TC. To me, building on top means building on top of the languages (which users will have to invest a lot of work in learning) as well, rather than having a completely different language and only using the underlying implementation(s). 1. Murano application developers (publishers) are using Heat directly. It is not the case that Murano executes some HOT templates under the hood but HOT templates are essential part of application definition. Developers still write HOT templates in HOT syntax 2. Please get me right here. It is not that we wanted to develop another language just for fun or to stand out from community. It is not that we wrote DSL and then started to think how we gonna use it and prove to others that is needed. We completely understand all concerns regarding new language. The only reason is that we had very concrete list of problems and use cases that we wanted to address in Murano. We did investigate on using HOT, Mistral, JavaScript, Lua and BPEL and we found overwhelming obstacles with each of those approaches. It don't pretend that having own DSL is good. Just that not having it is much worse. I'm also not a HOT expert as you are and thus can be (partially) wrong about HOT and not aware of some of its power features. If so as a technical guys we would quickly come to consensus. Now I would like to clarify why have we chosen imperative approach with DSL. You see a DSL as an alternative to HOT but it is not. DSL is alternative to Python-encoded resources in Heat (heat/engine/resources/*.py). Imagine how Heat would look like if you let untrusted users to upload Python plugins to Heat engine and load them on the fly. Heat resources are written in Python which is imperative language. So that MuranoPL for the same reason. We had this exact problem in Heat, and we decided to solve it with... HOT: http://lists.openstack.org/__pipermail/openstack-dev/2013-__April/007989.html http://lists.openstack.org/pipermail/openstack-dev/2013-April/007989.html If I may be so bold as to quote myself: ...no cloud operator in the world - not even your friendly local IT department - is going to let users upload Python code to run in-memory in their orchestration engine along with all of the other users' code. If only there were some sort of language for defining OpenStack services that could be safely executed by users... Of course that's exactly what we're all about on this project :). So my proposal is to allow users to define their own resource types using a Heat template. Excellent quotes! I can sign under each of them. No one would ever allow to upload Python code so we haven't used Python (and it is not the only reason not to use it) There's a fine line between a domain-specific programming language (as opposed to a domain-specific modelling language like HOT) and a general-purpose programming language. The reason no-one would run untrusted Python code is that Python is a programming language, not that it is general-purpose. If only there were some sort of language for defining OpenStack services that could be safely executed by users... - we created such language. And we found we already had one, that we successfully re-used for the purpose :) It is exactly what you wanted to have then and now when it exists you say it is useless. How this ironic. I'm not saying it's useless; I'm saying that it's *too* powerful for the job. It's clear that you guys have done amazing work and I don't want to criticise it, especially after having become aware of it only very late, but I also have serious reservations as to whether this is the right solution for OpenStack. It may be conceitedly to assume that
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Hi Zane, Thank you for the deep analysis. I agree that doing these deep technical debates around languages hides the real problem we should focus on. Thank you for bringing back to the light. You are right, TC highlighted that Murano did a huge step forward up to stack probably passing some important areas which should be covered in OpenStack before that. Workflows area is one of them. I think we need bring Mistral team here to find out how to align this workflows stuff with all three project Heat, Murano and Mistral. As Heat also wants to have a hooks mechanism to work with external workflow system and Murano uses Heat and workflows I think it will a lot of sense to design this properly together. It makes sense to bring TOSCA guys too as they have similar concepts in standard and Heat eventually will support TOSCA format. Having this aligned I see a Murano package as an archive with all necessary definitions and resources and Murano service will just properly pass them to related services like Heat and Mistral. I think then Murano DSL will be much more simple and probably will be closer to declarative format with some specific data operations. What will be the best way to do this? I know that Heat team is pretty busy now with Icehouse release which will happen pretty soon. I would like to have some draft before summit in Atlanta. We can polish them on the summit sessions but before some home work it will be hard. Thanks Georgy On Thu, Mar 20, 2014 at 5:28 PM, Zane Bitter zbit...@redhat.com wrote: On 19/03/14 17:13, Stan Lagun wrote: On Wed, Mar 19, 2014 at 9:18 PM, Zane Bitter zbit...@redhat.com mailto:zbit...@redhat.com wrote: On 19/03/14 05:00, Stan Lagun wrote: Steven, Agree with your opinion on HOT expansion. I see that inclusion of imperative workflows and ALM would require major Heat redesign and probably would be impossible without loosing compatibility with previous HOT syntax. It would blur Heat mission, confuse current users and rise a lot of questions what should and what should not be in Heat. Thats why we chose to built a system on top of Heat rather then expending HOT. +1, I agree (as we have discussed before) that it would be a mistake to shoehorn workflow stuff into Heat. I do think we should implement the hooks I mentioned at the start of this thread to allow tighter integration between Heat and a workflow engine (i.e. Mistral). So building a system on top of Heat is good. Building it on top of Mistral as well would also be good, and that was part of the feedback from the TC. To me, building on top means building on top of the languages (which users will have to invest a lot of work in learning) as well, rather than having a completely different language and only using the underlying implementation(s). 1. Murano application developers (publishers) are using Heat directly. It is not the case that Murano executes some HOT templates under the hood but HOT templates are essential part of application definition. Developers still write HOT templates in HOT syntax 2. Please get me right here. It is not that we wanted to develop another language just for fun or to stand out from community. It is not that we wrote DSL and then started to think how we gonna use it and prove to others that is needed. We completely understand all concerns regarding new language. The only reason is that we had very concrete list of problems and use cases that we wanted to address in Murano. We did investigate on using HOT, Mistral, JavaScript, Lua and BPEL and we found overwhelming obstacles with each of those approaches. It don't pretend that having own DSL is good. Just that not having it is much worse. I'm also not a HOT expert as you are and thus can be (partially) wrong about HOT and not aware of some of its power features. If so as a technical guys we would quickly come to consensus. Now I would like to clarify why have we chosen imperative approach with DSL. You see a DSL as an alternative to HOT but it is not. DSL is alternative to Python-encoded resources in Heat (heat/engine/resources/*.py). Imagine how Heat would look like if you let untrusted users to upload Python plugins to Heat engine and load them on the fly. Heat resources are written in Python which is imperative language. So that MuranoPL for the same reason. We had this exact problem in Heat, and we decided to solve it with... HOT: http://lists.openstack.org/__pipermail/openstack-dev/2013-_ _April/007989.html http://lists.openstack.org/pipermail/openstack-dev/2013- April/007989.html If I may be so bold as to quote myself: ...no cloud operator in the world - not even your friendly local IT department - is
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
: Georgy, In consideration of the can you express it instead of the who will generate it, I see Heat's HOT evolving to support the expression of complex multi-tier architectures and applications (I would argue you can already do this today, perhaps with some additional features desired, e.g. Ability to define cloud workflows and workflow execution rules which could come when we have a workflow service like Mistral). Therefore, I would encourage Murano contributors to consider whether they can help make Heat sufficiently cover desired use cases. I have never viewed Heat templates as isolated components of a multi-tier architecture. Instead, a single template or a combination of master/subordinate templates together (using references, nesting, or inclusion) could express the complete architecture, both infrastructure and applications. If I've read your previous comments and threads correctly, you desire a way to express System Lifecycle Management across multiple related applications or components, whereby you view the System as a grouping of independently developed and/or deployed (but systematically related) components, whereby you view Components as individual disconnected Heat templates that independently describe different application stacks of the System. Did I get that correct? If so, perhaps the discussion here is one of scope of what can or should be expressed in a Heat template. Is it correct to state that your argument is that a separate system (such as Murano) should be used to express System Lifecycle Management as I've defined it here? If so, why could we not use the Heat DSL to also define the System? The System definition could be logically separated out into its own text file... But, we'd have a common DSL syntax and semantics for both lower level and higher level component interaction (a building block effect of sorts). As for who will generate it, ( with it being the Heat multi-tier application/infrastructure definition) I think that question will go through a lot more evolution and could be any number of sources: e.g. Solum, Murano, Horizon, Template Author with a text editor, etc. Basically, I'm a +1 for as few DSLs as possible. I support the position that we should evolve HOT if needed vs. having two separate DSLs that are both related to expressing application and infrastructure semantics. Workflow is quite interesting ... Should we be able to express imperative workflow semantics in HOT? Or, should we only be able to declare workflow configurations that get configured in a service like Mistral whereby Mistral's execution of a workflow may need to invoke Heat hooks or Stack Updates? Or, some other solution? I look forward to a design discussion on all this at the summit... This is fun stuff to think about! -Keith From: Georgy Okrokvertskhov gokrokvertsk...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Tuesday, March 18, 2014 1:49 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? I see this in the following way - who will generate HOT template for my complex multi-tier applications when I have only templates for components? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
On 19 Mar 2014, at 16:00, Stan Lagun sla...@mirantis.com wrote: We want application authors to be able to express application deployment and maintenance logic of any complexity. This may involve communication with 3rd party REST services (APIs of applications being deployed, external services like DNS server API, application licensing server API, billing systems, some hardware component APIs etc) and internal OpenStack services like Trove, Sahara, Marconi and others including those that are not incubated yet and those to come in the future. You cannot have such things in HOT and when you required to you need to develop custom resource in Python. Independence on custom plugins is not good for Murano because they cannot be uploaded by end users and thus he cannot write application definition that can be imported to/run on any cloud and need to convince cloud administrator to install his Python plugin (something that is unimaginable in real life). +1. Makes perfect sense to me. Because DSL is a way to write custom resources (in Heats terminology) it has to be Turing-complete and have all the characteristics of general-purpose language. It also has to have domain-specific features because we cannot expect that DSL users would be as skilled as Heat developers and could write such resources without knowledge on hosting engine architecture and internals. +1 Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
is that a separate system (such as Murano) should be used to express System Lifecycle Management as I've defined it here? If so, why could we not use the Heat DSL to also define the System? The System definition could be logically separated out into its own text file... But, we'd have a common DSL syntax and semantics for both lower level and higher level component interaction (a building block effect of sorts). As for who will generate it, ( with it being the Heat multi-tier application/infrastructure definition) I think that question will go through a lot more evolution and could be any number of sources: e.g. Solum, Murano, Horizon, Template Author with a text editor, etc. Basically, I'm a +1 for as few DSLs as possible. I support the position that we should evolve HOT if needed vs. having two separate DSLs that are both related to expressing application and infrastructure semantics. Workflow is quite interesting ... Should we be able to express imperative workflow semantics in HOT? Or, should we only be able to declare workflow configurations that get configured in a service like Mistral whereby Mistral's execution of a workflow may need to invoke Heat hooks or Stack Updates? Or, some other solution? I look forward to a design discussion on all this at the summit... This is fun stuff to think about! -Keith From: Georgy Okrokvertskhov gokrokvertsk...@mirantis.com mailto:gokrokvertsk...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.__openstack.org mailto:openstack-dev@lists.openstack.org Date: Tuesday, March 18, 2014 1:49 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.__openstack.org mailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? I see this in the following way - who will generate HOT template for my complex multi-tier applications when I have only templates for components? _ OpenStack-dev mailing list OpenStack-dev@lists.openstack.__org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _ OpenStack-dev mailing list OpenStack-dev@lists.openstack.__org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _ OpenStack-dev mailing list OpenStack-dev@lists.openstack.__org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com http://www.mirantis.com/ sla...@mirantis.com mailto:sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
effect of sorts). As for who will generate it, ( with it being the Heat multi-tier application/infrastructure definition) I think that question will go through a lot more evolution and could be any number of sources: e.g. Solum, Murano, Horizon, Template Author with a text editor, etc. Basically, I'm a +1 for as few DSLs as possible. I support the position that we should evolve HOT if needed vs. having two separate DSLs that are both related to expressing application and infrastructure semantics. Workflow is quite interesting ... Should we be able to express imperative workflow semantics in HOT? Or, should we only be able to declare workflow configurations that get configured in a service like Mistral whereby Mistral's execution of a workflow may need to invoke Heat hooks or Stack Updates? Or, some other solution? I look forward to a design discussion on all this at the summit... This is fun stuff to think about! -Keith From: Georgy Okrokvertskhov gokrokvertsk...@mirantis.com mailto:gokrokvertsk...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org mailto:openstack-dev@lists.openstack.org Date: Tuesday, March 18, 2014 1:49 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org mailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? I see this in the following way - who will generate HOT template for my complex multi-tier applications when I have only templates for components? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com http://www.mirantis.com/ sla...@mirantis.com mailto:sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Excerpts from Zane Bitter's message on 19/03/2014 18:18:34: From: Zane Bitter zbit...@redhat.com To: openstack-dev@lists.openstack.org Date: 19/03/2014 18:21 Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? On 19/03/14 05:00, Stan Lagun wrote: Steven, Agree with your opinion on HOT expansion. I see that inclusion of imperative workflows and ALM would require major Heat redesign and probably would be impossible without loosing compatibility with previous HOT syntax. It would blur Heat mission, confuse current users and rise a lot of questions what should and what should not be in Heat. Thats why we chose to built a system on top of Heat rather then expending HOT. +1, I agree (as we have discussed before) that it would be a mistake to shoehorn workflow stuff into Heat. I do think we should implement the hooks I mentioned at the start of this thread to allow tighter integration between Heat and a workflow engine (i.e. Mistral). +1 on not putting workflow stuff into Heat. Rather let's come up with a nice way of Heat and a workflow service to work together. That could be done in two ways: (1) let Heat hand off to a workflow service for certains tasks or (2) let people define workflow tasks that can easily work on Heat deployed resources. Maybe both make sense, but right now I am more leaning towards (2). So building a system on top of Heat is good. Building it on top of Mistral as well would also be good, and that was part of the feedback from the TC. To me, building on top means building on top of the languages (which users will have to invest a lot of work in learning) as well, rather than having a completely different language and only using the underlying implementation(s). That all sounds logical to me and would keep things clean, i.e. keep the HOT language clean by not mixing it with imperative expression, and keep the Heat engine clean by not blowing it up to act as a workflow engine. When I think about the two aspects that are being brought up in this thread (declarative description of a desired state and workflows) my thinking is that much (and actually as much as possible) can be done declaratively the way Heat does it with HOT. Then for bigger lifecycle management there will be a need for additional workflows on top, because at some point it will be hard to express management logic declaratively in a topology model. Those additional flows on-top will have to be aware of the instance created from a declarative template (i.e. a Heat stack) because it needs to act on the respective resources to do something in addition. So when thinking about a domain specific workflow language, it should be possible to define tasks (in a template aware manner) like on resource XYZ of the template, do something, or update resource XYZ of the template with this state, then do this etc. At runtime this would resolve to the actual resource instances created from the resource templates. Making such constructs available to the workflow authors would make sense. Having a workflow service able to execute this via the right underlying APIs would be the execution part. I think from an instance API perspective, Heat already brings a lot for this with the stack model, so workflow tasks could be written to use the stack API to access instance information. Things like update of resources is also something that is already there. BTW, we have a similar concept (or are working on a refinement of it based on latest discussions) in TOSCA and call it the plan portability API, i.e. an API that a declarative engine would expose so that fit-for-purpose workflow tasks can be defined on-top. Regards, Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Hi, I think notification mechanism proposed in Heat will work fine for integration with external workflows. The approach which uses workflows outside of Heat engine sounds consistent with our current approach in Murano. I am looking into new TOSCA yaml format and I also ask Mirantis management to consider joining OASIS. The decision is not made yet, but hopefully will be made on next week. We eager to jump onto TOSCA standard work and contribute plan related parts. Thanks Georgy On Wed, Mar 19, 2014 at 1:38 PM, Thomas Spatzier thomas.spatz...@de.ibm.com wrote: Excerpts from Zane Bitter's message on 19/03/2014 18:18:34: From: Zane Bitter zbit...@redhat.com To: openstack-dev@lists.openstack.org Date: 19/03/2014 18:21 Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? On 19/03/14 05:00, Stan Lagun wrote: Steven, Agree with your opinion on HOT expansion. I see that inclusion of imperative workflows and ALM would require major Heat redesign and probably would be impossible without loosing compatibility with previous HOT syntax. It would blur Heat mission, confuse current users and rise a lot of questions what should and what should not be in Heat. Thats why we chose to built a system on top of Heat rather then expending HOT. +1, I agree (as we have discussed before) that it would be a mistake to shoehorn workflow stuff into Heat. I do think we should implement the hooks I mentioned at the start of this thread to allow tighter integration between Heat and a workflow engine (i.e. Mistral). +1 on not putting workflow stuff into Heat. Rather let's come up with a nice way of Heat and a workflow service to work together. That could be done in two ways: (1) let Heat hand off to a workflow service for certains tasks or (2) let people define workflow tasks that can easily work on Heat deployed resources. Maybe both make sense, but right now I am more leaning towards (2). So building a system on top of Heat is good. Building it on top of Mistral as well would also be good, and that was part of the feedback from the TC. To me, building on top means building on top of the languages (which users will have to invest a lot of work in learning) as well, rather than having a completely different language and only using the underlying implementation(s). That all sounds logical to me and would keep things clean, i.e. keep the HOT language clean by not mixing it with imperative expression, and keep the Heat engine clean by not blowing it up to act as a workflow engine. When I think about the two aspects that are being brought up in this thread (declarative description of a desired state and workflows) my thinking is that much (and actually as much as possible) can be done declaratively the way Heat does it with HOT. Then for bigger lifecycle management there will be a need for additional workflows on top, because at some point it will be hard to express management logic declaratively in a topology model. Those additional flows on-top will have to be aware of the instance created from a declarative template (i.e. a Heat stack) because it needs to act on the respective resources to do something in addition. So when thinking about a domain specific workflow language, it should be possible to define tasks (in a template aware manner) like on resource XYZ of the template, do something, or update resource XYZ of the template with this state, then do this etc. At runtime this would resolve to the actual resource instances created from the resource templates. Making such constructs available to the workflow authors would make sense. Having a workflow service able to execute this via the right underlying APIs would be the execution part. I think from an instance API perspective, Heat already brings a lot for this with the stack model, so workflow tasks could be written to use the stack API to access instance information. Things like update of resources is also something that is already there. BTW, we have a similar concept (or are working on a refinement of it based on latest discussions) in TOSCA and call it the plan portability API, i.e. an API that a declarative engine would expose so that fit-for-purpose workflow tasks can be defined on-top. Regards, Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Management as I've defined it here? If so, why could we not use the Heat DSL to also define the System? The System definition could be logically separated out into its own text file... But, we'd have a common DSL syntax and semantics for both lower level and higher level component interaction (a building block effect of sorts). As for who will generate it, ( with it being the Heat multi-tier application/infrastructure definition) I think that question will go through a lot more evolution and could be any number of sources: e.g. Solum, Murano, Horizon, Template Author with a text editor, etc. Basically, I'm a +1 for as few DSLs as possible. I support the position that we should evolve HOT if needed vs. having two separate DSLs that are both related to expressing application and infrastructure semantics. Workflow is quite interesting ... Should we be able to express imperative workflow semantics in HOT? Or, should we only be able to declare workflow configurations that get configured in a service like Mistral whereby Mistral's execution of a workflow may need to invoke Heat hooks or Stack Updates? Or, some other solution? I look forward to a design discussion on all this at the summit... This is fun stuff to think about! -Keith From: Georgy Okrokvertskhov gokrokvertsk...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Tuesday, March 18, 2014 1:49 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? I see this in the following way - who will generate HOT template for my complex multi-tier applications when I have only templates for components? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Ability to hook up to application deployment is indeed a good thing. And it can be done both in HOT and MuranoPL. And Mistral is a good candidate to handle those hooks. But this is not a replacement for MuranoPL but an addition to it. The problem with hooks is that you cannot hoot just into arbitrary place in deployment workflow. And the author of Python code may not expose the exact hook that you need. Hooks can work for logging purposes or triggering some additional workflows but are not good for customization of your workflow from outside. Hooked code may not have access to all engine's internal state and workflow context and have even less chances to modify it in a safe manner On Thu, Mar 20, 2014 at 1:21 AM, Georgy Okrokvertskhov gokrokvertsk...@mirantis.com wrote: Hi, I think notification mechanism proposed in Heat will work fine for integration with external workflows. The approach which uses workflows outside of Heat engine sounds consistent with our current approach in Murano. I am looking into new TOSCA yaml format and I also ask Mirantis management to consider joining OASIS. The decision is not made yet, but hopefully will be made on next week. We eager to jump onto TOSCA standard work and contribute plan related parts. Thanks Georgy On Wed, Mar 19, 2014 at 1:38 PM, Thomas Spatzier thomas.spatz...@de.ibm.com wrote: Excerpts from Zane Bitter's message on 19/03/2014 18:18:34: From: Zane Bitter zbit...@redhat.com To: openstack-dev@lists.openstack.org Date: 19/03/2014 18:21 Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? On 19/03/14 05:00, Stan Lagun wrote: Steven, Agree with your opinion on HOT expansion. I see that inclusion of imperative workflows and ALM would require major Heat redesign and probably would be impossible without loosing compatibility with previous HOT syntax. It would blur Heat mission, confuse current users and rise a lot of questions what should and what should not be in Heat. Thats why we chose to built a system on top of Heat rather then expending HOT. +1, I agree (as we have discussed before) that it would be a mistake to shoehorn workflow stuff into Heat. I do think we should implement the hooks I mentioned at the start of this thread to allow tighter integration between Heat and a workflow engine (i.e. Mistral). +1 on not putting workflow stuff into Heat. Rather let's come up with a nice way of Heat and a workflow service to work together. That could be done in two ways: (1) let Heat hand off to a workflow service for certains tasks or (2) let people define workflow tasks that can easily work on Heat deployed resources. Maybe both make sense, but right now I am more leaning towards (2). So building a system on top of Heat is good. Building it on top of Mistral as well would also be good, and that was part of the feedback from the TC. To me, building on top means building on top of the languages (which users will have to invest a lot of work in learning) as well, rather than having a completely different language and only using the underlying implementation(s). That all sounds logical to me and would keep things clean, i.e. keep the HOT language clean by not mixing it with imperative expression, and keep the Heat engine clean by not blowing it up to act as a workflow engine. When I think about the two aspects that are being brought up in this thread (declarative description of a desired state and workflows) my thinking is that much (and actually as much as possible) can be done declaratively the way Heat does it with HOT. Then for bigger lifecycle management there will be a need for additional workflows on top, because at some point it will be hard to express management logic declaratively in a topology model. Those additional flows on-top will have to be aware of the instance created from a declarative template (i.e. a Heat stack) because it needs to act on the respective resources to do something in addition. So when thinking about a domain specific workflow language, it should be possible to define tasks (in a template aware manner) like on resource XYZ of the template, do something, or update resource XYZ of the template with this state, then do this etc. At runtime this would resolve to the actual resource instances created from the resource templates. Making such constructs available to the workflow authors would make sense. Having a workflow service able to execute this via the right underlying APIs would be the execution part. I think from an instance API perspective, Heat already brings a lot for this with the stack model, so workflow tasks could be written to use the stack API to access instance information. Things like update of resources is also something that is already there. BTW, we have a similar concept (or are working on a refinement of it based on latest discussions) in TOSCA and call it the plan
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
(I added a couple of tags to the subject; hope this doesn't screw up
anyone's threading.)
On 09/03/14 16:26, Joshua Harlow wrote:
I'd be very interested in knowing the resource controls u plan to add.
Memory, CPU...
I'm still trying to figure out where something like
https://github.com/istalker2/MuranoDsl/blob/master/meta/com.mirantis.murano.demoApp.DemoInstance/manifest.yaml
would be beneficial, why not just spend effort sand boxing lua,
python... Instead of spending effort on creating a new language and then
having to sandbox it as well... Especially if u picked languages that
are made to be sandboxed from the start (not python)...
-1 to using a full-blown programming language like Python over a DSL, but...
Who is going to train people on muranoPL, write language books and
tutorials when the same amount of work has already been done for 10+
years for other languages
I fail to see where muranoPL is a DSL when it contains a full language
already with functions, objects, namespaces, conditionals... (what is
the domain of it?), maybe I'm just confused though (quite possible, haha).
...I'm inclined to agree with this. Whenever you find yourself
implementing a Turing-complete Object-Oriented DSL... well, you'd at
least want to stop and think very carefully about whether you might have
taken a wrong turn somewhere.
Does this not seem awkward to anyone else??
It does seem really awkward to me (and not just because of all the
$signs), because it's duplicating basically all of the functionality of
Heat. e.g. in MuranoPL you have:
Properties:
name:
Contract: $.string().notNull()
whereas in HOT this would be:
parameters:
name:
type: string
constraints:
- length: {min: 1}
In MuranoPL you reference it using $this.name, vs. HOT using
{get_param: name}.
Note that HOT (a) already exists in OpenStack, and (b) is being
developed in conjunction with TOSCA folks to ensure easy translation
to/from TOSCA Simple Profile YAML rendering.
Looking at e.g. [1], more or less everything in here can be done
already inside a Heat template, using get_file and str_replace.
It sounds like this is a DSL in which you write everything imperatively,
then it gets converted behind the scenes into a declarative model in a
completely different language (but not using any of the advanced
features of that language) and passed to Heat, which turns it back into
a workflow to execute. That seems bizarre to me. Surely Murano should be
focusing on filling the gaps in Heat, rather than reimplementing it in a
different paradigm?
What I'm imagining here is something along the lines of:
- Heat implements hooks to customise its workflows, as proposed in [2], [3].
- Deployments are defined declaratively using HOT syntax.
- Workflows - to wrap the deployment operations, to customise the
deployment and to perform lifecycle operations like backups - are
defined using a Mistral DSL (I assume this exists already? I haven't
looked into it).
- Murano provides a way to bundle the various workflow definitions, HOT
models, and other files into an application package.
Can anybody enlighten me as to what features would be missing from this
that would warrant creating a new programming language?
thanks,
Zane.
[1]
https://github.com/istalker2/MuranoDsl/blob/master/meta/com.mirantis.murano.demoApp.DemoInstance/manifest.yaml
[2]
http://lists.openstack.org/pipermail/openstack-dev/2014-February/026329.html
[3]
http://lists.openstack.org/pipermail/openstack-dev/2014-March/030228.html
Sent from my really tiny device...
On Mar 8, 2014, at 10:44 PM, Stan Lagun sla...@mirantis.com
mailto:sla...@mirantis.com wrote:
First of all MuranoPL is not a host but hosted language. It never
intended to replace Python and if Python can do the job it is probably
better than MuranoPL for that job.
The problem with Python is that you cannot have Python code as a part
of your DSL if you need to evaluate that DSL on server-side. Using
Python's eval() is not secure. And you don't have enough control on
what evaled code is allowed to do. MuranoPL on the contrary are fully
sandboxed. You have absolute control over what functions/methods/APIs
are exposed to DSL and DSL code can do nothing except for what it
allowed to do. Besides you typically do want your DSL to be
domain-specific so general-purpose language like Python can be suboptimal.
I don't say MuranoPL is good for all projects. It has many
Murano-specific things after all. In most cases you don't need all
those OOP features that MuranoPL has. But the code organization, how
it uses YAML, block structures and especially YAQL expressions can be
of a great value to many projects
For examples of MuranoPL classes you can browse
https://github.com/istalker2/MuranoDsl/tree/master/meta folder. This
is my private repository that I was using to develop PoC for MuranoPL
engine. We are on the way to create production-quality implementation
with unit-tests etc. in
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Hi Zane,
Thank you for your feedback. This is very important for us to find out how
our approach can be aligned with existing solutions. Let me try to explain
how we come up to specific solutions.
It does seem really awkward to me (and not just because of all the
$signs), because it's duplicating basically all of the functionality of
Heat. e.g. in MuranoPL you have:
Properties:
name:
Contract: $.string().notNull()
whereas in HOT this would be:
parameters:
name:
type: string
constraints:
- length: {min: 1}
First of all parameters are the common entities in all different services.
Some of them use config files, some of them use yaml formats. I don't see
that this is a significant overlap with Heat. While Murano parameters can
be directly passed to Heat template generated, there is a major difference
between Murano parameters and Heat parameters. We use these parameters to
bind different entities together. The good example is here [1][2]
Properties:
name:
Contract: $.string().notNull()
primaryController:
Contract: $.class(PrimaryController).notNull()
secondaryControllers:
Contract: [$.class(SecondaryController).notNull()]
As you see, parameters here are used to bind different applications as
PrimaryController and Secondary controllers are different VMs with
different configurations.
Murano can't use Heat parameters for that as we need to bind
applications before Heat template is generated. If we even try to use
Heat for keeping such parameters it will lead to Murano parsing and
processing Heat templates which we want to avoid.
Looking at e.g. [1], more or less everything in here can be done already
inside a Heat template, using get_file and str_replace.
This definitely can be done inside Heat template, but these operations
(str_replace) are not specific to Heat. As we use these variables
before HOT template creation it is better to do string operations
inside the service then pass this to another service. As we can't
expose python functions for that we just wrap them in our DSL syntax
which is just a couple lines of Python code around pythons string
methods.
It sounds like this is a DSL in which you write everything imperatively,
then it gets converted behind the scenes into a declarative model in a
completely different language (but not using any of the advanced features
of that language) and passed to Heat, which turns it back into a workflow
to execute. That seems bizarre to me. Surely Murano should be focusing on
filling the gaps in Heat, rather than reimplementing it in a different
paradigm?
We will be very happy to fill gaps in the HOT syntax to extend it. At the
same time, I need to mention, that the picture you drew does not perfectly
reflects what we are doing. Murano uses its own definition of application
which covers the specific aspects of application - actions and application
binding. As a part of application definition we store Heat snippets as
files. We don't hide anything from Heat under the Murano layer. All
features available in Heat can be used in Murano as user has an ability to
add Heat template to the application definition. As Heat main feature is to
manage different resource we pass this tasks to Heat. Murano does not do
any kind of infrastructure resource management outside of the Heat.
Heat template generation is just a first step in Application life. Murano
provides a way to define workflows for different aspects
of application which can be invoked by Heat during generation or out of
Heat by Mistral, Ceilometer and other services, including Murano.
What I'm imagining here is something along the lines of:
- Heat implements hooks to customise its workflows, as proposed in [2],
[3].
- Deployments are defined declaratively using HOT syntax.
- Workflows - to wrap the deployment operations, to customise the
deployment and to perform lifecycle operations like backups - are defined
using a Mistral DSL (I assume this exists already? I haven't looked into
it).
- Murano provides a way to bundle the various workflow definitions, HOT
models, and other files into an application package.
Murano is not focused on deployment itself. As soon as we have HOT Murano
is responsible to generate proper HOT template to make actual deployments.
As soon as HOT Software components are ready, application publisher will
use them in Heat templates.
Can anybody enlighten me as to what features would be missing from this
that would warrant creating a new programming language?
I see this in the following way - who will generate HOT template for my
complex multi-tier applications when I have only templates for components?
It looks like I will have to write a new template by myself understanding
all possible relation between components and their parameters. Then I have
to add some specific workflows to Mistral and somehow bind them to the
resources deployed by Heat. Then I have to understand what kind of
notifications to expect and somehow react on them
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Georgy, In consideration of the can you express it instead of the who will generate it, I see Heat's HOT evolving to support the expression of complex multi-tier architectures and applications (I would argue you can already do this today, perhaps with some additional features desired, e.g. Ability to define cloud workflows and workflow execution rules which could come when we have a workflow service like Mistral). Therefore, I would encourage Murano contributors to consider whether they can help make Heat sufficiently cover desired use cases. I have never viewed Heat templates as isolated components of a multi-tier architecture. Instead, a single template or a combination of master/subordinate templates together (using references, nesting, or inclusion) could express the complete architecture, both infrastructure and applications. If I've read your previous comments and threads correctly, you desire a way to express System Lifecycle Management across multiple related applications or components, whereby you view the System as a grouping of independently developed and/or deployed (but systematically related) components, whereby you view Components as individual disconnected Heat templates that independently describe different application stacks of the System. Did I get that correct? If so, perhaps the discussion here is one of scope of what can or should be expressed in a Heat template. Is it correct to state that your argument is that a separate system (such as Murano) should be used to express System Lifecycle Management as I've defined it here? If so, why could we not use the Heat DSL to also define the System? The System definition could be logically separated out into its own text file... But, we'd have a common DSL syntax and semantics for both lower level and higher level component interaction (a building block effect of sorts). As for who will generate it, ( with it being the Heat multi-tier application/infrastructure definition) I think that question will go through a lot more evolution and could be any number of sources: e.g. Solum, Murano, Horizon, Template Author with a text editor, etc. Basically, I'm a +1 for as few DSLs as possible. I support the position that we should evolve HOT if needed vs. having two separate DSLs that are both related to expressing application and infrastructure semantics. Workflow is quite interesting ... Should we be able to express imperative workflow semantics in HOT? Or, should we only be able to declare workflow configurations that get configured in a service like Mistral whereby Mistral's execution of a workflow may need to invoke Heat hooks or Stack Updates? Or, some other solution? I look forward to a design discussion on all this at the summit... This is fun stuff to think about! -Keith From: Georgy Okrokvertskhov gokrokvertsk...@mirantis.commailto:gokrokvertsk...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Tuesday, March 18, 2014 1:49 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? I see this in the following way - who will generate HOT template for my complex multi-tier applications when I have only templates for components? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
: [openstack-dev] [Murano][Heat] MuranoPL questions? I see this in the following way - who will generate HOT template for my complex multi-tier applications when I have only templates for components? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Hi Ruslan, I did not intend to suggest that definition of things like billing rules should necessarily be supported syntax in Heat. Murano is certainly able to develop whatever features it would like, including an alternative DSL. I have a preference towards minimizing DSLs across the OpenStack ecosystem, if possible. I hope that helps clear up my position. If the goal is to specify pertinent information that a billing system could consume, I default back to wanting to specify/store that information as associated data in the catalog with the application definition (e.g. HOT artifact in Glance). If the billing/rules logic crosses multiple HOT artifacts, I find that very interesting and would enjoy reading about a specific use-case. As for Trove and Savanna, I view these as application Services. Service registry/discovery isn't what I thought Murano was aiming to be, but I've been wrong about what Murano is/isn't/wants-to-be many times before, so my apologies if I'm misunderstanding again. I am a bit more confused now, however, given that discovery/definition of exposed software services is entering the conversation? Thank you for considering my comments, -Keith On 3/18/14 7:32 PM, Ruslan Kamaldinov rkamaldi...@mirantis.com wrote: - definition of an application which is already exposed via REST API. Think of something like Sahara (ex. Savanna) or Trove developed in-house for internal company needs. app publishers wouldn't be happy if they'll be forced to develop a new resource for Heat - definition of billing rules for an application ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
? Or, should we only be able to declare workflow configurations that get configured in a service like Mistral whereby Mistral's execution of a workflow may need to invoke Heat hooks or Stack Updates? Or, some other solution? I look forward to a design discussion on all this at the summit... This is fun stuff to think about! -Keith From: Georgy Okrokvertskhov gokrokvertsk...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Tuesday, March 18, 2014 1:49 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? I see this in the following way - who will generate HOT template for my complex multi-tier applications when I have only templates for components? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
of concerns is best for OpenStack and should be the preferred development path. Regards, -steve [1] http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two Thanks, Ruslan On Wed, Mar 19, 2014 at 2:07 AM, Keith Bray keith.b...@rackspace.com wrote: Georgy, In consideration of the can you express it instead of the who will generate it, I see Heat's HOT evolving to support the expression of complex multi-tier architectures and applications (I would argue you can already do this today, perhaps with some additional features desired, e.g. Ability to define cloud workflows and workflow execution rules which could come when we have a workflow service like Mistral). Therefore, I would encourage Murano contributors to consider whether they can help make Heat sufficiently cover desired use cases. I have never viewed Heat templates as isolated components of a multi-tier architecture. Instead, a single template or a combination of master/subordinate templates together (using references, nesting, or inclusion) could express the complete architecture, both infrastructure and applications. If I've read your previous comments and threads correctly, you desire a way to express System Lifecycle Management across multiple related applications or components, whereby you view the System as a grouping of independently developed and/or deployed (but systematically related) components, whereby you view Components as individual disconnected Heat templates that independently describe different application stacks of the System. Did I get that correct? If so, perhaps the discussion here is one of scope of what can or should be expressed in a Heat template. Is it correct to state that your argument is that a separate system (such as Murano) should be used to express System Lifecycle Management as I've defined it here? If so, why could we not use the Heat DSL to also define the System? The System definition could be logically separated out into its own text file... But, we'd have a common DSL syntax and semantics for both lower level and higher level component interaction (a building block effect of sorts). As for who will generate it, ( with it being the Heat multi-tier application/infrastructure definition) I think that question will go through a lot more evolution and could be any number of sources: e.g. Solum, Murano, Horizon, Template Author with a text editor, etc. Basically, I'm a +1 for as few DSLs as possible. I support the position that we should evolve HOT if needed vs. having two separate DSLs that are both related to expressing application and infrastructure semantics. Workflow is quite interesting ... Should we be able to express imperative workflow semantics in HOT? Or, should we only be able to declare workflow configurations that get configured in a service like Mistral whereby Mistral's execution of a workflow may need to invoke Heat hooks or Stack Updates? Or, some other solution? I look forward to a design discussion on all this at the summit... This is fun stuff to think about! -Keith From: Georgy Okrokvertskhov gokrokvertsk...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Tuesday, March 18, 2014 1:49 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? I see this in the following way - who will generate HOT template for my complex multi-tier applications when I have only templates for components? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
