Re: [openstack-dev] [Murano] Need a new DSL for Murano

2014-02-24 Thread Alexander Tivelkov 
Hi Dmitry,

I agree with you on this vision, however we have to think more on the
terminology: "Service Catalog" in OpenStack relates to Keystone (where by
"Service" we mean Openstack's infrastructure-level services).
I understand your concerns on "runtime lifecycle" vs "code-to-binary"
lifecycle though - it is absolutely valid, and we do not want to have any
overlap with Solum in this matter.

--
Regards,
Alexander Tivelkov


On Mon, Feb 24, 2014 at 1:47 PM, Dmitry  wrote:

> I think this is the great new service which will accompany OpenStack Solum
> similarly as Bosh is accompanying Cloud Foundry and CloudForms is
> accompanying OpenShift.
> I wouldn't call it the Application Catalog but the Service Catalog,
> because of the primary focus on the Service Life-cycle management (in
> opposite to application lifecycle management which is focused on
> "code-to-binary", execution, remote debugging and log grabbing etc).
> In order to make Murano the real Service Catalog, it should support (over
> DSL) run-time events processing such as service scaling (up/out/in/down),
> healing, replication, live-migration etc.
> In addition, it should support a "template" creation which could be used
> by Solum similar to Heroku BuildPack.
> I would happy to hear your opinion on how you envision the Murano's
> roadmap.
>
> Thanks,
> Dmitry
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Murano] Need a new DSL for Murano

2014-02-24 Thread Dmitry 
I think this is the great new service which will accompany OpenStack Solum
similarly as Bosh is accompanying Cloud Foundry and CloudForms is
accompanying OpenShift.
I wouldn't call it the Application Catalog but the Service Catalog, because
of the primary focus on the Service Life-cycle management (in opposite to
application lifecycle management which is focused on "code-to-binary",
execution, remote debugging and log grabbing etc).
In order to make Murano the real Service Catalog, it should support (over
DSL) run-time events processing such as service scaling (up/out/in/down),
healing, replication, live-migration etc.
In addition, it should support a "template" creation which could be used by
Solum similar to Heroku BuildPack.
I would happy to hear your opinion on how you envision the Murano's roadmap.

Thanks,
Dmitry
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Murano] Need a new DSL for Murano

2014-02-18 Thread Georgy Okrokvertskhov 
>My observation has been that Murano has changed from a Windows focused
Deployment Service >to a Metadata Application Catalog Workflow thing (I
fully admit this may be an invalid >observation).  It's unclear to me what
OpenStack pain/use-cases is to be solved by "complex >object composition,
description of data types, contracts..."

Murano is intended to provide high level definition of Applications which
will include description of specific application requirements (like input
parameters, external dependencies) as well as low level scripts, heat
snippets and workflows which will be required to manage application.

Object composition is required to address application diversity. We expect
to be an integration layer for numerous different application from
different OS and areas like WebServices, BigData, SAP, Enterprise specific
apps. In order to decrease amount of work for Application author we will
provide a way to reuse existing applications by extending them via object
composition. Inside Murano we provide a library of workflows and
requirements for general application classes. This library has workflows
for instance creation, networking and app deployments. This will help
application author to write only application specific stuff. For example if
I need to publish my web service based on Tomcat I will create a new
application class which has tomcat as a parent and then I will add my
application specific parameters like App URL and will add deployment script
which will download App war file and put it to Tomcat app directory. All
other stuff will be done automatically by existing workflows for tomcat
application.

You can imagine this as a nested Heat template inclusion but with ability
to override and\or extend it. The ability to add a workflow actually looks
like a dynamic Heat resource type generation as you can specify actions and
associated workflows. These actions can be triggered by external events to
provide an ability of application life cycle management.

Thanks
Georgy


On Mon, Feb 17, 2014 at 4:41 PM, Keith Bray wrote:

>
>  Can someone elaborate further on the things that Murano is intended to
> solve within the OpenStack ecosystem?   My observation has been that Murano
> has changed from a Windows focused Deployment Service to a Metadata
> Application Catalog Workflow thing (I fully admit this may be an invalid
> observation).  It's unclear to me what OpenStack pain/use-cases is to be
> solved by "complex object composition, description of data types,
> contracts..."
>
>  Your thoughts would be much appreciated.
>
>  Thanks,
> -Keith
>
>   From: Renat Akhmerov 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Monday, February 17, 2014 1:33 AM
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [Murano] Need a new DSL for Murano
>
>   Clint,
>
>  We're collaborating with Murano. We may need to do it in a way that
> others could see it though. There are several things here:
>
>- Murano doesn't really have a "workflow engine" similar to Mistral's.
>People get confused with that but it's just a legacy terminology, I think
>Murano folks were going to rename this component to be more precise about
>it.
>- Mistral DSL doesn't seem to be a good option for solving tasks that
>Murano is intended to solve. Specifically I mean things like complex object
>composition, description of data types, contracts and so on. Like Alex and
>Stan mentioned Murano DSL tends to grow into a full programming language
>.
>- Most likely Mistral will be used in Murano for implementation, at
>least we see where it would be valuable. But Mistral is not so matured yet,
>we need to keep working hard and be patient :)
>
>
>  Anyway, we keep thinking on how to make both languages look similar or
> at least the possibility to use them seamlessly, if needed (call Mistral
> workflows from Murano DSL or vice versa).
>
>   Renat Akhmerov
> @ Mirantis Inc.
>
>  On 16 Feb 2014, at 05:48, Clint Byrum  wrote:
>
> Excerpts from Alexander Tivelkov's message of 2014-02-14 18:17:10 -0800:
>
> Hi folks,
>
> Murano matures, and we are getting more and more feedback from our early
> adopters. The overall reception is very positive, but at the same time
> there are some complaints as well. By now the most significant complaint is
> is hard to write workflows for application deployment and maintenance.
>
> Current version of workflow definition markup really have some design
> drawbacks which limit its potential adoption. They are caused by the fa

Re: [openstack-dev] [Murano] Need a new DSL for Murano

2014-02-17 Thread Stan Lagun 
Keith,

I believe it is the same place Marketplace has in AWS or OpenShift
Application Gallery to Red Hat OpenShift.
The idea was of having free/open app catalog that cloud admins can fill
with application metadata describing those apps and then end users
construct environments from those applications in user friendly UI.

It is not ready recipes like Heat templates are that you only have to fill
several parameters and the whole stack created for you but a constructor
(think LEGO) for applications.

Lets take a normal non-tech user (who cannot program/write
JSON/YAML/whatever templates etc.) who wants to install Drupal on his
OpenStack account.

With Heat it would require finding HOT template for Drupal, answer several
questions and click one button in Thermal. Then new Heat stack would be
created with typically one VM for Drupal and another for MySQL.

With Murano UI user finds Drupal in App Catalog and puts it onto
environment. Now if he clicks "deploy" at this point the result would be
the same as in Heat. But he also can edit what was generated. For example
UI shows that Drupal requires a database that can be MySQL or PostgreSQL.
Instead of vanilla MySQL user can go to App Catalog and peak Galera MySQL
or PostgreSQL or something else that is compatible with those 2 DBMS. Or he
can reuse database server that already exist in his environment. Drupal is
hosted on web server so again user can replace Apache with Ngnix or host
Drupal on some existing web server. The same goes for VM instances - user
may choose to have both database and Drupal on the same VM.

Now if you take all the components involved with even simple app like
Drupal (Drupal, web server, PHP module for web server, database, VMs,
operating systems, instance flavors) there are hundreds of combinations
that can be constructed. With the Heat it would require hundreds of HOT
templates (even if we forget that Heat cannot refer to resources located in
another stack). And if user later decides to add load balancer to his
environment to balance Drupal instances he is in trouble because load
balancer brings another set of combinations (different LB implementations,
underlying operating system, flavor) and things become unmanageable.

So Heat is good for DevOps that can write HOT templates for exact task with
all the control possible. And Murano is for the rest of us. It is something
cloud providers can expose to users that do not know JSON, shell, puppet
etc but want to install complex applications and still have control on it.



On Tue, Feb 18, 2014 at 4:41 AM, Keith Bray wrote:

>
>  Can someone elaborate further on the things that Murano is intended to
> solve within the OpenStack ecosystem?   My observation has been that Murano
> has changed from a Windows focused Deployment Service to a Metadata
> Application Catalog Workflow thing (I fully admit this may be an invalid
> observation).  It's unclear to me what OpenStack pain/use-cases is to be
> solved by "complex object composition, description of data types,
> contracts..."
>
>  Your thoughts would be much appreciated.
>
>  Thanks,
> -Keith
>
>   From: Renat Akhmerov 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Monday, February 17, 2014 1:33 AM
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [Murano] Need a new DSL for Murano
>
>   Clint,
>
>  We're collaborating with Murano. We may need to do it in a way that
> others could see it though. There are several things here:
>
>- Murano doesn't really have a "workflow engine" similar to Mistral's.
>People get confused with that but it's just a legacy terminology, I think
>Murano folks were going to rename this component to be more precise about
>it.
>- Mistral DSL doesn't seem to be a good option for solving tasks that
>Murano is intended to solve. Specifically I mean things like complex object
>composition, description of data types, contracts and so on. Like Alex and
>Stan mentioned Murano DSL tends to grow into a full programming language
>.
>- Most likely Mistral will be used in Murano for implementation, at
>least we see where it would be valuable. But Mistral is not so matured yet,
>we need to keep working hard and be patient :)
>
>
>  Anyway, we keep thinking on how to make both languages look similar or
> at least the possibility to use them seamlessly, if needed (call Mistral
> workflows from Murano DSL or vice versa).
>
>   Renat Akhmerov
> @ Mirantis Inc.
>
>  On 16 Feb 2014, at 05:48, Clint Byrum  wrote:
>
> Excerpts from Alexander Tivelkov's message of 2014-02-14 18:17:10 -08

Re: [openstack-dev] [Murano] Need a new DSL for Murano

2014-02-17 Thread Keith Bray 

Can someone elaborate further on the things that Murano is intended to solve 
within the OpenStack ecosystem?   My observation has been that Murano has 
changed from a Windows focused Deployment Service to a Metadata Application 
Catalog Workflow thing (I fully admit this may be an invalid observation).  
It's unclear to me what OpenStack pain/use-cases is to be solved by "complex 
object composition, description of data types, contracts..."

Your thoughts would be much appreciated.

Thanks,
-Keith

From: Renat Akhmerov mailto:rakhme...@mirantis.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Date: Monday, February 17, 2014 1:33 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Murano] Need a new DSL for Murano

Clint,

We're collaborating with Murano. We may need to do it in a way that others 
could see it though. There are several things here:

  *   Murano doesn’t really have a “workflow engine” similar to Mistral’s. 
People get confused with that but it’s just a legacy terminology, I think 
Murano folks were going to rename this component to be more precise about it.
  *   Mistral DSL doesn’t seem to be a good option for solving tasks that 
Murano is intended to solve. Specifically I mean things like complex object 
composition, description of data types, contracts and so on. Like Alex and Stan 
mentioned Murano DSL tends to grow into a full programming language.
  *   Most likely Mistral will be used in Murano for implementation, at least 
we see where it would be valuable. But Mistral is not so matured yet, we need 
to keep working hard and be patient :)

Anyway, we keep thinking on how to make both languages look similar or at least 
the possibility to use them seamlessly, if needed (call Mistral workflows from 
Murano DSL or vice versa).

Renat Akhmerov
@ Mirantis Inc.

On 16 Feb 2014, at 05:48, Clint Byrum 
mailto:cl...@fewbar.com>> wrote:

Excerpts from Alexander Tivelkov's message of 2014-02-14 18:17:10 -0800:
Hi folks,

Murano matures, and we are getting more and more feedback from our early
adopters. The overall reception is very positive, but at the same time
there are some complaints as well. By now the most significant complaint is
is hard to write workflows for application deployment and maintenance.

Current version of workflow definition markup really have some design
drawbacks which limit its potential adoption. They are caused by the fact
that it was never intended for use for Application Catalog use-cases.


Just curious, is there any reason you're not collaborating on Mistral
for this rather than both having a workflow engine?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Murano] Need a new DSL for Murano

2014-02-16 Thread Renat Akhmerov 
Clint, 

We're collaborating with Murano. We may need to do it in a way that others 
could see it though. There are several things here:
Murano doesn’t really have a “workflow engine” similar to Mistral’s. People get 
confused with that but it’s just a legacy terminology, I think Murano folks 
were going to rename this component to be more precise about it.
Mistral DSL doesn’t seem to be a good option for solving tasks that Murano is 
intended to solve. Specifically I mean things like complex object composition, 
description of data types, contracts and so on. Like Alex and Stan mentioned 
Murano DSL tends to grow into a full programming language.
Most likely Mistral will be used in Murano for implementation, at least we see 
where it would be valuable. But Mistral is not so matured yet, we need to keep 
working hard and be patient :)

Anyway, we keep thinking on how to make both languages look similar or at least 
the possibility to use them seamlessly, if needed (call Mistral workflows from 
Murano DSL or vice versa).

Renat Akhmerov
@ Mirantis Inc.

On 16 Feb 2014, at 05:48, Clint Byrum  wrote:

> Excerpts from Alexander Tivelkov's message of 2014-02-14 18:17:10 -0800:
>> Hi folks,
>> 
>> Murano matures, and we are getting more and more feedback from our early
>> adopters. The overall reception is very positive, but at the same time
>> there are some complaints as well. By now the most significant complaint is
>> is hard to write workflows for application deployment and maintenance.
>> 
>> Current version of workflow definition markup really have some design
>> drawbacks which limit its potential adoption. They are caused by the fact
>> that it was never intended for use for Application Catalog use-cases.
>> 
> 
> Just curious, is there any reason you're not collaborating on Mistral
> for this rather than both having a workflow engine?
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Murano] Need a new DSL for Murano

2014-02-15 Thread Clint Byrum 
Excerpts from Alexander Tivelkov's message of 2014-02-14 18:17:10 -0800:
> Hi folks,
> 
> Murano matures, and we are getting more and more feedback from our early
> adopters. The overall reception is very positive, but at the same time
> there are some complaints as well. By now the most significant complaint is
> is hard to write workflows for application deployment and maintenance.
> 
> Current version of workflow definition markup really have some design
> drawbacks which limit its potential adoption. They are caused by the fact
> that it was never intended for use for Application Catalog use-cases.
> 

Just curious, is there any reason you're not collaborating on Mistral
for this rather than both having a workflow engine?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Murano] Need a new DSL for Murano

2014-02-15 Thread Stan Lagun 
Just to add my 2 cents.

1. YAML is already familiar to OpenStack developers as Heat and others use
it. So at least the syntax (not to mess with semantics) doesn't have to be
learned.

2. YAML parser is very flexible an can be extended with additional types or
constructs like Key:  to include content of other file etc.
Murano DSL may operate on already deserialized (parsed) data leaving yaml
files access to hosting operation. Thus the engine itself can be
independent from how and where the YAML files are stored. This is very good
for App Catalog that stores its data in Glance. Also it always possible to
serialize it to some other format (XML, JSON, whatever) if it required for
some purpose

3. YAML declarations can be processed by Murano dashboard, 3-rd party
software and various tooling for Murano. If it wasn't YAML but some
handcrafted syntax then we would have to provide them with embeddable
parser for that syntax

4. Implementing parser for full-blown language is not a trivial task to say
the least. It would require much more time for development and greatly
increase probability to shoot ourself in the foot. And we really don't want
to have like a year between Murano versions.


On Sat, Feb 15, 2014 at 12:18 PM, Alexander Tivelkov  wrote:

> Hi Joshua,
>
> Thank you very much for you feedback!
> This is really a great question, and it was the first question we've asked
> ourselves when we started thinking about this new design.
> We've considered both options: to have our own syntax (python-like,
> java-like, something-else-like) or to use YAML. We've chosen the latest,
> and here are the reasons.
>
> The most important moment: this is not a general-purpose language, but a
> domain-specific one. And Murano's domain is manipulating with complex
> nested data-structures, which are usually represented by YAML. Murano's
> environments are in YAML. Murano's VM-side commands are wrapped into YAML.
> The primary building blocks of Murano - Heat templates - are written on
> YAML. Actually, at a simplified level murano's workflows can be thought of
> as algorithms that just generate yaml fragments. So, it would be beneficial
> for Murano to manipulate with YAML-constructs at the DSL level. If we use
> YAML notation, yaml-blocks become first-class citizens in the language,
> while in a regular python-like language they would be just
> formatted-strings. For example, look at this code snippet which generates a
> heat template to deploy an instance:
> http://paste.openstack.org/show/65725/
> As you may see, the code on lines 7-11 is a Heat-template, seamlessly
> embedded inside Murano's workflow code. It has the variables right inside
> the template, and the Murano engine will substitute them with a
> user-specified (or workflow-computed) data
>
> Another reason for YAML: the notation is very easy to extend: you'll just
> have to add some new predefined key and a handler for its value: the format
> will not be broken, so older code will run out of the box, and even the
> newer code will most probably run fine on the older parser (the unknown
> sections will simply be ignored). This will allow us to extend the language
> without making any breaking-changes. The regular languages do not provide
> this flexibility: they will have problems if detecting unrecognised lexems
> or constructs.
>
> Last but not least: the perception. You are absolutely right when you say
> that this is actually a full programming language. However, we don't want
> to rush all its capabilities to unprepared developers. If some developer
> does not want any complexity, they may think about it as about some fancy
> configuration markup language: a declarative, heat-template-like header
> followed by a sequence of simple actions. And only if needed the power
> comes at your service: variable assignments, flow control, flexible data
> contracts, complex compositions, inheritance trees.. I can imagine a lot of
> scary stuff here J
> But at the same time, YAML's indent-based syntax will look familiar to
> python developers.
>
> Yes, everything comes at cost, and yaml may seem a bit bulky at the first
> glance. But I believe that people will get used to it soon enough, and the
> benefits are really important.
>
>
> I hope this answers your concern. We'll come up with more examples and
> ideas: this thing has just emerged, nothing is set in stone yet, I am
> actively seeking for feedback and ideas.  So thanks a loot for your
> question, I really appreciate it.
>
>
>
> --
> Regards,
> Alexander Tivelkov
>
>
> On Fri, Feb 14, 2014 at 6:41 PM, Joshua Harlow wrote:
>
>>  An honest question,
>>
>>  U are mentioning what appears to be the basis for a full programming
>> language (variables, calling other workflows - similar to functions) but
>> then u mention this is being stuffed into yaml.
>>
>>  Why?
>>
>>  It appears like u might as well spend the effort and define a grammar
>> and simplistic language that is not stuffed inside yaml. Shoving one into

Re: [openstack-dev] [Murano] Need a new DSL for Murano

2014-02-15 Thread Alexander Tivelkov 
Hi Joshua,

Thank you very much for you feedback!
This is really a great question, and it was the first question we've asked
ourselves when we started thinking about this new design.
We've considered both options: to have our own syntax (python-like,
java-like, something-else-like) or to use YAML. We've chosen the latest,
and here are the reasons.

The most important moment: this is not a general-purpose language, but a
domain-specific one. And Murano's domain is manipulating with complex
nested data-structures, which are usually represented by YAML. Murano's
environments are in YAML. Murano's VM-side commands are wrapped into YAML.
The primary building blocks of Murano - Heat templates - are written on
YAML. Actually, at a simplified level murano's workflows can be thought of
as algorithms that just generate yaml fragments. So, it would be beneficial
for Murano to manipulate with YAML-constructs at the DSL level. If we use
YAML notation, yaml-blocks become first-class citizens in the language,
while in a regular python-like language they would be just
formatted-strings. For example, look at this code snippet which generates a
heat template to deploy an instance: http://paste.openstack.org/show/65725/
As you may see, the code on lines 7-11 is a Heat-template, seamlessly
embedded inside Murano's workflow code. It has the variables right inside
the template, and the Murano engine will substitute them with a
user-specified (or workflow-computed) data

Another reason for YAML: the notation is very easy to extend: you'll just
have to add some new predefined key and a handler for its value: the format
will not be broken, so older code will run out of the box, and even the
newer code will most probably run fine on the older parser (the unknown
sections will simply be ignored). This will allow us to extend the language
without making any breaking-changes. The regular languages do not provide
this flexibility: they will have problems if detecting unrecognised lexems
or constructs.

Last but not least: the perception. You are absolutely right when you say
that this is actually a full programming language. However, we don't want
to rush all its capabilities to unprepared developers. If some developer
does not want any complexity, they may think about it as about some fancy
configuration markup language: a declarative, heat-template-like header
followed by a sequence of simple actions. And only if needed the power
comes at your service: variable assignments, flow control, flexible data
contracts, complex compositions, inheritance trees.. I can imagine a lot of
scary stuff here J
But at the same time, YAML's indent-based syntax will look familiar to
python developers.

Yes, everything comes at cost, and yaml may seem a bit bulky at the first
glance. But I believe that people will get used to it soon enough, and the
benefits are really important.


I hope this answers your concern. We'll come up with more examples and
ideas: this thing has just emerged, nothing is set in stone yet, I am
actively seeking for feedback and ideas.  So thanks a loot for your
question, I really appreciate it.



--
Regards,
Alexander Tivelkov


On Fri, Feb 14, 2014 at 6:41 PM, Joshua Harlow wrote:

>  An honest question,
>
>  U are mentioning what appears to be the basis for a full programming
> language (variables, calling other workflows - similar to functions) but
> then u mention this is being stuffed into yaml.
>
>  Why?
>
>  It appears like u might as well spend the effort and define a grammar
> and simplistic language that is not stuffed inside yaml. Shoving one into
> yaml syntax seems like it gets u none of the benefits of syntax checking,
> parsing, validation (highlighting...) and all the pain of yaml.
>
>  Something doesn't seem right about the approach of creating languages
> inside the yaml format (in a way it becomes like xsl, yet xsl at least has
> a spec and is well defined).
>
>  My 2 cents
>
> Sent from my really tiny device...
>
> On Feb 14, 2014, at 7:22 PM, "Alexander Tivelkov" 
> wrote:
>
>Hi folks,
>
>  Murano matures, and we are getting more and more feedback from our early
> adopters. The overall reception is very positive, but at the same time
> there are some complaints as well. By now the most significant complaint is
> is hard to write workflows for application deployment and maintenance.
>
> Current version of workflow definition markup really have some design
> drawbacks which limit its potential adoption. They are caused by the fact
> that it was never intended for use for Application Catalog use-cases.
>
>  I'll briefly touch these drawbacks first:
>
>
>1. Murano's workflow engine is actually a state machine, however the
>workflow markup does not explicitly define the states and transitions.
>2. There is no data isolation within any environment, which causes
>both potential security vulnerabilities and unpredictable workflow
>behaviours.
>3. There is no easy way to reuse the wo

Re: [openstack-dev] [Murano] Need a new DSL for Murano

2014-02-14 Thread Joshua Harlow 
An honest question,

U are mentioning what appears to be the basis for a full programming language 
(variables, calling other workflows - similar to functions) but then u mention 
this is being stuffed into yaml.

Why?

It appears like u might as well spend the effort and define a grammar and 
simplistic language that is not stuffed inside yaml. Shoving one into yaml 
syntax seems like it gets u none of the benefits of syntax checking, parsing, 
validation (highlighting...) and all the pain of yaml.

Something doesn't seem right about the approach of creating languages inside 
the yaml format (in a way it becomes like xsl, yet xsl at least has a spec and 
is well defined).

My 2 cents

Sent from my really tiny device...

On Feb 14, 2014, at 7:22 PM, "Alexander Tivelkov" 
mailto:ativel...@mirantis.com>> wrote:


Hi folks,


Murano matures, and we are getting more and more feedback from our early 
adopters. The overall reception is very positive, but at the same time there 
are some complaints as well. By now the most significant complaint is is hard 
to write workflows for application deployment and maintenance.

Current version of workflow definition markup really have some design drawbacks 
which limit its potential adoption. They are caused by the fact that it was 
never intended for use for Application Catalog use-cases.


I'll briefly touch these drawbacks first:

  1.  Murano's workflow engine is actually a state machine, however the 
workflow markup does not explicitly define the states and transitions.
  2.  There is no data isolation within any environment, which causes both 
potential security vulnerabilities and unpredictable workflow behaviours.
  3.  There is no easy way to reuse the workflows and their related procedures 
between several applications.
  4.  The markup uses JSONPath, which relies on Python’s 'eval' function. This 
is insecure and has to be avoided.
  5.  5. The workflow markup is XML-based, which is not a common practice in 
the OpenStack community.

So, it turns out that we have to design and implement a new workflow definition 
notation, which will not have any of the issues mentioned above.

At the same time, it should still allow to fully specify the configuration of 
any third-party Application, its dependencies with other Applications and 
define specific actions which are required for Application deployment, 
configuration and life cycle management.

This new notation should allow to do the following:


  *   List all the required configuration parameters and dependencies for a 
given application

  *   Validate user input and match it to the defined parameters

  *   Define specific deployment actions and their execution order

  *   Define behaviors to handle the events of changes in application’s 
environment


Also, it should satisfy the following requirements:


  *   Minimize the amount of configuration for common application parts, i.e. 
reuse existing configuration parts and add only difference specific to the 
application.

  *   Allow to use different deployment tools with using the same markup 
constructs. i.e. provide a high-level abstraction on the underlying tools 
(heat, shell, chef, puppet etc)

  *   For security reasons it should NOT allow to execute arbitrary operations 
- i.e. should allow to run only predefined set of meaningful configuration 
actions.



So, I would suggest to introduce a simple and domain specific notation which 
would satisfy these needs:

  *   Application dependencies and configuration properties are defined 
declaratively, in a way similar to how it is done in Heat templates.

  *   Each property has special constraints and rules, allowing to validate the 
input and applications relationship within the environment.

  *   The workflows are defined in imperative way: as a sequence of actions or 
method calls. This may include assigning data variables or calling the 
workflows of other applications.

  *   All of these may be packaged in a YAML format. The example may look 
something like this [1]


The final version may become a bit more complicated, but as the starting point 
this should look fine. I suggest to cover this in more details on our next IRC 
meeting on Tuesday.


Any feedback or suggestions are appreciated.



[1] https://etherpad.openstack.org/p/murano-new-dsl-example

--
Regards,
Alexander Tivelkov
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Murano] Need a new DSL for Murano

2014-02-14 Thread Alexander Tivelkov 
Hi folks,

Murano matures, and we are getting more and more feedback from our early
adopters. The overall reception is very positive, but at the same time
there are some complaints as well. By now the most significant complaint is
is hard to write workflows for application deployment and maintenance.

Current version of workflow definition markup really have some design
drawbacks which limit its potential adoption. They are caused by the fact
that it was never intended for use for Application Catalog use-cases.

I'll briefly touch these drawbacks first:


   1. Murano's workflow engine is actually a state machine, however the
   workflow markup does not explicitly define the states and transitions.
   2. There is no data isolation within any environment, which causes both
   potential security vulnerabilities and unpredictable workflow behaviours.
   3. There is no easy way to reuse the workflows and their related
   procedures between several applications.
   4. The markup uses JSONPath, which relies on Python's 'eval' function.
   This is insecure and has to be avoided.
   5. 5. The workflow markup is XML-based, which is not a common practice
   in the OpenStack community.

So, it turns out that we have to design and implement a new workflow
definition notation, which will not have any of the issues mentioned above.

At the same time, it should still allow to fully specify the configuration
of any third-party Application, its dependencies with other Applications
and define specific actions which are required for Application deployment,
configuration and life cycle management.

This new notation should allow to do the following:


   -

   List all the required configuration parameters and dependencies for a
   given application
   -

   Validate user input and match it to the defined parameters
   -

   Define specific deployment actions and their execution order
   -

   Define behaviors to handle the events of changes in application's
   environment


Also, it should satisfy the following requirements:


   -

   Minimize the amount of configuration for common application parts, i.e.
   reuse existing configuration parts and add only difference specific to the
   application.
   -

   Allow to use different deployment tools with using the same markup
   constructs. i.e. provide a high-level abstraction on the underlying tools
   (heat, shell, chef, puppet etc)
   -

   For security reasons it should NOT allow to execute arbitrary operations
   - i.e. should allow to run only predefined set of meaningful configuration
   actions.



So, I would suggest to introduce a simple and domain specific notation
which would satisfy these needs:

   -

   Application dependencies and configuration properties are defined
   declaratively, in a way similar to how it is done in Heat templates.
   -

   Each property has special constraints and rules, allowing to validate
   the input and applications relationship within the environment.
   -

   The workflows are defined in imperative way: as a sequence of actions or
   method calls. This may include assigning data variables or calling the
   workflows of other applications.
   -

   All of these may be packaged in a YAML format. The example may look
   something like this [1]


The final version may become a bit more complicated, but as the starting
point this should look fine. I suggest to cover this in more details on our
next IRC meeting on Tuesday.

Any feedback or suggestions are appreciated.


[1] https://etherpad.openstack.org/p/murano-new-dsl-example

--
Regards,
Alexander Tivelkov
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev