On 19/03/14 05:00, Stan Lagun wrote:
Steven,
Agree with your opinion on HOT expansion. I see that inclusion of
imperative workflows and ALM would require major Heat redesign and
probably would be impossible without loosing compatibility with previous
HOT syntax. It would blur Heat mission, confuse current users and rise a
lot of questions what should and what should not be in Heat. Thats why
we chose to built a system on top of Heat rather then expending HOT.
+1, I agree (as we have discussed before) that it would be a mistake to
shoehorn workflow stuff into Heat. I do think we should implement the
hooks I mentioned at the start of this thread to allow tighter
integration between Heat and a workflow engine (i.e. Mistral).
So building a system on top of Heat is good. Building it on top of
Mistral as well would also be good, and that was part of the feedback
from the TC.
To me, building on top means building on top of the languages (which
users will have to invest a lot of work in learning) as well, rather
than having a completely different language and only using the
underlying implementation(s).
Now I would like to clarify why have we chosen imperative approach with DSL.
You see a DSL as an alternative to HOT but it is not. DSL is alternative
to Python-encoded resources in Heat (heat/engine/resources/*.py).
Imagine how Heat would look like if you let untrusted users to upload
Python plugins to Heat engine and load them on the fly. Heat resources
are written in Python which is imperative language. So that MuranoPL for
the same reason.
We had this exact problem in Heat, and we decided to solve it with...
HOT:
http://lists.openstack.org/pipermail/openstack-dev/2013-April/007989.html
If I may be so bold as to quote myself:
"...no cloud operator in the world - not even your friendly local IT
department - is going to let users upload Python code to run in-memory
in their orchestration engine along with all of the other users' code.
"If only there were some sort of language for defining OpenStack
services that could be safely executed by users...
"Of course that's exactly what we're all about on this project :). So my
proposal is to allow users to define their own resource types using a
Heat template."
Which we did:
https://blueprints.launchpad.net/heat/+spec/provider-resource
We want application authors to be able to express application deployment
and maintenance logic of any complexity. This may involve communication
with 3rd party REST services (APIs of applications being deployed,
external services like DNS server API, application licensing server API,
billing systems, some hardware component APIs etc) and internal
OpenStack services like Trove, Sahara, Marconi and others including
those that are not incubated yet and those to come in the future. You
cannot have such things in HOT and when you required to you need to
develop custom resource in Python. Independence on custom plugins is
not good for Murano because they cannot be uploaded by end users and
thus he cannot write application definition that can be imported to/run
on any cloud and need to convince cloud administrator to install his
Python plugin (something that is unimaginable in real life).
Shouldn't Mistral be able to do all of those same things too?
Talking to existing OpenStack services doesn't seem hard - you could
write plugins for those, or save time (and save the user learning
another language) by using python-openstackclient and its syntax.
For everything else you have a small number of generic operations - e.g.
post to a Marconi queue (ReST API calls to untrusted services are
problematic from a security perspective) - and allow the user to handle
them in their own code in a language of their choice running either on
their own machine or on a sandboxed, metered Compute server, rather than
in a custom Turing-complete DSL running unmetered on the Murano server.
Because DSL is a way to write custom resources (in Heats terminology) it
has to be Turing-complete and have all the characteristics of
general-purpose language. It also has to have domain-specific features
because we cannot expect that DSL users would be as skilled as Heat
developers and could write such resources without knowledge on hosting
engine architecture and internals.
HOT DSL is declarative because all the imperative stuff is hardcoded
into Heat engine. Thus all is left for HOT is to define "state of the
world" - desired outcome. That is analogous to Object Model in Murano
(see [1]). It is Object Model that can be compared to HOT, not DSL. As
you can see it not more complex than HOT. Object Model is what end-user
produces in Murano. And he event don't need to write it cause it can be
composed in UI.
Now because DSL provides not only a way to write sandboxed isolated code
but also a lot of declarations (classes, properties, parameters,
inheritance and contracts) that are mostly not present in Python we
don't need Parameters or Output sections in Object Model because all of
this can be inferred from resource (classes) DSL declaration. Another
consequence is that most of the things that can be written wrong in HOT
can be verified on client side by validating classes' contracts without
trying to deploy the stack and then go through error log debugging.
This is being worked on:
https://blueprints.launchpad.net/heat/+spec/param-constraints
Because all resources' attributes types their constraints are known in
advance (note that resource attribute may be a reference to another
resource with constraints on that reference like "I want any (regular,
Galera etc) MySQL implementation") UI knows how to correctly compose the
environment and can point out your mistakes at design time. This is
similar to how statically typed languages like C++/Java can do a lot of
validation at compile time rather then in runtime as in Python.
Personally I would love to see many of this features in HOT. What is
your vision on this? What of the mentioned above can be contributed to
Heat? We definitely would like to integrate more with HOT and eliminate
all duplications between projects. I think that Murano and Heat are
complimentary products that can effectively coexist. Murano provides
access to all HOT features and relies on Heat for most of its
activities. I believe that we need to find an optimal way to integrate
Heat, Murano, Mistral, Solum, Heater, TOSCA, do some integration between
ex-Thermal and Murano Dashboard, be united regarding Glance usage for
metadata and so on.
+1
To me that implies that Murano should be a relatively thin wrapper that
ties together HOT and Mistral's DSL.
We are okay with throwing MuranoPL out if the issues
it solves would be addressed by HOT.
If you have a vision on how HOT can address the same domain MuranoPL
does or any plans for such features in upcoming Heat releases I would
ask you to share it.
[1] https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Object_model
On Wed, Mar 19, 2014 at 8:06 AM, Steven Dake <sd...@redhat.com
<mailto:sd...@redhat.com>> wrote:
Ruslan,
Some of my thoughts on the evolution of the HOT DSL to date.
On 03/18/2014 05:32 PM, Ruslan Kamaldinov wrote:
Here is my 2 cents:
I personally think that evolving Heat/HOT to what Murano needs
for it's use
cases is the best way to make PaaS layer of OpenStack to look
and feel as a
complete and fully integrated solution.
Standardising these things in a project like TOSCA is another
direction we all
should follow. I think that TOSCA is the place where developers
(like us),
application developers and enterprises can collaborate to
produce a common
standard for application lifecycle management in the clouds.
But before Murano contributors jump into direction of extending
HOT to the goal
of application (or system) lifecycle management, we need an
agreement that this
is the right direction for Heat/HOT/DSL and the Orchestration
program. There are
a lot of use cases that current HOT doesn't seem to be the right
tool to solve.
As it was said before, it's not a problem to collaborate on
extending it those
use cases. I'm just unsure if Heat team would like these use
cases to be solved
with Heat/HOT/DSL. For instance:
- definition of an application which is already exposed via REST
API. Think of
something like Sahara (ex. Savanna) or Trove developed
in-house for internal
company needs. app publishers wouldn't be happy if they'll
be forced to
develop a new resource for Heat
- definition of billing rules for an application
If everyone agrees that this is the direction we all should
follow, that we
should expand HOT/DSL to that scope, that HOT should be the
answer on "can you
express it?", then awesome - we can start speaking about
implementation details.
If it's not the direction these projects should follow then at
least finding
where Heat ends and Murano starts to avoid any functionality
duplication would
be great.
The HOT DSL for the most part, either by design or subconscious
development choices, enables the application of Miller's Law[1] in a
positive way. HOT as a DSL takes less then a few hours to learn and
use effectively. Its relative simplicity is its *key* advantage as
a DSL. DSL's by their very nature declare a desired state. It is
the responsibility of the DSL processor to convert that desired
state into reality. On a fundamental level, this is precisely what
Heat does.
A DSL by its very definition is meant to express a desired outcome
without specifying the intermediate steps. To express the
intermediate steps would require recording state in variables and
offering conditional operations on those variables. This implies
individual steps in the processing of the input to the language. If
HOT were to add these sorts of features, it would no longer be a
DSL, but a general purpose language (perhaps less general purpose
then python or C). A DSL is by definition a declarative language. I
don't like the idea of expanding the scope of HOT to add an
imperative model of operation.
Learning imperative languages takes inordinately more time and
brainpower then learning declarative languages, especially those
which generally follow the advantages provided by languages
operating inside the constraints of Miller's Law. We want Heat to
be dead simple to explain and learn. Realistically I'd like folks
to be able to write a template in under an hour with 15 minutes of
explanation, and I think we have hit that mark.
The idea of expanding the scope of the Heat APIs and engine to
include ALM and Workflow don't make sense to me from an engineering
perspective. It over-complicates the code base. I know we have
already covered those thoughts in detail on the mailing list
previously and the Murano folks agree that is a bad idea.
I see a parallel between expanding the scope of HOT to support ALM
and Workflow and expanding the scope of the heat-engine in the same
fashion that is not appealing. What would make more sense is to
follow the general laws of Unix (do one thing, do it well) and layer
these other possibly imperative languages on top of Heat using HOT
and the Heat APIs to implement such imperative programming models.
Then if someone really wanted to invest in the complexity of ALM or
Workflow, they may be more willing to invest in learning the
complexity of a new imperative programming language.
My personal opinion is expanding the scope of HOT to include
imperative programming models is not desirable for Heat in
isolation. I understand such an outcome may be appealing as a
holistic approach to handling the entire orchestration space, but
feel the costs of learning an imperative model for HOT do not pay
for the advantages of having only one language to program all the
things.
I see no issue with HOT remaining simple and tidy focused entirely
on orchestration (taking a desired state and converting that into
reality) with some other imperative language layered on top to
handle workflow and ALM. I believe this separation of concerns is
best for OpenStack and should be the preferred development path.
Regards,
-steve
[1]
http://en.wikipedia.org/wiki/__The_Magical_Number_Seven,___Plus_or_Minus_Two
<http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two>
Thanks,
Ruslan
On Wed, Mar 19, 2014 at 2:07 AM, Keith Bray
<keith.b...@rackspace.com <mailto:keith.b...@rackspace.com>> wrote:
Georgy,
In consideration of the "can you express it" instead of the
"who will
generate it," I see Heat's HOT evolving to support the
expression of complex
multi-tier architectures and applications (I would argue you
can already do
this today, perhaps with some additional features desired,
e.g. Ability to
define cloud workflows and workflow execution rules which
could come when we
have a workflow service like Mistral). Therefore, I would
encourage Murano
contributors to consider whether they can help make Heat
sufficiently cover
desired use cases. I have never viewed Heat templates as
isolated
components of a multi-tier architecture. Instead, a single
template or a
combination of master/subordinate templates together (using
references,
nesting, or inclusion) could express the complete
architecture, both
infrastructure and applications.
If I've read your previous comments and threads correctly,
you desire a way
to express System Lifecycle Management across multiple
related applications
or components, whereby you view the System as a grouping of
independently
developed and/or deployed (but systematically related)
"components," whereby
you view Components as individual disconnected Heat
templates that
independently describe different application stacks of the
System. Did I
get that correct? If so, perhaps the discussion here is
one of "scope" of
what can or should be expressed in a Heat template. Is it
correct to state
that your argument is that a separate system (such as
Murano) should be used
to express System Lifecycle Management as I've defined it
here? If so, why
could we not use the Heat DSL to also define the System?
The System
definition could be logically separated out into its own
text file... But,
we'd have a common DSL syntax and semantics for both lower
level and higher
level component interaction (a building block effect of sorts).
As for "who will generate it," ( with "it" being the Heat
multi-tier
application/infrastructure definition) I think that question
will go through
a lot more evolution and could be any number of sources:
e.g. Solum, Murano,
Horizon, Template Author with a text editor, etc.
Basically, I'm a +1 for as few DSLs as possible. I support
the position that
we should evolve HOT if needed vs. having two separate DSLs
that are both
related to expressing application and infrastructure semantics.
Workflow is quite interesting ... Should we be able to
express imperative
workflow semantics in HOT? Or, should we only be able to
declare workflow
configurations that get configured in a service like Mistral
whereby
Mistral's execution of a workflow may need to invoke Heat
hooks or Stack
Updates? Or, some other solution?
I look forward to a design discussion on all this at the
summit... This is
fun stuff to think about!
-Keith
From: Georgy Okrokvertskhov <gokrokvertsk...@mirantis.com
<mailto:gokrokvertsk...@mirantis.com>>
Reply-To: "OpenStack Development Mailing List (not for usage
questions)"
<openstack-dev@lists.__openstack.org
<mailto:openstack-dev@lists.openstack.org>>
Date: Tuesday, March 18, 2014 1:49 PM
To: "OpenStack Development Mailing List (not for usage
questions)"
<openstack-dev@lists.__openstack.org
<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
I see this in the following way - who will generate HOT
template for my
complex multi-tier applications when I have only templates
for components?
_________________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org
<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_________________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org
<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_________________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org
<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
--
Sincerely yours
Stanislav (Stan) Lagun
Senior Developer
Mirantis
35b/3, Vorontsovskaya St.
Moscow, Russia
Skype: stanlagun
www.mirantis.com <http://www.mirantis.com/>
sla...@mirantis.com <mailto:sla...@mirantis.com>
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
