On 21/03/14 18:58, Stan Lagun wrote:
Zane,
I appreciate your explanations on Heat/HOT. This really makes sense.
I didn't mean to say that MuranoPL is better for Heat. Actually HOT is
good for Heat's mission. I completely acknowledge it.
I've tried to avoid comparison between languages and I'm sorry if it
felt that way. This is not productive as I don't offer you to replace
No problem, I didn't interpret it that way. I just wanted to use it as
an opportunity to document a bit more information about Heat works that
may be helpful to you.
HOT with MuranoPL (although I believe that certain elements of MuranoPL
syntax can be contributed to HOT and be valuable addition there). Also
people tend to protect what they have developed and invested into and to
be fair this is what we did in this thread to great extent.
Yes, we are all human and this is an unavoidable part of life :)
What I'm trying to achieve is that you and the rest of Heat team
understand why it was designed the way it is. I don't feel that Murano
can become full-fledged member of OpenStack ecosystem without a bless
from Heat team. And it would be even better if we agree on certain
design, join our efforts and contribute to each other for sake of
Orchestration program.
Thanks. To be clear, this is what I am looking for. At the beginning of
this thread I proposed a very simple possible implementation, and I'm
wanting folks to tell me what is missing from that model that would
justify a more complicated approach.
I'm sorry for long mail texts written in not-so-good English and
appreciate you patience reading and answering them.
Having said that let me step backward and explain our design decisions.
Cloud administrators are usually technical guys that are capable of
learning HOT and writing YAML templates. They know exact configuration
of their cloud (what services are available, what is the version of
OpenStack cloud is running) and generally understands how OpenStack
works. They also know about software they intent to install. If such guy
wants to install Drupal he knows exactly that he needs HOT template
describing Fedora VM with Apache + PHP + MySQL + Drupal itself. It is
not a problem for him to write such HOT template.
I'm aware that TOSCA has these types of constraints, and in fact I
suggested to the TOSCA TC that maybe this is where we should draw the
line between Heat and some TOSCA-compatible service: HOT should be a
concrete description of exactly what you're going to get, whereas some
other service (in this case Murano) would act as the constraints solver.
e.g. something like an image name would not be hardcoded in a Murano
template, you have some constraints about which operating system and
what versions should be allowed, and it would pick one and pass it to
Heat. So I am interested in this approach.
The worst outcome here would be to end up with something that was
equivalent to TOSCA but not easily translatable to the TOSCA Simple
Profile YAML format (currently a Working Draft). Where 'easily
translatable' preferably means 'by just changing some names'. I can't
comment on whether this is the case as things stand.
Note that such template would be designed for very particular
configuration. There are hundreds of combinations that may be used to
install that Drupal - use RHEL/Windows/etc instead of Fedora, use
ngnix/IIS/etc instead of Apache, use FastCGI instead of mod_php,
PostgreSQL instead of MySQL. You may choose to have all software on
single VM or have one VM for database and another for Drupal. There are
also constraints to those combinations. For example you cannot have
Fedora + IIS on the same VM. You cannot have Apache and Drupal on
different VMs.
So the HOT template represent fixed combination of those software
components. HOT may have input parameters like "username" or
"dbImageName" but the overall structure of template is fixed. You cannot
have template that choses whether to use Windows or Linux based on
parameter value.
As Thomas mentioned, CloudFormation now has conditionals. If we were to
add this feature in HOT (which seems likely), you would actually be able
to do that.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html
You cannot write HOT that accepts number of instances
it allowed to create and then decide what would be installed on each of
them. This is just not needed for Heat users.
The path we've gone down in Heat is to introduce software component
resources so that the definition of the software component to install is
separated from the definition of the server it runs on, and may even be
in a separate template file.
So you can write a template that launches a single server with (e.g.)
both Wordpress and MySQL installed, or you can have a separate template
(though with the conditional stuff above it could theoretically be the
same template even) that launches two servers with Wordpress on one and
MySQL on the other, and in both cases you can use the same definitions
of Wordpress and MySQL without having to copy-and-paste.
With Murano the picture looks the opposite. Typical Murano user is a guy
who bought account from cloud hosting vendor (cloud operator) and want
to run some software in the cloud. He may not even be aware that it is
OpenStack. He knows nothing about programming in general and Heat in
particular. He doesn't want to write YAMLs. He may not know how exactly
Drupal is installed and what components it consists of.
So what he does is he goes to his cloud (Murano) dashboard, browses
through application catalog, finds Drupal and drags it onto his
environment board (think like Visio-style designer). He can stop at this
point, click "deploy" button and the system will deploy Drupal. In
another words the system (or maybe better to say cloud operator or
application developer) decides what set of components is going to be
installed (like 1 Fedora VM for MySQL and 1 CentOS VM for
Apache-PHP-Drupal). But user may decide he wants to customize his
environment. He digs down and sees that Drupal requires database
instance and the default is MySQL. He clicks on a button to see what are
other options available for that role.
I mentioned on IRC the other day that I find this problem fundamentally
un-interesting, and I'll explain why. I should note that this is
strictly my personal opinion, probably in no way resembles the views of
anyone else involved in Heat, and is unlikely to persuade anyone.
This guy, your hypothetical typical Murano user. Two minutes ago he
didn't know what OpenStack is, or that Drupal requires a database. But
now it's suddenly *imperative* that he be able to swap out MySQL for
Postgres in this pre-packaged application???
This problem is not interesting to me because nobody has this problem.
Except maybe bored micromanagers looking to mark their territory by
putting their stamp on some part of a project. (See
http://programmers.stackexchange.com/a/122148.) And I'm not interested
in them either.
(Looking up the appropriate Dilbert references to insert here is left as
an exercise for the reader.)
If the developer wants to support a MySQL version and a Postgres
version, let them release two versions. They will know, of course, that
the number of things they need to test increases combinatorially with
the number of options they provide, and may well not be as keen as you
think to make everything customisable. (Of course the right answer here,
and the one that substantially all application authors will choose, is
to use whatever built-in DBaaS the operator provides.)
In Heat HOT developer is the user. But in Murano those are completely
different roles. There are developers that write application definitions
I don't really see a difference. In both HOT and Murano the person
launching the application may or may not be the same person who wrote
the application definition. There's no reason e.g. that both should not
be made available in a Glance-like application catalog.
(that is DSL code) and there are end users who compose environments from
those applications (components). Application developers may have nothing
to do with particular cloud their application deployed on. As for Drupal
application the developer knows that Drupal can be run with MySQL or
PostgreSQL. But there may be many compatible implementations of those
DBMSes - Galera MySQL, TroveMySQL, MMM MySQL etc. So to get a list of
what components can be placed in a database role Murano needs to look at
all applications in Application Catalog and find which of them are
compatible with MySQL and PostgreSQL so that user could choose what
implementation is better suits his needs (trade performance for
high-availability etc.).
Who are these people, who didn't know, or barely knew, that Drupal used
a database but have the in-depth technical knowledge to make
finely-nuanced trade-offs between performance and high-availability for
a Drupal database? That such people exist in large numbers beggars belief.
User can go deeper and to decide that he wants that MySQL instance (this
can be 1 or more VMs depending on implementation) to be shared between
Drupal and another application in that environment (say WordPress). He
can go even deeper to VM level and decide that he wants to have
WordPress, Drupal, Apcache and slave MySQL node on one VM and MySQL
master node on another.
The ultimate goal is to create sort of construction kit for
applications. Cloud LEGO.
Ever tried to build anything substantial, with a lot of moving parts, in
Lego? I have. Spoiler Alert: What happens is that all of the tolerances
start to stack up and forces start to build up. Everything that moves
jams, and everything else becomes really fragile.
And let users build environments of any
complexity from small components while providing reasonable default so
that one-click deployment would also be possible. And such system to be
useful the design process need to be guided and driven by UI. System
must know what combination of components are possible and what are not
and not let user create Microsoft IIS hosted on Fedora.
If I may go all editorial on you again, this sounds like the same thing
we've been hearing since the 1970s: "When everything is object oriented,
non-technical users will be able to program just by plugging together
existing chunks of code." Except it hasn't ever worked. 35+ years. No
results.
This leads us to following design decisions:
a. We need a DSL that would allow describing those small components
independently in such a way that the environment could be composed from
components written by different people that are not aware of each other
b. Components need to have properties (parameters in Heat terms).
Properties may be of different types (strings, numbers, booleans etc.).
There can be constraints on property values ("not null", "length > 5").
c. Components may reference (require, make use of) other components
(Drupal uses MySQL, hosted on Apache etc). There can be constraints for
those references.
d. Constraints for both properties and references can be *very* complex
and indirect. Like "IIS can be deployed on any VM who's image is known
to be Windows with version >= WinServer2003". Or "my app requires 10 to
20 Windows instances that are all member of the same ActiveDirectory
domain". Or "I have 2 properties: minInstanceCount and maxInstanceCount
and constraint maxInstaceCount >= minInstanceCount". So we need a
language to express conditions of any complexity.
e. Both properties and references must be declared in a way that would
make possible for UI dashboard to discover their existence, ask user
appropriate input and validate the constraints on the fly.
f. Properties and especially references can have complex structure
(lists, maps, combination of them) rather than plain scalars. For
example ActiveDirectory consists of (e.g. references) *list* of domain
controllers. Constraints must respect that and be capable of validating
things like len(list) >= 2, all elements in list have some predicate
true, there is at least one element with desired set of properties etc.
g. Language need to have some sort mechanism to express polymorphism.
E,g, GaleraMySQL *is* MySQL (and thus can be used anywhere where "just
MySQL" can be). This also implies ability to have declare interfaces
(class contracts) and ability to implement those interface to that it
would be possible that two components are conform to the same interface.
h. If we want components to be reusable and easy to write we need to
have ability to one component to inherit/extend another. Inheritance
implies ability to override and customize some methods
i. We don't want to do deployment ourself. We don't intend to replace
Heat, Puppet etc. So we need a way to construct Heat templates that will
do the job. We need to map component's properties to
HOT parameters. This requires a language to express those bindings
j. Because there can be lists of references (or properties) it should be
possible to express something like
for(domain_controller in self.controllers):
hot_template.insert(resources_for_particular_controller)
or
if self.deploy_on_windows:
generate_windows_hot_template()
else:
generate_linux_hot_template()
or
for i in range(self.instance_count):
hot_template.insert(generate_instance())
k. Need to provide most simple method to compose stack from pre-made HOT
snippets and avoid dynamic generation when its not really needed
l. Need to have some standard format to represent view of the world that
is what was designed in dashboard. In Murano it is JSON that describes
object graph with object property values and references to another
objects. This is somewhat similar to simple HOT template but with
hierarchical structure
I think at this point it becomes obvious that above describes
object-oriented Turing-complete language. And the most important
property of that language is that *everything* is declared in a way
useful to be accessed from dashboard/app catalog. You not often see
properties/references/constraints declared in dynamic language. I've
answered several times already why this language cannot be JavaScript or
Lua. And this is just on the surface. There are many things under the
hood that makes MuranoPL stand out from all embeddable languages I know
and thats for a good reason.
And I didn't even mentioned ALM aspects, orchestration, and many many
other things that go far beyond what can be expressed by HOT-style DSL
Maybe you _should_ mention those, because those sound like the
interesting things - the things HOT is missing and should be added
either to HOT or to some encapsulating format like one provided by Murano.
-=-=-=-=-=-=-=-
Now few words ( :-))) on Mistral. In theory Mistral could be capable to
handle MuranoPL workflows and do what MuranoPL does on his own. But here
are the obstacles on this path:
1. Mistral follows Amazon SWF model and not BPEL model. 90% of SWF's
power comes from the fact the you don't work with it directly. With SWF
you use Java bindings/SDK and not consume it directly. So you still
write code in Turing-complete sophisticated language Java
Or, indeed, any language of your choosing.
2. Our initial idea was to translate MuranoPL into Mistral workflows
(similar to compiling C++ to assembly) but it turns to be too difficult
to implement
3. The DSL Mistral currently has is very very limited and academic. You
have to be real genius with a twisted mind to write real-life workflows
with Mistral DSL.
If this is correct then it sounds as all of us should be making sorting
out the Mistral DSL our top priority.
4. Mistral is at PoC stage and it would take him years to grow to
something that is as powerful as MuranoPL (or maybe BPEL)
IMHO there's good reason to hope that it *never* gets as powerful as those.
5. Mistral does not need all that OOP stuff, inheritance, constraints
etc. And in practice it is very hard to separate declarations from the
workflow code as they are very coupled in their nature
Why? Because of all the $this stuff? From the few I have looked at, all
of the stuff that is being done imperatively in the workflows using the
object's properties could have been done declaratively in Heat, just
passing the properties as parameters to a template.
6. DSL is a hart and brain of Murano. It is the most vital part of it.
You cannot develop system of such level as Murano without having control
over it. Any decision taken by Mistral team can become fatal for Murano.
You depend on parts of HOT already, don't you have the same issue with Heat?
Anyway I hope that some day we will be able to somehow merge our
DSLs/codebase/whatever
This seems unlikely to happen by accident.
I think the difference between the approach you've taken and what I'm
advocating for is that it's very top-down. Start with a problem, design
a DSL around it, stick it in OpenStack and "hope that some day we will
be able to somehow merge". I would much prefer a more bottom-up
approach. We have these tools already (or in the case of Mistral, we're
in the process of developing them)... in what ways can we extend them
incrementally to help them cover a wider range of use cases? (Folks who
were around a year ago when talk of moving Heat closer to TOSCA first
began will tell you that I have been pretty consistent in pushing this
approach.)
cheers,
Zane.
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
