Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Zane Bitter]

Late to the party, but this comparison seems misleading to me...

On 26/01/16 04:46, Steven Hardy wrote:

It's one more thing, which is already maintained and has an active
community, vs yet-another-bespoke-special-to-tripleo-thing.  IMHO we have
*way*  too many tripleo specific things already.

However, lets look at the "python knowledge" thing in a bit more detail.

Let's say, as an operator I want to wire in a HTTP call to an internal asset
management system.  The requirement is to log an HTTP call with some
content every time an overcloud is deployed or updated.  (This sort of
requirement is*very*  common in enterprise environments IME)

In the mistral case[1], the modification would look something like:

http_task:
   action: std.http url='assets.foo.com' 

You'd simply add two lines to your TripleO deployment workflow yaml[2]:

Now, consider the bespoke API case.  You have to do some or all of the
following:

- Find the python code which handles deployment and implements the workflow
- Pull and fork the code base, resolve any differences between the upstream
   version and whatever pacakged version you're running
- Figure out how to either hack in your HTTP calls via a python library, or
   build a new plugin mechanism to enable out-of-tree deployment hooks
- Figure out a bunch of complex stuff to write unit tests, battle for
   weeks/months to get your code accepted upstream (or, maintain the fork
   forever and deal with rebasing, packaging, and the fact that your entire
   API is no longer supported by your vendor because you hacked on it)


If I were doing it I would write a piece of WSGI middleware - a highly 
standardised thing, not specific to TripleO or even OpenStack, that a 
non-python-ninja could easily figure out from StackOverflow - then 
deploy it on the undercloud machine and add it into the paste pipeline.


  class AssetControl(wsgi.Middleware):
  def process_request(self, req):
  requests.get('assets.foo.com', data={'some':'arguments'})

It's true that the 'deploy it on the machine' step is probably more 
complicated than the 'upload a new workflow' one. OTOH most sysadmins 
are *really* good at installing stuff on a machine, and there is a HUGE 
advantage in not ever having to merge your forked workflow definitions.



Which of these is most accessible to a traditional non-python-ninja
sysadmin?


Given the above, I would genuinely have to say the second. WSGI and 
Requests are *very* well documented *everywhere*.


Though the biggest difference, I suspect, comes when you have to 
incorporate some logic in there. Say you want to log the request to a 
different server when the user's manager's oldest pet's middle name 
begins with 'Q' or something. (I would venture to speculate that this 
kind of requirement is, ahem, not all that uncommon in enterprise 
environments either ;) In Python this is pretty trivial and you always 
have StackOverflow to help when you get stuck; if you're having to 
implement it in some obscure DSL that knows nothing about your 
application then you could be in for a world of hurt.


cheers,
Zane.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Mon, 2016-02-01 at 15:17 +0600, Renat Akhmerov wrote:
> Hi,
> 
> I’ve read only part of letters in this huge interesting thread so far
> but I’d like to try to jump in and give some comments.
> 
> API
> I personally don’t support using Mistral API as is. Maybe you came to
> agreement already about that, don’t know. I think that API should
> reflect user needs in specific functionality in the most suitable and
> natural way and provide an abstraction over implementation details
> such as a backend technology (that can easily change).
> If there are processes though on the backend that need to be HA,
> stateful and you need to have a fine-grained control over them (stop,
> resume, etc.) and monitoring of all the state then I’d recommend
> consider Mistral for sure.

Although this thread has gone in many directions I think the primary
need we are looking for with Mistral is to help us expose deployment
workflows to both a CLI and UI. Some of the workflows do have state
(say a set of templates, and parameters) which we would like to be able
to manage equally well regardless of which tool the end user chooses
(again CLI or UI). The benefit of Mistral is that as an in-cloud
generic workflow tool it sits nicely within the TripleO stack, allows
us to customize what we need without writing boilerplate code we would
prefer not to maintain. And because it is generic it can do things like
call Heat, or be called by Heat all natively.

> 
> Mistral & Ansible comparison
> IMO, it’s not 100% correct to compare Mistral with Ansible for a
> number of reasons:
> Ansible is a less general technology, it’s sharpened for
> configuration management and deployment tasks hence it has lots and
> lots of specific things in the language (mostly properties).
> Mistral workflow language is not 100% replica of Ansible Playbooks,
> there are significant differences in concepts, data model, execution
> model (e.g. tasks always run sequentially in Ansible whereas in
> Mistral they can be parallelized in a number of ways). Mistral
> provides graph-based workflows.
> Mistral in its core assumes pluggability for different workflow types
> each one of them can have absolutely different semantics. For
> example, currently it has workflow types “direct” (default) and
> “reverse” that works according to a different logic. It’s pretty easy
> to extend it with something else, for example, task priority based
> workflow.
> As I mentioned before Mistral is mostly about state management: all
> workflows, subworkflows, tasks and actions have state observable
> through API. From this standpoint, it is more like taskflow with an
> important difference that Mistral is a service. It provides
> functionality to manage life cycle of workflows such as stop, resume,
> recover from errors. It also provide various policies that can be
> applied to workflow tasks such as “retry”, “timeout”, “pause-before”
> etc.
> As far as I understand there’s a serious difference in Ansible and
> Mistral architecture. Mistral is naturally based on asynchronous
> processing model that makes it possible to have asynchronous actions
> w/o having to use polls and allows engine to be scalable naturally.
> In other words, each engine instance is stateless.
> 
> As far as languages, it requires significant work on comparison. In a
> nutshell, Ansible has a lot of stuff that’s missing in Mistral and
> vice versa. For example, Ansible has lots of nice things like various
> looping capabilites expressed as “with_XXX” whereas Mistral can only
> iterate over lists.
> 
> Thanks

Thanks for this summary. Lots of good points in here and I think
perhaps it would be nice to see some side by side examples of trying to
use the tools for various things. Highlighting where Mistral and or
Ansible excel in certain areas. The biggest missing feature we'd need
for Ansible to be a solution for us would be an API (aka something like
Tower). Without that or something equivalent (like a Mistral or custom
generic API on top of Ansible) I'm not sure we could consider Ansible
as a solution for our needs at the moment.

Dan

> 
> Will keep reading...
> 
> Renat Akhmerov
> @ Mirantis Inc.
> 
> _
> _
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Renat Akhmerov]

Hi,

I’ve read only part of letters in this huge interesting thread so far but I’d 
like to try to jump in and give some comments.

API
I personally don’t support using Mistral API as is. Maybe you came to agreement 
already about that, don’t know. I think that API should reflect user needs in 
specific functionality in the most suitable and natural way and provide an 
abstraction over implementation details such as a backend technology (that can 
easily change).
If there are processes though on the backend that need to be HA, stateful and 
you need to have a fine-grained control over them (stop, resume, etc.) and 
monitoring of all the state then I’d recommend consider Mistral for sure.

Mistral & Ansible comparison
IMO, it’s not 100% correct to compare Mistral with Ansible for a number of 
reasons:
Ansible is a less general technology, it’s sharpened for configuration 
management and deployment tasks hence it has lots and lots of specific things 
in the language (mostly properties).
Mistral workflow language is not 100% replica of Ansible Playbooks, there are 
significant differences in concepts, data model, execution model (e.g. tasks 
always run sequentially in Ansible whereas in Mistral they can be parallelized 
in a number of ways). Mistral provides graph-based workflows.
Mistral in its core assumes pluggability for different workflow types each one 
of them can have absolutely different semantics. For example, currently it has 
workflow types “direct” (default) and “reverse” that works according to a 
different logic. It’s pretty easy to extend it with something else, for 
example, task priority based workflow.
As I mentioned before Mistral is mostly about state management: all workflows, 
subworkflows, tasks and actions have state observable through API. From this 
standpoint, it is more like taskflow with an important difference that Mistral 
is a service. It provides functionality to manage life cycle of workflows such 
as stop, resume, recover from errors. It also provide various policies that can 
be applied to workflow tasks such as “retry”, “timeout”, “pause-before” etc.
As far as I understand there’s a serious difference in Ansible and Mistral 
architecture. Mistral is naturally based on asynchronous processing model that 
makes it possible to have asynchronous actions w/o having to use polls and 
allows engine to be scalable naturally. In other words, each engine instance is 
stateless.

As far as languages, it requires significant work on comparison. In a nutshell, 
Ansible has a lot of stuff that’s missing in Mistral and vice versa. For 
example, Ansible has lots of nice things like various looping capabilites 
expressed as “with_XXX” whereas Mistral can only iterate over lists.

Thanks

Will keep reading...

Renat Akhmerov
@ Mirantis Inc.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Renat Akhmerov]

> On 02 Feb 2016, at 03:55, Dan Prince  wrote:
> 
> On Mon, 2016-02-01 at 15:17 +0600, Renat Akhmerov wrote:
>> Hi,
>> 
>> I’ve read only part of letters in this huge interesting thread so far
>> but I’d like to try to jump in and give some comments.
>> 
>> API
>> I personally don’t support using Mistral API as is. Maybe you came to
>> agreement already about that, don’t know. I think that API should
>> reflect user needs in specific functionality in the most suitable and
>> natural way and provide an abstraction over implementation details
>> such as a backend technology (that can easily change).
>> If there are processes though on the backend that need to be HA,
>> stateful and you need to have a fine-grained control over them (stop,
>> resume, etc.) and monitoring of all the state then I’d recommend
>> consider Mistral for sure.
> 
> Although this thread has gone in many directions I think the primary
> need we are looking for with Mistral is to help us expose deployment
> workflows to both a CLI and UI. Some of the workflows do have state
> (say a set of templates, and parameters) which we would like to be able
> to manage equally well regardless of which tool the end user chooses
> (again CLI or UI). The benefit of Mistral is that as an in-cloud
> generic workflow tool it sits nicely within the TripleO stack, allows
> us to customize what we need without writing boilerplate code we would
> prefer not to maintain. And because it is generic it can do things like
> call Heat, or be called by Heat all natively.

Yeah, makes perfect sense to me. It’s one of the cool things that workflows
in general provide: ability to build a template of a process where steps could
be reimplemented in different ways.

Renat Akhmerov
@ Mirantis Inc.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Renat Akhmerov]

> On 22 Jan 2016, at 20:01, Dan Prince  wrote:
> 
> https://github.com/dprince/tripleo-common/tree/mistral

Just a couple of cosmetic notes:
* this example can be slightly simplified by removing “type: direct” since by 
default workflows are of “direct” type. 
* action params don’t have to go on a single line (sometimes bad for 
readability), instead we can use “input:” task 
property and specify params as a regular dict.

Thanks

Renat Akhmerov
@ Mirantis Inc.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Steven Hardy]

On Wed, Jan 27, 2016 at 10:22:54AM -0500, Dan Prince wrote:
> On Wed, 2016-01-27 at 09:36 -0500, Dan Prince wrote:
> > On Wed, 2016-01-27 at 14:32 +0100, Jiri Tomasek wrote:
> > > On 01/26/2016 09:05 PM, Ben Nemec wrote:
> > > > On 01/25/2016 04:36 PM, Dan Prince wrote:
> > > > > On Mon, 2016-01-25 at 15:31 -0600, Ben Nemec wrote:
> > > > > > On 01/22/2016 06:19 PM, Dan Prince wrote:
> > > > > > > On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> > > > > > > > So I haven't weighed in on this yet, in part because I
> > > > > > > > was
> > > > > > > > on
> > > > > > > > vacation
> > > > > > > > when it was first proposed and missed a lot of the
> > > > > > > > initial
> > > > > > > > discussion,
> > > > > > > > and also because I wanted to take some time to order my
> > > > > > > > thoughts
> > > > > > > > on
> > > > > > > > it.
> > > > > > > >   Also because my initial reaction...was not conducive to
> > > > > > > > calm and
> > > > > > > > rational discussion. ;-)
> > > > > > > > 
> > > > > > > > The tldr is that I don't like it.  To explain why, I'm
> > > > > > > > going to
> > > > > > > > make
> > > > > > > > a
> > > > > > > > list (everyone loves lists, right? Top $NUMBER reasons we
> > > > > > > > should
> > > > > > > > stop
> > > > > > > > expecting other people to write our API for us):
> > > > > > > > 
> > > > > > > > 1) We've been down this road before.  Except last time it
> > > > > > > > was
> > > > > > > > with
> > > > > > > > Heat.
> > > > > > > >   I'm being somewhat tongue-in-cheek here, but expecting
> > > > > > > > a
> > > > > > > > general
> > > > > > > > service to provide us a user-friendly API for our
> > > > > > > > specific
> > > > > > > > use
> > > > > > > > case
> > > > > > > > just
> > > > > > > > doesn't make sense to me.
> > > > > > > We've been down this road with Heat yes. But we are
> > > > > > > currently
> > > > > > > using
> > > > > > > Heat for some things that we arguable should be (a
> > > > > > > workflows
> > > > > > > tool
> > > > > > > might
> > > > > > > help offload some stuff out of Heat). Also we haven't
> > > > > > > implemented
> > > > > > > custom Heat resources for TripleO either. There are mixed
> > > > > > > opinions
> > > > > > > on
> > > > > > > this but plugging in your code to a generic API is quite
> > > > > > > nice
> > > > > > > sometimes.
> > > > > > > 
> > > > > > > That is the beauty of Mistral I think. Unlike Heat it
> > > > > > > actually
> > > > > > > encourages you to customize it with custom Python actions.
> > > > > > > Anything
> > > > > > > we
> > > > > > > want in tripleo-common can become our own Mistral action
> > > > > > > (these get
> > > > > > > registered with stevedore entry points so we'd own the
> > > > > > > code)
> > > > > > > and
> > > > > > > the
> > > > > > > YAML workflows just tie them together via tasks.
> > > > > > > 
> > > > > > > We don't have to go off and build our own proxy deployment
> > > > > > > workflow
> > > > > > > API. The structure to do just about anything we need
> > > > > > > already
> > > > > > > exists
> > > > > > > so
> > > > > > > why not go and use it?
> > > > > > > 
> > > > > > > > 2) The TripleO API is not a workflow API.  I also largely
> > > > > > > > missed
> > > > > > > > this
> > > > > > > > discussion, but the TripleO API is a _Deployment_
> > > > > > > > API.  In
> > > > > > > > some
> > > > > > > > cases
> > > > > > > > there also happens to be a workflow going on behind the
> > > > > > > > scenes,
> > > > > > > > but
> > > > > > > > honestly that's not something I want our users to have to
> > > > > > > > care
> > > > > > > > about.
> > > > > > > Agree that users don't have to care about this.
> > > > > > > 
> > > > > > > Users can get as involved as they want here. Most users I
> > > > > > > think
> > > > > > > will
> > > > > > > use python-tripleoclient to drive the deployment or the new
> > > > > > > UI.
> > > > > > > They
> > > > > > > don't have to interact with Mistral directly unless they
> > > > > > > really
> > > > > > > want
> > > > > > > to. So whether we choose to build our own API or use a
> > > > > > > generic one
> > > > > > > I
> > > > > > > think this point is mute.
> > > > > > Okay, I think this is a very fundamental point, and I believe
> > > > > > it gets
> > > > > > right to the heart of my objection to the proposed change.
> > > > > > 
> > > > > > When I hear you say that users will use tripleoclient to talk
> > > > > > to
> > > > > > Mistral, it raises a big flag.  Then I look at something like
> > > > > > https://github.com/dprince/python-tripleoclient/commit/77ffd2
> > > > > > fa
> > > > > > 7b1642
> > > > > > b9f05713ca30b8a27ec4b322b7
> > > > > > and the flag gets bigger.
> > > > > > 
> > > > > > The thing is that there's a whole bunch of business logic
> > > > > > currently
> > > > > > sitting in the client that shouldn't/can't be there.  There
> > > > > > are
> > > > > > historical reasons for it, but the important thing is that
> > > > > > the
> > > > > > 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Tzu-Mainn Chen]

> On 27 January 2016 at 08:10, Tzu-Mainn Chen < tzuma...@redhat.com > wrote:

> > > Okay, so I initially thought we weren't making much progress on this
> 
> > > discussion, but after some more thought and reading of the existing PoC,
> 
> > > we're (maybe?) less far apart than I initially thought.
> 
> > >
> 
> > > I think there are kind of three different designs being discussed.
> 
> > >
> 
> > > 1) Rewrite a bunch of stuff into MistrYAML, with the idea that users
> 
> > > could edit our workflows. I think this is what I've been most
> 
> > > strenuously objecting to, and for the most part my previous arguments
> 
> > > pertain to this model.
> 
> > >
> 
> > > 2) However, I think there's another thing going on/planned with at least
> 
> > > some of the actions. It sounds like some of our workflows are going to
> 
> > > essentially be a single action that just passes the REST params into our
> 
> > > Python code. This sort of API as a Service would be more palatable to
> 
> > > me, as it doesn't really split our implementation between YAML and
> 
> > > Python (the YAML is pretty much only defining the REST API in this
> 
> > > model), but it still gives us a quick and easy REST interface to the
> 
> > > existing code. It also keeps a certain amount of separation between
> 
> > > Mistral and the TripleO code in case we decide some day that we need a
> 
> > > proper API service and need to swap out the Mistral frontend for a
> 
> > > different one. This should also be the easiest to implement since it
> 
> > > doesn't involve rewriting anything - we're mostly just moving the
> 
> > > existing code into Mistral actions and creating some pretty trivial
> 
> > > Mistral workflows.
> 
> > >
> 
> > > 3) The thing I _want_ to see, which is a regular Python-based API
> 
> > > service. Again, you can kind of see my arguments around why I think we
> 
> > > should do this elsewhere in the thread. It's also worth noting that
> 
> > > there is already an initial implementation of this proposed to
> 
> > > tripleo-common, so it's not like we'd be starting from zero here either.
> 
> > >
> 
> > > I'm still not crazy about 2, but if it lets me stop spending excessive
> 
> > > amounts of time on this topic it might be worth it. :-)
> 
> > >
> 

> > I'm kinda with Ben here; I'm strongly for 3), but 2) is okay-ish - with a
> 
> > few caveats. This thread has raised a lot of interesting points that, if
> 
> > clarified, might help me feel more comfortable about 2), so I'm hoping
> 
> > that Dan/Steve, you'd be willing to help me understand a few things:
> 

> > a) One argument against the TripleO API is that the Tuskar API tied us
> 
> > far too strongly with one way of doing things. However, the Mistral
> 
> > solution would create a set of workflows that essentially have the same
> 
> > interface as the TripleO API, correct? If so, would we support those
> 
> > workflows the same as if they were an API, with extensive documentation
> 
> > and guaranteeing stability from version to version of TripleO?
> 

> I believe we would, yes. This needs to be a stable interface, if we really
> need
> to make breaking changes then we could use versions in the workflow names
> which Dan suggested at one point.

> > b) For simple features that we might expose through the Mistral API as
> 
> > one-step workflows calling a single function (getting parameters for a
> 
> > deployment, say): when we expose these features through the CLI, would we
> 
> > also enforce the CLI going through Mistral to access those features rather
> 
> > than calling that single function?
> 

> I think we should, it is just much simpler to have everyone use the same
> interface
> than decide of a case by case basis. However, I could be persuaded otherwise
> if others object to this.

> > c) Is there consideration to the fact that multiple OpenStack projects
> 
> > have created their own REST APIs to the point that seems like more of
> 
> > a known technology than using Mistral to front everything? Or are we
> 
> > going to argue that other projects should also switch to using Mistral?
> 

> Projects are so different, that I don't think this can really be compared so
> widely. It doesn't really make sense. Projects should be free to choose
> which approach suits them best.

So what makes TripleO different here? I think this is fundamental to my 
objection, so 
maybe if I understood this point my objections would fall away. 

Mainn 

> > d) If we proceed down the Mistral path and run into issues, is there a
> 
> > point when we'd be willing to back away?
> 

> I don't imagine anyone wants to beat a dead horse. So, yeah, if it doesn't
> work out we should reevaluate things. I feel like we have constantly been
> doing this for some time (for better or worse).

> I am still of the opinion that we can always add an API in-front of Mistral
> later if it doesn't work out well. If we start doing this early enough in the
> cycle that should give us time to have feedback 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Tzu-Mainn Chen]

> Okay, so I initially thought we weren't making much progress on this
> discussion, but after some more thought and reading of the existing PoC,
> we're (maybe?) less far apart than I initially thought.
> 
> I think there are kind of three different designs being discussed.
> 
> 1) Rewrite a bunch of stuff into MistrYAML, with the idea that users
> could edit our workflows.  I think this is what I've been most
> strenuously objecting to, and for the most part my previous arguments
> pertain to this model.
> 
> 2) However, I think there's another thing going on/planned with at least
> some of the actions.  It sounds like some of our workflows are going to
> essentially be a single action that just passes the REST params into our
> Python code.  This sort of API as a Service would be more palatable to
> me, as it doesn't really split our implementation between YAML and
> Python (the YAML is pretty much only defining the REST API in this
> model), but it still gives us a quick and easy REST interface to the
> existing code.  It also keeps a certain amount of separation between
> Mistral and the TripleO code in case we decide some day that we need a
> proper API service and need to swap out the Mistral frontend for a
> different one.  This should also be the easiest to implement since it
> doesn't involve rewriting anything - we're mostly just moving the
> existing code into Mistral actions and creating some pretty trivial
> Mistral workflows.
> 
> 3) The thing I _want_ to see, which is a regular Python-based API
> service.  Again, you can kind of see my arguments around why I think we
> should do this elsewhere in the thread.  It's also worth noting that
> there is already an initial implementation of this proposed to
> tripleo-common, so it's not like we'd be starting from zero here either.
> 
> I'm still not crazy about 2, but if it lets me stop spending excessive
> amounts of time on this topic it might be worth it. :-)
> 

I'm kinda with Ben here; I'm strongly for 3), but 2) is okay-ish - with a
few caveats.  This thread has raised a lot of interesting points that, if
clarified, might help me feel more comfortable about 2), so I'm hoping
that Dan/Steve, you'd be willing to help me understand a few things:

a) One argument against the TripleO API is that the Tuskar API tied us
far too strongly with one way of doing things.  However, the Mistral
solution would create a set of workflows that essentially have the same
interface as the TripleO API, correct?  If so, would we support those
workflows the same as if they were an API, with extensive documentation
and guaranteeing stability from version to version of TripleO?

b) For simple features that we might expose through the Mistral API as
one-step workflows calling a single function (getting parameters for a
deployment, say): when we expose these features through the CLI, would we
also enforce the CLI going through Mistral to access those features rather
than calling that single function?

c) Is there consideration to the fact that multiple OpenStack projects
have created their own REST APIs to the point that seems like more of
a known technology than using Mistral to front everything?  Or are we
going to argue that other projects should also switch to using Mistral?

d) If we proceed down the Mistral path and run into issues, is there a
point when we'd be willing to back away?


Mainn

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Wed, 2016-01-27 at 09:36 -0500, Dan Prince wrote:
> On Wed, 2016-01-27 at 14:32 +0100, Jiri Tomasek wrote:
> > On 01/26/2016 09:05 PM, Ben Nemec wrote:
> > > On 01/25/2016 04:36 PM, Dan Prince wrote:
> > > > On Mon, 2016-01-25 at 15:31 -0600, Ben Nemec wrote:
> > > > > On 01/22/2016 06:19 PM, Dan Prince wrote:
> > > > > > On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> > > > > > > So I haven't weighed in on this yet, in part because I
> > > > > > > was
> > > > > > > on
> > > > > > > vacation
> > > > > > > when it was first proposed and missed a lot of the
> > > > > > > initial
> > > > > > > discussion,
> > > > > > > and also because I wanted to take some time to order my
> > > > > > > thoughts
> > > > > > > on
> > > > > > > it.
> > > > > > >   Also because my initial reaction...was not conducive to
> > > > > > > calm and
> > > > > > > rational discussion. ;-)
> > > > > > > 
> > > > > > > The tldr is that I don't like it.  To explain why, I'm
> > > > > > > going to
> > > > > > > make
> > > > > > > a
> > > > > > > list (everyone loves lists, right? Top $NUMBER reasons we
> > > > > > > should
> > > > > > > stop
> > > > > > > expecting other people to write our API for us):
> > > > > > > 
> > > > > > > 1) We've been down this road before.  Except last time it
> > > > > > > was
> > > > > > > with
> > > > > > > Heat.
> > > > > > >   I'm being somewhat tongue-in-cheek here, but expecting
> > > > > > > a
> > > > > > > general
> > > > > > > service to provide us a user-friendly API for our
> > > > > > > specific
> > > > > > > use
> > > > > > > case
> > > > > > > just
> > > > > > > doesn't make sense to me.
> > > > > > We've been down this road with Heat yes. But we are
> > > > > > currently
> > > > > > using
> > > > > > Heat for some things that we arguable should be (a
> > > > > > workflows
> > > > > > tool
> > > > > > might
> > > > > > help offload some stuff out of Heat). Also we haven't
> > > > > > implemented
> > > > > > custom Heat resources for TripleO either. There are mixed
> > > > > > opinions
> > > > > > on
> > > > > > this but plugging in your code to a generic API is quite
> > > > > > nice
> > > > > > sometimes.
> > > > > > 
> > > > > > That is the beauty of Mistral I think. Unlike Heat it
> > > > > > actually
> > > > > > encourages you to customize it with custom Python actions.
> > > > > > Anything
> > > > > > we
> > > > > > want in tripleo-common can become our own Mistral action
> > > > > > (these get
> > > > > > registered with stevedore entry points so we'd own the
> > > > > > code)
> > > > > > and
> > > > > > the
> > > > > > YAML workflows just tie them together via tasks.
> > > > > > 
> > > > > > We don't have to go off and build our own proxy deployment
> > > > > > workflow
> > > > > > API. The structure to do just about anything we need
> > > > > > already
> > > > > > exists
> > > > > > so
> > > > > > why not go and use it?
> > > > > > 
> > > > > > > 2) The TripleO API is not a workflow API.  I also largely
> > > > > > > missed
> > > > > > > this
> > > > > > > discussion, but the TripleO API is a _Deployment_
> > > > > > > API.  In
> > > > > > > some
> > > > > > > cases
> > > > > > > there also happens to be a workflow going on behind the
> > > > > > > scenes,
> > > > > > > but
> > > > > > > honestly that's not something I want our users to have to
> > > > > > > care
> > > > > > > about.
> > > > > > Agree that users don't have to care about this.
> > > > > > 
> > > > > > Users can get as involved as they want here. Most users I
> > > > > > think
> > > > > > will
> > > > > > use python-tripleoclient to drive the deployment or the new
> > > > > > UI.
> > > > > > They
> > > > > > don't have to interact with Mistral directly unless they
> > > > > > really
> > > > > > want
> > > > > > to. So whether we choose to build our own API or use a
> > > > > > generic one
> > > > > > I
> > > > > > think this point is mute.
> > > > > Okay, I think this is a very fundamental point, and I believe
> > > > > it gets
> > > > > right to the heart of my objection to the proposed change.
> > > > > 
> > > > > When I hear you say that users will use tripleoclient to talk
> > > > > to
> > > > > Mistral, it raises a big flag.  Then I look at something like
> > > > > https://github.com/dprince/python-tripleoclient/commit/77ffd2
> > > > > fa
> > > > > 7b1642
> > > > > b9f05713ca30b8a27ec4b322b7
> > > > > and the flag gets bigger.
> > > > > 
> > > > > The thing is that there's a whole bunch of business logic
> > > > > currently
> > > > > sitting in the client that shouldn't/can't be there.  There
> > > > > are
> > > > > historical reasons for it, but the important thing is that
> > > > > the
> > > > > current
> > > > > client architecture is terribly flawed.  Business logic
> > > > > should
> > > > > never
> > > > > live in the client like it does today.
> > > > Totally agree here. In fact I have removed business logic from
> > > > python-
> > > > tripleoclient in this patch and moved it into a Mistral action.
> 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Tzu-Mainn Chen]

> On Wed, 2016-01-27 at 09:36 -0500, Dan Prince wrote:
> > On Wed, 2016-01-27 at 14:32 +0100, Jiri Tomasek wrote:
> > > On 01/26/2016 09:05 PM, Ben Nemec wrote:
> > > > On 01/25/2016 04:36 PM, Dan Prince wrote:
> > > > > On Mon, 2016-01-25 at 15:31 -0600, Ben Nemec wrote:
> > > > > > On 01/22/2016 06:19 PM, Dan Prince wrote:
> > > > > > > On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> > > > > > > > So I haven't weighed in on this yet, in part because I
> > > > > > > > was
> > > > > > > > on
> > > > > > > > vacation
> > > > > > > > when it was first proposed and missed a lot of the
> > > > > > > > initial
> > > > > > > > discussion,
> > > > > > > > and also because I wanted to take some time to order my
> > > > > > > > thoughts
> > > > > > > > on
> > > > > > > > it.
> > > > > > > >   Also because my initial reaction...was not conducive to
> > > > > > > > calm and
> > > > > > > > rational discussion. ;-)
> > > > > > > > 
> > > > > > > > The tldr is that I don't like it.  To explain why, I'm
> > > > > > > > going to
> > > > > > > > make
> > > > > > > > a
> > > > > > > > list (everyone loves lists, right? Top $NUMBER reasons we
> > > > > > > > should
> > > > > > > > stop
> > > > > > > > expecting other people to write our API for us):
> > > > > > > > 
> > > > > > > > 1) We've been down this road before.  Except last time it
> > > > > > > > was
> > > > > > > > with
> > > > > > > > Heat.
> > > > > > > >   I'm being somewhat tongue-in-cheek here, but expecting
> > > > > > > > a
> > > > > > > > general
> > > > > > > > service to provide us a user-friendly API for our
> > > > > > > > specific
> > > > > > > > use
> > > > > > > > case
> > > > > > > > just
> > > > > > > > doesn't make sense to me.
> > > > > > > We've been down this road with Heat yes. But we are
> > > > > > > currently
> > > > > > > using
> > > > > > > Heat for some things that we arguable should be (a
> > > > > > > workflows
> > > > > > > tool
> > > > > > > might
> > > > > > > help offload some stuff out of Heat). Also we haven't
> > > > > > > implemented
> > > > > > > custom Heat resources for TripleO either. There are mixed
> > > > > > > opinions
> > > > > > > on
> > > > > > > this but plugging in your code to a generic API is quite
> > > > > > > nice
> > > > > > > sometimes.
> > > > > > > 
> > > > > > > That is the beauty of Mistral I think. Unlike Heat it
> > > > > > > actually
> > > > > > > encourages you to customize it with custom Python actions.
> > > > > > > Anything
> > > > > > > we
> > > > > > > want in tripleo-common can become our own Mistral action
> > > > > > > (these get
> > > > > > > registered with stevedore entry points so we'd own the
> > > > > > > code)
> > > > > > > and
> > > > > > > the
> > > > > > > YAML workflows just tie them together via tasks.
> > > > > > > 
> > > > > > > We don't have to go off and build our own proxy deployment
> > > > > > > workflow
> > > > > > > API. The structure to do just about anything we need
> > > > > > > already
> > > > > > > exists
> > > > > > > so
> > > > > > > why not go and use it?
> > > > > > > 
> > > > > > > > 2) The TripleO API is not a workflow API.  I also largely
> > > > > > > > missed
> > > > > > > > this
> > > > > > > > discussion, but the TripleO API is a _Deployment_
> > > > > > > > API.  In
> > > > > > > > some
> > > > > > > > cases
> > > > > > > > there also happens to be a workflow going on behind the
> > > > > > > > scenes,
> > > > > > > > but
> > > > > > > > honestly that's not something I want our users to have to
> > > > > > > > care
> > > > > > > > about.
> > > > > > > Agree that users don't have to care about this.
> > > > > > > 
> > > > > > > Users can get as involved as they want here. Most users I
> > > > > > > think
> > > > > > > will
> > > > > > > use python-tripleoclient to drive the deployment or the new
> > > > > > > UI.
> > > > > > > They
> > > > > > > don't have to interact with Mistral directly unless they
> > > > > > > really
> > > > > > > want
> > > > > > > to. So whether we choose to build our own API or use a
> > > > > > > generic one
> > > > > > > I
> > > > > > > think this point is mute.
> > > > > > Okay, I think this is a very fundamental point, and I believe
> > > > > > it gets
> > > > > > right to the heart of my objection to the proposed change.
> > > > > > 
> > > > > > When I hear you say that users will use tripleoclient to talk
> > > > > > to
> > > > > > Mistral, it raises a big flag.  Then I look at something like
> > > > > > https://github.com/dprince/python-tripleoclient/commit/77ffd2
> > > > > > fa
> > > > > > 7b1642
> > > > > > b9f05713ca30b8a27ec4b322b7
> > > > > > and the flag gets bigger.
> > > > > > 
> > > > > > The thing is that there's a whole bunch of business logic
> > > > > > currently
> > > > > > sitting in the client that shouldn't/can't be there.  There
> > > > > > are
> > > > > > historical reasons for it, but the important thing is that
> > > > > > the
> > > > > > current
> > > > > > client architecture is terribly flawed. 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Jiri Tomasek]

On 01/27/2016 03:36 PM, Dan Prince wrote:

On Wed, 2016-01-27 at 14:32 +0100, Jiri Tomasek wrote:

On 01/26/2016 09:05 PM, Ben Nemec wrote:

On 01/25/2016 04:36 PM, Dan Prince wrote:

On Mon, 2016-01-25 at 15:31 -0600, Ben Nemec wrote:

On 01/22/2016 06:19 PM, Dan Prince wrote:

On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:

So I haven't weighed in on this yet, in part because I was
on
vacation
when it was first proposed and missed a lot of the initial
discussion,
and also because I wanted to take some time to order my
thoughts
on
it.
   Also because my initial reaction...was not conducive to
calm and
rational discussion. ;-)

The tldr is that I don't like it.  To explain why, I'm
going to
make
a
list (everyone loves lists, right? Top $NUMBER reasons we
should
stop
expecting other people to write our API for us):

1) We've been down this road before.  Except last time it
was
with
Heat.
   I'm being somewhat tongue-in-cheek here, but expecting a
general
service to provide us a user-friendly API for our specific
use
case
just
doesn't make sense to me.

We've been down this road with Heat yes. But we are currently
using
Heat for some things that we arguable should be (a workflows
tool
might
help offload some stuff out of Heat). Also we haven't
implemented
custom Heat resources for TripleO either. There are mixed
opinions
on
this but plugging in your code to a generic API is quite nice
sometimes.

That is the beauty of Mistral I think. Unlike Heat it
actually
encourages you to customize it with custom Python actions.
Anything
we
want in tripleo-common can become our own Mistral action
(these get
registered with stevedore entry points so we'd own the code)
and
the
YAML workflows just tie them together via tasks.

We don't have to go off and build our own proxy deployment
workflow
API. The structure to do just about anything we need already
exists
so
why not go and use it?


2) The TripleO API is not a workflow API.  I also largely
missed
this
discussion, but the TripleO API is a _Deployment_ API.  In
some
cases
there also happens to be a workflow going on behind the
scenes,
but
honestly that's not something I want our users to have to
care
about.

Agree that users don't have to care about this.

Users can get as involved as they want here. Most users I
think
will
use python-tripleoclient to drive the deployment or the new
UI.
They
don't have to interact with Mistral directly unless they
really
want
to. So whether we choose to build our own API or use a
generic one
I
think this point is mute.

Okay, I think this is a very fundamental point, and I believe
it gets
right to the heart of my objection to the proposed change.

When I hear you say that users will use tripleoclient to talk
to
Mistral, it raises a big flag.  Then I look at something like
https://github.com/dprince/python-tripleoclient/commit/77ffd2fa
7b1642
b9f05713ca30b8a27ec4b322b7
and the flag gets bigger.

The thing is that there's a whole bunch of business logic
currently
sitting in the client that shouldn't/can't be there.  There are
historical reasons for it, but the important thing is that the
current
client architecture is terribly flawed.  Business logic should
never
live in the client like it does today.

Totally agree here. In fact I have removed business logic from
python-
tripleoclient in this patch and moved it into a Mistral action.
Which
can then be used via a stable API from anywhere.


Looking at that change, I see a bunch of business logic around
taking
our configuration and passing it to Mistral.  In order for us
to do
something like that and have a sustainable GUI, that code _has_
to
live
behind an API that the GUI and CLI alike can call.  If we ask
the GUI
to
re-implement that code, then we're doomed to divergence between
the
CLI
and GUI code and we'll most likely end up back where we are
with a
GUI
that can't deploy half of our features because they were
implemented
solely with the CLI in mind and made assumptions the GUI can't
meet.

The latest feedback I've gotten from working with the UI
developers on
this was that we should have a workflow to create the
environment. That
would get called via the Mistral API via python-tripleoclient and
any
sort of UI you could imagine and would essentially give us a
stable
environment interface.

Anything that requires tripleoclient means !GUI though.  I know the
current GUI still has a bunch of dependencies on the CLI, but that
seems
like something we need to fix, not a pattern to repeat.  I still
think
any sentence containing "call Mistral via tripleoclient" is
indicative
of a problem in the design.

I am not sure I understand the argument here.

Regardless of which API we use (Mistral API or TripleO API) GUI is
going
to call the API and tripleoclient (CLI) is going to call the API
(through mistralclient - impl. detail).

GUI can't and does not call API through tripleoclient. This is why
the
work on extracting the common business logic to tripleo-common
happened.
So tripleo-common is 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Tue, 2016-01-26 at 14:05 -0600, Ben Nemec wrote:
> On 01/25/2016 04:36 PM, Dan Prince wrote:
> > On Mon, 2016-01-25 at 15:31 -0600, Ben Nemec wrote:
> > > On 01/22/2016 06:19 PM, Dan Prince wrote:
> > > > On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> > > > > So I haven't weighed in on this yet, in part because I was on
> > > > > vacation
> > > > > when it was first proposed and missed a lot of the initial
> > > > > discussion,
> > > > > and also because I wanted to take some time to order my
> > > > > thoughts
> > > > > on
> > > > > it.
> > > > >  Also because my initial reaction...was not conducive to calm
> > > > > and
> > > > > rational discussion. ;-)
> > > > > 
> > > > > The tldr is that I don't like it.  To explain why, I'm going
> > > > > to
> > > > > make
> > > > > a
> > > > > list (everyone loves lists, right? Top $NUMBER reasons we
> > > > > should
> > > > > stop
> > > > > expecting other people to write our API for us):
> > > > > 
> > > > > 1) We've been down this road before.  Except last time it was
> > > > > with
> > > > > Heat.
> > > > >  I'm being somewhat tongue-in-cheek here, but expecting a
> > > > > general
> > > > > service to provide us a user-friendly API for our specific
> > > > > use
> > > > > case
> > > > > just
> > > > > doesn't make sense to me.
> > > > 
> > > > We've been down this road with Heat yes. But we are currently
> > > > using
> > > > Heat for some things that we arguable should be (a workflows
> > > > tool
> > > > might
> > > > help offload some stuff out of Heat). Also we haven't
> > > > implemented
> > > > custom Heat resources for TripleO either. There are mixed
> > > > opinions
> > > > on
> > > > this but plugging in your code to a generic API is quite nice
> > > > sometimes.
> > > > 
> > > > That is the beauty of Mistral I think. Unlike Heat it actually
> > > > encourages you to customize it with custom Python actions.
> > > > Anything
> > > > we
> > > > want in tripleo-common can become our own Mistral action (these
> > > > get
> > > > registered with stevedore entry points so we'd own the code)
> > > > and
> > > > the
> > > > YAML workflows just tie them together via tasks.
> > > > 
> > > > We don't have to go off and build our own proxy deployment
> > > > workflow
> > > > API. The structure to do just about anything we need already
> > > > exists
> > > > so
> > > > why not go and use it?
> > > > 
> > > > > 
> > > > > 2) The TripleO API is not a workflow API.  I also largely
> > > > > missed
> > > > > this
> > > > > discussion, but the TripleO API is a _Deployment_ API.  In
> > > > > some
> > > > > cases
> > > > > there also happens to be a workflow going on behind the
> > > > > scenes,
> > > > > but
> > > > > honestly that's not something I want our users to have to
> > > > > care
> > > > > about.
> > > > 
> > > > Agree that users don't have to care about this.
> > > > 
> > > > Users can get as involved as they want here. Most users I think
> > > > will
> > > > use python-tripleoclient to drive the deployment or the new UI.
> > > > They
> > > > don't have to interact with Mistral directly unless they really
> > > > want
> > > > to. So whether we choose to build our own API or use a generic
> > > > one
> > > > I
> > > > think this point is mute.
> > > 
> > > Okay, I think this is a very fundamental point, and I believe it
> > > gets
> > > right to the heart of my objection to the proposed change.
> > > 
> > > When I hear you say that users will use tripleoclient to talk to
> > > Mistral, it raises a big flag.  Then I look at something like
> > > https://github.com/dprince/python-tripleoclient/commit/77ffd2fa7b
> > > 1642
> > > b9f05713ca30b8a27ec4b322b7
> > > and the flag gets bigger.
> > > 
> > > The thing is that there's a whole bunch of business logic
> > > currently
> > > sitting in the client that shouldn't/can't be there.  There are
> > > historical reasons for it, but the important thing is that the
> > > current
> > > client architecture is terribly flawed.  Business logic should
> > > never
> > > live in the client like it does today.
> > 
> > Totally agree here. In fact I have removed business logic from
> > python-
> > tripleoclient in this patch and moved it into a Mistral action.
> > Which
> > can then be used via a stable API from anywhere.
> > 
> > > 
> > > Looking at that change, I see a bunch of business logic around
> > > taking
> > > our configuration and passing it to Mistral.  In order for us to
> > > do
> > > something like that and have a sustainable GUI, that code _has_
> > > to
> > > live
> > > behind an API that the GUI and CLI alike can call.  If we ask the
> > > GUI
> > > to
> > > re-implement that code, then we're doomed to divergence between
> > > the
> > > CLI
> > > and GUI code and we'll most likely end up back where we are with
> > > a
> > > GUI
> > > that can't deploy half of our features because they were
> > > implemented
> > > solely with the CLI in mind and made assumptions the GUI can't
> > > meet.
> > 
> > The 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dougal Matthews]

On 26 January 2016 at 16:01, Ben Nemec  wrote:

> On 01/26/2016 03:46 AM, Steven Hardy wrote:
> > On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
> >> On 01/25/2016 03:56 PM, Steven Hardy wrote:
> >>> On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
>  So I haven't weighed in on this yet, in part because I was on vacation
>  when it was first proposed and missed a lot of the initial discussion,
>  and also because I wanted to take some time to order my thoughts on
> it.
>   Also because my initial reaction...was not conducive to calm and
>  rational discussion. ;-)
> 
>  The tldr is that I don't like it.  To explain why, I'm going to make a
>  list (everyone loves lists, right? Top $NUMBER reasons we should stop
>  expecting other people to write our API for us):
> 
>  1) We've been down this road before.  Except last time it was with
> Heat.
>   I'm being somewhat tongue-in-cheek here, but expecting a general
>  service to provide us a user-friendly API for our specific use case
> just
>  doesn't make sense to me.
> >>>
> >>> Actually, we've been down this road before with Tuskar, and discovered
> that
> >>> designing and maintaining a bespoke API for TripleO is really hard.
> >>
> >> My takeaway from Tuskar was that designing an API that none of the
> >> developers on the project use is doomed to fail.  Tuskar also suffered
> >> from a lack of some features in Heat that the new API is explicitly
> >> depending on in an attempt to avoid many of the problems Tuskar had.
> >>
> >> Problem #1 is still developer apathy IMHO though.
> >
> > I think the main issue is developer capacity - we're a small community
> and
> > I for one am worried about the effort involved with building and
> > maintaining a bespoke API - thus this whole discussion is essentially
> about
> > finding a quicker and easier way to meet the needs of those needing an
> API.
> >
> > In terms of apathy, I think as a developer I don't need an abstraction
> > between me, my templates and heat.  Some advanced operators will feel
> > likewise, others won't.  What I would find useful sometimes is a general
> > purpose workflow engine, which is where I think the more pluggable
> mistral
> > based solution may have advantages in terms of developer and advanced
> > operator uptake.
>
> The API is for end users, not developers.  tripleo-incubator was easily
> hackable for developers and power users.  It was unusable for everyone
> else.
>

Doesn't it depend on what you mean by end users? I would argue that
typically the CLI and UI will be for end users. Then API is for end users
that are also developers. I don't imagine we are going to suggest
non-developer end users directly use the API.


>
> >>> I somewhat agree that heat as an API is insufficient, but that doesn't
> >>> necessarily imply you have to have a TripleO specific abstraction, just
> >>> that *an* abstraction is required.
> >>>
>  2) The TripleO API is not a workflow API.  I also largely missed this
>  discussion, but the TripleO API is a _Deployment_ API.  In some cases
>  there also happens to be a workflow going on behind the scenes, but
>  honestly that's not something I want our users to have to care about.
> >>>
> >>> How would you differentiate between "deployment" in a generic sense in
> >>> contrast to a generic workflow?
> >>>
> >>> Every deployment I can think of involves a series of steps, involving
> some
> >>> choices and interactions with other services.  That *is* a workflow?
> >>
> >> Well, I mean if we want to take this to extremes then pretty much every
> >> API is a workflow API.  You make a REST call, a "workflow" happens in
> >> the service, and you get back a result.
> >>
> >> Let me turn this around: Would you implement Heat's API on Mistral?  All
> >> that happens when I call Heat is that a series of OpenStack calls are
> >> made from heat-engine, after all.  Or is that a gross oversimplification
> >> of what's happening?  I could argue that the same is true of this
> >> discussion. :-)
> >
> > As Hugh has mentioned the main thing Heat does is actually manage
> > dependencies.  It processes the templates, builds a graph, then walks the
> > graph running a "workflow" to create/update/delete/etc each resource.
> >
> > I could imagine a future where we interface to some external workflow
> tool to
> > e.g do each resource action (e.g create a nova server, poll until it's
> active),
> > however that's actually a pretty high overhead approach, and it'd
> probably
> > be better to move towards better use of notifications instead (e.g less
> > internal workflow)
> >
>  3) It ties us 100% to a given implementation.  If Mistral proves to
> be a
>  poor choice for some reason, or insufficient for a particular use
> case,
>  we have no alternative.  If we have an API and decide to change our
>  implementation, nobody has to know or care.  

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Wed, 2016-01-27 at 14:32 +0100, Jiri Tomasek wrote:
> On 01/26/2016 09:05 PM, Ben Nemec wrote:
> > On 01/25/2016 04:36 PM, Dan Prince wrote:
> > > On Mon, 2016-01-25 at 15:31 -0600, Ben Nemec wrote:
> > > > On 01/22/2016 06:19 PM, Dan Prince wrote:
> > > > > On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> > > > > > So I haven't weighed in on this yet, in part because I was
> > > > > > on
> > > > > > vacation
> > > > > > when it was first proposed and missed a lot of the initial
> > > > > > discussion,
> > > > > > and also because I wanted to take some time to order my
> > > > > > thoughts
> > > > > > on
> > > > > > it.
> > > > > >   Also because my initial reaction...was not conducive to
> > > > > > calm and
> > > > > > rational discussion. ;-)
> > > > > > 
> > > > > > The tldr is that I don't like it.  To explain why, I'm
> > > > > > going to
> > > > > > make
> > > > > > a
> > > > > > list (everyone loves lists, right? Top $NUMBER reasons we
> > > > > > should
> > > > > > stop
> > > > > > expecting other people to write our API for us):
> > > > > > 
> > > > > > 1) We've been down this road before.  Except last time it
> > > > > > was
> > > > > > with
> > > > > > Heat.
> > > > > >   I'm being somewhat tongue-in-cheek here, but expecting a
> > > > > > general
> > > > > > service to provide us a user-friendly API for our specific
> > > > > > use
> > > > > > case
> > > > > > just
> > > > > > doesn't make sense to me.
> > > > > We've been down this road with Heat yes. But we are currently
> > > > > using
> > > > > Heat for some things that we arguable should be (a workflows
> > > > > tool
> > > > > might
> > > > > help offload some stuff out of Heat). Also we haven't
> > > > > implemented
> > > > > custom Heat resources for TripleO either. There are mixed
> > > > > opinions
> > > > > on
> > > > > this but plugging in your code to a generic API is quite nice
> > > > > sometimes.
> > > > > 
> > > > > That is the beauty of Mistral I think. Unlike Heat it
> > > > > actually
> > > > > encourages you to customize it with custom Python actions.
> > > > > Anything
> > > > > we
> > > > > want in tripleo-common can become our own Mistral action
> > > > > (these get
> > > > > registered with stevedore entry points so we'd own the code)
> > > > > and
> > > > > the
> > > > > YAML workflows just tie them together via tasks.
> > > > > 
> > > > > We don't have to go off and build our own proxy deployment
> > > > > workflow
> > > > > API. The structure to do just about anything we need already
> > > > > exists
> > > > > so
> > > > > why not go and use it?
> > > > > 
> > > > > > 2) The TripleO API is not a workflow API.  I also largely
> > > > > > missed
> > > > > > this
> > > > > > discussion, but the TripleO API is a _Deployment_ API.  In
> > > > > > some
> > > > > > cases
> > > > > > there also happens to be a workflow going on behind the
> > > > > > scenes,
> > > > > > but
> > > > > > honestly that's not something I want our users to have to
> > > > > > care
> > > > > > about.
> > > > > Agree that users don't have to care about this.
> > > > > 
> > > > > Users can get as involved as they want here. Most users I
> > > > > think
> > > > > will
> > > > > use python-tripleoclient to drive the deployment or the new
> > > > > UI.
> > > > > They
> > > > > don't have to interact with Mistral directly unless they
> > > > > really
> > > > > want
> > > > > to. So whether we choose to build our own API or use a
> > > > > generic one
> > > > > I
> > > > > think this point is mute.
> > > > Okay, I think this is a very fundamental point, and I believe
> > > > it gets
> > > > right to the heart of my objection to the proposed change.
> > > > 
> > > > When I hear you say that users will use tripleoclient to talk
> > > > to
> > > > Mistral, it raises a big flag.  Then I look at something like
> > > > https://github.com/dprince/python-tripleoclient/commit/77ffd2fa
> > > > 7b1642
> > > > b9f05713ca30b8a27ec4b322b7
> > > > and the flag gets bigger.
> > > > 
> > > > The thing is that there's a whole bunch of business logic
> > > > currently
> > > > sitting in the client that shouldn't/can't be there.  There are
> > > > historical reasons for it, but the important thing is that the
> > > > current
> > > > client architecture is terribly flawed.  Business logic should
> > > > never
> > > > live in the client like it does today.
> > > Totally agree here. In fact I have removed business logic from
> > > python-
> > > tripleoclient in this patch and moved it into a Mistral action.
> > > Which
> > > can then be used via a stable API from anywhere.
> > > 
> > > > Looking at that change, I see a bunch of business logic around
> > > > taking
> > > > our configuration and passing it to Mistral.  In order for us
> > > > to do
> > > > something like that and have a sustainable GUI, that code _has_
> > > > to
> > > > live
> > > > behind an API that the GUI and CLI alike can call.  If we ask
> > > > the GUI
> > > > to
> > > > re-implement that code, 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dougal Matthews]

On 27 January 2016 at 08:10, Tzu-Mainn Chen  wrote:

> > Okay, so I initially thought we weren't making much progress on this
> > discussion, but after some more thought and reading of the existing PoC,
> > we're (maybe?) less far apart than I initially thought.
> >
> > I think there are kind of three different designs being discussed.
> >
> > 1) Rewrite a bunch of stuff into MistrYAML, with the idea that users
> > could edit our workflows.  I think this is what I've been most
> > strenuously objecting to, and for the most part my previous arguments
> > pertain to this model.
> >
> > 2) However, I think there's another thing going on/planned with at least
> > some of the actions.  It sounds like some of our workflows are going to
> > essentially be a single action that just passes the REST params into our
> > Python code.  This sort of API as a Service would be more palatable to
> > me, as it doesn't really split our implementation between YAML and
> > Python (the YAML is pretty much only defining the REST API in this
> > model), but it still gives us a quick and easy REST interface to the
> > existing code.  It also keeps a certain amount of separation between
> > Mistral and the TripleO code in case we decide some day that we need a
> > proper API service and need to swap out the Mistral frontend for a
> > different one.  This should also be the easiest to implement since it
> > doesn't involve rewriting anything - we're mostly just moving the
> > existing code into Mistral actions and creating some pretty trivial
> > Mistral workflows.
> >
> > 3) The thing I _want_ to see, which is a regular Python-based API
> > service.  Again, you can kind of see my arguments around why I think we
> > should do this elsewhere in the thread.  It's also worth noting that
> > there is already an initial implementation of this proposed to
> > tripleo-common, so it's not like we'd be starting from zero here either.
> >
> > I'm still not crazy about 2, but if it lets me stop spending excessive
> > amounts of time on this topic it might be worth it. :-)
> >
>
> I'm kinda with Ben here; I'm strongly for 3), but 2) is okay-ish - with a
> few caveats.  This thread has raised a lot of interesting points that, if
> clarified, might help me feel more comfortable about 2), so I'm hoping
> that Dan/Steve, you'd be willing to help me understand a few things:
>
> a) One argument against the TripleO API is that the Tuskar API tied us
> far too strongly with one way of doing things.  However, the Mistral
> solution would create a set of workflows that essentially have the same
> interface as the TripleO API, correct?  If so, would we support those
> workflows the same as if they were an API, with extensive documentation
> and guaranteeing stability from version to version of TripleO?
>

I believe we would, yes. This needs to be a stable interface, if we really
need
to make breaking changes then we could use versions in the workflow names
which Dan suggested at one point.


b) For simple features that we might expose through the Mistral API as
> one-step workflows calling a single function (getting parameters for a
> deployment, say): when we expose these features through the CLI, would we
> also enforce the CLI going through Mistral to access those features rather
> than calling that single function?
>

I think we should, it is just much simpler to have everyone use the same
interface
than decide of a case by case basis. However, I could be persuaded otherwise
if others object to this.


c) Is there consideration to the fact that multiple OpenStack projects
> have created their own REST APIs to the point that seems like more of
> a known technology than using Mistral to front everything?  Or are we
> going to argue that other projects should also switch to using Mistral?
>

Projects are so different, that I don't think this can really be compared so
widely. It doesn't really make sense. Projects should be free to choose
which approach suits them best.



> d) If we proceed down the Mistral path and run into issues, is there a
> point when we'd be willing to back away?
>

I don't imagine anyone wants to beat a dead horse. So, yeah, if it doesn't
work out we should reevaluate things. I feel like we have constantly been
doing this for some time (for better or worse).

I am still of the opinion that we can always add an API in-front of Mistral
later if it doesn't work out well. If we start doing this early enough in
the
cycle that should give us time to have feedback and see. It might even
be worth somebody creating a POC API that uses the Mistral workflows
to explore the idea.



>
>
> Mainn
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Jiri Tomasek]

On 01/26/2016 09:05 PM, Ben Nemec wrote:

On 01/25/2016 04:36 PM, Dan Prince wrote:

On Mon, 2016-01-25 at 15:31 -0600, Ben Nemec wrote:

On 01/22/2016 06:19 PM, Dan Prince wrote:

On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:

So I haven't weighed in on this yet, in part because I was on
vacation
when it was first proposed and missed a lot of the initial
discussion,
and also because I wanted to take some time to order my thoughts
on
it.
  Also because my initial reaction...was not conducive to calm and
rational discussion. ;-)

The tldr is that I don't like it.  To explain why, I'm going to
make
a
list (everyone loves lists, right? Top $NUMBER reasons we should
stop
expecting other people to write our API for us):

1) We've been down this road before.  Except last time it was
with
Heat.
  I'm being somewhat tongue-in-cheek here, but expecting a general
service to provide us a user-friendly API for our specific use
case
just
doesn't make sense to me.

We've been down this road with Heat yes. But we are currently using
Heat for some things that we arguable should be (a workflows tool
might
help offload some stuff out of Heat). Also we haven't implemented
custom Heat resources for TripleO either. There are mixed opinions
on
this but plugging in your code to a generic API is quite nice
sometimes.

That is the beauty of Mistral I think. Unlike Heat it actually
encourages you to customize it with custom Python actions. Anything
we
want in tripleo-common can become our own Mistral action (these get
registered with stevedore entry points so we'd own the code) and
the
YAML workflows just tie them together via tasks.

We don't have to go off and build our own proxy deployment workflow
API. The structure to do just about anything we need already exists
so
why not go and use it?


2) The TripleO API is not a workflow API.  I also largely missed
this
discussion, but the TripleO API is a _Deployment_ API.  In some
cases
there also happens to be a workflow going on behind the scenes,
but
honestly that's not something I want our users to have to care
about.

Agree that users don't have to care about this.

Users can get as involved as they want here. Most users I think
will
use python-tripleoclient to drive the deployment or the new UI.
They
don't have to interact with Mistral directly unless they really
want
to. So whether we choose to build our own API or use a generic one
I
think this point is mute.

Okay, I think this is a very fundamental point, and I believe it gets
right to the heart of my objection to the proposed change.

When I hear you say that users will use tripleoclient to talk to
Mistral, it raises a big flag.  Then I look at something like
https://github.com/dprince/python-tripleoclient/commit/77ffd2fa7b1642
b9f05713ca30b8a27ec4b322b7
and the flag gets bigger.

The thing is that there's a whole bunch of business logic currently
sitting in the client that shouldn't/can't be there.  There are
historical reasons for it, but the important thing is that the
current
client architecture is terribly flawed.  Business logic should never
live in the client like it does today.

Totally agree here. In fact I have removed business logic from python-
tripleoclient in this patch and moved it into a Mistral action. Which
can then be used via a stable API from anywhere.


Looking at that change, I see a bunch of business logic around taking
our configuration and passing it to Mistral.  In order for us to do
something like that and have a sustainable GUI, that code _has_ to
live
behind an API that the GUI and CLI alike can call.  If we ask the GUI
to
re-implement that code, then we're doomed to divergence between the
CLI
and GUI code and we'll most likely end up back where we are with a
GUI
that can't deploy half of our features because they were implemented
solely with the CLI in mind and made assumptions the GUI can't meet.

The latest feedback I've gotten from working with the UI developers on
this was that we should have a workflow to create the environment. That
would get called via the Mistral API via python-tripleoclient and any
sort of UI you could imagine and would essentially give us a stable
environment interface.

Anything that requires tripleoclient means !GUI though.  I know the
current GUI still has a bunch of dependencies on the CLI, but that seems
like something we need to fix, not a pattern to repeat.  I still think
any sentence containing "call Mistral via tripleoclient" is indicative
of a problem in the design.


I am not sure I understand the argument here.

Regardless of which API we use (Mistral API or TripleO API) GUI is going 
to call the API and tripleoclient (CLI) is going to call the API 
(through mistralclient - impl. detail).


GUI can't and does not call API through tripleoclient. This is why the 
work on extracting the common business logic to tripleo-common happened. 
So tripleo-common is the place which holds the business logic.


The proposed API (in the spec) is supposed only to 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Steven Hardy]

On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
> On 01/25/2016 03:56 PM, Steven Hardy wrote:
> > On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
> >> So I haven't weighed in on this yet, in part because I was on vacation
> >> when it was first proposed and missed a lot of the initial discussion,
> >> and also because I wanted to take some time to order my thoughts on it.
> >>  Also because my initial reaction...was not conducive to calm and
> >> rational discussion. ;-)
> >>
> >> The tldr is that I don't like it.  To explain why, I'm going to make a
> >> list (everyone loves lists, right? Top $NUMBER reasons we should stop
> >> expecting other people to write our API for us):
> >>
> >> 1) We've been down this road before.  Except last time it was with Heat.
> >>  I'm being somewhat tongue-in-cheek here, but expecting a general
> >> service to provide us a user-friendly API for our specific use case just
> >> doesn't make sense to me.
> > 
> > Actually, we've been down this road before with Tuskar, and discovered that
> > designing and maintaining a bespoke API for TripleO is really hard.
> 
> My takeaway from Tuskar was that designing an API that none of the
> developers on the project use is doomed to fail.  Tuskar also suffered
> from a lack of some features in Heat that the new API is explicitly
> depending on in an attempt to avoid many of the problems Tuskar had.
> 
> Problem #1 is still developer apathy IMHO though.

I think the main issue is developer capacity - we're a small community and
I for one am worried about the effort involved with building and
maintaining a bespoke API - thus this whole discussion is essentially about
finding a quicker and easier way to meet the needs of those needing an API.

In terms of apathy, I think as a developer I don't need an abstraction
between me, my templates and heat.  Some advanced operators will feel
likewise, others won't.  What I would find useful sometimes is a general
purpose workflow engine, which is where I think the more pluggable mistral
based solution may have advantages in terms of developer and advanced
operator uptake.

> > I somewhat agree that heat as an API is insufficient, but that doesn't
> > necessarily imply you have to have a TripleO specific abstraction, just
> > that *an* abstraction is required.
> > 
> >> 2) The TripleO API is not a workflow API.  I also largely missed this
> >> discussion, but the TripleO API is a _Deployment_ API.  In some cases
> >> there also happens to be a workflow going on behind the scenes, but
> >> honestly that's not something I want our users to have to care about.
> > 
> > How would you differentiate between "deployment" in a generic sense in
> > contrast to a generic workflow?
> > 
> > Every deployment I can think of involves a series of steps, involving some
> > choices and interactions with other services.  That *is* a workflow?
> 
> Well, I mean if we want to take this to extremes then pretty much every
> API is a workflow API.  You make a REST call, a "workflow" happens in
> the service, and you get back a result.
> 
> Let me turn this around: Would you implement Heat's API on Mistral?  All
> that happens when I call Heat is that a series of OpenStack calls are
> made from heat-engine, after all.  Or is that a gross oversimplification
> of what's happening?  I could argue that the same is true of this
> discussion. :-)

As Hugh has mentioned the main thing Heat does is actually manage
dependencies.  It processes the templates, builds a graph, then walks the
graph running a "workflow" to create/update/delete/etc each resource.

I could imagine a future where we interface to some external workflow tool to
e.g do each resource action (e.g create a nova server, poll until it's active),
however that's actually a pretty high overhead approach, and it'd probably
be better to move towards better use of notifications instead (e.g less
internal workflow)

> >> 3) It ties us 100% to a given implementation.  If Mistral proves to be a
> >> poor choice for some reason, or insufficient for a particular use case,
> >> we have no alternative.  If we have an API and decide to change our
> >> implementation, nobody has to know or care.  This is kind of the whole
> >> point of having an API - it shields users from all the nasty
> >> implementation details under the surface.
> > 
> > This is a valid argument, but building (and maintining, forever) a bespoke
> > API is a high cost to pay for this additional degree of abstraction, and
> > when you think of the target audience, I'm not certain it's entirely
> > justified (or, honestly, if our community can bear that overhead);
> > 
> > For example, take other single-purpose "deployment" projects, such as
> > Sahara, Magnum, perhaps Trove.  These are designed primarily as user-facing
> > API's, where the services will ultimately be consumed by public and private
> > cloud customers.
> > 
> > Contrast with TripleO, where our target audience is, for the 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Jiri Tomasek]

On 01/14/2016 12:54 PM, Steven Hardy wrote:

On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:

Hey all,

I realize now from the title of the other TripleO/Mistral thread [1] that
the discussion there may have gotten confused.  I think using Mistral for
TripleO processes that are obviously workflows - stack deployment, node
registration - makes perfect sense.  That thread is exploring practicalities
for doing that, and I think that's great work.

What I inappropriately started to address in that thread was a somewhat
orthogonal point that Dan asked in his original email, namely:

"what it might look like if we were to use Mistral as a replacement for the
TripleO API entirely"

I'd like to create this thread to talk about that; more of a 'should we'
than 'can we'.  And to do that, I want to indulge in a thought exercise
stemming from an IRC discussion with Dan and others.  All, please correct me
if I've misstated anything.

The IRC discussion revolved around one use case: deploying a Heat stack
directly from a Swift container.  With an updated patch, the Heat CLI can
support this functionality natively.  Then we don't need a TripleO API; we
can use Mistral to access that functionality, and we're done, with no need
for additional code within TripleO.  And, as I understand it, that's the
true motivation for using Mistral instead of a TripleO API: avoiding custom
code within TripleO.

That's definitely a worthy goal... except from my perspective, the story
doesn't quite end there.  A GUI needs additional functionality, which boils
down to: understanding the Heat deployment templates in order to provide
options for a user; and persisting those options within a Heat environment
file.

Right away I think we hit a problem.  Where does the code for 'understanding
options' go?  Much of that understanding comes from the capabilities map
in tripleo-heat-templates [2]; it would make sense to me that responsibility
for that would fall to a TripleO library.

Still, perhaps we can limit the amount of TripleO code.  So to give API
access to 'getDeploymentOptions', we can create a Mistral workflow.

   Retrieve Heat templates from Swift -> Parse capabilities map

Which is fine-ish, except from an architectural perspective
'getDeploymentOptions' violates the abstraction layer between storage and
business logic, a problem that is compounded because 'getDeploymentOptions'
is not the only functionality that accesses the Heat templates and needs
exposure through an API.  And, as has been discussed on a separate TripleO
thread, we're not even sure Swift is sufficient for our needs; one possible
consideration right now is allowing deployment from templates stored in
multiple places, such as the file system or git.

Actually, that whole capabilities map thing is a workaround for a missing
feature in Heat, which I have proposed, but am having a hard time reaching
consensus on within the Heat community:

https://review.openstack.org/#/c/196656/

Given that is a large part of what's anticipated to be provided by the
proposed TripleO API, I'd welcome feedback and collaboration so we can move
that forward, vs solving only for TripleO.


Yes, the original intent was to provide user with a means to safely 
construct the deployment template tree which iiuc is what this proposed 
feature provides. Then in the process of figuring out how to provide 
user with a reasonable choices for designing the deployment, I realized 
that more natural way is to provide the choices on the environments 
level (which is what we already do in THT - provide the list of 
alternative/complementary environments) The capabilities map provides a 
list of environments which user is able to choose from when designing 
the deployment supporting the choice with environment description and 
information about whether the environment is required to use, if it 
depends on the use of other environment or if the environment is one of 
the mutually exclusive choices.


So in addition to what the mentioned spec defines, to implement a 
replacement for capabilities_map in Heat, we'd need a way to specify 
environment description, other environments dependency, mutual 
exclusivity to other environments etc from within environment itself. 
Heat then could return a response which resembles to what 
capabilities_map provides now.


Maybe the current spec is sufficient and based on the information that 
this spec brings Heat would already be able to identify the 
inter-environment behavior based on parsing the resource_registry in the 
environments? Only thing we'd need to add is a 'description' to 
environment file.


-- Jirka




Are we going to have duplicate 'getDeploymentOptions' workflows for each
storage mechanism?  If we consolidate the storage code within a TripleO
library, do we really need a *workflow* to call a single function?  Is a
thin TripleO API that contains no additional business logic really so bad
at that point?

Actually, this is an argument 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Steven Hardy]

On Tue, Jan 26, 2016 at 07:40:05AM -0500, James Slagle wrote:
>On Tue, Jan 26, 2016 at 4:46 AM, Steven Hardy  wrote:
> 
>  On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
>  > On 01/25/2016 03:56 PM, Steven Hardy wrote:
>  > > On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
>  > >> So I haven't weighed in on this yet, in part because I was on
>  vacation
>  > >> when it was first proposed and missed a lot of the initial
>  discussion,
>  > >> and also because I wanted to take some time to order my thoughts on
>  it.
>  > >>  Also because my initial reaction...was not conducive to calm and
>  > >> rational discussion. ;-)
>  > >>
>  > >> The tldr is that I don't like it.  To explain why, I'm going to
>  make a
>  > >> list (everyone loves lists, right? Top $NUMBER reasons we should
>  stop
>  > >> expecting other people to write our API for us):
>  > >>
>  > >> 1) We've been down this road before.  Except last time it was with
>  Heat.
>  > >>  I'm being somewhat tongue-in-cheek here, but expecting a general
>  > >> service to provide us a user-friendly API for our specific use case
>  just
>  > >> doesn't make sense to me.
>  > >
>  > > Actually, we've been down this road before with Tuskar, and
>  discovered that
>  > > designing and maintaining a bespoke API for TripleO is really hard.
>  >
>  > My takeaway from Tuskar was that designing an API that none of the
>  > developers on the project use is doomed to fail.  Tuskar also
>  suffered
>  > from a lack of some features in Heat that the new API is explicitly
>  > depending on in an attempt to avoid many of the problems Tuskar had.
>  >
>  > Problem #1 is still developer apathy IMHO though.
> 
>  I think the main issue is developer capacity - we're a small community
>  and
>  I for one am worried about the effort involved with building and
>  maintaining a bespoke API - thus this whole discussion is essentially
>  about
>  finding a quicker and easier way to meet the needs of those needing an
>  API.
> 
>  In terms of apathy, I think as a developer I don't need an abstraction
>  between me, my templates and heat.  Some advanced operators will feel
>  likewise, others won't.  What I would find useful sometimes is a
>  general
>  purpose workflow engine, which is where I think the more pluggable
>  mistral
>  based solution may have advantages in terms of developer and advanced
>  operator uptake.
>  > > I somewhat agree that heat as an API is insufficient, but that
>  doesn't
>  > > necessarily imply you have to have a TripleO specific abstraction,
>  just
>  > > that *an* abstraction is required.
>  > >
>  > >> 2) The TripleO API is not a workflow API.  I also largely missed
>  this
>  > >> discussion, but the TripleO API is a _Deployment_ API.  In some
>  cases
>  > >> there also happens to be a workflow going on behind the scenes, but
>  > >> honestly that's not something I want our users to have to care
>  about.
>  > >
>  > > How would you differentiate between "deployment" in a generic sense
>  in
>  > > contrast to a generic workflow?
>  > >
>  > > Every deployment I can think of involves a series of steps,
>  involving some
>  > > choices and interactions with other services.  That *is* a
>  workflow?
>  >
>  > Well, I mean if we want to take this to extremes then pretty much
>  every
>  > API is a workflow API.  You make a REST call, a "workflow" happens in
>  > the service, and you get back a result.
>  >
>  > Let me turn this around: Would you implement Heat's API on Mistral? 
>  All
>  > that happens when I call Heat is that a series of OpenStack calls are
>  > made from heat-engine, after all.  Or is that a gross
>  oversimplification
>  > of what's happening?  I could argue that the same is true of this
>  > discussion. :-)
> 
>  As Hugh has mentioned the main thing Heat does is actually manage
>  dependencies.  It processes the templates, builds a graph, then walks
>  the
>  graph running a "workflow" to create/update/delete/etc each resource.
> 
>  I could imagine a future where we interface to some external workflow
>  tool to
>  e.g do each resource action (e.g create a nova server, poll until it's
>  active),
>  however that's actually a pretty high overhead approach, and it'd
>  probably
>  be better to move towards better use of notifications instead (e.g less
>  internal workflow)
>  > >> 3) It ties us 100% to a given implementation.  If Mistral proves
>  to be a
>  > >> poor choice for some reason, or insufficient for a particular use
>  case,
>  > >> we have no 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Tzu-Mainn Chen]

- Original Message -
> On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
> > On 01/25/2016 03:56 PM, Steven Hardy wrote:
> > > On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
> > >> So I haven't weighed in on this yet, in part because I was on vacation
> > >> when it was first proposed and missed a lot of the initial discussion,
> > >> and also because I wanted to take some time to order my thoughts on it.
> > >>  Also because my initial reaction...was not conducive to calm and
> > >> rational discussion. ;-)
> > >>
> > >> The tldr is that I don't like it.  To explain why, I'm going to make a
> > >> list (everyone loves lists, right? Top $NUMBER reasons we should stop
> > >> expecting other people to write our API for us):
> > >>
> > >> 1) We've been down this road before.  Except last time it was with Heat.
> > >>  I'm being somewhat tongue-in-cheek here, but expecting a general
> > >> service to provide us a user-friendly API for our specific use case just
> > >> doesn't make sense to me.
> > > 
> > > Actually, we've been down this road before with Tuskar, and discovered
> > > that
> > > designing and maintaining a bespoke API for TripleO is really hard.
> > 
> > My takeaway from Tuskar was that designing an API that none of the
> > developers on the project use is doomed to fail.  Tuskar also suffered
> > from a lack of some features in Heat that the new API is explicitly
> > depending on in an attempt to avoid many of the problems Tuskar had.
> > 
> > Problem #1 is still developer apathy IMHO though.
> 
> I think the main issue is developer capacity - we're a small community and
> I for one am worried about the effort involved with building and
> maintaining a bespoke API - thus this whole discussion is essentially about
> finding a quicker and easier way to meet the needs of those needing an API.
> 

Just a quick note about this; developer capacity works both ways.  On a
practical level, if we were to get involved with Mistral wouldn't we need
developers to get deeply involved with the Mistral community?  If Mistral
were to become the effective REST API interface for the whole deployment
API, bugs like https://bugs.launchpad.net/mistral/+bug/1423054 which affect
load would have to be fixed, right?

Mainn

>
> In terms of apathy, I think as a developer I don't need an abstraction
> between me, my templates and heat.  Some advanced operators will feel
> likewise, others won't.  What I would find useful sometimes is a general
> purpose workflow engine, which is where I think the more pluggable mistral
> based solution may have advantages in terms of developer and advanced
> operator uptake.
> 
> > > I somewhat agree that heat as an API is insufficient, but that doesn't
> > > necessarily imply you have to have a TripleO specific abstraction, just
> > > that *an* abstraction is required.
> > > 
> > >> 2) The TripleO API is not a workflow API.  I also largely missed this
> > >> discussion, but the TripleO API is a _Deployment_ API.  In some cases
> > >> there also happens to be a workflow going on behind the scenes, but
> > >> honestly that's not something I want our users to have to care about.
> > > 
> > > How would you differentiate between "deployment" in a generic sense in
> > > contrast to a generic workflow?
> > > 
> > > Every deployment I can think of involves a series of steps, involving
> > > some
> > > choices and interactions with other services.  That *is* a workflow?
> > 
> > Well, I mean if we want to take this to extremes then pretty much every
> > API is a workflow API.  You make a REST call, a "workflow" happens in
> > the service, and you get back a result.
> > 
> > Let me turn this around: Would you implement Heat's API on Mistral?  All
> > that happens when I call Heat is that a series of OpenStack calls are
> > made from heat-engine, after all.  Or is that a gross oversimplification
> > of what's happening?  I could argue that the same is true of this
> > discussion. :-)
> 
> As Hugh has mentioned the main thing Heat does is actually manage
> dependencies.  It processes the templates, builds a graph, then walks the
> graph running a "workflow" to create/update/delete/etc each resource.
> 
> I could imagine a future where we interface to some external workflow tool to
> e.g do each resource action (e.g create a nova server, poll until it's
> active),
> however that's actually a pretty high overhead approach, and it'd probably
> be better to move towards better use of notifications instead (e.g less
> internal workflow)
> 
> > >> 3) It ties us 100% to a given implementation.  If Mistral proves to be a
> > >> poor choice for some reason, or insufficient for a particular use case,
> > >> we have no alternative.  If we have an API and decide to change our
> > >> implementation, nobody has to know or care.  This is kind of the whole
> > >> point of having an API - it shields users from all the nasty
> > >> implementation details under the surface.
> > > 
> > > 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[James Slagle]

On Tue, Jan 26, 2016 at 9:08 AM, Steven Hardy  wrote:

> On Tue, Jan 26, 2016 at 07:40:05AM -0500, James Slagle wrote:
> >On Tue, Jan 26, 2016 at 4:46 AM, Steven Hardy 
> wrote:
> >
> >  On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
> >  > On 01/25/2016 03:56 PM, Steven Hardy wrote:
> >  > > On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
> >  > >> So I haven't weighed in on this yet, in part because I was on
> >  vacation
> >  > >> when it was first proposed and missed a lot of the initial
> >  discussion,
> >  > >> and also because I wanted to take some time to order my
> thoughts on
> >  it.
> >  > >>Â  Also because my initial reaction...was not conducive to calm
> and
> >  > >> rational discussion. ;-)
> >  > >>
> >  > >> The tldr is that I don't like it.  To explain why, I'm going
> to
> >  make a
> >  > >> list (everyone loves lists, right? Top $NUMBER reasons we
> should
> >  stop
> >  > >> expecting other people to write our API for us):
> >  > >>
> >  > >> 1) We've been down this road before.  Except last time it was
> with
> >  Heat.
> >  > >>Â  I'm being somewhat tongue-in-cheek here, but expecting a
> general
> >  > >> service to provide us a user-friendly API for our specific use
> case
> >  just
> >  > >> doesn't make sense to me.
> >  > >
> >  > > Actually, we've been down this road before with Tuskar, and
> >  discovered that
> >  > > designing and maintaining a bespoke API for TripleO is really
> hard.
> >  >
> >  > My takeaway from Tuskar was that designing an API that none of the
> >  > developers on the project use is doomed to fail.  Tuskar also
> >  suffered
> >  > from a lack of some features in Heat that the new API is
> explicitly
> >  > depending on in an attempt to avoid many of the problems Tuskar
> had.
> >  >
> >  > Problem #1 is still developer apathy IMHO though.
> >
> >  I think the main issue is developer capacity - we're a small
> community
> >  and
> >  I for one am worried about the effort involved with building and
> >  maintaining a bespoke API - thus this whole discussion is
> essentially
> >  about
> >  finding a quicker and easier way to meet the needs of those needing
> an
> >  API.
> >
> >  In terms of apathy, I think as a developer I don't need an
> abstraction
> >  between me, my templates and heat.  Some advanced operators will
> feel
> >  likewise, others won't.  What I would find useful sometimes is a
> >  general
> >  purpose workflow engine, which is where I think the more pluggable
> >  mistral
> >  based solution may have advantages in terms of developer and
> advanced
> >  operator uptake.
> >  > > I somewhat agree that heat as an API is insufficient, but that
> >  doesn't
> >  > > necessarily imply you have to have a TripleO specific
> abstraction,
> >  just
> >  > > that *an* abstraction is required.
> >  > >
> >  > >> 2) The TripleO API is not a workflow API.  I also largely
> missed
> >  this
> >  > >> discussion, but the TripleO API is a _Deployment_ API.  In
> some
> >  cases
> >  > >> there also happens to be a workflow going on behind the
> scenes, but
> >  > >> honestly that's not something I want our users to have to care
> >  about.
> >  > >
> >  > > How would you differentiate between "deployment" in a generic
> sense
> >  in
> >  > > contrast to a generic workflow?
> >  > >
> >  > > Every deployment I can think of involves a series of steps,
> >  involving some
> >  > > choices and interactions with other services.  That *is* a
> >  workflow?
> >  >
> >  > Well, I mean if we want to take this to extremes then pretty much
> >  every
> >  > API is a workflow API.  You make a REST call, a "workflow"
> happens in
> >  > the service, and you get back a result.
> >  >
> >  > Let me turn this around: Would you implement Heat's API on
> Mistral?Â
> >  All
> >  > that happens when I call Heat is that a series of OpenStack calls
> are
> >  > made from heat-engine, after all.  Or is that a gross
> >  oversimplification
> >  > of what's happening?  I could argue that the same is true of this
> >  > discussion. :-)
> >
> >  As Hugh has mentioned the main thing Heat does is actually manage
> >  dependencies.  It processes the templates, builds a graph, then
> walks
> >  the
> >  graph running a "workflow" to create/update/delete/etc each
> resource.
> >
> >  I could imagine a future where we interface to some external
> workflow
> >  tool to
> >  e.g do each resource action (e.g create a nova server, poll until
> it's
> >  active),
> >  however that's actually a pretty high overhead approach, and it'd
> 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Steven Hardy]

On Tue, Jan 26, 2016 at 09:23:00AM -0500, Tzu-Mainn Chen wrote:
> - Original Message -
> > On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
> > > On 01/25/2016 03:56 PM, Steven Hardy wrote:
> > > > On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
> > > >> So I haven't weighed in on this yet, in part because I was on vacation
> > > >> when it was first proposed and missed a lot of the initial discussion,
> > > >> and also because I wanted to take some time to order my thoughts on it.
> > > >>  Also because my initial reaction...was not conducive to calm and
> > > >> rational discussion. ;-)
> > > >>
> > > >> The tldr is that I don't like it.  To explain why, I'm going to make a
> > > >> list (everyone loves lists, right? Top $NUMBER reasons we should stop
> > > >> expecting other people to write our API for us):
> > > >>
> > > >> 1) We've been down this road before.  Except last time it was with 
> > > >> Heat.
> > > >>  I'm being somewhat tongue-in-cheek here, but expecting a general
> > > >> service to provide us a user-friendly API for our specific use case 
> > > >> just
> > > >> doesn't make sense to me.
> > > > 
> > > > Actually, we've been down this road before with Tuskar, and discovered
> > > > that
> > > > designing and maintaining a bespoke API for TripleO is really hard.
> > > 
> > > My takeaway from Tuskar was that designing an API that none of the
> > > developers on the project use is doomed to fail.  Tuskar also suffered
> > > from a lack of some features in Heat that the new API is explicitly
> > > depending on in an attempt to avoid many of the problems Tuskar had.
> > > 
> > > Problem #1 is still developer apathy IMHO though.
> > 
> > I think the main issue is developer capacity - we're a small community and
> > I for one am worried about the effort involved with building and
> > maintaining a bespoke API - thus this whole discussion is essentially about
> > finding a quicker and easier way to meet the needs of those needing an API.
> > 
> 
> Just a quick note about this; developer capacity works both ways.  On a
> practical level, if we were to get involved with Mistral wouldn't we need
> developers to get deeply involved with the Mistral community?  If Mistral
> were to become the effective REST API interface for the whole deployment
> API, bugs like https://bugs.launchpad.net/mistral/+bug/1423054 which affect
> load would have to be fixed, right?

Well, sure the advantage of not solving only for TripleO is that some
time can presumably be spent working on helping to improve Mistral instead
of writing a new API completely from scratch.  In general this is a good
thing, and to be encouraged, right? :)

You've mentioned that bug before, have you seen this?

http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html

I'm not sure it's even a bug, it's not confirmed as such and it sounds like
the configuration issues mentioned in that thread to me.

But, sure, there will be bugs, are you suggesting we'll have less if we
start from scratch with a TripleO specific API?  Be interested to
understand your reasoning if so :)

Cheers,

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Tue, 2016-01-26 at 09:56 -0500, James Slagle wrote:
> 
> 
> On Tue, Jan 26, 2016 at 9:08 AM, Steven Hardy 
> wrote:
> > On Tue, Jan 26, 2016 at 07:40:05AM -0500, James Slagle wrote:
> > >    On Tue, Jan 26, 2016 at 4:46 AM, Steven Hardy  > om> wrote:
> > >
> > >      On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
> > >      > On 01/25/2016 03:56 PM, Steven Hardy wrote:
> > >      > > On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec
> > wrote:
> > >      > >> So I haven't weighed in on this yet, in part because I
> > was on
> > >      vacation
> > >      > >> when it was first proposed and missed a lot of the
> > initial
> > >      discussion,
> > >      > >> and also because I wanted to take some time to order my
> > thoughts on
> > >      it.
> > >      > >>  Also because my initial reaction...was not conducive
> > to calm and
> > >      > >> rational discussion. ;-)
> > >      > >>
> > >      > >> The tldr is that I don't like it.  To explain why, I'm
> > going to
> > >      make a
> > >      > >> list (everyone loves lists, right? Top $NUMBER reasons
> > we should
> > >      stop
> > >      > >> expecting other people to write our API for us):
> > >      > >>
> > >      > >> 1) We've been down this road before.  Except last time
> > it was with
> > >      Heat.
> > >      > >>  I'm being somewhat tongue-in-cheek here, but
> > expecting a general
> > >      > >> service to provide us a user-friendly API for our
> > specific use case
> > >      just
> > >      > >> doesn't make sense to me.
> > >      > >
> > >      > > Actually, we've been down this road before with Tuskar,
> > and
> > >      discovered that
> > >      > > designing and maintaining a bespoke API for TripleO is
> > really hard.
> > >      >
> > >      > My takeaway from Tuskar was that designing an API that
> > none of the
> > >      > developers on the project use is doomed to fail.  Tuskar
> > also
> > >      suffered
> > >      > from a lack of some features in Heat that the new API is
> > explicitly
> > >      > depending on in an attempt to avoid many of the problems
> > Tuskar had.
> > >      >
> > >      > Problem #1 is still developer apathy IMHO though.
> > >
> > >      I think the main issue is developer capacity - we're a small
> > community
> > >      and
> > >      I for one am worried about the effort involved with building
> > and
> > >      maintaining a bespoke API - thus this whole discussion is
> > essentially
> > >      about
> > >      finding a quicker and easier way to meet the needs of those
> > needing an
> > >      API.
> > >
> > >      In terms of apathy, I think as a developer I don't need an
> > abstraction
> > >      between me, my templates and heat.  Some advanced operators
> > will feel
> > >      likewise, others won't.  What I would find useful sometimes
> > is a
> > >      general
> > >      purpose workflow engine, which is where I think the more
> > pluggable
> > >      mistral
> > >      based solution may have advantages in terms of developer and
> > advanced
> > >      operator uptake.
> > >      > > I somewhat agree that heat as an API is insufficient,
> > but that
> > >      doesn't
> > >      > > necessarily imply you have to have a TripleO specific
> > abstraction,
> > >      just
> > >      > > that *an* abstraction is required.
> > >      > >
> > >      > >> 2) The TripleO API is not a workflow API.  I also
> > largely missed
> > >      this
> > >      > >> discussion, but the TripleO API is a _Deployment_
> > API.  In some
> > >      cases
> > >      > >> there also happens to be a workflow going on behind the
> > scenes, but
> > >      > >> honestly that's not something I want our users to have
> > to care
> > >      about.
> > >      > >
> > >      > > How would you differentiate between "deployment" in a
> > generic sense
> > >      in
> > >      > > contrast to a generic workflow?
> > >      > >
> > >      > > Every deployment I can think of involves a series of
> > steps,
> > >      involving some
> > >      > > choices and interactions with other services.  That
> > *is* a
> > >      workflow?
> > >      >
> > >      > Well, I mean if we want to take this to extremes then
> > pretty much
> > >      every
> > >      > API is a workflow API.  You make a REST call, a
> > "workflow" happens in
> > >      > the service, and you get back a result.
> > >      >
> > >      > Let me turn this around: Would you implement Heat's API on
> > Mistral? 
> > >      All
> > >      > that happens when I call Heat is that a series of
> > OpenStack calls are
> > >      > made from heat-engine, after all.  Or is that a gross
> > >      oversimplification
> > >      > of what's happening?  I could argue that the same is true
> > of this
> > >      > discussion. :-)
> > >
> > >      As Hugh has mentioned the main thing Heat does is actually
> > manage
> > >      dependencies.  It processes the templates, builds a graph,
> > then walks
> > 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Tzu-Mainn Chen]

- Original Message -
> On Tue, Jan 26, 2016 at 09:23:00AM -0500, Tzu-Mainn Chen wrote:
> > - Original Message -
> > > On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
> > > > On 01/25/2016 03:56 PM, Steven Hardy wrote:
> > > > > On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
> > > > >> So I haven't weighed in on this yet, in part because I was on
> > > > >> vacation
> > > > >> when it was first proposed and missed a lot of the initial
> > > > >> discussion,
> > > > >> and also because I wanted to take some time to order my thoughts on
> > > > >> it.
> > > > >>  Also because my initial reaction...was not conducive to calm and
> > > > >> rational discussion. ;-)
> > > > >>
> > > > >> The tldr is that I don't like it.  To explain why, I'm going to make
> > > > >> a
> > > > >> list (everyone loves lists, right? Top $NUMBER reasons we should
> > > > >> stop
> > > > >> expecting other people to write our API for us):
> > > > >>
> > > > >> 1) We've been down this road before.  Except last time it was with
> > > > >> Heat.
> > > > >>  I'm being somewhat tongue-in-cheek here, but expecting a general
> > > > >> service to provide us a user-friendly API for our specific use case
> > > > >> just
> > > > >> doesn't make sense to me.
> > > > > 
> > > > > Actually, we've been down this road before with Tuskar, and
> > > > > discovered
> > > > > that
> > > > > designing and maintaining a bespoke API for TripleO is really hard.
> > > > 
> > > > My takeaway from Tuskar was that designing an API that none of the
> > > > developers on the project use is doomed to fail.  Tuskar also suffered
> > > > from a lack of some features in Heat that the new API is explicitly
> > > > depending on in an attempt to avoid many of the problems Tuskar had.
> > > > 
> > > > Problem #1 is still developer apathy IMHO though.
> > > 
> > > I think the main issue is developer capacity - we're a small community
> > > and
> > > I for one am worried about the effort involved with building and
> > > maintaining a bespoke API - thus this whole discussion is essentially
> > > about
> > > finding a quicker and easier way to meet the needs of those needing an
> > > API.
> > > 
> > 
> > Just a quick note about this; developer capacity works both ways.  On a
> > practical level, if we were to get involved with Mistral wouldn't we need
> > developers to get deeply involved with the Mistral community?  If Mistral
> > were to become the effective REST API interface for the whole deployment
> > API, bugs like https://bugs.launchpad.net/mistral/+bug/1423054 which affect
> > load would have to be fixed, right?
> 
> Well, sure the advantage of not solving only for TripleO is that some
> time can presumably be spent working on helping to improve Mistral instead
> of writing a new API completely from scratch.  In general this is a good
> thing, and to be encouraged, right? :)
>

I don't think this is an entirely fair comparison.  Using Mistral as our REST
API feels like a new use of Mistral, and we're not expecting other OpenStack
projects to follow suit, are we?  I think there may be unknown consequences to
putting a workflow engine in front of every single REST API request.

> 
> You've mentioned that bug before, have you seen this?
> 
> http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html
> 
> I'm not sure it's even a bug, it's not confirmed as such and it sounds like
> the configuration issues mentioned in that thread to me.
> 

Fair enough, thanks for pointing that out!

> But, sure, there will be bugs, are you suggesting we'll have less if we
> start from scratch with a TripleO specific API?  Be interested to
> understand your reasoning if so :)
> 

I think there's a slightly subtle point being missed.  Sure, the Mistral API
exists, but presumably we'd create workflows corresponding to each proposed
TripleO API function.  Those workflows would be our true 'API', and I expect
that TripleO would have to provide a guarantee of stability there.  I would
expect development of those workflows to have roughly the same amount of
issues as creating a TripleO API.

If we're talking solely about creating a REST API - then yeah, I think that
creating a REST API is well-demonstrated throughout OpenStack, and using
Mistral as the REST API interface is less so.


Mainn

> Cheers,
> 
> Steve
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[James Slagle]

On Tue, Jan 26, 2016 at 10:27 AM, Dan Prince  wrote:

> On Tue, 2016-01-26 at 09:56 -0500, James Slagle wrote:
> >
> >
> > On Tue, Jan 26, 2016 at 9:08 AM, Steven Hardy 
> > wrote:
> > > On Tue, Jan 26, 2016 at 07:40:05AM -0500, James Slagle wrote:
> > > >On Tue, Jan 26, 2016 at 4:46 AM, Steven Hardy  > > om> wrote:
> > > >
> > > >  On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
> > > >  > On 01/25/2016 03:56 PM, Steven Hardy wrote:
> > > >  > > On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec
> > > wrote:
> > > >  > >> So I haven't weighed in on this yet, in part because I
> > > was on
> > > >  vacation
> > > >  > >> when it was first proposed and missed a lot of the
> > > initial
> > > >  discussion,
> > > >  > >> and also because I wanted to take some time to order my
> > > thoughts on
> > > >  it.
> > > >  > >>Â  Also because my initial reaction...was not conducive
> > > to calm and
> > > >  > >> rational discussion. ;-)
> > > >  > >>
> > > >  > >> The tldr is that I don't like it.  To explain why, I'm
> > > going to
> > > >  make a
> > > >  > >> list (everyone loves lists, right? Top $NUMBER reasons
> > > we should
> > > >  stop
> > > >  > >> expecting other people to write our API for us):
> > > >  > >>
> > > >  > >> 1) We've been down this road before.  Except last time
> > > it was with
> > > >  Heat.
> > > >  > >>Â  I'm being somewhat tongue-in-cheek here, but
> > > expecting a general
> > > >  > >> service to provide us a user-friendly API for our
> > > specific use case
> > > >  just
> > > >  > >> doesn't make sense to me.
> > > >  > >
> > > >  > > Actually, we've been down this road before with Tuskar,
> > > and
> > > >  discovered that
> > > >  > > designing and maintaining a bespoke API for TripleO is
> > > really hard.
> > > >  >
> > > >  > My takeaway from Tuskar was that designing an API that
> > > none of the
> > > >  > developers on the project use is doomed to fail.  Tuskar
> > > also
> > > >  suffered
> > > >  > from a lack of some features in Heat that the new API is
> > > explicitly
> > > >  > depending on in an attempt to avoid many of the problems
> > > Tuskar had.
> > > >  >
> > > >  > Problem #1 is still developer apathy IMHO though.
> > > >
> > > >  I think the main issue is developer capacity - we're a small
> > > community
> > > >  and
> > > >  I for one am worried about the effort involved with building
> > > and
> > > >  maintaining a bespoke API - thus this whole discussion is
> > > essentially
> > > >  about
> > > >  finding a quicker and easier way to meet the needs of those
> > > needing an
> > > >  API.
> > > >
> > > >  In terms of apathy, I think as a developer I don't need an
> > > abstraction
> > > >  between me, my templates and heat.  Some advanced operators
> > > will feel
> > > >  likewise, others won't.  What I would find useful sometimes
> > > is a
> > > >  general
> > > >  purpose workflow engine, which is where I think the more
> > > pluggable
> > > >  mistral
> > > >  based solution may have advantages in terms of developer and
> > > advanced
> > > >  operator uptake.
> > > >  > > I somewhat agree that heat as an API is insufficient,
> > > but that
> > > >  doesn't
> > > >  > > necessarily imply you have to have a TripleO specific
> > > abstraction,
> > > >  just
> > > >  > > that *an* abstraction is required.
> > > >  > >
> > > >  > >> 2) The TripleO API is not a workflow API.  I also
> > > largely missed
> > > >  this
> > > >  > >> discussion, but the TripleO API is a _Deployment_
> > > API.  In some
> > > >  cases
> > > >  > >> there also happens to be a workflow going on behind the
> > > scenes, but
> > > >  > >> honestly that's not something I want our users to have
> > > to care
> > > >  about.
> > > >  > >
> > > >  > > How would you differentiate between "deployment" in a
> > > generic sense
> > > >  in
> > > >  > > contrast to a generic workflow?
> > > >  > >
> > > >  > > Every deployment I can think of involves a series of
> > > steps,
> > > >  involving some
> > > >  > > choices and interactions with other services.  That
> > > *is* a
> > > >  workflow?
> > > >  >
> > > >  > Well, I mean if we want to take this to extremes then
> > > pretty much
> > > >  every
> > > >  > API is a workflow API.  You make a REST call, a
> > > "workflow" happens in
> > > >  > the service, and you get back a result.
> > > >  >
> > > >  > Let me turn this around: Would you implement Heat's API on
> > > Mistral?Â
> > > >  All
> > > >  > that happens when I call Heat is that a series of
> > > OpenStack calls are
> > > >  > made from heat-engine, after all.  Or 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Mon, 2016-01-25 at 15:08 -0500, James Slagle wrote:
> 
> 
> On Mon, Jan 25, 2016 at 1:09 PM, Dan Prince 
> wrote:
> >  
> > As for tripleo-heat-templates... sure it hasn't been an entirely
> > smooth
> > ride. I think a lot of the pain has to do with a lack of CI
> > coverage on
> > proper upgrade paths upstream. I think this has less to do with the
> > architecture (certainly not the fact that it uses YAML) and more to
> > do
> > with the fact that we may have cut corners, jammed a bunch of
> > features
> > in really fast, and don't have CI to cover proper upgrades jobs.
> I think it has a lot less to do with that then it may appear on the
> surface. Even if we had 100% functional test coverage of upgrade
> paths, that does nothing to solve for scenarios where the templates
> have been so heavily customized that stack updates can not be
> guaranteed to work, or worse, destroy the cloud.
> 
> The most we could do in those scenarios is better detection around
> what stack changes are going to be done by an update before anything
> is actually applied. And if something is going to be destructive or
> is not expected, we refuse to proceed. That would definitely help us
> as developers to know if our own changes are breaking upgrades.
> 
> But, that doesn't really help the user get their cloud upgraded if
> they've customized templates themselves, other than they know that
> they must undo whatever custom changes they made, and if they made
> those for valid reasons and can't undo, then they are stuck.
> 
> They've taken what may appear as a "feature" of the system --
> customizing templates to add support for new things -- and turned it
> into an anti-feature, now they can't upgrade without writing upgrade
> support for whatever they added themself. Perhaps having to undo new
> features we as TripleO developers added for users in the first place.
> 
> I don't think that's a supportable path for users of TripleO, nor
> sustainable as a community, so editing templates should be advertised
> as unsupported.
> 
>  
> >  
> > The fact that projects like Heat, Mistral, and Ansible use YAML is
> > to
> > our advantage as developers. I think it allows us to work faster by
> > avoiding rather large amounts of code to represent the same sorts
> > of
> > things. Please, lets not argue that by writing things in Python we
> > deter end users from editing things we don't want them too. Instead
> > let
> > the choice be ours... not as Python developers... but as TripleO
> > deployment developers who choose the write tools for the right
> > jobs.
> > 
> 
> Agreed, we can't deter anyone from editing python anymore than we
> could yaml. 
> 
> But it's not about dictating what people do. It's about what we
> support as a project for users and operators. This is something we
> can define very clearly so that users aren't surprised. At least they
> would have some way of knowing if they are using the system as
> intended.
> 
> I don't think we should support/encourage editing the yaml workflows
> by end users if we choose a yaml based workflow tool, any more than
> we should support editing python code by users if we wrote our own
> REST API.
> 
> It's not clear to me in this discussion when the argument about yaml
> files being easier to edit so Mistral/Ansible is a better choice...if
> we're talking about developers editing yaml files, or users editing
> yaml files.
> 
> If it's the former (developers), I agree it's a benefit for us.
> 
> If it's the latter (users), then personally I don't see that as a
> valid argument at all since I view it as something we should never
> do.

/me missed this in his previous reply

Its both.

For the latter we want our architecture to be something that users can
"contribute" to. All edits aren't bad. One example: There are some
cases today where we designed our templates as examples (potentially
needing edits). The network configuration templates are an example
here. No harm in adding an extra NIC to a bond or this sort of thing if
the operator/user knows that is how they need to manage their physical
hardware. The smart operator would perhaps build an automatic tool to
help generate these sorts of templates. And perhaps we need a tool to
help generate/validate these templates... but today the lay of the land
is it is okay to customize some of these templates because they were in
fact designed to be customized (os-net-config, etc.)

Dan

> 
>  
> >  >
> > > So whether the API is a new TripleO Rest API, or the Mistral Rest
> > > API, I hope we can agree that the API is the stable part, and the
> > > implementation is not meant to be edited by users.
> > 
> > We do totally agree on the stable API part. :)
> 
> To be clear, are you saying we should have a stable API, and then
> support users in editing the implementation of that API (yaml
> workflows)?
> 
> For example, we have a stable API such as:
> 
> POST /workbooks/deploy-cloud/executions
> 
> that might do totally 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Tue, 2016-01-26 at 09:23 -0500, Tzu-Mainn Chen wrote:
> - Original Message -
> > On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
> > > On 01/25/2016 03:56 PM, Steven Hardy wrote:
> > > > On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
> > > > > So I haven't weighed in on this yet, in part because I was on
> > > > > vacation
> > > > > when it was first proposed and missed a lot of the initial
> > > > > discussion,
> > > > > and also because I wanted to take some time to order my
> > > > > thoughts on it.
> > > > >  Also because my initial reaction...was not conducive to calm
> > > > > and
> > > > > rational discussion. ;-)
> > > > > 
> > > > > The tldr is that I don't like it.  To explain why, I'm going
> > > > > to make a
> > > > > list (everyone loves lists, right? Top $NUMBER reasons we
> > > > > should stop
> > > > > expecting other people to write our API for us):
> > > > > 
> > > > > 1) We've been down this road before.  Except last time it was
> > > > > with Heat.
> > > > >  I'm being somewhat tongue-in-cheek here, but expecting a
> > > > > general
> > > > > service to provide us a user-friendly API for our specific
> > > > > use case just
> > > > > doesn't make sense to me.
> > > > 
> > > > Actually, we've been down this road before with Tuskar, and
> > > > discovered
> > > > that
> > > > designing and maintaining a bespoke API for TripleO is really
> > > > hard.
> > > 
> > > My takeaway from Tuskar was that designing an API that none of
> > > the
> > > developers on the project use is doomed to fail.  Tuskar also
> > > suffered
> > > from a lack of some features in Heat that the new API is
> > > explicitly
> > > depending on in an attempt to avoid many of the problems Tuskar
> > > had.
> > > 
> > > Problem #1 is still developer apathy IMHO though.
> > 
> > I think the main issue is developer capacity - we're a small
> > community and
> > I for one am worried about the effort involved with building and
> > maintaining a bespoke API - thus this whole discussion is
> > essentially about
> > finding a quicker and easier way to meet the needs of those needing
> > an API.
> > 
> 
> Just a quick note about this; developer capacity works both ways.  On
> a
> practical level, if we were to get involved with Mistral wouldn't we
> need
> developers to get deeply involved with the Mistral community?  If
> Mistral
> were to become the effective REST API interface for the whole
> deployment
> API, bugs like https://bugs.launchpad.net/mistral/+bug/1423054 which 
> affect
> load would have to be fixed, right?

Getting involved in a community is an opportunity. Yes, there are bugs
that might need to be fixed but it also means that our (smaller)
TripleO team isn't the only team who will help fix them.

You are entitle to your opinion but I don't find any of the bugs you've
linked into these discussions to be particularly relevant. Any tool we
use, including one that we build ourselves would likely have bugs
associated with them as well. Given our version is just getting started
I honestly think we might have more, not less issues if we choose to go
it alone.

Dan


> 
> Mainn
> 
> > 
> > In terms of apathy, I think as a developer I don't need an
> > abstraction
> > between me, my templates and heat.  Some advanced operators will
> > feel
> > likewise, others won't.  What I would find useful sometimes is a
> > general
> > purpose workflow engine, which is where I think the more pluggable
> > mistral
> > based solution may have advantages in terms of developer and
> > advanced
> > operator uptake.
> > 
> > > > I somewhat agree that heat as an API is insufficient, but that
> > > > doesn't
> > > > necessarily imply you have to have a TripleO specific
> > > > abstraction, just
> > > > that *an* abstraction is required.
> > > > 
> > > > > 2) The TripleO API is not a workflow API.  I also largely
> > > > > missed this
> > > > > discussion, but the TripleO API is a _Deployment_ API.  In
> > > > > some cases
> > > > > there also happens to be a workflow going on behind the
> > > > > scenes, but
> > > > > honestly that's not something I want our users to have to
> > > > > care about.
> > > > 
> > > > How would you differentiate between "deployment" in a generic
> > > > sense in
> > > > contrast to a generic workflow?
> > > > 
> > > > Every deployment I can think of involves a series of steps,
> > > > involving
> > > > some
> > > > choices and interactions with other services.  That *is* a
> > > > workflow?
> > > 
> > > Well, I mean if we want to take this to extremes then pretty much
> > > every
> > > API is a workflow API.  You make a REST call, a "workflow"
> > > happens in
> > > the service, and you get back a result.
> > > 
> > > Let me turn this around: Would you implement Heat's API on
> > > Mistral?  All
> > > that happens when I call Heat is that a series of OpenStack calls
> > > are
> > > made from heat-engine, after all.  Or is that a gross
> > > oversimplification
> > > of what's 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Ben Nemec]

On 01/26/2016 03:46 AM, Steven Hardy wrote:
> On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
>> On 01/25/2016 03:56 PM, Steven Hardy wrote:
>>> On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
 So I haven't weighed in on this yet, in part because I was on vacation
 when it was first proposed and missed a lot of the initial discussion,
 and also because I wanted to take some time to order my thoughts on it.
  Also because my initial reaction...was not conducive to calm and
 rational discussion. ;-)

 The tldr is that I don't like it.  To explain why, I'm going to make a
 list (everyone loves lists, right? Top $NUMBER reasons we should stop
 expecting other people to write our API for us):

 1) We've been down this road before.  Except last time it was with Heat.
  I'm being somewhat tongue-in-cheek here, but expecting a general
 service to provide us a user-friendly API for our specific use case just
 doesn't make sense to me.
>>>
>>> Actually, we've been down this road before with Tuskar, and discovered that
>>> designing and maintaining a bespoke API for TripleO is really hard.
>>
>> My takeaway from Tuskar was that designing an API that none of the
>> developers on the project use is doomed to fail.  Tuskar also suffered
>> from a lack of some features in Heat that the new API is explicitly
>> depending on in an attempt to avoid many of the problems Tuskar had.
>>
>> Problem #1 is still developer apathy IMHO though.
> 
> I think the main issue is developer capacity - we're a small community and
> I for one am worried about the effort involved with building and
> maintaining a bespoke API - thus this whole discussion is essentially about
> finding a quicker and easier way to meet the needs of those needing an API.
> 
> In terms of apathy, I think as a developer I don't need an abstraction
> between me, my templates and heat.  Some advanced operators will feel
> likewise, others won't.  What I would find useful sometimes is a general
> purpose workflow engine, which is where I think the more pluggable mistral
> based solution may have advantages in terms of developer and advanced
> operator uptake.

The API is for end users, not developers.  tripleo-incubator was easily
hackable for developers and power users.  It was unusable for everyone else.

> 
>>> I somewhat agree that heat as an API is insufficient, but that doesn't
>>> necessarily imply you have to have a TripleO specific abstraction, just
>>> that *an* abstraction is required.
>>>
 2) The TripleO API is not a workflow API.  I also largely missed this
 discussion, but the TripleO API is a _Deployment_ API.  In some cases
 there also happens to be a workflow going on behind the scenes, but
 honestly that's not something I want our users to have to care about.
>>>
>>> How would you differentiate between "deployment" in a generic sense in
>>> contrast to a generic workflow?
>>>
>>> Every deployment I can think of involves a series of steps, involving some
>>> choices and interactions with other services.  That *is* a workflow?
>>
>> Well, I mean if we want to take this to extremes then pretty much every
>> API is a workflow API.  You make a REST call, a "workflow" happens in
>> the service, and you get back a result.
>>
>> Let me turn this around: Would you implement Heat's API on Mistral?  All
>> that happens when I call Heat is that a series of OpenStack calls are
>> made from heat-engine, after all.  Or is that a gross oversimplification
>> of what's happening?  I could argue that the same is true of this
>> discussion. :-)
> 
> As Hugh has mentioned the main thing Heat does is actually manage
> dependencies.  It processes the templates, builds a graph, then walks the
> graph running a "workflow" to create/update/delete/etc each resource.
> 
> I could imagine a future where we interface to some external workflow tool to
> e.g do each resource action (e.g create a nova server, poll until it's 
> active),
> however that's actually a pretty high overhead approach, and it'd probably
> be better to move towards better use of notifications instead (e.g less
> internal workflow)
> 
 3) It ties us 100% to a given implementation.  If Mistral proves to be a
 poor choice for some reason, or insufficient for a particular use case,
 we have no alternative.  If we have an API and decide to change our
 implementation, nobody has to know or care.  This is kind of the whole
 point of having an API - it shields users from all the nasty
 implementation details under the surface.
>>>
>>> This is a valid argument, but building (and maintining, forever) a bespoke
>>> API is a high cost to pay for this additional degree of abstraction, and
>>> when you think of the target audience, I'm not certain it's entirely
>>> justified (or, honestly, if our community can bear that overhead);
>>>
>>> For example, take other single-purpose "deployment" projects, such 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[James Slagle]

On Tue, Jan 26, 2016 at 10:46 AM, Dan Prince  wrote:

> On Mon, 2016-01-25 at 15:08 -0500, James Slagle wrote:
> >
> >
> > On Mon, Jan 25, 2016 at 1:09 PM, Dan Prince 
> > wrote:
> > >
> > > As for tripleo-heat-templates... sure it hasn't been an entirely
> > > smooth
> > > ride. I think a lot of the pain has to do with a lack of CI
> > > coverage on
> > > proper upgrade paths upstream. I think this has less to do with the
> > > architecture (certainly not the fact that it uses YAML) and more to
> > > do
> > > with the fact that we may have cut corners, jammed a bunch of
> > > features
> > > in really fast, and don't have CI to cover proper upgrades jobs.
> > I think it has a lot less to do with that then it may appear on the
> > surface. Even if we had 100% functional test coverage of upgrade
> > paths, that does nothing to solve for scenarios where the templates
> > have been so heavily customized that stack updates can not be
> > guaranteed to work, or worse, destroy the cloud.
> >
> > The most we could do in those scenarios is better detection around
> > what stack changes are going to be done by an update before anything
> > is actually applied. And if something is going to be destructive or
> > is not expected, we refuse to proceed. That would definitely help us
> > as developers to know if our own changes are breaking upgrades.
> >
> > But, that doesn't really help the user get their cloud upgraded if
> > they've customized templates themselves, other than they know that
> > they must undo whatever custom changes they made, and if they made
> > those for valid reasons and can't undo, then they are stuck.
> >
> > They've taken what may appear as a "feature" of the system --
> > customizing templates to add support for new things -- and turned it
> > into an anti-feature, now they can't upgrade without writing upgrade
> > support for whatever they added themself. Perhaps having to undo new
> > features we as TripleO developers added for users in the first place.
> >
> > I don't think that's a supportable path for users of TripleO, nor
> > sustainable as a community, so editing templates should be advertised
> > as unsupported.
> >
> >
> > >
> > > The fact that projects like Heat, Mistral, and Ansible use YAML is
> > > to
> > > our advantage as developers. I think it allows us to work faster by
> > > avoiding rather large amounts of code to represent the same sorts
> > > of
> > > things. Please, lets not argue that by writing things in Python we
> > > deter end users from editing things we don't want them too. Instead
> > > let
> > > the choice be ours... not as Python developers... but as TripleO
> > > deployment developers who choose the write tools for the right
> > > jobs.
> > >
> >
> > Agreed, we can't deter anyone from editing python anymore than we
> > could yaml.
> >
> > But it's not about dictating what people do. It's about what we
> > support as a project for users and operators. This is something we
> > can define very clearly so that users aren't surprised. At least they
> > would have some way of knowing if they are using the system as
> > intended.
> >
> > I don't think we should support/encourage editing the yaml workflows
> > by end users if we choose a yaml based workflow tool, any more than
> > we should support editing python code by users if we wrote our own
> > REST API.
> >
> > It's not clear to me in this discussion when the argument about yaml
> > files being easier to edit so Mistral/Ansible is a better choice...if
> > we're talking about developers editing yaml files, or users editing
> > yaml files.
> >
> > If it's the former (developers), I agree it's a benefit for us.
> >
> > If it's the latter (users), then personally I don't see that as a
> > valid argument at all since I view it as something we should never
> > do.
>
> /me missed this in his previous reply
>
> Its both.
>
> For the latter we want our architecture to be something that users can
> "contribute" to. All edits aren't bad. One example: There are some
> cases today where we designed our templates as examples (potentially
> needing edits). The network configuration templates are an example
> here. No harm in adding an extra NIC to a bond or this sort of thing if
> the operator/user knows that is how they need to manage their physical
> hardware. The smart operator would perhaps build an automatic tool to
> help generate these sorts of templates. And perhaps we need a tool to
> help generate/validate these templates... but today the lay of the land
> is it is okay to customize some of these templates because they were in
> fact designed to be customized (os-net-config, etc.)
>

​What I'm arguing for then is that the distinction and the interface be
defined. The reason being that today in tripleo-heat-templates it is not
defined and we've left users in limbo. You say the nic config templates are
examples, that's fine. But we haven't defined under what circumstances we,
as 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Steven Hardy]

On Tue, Jan 26, 2016 at 10:01:56AM -0600, Ben Nemec wrote:
> On 01/26/2016 03:46 AM, Steven Hardy wrote:
> > On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
> >> On 01/25/2016 03:56 PM, Steven Hardy wrote:
> >>> On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
>  So I haven't weighed in on this yet, in part because I was on vacation
>  when it was first proposed and missed a lot of the initial discussion,
>  and also because I wanted to take some time to order my thoughts on it.
>   Also because my initial reaction...was not conducive to calm and
>  rational discussion. ;-)
> 
>  The tldr is that I don't like it.  To explain why, I'm going to make a
>  list (everyone loves lists, right? Top $NUMBER reasons we should stop
>  expecting other people to write our API for us):
> 
>  1) We've been down this road before.  Except last time it was with Heat.
>   I'm being somewhat tongue-in-cheek here, but expecting a general
>  service to provide us a user-friendly API for our specific use case just
>  doesn't make sense to me.
> >>>
> >>> Actually, we've been down this road before with Tuskar, and discovered 
> >>> that
> >>> designing and maintaining a bespoke API for TripleO is really hard.
> >>
> >> My takeaway from Tuskar was that designing an API that none of the
> >> developers on the project use is doomed to fail.  Tuskar also suffered
> >> from a lack of some features in Heat that the new API is explicitly
> >> depending on in an attempt to avoid many of the problems Tuskar had.
> >>
> >> Problem #1 is still developer apathy IMHO though.
> > 
> > I think the main issue is developer capacity - we're a small community and
> > I for one am worried about the effort involved with building and
> > maintaining a bespoke API - thus this whole discussion is essentially about
> > finding a quicker and easier way to meet the needs of those needing an API.
> > 
> > In terms of apathy, I think as a developer I don't need an abstraction
> > between me, my templates and heat.  Some advanced operators will feel
> > likewise, others won't.  What I would find useful sometimes is a general
> > purpose workflow engine, which is where I think the more pluggable mistral
> > based solution may have advantages in terms of developer and advanced
> > operator uptake.
> 
> The API is for end users, not developers.  tripleo-incubator was easily
> hackable for developers and power users.  It was unusable for everyone else.

I'm sorry but you are contradicting your previous "doomed to fail"
assertion here.

Maybe it's time we took a step back, away from the invariably flamey
discussion re implementation details, and focus on what the actual
requirement is.

1. We need an opinionated and strongly versioned abstraction for UIs and
non-advanced CLI use-cases.

2. We also need that abstraction to add some value to advanced users and
developers, or they won't use it.

How can we do both?

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Ben Nemec]

On 01/25/2016 04:36 PM, Dan Prince wrote:
> On Mon, 2016-01-25 at 15:31 -0600, Ben Nemec wrote:
>> On 01/22/2016 06:19 PM, Dan Prince wrote:
>>> On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
 So I haven't weighed in on this yet, in part because I was on
 vacation
 when it was first proposed and missed a lot of the initial
 discussion,
 and also because I wanted to take some time to order my thoughts
 on
 it.
  Also because my initial reaction...was not conducive to calm and
 rational discussion. ;-)

 The tldr is that I don't like it.  To explain why, I'm going to
 make
 a
 list (everyone loves lists, right? Top $NUMBER reasons we should
 stop
 expecting other people to write our API for us):

 1) We've been down this road before.  Except last time it was
 with
 Heat.
  I'm being somewhat tongue-in-cheek here, but expecting a general
 service to provide us a user-friendly API for our specific use
 case
 just
 doesn't make sense to me.
>>>
>>> We've been down this road with Heat yes. But we are currently using
>>> Heat for some things that we arguable should be (a workflows tool
>>> might
>>> help offload some stuff out of Heat). Also we haven't implemented
>>> custom Heat resources for TripleO either. There are mixed opinions
>>> on
>>> this but plugging in your code to a generic API is quite nice
>>> sometimes.
>>>
>>> That is the beauty of Mistral I think. Unlike Heat it actually
>>> encourages you to customize it with custom Python actions. Anything
>>> we
>>> want in tripleo-common can become our own Mistral action (these get
>>> registered with stevedore entry points so we'd own the code) and
>>> the
>>> YAML workflows just tie them together via tasks.
>>>
>>> We don't have to go off and build our own proxy deployment workflow
>>> API. The structure to do just about anything we need already exists
>>> so
>>> why not go and use it?
>>>

 2) The TripleO API is not a workflow API.  I also largely missed
 this
 discussion, but the TripleO API is a _Deployment_ API.  In some
 cases
 there also happens to be a workflow going on behind the scenes,
 but
 honestly that's not something I want our users to have to care
 about.
>>>
>>> Agree that users don't have to care about this.
>>>
>>> Users can get as involved as they want here. Most users I think
>>> will
>>> use python-tripleoclient to drive the deployment or the new UI.
>>> They
>>> don't have to interact with Mistral directly unless they really
>>> want
>>> to. So whether we choose to build our own API or use a generic one
>>> I
>>> think this point is mute.
>>
>> Okay, I think this is a very fundamental point, and I believe it gets
>> right to the heart of my objection to the proposed change.
>>
>> When I hear you say that users will use tripleoclient to talk to
>> Mistral, it raises a big flag.  Then I look at something like
>> https://github.com/dprince/python-tripleoclient/commit/77ffd2fa7b1642
>> b9f05713ca30b8a27ec4b322b7
>> and the flag gets bigger.
>>
>> The thing is that there's a whole bunch of business logic currently
>> sitting in the client that shouldn't/can't be there.  There are
>> historical reasons for it, but the important thing is that the
>> current
>> client architecture is terribly flawed.  Business logic should never
>> live in the client like it does today.
> 
> Totally agree here. In fact I have removed business logic from python-
> tripleoclient in this patch and moved it into a Mistral action. Which
> can then be used via a stable API from anywhere.
> 
>>
>> Looking at that change, I see a bunch of business logic around taking
>> our configuration and passing it to Mistral.  In order for us to do
>> something like that and have a sustainable GUI, that code _has_ to
>> live
>> behind an API that the GUI and CLI alike can call.  If we ask the GUI
>> to
>> re-implement that code, then we're doomed to divergence between the
>> CLI
>> and GUI code and we'll most likely end up back where we are with a
>> GUI
>> that can't deploy half of our features because they were implemented
>> solely with the CLI in mind and made assumptions the GUI can't meet.
> 
> The latest feedback I've gotten from working with the UI developers on
> this was that we should have a workflow to create the environment. That
> would get called via the Mistral API via python-tripleoclient and any
> sort of UI you could imagine and would essentially give us a stable
> environment interface.

Anything that requires tripleoclient means !GUI though.  I know the
current GUI still has a bunch of dependencies on the CLI, but that seems
like something we need to fix, not a pattern to repeat.  I still think
any sentence containing "call Mistral via tripleoclient" is indicative
of a problem in the design.

> 
> This would also allow us to version the types of Mistral environments
> we create for use with workflows that support the 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[James Slagle]

On Tue, Jan 26, 2016 at 4:46 AM, Steven Hardy  wrote:

> On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
> > On 01/25/2016 03:56 PM, Steven Hardy wrote:
> > > On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
> > >> So I haven't weighed in on this yet, in part because I was on vacation
> > >> when it was first proposed and missed a lot of the initial discussion,
> > >> and also because I wanted to take some time to order my thoughts on
> it.
> > >>  Also because my initial reaction...was not conducive to calm and
> > >> rational discussion. ;-)
> > >>
> > >> The tldr is that I don't like it.  To explain why, I'm going to make a
> > >> list (everyone loves lists, right? Top $NUMBER reasons we should stop
> > >> expecting other people to write our API for us):
> > >>
> > >> 1) We've been down this road before.  Except last time it was with
> Heat.
> > >>  I'm being somewhat tongue-in-cheek here, but expecting a general
> > >> service to provide us a user-friendly API for our specific use case
> just
> > >> doesn't make sense to me.
> > >
> > > Actually, we've been down this road before with Tuskar, and discovered
> that
> > > designing and maintaining a bespoke API for TripleO is really hard.
> >
> > My takeaway from Tuskar was that designing an API that none of the
> > developers on the project use is doomed to fail.  Tuskar also suffered
> > from a lack of some features in Heat that the new API is explicitly
> > depending on in an attempt to avoid many of the problems Tuskar had.
> >
> > Problem #1 is still developer apathy IMHO though.
>
> I think the main issue is developer capacity - we're a small community and
> I for one am worried about the effort involved with building and
> maintaining a bespoke API - thus this whole discussion is essentially about
> finding a quicker and easier way to meet the needs of those needing an API.
>
> In terms of apathy, I think as a developer I don't need an abstraction
> between me, my templates and heat.  Some advanced operators will feel
> likewise, others won't.  What I would find useful sometimes is a general
> purpose workflow engine, which is where I think the more pluggable mistral
> based solution may have advantages in terms of developer and advanced
> operator uptake.
>
> > > I somewhat agree that heat as an API is insufficient, but that doesn't
> > > necessarily imply you have to have a TripleO specific abstraction, just
> > > that *an* abstraction is required.
> > >
> > >> 2) The TripleO API is not a workflow API.  I also largely missed this
> > >> discussion, but the TripleO API is a _Deployment_ API.  In some cases
> > >> there also happens to be a workflow going on behind the scenes, but
> > >> honestly that's not something I want our users to have to care about.
> > >
> > > How would you differentiate between "deployment" in a generic sense in
> > > contrast to a generic workflow?
> > >
> > > Every deployment I can think of involves a series of steps, involving
> some
> > > choices and interactions with other services.  That *is* a workflow?
> >
> > Well, I mean if we want to take this to extremes then pretty much every
> > API is a workflow API.  You make a REST call, a "workflow" happens in
> > the service, and you get back a result.
> >
> > Let me turn this around: Would you implement Heat's API on Mistral?  All
> > that happens when I call Heat is that a series of OpenStack calls are
> > made from heat-engine, after all.  Or is that a gross oversimplification
> > of what's happening?  I could argue that the same is true of this
> > discussion. :-)
>
> As Hugh has mentioned the main thing Heat does is actually manage
> dependencies.  It processes the templates, builds a graph, then walks the
> graph running a "workflow" to create/update/delete/etc each resource.
>
> I could imagine a future where we interface to some external workflow tool
> to
> e.g do each resource action (e.g create a nova server, poll until it's
> active),
> however that's actually a pretty high overhead approach, and it'd probably
> be better to move towards better use of notifications instead (e.g less
> internal workflow)
>
> > >> 3) It ties us 100% to a given implementation.  If Mistral proves to
> be a
> > >> poor choice for some reason, or insufficient for a particular use
> case,
> > >> we have no alternative.  If we have an API and decide to change our
> > >> implementation, nobody has to know or care.  This is kind of the whole
> > >> point of having an API - it shields users from all the nasty
> > >> implementation details under the surface.
> > >
> > > This is a valid argument, but building (and maintining, forever) a
> bespoke
> > > API is a high cost to pay for this additional degree of abstraction,
> and
> > > when you think of the target audience, I'm not certain it's entirely
> > > justified (or, honestly, if our community can bear that overhead);
> > >
> > > For example, take other single-purpose "deployment" projects, such 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Clint Byrum]

Excerpts from Jiri Tomasek's message of 2016-01-25 06:09:33 -0800:
> On 01/25/2016 12:42 AM, Clint Byrum wrote:
> > Excerpts from Dan Prince's message of 2016-01-22 16:19:07 -0800:
> >> On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> >>> So I haven't weighed in on this yet, in part because I was on
> >>> vacation
> >>> when it was first proposed and missed a lot of the initial
> >>> discussion,
> >>> and also because I wanted to take some time to order my thoughts on
> >>> it.
> >>>   Also because my initial reaction...was not conducive to calm and
> >>> rational discussion. ;-)
> >>>
> >>> The tldr is that I don't like it.  To explain why, I'm going to make
> >>> a
> >>> list (everyone loves lists, right? Top $NUMBER reasons we should stop
> >>> expecting other people to write our API for us):
> >>>
> >>> 1) We've been down this road before.  Except last time it was with
> >>> Heat.
> >>>   I'm being somewhat tongue-in-cheek here, but expecting a general
> >>> service to provide us a user-friendly API for our specific use case
> >>> just
> >>> doesn't make sense to me.
> >> We've been down this road with Heat yes. But we are currently using
> >> Heat for some things that we arguable should be (a workflows tool might
> >> help offload some stuff out of Heat). Also we haven't implemented
> >> custom Heat resources for TripleO either. There are mixed opinions on
> >> this but plugging in your code to a generic API is quite nice
> >> sometimes.
> >>
> >> That is the beauty of Mistral I think. Unlike Heat it actually
> >> encourages you to customize it with custom Python actions. Anything we
> >> want in tripleo-common can become our own Mistral action (these get
> >> registered with stevedore entry points so we'd own the code) and the
> >> YAML workflows just tie them together via tasks.
> >>
> > Since TripleO's undercloud is a special, single-purpose cloud just for
> > deploying OpenStack, I think that is a perfectly fine idea. But it also
> > somewhat invalidates the reasons to use Mistral. If you can control all
> > the pieces, IMO you may want to use the best tool for that job, not the
> > one that is the most OpenStack. The virtuous circle only works if you're
> > actually improving the OpenStack components for their general use cases.
> 
> Maybe it is just me but I'd like to point out that we'd not be 
> changing/updating Mistral API to serve our purpose. We'd just write our 
> own custom Mistral Actions and use Mistral API to organize them in 
> workflows and run them.
> 

If you're doing it with a stevedore entry point, you're making a custom
piece of python which any other Mistral service of the same version would
not have.

> Mistral seems to be best _existent_ and _usable_ tool for the job which 
> is IMO very important point.
> 

Agreed, that's a very big +1. However, as in poker, before pushing in
all the chips, be sure the player next to you is actually beat, and not
just playing it slow.

> >
> >> We don't have to go off and build our own proxy deployment workflow
> >> API. The structure to do just about anything we need already exists so
> >> why not go and use it?
> >>
> >>> 2) The TripleO API is not a workflow API.  I also largely missed this
> >>> discussion, but the TripleO API is a _Deployment_ API.  In some cases
> >>> there also happens to be a workflow going on behind the scenes, but
> >>> honestly that's not something I want our users to have to care about.
> >> Agree that users don't have to care about this.
> >>
> >> Users can get as involved as they want here. Most users I think will
> >> use python-tripleoclient to drive the deployment or the new UI. They
> >> don't have to interact with Mistral directly unless they really want
> >> to. So whether we choose to build our own API or use a generic one I
> >> think this point is mute.
> >>
> >>> 3) It ties us 100% to a given implementation.  If Mistral proves to
> >>> be a
> >>> poor choice for some reason, or insufficient for a particular use
> >>> case,
> >>> we have no alternative.  If we have an API and decide to change our
> >>> implementation, nobody has to know or care.  This is kind of the
> >>> whole
> >>> point of having an API - it shields users from all the nasty
> >>> implementation details under the surface.
> >>
> >> Mistal's API is a generic workflow API. It is very much the same layer
> >> that I think we would get if we were to integrate with something like
> >> Ansible Tower... except that Mistral is part of OpenStack. It
> >> integrates very nicely with OpenStack services and is very customizable
> >> with custom actions. The fact that Mistral sits much closer to
> >> OpenStack and is essentially a light shim on top of it is to our
> >> advantage (being TripleO). To think that we can build up a proxy API in
> >> such a manner that we might be able to swap in an entirely new backend
> >> (without even having a fully implement backend yet to begin with) is
> >> for me a bit of a stretch. We've got a lot of 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[James Slagle]

On Mon, Jan 25, 2016 at 6:56 AM, Dougal Matthews  wrote:

> Great post Ben, I largely agree with what you are saying, a lot of your
> points
> are valid concerns that I share. Just a couple of comments inline as I try
> to
> avoid what others have said.
>
> On 22 January 2016 at 17:24, Ben Nemec  wrote:
>
>> So I haven't weighed in on this yet, in part because I was on vacation
>> when it was first proposed and missed a lot of the initial discussion,
>> and also because I wanted to take some time to order my thoughts on it.
>>  Also because my initial reaction...was not conducive to calm and
>> rational discussion. ;-)
>>
>> The tldr is that I don't like it.  To explain why, I'm going to make a
>> list (everyone loves lists, right? Top $NUMBER reasons we should stop
>> expecting other people to write our API for us):
>>
>> 1) We've been down this road before.  Except last time it was with Heat.
>>  I'm being somewhat tongue-in-cheek here, but expecting a general
>> service to provide us a user-friendly API for our specific use case just
>> doesn't make sense to me.
>>
>
> I don't really expect the API to be used directly by many users - how many
> users directly used the Tuskar API? I suspect they all used it via the CLI
> tools or the GUI. I could be completely wrong with the assumption but I
> think
> the primary entry points will be the CLI and GUI we expose. Only advanced
> users would need to consider Mistral.
>

If we don't expect the API to be used directly by many users, then I
question why we're discussing building one. Is it even a useful
abstraction? Or, are we just solving for the problem that the gui needs to
talk http to something?

Tuskar was the anti-pattern, not something to try and do the same again. It
wasn't just the API that wasn't heavily used, it was the CLI as well. Most
TripleO developers didn't use it in their own development, most users
didn't use it, and it wasn't used in tripleo-ci. Eventually, TripleO moved
on, and Tuskar was left behind and stopped being useful.

We ought to focus on getting the API correct and user friendly, then the
CLI will follow as almost a boilerplate addition.

If Mistral doesn't offer the general usability we'd like to see out of a
TripleO deployment API, then it isn't really a choice imo. I'm not saying
it doesn't ftr.

I'm just trying to understand the assumptions of the debate about the
choice. Because to me, it seems like there are some fundamental differences
of opinion as to why we even need an API, or how it would be used. And
those differences are driving a lot of the discussion about TripleO API or
Mistral. It feels a little bit like it's less of choosing the right tool,
and more about how do we expect either of those tools to actually be used.
If we could agree on that, then maybe it makes the technical choice much
easier to make?

I just don't want to see TripleO offer an API we don't really anticipate
anyone using other than through a CLI/GUI. I'd like to see us offer a
stable interface, and have that be the API (TripleO or Mistral), and it be
something that people want/understand how to use and it be well supported
that they use it directly (without editing yaml, see my other reply about
that).



>
> Anyone directly interfacing with the API's will need to learn a ton of
> services,
> I don't think Mistral adds much more complexity here.
>
>
>
>> 2) The TripleO API is not a workflow API.  I also largely missed this
>> discussion, but the TripleO API is a _Deployment_ API.  In some cases
>> there also happens to be a workflow going on behind the scenes, but
>> honestly that's not something I want our users to have to care about.
>>
>
> Yeah, I agree that one of the downsides of the Mistral approach is that we
> blur the lines between these two.
>
>
>>
>> 3) It ties us 100% to a given implementation.  If Mistral proves to be a
>> poor choice for some reason, or insufficient for a particular use case,
>> we have no alternative.  If we have an API and decide to change our
>> implementation, nobody has to know or care.  This is kind of the whole
>> point of having an API - it shields users from all the nasty
>> implementation details under the surface.
>>
>> 4) It raises the bar even further for both new deployers and developers.
>>  You already need to have a pretty firm grasp of Puppet and Heat
>> templates to understand how our stuff works, not to mention a decent
>> understanding of quite a number of OpenStack services.
>>
>> This presents a big chicken and egg problem for people new to OpenStack.
>>  It's great that we're based on OpenStack and that allows people to peek
>> under the hood and do some tinkering, but it can't be required for
>> everyone.  A lot of our deployers are going to have little to no
>> OpenStack experience, and TripleO is already a daunting task for those
>> people (hell, it's daunting for people who _are_ experienced).
>>
>> 5) What does reimplementing all of our tested, 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[James Slagle]

On Mon, Jan 25, 2016 at 9:09 AM, Jiri Tomasek  wrote:

> On 01/25/2016 12:42 AM, Clint Byrum wrote:
>
>> Excerpts from Dan Prince's message of 2016-01-22 16:19:07 -0800:
>>
>
>> I actually think rather than shielding users we should be more
>>> transparent about the actual workflows that are driving deployment.
>>> Smaller more focused workflows that we string together to drive the
>>> deployment.
>>>
>>> I tend to agree, especially when your workflows may need to be
>> customized.
>>
>
> Customizing underlining deployment workflows is IMO potentially very
> dangerous regarding updates etc. and should be avoided as much as possible.


​+1 to Jiri.

I was in the process of replying about the same point :).

It should not be the intent that TripleO *users* or *operators* are
encouraged or supported to edit the implementation, whether it be yaml
workflows or python code. If it is, then we've failed at building a stable
interface.

If we go with a workflow tool that is backed by yaml defined workflows,
then those workflows are implementations of an abstraction. In the same way
that python code is an implementation of a Restful API if we did a new
TripleO API. As a user, no one would have the expectation that editing that
python code is encouraged or supported. It ought to be the same with the
yaml workflows if we go that route -- they aren't meant to be modified by
users/operators.

We have direct experience with this already...tripleo-heat-templates. We
have kind of tried to define a "stable" interface around a pile of yaml
files already, and more or less failed. We know that we can't provide
reliable updates/upgrades if we don't know how the templates have been
modified. There could be many reasons for this: we weren't disciplined
enough in code reviews to keep from breaking ourselves, we lack the
knowledge to recognize a breaking change in the templates, or just
generally that it's really not well known/accepted how to even build a
stable interface around yaml files. Whereas, it's quite universally well
known and understood how to build stable REST API's.

So whether the API is a new TripleO Rest API, or the Mistral Rest API, I
hope we can agree that the API is the stable part, and the implementation
is not meant to be edited by users.

If we can't agree on that, then we aren't even really discussing the same
point IMO. As it's not really a question of which is the better tool,
TripleO API / Mistral / Ansible, it's actually a question of "how is
TripleO meant to be used".


-- 
-- James Slagle
--
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[James Slagle]

On Mon, Jan 25, 2016 at 1:09 PM, Dan Prince  wrote:

>
> As for tripleo-heat-templates... sure it hasn't been an entirely smooth
> ride. I think a lot of the pain has to do with a lack of CI coverage on
> proper upgrade paths upstream. I think this has less to do with the
> architecture (certainly not the fact that it uses YAML) and more to do
> with the fact that we may have cut corners, jammed a bunch of features
> in really fast, and don't have CI to cover proper upgrades jobs.
>

​I think it has a lot less to do with that then it may appear on the
surface. Even if we had 100% functional test coverage of upgrade paths,
that does nothing to solve for scenarios where the templates have been so
heavily customized that stack updates can not be guaranteed to work, or
worse, destroy the cloud.

The most we could do in those scenarios is better detection around what
stack changes are going to be done by an update before anything is actually
applied. And if something is going to be destructive or is not expected, we
refuse to proceed. That would definitely help us as developers to know if
our own changes are breaking upgrades.

But, that doesn't really help the user get their cloud upgraded if they've
customized templates themselves, other than they know that they must undo
whatever custom changes they made, and if they made those for valid reasons
and can't undo, then they are stuck.

They've taken what may appear as a "feature" of the system -- customizing
templates to add support for new things -- and turned it into an
anti-feature, now they can't upgrade without writing upgrade support for
whatever they added themself. Perhaps having to undo new features we as
TripleO developers added for users in the first place.

I don't think that's a supportable path for users of TripleO, nor
sustainable as a community, so editing templates should be advertised as
unsupported.
​


>
> The fact that projects like Heat, Mistral, and Ansible use YAML is to
> our advantage as developers. I think it allows us to work faster by
> avoiding rather large amounts of code to represent the same sorts of
> things. Please, lets not argue that by writing things in Python we
> deter end users from editing things we don't want them too. Instead let
> the choice be ours... not as Python developers... but as TripleO
> deployment developers who choose the write tools for the right jobs.
>


Agreed, we can't deter anyone from editing python anymore than we could
yaml.

But it's not about dictating what people do. It's about what we support as
a project for users and operators. This is something we can define very
clearly so that users aren't surprised. At least they would have some way
of knowing if they are using the system as intended.

I don't think we should support/encourage editing the yaml workflows by end
users if ​we choose a yaml based workflow tool, any more than we should
support editing python code by users if we wrote our own REST API.

It's not clear to me in this discussion when the argument about yaml files
being easier to edit so Mistral/Ansible is a better choice...if we're
talking about developers editing yaml files, or users editing yaml files.

If it's the former (developers), I agree it's a benefit for us.

If it's the latter (users), then personally I don't see that as a valid
argument at all since I view it as something we should never do.



> >
> > So whether the API is a new TripleO Rest API, or the Mistral Rest
> > API, I hope we can agree that the API is the stable part, and the
> > implementation is not meant to be edited by users.
>
> We do totally agree on the stable API part. :)
>


​To be clear, are you saying we should have a stable API, and then support
users in editing the implementation of that API (yaml workflows)?​

For example, we have a stable API such as:

POST /workbooks/deploy-cloud/executions

that might do totally different things across deployed TripleO clouds of
the same released version, depending on how the user has customized the
deploy-cloud workbook?

-- 
-- James Slagle
--
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Mon, 2016-01-25 at 06:55 -0800, Clint Byrum wrote:
> Excerpts from Jiri Tomasek's message of 2016-01-25 06:09:33 -0800:
> > On 01/25/2016 12:42 AM, Clint Byrum wrote:
> > > Excerpts from Dan Prince's message of 2016-01-22 16:19:07 -0800:
> > > > On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> > > > > So I haven't weighed in on this yet, in part because I was on
> > > > > vacation
> > > > > when it was first proposed and missed a lot of the initial
> > > > > discussion,
> > > > > and also because I wanted to take some time to order my
> > > > > thoughts on
> > > > > it.
> > > > >   Also because my initial reaction...was not conducive to
> > > > > calm and
> > > > > rational discussion. ;-)
> > > > > 
> > > > > The tldr is that I don't like it.  To explain why, I'm going
> > > > > to make
> > > > > a
> > > > > list (everyone loves lists, right? Top $NUMBER reasons we
> > > > > should stop
> > > > > expecting other people to write our API for us):
> > > > > 
> > > > > 1) We've been down this road before.  Except last time it was
> > > > > with
> > > > > Heat.
> > > > >   I'm being somewhat tongue-in-cheek here, but expecting a
> > > > > general
> > > > > service to provide us a user-friendly API for our specific
> > > > > use case
> > > > > just
> > > > > doesn't make sense to me.
> > > > We've been down this road with Heat yes. But we are currently
> > > > using
> > > > Heat for some things that we arguable should be (a workflows
> > > > tool might
> > > > help offload some stuff out of Heat). Also we haven't
> > > > implemented
> > > > custom Heat resources for TripleO either. There are mixed
> > > > opinions on
> > > > this but plugging in your code to a generic API is quite nice
> > > > sometimes.
> > > > 
> > > > That is the beauty of Mistral I think. Unlike Heat it actually
> > > > encourages you to customize it with custom Python actions.
> > > > Anything we
> > > > want in tripleo-common can become our own Mistral action (these
> > > > get
> > > > registered with stevedore entry points so we'd own the code)
> > > > and the
> > > > YAML workflows just tie them together via tasks.
> > > > 
> > > Since TripleO's undercloud is a special, single-purpose cloud
> > > just for
> > > deploying OpenStack, I think that is a perfectly fine idea. But
> > > it also
> > > somewhat invalidates the reasons to use Mistral. If you can
> > > control all
> > > the pieces, IMO you may want to use the best tool for that job,
> > > not the
> > > one that is the most OpenStack. The virtuous circle only works if
> > > you're
> > > actually improving the OpenStack components for their general use
> > > cases.
> > 
> > Maybe it is just me but I'd like to point out that we'd not be 
> > changing/updating Mistral API to serve our purpose. We'd just write
> > our 
> > own custom Mistral Actions and use Mistral API to organize them in 
> > workflows and run them.
> > 
> 
> If you're doing it with a stevedore entry point, you're making a
> custom
> piece of python which any other Mistral service of the same version
> would
> not have.
> 
> > Mistral seems to be best _existent_ and _usable_ tool for the job
> > which 
> > is IMO very important point.
> > 
> 
> Agreed, that's a very big +1. However, as in poker, before pushing in
> all the chips, be sure the player next to you is actually beat, and
> not
> just playing it slow.
> 
> > > 
> > > > We don't have to go off and build our own proxy deployment
> > > > workflow
> > > > API. The structure to do just about anything we need already
> > > > exists so
> > > > why not go and use it?
> > > > 
> > > > > 2) The TripleO API is not a workflow API.  I also largely
> > > > > missed this
> > > > > discussion, but the TripleO API is a _Deployment_ API.  In
> > > > > some cases
> > > > > there also happens to be a workflow going on behind the
> > > > > scenes, but
> > > > > honestly that's not something I want our users to have to
> > > > > care about.
> > > > Agree that users don't have to care about this.
> > > > 
> > > > Users can get as involved as they want here. Most users I think
> > > > will
> > > > use python-tripleoclient to drive the deployment or the new UI.
> > > > They
> > > > don't have to interact with Mistral directly unless they really
> > > > want
> > > > to. So whether we choose to build our own API or use a generic
> > > > one I
> > > > think this point is mute.
> > > > 
> > > > > 3) It ties us 100% to a given implementation.  If Mistral
> > > > > proves to
> > > > > be a
> > > > > poor choice for some reason, or insufficient for a particular
> > > > > use
> > > > > case,
> > > > > we have no alternative.  If we have an API and decide to
> > > > > change our
> > > > > implementation, nobody has to know or care.  This is kind of
> > > > > the
> > > > > whole
> > > > > point of having an API - it shields users from all the nasty
> > > > > implementation details under the surface.
> > > > 
> > > > Mistal's API is a generic workflow API. It is very much the
> > > > 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Mon, 2016-01-25 at 09:32 -0500, James Slagle wrote:
> 
> 
> On Mon, Jan 25, 2016 at 9:09 AM, Jiri Tomasek 
> wrote:
> > On 01/25/2016 12:42 AM, Clint Byrum wrote:
> > >  Excerpts from Dan Prince's message of 2016-01-22 16:19:07 -0800:
> > > 
> > >  
> > > >  I actually think rather than shielding users we should be more
> > > > transparent about the actual workflows that are driving
> > > > deployment.
> > > > Smaller more focused workflows that we string together to drive
> > > > the
> > > > deployment.
> > > > 
> > > > 
> > > I tend to agree, especially when your workflows may need to be
> > > customized.
> > > 
> > Customizing underlining deployment workflows is IMO potentially
> > very dangerous regarding updates etc. and should be avoided as much
> > as possible.
> +1 to Jiri.
> 
> I was in the process of replying about the same point :).
> 
> It should not be the intent that TripleO *users* or *operators* are
> encouraged or supported to edit the implementation, whether it be
> yaml workflows or python code. If it is, then we've failed at
> building a stable interface.
> 
> If we go with a workflow tool that is backed by yaml defined
> workflows, then those workflows are implementations of an
> abstraction. In the same way that python code is an implementation of
> a Restful API if we did a new TripleO API. As a user, no one would
> have the expectation that editing that python code is encouraged or
> supported. It ought to be the same with the yaml workflows if we go
> that route -- they aren't meant to be modified by users/operators.
> 
> We have direct experience with this already...tripleo-heat-templates. 
> We have kind of tried to define a "stable" interface around a pile of
> yaml files already, and more or less failed. We know that we can't
> provide reliable updates/upgrades if we don't know how the templates
> have been modified. There could be many reasons for this: we weren't
> disciplined enough in code reviews to keep from breaking ourselves,
> we lack the knowledge to recognize a breaking change in the
> templates, or just generally that it's really not well known/accepted
> how to even build a stable interface around yaml files.
>  Whereas, it's quite universally well known and understood how to
> build stable REST API's.

Here is an example of a versioned YAML workflow in Mistral:

https://github.com/dprince/tripleo-common/blob/mistral/workflows/overcl
oud_deploy.yaml#L3

Basically what I'm suggesting is that we name our workflows like this:

tripleo.overcloud_deploy.v1.deploy
tripleo.overcloud_deploy.v1.get_parameters
etc

I would actually call this a stable API that is implemented within a
generic workflow tool (which also has its own stable API).



As for tripleo-heat-templates... sure it hasn't been an entirely smooth
ride. I think a lot of the pain has to do with a lack of CI coverage on
proper upgrade paths upstream. I think this has less to do with the
architecture (certainly not the fact that it uses YAML) and more to do
with the fact that we may have cut corners, jammed a bunch of features
in really fast, and don't have CI to cover proper upgrades jobs.

The fact that projects like Heat, Mistral, and Ansible use YAML is to
our advantage as developers. I think it allows us to work faster by
avoiding rather large amounts of code to represent the same sorts of
things. Please, lets not argue that by writing things in Python we
deter end users from editing things we don't want them too. Instead let
the choice be ours... not as Python developers... but as TripleO
deployment developers who choose the write tools for the right jobs.

> 
> So whether the API is a new TripleO Rest API, or the Mistral Rest
> API, I hope we can agree that the API is the stable part, and the
> implementation is not meant to be edited by users.

We do totally agree on the stable API part. :)

> 
> If we can't agree on that, then we aren't even really discussing the
> same point IMO. As it's not really a question of which is the better
> tool, TripleO API / Mistral / Ansible, it's actually a question of
> "how is TripleO meant to be used".
> 
> 
> _
> _
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Sun, 2016-01-24 at 15:42 -0800, Clint Byrum wrote:
> Excerpts from Dan Prince's message of 2016-01-22 16:19:07 -0800:
> > On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> > > So I haven't weighed in on this yet, in part because I was on
> > > vacation
> > > when it was first proposed and missed a lot of the initial
> > > discussion,
> > > and also because I wanted to take some time to order my thoughts
> > > on
> > > it.
> > >  Also because my initial reaction...was not conducive to calm and
> > > rational discussion. ;-)
> > > 
> > > The tldr is that I don't like it.  To explain why, I'm going to
> > > make
> > > a
> > > list (everyone loves lists, right? Top $NUMBER reasons we should
> > > stop
> > > expecting other people to write our API for us):
> > > 
> > > 1) We've been down this road before.  Except last time it was
> > > with
> > > Heat.
> > >  I'm being somewhat tongue-in-cheek here, but expecting a general
> > > service to provide us a user-friendly API for our specific use
> > > case
> > > just
> > > doesn't make sense to me.
> > 
> > We've been down this road with Heat yes. But we are currently using
> > Heat for some things that we arguable should be (a workflows tool
> > might
> > help offload some stuff out of Heat). Also we haven't implemented
> > custom Heat resources for TripleO either. There are mixed opinions
> > on
> > this but plugging in your code to a generic API is quite nice
> > sometimes.
> > 
> > That is the beauty of Mistral I think. Unlike Heat it actually
> > encourages you to customize it with custom Python actions. Anything
> > we
> > want in tripleo-common can become our own Mistral action (these get
> > registered with stevedore entry points so we'd own the code) and
> > the
> > YAML workflows just tie them together via tasks.
> > 
> 
> Since TripleO's undercloud is a special, single-purpose cloud just
> for
> deploying OpenStack, I think that is a perfectly fine idea. But it
> also
> somewhat invalidates the reasons to use Mistral. If you can control
> all
> the pieces, IMO you may want to use the best tool for that job, not
> the
> one that is the most OpenStack. The virtuous circle only works if
> you're
> actually improving the OpenStack components for their general use
> cases.

Very true. And I think the goal would be that in general we try to
improve OpenStack components first. We don't want custom actions for
everything. Ideally core OpenStack services would support what we need
instead of us having to roll our own actions for things. However that
is sometimes easier said than done so we need to have a "release valve"
to move forwards.

Again, this is why I really like Mistral. It integrates directly with a
slew of OpenStack services by simply providing a lightweight glue
wrapper on top of the various python-* clients:

http://git.openstack.org/cgit/openstack/mistral/tree/mistral/actions/op
enstack/mapping.json

So yeah, instead of creating our own massive pile of custom TripleO
actions lets first ask the question does this code actually belong in
the upstream python client library? And would putting it there be a
generally useful addition to everyone?

Using a combination of both these approaches however I think we could
quickly move forwards and build the deployment workflows we need in the
realm of generic workflow tools.

Dan


> 
> > We don't have to go off and build our own proxy deployment workflow
> > API. The structure to do just about anything we need already exists
> > so
> > why not go and use it?
> > 
> > > 
> > > 2) The TripleO API is not a workflow API.  I also largely missed
> > > this
> > > discussion, but the TripleO API is a _Deployment_ API.  In some
> > > cases
> > > there also happens to be a workflow going on behind the scenes,
> > > but
> > > honestly that's not something I want our users to have to care
> > > about.
> > 
> > Agree that users don't have to care about this.
> > 
> > Users can get as involved as they want here. Most users I think
> > will
> > use python-tripleoclient to drive the deployment or the new UI.
> > They
> > don't have to interact with Mistral directly unless they really
> > want
> > to. So whether we choose to build our own API or use a generic one
> > I
> > think this point is mute.
> > 
> > > 
> > > 3) It ties us 100% to a given implementation.  If Mistral proves
> > > to
> > > be a
> > > poor choice for some reason, or insufficient for a particular use
> > > case,
> > > we have no alternative.  If we have an API and decide to change
> > > our
> > > implementation, nobody has to know or care.  This is kind of the
> > > whole
> > > point of having an API - it shields users from all the nasty
> > > implementation details under the surface.
> > 
> > 
> > Mistal's API is a generic workflow API. It is very much the same
> > layer
> > that I think we would get if we were to integrate with something
> > like
> > Ansible Tower... except that Mistral is part of OpenStack. It
> > integrates very nicely with 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Steven Hardy]

On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
> So I haven't weighed in on this yet, in part because I was on vacation
> when it was first proposed and missed a lot of the initial discussion,
> and also because I wanted to take some time to order my thoughts on it.
>  Also because my initial reaction...was not conducive to calm and
> rational discussion. ;-)
> 
> The tldr is that I don't like it.  To explain why, I'm going to make a
> list (everyone loves lists, right? Top $NUMBER reasons we should stop
> expecting other people to write our API for us):
> 
> 1) We've been down this road before.  Except last time it was with Heat.
>  I'm being somewhat tongue-in-cheek here, but expecting a general
> service to provide us a user-friendly API for our specific use case just
> doesn't make sense to me.

Actually, we've been down this road before with Tuskar, and discovered that
designing and maintaining a bespoke API for TripleO is really hard.

I somewhat agree that heat as an API is insufficient, but that doesn't
necessarily imply you have to have a TripleO specific abstraction, just
that *an* abstraction is required.

> 2) The TripleO API is not a workflow API.  I also largely missed this
> discussion, but the TripleO API is a _Deployment_ API.  In some cases
> there also happens to be a workflow going on behind the scenes, but
> honestly that's not something I want our users to have to care about.

How would you differentiate between "deployment" in a generic sense in
contrast to a generic workflow?

Every deployment I can think of involves a series of steps, involving some
choices and interactions with other services.  That *is* a workflow?

> 3) It ties us 100% to a given implementation.  If Mistral proves to be a
> poor choice for some reason, or insufficient for a particular use case,
> we have no alternative.  If we have an API and decide to change our
> implementation, nobody has to know or care.  This is kind of the whole
> point of having an API - it shields users from all the nasty
> implementation details under the surface.

This is a valid argument, but building (and maintining, forever) a bespoke
API is a high cost to pay for this additional degree of abstraction, and
when you think of the target audience, I'm not certain it's entirely
justified (or, honestly, if our community can bear that overhead);

For example, take other single-purpose "deployment" projects, such as
Sahara, Magnum, perhaps Trove.  These are designed primarily as user-facing
API's, where the services will ultimately be consumed by public and private
cloud customers.

Contrast with TripleO, where our target audience is, for the most part,
sysadmins who deploy and maintain an openstack deployment over a long
period of time.  There are two main groups here:

1. PoC "getting started" folks who need a very simple on-ramp (generalizing
somewhat, the audience for the opinionated deployments driven via UI's)

2. Seasoned sysadmins who want plugability, control and flexibility above
all else (and, simplicity and lack of extraneous abstractions)

A bespoke API potentially has a fairly high value to (1), but a very low or
even negative value to (2).  Which is why this is turning out to be a tough
and polarized discussion, unfortunately.

> 4) It raises the bar even further for both new deployers and developers.
>  You already need to have a pretty firm grasp of Puppet and Heat
> templates to understand how our stuff works, not to mention a decent
> understanding of quite a number of OpenStack services.

I'm not really sure if a bespoke WSGI app vs an existing one (mistral)
really makes much difference at all wrt raising the bar.  I see it
primarily as in implementation detail tbh.

Looking at the prototypes Dan has done, it *is* possible to define a much
more strongly versioned API with Mistral than it is with Heat, which is the
main requirement in terms of lowering the bar (stable, relatively
opinionated by default and easy to use).

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Mon, 2016-01-25 at 15:08 -0500, James Slagle wrote:
> 
> 
> On Mon, Jan 25, 2016 at 1:09 PM, Dan Prince 
> wrote:
> >  
> > As for tripleo-heat-templates... sure it hasn't been an entirely
> > smooth
> > ride. I think a lot of the pain has to do with a lack of CI
> > coverage on
> > proper upgrade paths upstream. I think this has less to do with the
> > architecture (certainly not the fact that it uses YAML) and more to
> > do
> > with the fact that we may have cut corners, jammed a bunch of
> > features
> > in really fast, and don't have CI to cover proper upgrades jobs.
> I think it has a lot less to do with that then it may appear on the
> surface. Even if we had 100% functional test coverage of upgrade
> paths, that does nothing to solve for scenarios where the templates
> have been so heavily customized that stack updates can not be
> guaranteed to work, or worse, destroy the cloud.
> 
> The most we could do in those scenarios is better detection around
> what stack changes are going to be done by an update before anything
> is actually applied. And if something is going to be destructive or
> is not expected, we refuse to proceed. That would definitely help us
> as developers to know if our own changes are breaking upgrades.
> 
> But, that doesn't really help the user get their cloud upgraded if
> they've customized templates themselves, other than they know that
> they must undo whatever custom changes they made, and if they made
> those for valid reasons and can't undo, then they are stuck.
> 
> They've taken what may appear as a "feature" of the system --
> customizing templates to add support for new things -- and turned it
> into an anti-feature, now they can't upgrade without writing upgrade
> support for whatever they added themself. Perhaps having to undo new
> features we as TripleO developers added for users in the first place.
> 
> I don't think that's a supportable path for users of TripleO, nor
> sustainable as a community, so editing templates should be advertised
> as unsupported.
> 
>  
> >  
> > The fact that projects like Heat, Mistral, and Ansible use YAML is
> > to
> > our advantage as developers. I think it allows us to work faster by
> > avoiding rather large amounts of code to represent the same sorts
> > of
> > things. Please, lets not argue that by writing things in Python we
> > deter end users from editing things we don't want them too. Instead
> > let
> > the choice be ours... not as Python developers... but as TripleO
> > deployment developers who choose the write tools for the right
> > jobs.
> > 
> 
> Agreed, we can't deter anyone from editing python anymore than we
> could yaml. 
> 
> But it's not about dictating what people do. It's about what we
> support as a project for users and operators. This is something we
> can define very clearly so that users aren't surprised. At least they
> would have some way of knowing if they are using the system as
> intended.
> 
> I don't think we should support/encourage editing the yaml workflows
> by end users if we choose a yaml based workflow tool, any more than
> we should support editing python code by users if we wrote our own
> REST API.
> 
> It's not clear to me in this discussion when the argument about yaml
> files being easier to edit so Mistral/Ansible is a better choice...if
> we're talking about developers editing yaml files, or users editing
> yaml files.
> 
> If it's the former (developers), I agree it's a benefit for us.

Its the former. YAML is primarily a benefit for us as developers. And
to go back to my original statement my preference would be that we
develop them as a set of smaller more focused workflows that are more
transparent to the end user such that they know what will happen when
they get executed.

> 
> If it's the latter (users), then personally I don't see that as a
> valid argument at all since I view it as something we should never
> do.
> 
>  
> >  >
> > > So whether the API is a new TripleO Rest API, or the Mistral Rest
> > > API, I hope we can agree that the API is the stable part, and the
> > > implementation is not meant to be edited by users.
> > 
> > We do totally agree on the stable API part. :)
> 
> To be clear, are you saying we should have a stable API, and then
> support users in editing the implementation of that API (yaml
> workflows)?

What I was agreeing to was the desire to maintain a set of stable
workflows that we support. Be it in the form of a generic workflow API
(my preference) or an API that we build up from scratch. Regardless of
choice the goal would be to create stable workflows that could be
versioned so as not to pull the rug on previous implementations.

> 
> For example, we have a stable API such as:
> 
> POST /workbooks/deploy-cloud/executions
> 
> that might do totally different things across deployed TripleO clouds
> of the same released version, depending on how the user has
> customized the deploy-cloud workbook?
> 
> -- 
> -- James 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Ben Nemec]

On 01/25/2016 05:56 AM, Dougal Matthews wrote:
> Great post Ben, I largely agree with what you are saying, a lot of your
> points
> are valid concerns that I share. Just a couple of comments inline as I
> try to
> avoid what others have said.
> 
> On 22 January 2016 at 17:24, Ben Nemec  > wrote:
> 
> So I haven't weighed in on this yet, in part because I was on vacation
> when it was first proposed and missed a lot of the initial discussion,
> and also because I wanted to take some time to order my thoughts on it.
>  Also because my initial reaction...was not conducive to calm and
> rational discussion. ;-)
> 
> The tldr is that I don't like it.  To explain why, I'm going to make a
> list (everyone loves lists, right? Top $NUMBER reasons we should stop
> expecting other people to write our API for us):
> 
> 1) We've been down this road before.  Except last time it was with Heat.
>  I'm being somewhat tongue-in-cheek here, but expecting a general
> service to provide us a user-friendly API for our specific use case just
> doesn't make sense to me.
> 
> 
> I don't really expect the API to be used directly by many users - how many
> users directly used the Tuskar API? I suspect they all used it via the CLI
> tools or the GUI. I could be completely wrong with the assumption but I
> think
> the primary entry points will be the CLI and GUI we expose. Only advanced
> users would need to consider Mistral.

I touched on this in my reply to Dan as well, but I don't believe this
is true.  The CLI needs to be a very thin wrapper around whatever REST
API we're calling.  Basically it should be taking params from the
command-line and passing them through to $API.  If it does _any_
processing of them, then that's something the GUI has to duplicate
(assuming it can) and we have an architectural problem.

That being said, it does appear as though I misunderstood the initial
proposal.  I thought we were talking about replacing openstack overcloud
* (and friends) commands with mistral calls, but it appears the proposal
was to simply pull certain pieces of the commands out and replace them
with Mistral (which as I noted in my other reply doesn't actually solve
the problem the TripleO API was trying to solve), so it's possible some
of the things I was arguing against weren't actually being proposed. :-)

> 
> Anyone directly interfacing with the API's will need to learn a ton of
> services,
> I don't think Mistral adds much more complexity here.
> 
>  
> 
> 2) The TripleO API is not a workflow API.  I also largely missed this
> discussion, but the TripleO API is a _Deployment_ API.  In some cases
> there also happens to be a workflow going on behind the scenes, but
> honestly that's not something I want our users to have to care about.
> 
> 
> Yeah, I agree that one of the downsides of the Mistral approach is that we
> blur the lines between these two.
>  
> 
> 
> 3) It ties us 100% to a given implementation.  If Mistral proves to be a
> poor choice for some reason, or insufficient for a particular use case,
> we have no alternative.  If we have an API and decide to change our
> implementation, nobody has to know or care.  This is kind of the whole
> point of having an API - it shields users from all the nasty
> implementation details under the surface.
> 
> 4) It raises the bar even further for both new deployers and developers.
>  You already need to have a pretty firm grasp of Puppet and Heat
> templates to understand how our stuff works, not to mention a decent
> understanding of quite a number of OpenStack services.
> 
> This presents a big chicken and egg problem for people new to OpenStack.
>  It's great that we're based on OpenStack and that allows people to peek
> under the hood and do some tinkering, but it can't be required for
> everyone.  A lot of our deployers are going to have little to no
> OpenStack experience, and TripleO is already a daunting task for those
> people (hell, it's daunting for people who _are_ experienced).
> 
> 5) What does reimplementing all of our tested, well-understood Python
> into a new YAML format gain us?  This is maybe the biggest thing I'm
> missing from this whole discussion.  We lose a bunch of things (ease of
> transition from other Python projects, excellent existing testing
> framework, etc.), but what are we actually gaining other than the
> ability to say that we use N + 1 OpenStack services?  Because we're way
> past the point where "It's OpenStack deploying OpenStack" is sufficient
> reason for people to pay attention to us.  We need less "Ooh, neat" and
> more "Ooh, that's easy to use and works well."  It's still not clear to
> me that Mistral helps in any way with the latter.
> 
> 6) On the testing note, how do we test these workflows?  Do we know 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Ben Nemec]

On 01/22/2016 06:19 PM, Dan Prince wrote:
> On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
>> So I haven't weighed in on this yet, in part because I was on
>> vacation
>> when it was first proposed and missed a lot of the initial
>> discussion,
>> and also because I wanted to take some time to order my thoughts on
>> it.
>>  Also because my initial reaction...was not conducive to calm and
>> rational discussion. ;-)
>>
>> The tldr is that I don't like it.  To explain why, I'm going to make
>> a
>> list (everyone loves lists, right? Top $NUMBER reasons we should stop
>> expecting other people to write our API for us):
>>
>> 1) We've been down this road before.  Except last time it was with
>> Heat.
>>  I'm being somewhat tongue-in-cheek here, but expecting a general
>> service to provide us a user-friendly API for our specific use case
>> just
>> doesn't make sense to me.
> 
> We've been down this road with Heat yes. But we are currently using
> Heat for some things that we arguable should be (a workflows tool might
> help offload some stuff out of Heat). Also we haven't implemented
> custom Heat resources for TripleO either. There are mixed opinions on
> this but plugging in your code to a generic API is quite nice
> sometimes.
> 
> That is the beauty of Mistral I think. Unlike Heat it actually
> encourages you to customize it with custom Python actions. Anything we
> want in tripleo-common can become our own Mistral action (these get
> registered with stevedore entry points so we'd own the code) and the
> YAML workflows just tie them together via tasks.
> 
> We don't have to go off and build our own proxy deployment workflow
> API. The structure to do just about anything we need already exists so
> why not go and use it?
> 
>>
>> 2) The TripleO API is not a workflow API.  I also largely missed this
>> discussion, but the TripleO API is a _Deployment_ API.  In some cases
>> there also happens to be a workflow going on behind the scenes, but
>> honestly that's not something I want our users to have to care about.
> 
> Agree that users don't have to care about this.
> 
> Users can get as involved as they want here. Most users I think will
> use python-tripleoclient to drive the deployment or the new UI. They
> don't have to interact with Mistral directly unless they really want
> to. So whether we choose to build our own API or use a generic one I
> think this point is mute.

Okay, I think this is a very fundamental point, and I believe it gets
right to the heart of my objection to the proposed change.

When I hear you say that users will use tripleoclient to talk to
Mistral, it raises a big flag.  Then I look at something like
https://github.com/dprince/python-tripleoclient/commit/77ffd2fa7b1642b9f05713ca30b8a27ec4b322b7
and the flag gets bigger.

The thing is that there's a whole bunch of business logic currently
sitting in the client that shouldn't/can't be there.  There are
historical reasons for it, but the important thing is that the current
client architecture is terribly flawed.  Business logic should never
live in the client like it does today.

Looking at that change, I see a bunch of business logic around taking
our configuration and passing it to Mistral.  In order for us to do
something like that and have a sustainable GUI, that code _has_ to live
behind an API that the GUI and CLI alike can call.  If we ask the GUI to
re-implement that code, then we're doomed to divergence between the CLI
and GUI code and we'll most likely end up back where we are with a GUI
that can't deploy half of our features because they were implemented
solely with the CLI in mind and made assumptions the GUI can't meet.

As I said, this is a really fundamental part of the argument for
creating a REST API for TripleO.  A huge reason Tuskar UI didn't work
was that it had to reimplement all of the logic in tripleoclient.  Two
parallel implementations in different languages is not a sustainable
model of development, and on top of that developers will always focus on
the CLI, which can do a lot of things the UI can't.  That was the straw
that broke Tuskar UI's back in the end - new features like network
isolation and Ceph were designed for the CLI, and had requirements the
UI simply couldn't meet in a sane fashion.

It's not like we undertook the task of writing an API lightly.  In fact,
I initially argued against it myself, but after talking to the GUI folks
it was explained that just sticking all of our code in a Python library
doesn't actually solve their problems.  They need something they can
talk to (read: a REST API) that can handle the business logic.  This is
the problem the TripleO API was designed to solve, not simply the task
of running some pre-defined OpenStack API calls.  Which is why one of my
first points was "is not a workflow API".

I realize I've now typed enough that everyone probably tuned out a few
paragraphs ago, but I hope somewhere in that wall of text I've explained
what I see as a disconnect 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Steven Hardy]

On Mon, Jan 25, 2016 at 03:08:07PM -0500, James Slagle wrote:
>On Mon, Jan 25, 2016 at 1:09 PM, Dan Prince  wrote:
> 
>  As for tripleo-heat-templates... sure it hasn't been an entirely smooth
>  ride. I think a lot of the pain has to do with a lack of CI coverage on
>  proper upgrade paths upstream. I think this has less to do with the
>  architecture (certainly not the fact that it uses YAML) and more to do
>  with the fact that we may have cut corners, jammed a bunch of features
>  in really fast, and don't have CI to cover proper upgrades jobs.
> 
>â**I think it has a lot less to do with that then it may appear on the
>surface. Even if we had 100% functional test coverage of upgrade paths,
>that does nothing to solve for scenarios where the templates have been so
>heavily customized that stack updates can not be guaranteed to work, or
>worse, destroy the cloud.

>The most we could do in those scenarios is better detection around what
>stack changes are going to be done by an update before anything is
>actually applied. And if something is going to be destructive or is not
>expected, we refuse to proceed. That would definitely help us as
>developers to know if our own changes are breaking upgrades.

FYI I'm working on getting these patches landed, then we can wire exactly
this preview capability in to TripleO:

https://review.openstack.org/#/c/268997/
https://review.openstack.org/#/c/269176/

I think there are some other easy-wins, such as introducing a
userdata_update_policy property to OS::Nova::Server, which would mean you
aren't forced to rebuild an entire ResourceGroup just because user-data for
new instances changes.  Similarly, introduce an "ignore" option for
existing flavor/image update policy properties.

But, yeah, we need to get way better at testing and detecting these sorts
of consequences, and loudly warning the operator before they happen.

>But, that doesn't really help the user get their cloud upgraded if they've
>customized templates themselves, other than they know that they must undo
>whatever custom changes they made, and if they made those for valid
>reasons and can't undo, then they are stuck.
> 
>They've taken what may appear as a "feature" of the system -- customizing
>templates to add support for new things -- and turned it into an
>anti-feature, now they can't upgrade without writing upgrade support for
>whatever they added themself. Perhaps having to undo new features we as
>TripleO developers added for users in the first place.

FWIW I think you're overstating things here, we've got a well defined and
documented [1] set of ExtraConfig interfaces which should be safe to use
over upgrades (we need better CI testing of them tho).

If folks start hacking on and relying on internal implementation details of
any other project, they would expect upgrade pain, and TripleO is no
different - if you start hacking on undocumented internal interfaces,
you'll likely have more work to do come upgrade time.  Try forking a bunch
of puppet modules, same problem.

If anything we need to evolve this into an even more strongly defined
"plugin" type interface, where explicit steps are supported and folks can
interface their stuff (Dan's composable roles prototype[2] already makes
progress towards this btw).

Cheers,

Steve

[1] "Node customization and Third-Party Integration"
http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/extra_config.html

[2] https://review.openstack.org/#/c/236243/11/puppet/roles/README.rst

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Mon, 2016-01-25 at 15:31 -0600, Ben Nemec wrote:
> On 01/22/2016 06:19 PM, Dan Prince wrote:
> > On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> > > So I haven't weighed in on this yet, in part because I was on
> > > vacation
> > > when it was first proposed and missed a lot of the initial
> > > discussion,
> > > and also because I wanted to take some time to order my thoughts
> > > on
> > > it.
> > >  Also because my initial reaction...was not conducive to calm and
> > > rational discussion. ;-)
> > > 
> > > The tldr is that I don't like it.  To explain why, I'm going to
> > > make
> > > a
> > > list (everyone loves lists, right? Top $NUMBER reasons we should
> > > stop
> > > expecting other people to write our API for us):
> > > 
> > > 1) We've been down this road before.  Except last time it was
> > > with
> > > Heat.
> > >  I'm being somewhat tongue-in-cheek here, but expecting a general
> > > service to provide us a user-friendly API for our specific use
> > > case
> > > just
> > > doesn't make sense to me.
> > 
> > We've been down this road with Heat yes. But we are currently using
> > Heat for some things that we arguable should be (a workflows tool
> > might
> > help offload some stuff out of Heat). Also we haven't implemented
> > custom Heat resources for TripleO either. There are mixed opinions
> > on
> > this but plugging in your code to a generic API is quite nice
> > sometimes.
> > 
> > That is the beauty of Mistral I think. Unlike Heat it actually
> > encourages you to customize it with custom Python actions. Anything
> > we
> > want in tripleo-common can become our own Mistral action (these get
> > registered with stevedore entry points so we'd own the code) and
> > the
> > YAML workflows just tie them together via tasks.
> > 
> > We don't have to go off and build our own proxy deployment workflow
> > API. The structure to do just about anything we need already exists
> > so
> > why not go and use it?
> > 
> > > 
> > > 2) The TripleO API is not a workflow API.  I also largely missed
> > > this
> > > discussion, but the TripleO API is a _Deployment_ API.  In some
> > > cases
> > > there also happens to be a workflow going on behind the scenes,
> > > but
> > > honestly that's not something I want our users to have to care
> > > about.
> > 
> > Agree that users don't have to care about this.
> > 
> > Users can get as involved as they want here. Most users I think
> > will
> > use python-tripleoclient to drive the deployment or the new UI.
> > They
> > don't have to interact with Mistral directly unless they really
> > want
> > to. So whether we choose to build our own API or use a generic one
> > I
> > think this point is mute.
> 
> Okay, I think this is a very fundamental point, and I believe it gets
> right to the heart of my objection to the proposed change.
> 
> When I hear you say that users will use tripleoclient to talk to
> Mistral, it raises a big flag.  Then I look at something like
> https://github.com/dprince/python-tripleoclient/commit/77ffd2fa7b1642
> b9f05713ca30b8a27ec4b322b7
> and the flag gets bigger.
> 
> The thing is that there's a whole bunch of business logic currently
> sitting in the client that shouldn't/can't be there.  There are
> historical reasons for it, but the important thing is that the
> current
> client architecture is terribly flawed.  Business logic should never
> live in the client like it does today.

Totally agree here. In fact I have removed business logic from python-
tripleoclient in this patch and moved it into a Mistral action. Which
can then be used via a stable API from anywhere.

> 
> Looking at that change, I see a bunch of business logic around taking
> our configuration and passing it to Mistral.  In order for us to do
> something like that and have a sustainable GUI, that code _has_ to
> live
> behind an API that the GUI and CLI alike can call.  If we ask the GUI
> to
> re-implement that code, then we're doomed to divergence between the
> CLI
> and GUI code and we'll most likely end up back where we are with a
> GUI
> that can't deploy half of our features because they were implemented
> solely with the CLI in mind and made assumptions the GUI can't meet.

The latest feedback I've gotten from working with the UI developers on
this was that we should have a workflow to create the environment. That
would get called via the Mistral API via python-tripleoclient and any
sort of UI you could imagine and would essentially give us a stable
environment interface.

This would also allow us to version the types of Mistral environments
we create for use with workflows that support the various version
(should we choose to take it to this level).

Rather than focus on the environments mechanism I rather meant this
prototype to be a sort of demonstration to show how we could call a
workflow, how the code would cleanly move out of python-tripleoclient
and into tripleo-common where it becomes a Mistral action, etc. I
needed the environment too... 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Ben Nemec]

On 01/25/2016 03:56 PM, Steven Hardy wrote:
> On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
>> So I haven't weighed in on this yet, in part because I was on vacation
>> when it was first proposed and missed a lot of the initial discussion,
>> and also because I wanted to take some time to order my thoughts on it.
>>  Also because my initial reaction...was not conducive to calm and
>> rational discussion. ;-)
>>
>> The tldr is that I don't like it.  To explain why, I'm going to make a
>> list (everyone loves lists, right? Top $NUMBER reasons we should stop
>> expecting other people to write our API for us):
>>
>> 1) We've been down this road before.  Except last time it was with Heat.
>>  I'm being somewhat tongue-in-cheek here, but expecting a general
>> service to provide us a user-friendly API for our specific use case just
>> doesn't make sense to me.
> 
> Actually, we've been down this road before with Tuskar, and discovered that
> designing and maintaining a bespoke API for TripleO is really hard.

My takeaway from Tuskar was that designing an API that none of the
developers on the project use is doomed to fail.  Tuskar also suffered
from a lack of some features in Heat that the new API is explicitly
depending on in an attempt to avoid many of the problems Tuskar had.

Problem #1 is still developer apathy IMHO though.

> 
> I somewhat agree that heat as an API is insufficient, but that doesn't
> necessarily imply you have to have a TripleO specific abstraction, just
> that *an* abstraction is required.
> 
>> 2) The TripleO API is not a workflow API.  I also largely missed this
>> discussion, but the TripleO API is a _Deployment_ API.  In some cases
>> there also happens to be a workflow going on behind the scenes, but
>> honestly that's not something I want our users to have to care about.
> 
> How would you differentiate between "deployment" in a generic sense in
> contrast to a generic workflow?
> 
> Every deployment I can think of involves a series of steps, involving some
> choices and interactions with other services.  That *is* a workflow?

Well, I mean if we want to take this to extremes then pretty much every
API is a workflow API.  You make a REST call, a "workflow" happens in
the service, and you get back a result.

Let me turn this around: Would you implement Heat's API on Mistral?  All
that happens when I call Heat is that a series of OpenStack calls are
made from heat-engine, after all.  Or is that a gross oversimplification
of what's happening?  I could argue that the same is true of this
discussion. :-)

> 
>> 3) It ties us 100% to a given implementation.  If Mistral proves to be a
>> poor choice for some reason, or insufficient for a particular use case,
>> we have no alternative.  If we have an API and decide to change our
>> implementation, nobody has to know or care.  This is kind of the whole
>> point of having an API - it shields users from all the nasty
>> implementation details under the surface.
> 
> This is a valid argument, but building (and maintining, forever) a bespoke
> API is a high cost to pay for this additional degree of abstraction, and
> when you think of the target audience, I'm not certain it's entirely
> justified (or, honestly, if our community can bear that overhead);
> 
> For example, take other single-purpose "deployment" projects, such as
> Sahara, Magnum, perhaps Trove.  These are designed primarily as user-facing
> API's, where the services will ultimately be consumed by public and private
> cloud customers.
> 
> Contrast with TripleO, where our target audience is, for the most part,
> sysadmins who deploy and maintain an openstack deployment over a long
> period of time.  There are two main groups here:
> 
> 1. PoC "getting started" folks who need a very simple on-ramp (generalizing
> somewhat, the audience for the opinionated deployments driven via UI's)
> 
> 2. Seasoned sysadmins who want plugability, control and flexibility above
> all else (and, simplicity and lack of extraneous abstractions)
> 
> A bespoke API potentially has a fairly high value to (1), but a very low or
> even negative value to (2).  Which is why this is turning out to be a tough
> and polarized discussion, unfortunately.

Well, to be honest I'm not sure we can satisfy the second type of user
with what we have today anyway.  Our Heat-driven puppet is hardly
lightweight or simple, and there are extraneous abstractions all over
the place (see also every place that we have a Heat template param that
exists solely to get plugged into a puppet hiera file :-).

To me, this is mostly an artifact of the original intent of the Heat
templates being _the_ abstraction that would then be translated into
os-*-config, puppet, or [insert deployment tool of choice] by the
templates, and I have to admit I'm not sure how to fix it for these users.

So I guess the question is, how does having an API hurt those power
users?  They'll still be able/have to edit Heat templates to deploy

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Hugh Brock]

hI can't speak to any of the concerns below from an engineering
standpoint, so I won't try. I do have the benefit of an earful from
various tripleo users, and also of watching a lot of the history of
tripleo development, so I will comment from that standpoint.

On Tue, Jan 26, 2016 at 12:45 AM, Ben Nemec  wrote:
> On 01/25/2016 03:56 PM, Steven Hardy wrote:
>> On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
>>> So I haven't weighed in on this yet, in part because I was on vacation
>>> when it was first proposed and missed a lot of the initial discussion,
>>> and also because I wanted to take some time to order my thoughts on it.
>>>  Also because my initial reaction...was not conducive to calm and
>>> rational discussion. ;-)
>>>
>>> The tldr is that I don't like it.  To explain why, I'm going to make a
>>> list (everyone loves lists, right? Top $NUMBER reasons we should stop
>>> expecting other people to write our API for us):
>>>
>>> 1) We've been down this road before.  Except last time it was with Heat.
>>>  I'm being somewhat tongue-in-cheek here, but expecting a general
>>> service to provide us a user-friendly API for our specific use case just
>>> doesn't make sense to me.
>>
>> Actually, we've been down this road before with Tuskar, and discovered that
>> designing and maintaining a bespoke API for TripleO is really hard.
>
> My takeaway from Tuskar was that designing an API that none of the
> developers on the project use is doomed to fail.  Tuskar also suffered
> from a lack of some features in Heat that the new API is explicitly
> depending on in an attempt to avoid many of the problems Tuskar had.
>
> Problem #1 is still developer apathy IMHO though.

Tuskar API v2's primary function was to eliminate the need for
merge.py by providing a sane method to combine bits of heat templates
into a coherent whole.

Later, Heat (sensibly, I think) gained the ability to combine its own
templates and manage parameter inheritance. Once that happened Tuskar
was really just getting in the way, even more so when we started
needing relatively large environment files. It was a good engineering
effort that was rendered obsolete by developments around it. We should
not regret it; nor should we mourn it.

>>
>> I somewhat agree that heat as an API is insufficient, but that doesn't
>> necessarily imply you have to have a TripleO specific abstraction, just
>> that *an* abstraction is required.
>>
>>> 2) The TripleO API is not a workflow API.  I also largely missed this
>>> discussion, but the TripleO API is a _Deployment_ API.  In some cases
>>> there also happens to be a workflow going on behind the scenes, but
>>> honestly that's not something I want our users to have to care about.
>>
>> How would you differentiate between "deployment" in a generic sense in
>> contrast to a generic workflow?
>>
>> Every deployment I can think of involves a series of steps, involving some
>> choices and interactions with other services.  That *is* a workflow?
>
> Well, I mean if we want to take this to extremes then pretty much every
> API is a workflow API.  You make a REST call, a "workflow" happens in
> the service, and you get back a result.
>
> Let me turn this around: Would you implement Heat's API on Mistral?  All
> that happens when I call Heat is that a series of OpenStack calls are
> made from heat-engine, after all.  Or is that a gross oversimplification
> of what's happening?  I could argue that the same is true of this
> discussion. :-)

It is fairly clear to me from sitting next to the guy who has had to
implement a big chunk of the update and upgrade code, that Heat is not
good at workflow. This is well understood by everyone and should not
surprise us. It does not make any sense to talk about Heat "making a
series of calls" because "a series" is precisely what Heat does not do
-- it does "a graph" of things, if anything. There is nothing wrong
with this. If you want to do a series of things, however, you need a
different tool.

>>
>>> 3) It ties us 100% to a given implementation.  If Mistral proves to be a
>>> poor choice for some reason, or insufficient for a particular use case,
>>> we have no alternative.  If we have an API and decide to change our
>>> implementation, nobody has to know or care.  This is kind of the whole
>>> point of having an API - it shields users from all the nasty
>>> implementation details under the surface.
>>
>> This is a valid argument, but building (and maintining, forever) a bespoke
>> API is a high cost to pay for this additional degree of abstraction, and
>> when you think of the target audience, I'm not certain it's entirely
>> justified (or, honestly, if our community can bear that overhead);
>>
>> For example, take other single-purpose "deployment" projects, such as
>> Sahara, Magnum, perhaps Trove.  These are designed primarily as user-facing
>> API's, where the services will ultimately be consumed by public and private
>> cloud customers.
>>

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dougal Matthews]

Great post Ben, I largely agree with what you are saying, a lot of your
points
are valid concerns that I share. Just a couple of comments inline as I try
to
avoid what others have said.

On 22 January 2016 at 17:24, Ben Nemec  wrote:

> So I haven't weighed in on this yet, in part because I was on vacation
> when it was first proposed and missed a lot of the initial discussion,
> and also because I wanted to take some time to order my thoughts on it.
>  Also because my initial reaction...was not conducive to calm and
> rational discussion. ;-)
>
> The tldr is that I don't like it.  To explain why, I'm going to make a
> list (everyone loves lists, right? Top $NUMBER reasons we should stop
> expecting other people to write our API for us):
>
> 1) We've been down this road before.  Except last time it was with Heat.
>  I'm being somewhat tongue-in-cheek here, but expecting a general
> service to provide us a user-friendly API for our specific use case just
> doesn't make sense to me.
>

I don't really expect the API to be used directly by many users - how many
users directly used the Tuskar API? I suspect they all used it via the CLI
tools or the GUI. I could be completely wrong with the assumption but I
think
the primary entry points will be the CLI and GUI we expose. Only advanced
users would need to consider Mistral.

Anyone directly interfacing with the API's will need to learn a ton of
services,
I don't think Mistral adds much more complexity here.



> 2) The TripleO API is not a workflow API.  I also largely missed this
> discussion, but the TripleO API is a _Deployment_ API.  In some cases
> there also happens to be a workflow going on behind the scenes, but
> honestly that's not something I want our users to have to care about.
>

Yeah, I agree that one of the downsides of the Mistral approach is that we
blur the lines between these two.


>
> 3) It ties us 100% to a given implementation.  If Mistral proves to be a
> poor choice for some reason, or insufficient for a particular use case,
> we have no alternative.  If we have an API and decide to change our
> implementation, nobody has to know or care.  This is kind of the whole
> point of having an API - it shields users from all the nasty
> implementation details under the surface.
>
> 4) It raises the bar even further for both new deployers and developers.
>  You already need to have a pretty firm grasp of Puppet and Heat
> templates to understand how our stuff works, not to mention a decent
> understanding of quite a number of OpenStack services.
>
> This presents a big chicken and egg problem for people new to OpenStack.
>  It's great that we're based on OpenStack and that allows people to peek
> under the hood and do some tinkering, but it can't be required for
> everyone.  A lot of our deployers are going to have little to no
> OpenStack experience, and TripleO is already a daunting task for those
> people (hell, it's daunting for people who _are_ experienced).
>
> 5) What does reimplementing all of our tested, well-understood Python
> into a new YAML format gain us?  This is maybe the biggest thing I'm
> missing from this whole discussion.  We lose a bunch of things (ease of
> transition from other Python projects, excellent existing testing
> framework, etc.), but what are we actually gaining other than the
> ability to say that we use N + 1 OpenStack services?  Because we're way
> past the point where "It's OpenStack deploying OpenStack" is sufficient
> reason for people to pay attention to us.  We need less "Ooh, neat" and
> more "Ooh, that's easy to use and works well."  It's still not clear to
> me that Mistral helps in any way with the latter.
>
> 6) On the testing note, how do we test these workflows?  Do we know what
> happens when step X fails?  How do we test that they handle it properly
> in an automated and repeatable way?  In Python these are largely easy
> questions to answer: unit tests.  How do you unit test YAML?  This is a
> big reason I'm not even crazy about having Mistral on the back end of a
> TripleO API.  We'd be going from code that we can test and prove works
> in a variety of scenarios, to YAML that is tested and proven to work in
> exactly the three scenarios we run in CI.  This is basically the same
> situation we had with tripleo-incubator, and it was bad there too.
>
> I dunno.  Maybe I'm too late to this party to have any impact on the
> discussion, but I very much do not like the direction we're going and I
> would be remiss if I didn't at least point out my concerns with it.
>
> -Ben
>
> On 01/13/2016 03:41 AM, Tzu-Mainn Chen wrote:
> > Hey all,
> >
> > I realize now from the title of the other TripleO/Mistral thread [1] that
> > the discussion there may have gotten confused.  I think using Mistral for
> > TripleO processes that are obviously workflows - stack deployment, node
> > registration - makes perfect sense.  That thread is exploring
> practicalities
> > for doing that, and I 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Adam Young]

On 01/22/2016 09:01 AM, Dan Prince wrote:

I gave a couple of ad-hoc demo's of my Mistral dev environment this
week. For those interested in tinkering with TripleO Mistral actions in
tripleo-common here are links to the code I used for those. Still a
rough prototype but hopefully demonstrates how these things might work:

https://github.com/dprince/tripleo-common/tree/mistral

https://github.com/dprince/python-tripleoclient/tree/mistral

The first commit in each of those basically explains what was changed.

Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Regardless of the underlying implementation, the CLI for Tripleo should 
be the openstack common CLI, and the operations should be specifically 
named for what they are doing, not the implementation.



openstack undercloud install
openstack overcloud install

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Clint Byrum]

Excerpts from Dan Prince's message of 2016-01-22 16:19:07 -0800:
> On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> > So I haven't weighed in on this yet, in part because I was on
> > vacation
> > when it was first proposed and missed a lot of the initial
> > discussion,
> > and also because I wanted to take some time to order my thoughts on
> > it.
> >  Also because my initial reaction...was not conducive to calm and
> > rational discussion. ;-)
> > 
> > The tldr is that I don't like it.  To explain why, I'm going to make
> > a
> > list (everyone loves lists, right? Top $NUMBER reasons we should stop
> > expecting other people to write our API for us):
> > 
> > 1) We've been down this road before.  Except last time it was with
> > Heat.
> >  I'm being somewhat tongue-in-cheek here, but expecting a general
> > service to provide us a user-friendly API for our specific use case
> > just
> > doesn't make sense to me.
> 
> We've been down this road with Heat yes. But we are currently using
> Heat for some things that we arguable should be (a workflows tool might
> help offload some stuff out of Heat). Also we haven't implemented
> custom Heat resources for TripleO either. There are mixed opinions on
> this but plugging in your code to a generic API is quite nice
> sometimes.
> 
> That is the beauty of Mistral I think. Unlike Heat it actually
> encourages you to customize it with custom Python actions. Anything we
> want in tripleo-common can become our own Mistral action (these get
> registered with stevedore entry points so we'd own the code) and the
> YAML workflows just tie them together via tasks.
> 

Since TripleO's undercloud is a special, single-purpose cloud just for
deploying OpenStack, I think that is a perfectly fine idea. But it also
somewhat invalidates the reasons to use Mistral. If you can control all
the pieces, IMO you may want to use the best tool for that job, not the
one that is the most OpenStack. The virtuous circle only works if you're
actually improving the OpenStack components for their general use cases.

> We don't have to go off and build our own proxy deployment workflow
> API. The structure to do just about anything we need already exists so
> why not go and use it?
> 
> > 
> > 2) The TripleO API is not a workflow API.  I also largely missed this
> > discussion, but the TripleO API is a _Deployment_ API.  In some cases
> > there also happens to be a workflow going on behind the scenes, but
> > honestly that's not something I want our users to have to care about.
> 
> Agree that users don't have to care about this.
> 
> Users can get as involved as they want here. Most users I think will
> use python-tripleoclient to drive the deployment or the new UI. They
> don't have to interact with Mistral directly unless they really want
> to. So whether we choose to build our own API or use a generic one I
> think this point is mute.
> 
> > 
> > 3) It ties us 100% to a given implementation.  If Mistral proves to
> > be a
> > poor choice for some reason, or insufficient for a particular use
> > case,
> > we have no alternative.  If we have an API and decide to change our
> > implementation, nobody has to know or care.  This is kind of the
> > whole
> > point of having an API - it shields users from all the nasty
> > implementation details under the surface.
> 
> 
> Mistal's API is a generic workflow API. It is very much the same layer
> that I think we would get if we were to integrate with something like
> Ansible Tower... except that Mistral is part of OpenStack. It
> integrates very nicely with OpenStack services and is very customizable
> with custom actions. The fact that Mistral sits much closer to
> OpenStack and is essentially a light shim on top of it is to our
> advantage (being TripleO). To think that we can build up a proxy API in
> such a manner that we might be able to swap in an entirely new backend
> (without even having a fully implement backend yet to begin with) is
> for me a bit of a stretch. We've got a lot of "TripleO API" maturing
> before we'll get to this point. Which is why I lean towards using a
> generic workflow API to accomplis the same task.
> 

Ansible 2.0 integrates extremely well with OpenStack too. Even better,
you don't need an inside-the-cloud service to use it to interact
with OpenStack. Now, sometimes you want an "aaS" helping you, and as
you've alluded to, Tower is sort of like that. As is Mistral for its
language. I've suggested before that Mistral would be well served by
adopting Ansible as a language for workflow, since the concepts they're
encoding are very similar. In fact, learning Mistral's DSL feels like
learning Spanish after already having studied Latin. The structure is
the same, the meanings are the same. It's just different words.

> I actually think rather than shielding users we should be more
> transparent about the actual workflows that are driving deployment.
> Smaller more focused workflows that we string together to drive the

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Jay Dobies]

I fall very much in the same mentality as Ben. I'm +1 to all of his 
points, with a few comments inline.


On 01/22/2016 12:24 PM, Ben Nemec wrote:

So I haven't weighed in on this yet, in part because I was on vacation
when it was first proposed and missed a lot of the initial discussion,
and also because I wanted to take some time to order my thoughts on it.
  Also because my initial reaction...was not conducive to calm and
rational discussion. ;-)

The tldr is that I don't like it.  To explain why, I'm going to make a
list (everyone loves lists, right? Top $NUMBER reasons we should stop
expecting other people to write our API for us):

1) We've been down this road before.  Except last time it was with Heat.
  I'm being somewhat tongue-in-cheek here, but expecting a general
service to provide us a user-friendly API for our specific use case just
doesn't make sense to me.


I think it's important to think about outside integrations here. The 
current model is to tell other clients to manipulate Heat environments 
and understand how to parse/inspect templates*. Now it will be to 
understand/parse/manipulate Mistral workflows. Neither of those are 
conducive to the types of UI wireframes we've proposed in the past, much 
less friendly to completely outside integrators.


* I realize some of that inspection is moving into Heat, but it's still 
at the mechanical template level rather than providing insight into how 
to actually use them.



2) The TripleO API is not a workflow API.  I also largely missed this
discussion, but the TripleO API is a _Deployment_ API.  In some cases
there also happens to be a workflow going on behind the scenes, but
honestly that's not something I want our users to have to care about.


I'm glad Ben mentioned this, because I always viewed the workflow 
aspects as a subset of what actually needs to be done.



3) It ties us 100% to a given implementation.  If Mistral proves to be a
poor choice for some reason, or insufficient for a particular use case,
we have no alternative.  If we have an API and decide to change our
implementation, nobody has to know or care.  This is kind of the whole
point of having an API - it shields users from all the nasty
implementation details under the surface.


I strongly agree with this one. It's not even generic speculation; we've 
told people in the past to deal with Heat templates and now we're 
telling them to deal with workflows. We already have a history of the 
backend changing and an API would give us much more flexibility (and 
less annoyed users).



4) It raises the bar even further for both new deployers and developers.
  You already need to have a pretty firm grasp of Puppet and Heat
templates to understand how our stuff works, not to mention a decent
understanding of quite a number of OpenStack services.

This presents a big chicken and egg problem for people new to OpenStack.
  It's great that we're based on OpenStack and that allows people to peek
under the hood and do some tinkering, but it can't be required for
everyone.  A lot of our deployers are going to have little to no
OpenStack experience, and TripleO is already a daunting task for those
people (hell, it's daunting for people who _are_ experienced).

5) What does reimplementing all of our tested, well-understood Python
into a new YAML format gain us?  This is maybe the biggest thing I'm
missing from this whole discussion.  We lose a bunch of things (ease of
transition from other Python projects, excellent existing testing
framework, etc.), but what are we actually gaining other than the
ability to say that we use N + 1 OpenStack services?  Because we're way
past the point where "It's OpenStack deploying OpenStack" is sufficient
reason for people to pay attention to us.  We need less "Ooh, neat" and
more "Ooh, that's easy to use and works well."  It's still not clear to
me that Mistral helps in any way with the latter.

6) On the testing note, how do we test these workflows?  Do we know what
happens when step X fails?  How do we test that they handle it properly
in an automated and repeatable way?  In Python these are largely easy
questions to answer: unit tests.  How do you unit test YAML?  This is a
big reason I'm not even crazy about having Mistral on the back end of a
TripleO API.  We'd be going from code that we can test and prove works
in a variety of scenarios, to YAML that is tested and proven to work in
exactly the three scenarios we run in CI.  This is basically the same
situation we had with tripleo-incubator, and it was bad there too.

I dunno.  Maybe I'm too late to this party to have any impact on the
discussion, but I very much do not like the direction we're going and I
would be remiss if I didn't at least point out my concerns with it.

-Ben

On 01/13/2016 03:41 AM, Tzu-Mainn Chen wrote:

Hey all,

I realize now from the title of the other TripleO/Mistral thread [1] that
the discussion there may have gotten confused.  I think using Mistral for
TripleO 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Ben Nemec]

On 01/22/2016 12:30 PM, Ryan Brady wrote:
> 
> 
> On Fri, Jan 22, 2016 at 12:24 PM, Ben Nemec  > wrote:
> 
> So I haven't weighed in on this yet, in part because I was on vacation
> when it was first proposed and missed a lot of the initial discussion,
> and also because I wanted to take some time to order my thoughts on it.
>  Also because my initial reaction...was not conducive to calm and
> rational discussion. ;-)
> 
> The tldr is that I don't like it.  To explain why, I'm going to make a
> list (everyone loves lists, right? Top $NUMBER reasons we should stop
> expecting other people to write our API for us):
> 
> 1) We've been down this road before.  Except last time it was with Heat.
>  I'm being somewhat tongue-in-cheek here, but expecting a general
> service to provide us a user-friendly API for our specific use case just
> doesn't make sense to me.
> 
> 
> The UI for is the CLI or the GUI right?  The typical method for
> interacting with TripleO seems defined that way.  If we're worried about
> it being friendly enough to integrate with I'm sure it could be easily
> explained in a doc.  Underneath the difference is calling to
> :8989/v2/executions instead of :8585/v1/deploy.  I'm not convinced
> there's a large enough difference here to say that one is more
> user-friendly or not than the other.

The difference is in having parameters passed via a well-defined
JSON-based API like every other OpenStack service, or having parameters
passed via a loosely defined "API" in the Mistral YAML.  The latter is
what we're currently doing for the Heat templates, and I don't think
it's particularly good there either.  We make changes to the template
API all the time that break our consumers (I know this because I am one
of them, and I'm constantly having to rebase my nic templates because of
interface changes).

>  
> 
> 2) The TripleO API is not a workflow API.  I also largely missed this
> discussion, but the TripleO API is a _Deployment_ API.  In some cases
> there also happens to be a workflow going on behind the scenes, but
> honestly that's not something I want our users to have to care about.
> 
> 3) It ties us 100% to a given implementation.  If Mistral proves to be a
> poor choice for some reason, or insufficient for a particular use case,
> we have no alternative.  If we have an API and decide to change our
> implementation, nobody has to know or care.  This is kind of the whole
> point of having an API - it shields users from all the nasty
> implementation details under the surface.
> 
> 4) It raises the bar even further for both new deployers and developers.
>  You already need to have a pretty firm grasp of Puppet and Heat
> templates to understand how our stuff works, not to mention a decent
> understanding of quite a number of OpenStack services.
> 
> 
> And an additional 14 or so active projects TripleO created, right?

It's a stupidly complex environment, yes. :-)

>  
> 
> 
> This presents a big chicken and egg problem for people new to OpenStack.
>  It's great that we're based on OpenStack and that allows people to peek
> under the hood and do some tinkering, but it can't be required for
> everyone.  A lot of our deployers are going to have little to no
> OpenStack experience, and TripleO is already a daunting task for those
> people (hell, it's daunting for people who _are_ experienced).
> 
> 
> The original vision statement [1] for TripleO has not changed in some
> time.  "TripleO is a program aimed at installing, upgrading and
> operating OpenStack clouds using OpenStack's own cloud facilities as the
> foundations".  I'm not disagreeing that TripleO *could* be a deployment
> API, but that's not the language used in the wiki and if the goal is an
> API then maybe this needs to be formalized to get everyone in the
> project on the same page.  Following along with the statement as it
> stands, using existing OpenStack services sounds like a mandate.  How do
> we decide as a project when something within OpenStack can be improved
> or when we just create something new?  How do we convey that
> justification to both developers and users? 

"using existing OpenStack services sounds like a mandate." Where do you
draw the line on that?  Taken to extremes, now that Fuel is big tent we
should just use that and all go home. ;-)

I think there's got to be some qualification of that statement, and I
think it comes down to using OpenStack services where it makes sense.
I'm arguing that in this case it doesn't.

There's another important part of this that I think we also need to
discuss, which is the Red Hat acquisition of Ansible.  While it's true
that today Ansible doesn't have an open API service, it seems reasonable
(though I _do not_ have any inside information about this) to assume
that Ansible Tower will be open 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Ryan Brady]

On Fri, Jan 22, 2016 at 12:24 PM, Ben Nemec  wrote:

> So I haven't weighed in on this yet, in part because I was on vacation
> when it was first proposed and missed a lot of the initial discussion,
> and also because I wanted to take some time to order my thoughts on it.
>  Also because my initial reaction...was not conducive to calm and
> rational discussion. ;-)
>
> The tldr is that I don't like it.  To explain why, I'm going to make a
> list (everyone loves lists, right? Top $NUMBER reasons we should stop
> expecting other people to write our API for us):
>
> 1) We've been down this road before.  Except last time it was with Heat.
>  I'm being somewhat tongue-in-cheek here, but expecting a general
> service to provide us a user-friendly API for our specific use case just
> doesn't make sense to me.
>
>
The UI for is the CLI or the GUI right?  The typical method for interacting
with TripleO seems defined that way.  If we're worried about it being
friendly enough to integrate with I'm sure it could be easily explained in
a doc.  Underneath the difference is calling to :8989/v2/executions instead
of :8585/v1/deploy.  I'm not convinced there's a large enough difference
here to say that one is more user-friendly or not than the other.


> 2) The TripleO API is not a workflow API.  I also largely missed this
> discussion, but the TripleO API is a _Deployment_ API.  In some cases
> there also happens to be a workflow going on behind the scenes, but
> honestly that's not something I want our users to have to care about.
>
> 3) It ties us 100% to a given implementation.  If Mistral proves to be a
> poor choice for some reason, or insufficient for a particular use case,
> we have no alternative.  If we have an API and decide to change our
> implementation, nobody has to know or care.  This is kind of the whole
> point of having an API - it shields users from all the nasty
> implementation details under the surface.
>
> 4) It raises the bar even further for both new deployers and developers.
>  You already need to have a pretty firm grasp of Puppet and Heat
> templates to understand how our stuff works, not to mention a decent
> understanding of quite a number of OpenStack services.
>

And an additional 14 or so active projects TripleO created, right?


>
> This presents a big chicken and egg problem for people new to OpenStack.
>  It's great that we're based on OpenStack and that allows people to peek
> under the hood and do some tinkering, but it can't be required for
> everyone.  A lot of our deployers are going to have little to no
> OpenStack experience, and TripleO is already a daunting task for those
> people (hell, it's daunting for people who _are_ experienced).
>

The original vision statement [1] for TripleO has not changed in some
time.  "TripleO is a program aimed at installing, upgrading and operating
OpenStack clouds using OpenStack's own cloud facilities as the
foundations".  I'm not disagreeing that TripleO *could* be a deployment
API, but that's not the language used in the wiki and if the goal is an API
then maybe this needs to be formalized to get everyone in the project on
the same page.  Following along with the statement as it stands, using
existing OpenStack services sounds like a mandate.  How do we decide as a
project when something within OpenStack can be improved or when we just
create something new?  How do we convey that justification to both
developers and users?



>
> 5) What does reimplementing all of our tested, well-understood Python
> into a new YAML format gain us?  This is maybe the biggest thing I'm
> missing from this whole discussion.


So workflows are in yaml, but the actions are still in python.  The actions
have an __init__ and a run method which looks testable to me.


> We lose a bunch of things (ease of
> transition from other Python projects, excellent existing testing
> framework, etc.), but what are we actually gaining other than the
> ability to say that we use N + 1 OpenStack services?  Because we're way
> past the point where "It's OpenStack deploying OpenStack" is sufficient
> reason for people to pay attention to us.  We need less "Ooh, neat" and
> more "Ooh, that's easy to use and works well."  It's still not clear to
> me that Mistral helps in any way with the latter.
>
> 6) On the testing note, how do we test these workflows?  Do we know what
> happens when step X fails?  How do we test that they handle it properly
> in an automated and repeatable way?  In Python these are largely easy
> questions to answer: unit tests.  How do you unit test YAML?  This is a
> big reason I'm not even crazy about having Mistral on the back end of a
> TripleO API.  We'd be going from code that we can test and prove works
> in a variety of scenarios, to YAML that is tested and proven to work in
> exactly the three scenarios we run in CI.  This is basically the same
> situation we had with tripleo-incubator, and it was bad there too.
>
>
If a 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> So I haven't weighed in on this yet, in part because I was on
> vacation
> when it was first proposed and missed a lot of the initial
> discussion,
> and also because I wanted to take some time to order my thoughts on
> it.
>  Also because my initial reaction...was not conducive to calm and
> rational discussion. ;-)
> 
> The tldr is that I don't like it.  To explain why, I'm going to make
> a
> list (everyone loves lists, right? Top $NUMBER reasons we should stop
> expecting other people to write our API for us):
> 
> 1) We've been down this road before.  Except last time it was with
> Heat.
>  I'm being somewhat tongue-in-cheek here, but expecting a general
> service to provide us a user-friendly API for our specific use case
> just
> doesn't make sense to me.

We've been down this road with Heat yes. But we are currently using
Heat for some things that we arguable should be (a workflows tool might
help offload some stuff out of Heat). Also we haven't implemented
custom Heat resources for TripleO either. There are mixed opinions on
this but plugging in your code to a generic API is quite nice
sometimes.

That is the beauty of Mistral I think. Unlike Heat it actually
encourages you to customize it with custom Python actions. Anything we
want in tripleo-common can become our own Mistral action (these get
registered with stevedore entry points so we'd own the code) and the
YAML workflows just tie them together via tasks.

We don't have to go off and build our own proxy deployment workflow
API. The structure to do just about anything we need already exists so
why not go and use it?

> 
> 2) The TripleO API is not a workflow API.  I also largely missed this
> discussion, but the TripleO API is a _Deployment_ API.  In some cases
> there also happens to be a workflow going on behind the scenes, but
> honestly that's not something I want our users to have to care about.

Agree that users don't have to care about this.

Users can get as involved as they want here. Most users I think will
use python-tripleoclient to drive the deployment or the new UI. They
don't have to interact with Mistral directly unless they really want
to. So whether we choose to build our own API or use a generic one I
think this point is mute.

> 
> 3) It ties us 100% to a given implementation.  If Mistral proves to
> be a
> poor choice for some reason, or insufficient for a particular use
> case,
> we have no alternative.  If we have an API and decide to change our
> implementation, nobody has to know or care.  This is kind of the
> whole
> point of having an API - it shields users from all the nasty
> implementation details under the surface.


Mistal's API is a generic workflow API. It is very much the same layer
that I think we would get if we were to integrate with something like
Ansible Tower... except that Mistral is part of OpenStack. It
integrates very nicely with OpenStack services and is very customizable
with custom actions. The fact that Mistral sits much closer to
OpenStack and is essentially a light shim on top of it is to our
advantage (being TripleO). To think that we can build up a proxy API in
such a manner that we might be able to swap in an entirely new backend
(without even having a fully implement backend yet to begin with) is
for me a bit of a stretch. We've got a lot of "TripleO API" maturing
before we'll get to this point. Which is why I lean towards using a
generic workflow API to accomplis the same task.

I actually think rather than shielding users we should be more
transparent about the actual workflows that are driving deployment.
Smaller more focused workflows that we string together to drive the
deployment.

> 
> 4) It raises the bar even further for both new deployers and
> developers.
>  You already need to have a pretty firm grasp of Puppet and Heat
> templates to understand how our stuff works, not to mention a decent
> understanding of quite a number of OpenStack services.
> 
> This presents a big chicken and egg problem for people new to
> OpenStack.
>  It's great that we're based on OpenStack and that allows people to
> peek
> under the hood and do some tinkering, but it can't be required for
> everyone.  A lot of our deployers are going to have little to no
> OpenStack experience, and TripleO is already a daunting task for
> those
> people (hell, it's daunting for people who _are_ experienced).
> 

And on the flipside you will get more of a community around using an
OpenStack project than you ever would going off and building your own
"Deployment/Workflow API". 

I would actually argue this is less of a deployers thing and more of a
development tool choice. IMO most deployers will use python-
tripleoclient or some UI and not mistralclient directly. The code I've
posted this week shows a prototype of just this, Mistral is swapped in
such that you would never know it was involved because python-
tripleoclient works like it always did. Deployers use 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

I gave a couple of ad-hoc demo's of my Mistral dev environment this
week. For those interested in tinkering with TripleO Mistral actions in
tripleo-common here are links to the code I used for those. Still a
rough prototype but hopefully demonstrates how these things might work:

https://github.com/dprince/tripleo-common/tree/mistral

https://github.com/dprince/python-tripleoclient/tree/mistral

The first commit in each of those basically explains what was changed.

Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Ben Nemec]

So I haven't weighed in on this yet, in part because I was on vacation
when it was first proposed and missed a lot of the initial discussion,
and also because I wanted to take some time to order my thoughts on it.
 Also because my initial reaction...was not conducive to calm and
rational discussion. ;-)

The tldr is that I don't like it.  To explain why, I'm going to make a
list (everyone loves lists, right? Top $NUMBER reasons we should stop
expecting other people to write our API for us):

1) We've been down this road before.  Except last time it was with Heat.
 I'm being somewhat tongue-in-cheek here, but expecting a general
service to provide us a user-friendly API for our specific use case just
doesn't make sense to me.

2) The TripleO API is not a workflow API.  I also largely missed this
discussion, but the TripleO API is a _Deployment_ API.  In some cases
there also happens to be a workflow going on behind the scenes, but
honestly that's not something I want our users to have to care about.

3) It ties us 100% to a given implementation.  If Mistral proves to be a
poor choice for some reason, or insufficient for a particular use case,
we have no alternative.  If we have an API and decide to change our
implementation, nobody has to know or care.  This is kind of the whole
point of having an API - it shields users from all the nasty
implementation details under the surface.

4) It raises the bar even further for both new deployers and developers.
 You already need to have a pretty firm grasp of Puppet and Heat
templates to understand how our stuff works, not to mention a decent
understanding of quite a number of OpenStack services.

This presents a big chicken and egg problem for people new to OpenStack.
 It's great that we're based on OpenStack and that allows people to peek
under the hood and do some tinkering, but it can't be required for
everyone.  A lot of our deployers are going to have little to no
OpenStack experience, and TripleO is already a daunting task for those
people (hell, it's daunting for people who _are_ experienced).

5) What does reimplementing all of our tested, well-understood Python
into a new YAML format gain us?  This is maybe the biggest thing I'm
missing from this whole discussion.  We lose a bunch of things (ease of
transition from other Python projects, excellent existing testing
framework, etc.), but what are we actually gaining other than the
ability to say that we use N + 1 OpenStack services?  Because we're way
past the point where "It's OpenStack deploying OpenStack" is sufficient
reason for people to pay attention to us.  We need less "Ooh, neat" and
more "Ooh, that's easy to use and works well."  It's still not clear to
me that Mistral helps in any way with the latter.

6) On the testing note, how do we test these workflows?  Do we know what
happens when step X fails?  How do we test that they handle it properly
in an automated and repeatable way?  In Python these are largely easy
questions to answer: unit tests.  How do you unit test YAML?  This is a
big reason I'm not even crazy about having Mistral on the back end of a
TripleO API.  We'd be going from code that we can test and prove works
in a variety of scenarios, to YAML that is tested and proven to work in
exactly the three scenarios we run in CI.  This is basically the same
situation we had with tripleo-incubator, and it was bad there too.

I dunno.  Maybe I'm too late to this party to have any impact on the
discussion, but I very much do not like the direction we're going and I
would be remiss if I didn't at least point out my concerns with it.

-Ben

On 01/13/2016 03:41 AM, Tzu-Mainn Chen wrote:
> Hey all,
> 
> I realize now from the title of the other TripleO/Mistral thread [1] that
> the discussion there may have gotten confused.  I think using Mistral for
> TripleO processes that are obviously workflows - stack deployment, node
> registration - makes perfect sense.  That thread is exploring practicalities
> for doing that, and I think that's great work.
> 
> What I inappropriately started to address in that thread was a somewhat
> orthogonal point that Dan asked in his original email, namely:
> 
> "what it might look like if we were to use Mistral as a replacement for the
> TripleO API entirely"
> 
> I'd like to create this thread to talk about that; more of a 'should we'
> than 'can we'.  And to do that, I want to indulge in a thought exercise
> stemming from an IRC discussion with Dan and others.  All, please correct me
> if I've misstated anything.
> 
> The IRC discussion revolved around one use case: deploying a Heat stack
> directly from a Swift container.  With an updated patch, the Heat CLI can
> support this functionality natively.  Then we don't need a TripleO API; we
> can use Mistral to access that functionality, and we're done, with no need
> for additional code within TripleO.  And, as I understand it, that's the
> true motivation for using Mistral instead of a TripleO API: avoiding custom

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dougal Matthews]

On 21 January 2016 at 14:46, Dougal Matthews  wrote:

>
>
> On 20 January 2016 at 20:05, Tzu-Mainn Chen  wrote:
>
>> - Original Message -
>> > On 18.1.2016 19:49, Tzu-Mainn Chen wrote:
>> > > - Original Message -
>> > >> On Thu, 2016-01-14 at 16:04 -0500, Tzu-Mainn Chen wrote:
>> > >>>
>> > >>> - Original Message -
>> >  On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:
>> > > Hey all,
>> > >
>> > > I realize now from the title of the other TripleO/Mistral thread
>> > > [1] that
>> > > the discussion there may have gotten confused.  I think using
>> > > Mistral for
>> > > TripleO processes that are obviously workflows - stack
>> > > deployment, node
>> > > registration - makes perfect sense.  That thread is exploring
>> > > practicalities
>> > > for doing that, and I think that's great work.
>> > >
>> > > What I inappropriately started to address in that thread was a
>> > > somewhat
>> > > orthogonal point that Dan asked in his original email, namely:
>> > >
>> > > "what it might look like if we were to use Mistral as a
>> > > replacement for the
>> > > TripleO API entirely"
>> > >
>> > > I'd like to create this thread to talk about that; more of a
>> > > 'should we'
>> > > than 'can we'.  And to do that, I want to indulge in a thought
>> > > exercise
>> > > stemming from an IRC discussion with Dan and others.  All, please
>> > > correct
>> > > me
>> > > if I've misstated anything.
>> > >
>> > > The IRC discussion revolved around one use case: deploying a Heat
>> > > stack
>> > > directly from a Swift container.  With an updated patch, the Heat
>> > > CLI can
>> > > support this functionality natively.  Then we don't need a
>> > > TripleO API; we
>> > > can use Mistral to access that functionality, and we're done,
>> > > with no need
>> > > for additional code within TripleO.  And, as I understand it,
>> > > that's the
>> > > true motivation for using Mistral instead of a TripleO API:
>> > > avoiding custom
>> > > code within TripleO.
>> > >
>> > > That's definitely a worthy goal... except from my perspective,
>> > > the story
>> > > doesn't quite end there.  A GUI needs additional functionality,
>> > > which boils
>> > > down to: understanding the Heat deployment templates in order to
>> > > provide
>> > > options for a user; and persisting those options within a Heat
>> > > environment
>> > > file.
>> > >
>> > > Right away I think we hit a problem.  Where does the code for
>> > > 'understanding
>> > > options' go?  Much of that understanding comes from the
>> > > capabilities map
>> > > in tripleo-heat-templates [2]; it would make sense to me that
>> > > responsibility
>> > > for that would fall to a TripleO library.
>> > >
>> > > Still, perhaps we can limit the amount of TripleO code.  So to
>> > > give API
>> > > access to 'getDeploymentOptions', we can create a Mistral
>> > > workflow.
>> > >
>> > >Retrieve Heat templates from Swift -> Parse capabilities map
>> > >
>> > > Which is fine-ish, except from an architectural perspective
>> > > 'getDeploymentOptions' violates the abstraction layer between
>> > > storage and
>> > > business logic, a problem that is compounded because
>> > > 'getDeploymentOptions'
>> > > is not the only functionality that accesses the Heat templates
>> > > and needs
>> > > exposure through an API.  And, as has been discussed on a
>> > > separate TripleO
>> > > thread, we're not even sure Swift is sufficient for our needs;
>> > > one possible
>> > > consideration right now is allowing deployment from templates
>> > > stored in
>> > > multiple places, such as the file system or git.
>> > 
>> >  Actually, that whole capabilities map thing is a workaround for a
>> >  missing
>> >  feature in Heat, which I have proposed, but am having a hard time
>> >  reaching
>> >  consensus on within the Heat community:
>> > 
>> >  https://review.openstack.org/#/c/196656/
>> > 
>> >  Given that is a large part of what's anticipated to be provided by
>> >  the
>> >  proposed TripleO API, I'd welcome feedback and collaboration so we
>> >  can move
>> >  that forward, vs solving only for TripleO.
>> > 
>> > > Are we going to have duplicate 'getDeploymentOptions' workflows
>> > > for each
>> > > storage mechanism?  If we consolidate the storage code within a
>> > > TripleO
>> > > library, do we really need a *workflow* to call a single
>> > > function?  Is a
>> > > thin TripleO API that contains no additional business logic
>> > > really so bad
>> > > at that point?
>> > 
>> >  Actually, 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Tzu-Mainn Chen]

- Original Message -

> On 21 January 2016 at 14:46, Dougal Matthews < dou...@redhat.com > wrote:

> > On 20 January 2016 at 20:05, Tzu-Mainn Chen < tzuma...@redhat.com > wrote:
> 

> > > - Original Message -
> > 
> 
> > > > On 18.1.2016 19:49, Tzu-Mainn Chen wrote:
> > 
> 
> > > > > - Original Message -
> > 
> 
> > > > >> On Thu, 2016-01-14 at 16:04 -0500, Tzu-Mainn Chen wrote:
> > 
> 
> > > > >>>
> > 
> 
> > > > >>> - Original Message -
> > 
> 
> > > >  On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:
> > 
> 
> > > > > Hey all,
> > 
> 
> > > > >
> > 
> 
> > > > > I realize now from the title of the other TripleO/Mistral thread
> > 
> 
> > > > > [1] that
> > 
> 
> > > > > the discussion there may have gotten confused. I think using
> > 
> 
> > > > > Mistral for
> > 
> 
> > > > > TripleO processes that are obviously workflows - stack
> > 
> 
> > > > > deployment, node
> > 
> 
> > > > > registration - makes perfect sense. That thread is exploring
> > 
> 
> > > > > practicalities
> > 
> 
> > > > > for doing that, and I think that's great work.
> > 
> 
> > > > >
> > 
> 
> > > > > What I inappropriately started to address in that thread was a
> > 
> 
> > > > > somewhat
> > 
> 
> > > > > orthogonal point that Dan asked in his original email, namely:
> > 
> 
> > > > >
> > 
> 
> > > > > "what it might look like if we were to use Mistral as a
> > 
> 
> > > > > replacement for the
> > 
> 
> > > > > TripleO API entirely"
> > 
> 
> > > > >
> > 
> 
> > > > > I'd like to create this thread to talk about that; more of a
> > 
> 
> > > > > 'should we'
> > 
> 
> > > > > than 'can we'. And to do that, I want to indulge in a thought
> > 
> 
> > > > > exercise
> > 
> 
> > > > > stemming from an IRC discussion with Dan and others. All, please
> > 
> 
> > > > > correct
> > 
> 
> > > > > me
> > 
> 
> > > > > if I've misstated anything.
> > 
> 
> > > > >
> > 
> 
> > > > > The IRC discussion revolved around one use case: deploying a Heat
> > 
> 
> > > > > stack
> > 
> 
> > > > > directly from a Swift container. With an updated patch, the Heat
> > 
> 
> > > > > CLI can
> > 
> 
> > > > > support this functionality natively. Then we don't need a
> > 
> 
> > > > > TripleO API; we
> > 
> 
> > > > > can use Mistral to access that functionality, and we're done,
> > 
> 
> > > > > with no need
> > 
> 
> > > > > for additional code within TripleO. And, as I understand it,
> > 
> 
> > > > > that's the
> > 
> 
> > > > > true motivation for using Mistral instead of a TripleO API:
> > 
> 
> > > > > avoiding custom
> > 
> 
> > > > > code within TripleO.
> > 
> 
> > > > >
> > 
> 
> > > > > That's definitely a worthy goal... except from my perspective,
> > 
> 
> > > > > the story
> > 
> 
> > > > > doesn't quite end there. A GUI needs additional functionality,
> > 
> 
> > > > > which boils
> > 
> 
> > > > > down to: understanding the Heat deployment templates in order to
> > 
> 
> > > > > provide
> > 
> 
> > > > > options for a user; and persisting those options within a Heat
> > 
> 
> > > > > environment
> > 
> 
> > > > > file.
> > 
> 
> > > > >
> > 
> 
> > > > > Right away I think we hit a problem. Where does the code for
> > 
> 
> > > > > 'understanding
> > 
> 
> > > > > options' go? Much of that understanding comes from the
> > 
> 
> > > > > capabilities map
> > 
> 
> > > > > in tripleo-heat-templates [2]; it would make sense to me that
> > 
> 
> > > > > responsibility
> > 
> 
> > > > > for that would fall to a TripleO library.
> > 
> 
> > > > >
> > 
> 
> > > > > Still, perhaps we can limit the amount of TripleO code. So to
> > 
> 
> > > > > give API
> > 
> 
> > > > > access to 'getDeploymentOptions', we can create a Mistral
> > 
> 
> > > > > workflow.
> > 
> 
> > > > >
> > 
> 
> > > > > Retrieve Heat templates from Swift -> Parse capabilities map
> > 
> 
> > > > >
> > 
> 
> > > > > Which is fine-ish, except from an architectural perspective
> > 
> 
> > > > > 'getDeploymentOptions' violates the abstraction layer between
> > 
> 
> > > > > storage and
> > 
> 
> > > > > business logic, a problem that is compounded because
> > 
> 
> > > > > 'getDeploymentOptions'
> > 
> 
> > > > > is not the only functionality that accesses the Heat templates
> > 
> 
> > > > > and needs
> > 
> 
> > > > > exposure through an API. And, as has been discussed on a
> > 
> 
> > > > > separate TripleO
> > 
> 
> > > > > thread, we're not even sure Swift is sufficient for our needs;
> > 
> 
> > > > > one possible
> > 
> 
> > > > > consideration right now is allowing deployment from templates
> > 
> 
> > > > > stored in
> > 
> 
> > > > > multiple places, such as the file system or git.
> > 
> 
> > > > 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dougal Matthews]

On 20 January 2016 at 20:05, Tzu-Mainn Chen  wrote:

> - Original Message -
> > On 18.1.2016 19:49, Tzu-Mainn Chen wrote:
> > > - Original Message -
> > >> On Thu, 2016-01-14 at 16:04 -0500, Tzu-Mainn Chen wrote:
> > >>>
> > >>> - Original Message -
> >  On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:
> > > Hey all,
> > >
> > > I realize now from the title of the other TripleO/Mistral thread
> > > [1] that
> > > the discussion there may have gotten confused.  I think using
> > > Mistral for
> > > TripleO processes that are obviously workflows - stack
> > > deployment, node
> > > registration - makes perfect sense.  That thread is exploring
> > > practicalities
> > > for doing that, and I think that's great work.
> > >
> > > What I inappropriately started to address in that thread was a
> > > somewhat
> > > orthogonal point that Dan asked in his original email, namely:
> > >
> > > "what it might look like if we were to use Mistral as a
> > > replacement for the
> > > TripleO API entirely"
> > >
> > > I'd like to create this thread to talk about that; more of a
> > > 'should we'
> > > than 'can we'.  And to do that, I want to indulge in a thought
> > > exercise
> > > stemming from an IRC discussion with Dan and others.  All, please
> > > correct
> > > me
> > > if I've misstated anything.
> > >
> > > The IRC discussion revolved around one use case: deploying a Heat
> > > stack
> > > directly from a Swift container.  With an updated patch, the Heat
> > > CLI can
> > > support this functionality natively.  Then we don't need a
> > > TripleO API; we
> > > can use Mistral to access that functionality, and we're done,
> > > with no need
> > > for additional code within TripleO.  And, as I understand it,
> > > that's the
> > > true motivation for using Mistral instead of a TripleO API:
> > > avoiding custom
> > > code within TripleO.
> > >
> > > That's definitely a worthy goal... except from my perspective,
> > > the story
> > > doesn't quite end there.  A GUI needs additional functionality,
> > > which boils
> > > down to: understanding the Heat deployment templates in order to
> > > provide
> > > options for a user; and persisting those options within a Heat
> > > environment
> > > file.
> > >
> > > Right away I think we hit a problem.  Where does the code for
> > > 'understanding
> > > options' go?  Much of that understanding comes from the
> > > capabilities map
> > > in tripleo-heat-templates [2]; it would make sense to me that
> > > responsibility
> > > for that would fall to a TripleO library.
> > >
> > > Still, perhaps we can limit the amount of TripleO code.  So to
> > > give API
> > > access to 'getDeploymentOptions', we can create a Mistral
> > > workflow.
> > >
> > >Retrieve Heat templates from Swift -> Parse capabilities map
> > >
> > > Which is fine-ish, except from an architectural perspective
> > > 'getDeploymentOptions' violates the abstraction layer between
> > > storage and
> > > business logic, a problem that is compounded because
> > > 'getDeploymentOptions'
> > > is not the only functionality that accesses the Heat templates
> > > and needs
> > > exposure through an API.  And, as has been discussed on a
> > > separate TripleO
> > > thread, we're not even sure Swift is sufficient for our needs;
> > > one possible
> > > consideration right now is allowing deployment from templates
> > > stored in
> > > multiple places, such as the file system or git.
> > 
> >  Actually, that whole capabilities map thing is a workaround for a
> >  missing
> >  feature in Heat, which I have proposed, but am having a hard time
> >  reaching
> >  consensus on within the Heat community:
> > 
> >  https://review.openstack.org/#/c/196656/
> > 
> >  Given that is a large part of what's anticipated to be provided by
> >  the
> >  proposed TripleO API, I'd welcome feedback and collaboration so we
> >  can move
> >  that forward, vs solving only for TripleO.
> > 
> > > Are we going to have duplicate 'getDeploymentOptions' workflows
> > > for each
> > > storage mechanism?  If we consolidate the storage code within a
> > > TripleO
> > > library, do we really need a *workflow* to call a single
> > > function?  Is a
> > > thin TripleO API that contains no additional business logic
> > > really so bad
> > > at that point?
> > 
> >  Actually, this is an argument for making the validation part of the
> >  deployment a workflow - then the interface with the storage
> >  mechanism
> >  becomes more easily pluggable vs baked 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Ryan Brady]

On Thu, Jan 21, 2016 at 10:29 AM, Tzu-Mainn Chen 
wrote:
> 
>
>
>
> On 21 January 2016 at 14:46, Dougal Matthews  wrote:
>>
>>
>>
>> On 20 January 2016 at 20:05, Tzu-Mainn Chen  wrote:
>>>
>>> - Original Message -
>>> > On 18.1.2016 19:49, Tzu-Mainn Chen wrote:
>>> > > - Original Message -
>>> > >> On Thu, 2016-01-14 at 16:04 -0500, Tzu-Mainn Chen wrote:
>>> > >>>
>>> > >>> - Original Message -
>>> >  On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:
>>> > > Hey all,
>>> > >
>>> > > I realize now from the title of the other TripleO/Mistral thread
>>> > > [1] that
>>> > > the discussion there may have gotten confused.  I think using
>>> > > Mistral for
>>> > > TripleO processes that are obviously workflows - stack
>>> > > deployment, node
>>> > > registration - makes perfect sense.  That thread is exploring
>>> > > practicalities
>>> > > for doing that, and I think that's great work.
>>> > >
>>> > > What I inappropriately started to address in that thread was a
>>> > > somewhat
>>> > > orthogonal point that Dan asked in his original email, namely:
>>> > >
>>> > > "what it might look like if we were to use Mistral as a
>>> > > replacement for the
>>> > > TripleO API entirely"
>>> > >
>>> > > I'd like to create this thread to talk about that; more of a
>>> > > 'should we'
>>> > > than 'can we'.  And to do that, I want to indulge in a thought
>>> > > exercise
>>> > > stemming from an IRC discussion with Dan and others.  All,
please
>>> > > correct
>>> > > me
>>> > > if I've misstated anything.
>>> > >
>>> > > The IRC discussion revolved around one use case: deploying a
Heat
>>> > > stack
>>> > > directly from a Swift container.  With an updated patch, the
Heat
>>> > > CLI can
>>> > > support this functionality natively.  Then we don't need a
>>> > > TripleO API; we
>>> > > can use Mistral to access that functionality, and we're done,
>>> > > with no need
>>> > > for additional code within TripleO.  And, as I understand it,
>>> > > that's the
>>> > > true motivation for using Mistral instead of a TripleO API:
>>> > > avoiding custom
>>> > > code within TripleO.
>>> > >
>>> > > That's definitely a worthy goal... except from my perspective,
>>> > > the story
>>> > > doesn't quite end there.  A GUI needs additional functionality,
>>> > > which boils
>>> > > down to: understanding the Heat deployment templates in order to
>>> > > provide
>>> > > options for a user; and persisting those options within a Heat
>>> > > environment
>>> > > file.
>>> > >
>>> > > Right away I think we hit a problem.  Where does the code for
>>> > > 'understanding
>>> > > options' go?  Much of that understanding comes from the
>>> > > capabilities map
>>> > > in tripleo-heat-templates [2]; it would make sense to me that
>>> > > responsibility
>>> > > for that would fall to a TripleO library.
>>> > >
>>> > > Still, perhaps we can limit the amount of TripleO code.  So to
>>> > > give API
>>> > > access to 'getDeploymentOptions', we can create a Mistral
>>> > > workflow.
>>> > >
>>> > >Retrieve Heat templates from Swift -> Parse capabilities map
>>> > >
>>> > > Which is fine-ish, except from an architectural perspective
>>> > > 'getDeploymentOptions' violates the abstraction layer between
>>> > > storage and
>>> > > business logic, a problem that is compounded because
>>> > > 'getDeploymentOptions'
>>> > > is not the only functionality that accesses the Heat templates
>>> > > and needs
>>> > > exposure through an API.  And, as has been discussed on a
>>> > > separate TripleO
>>> > > thread, we're not even sure Swift is sufficient for our needs;
>>> > > one possible
>>> > > consideration right now is allowing deployment from templates
>>> > > stored in
>>> > > multiple places, such as the file system or git.
>>> > 
>>> >  Actually, that whole capabilities map thing is a workaround for a
>>> >  missing
>>> >  feature in Heat, which I have proposed, but am having a hard time
>>> >  reaching
>>> >  consensus on within the Heat community:
>>> > 
>>> >  https://review.openstack.org/#/c/196656/
>>> > 
>>> >  Given that is a large part of what's anticipated to be provided
by
>>> >  the
>>> >  proposed TripleO API, I'd welcome feedback and collaboration so
we
>>> >  can move
>>> >  that forward, vs solving only for TripleO.
>>> > 
>>> > > Are we going to have duplicate 'getDeploymentOptions' workflows
>>> > > for each
>>> > > storage mechanism?  If we consolidate the storage code within a
>>> > > TripleO
>>> > > 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dougal Matthews]

On 20 January 2016 at 10:03, Jiří Stránský  wrote:

> On 18.1.2016 19:49, Tzu-Mainn Chen wrote:
>
>> - Original Message -
>>
>>> On Thu, 2016-01-14 at 16:04 -0500, Tzu-Mainn Chen wrote:
>>>

 - Original Message -

> On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:
>
>> Hey all,
>>
>> I realize now from the title of the other TripleO/Mistral thread
>> [1] that
>> the discussion there may have gotten confused.  I think using
>> Mistral for
>> TripleO processes that are obviously workflows - stack
>> deployment, node
>> registration - makes perfect sense.  That thread is exploring
>> practicalities
>> for doing that, and I think that's great work.
>>
>> What I inappropriately started to address in that thread was a
>> somewhat
>> orthogonal point that Dan asked in his original email, namely:
>>
>> "what it might look like if we were to use Mistral as a
>> replacement for the
>> TripleO API entirely"
>>
>> I'd like to create this thread to talk about that; more of a
>> 'should we'
>> than 'can we'.  And to do that, I want to indulge in a thought
>> exercise
>> stemming from an IRC discussion with Dan and others.  All, please
>> correct
>> me
>> if I've misstated anything.
>>
>> The IRC discussion revolved around one use case: deploying a Heat
>> stack
>> directly from a Swift container.  With an updated patch, the Heat
>> CLI can
>> support this functionality natively.  Then we don't need a
>> TripleO API; we
>> can use Mistral to access that functionality, and we're done,
>> with no need
>> for additional code within TripleO.  And, as I understand it,
>> that's the
>> true motivation for using Mistral instead of a TripleO API:
>> avoiding custom
>> code within TripleO.
>>
>> That's definitely a worthy goal... except from my perspective,
>> the story
>> doesn't quite end there.  A GUI needs additional functionality,
>> which boils
>> down to: understanding the Heat deployment templates in order to
>> provide
>> options for a user; and persisting those options within a Heat
>> environment
>> file.
>>
>> Right away I think we hit a problem.  Where does the code for
>> 'understanding
>> options' go?  Much of that understanding comes from the
>> capabilities map
>> in tripleo-heat-templates [2]; it would make sense to me that
>> responsibility
>> for that would fall to a TripleO library.
>>
>> Still, perhaps we can limit the amount of TripleO code.  So to
>> give API
>> access to 'getDeploymentOptions', we can create a Mistral
>> workflow.
>>
>>Retrieve Heat templates from Swift -> Parse capabilities map
>>
>> Which is fine-ish, except from an architectural perspective
>> 'getDeploymentOptions' violates the abstraction layer between
>> storage and
>> business logic, a problem that is compounded because
>> 'getDeploymentOptions'
>> is not the only functionality that accesses the Heat templates
>> and needs
>> exposure through an API.  And, as has been discussed on a
>> separate TripleO
>> thread, we're not even sure Swift is sufficient for our needs;
>> one possible
>> consideration right now is allowing deployment from templates
>> stored in
>> multiple places, such as the file system or git.
>>
>
> Actually, that whole capabilities map thing is a workaround for a
> missing
> feature in Heat, which I have proposed, but am having a hard time
> reaching
> consensus on within the Heat community:
>
> https://review.openstack.org/#/c/196656/
>
> Given that is a large part of what's anticipated to be provided by
> the
> proposed TripleO API, I'd welcome feedback and collaboration so we
> can move
> that forward, vs solving only for TripleO.
>
> Are we going to have duplicate 'getDeploymentOptions' workflows
>> for each
>> storage mechanism?  If we consolidate the storage code within a
>> TripleO
>> library, do we really need a *workflow* to call a single
>> function?  Is a
>> thin TripleO API that contains no additional business logic
>> really so bad
>> at that point?
>>
>
> Actually, this is an argument for making the validation part of the
> deployment a workflow - then the interface with the storage
> mechanism
> becomes more easily pluggable vs baked into an opaque-to-operators
> API.
>
> E.g, in the long term, imagine the capabilities feature exists in
> Heat, you
> then have a pre-deployment workflow that looks something like:
>
> 1. Retrieve golden templates from a template store
> 2. Pass templates to Heat, get capabilities map which defines
> features user

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Tzu-Mainn Chen]

- Original Message -
> On 18.1.2016 19:49, Tzu-Mainn Chen wrote:
> > - Original Message -
> >> On Thu, 2016-01-14 at 16:04 -0500, Tzu-Mainn Chen wrote:
> >>>
> >>> - Original Message -
>  On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:
> > Hey all,
> >
> > I realize now from the title of the other TripleO/Mistral thread
> > [1] that
> > the discussion there may have gotten confused.  I think using
> > Mistral for
> > TripleO processes that are obviously workflows - stack
> > deployment, node
> > registration - makes perfect sense.  That thread is exploring
> > practicalities
> > for doing that, and I think that's great work.
> >
> > What I inappropriately started to address in that thread was a
> > somewhat
> > orthogonal point that Dan asked in his original email, namely:
> >
> > "what it might look like if we were to use Mistral as a
> > replacement for the
> > TripleO API entirely"
> >
> > I'd like to create this thread to talk about that; more of a
> > 'should we'
> > than 'can we'.  And to do that, I want to indulge in a thought
> > exercise
> > stemming from an IRC discussion with Dan and others.  All, please
> > correct
> > me
> > if I've misstated anything.
> >
> > The IRC discussion revolved around one use case: deploying a Heat
> > stack
> > directly from a Swift container.  With an updated patch, the Heat
> > CLI can
> > support this functionality natively.  Then we don't need a
> > TripleO API; we
> > can use Mistral to access that functionality, and we're done,
> > with no need
> > for additional code within TripleO.  And, as I understand it,
> > that's the
> > true motivation for using Mistral instead of a TripleO API:
> > avoiding custom
> > code within TripleO.
> >
> > That's definitely a worthy goal... except from my perspective,
> > the story
> > doesn't quite end there.  A GUI needs additional functionality,
> > which boils
> > down to: understanding the Heat deployment templates in order to
> > provide
> > options for a user; and persisting those options within a Heat
> > environment
> > file.
> >
> > Right away I think we hit a problem.  Where does the code for
> > 'understanding
> > options' go?  Much of that understanding comes from the
> > capabilities map
> > in tripleo-heat-templates [2]; it would make sense to me that
> > responsibility
> > for that would fall to a TripleO library.
> >
> > Still, perhaps we can limit the amount of TripleO code.  So to
> > give API
> > access to 'getDeploymentOptions', we can create a Mistral
> > workflow.
> >
> >Retrieve Heat templates from Swift -> Parse capabilities map
> >
> > Which is fine-ish, except from an architectural perspective
> > 'getDeploymentOptions' violates the abstraction layer between
> > storage and
> > business logic, a problem that is compounded because
> > 'getDeploymentOptions'
> > is not the only functionality that accesses the Heat templates
> > and needs
> > exposure through an API.  And, as has been discussed on a
> > separate TripleO
> > thread, we're not even sure Swift is sufficient for our needs;
> > one possible
> > consideration right now is allowing deployment from templates
> > stored in
> > multiple places, such as the file system or git.
> 
>  Actually, that whole capabilities map thing is a workaround for a
>  missing
>  feature in Heat, which I have proposed, but am having a hard time
>  reaching
>  consensus on within the Heat community:
> 
>  https://review.openstack.org/#/c/196656/
> 
>  Given that is a large part of what's anticipated to be provided by
>  the
>  proposed TripleO API, I'd welcome feedback and collaboration so we
>  can move
>  that forward, vs solving only for TripleO.
> 
> > Are we going to have duplicate 'getDeploymentOptions' workflows
> > for each
> > storage mechanism?  If we consolidate the storage code within a
> > TripleO
> > library, do we really need a *workflow* to call a single
> > function?  Is a
> > thin TripleO API that contains no additional business logic
> > really so bad
> > at that point?
> 
>  Actually, this is an argument for making the validation part of the
>  deployment a workflow - then the interface with the storage
>  mechanism
>  becomes more easily pluggable vs baked into an opaque-to-operators
>  API.
> 
>  E.g, in the long term, imagine the capabilities feature exists in
>  Heat, you
>  then have a pre-deployment workflow that looks something like:
> 
>  1. Retrieve golden templates from a template store
>  2. Pass templates to Heat, get 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Jiří Stránský]

On 18.1.2016 19:49, Tzu-Mainn Chen wrote:

- Original Message -

On Thu, 2016-01-14 at 16:04 -0500, Tzu-Mainn Chen wrote:


- Original Message -

On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:

Hey all,

I realize now from the title of the other TripleO/Mistral thread
[1] that
the discussion there may have gotten confused.  I think using
Mistral for
TripleO processes that are obviously workflows - stack
deployment, node
registration - makes perfect sense.  That thread is exploring
practicalities
for doing that, and I think that's great work.

What I inappropriately started to address in that thread was a
somewhat
orthogonal point that Dan asked in his original email, namely:

"what it might look like if we were to use Mistral as a
replacement for the
TripleO API entirely"

I'd like to create this thread to talk about that; more of a
'should we'
than 'can we'.  And to do that, I want to indulge in a thought
exercise
stemming from an IRC discussion with Dan and others.  All, please
correct
me
if I've misstated anything.

The IRC discussion revolved around one use case: deploying a Heat
stack
directly from a Swift container.  With an updated patch, the Heat
CLI can
support this functionality natively.  Then we don't need a
TripleO API; we
can use Mistral to access that functionality, and we're done,
with no need
for additional code within TripleO.  And, as I understand it,
that's the
true motivation for using Mistral instead of a TripleO API:
avoiding custom
code within TripleO.

That's definitely a worthy goal... except from my perspective,
the story
doesn't quite end there.  A GUI needs additional functionality,
which boils
down to: understanding the Heat deployment templates in order to
provide
options for a user; and persisting those options within a Heat
environment
file.

Right away I think we hit a problem.  Where does the code for
'understanding
options' go?  Much of that understanding comes from the
capabilities map
in tripleo-heat-templates [2]; it would make sense to me that
responsibility
for that would fall to a TripleO library.

Still, perhaps we can limit the amount of TripleO code.  So to
give API
access to 'getDeploymentOptions', we can create a Mistral
workflow.

   Retrieve Heat templates from Swift -> Parse capabilities map

Which is fine-ish, except from an architectural perspective
'getDeploymentOptions' violates the abstraction layer between
storage and
business logic, a problem that is compounded because
'getDeploymentOptions'
is not the only functionality that accesses the Heat templates
and needs
exposure through an API.  And, as has been discussed on a
separate TripleO
thread, we're not even sure Swift is sufficient for our needs;
one possible
consideration right now is allowing deployment from templates
stored in
multiple places, such as the file system or git.


Actually, that whole capabilities map thing is a workaround for a
missing
feature in Heat, which I have proposed, but am having a hard time
reaching
consensus on within the Heat community:

https://review.openstack.org/#/c/196656/

Given that is a large part of what's anticipated to be provided by
the
proposed TripleO API, I'd welcome feedback and collaboration so we
can move
that forward, vs solving only for TripleO.


Are we going to have duplicate 'getDeploymentOptions' workflows
for each
storage mechanism?  If we consolidate the storage code within a
TripleO
library, do we really need a *workflow* to call a single
function?  Is a
thin TripleO API that contains no additional business logic
really so bad
at that point?


Actually, this is an argument for making the validation part of the
deployment a workflow - then the interface with the storage
mechanism
becomes more easily pluggable vs baked into an opaque-to-operators
API.

E.g, in the long term, imagine the capabilities feature exists in
Heat, you
then have a pre-deployment workflow that looks something like:

1. Retrieve golden templates from a template store
2. Pass templates to Heat, get capabilities map which defines
features user
must/may select.
3. Prompt user for input to select required capabilites
4. Pass user input to Heat, validate the configuration, get a
mapping of
required options for the selected capabilities (nested validation)
5. Push the validated pieces ("plan" in TripleO API terminology) to
a
template store

This is a pre-deployment validation workflow, and it's a superset
of the
getDeploymentOptions feature you refer to.

Historically, TripleO has had a major gap wrt workflow, meaning
that we've
always implemented it either via shell scripts (tripleo-incubator)
or
python code (tripleo-common/tripleo-client, potentially TripleO
API).

So I think what Dan is exploring is, how do we avoid reimplementing
a
workflow engine, when a project exists which already does that.


My gut reaction is to say that proposing Mistral in place of a
TripleO API
is to look at the engineering concerns from the wrong
direction.  The

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Tzu-Mainn Chen]

- Original Message -
> On Thu, 2016-01-14 at 16:04 -0500, Tzu-Mainn Chen wrote:
> > 
> > - Original Message -
> > > On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:
> > > > Hey all,
> > > > 
> > > > I realize now from the title of the other TripleO/Mistral thread
> > > > [1] that
> > > > the discussion there may have gotten confused.  I think using
> > > > Mistral for
> > > > TripleO processes that are obviously workflows - stack
> > > > deployment, node
> > > > registration - makes perfect sense.  That thread is exploring
> > > > practicalities
> > > > for doing that, and I think that's great work.
> > > > 
> > > > What I inappropriately started to address in that thread was a
> > > > somewhat
> > > > orthogonal point that Dan asked in his original email, namely:
> > > > 
> > > > "what it might look like if we were to use Mistral as a
> > > > replacement for the
> > > > TripleO API entirely"
> > > > 
> > > > I'd like to create this thread to talk about that; more of a
> > > > 'should we'
> > > > than 'can we'.  And to do that, I want to indulge in a thought
> > > > exercise
> > > > stemming from an IRC discussion with Dan and others.  All, please
> > > > correct
> > > > me
> > > > if I've misstated anything.
> > > > 
> > > > The IRC discussion revolved around one use case: deploying a Heat
> > > > stack
> > > > directly from a Swift container.  With an updated patch, the Heat
> > > > CLI can
> > > > support this functionality natively.  Then we don't need a
> > > > TripleO API; we
> > > > can use Mistral to access that functionality, and we're done,
> > > > with no need
> > > > for additional code within TripleO.  And, as I understand it,
> > > > that's the
> > > > true motivation for using Mistral instead of a TripleO API:
> > > > avoiding custom
> > > > code within TripleO.
> > > > 
> > > > That's definitely a worthy goal... except from my perspective,
> > > > the story
> > > > doesn't quite end there.  A GUI needs additional functionality,
> > > > which boils
> > > > down to: understanding the Heat deployment templates in order to
> > > > provide
> > > > options for a user; and persisting those options within a Heat
> > > > environment
> > > > file.
> > > > 
> > > > Right away I think we hit a problem.  Where does the code for
> > > > 'understanding
> > > > options' go?  Much of that understanding comes from the
> > > > capabilities map
> > > > in tripleo-heat-templates [2]; it would make sense to me that
> > > > responsibility
> > > > for that would fall to a TripleO library.
> > > > 
> > > > Still, perhaps we can limit the amount of TripleO code.  So to
> > > > give API
> > > > access to 'getDeploymentOptions', we can create a Mistral
> > > > workflow.
> > > > 
> > > >   Retrieve Heat templates from Swift -> Parse capabilities map
> > > > 
> > > > Which is fine-ish, except from an architectural perspective
> > > > 'getDeploymentOptions' violates the abstraction layer between
> > > > storage and
> > > > business logic, a problem that is compounded because
> > > > 'getDeploymentOptions'
> > > > is not the only functionality that accesses the Heat templates
> > > > and needs
> > > > exposure through an API.  And, as has been discussed on a
> > > > separate TripleO
> > > > thread, we're not even sure Swift is sufficient for our needs;
> > > > one possible
> > > > consideration right now is allowing deployment from templates
> > > > stored in
> > > > multiple places, such as the file system or git.
> > > 
> > > Actually, that whole capabilities map thing is a workaround for a
> > > missing
> > > feature in Heat, which I have proposed, but am having a hard time
> > > reaching
> > > consensus on within the Heat community:
> > > 
> > > https://review.openstack.org/#/c/196656/
> > > 
> > > Given that is a large part of what's anticipated to be provided by
> > > the
> > > proposed TripleO API, I'd welcome feedback and collaboration so we
> > > can move
> > > that forward, vs solving only for TripleO.
> > > 
> > > > Are we going to have duplicate 'getDeploymentOptions' workflows
> > > > for each
> > > > storage mechanism?  If we consolidate the storage code within a
> > > > TripleO
> > > > library, do we really need a *workflow* to call a single
> > > > function?  Is a
> > > > thin TripleO API that contains no additional business logic
> > > > really so bad
> > > > at that point?
> > > 
> > > Actually, this is an argument for making the validation part of the
> > > deployment a workflow - then the interface with the storage
> > > mechanism
> > > becomes more easily pluggable vs baked into an opaque-to-operators
> > > API.
> > > 
> > > E.g, in the long term, imagine the capabilities feature exists in
> > > Heat, you
> > > then have a pre-deployment workflow that looks something like:
> > > 
> > > 1. Retrieve golden templates from a template store
> > > 2. Pass templates to Heat, get capabilities map which defines
> > > features user
> > > must/may select.
> > > 3. Prompt user for 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Thu, 2016-01-14 at 16:04 -0500, Tzu-Mainn Chen wrote:
> 
> - Original Message -
> > On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:
> > > Hey all,
> > > 
> > > I realize now from the title of the other TripleO/Mistral thread
> > > [1] that
> > > the discussion there may have gotten confused.  I think using
> > > Mistral for
> > > TripleO processes that are obviously workflows - stack
> > > deployment, node
> > > registration - makes perfect sense.  That thread is exploring
> > > practicalities
> > > for doing that, and I think that's great work.
> > > 
> > > What I inappropriately started to address in that thread was a
> > > somewhat
> > > orthogonal point that Dan asked in his original email, namely:
> > > 
> > > "what it might look like if we were to use Mistral as a
> > > replacement for the
> > > TripleO API entirely"
> > > 
> > > I'd like to create this thread to talk about that; more of a
> > > 'should we'
> > > than 'can we'.  And to do that, I want to indulge in a thought
> > > exercise
> > > stemming from an IRC discussion with Dan and others.  All, please
> > > correct
> > > me
> > > if I've misstated anything.
> > > 
> > > The IRC discussion revolved around one use case: deploying a Heat
> > > stack
> > > directly from a Swift container.  With an updated patch, the Heat
> > > CLI can
> > > support this functionality natively.  Then we don't need a
> > > TripleO API; we
> > > can use Mistral to access that functionality, and we're done,
> > > with no need
> > > for additional code within TripleO.  And, as I understand it,
> > > that's the
> > > true motivation for using Mistral instead of a TripleO API:
> > > avoiding custom
> > > code within TripleO.
> > > 
> > > That's definitely a worthy goal... except from my perspective,
> > > the story
> > > doesn't quite end there.  A GUI needs additional functionality,
> > > which boils
> > > down to: understanding the Heat deployment templates in order to
> > > provide
> > > options for a user; and persisting those options within a Heat
> > > environment
> > > file.
> > > 
> > > Right away I think we hit a problem.  Where does the code for
> > > 'understanding
> > > options' go?  Much of that understanding comes from the
> > > capabilities map
> > > in tripleo-heat-templates [2]; it would make sense to me that
> > > responsibility
> > > for that would fall to a TripleO library.
> > > 
> > > Still, perhaps we can limit the amount of TripleO code.  So to
> > > give API
> > > access to 'getDeploymentOptions', we can create a Mistral
> > > workflow.
> > > 
> > >   Retrieve Heat templates from Swift -> Parse capabilities map
> > > 
> > > Which is fine-ish, except from an architectural perspective
> > > 'getDeploymentOptions' violates the abstraction layer between
> > > storage and
> > > business logic, a problem that is compounded because
> > > 'getDeploymentOptions'
> > > is not the only functionality that accesses the Heat templates
> > > and needs
> > > exposure through an API.  And, as has been discussed on a
> > > separate TripleO
> > > thread, we're not even sure Swift is sufficient for our needs;
> > > one possible
> > > consideration right now is allowing deployment from templates
> > > stored in
> > > multiple places, such as the file system or git.
> > 
> > Actually, that whole capabilities map thing is a workaround for a
> > missing
> > feature in Heat, which I have proposed, but am having a hard time
> > reaching
> > consensus on within the Heat community:
> > 
> > https://review.openstack.org/#/c/196656/
> > 
> > Given that is a large part of what's anticipated to be provided by
> > the
> > proposed TripleO API, I'd welcome feedback and collaboration so we
> > can move
> > that forward, vs solving only for TripleO.
> > 
> > > Are we going to have duplicate 'getDeploymentOptions' workflows
> > > for each
> > > storage mechanism?  If we consolidate the storage code within a
> > > TripleO
> > > library, do we really need a *workflow* to call a single
> > > function?  Is a
> > > thin TripleO API that contains no additional business logic
> > > really so bad
> > > at that point?
> > 
> > Actually, this is an argument for making the validation part of the
> > deployment a workflow - then the interface with the storage
> > mechanism
> > becomes more easily pluggable vs baked into an opaque-to-operators
> > API.
> > 
> > E.g, in the long term, imagine the capabilities feature exists in
> > Heat, you
> > then have a pre-deployment workflow that looks something like:
> > 
> > 1. Retrieve golden templates from a template store
> > 2. Pass templates to Heat, get capabilities map which defines
> > features user
> > must/may select.
> > 3. Prompt user for input to select required capabilites
> > 4. Pass user input to Heat, validate the configuration, get a
> > mapping of
> > required options for the selected capabilities (nested validation)
> > 5. Push the validated pieces ("plan" in TripleO API terminology) to
> > a
> > template 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Steven Hardy]

On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:
> Hey all,
> 
> I realize now from the title of the other TripleO/Mistral thread [1] that
> the discussion there may have gotten confused.  I think using Mistral for
> TripleO processes that are obviously workflows - stack deployment, node
> registration - makes perfect sense.  That thread is exploring practicalities
> for doing that, and I think that's great work.
> 
> What I inappropriately started to address in that thread was a somewhat
> orthogonal point that Dan asked in his original email, namely:
> 
> "what it might look like if we were to use Mistral as a replacement for the
> TripleO API entirely"
> 
> I'd like to create this thread to talk about that; more of a 'should we'
> than 'can we'.  And to do that, I want to indulge in a thought exercise
> stemming from an IRC discussion with Dan and others.  All, please correct me
> if I've misstated anything.
> 
> The IRC discussion revolved around one use case: deploying a Heat stack
> directly from a Swift container.  With an updated patch, the Heat CLI can
> support this functionality natively.  Then we don't need a TripleO API; we
> can use Mistral to access that functionality, and we're done, with no need
> for additional code within TripleO.  And, as I understand it, that's the
> true motivation for using Mistral instead of a TripleO API: avoiding custom
> code within TripleO.
> 
> That's definitely a worthy goal... except from my perspective, the story
> doesn't quite end there.  A GUI needs additional functionality, which boils
> down to: understanding the Heat deployment templates in order to provide
> options for a user; and persisting those options within a Heat environment
> file.
> 
> Right away I think we hit a problem.  Where does the code for 'understanding
> options' go?  Much of that understanding comes from the capabilities map
> in tripleo-heat-templates [2]; it would make sense to me that responsibility
> for that would fall to a TripleO library.
> 
> Still, perhaps we can limit the amount of TripleO code.  So to give API
> access to 'getDeploymentOptions', we can create a Mistral workflow.
> 
>   Retrieve Heat templates from Swift -> Parse capabilities map
> 
> Which is fine-ish, except from an architectural perspective
> 'getDeploymentOptions' violates the abstraction layer between storage and
> business logic, a problem that is compounded because 'getDeploymentOptions'
> is not the only functionality that accesses the Heat templates and needs
> exposure through an API.  And, as has been discussed on a separate TripleO
> thread, we're not even sure Swift is sufficient for our needs; one possible
> consideration right now is allowing deployment from templates stored in
> multiple places, such as the file system or git.

Actually, that whole capabilities map thing is a workaround for a missing
feature in Heat, which I have proposed, but am having a hard time reaching
consensus on within the Heat community:

https://review.openstack.org/#/c/196656/

Given that is a large part of what's anticipated to be provided by the
proposed TripleO API, I'd welcome feedback and collaboration so we can move
that forward, vs solving only for TripleO.

> Are we going to have duplicate 'getDeploymentOptions' workflows for each
> storage mechanism?  If we consolidate the storage code within a TripleO
> library, do we really need a *workflow* to call a single function?  Is a
> thin TripleO API that contains no additional business logic really so bad
> at that point?

Actually, this is an argument for making the validation part of the
deployment a workflow - then the interface with the storage mechanism
becomes more easily pluggable vs baked into an opaque-to-operators API.

E.g, in the long term, imagine the capabilities feature exists in Heat, you
then have a pre-deployment workflow that looks something like:

1. Retrieve golden templates from a template store
2. Pass templates to Heat, get capabilities map which defines features user
must/may select.
3. Prompt user for input to select required capabilites
4. Pass user input to Heat, validate the configuration, get a mapping of
required options for the selected capabilities (nested validation)
5. Push the validated pieces ("plan" in TripleO API terminology) to a
template store

This is a pre-deployment validation workflow, and it's a superset of the
getDeploymentOptions feature you refer to.

Historically, TripleO has had a major gap wrt workflow, meaning that we've
always implemented it either via shell scripts (tripleo-incubator) or
python code (tripleo-common/tripleo-client, potentially TripleO API).

So I think what Dan is exploring is, how do we avoid reimplementing a
workflow engine, when a project exists which already does that.

> My gut reaction is to say that proposing Mistral in place of a TripleO API
> is to look at the engineering concerns from the wrong direction.  The
> Mistral alternative comes from a desire to limit 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Tzu-Mainn Chen]

- Original Message -
> On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:
> > Hey all,
> > 
> > I realize now from the title of the other TripleO/Mistral thread [1] that
> > the discussion there may have gotten confused.  I think using Mistral for
> > TripleO processes that are obviously workflows - stack deployment, node
> > registration - makes perfect sense.  That thread is exploring
> > practicalities
> > for doing that, and I think that's great work.
> > 
> > What I inappropriately started to address in that thread was a somewhat
> > orthogonal point that Dan asked in his original email, namely:
> > 
> > "what it might look like if we were to use Mistral as a replacement for the
> > TripleO API entirely"
> > 
> > I'd like to create this thread to talk about that; more of a 'should we'
> > than 'can we'.  And to do that, I want to indulge in a thought exercise
> > stemming from an IRC discussion with Dan and others.  All, please correct
> > me
> > if I've misstated anything.
> > 
> > The IRC discussion revolved around one use case: deploying a Heat stack
> > directly from a Swift container.  With an updated patch, the Heat CLI can
> > support this functionality natively.  Then we don't need a TripleO API; we
> > can use Mistral to access that functionality, and we're done, with no need
> > for additional code within TripleO.  And, as I understand it, that's the
> > true motivation for using Mistral instead of a TripleO API: avoiding custom
> > code within TripleO.
> > 
> > That's definitely a worthy goal... except from my perspective, the story
> > doesn't quite end there.  A GUI needs additional functionality, which boils
> > down to: understanding the Heat deployment templates in order to provide
> > options for a user; and persisting those options within a Heat environment
> > file.
> > 
> > Right away I think we hit a problem.  Where does the code for
> > 'understanding
> > options' go?  Much of that understanding comes from the capabilities map
> > in tripleo-heat-templates [2]; it would make sense to me that
> > responsibility
> > for that would fall to a TripleO library.
> > 
> > Still, perhaps we can limit the amount of TripleO code.  So to give API
> > access to 'getDeploymentOptions', we can create a Mistral workflow.
> > 
> >   Retrieve Heat templates from Swift -> Parse capabilities map
> > 
> > Which is fine-ish, except from an architectural perspective
> > 'getDeploymentOptions' violates the abstraction layer between storage and
> > business logic, a problem that is compounded because 'getDeploymentOptions'
> > is not the only functionality that accesses the Heat templates and needs
> > exposure through an API.  And, as has been discussed on a separate TripleO
> > thread, we're not even sure Swift is sufficient for our needs; one possible
> > consideration right now is allowing deployment from templates stored in
> > multiple places, such as the file system or git.
> 
> Actually, that whole capabilities map thing is a workaround for a missing
> feature in Heat, which I have proposed, but am having a hard time reaching
> consensus on within the Heat community:
> 
> https://review.openstack.org/#/c/196656/
> 
> Given that is a large part of what's anticipated to be provided by the
> proposed TripleO API, I'd welcome feedback and collaboration so we can move
> that forward, vs solving only for TripleO.
> 
> > Are we going to have duplicate 'getDeploymentOptions' workflows for each
> > storage mechanism?  If we consolidate the storage code within a TripleO
> > library, do we really need a *workflow* to call a single function?  Is a
> > thin TripleO API that contains no additional business logic really so bad
> > at that point?
> 
> Actually, this is an argument for making the validation part of the
> deployment a workflow - then the interface with the storage mechanism
> becomes more easily pluggable vs baked into an opaque-to-operators API.
> 
> E.g, in the long term, imagine the capabilities feature exists in Heat, you
> then have a pre-deployment workflow that looks something like:
> 
> 1. Retrieve golden templates from a template store
> 2. Pass templates to Heat, get capabilities map which defines features user
> must/may select.
> 3. Prompt user for input to select required capabilites
> 4. Pass user input to Heat, validate the configuration, get a mapping of
> required options for the selected capabilities (nested validation)
> 5. Push the validated pieces ("plan" in TripleO API terminology) to a
> template store
> 
> This is a pre-deployment validation workflow, and it's a superset of the
> getDeploymentOptions feature you refer to.
> 
> Historically, TripleO has had a major gap wrt workflow, meaning that we've
> always implemented it either via shell scripts (tripleo-incubator) or
> python code (tripleo-common/tripleo-client, potentially TripleO API).
> 
> So I think what Dan is exploring is, how do we avoid reimplementing a
> workflow engine, when a project 

[openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Tzu-Mainn Chen]

Hey all,

I realize now from the title of the other TripleO/Mistral thread [1] that
the discussion there may have gotten confused.  I think using Mistral for
TripleO processes that are obviously workflows - stack deployment, node
registration - makes perfect sense.  That thread is exploring practicalities
for doing that, and I think that's great work.

What I inappropriately started to address in that thread was a somewhat
orthogonal point that Dan asked in his original email, namely:

"what it might look like if we were to use Mistral as a replacement for the
TripleO API entirely"

I'd like to create this thread to talk about that; more of a 'should we'
than 'can we'.  And to do that, I want to indulge in a thought exercise
stemming from an IRC discussion with Dan and others.  All, please correct me
if I've misstated anything.

The IRC discussion revolved around one use case: deploying a Heat stack
directly from a Swift container.  With an updated patch, the Heat CLI can
support this functionality natively.  Then we don't need a TripleO API; we
can use Mistral to access that functionality, and we're done, with no need
for additional code within TripleO.  And, as I understand it, that's the
true motivation for using Mistral instead of a TripleO API: avoiding custom
code within TripleO.

That's definitely a worthy goal... except from my perspective, the story
doesn't quite end there.  A GUI needs additional functionality, which boils
down to: understanding the Heat deployment templates in order to provide
options for a user; and persisting those options within a Heat environment
file.

Right away I think we hit a problem.  Where does the code for 'understanding
options' go?  Much of that understanding comes from the capabilities map
in tripleo-heat-templates [2]; it would make sense to me that responsibility
for that would fall to a TripleO library.

Still, perhaps we can limit the amount of TripleO code.  So to give API
access to 'getDeploymentOptions', we can create a Mistral workflow.

  Retrieve Heat templates from Swift -> Parse capabilities map

Which is fine-ish, except from an architectural perspective
'getDeploymentOptions' violates the abstraction layer between storage and
business logic, a problem that is compounded because 'getDeploymentOptions'
is not the only functionality that accesses the Heat templates and needs
exposure through an API.  And, as has been discussed on a separate TripleO
thread, we're not even sure Swift is sufficient for our needs; one possible
consideration right now is allowing deployment from templates stored in
multiple places, such as the file system or git.

Are we going to have duplicate 'getDeploymentOptions' workflows for each
storage mechanism?  If we consolidate the storage code within a TripleO
library, do we really need a *workflow* to call a single function?  Is a
thin TripleO API that contains no additional business logic really so bad
at that point?

My gut reaction is to say that proposing Mistral in place of a TripleO API
is to look at the engineering concerns from the wrong direction.  The
Mistral alternative comes from a desire to limit custom TripleO code at all
costs.  I think that is an extremely dangerous attitude that leads to
compromises and workarounds that will quickly lead to a shaky code base
full of design flaws that make it difficult to implement or extend any
functionality cleanly.

I think the correct attitude is to simply look at the problem we're
trying to solve and find the correct architecture.  For these get/set
methods that the API needs, it's pretty simple: storage -> some logic ->
a REST API.  Adding a workflow engine on top of that is unneeded, and I
believe that means it's an incorrect solution.


Thanks,
Tzu-Mainn Chen



[1] http://lists.openstack.org/pipermail/openstack-dev/2016-January/083757.html
[2] 
https://github.com/openstack/tripleo-heat-templates/blob/master/capabilities_map.yaml

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Tzu-Mainn Chen]

- Original Message -
> On Wed, 2016-01-13 at 04:41 -0500, Tzu-Mainn Chen wrote:
> > Hey all,
> > 
> > I realize now from the title of the other TripleO/Mistral thread [1]
> > that
> > the discussion there may have gotten confused.  I think using Mistral
> > for
> > TripleO processes that are obviously workflows - stack deployment,
> > node
> > registration - makes perfect sense.  That thread is exploring
> > practicalities
> > for doing that, and I think that's great work.
> > 
> > What I inappropriately started to address in that thread was a
> > somewhat
> > orthogonal point that Dan asked in his original email, namely:
> > 
> > "what it might look like if we were to use Mistral as a replacement
> > for the
> > TripleO API entirely"
> > 
> > I'd like to create this thread to talk about that; more of a 'should
> > we'
> > than 'can we'.  And to do that, I want to indulge in a thought
> > exercise
> > stemming from an IRC discussion with Dan and others.  All, please
> > correct me
> > if I've misstated anything.
> > 
> > The IRC discussion revolved around one use case: deploying a Heat
> > stack
> > directly from a Swift container.  With an updated patch, the Heat CLI
> > can
> > support this functionality natively.  Then we don't need a TripleO
> > API; we
> > can use Mistral to access that functionality, and we're done, with no
> > need
> > for additional code within TripleO.  And, as I understand it, that's
> > the
> > true motivation for using Mistral instead of a TripleO API: avoiding
> > custom
> > code within TripleO.
> 
> The True motivation for investigating Mistral was to counter the
> assertion that we needed to build our own REST API for workflows in the
> TripleO API spec. This:
> 
>  "We need a REST API that supports the overcloud deployment workflow."
> 
> https://review.openstack.org/#/c/230432/13/specs/mitaka/tripleo-overclo
> ud-deployment-api.rst
> 
> In doing that I was trying to wrap some of the existing code in
> tripleo-common in Mistral actions and it occurred to me that some
> things (like the ability to deploy heat templates from Swift) might
> benefit others more if we put them in the respective client libraries
> (like heatclient) instead of carrying them in tripleo-common where they
> only benefit us. Especially since in the heatclient case it already had
> a --template-object option to begin with (https://bugs.launchpad.net/py
> thon-heatclient/+bug/1532326)
> 
> This goes towards a similar pattern we follow with other components
> like Puppet which is that instead of trying to create our own
> functionality in puppet-tripleo we are trying to as much as possible
> add the features we need to the individual modules for each project
> (puppet-nova, puppet-swift, etc).
> 
> In other words avoiding custom code within TripleO is just good
> practice in general, not something that is specific to the Mistral vs
> TripleO API discussion. When someone looks at TripleO as a project I
> would very much like them to admire our architecture... not what we've
> had to build to support it. As an example I'm very glad to see TripleO
> be a bit more friendly to config management tooling... rather than
> trying to build our own version (os-apply-config, etc.). And adding
> code to the right place usually works out better for everyone and can
> help build up the larger OpenStack community too.
> 
> > 
> > That's definitely a worthy goal... except from my perspective, the
> > story
> > doesn't quite end there.  A GUI needs additional functionality, which
> > boils
> > down to: understanding the Heat deployment templates in order to
> > provide
> > options for a user; and persisting those options within a Heat
> > environment
> > file.
> 
> TripleO API was previously described as what would be our "workflows
> API" for TripleO. A places for common workflows that are used for the
> CLI, and UI together (one code path for all!).
> 
> What you describe here sounds like a different sort of idea which is a
> GUI helper sort of API. FWIW I think it is totally fine for a GUI to
> maintain its own API for caching reasons etc if the members of that
> team find it to be a requirement. Please don't feel that any of the
> workflow API comparisons block requirements for things to build a GUI
> though.
> 
> 
> > 
> > Right away I think we hit a problem.  Where does the code for
> > 'understanding
> > options' go?  Much of that understanding comes from the capabilities
> > map
> > in tripleo-heat-templates [2]; it would make sense to me that
> > responsibility
> > for that would fall to a TripleO library.
> > 
> > Still, perhaps we can limit the amount of TripleO code.  So to give
> > API
> > access to 'getDeploymentOptions', we can create a Mistral workflow.
> > 
> >   Retrieve Heat templates from Swift -> Parse capabilities map
> 
> I have no issues with this workflow if it helps the UI team to create
> say a cached copy of the capabilities map.
> 
> Similar to the above heatclient discussion I would 

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Dan Prince]

On Wed, 2016-01-13 at 04:41 -0500, Tzu-Mainn Chen wrote:
> Hey all,
> 
> I realize now from the title of the other TripleO/Mistral thread [1]
> that
> the discussion there may have gotten confused.  I think using Mistral
> for
> TripleO processes that are obviously workflows - stack deployment,
> node
> registration - makes perfect sense.  That thread is exploring
> practicalities
> for doing that, and I think that's great work.
> 
> What I inappropriately started to address in that thread was a
> somewhat
> orthogonal point that Dan asked in his original email, namely:
> 
> "what it might look like if we were to use Mistral as a replacement
> for the
> TripleO API entirely"
> 
> I'd like to create this thread to talk about that; more of a 'should
> we'
> than 'can we'.  And to do that, I want to indulge in a thought
> exercise
> stemming from an IRC discussion with Dan and others.  All, please
> correct me
> if I've misstated anything.
> 
> The IRC discussion revolved around one use case: deploying a Heat
> stack
> directly from a Swift container.  With an updated patch, the Heat CLI
> can
> support this functionality natively.  Then we don't need a TripleO
> API; we
> can use Mistral to access that functionality, and we're done, with no
> need
> for additional code within TripleO.  And, as I understand it, that's
> the
> true motivation for using Mistral instead of a TripleO API: avoiding
> custom
> code within TripleO.

The True motivation for investigating Mistral was to counter the
assertion that we needed to build our own REST API for workflows in the
TripleO API spec. This:

 "We need a REST API that supports the overcloud deployment workflow."

https://review.openstack.org/#/c/230432/13/specs/mitaka/tripleo-overclo
ud-deployment-api.rst

In doing that I was trying to wrap some of the existing code in
tripleo-common in Mistral actions and it occurred to me that some
things (like the ability to deploy heat templates from Swift) might
benefit others more if we put them in the respective client libraries
(like heatclient) instead of carrying them in tripleo-common where they
only benefit us. Especially since in the heatclient case it already had
a --template-object option to begin with (https://bugs.launchpad.net/py
thon-heatclient/+bug/1532326)

This goes towards a similar pattern we follow with other components
like Puppet which is that instead of trying to create our own
functionality in puppet-tripleo we are trying to as much as possible
add the features we need to the individual modules for each project
(puppet-nova, puppet-swift, etc).

In other words avoiding custom code within TripleO is just good
practice in general, not something that is specific to the Mistral vs
TripleO API discussion. When someone looks at TripleO as a project I
would very much like them to admire our architecture... not what we've
had to build to support it. As an example I'm very glad to see TripleO
be a bit more friendly to config management tooling... rather than
trying to build our own version (os-apply-config, etc.). And adding
code to the right place usually works out better for everyone and can
help build up the larger OpenStack community too.

> 
> That's definitely a worthy goal... except from my perspective, the
> story
> doesn't quite end there.  A GUI needs additional functionality, which
> boils
> down to: understanding the Heat deployment templates in order to
> provide
> options for a user; and persisting those options within a Heat
> environment
> file.

TripleO API was previously described as what would be our "workflows
API" for TripleO. A places for common workflows that are used for the
CLI, and UI together (one code path for all!).

What you describe here sounds like a different sort of idea which is a
GUI helper sort of API. FWIW I think it is totally fine for a GUI to
maintain its own API for caching reasons etc if the members of that
team find it to be a requirement. Please don't feel that any of the
workflow API comparisons block requirements for things to build a GUI
though.


> 
> Right away I think we hit a problem.  Where does the code for
> 'understanding
> options' go?  Much of that understanding comes from the capabilities
> map
> in tripleo-heat-templates [2]; it would make sense to me that
> responsibility
> for that would fall to a TripleO library.
> 
> Still, perhaps we can limit the amount of TripleO code.  So to give
> API
> access to 'getDeploymentOptions', we can create a Mistral workflow.
> 
>   Retrieve Heat templates from Swift -> Parse capabilities map

I have no issues with this workflow if it helps the UI team to create
say a cached copy of the capabilities map.

Similar to the above heatclient discussion I would ask why put this
into a TripleO library if we don't have to. Could this code perhaps be
made into a more generic Heat feature? If not immediately, then perhaps
eventually? What I mean is... it would be totally fine to write this
code in say TripleO common with an eye