More inline.

From: Dmitri Zimine <<>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
Date: Tuesday, April 1, 2014 at 2:59 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
Subject: Re: [openstack-dev] [Mistral][TaskFlow] Long running actions

On Apr 1, 2014, at 3:43 AM, Renat Akhmerov 
<<>> wrote:
On 25 Mar 2014, at 01:51, Joshua Harlow 
<<>> wrote:

The first execution model I would call the local execution model, this model 
involves forming tasks and flows and then executing them inside an application, 
that application is running for the duration of the workflow (although if it 
crashes it can re-establish the task and flows that it was doing and attempt to 
resume them). This could also be what openstack projects would call the 
'conductor' approach where nova, ironic, trove have a conductor which manages 
these long-running actions (the conductor is alive/running throughout the 
duration of these workflows, although it may be restarted while running). The 
restarting + resuming part is something that openstack hasn't handled so 
gracefully currently, typically requiring either some type of cleanup at 
restart (or by operations), with taskflow using this model the resumption part 
makes it possible to resume from the last saved state (this connects into the 
persistence model that taskflow uses, the state transitions, how execution 
occurrs itself...).

The second execution model is an extension of the first, whereby there is still 
a type of 'conductor' that is managing the life-time of the workflow, but 
instead of locally executing tasks in the conductor itself tasks are now 
executed on remote-workers (see
). The engine currently still is 'alive' for the life-time of the execution, 
although the work that it is doing is relatively minimal (since its not 
actually executing any task code, but proxying those requests to others works). 
The engine while running does the conducting of the remote-workers (saving 
persistence details, doing state-transtions, getting results, sending requests 
to workers…).

These two execution models are special cases of what you call “lazy execution 
model” (or passive as we call it). To illustrate this idea we can take a look 
at the first sequence diagram at [0], we basically will see the following 

1) engine --(task)--> queue --(task)--> worker
2) execute task
3) worker --(result)--> queue --(result)--> engine

This is how TaskFlow worker based model works.

If we loosen the requirement in 3) and assume that not only worker can send a 
task result back to engine we’ll get our passive model. Instead of worker it 
can be anything else (some external system) that knows how to make this call. A 
particular way is not too important, it can be a direct message or it can be 
hidden behind an API method. In Mistral it’s now a REST API method however 
we’re about to decouple engine from REST API so that engine is a standalone 
process and listens to a queue. So worker-based model is basically the same 
with the only strict requirement that only worker sends a result back.

In order to implement local execution model on top of “lazy execution model” we 
just need to abstract a transport (queue) so that we can use an in-process 
transport. That’s it. It’s what Mistral already has implemented. Again, we see 
that “lazy execution model” is more universal.

IMO this “lazy execution model” should be the main execution model that 
TaskFlow supports, others can be easily implemented on top of it. But the 
opposite assertion is wrong. IMO this is the most important obstacle in all our 
discussions, the reason why we don’t always understand each other well enough. 
I know it may be a lot of work to shift a paradigm in TaskFlow team but if we 
did that we would get enough freedom for using TaskFlow in lots of cases.

Let me know what you think. I might have missed something.

DZ: Interesting idea! So that other models of execution are based on lazy 
execution model? TaskFlow implements this, we can use it, and for other clients 
more convenient higher level execution models are provided? Interesting. Makes 
@Joshua? @Kirill? Others?

I think this is likely possible, which is simiar to whats in, engine types can be built from each other (and if 
we wanted to alter the structure that exists in taskflow) then sure. But see 
that message for more of my concerns around exposing that engine API to library 
users (I think it could have its usage in mistral to expose this, but I'm not 
sure its useful for elsewhere, and once its public engine API, its public for a 
very long time).

=== HA ===

So this is an interesting question, and to me is strongly connected to how your 
engines are executing (and the persistence and state-transitions that they go 
through while running). Without persistence of state and transitions there is 
no good way (a bad way of course can be created, by just redoing all the work, 
but that's not always feasible or the best option) to accomplish resuming in a 
sane manner and there is also imho no way to accomplish any type of automated 
HA of workflows.

Sure, no questions here.

Let me describe:

When you save the states of a workflow and any intermediate results of a 
workflow to some database (for example) and the engine (see above models) which 
is being used (for example the conductor type from above) the application 
containing that engine may be prone to crashes (or just being powered off due 
to software upgrades...). Since taskflows key primitives were made to allow for 
resuming when a crash occurs, it is relatively simple to allow another 
application (also running a conductor) to resume whatever that prior 
application was doing when it crashed. Now most users of taskflow don't want to 
have to do this resumption manually (although they can if they want) so it 
would be expected that the other versions of that application would be running 
would automatically 'know' how to 'take-over' the work of the failed 
application. This is where the concept of the taskflows 'jobboard' 
( comes into play, where a jobboard can be backed by 
something like zookeeper (which provides notifications of lock lose/release to 
others automatically). The jobboard is the place where the other applications 
would be looking to 'take-over' the other failed applications work (by using 
zookeeper 'atomic' primitives designed for this type of usage) and they would 
also be releasing the work back for others to 'take-over' when there own 
zookeeper connection is lost (zookeeper handles this this natively).

Instead of inventing this jobboard built with Zookeeper which is not always 
welcome in the community, why not just use a queue with message 
acknowledgements? In case if engine instance is processing a task and it fails 
a message broker just resubmits the corresponding message automatically. This 
is a standard, e.g. RabbitMQ feature. Am I missing something important?

Renat Akhmerov
@ Mirantis Inc.


OpenStack-dev mailing list<>

OpenStack-dev mailing list

Reply via email to