subject:"\[openstack\-dev\] \[Mistral\]"

Some notes:
Even though we use YAQL now our design is flexible enough to plug other ELs in.
If it tells you something in Amazon SWF a component that makes a decision about 
a further route is called Decider. 
This discussion about conditionals is surely important but it doesn’t matter 
too much if we don’t agree on that lazy execution model.

 Of course I'm trying to make the above not be its own micro-language as much 
 as possible (a switch object starts to act like one, sadly).

Why do you think it’s going to be a micro-language?

 [1] http://www.cs.cmu.edu/~aldrich/papers/onward2009-concurrency.pdf
 [2] 
 http://www.cs.ucf.edu/~dcm/Teaching/COT4810-Spring2011/Literature/DataFlowProgrammingLanguages.pdf

Cool, thanks!

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary

2014-04-15 Thread Kirill Izotov

Since you have mentioned that, I'm kind of interested, who else is going to 
benefit from this transition?  

I'm asking because everyone tells me that it's essential for us to work 
together, though my senses tells me that we are starting from different 
prerequisites, targeting different use cases and taking the entirely different 
approaches to solve the problem. Actually, I'd say we have more differences 
than the things we have in common.

Did you hear the same requests from anyone of your current users? May be we 
should invite them for discussion, just to make sure we would not have to redo 
it again.  

--  
Kirill Izotov


вторник, 15 апреля 2014 г. в 11:13, Joshua Harlow написал:

 Sure, its not the fully complete lazy_engine, but piece by piece we can get 
 there.
  
 Of course code/contributions are welcome, as such things will benefit more 
 than just mistral, but openstack as a whole :-)  
  
 -Josh  
  
 From: Kirill Izotov enyk...@stackstorm.com (mailto:enyk...@stackstorm.com)
 Date: Monday, April 14, 2014 at 9:02 PM
 To: Joshua Harlow harlo...@yahoo-inc.com (mailto:harlo...@yahoo-inc.com)
 Cc: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org)
 Subject: Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow 
 integration summary
  
  Thank for pointing that out, Joshua.  
   
  I had a look on [1] and it seems to me that it might actually do the trick 
  to some degree, though I'm afraid this is still not what we are looking 
  for. While Mistral is asynchronous and event-driven, this particular design 
  is not and would still force us to store the engine in memory and therefore 
  limit our means of scalability. The lazy engine (or better controller) I 
  have proposed is asynchronous at its core and would fit the needs for both 
  of us (since it's much easier to make sync from async, rather than 
  backwards).  
   
  Regarding the retries, while it might work with the current flow design, I 
  doubt it would work with conditional transitions. The attempt to build a 
  repeater by incapsulating the tasks into sub-flow will basically means that 
  every transition they produce will be in that flow and you can't leave it 
  until they are all finished. The whole idea of sub-flows within the scope 
  of direct conditional transitions is a bit unclear to me (and probably us 
  all) at the moment, though I'm trying to rely on them only as a means to 
  lesser the complexity.  
   
  [1] https://review.openstack.org/#/c/86470  
   
  --   
  Kirill Izotov
   
   
  пятница, 11 апреля 2014 г. в 23:47, Joshua Harlow написал:
   
   Thanks for the write-up krill.

   Also some adjustments,  

   Both points are good, and putting some of this on @ 
   https://etherpad.openstack.org/p/taskflow-mistral-details so that we can 
   have it actively noted (feel free to adjust it).  

   I think ivan is working on some docs/code/… for the lazy engine idea, so 
   hopefully we can get back soon with that. Lets see what comes out of that 
   effort and iterate on that.  

   For (2), our are mostly correct about unconditional execution although 
   [1] does now change this, and there are a few active reviews that are 
   being worked [3] on to fit this mistral use-case better. I believe [2] 
   can help move in this direction, ivans ideas I think will also push it a 
   little farther to. Of course lets work together to make sure they fit the 
   best so that taskflow  mistral  openstack can be the best it can be 
   (pigeons not included).  

   Can we also make sure the small issues are noted somewhere (maybe in the 
   above etherpad??). Thanks!  

   [1] https://wiki.openstack.org/wiki/TaskFlow#Retries  
   [2] https://review.openstack.org/#/c/86470
   [3] 
   https://review.openstack.org/#/q/status:open+project:openstack/taskflow,n,z
 

   From: Kirill Izotov enyk...@stackstorm.com 
   (mailto:enyk...@stackstorm.com)
   Reply-To: OpenStack Development Mailing List (not for usage questions) 
   openstack-dev@lists.openstack.org 
   (mailto:openstack-dev@lists.openstack.org)
   Date: Thursday, April 10, 2014 at 9:20 PM
   To: OpenStack-dev@lists.openstack.org 
   (mailto:OpenStack-dev@lists.openstack.org) 
   OpenStack-dev@lists.openstack.org 
   (mailto:OpenStack-dev@lists.openstack.org)
   Subject: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow 
   integration summary

Hi everyone,  
 
This is a summary to the prototype integration we did not too long ago: 
http://github.com/enykeev/mistral/pull/1. Hope it would shed some light 
on the aspects of the integration we are struggling with.  
 
There is a possibility to build Mistral on top of TaskFlow as a 
library, but in order to meet the requirements dictated by Mistral 
users and use cases, both Mistral and TaskFlow should change.  
 
There are two main sides of the story

Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary


On 15 Apr 2014, at 11:13, Joshua Harlow harlo...@yahoo-inc.com wrote:

 Sure, its not the fully complete lazy_engine, but piece by piece we can get 
 there.

Did you make any estimations when it could happen? :)

 Of course code/contributions are welcome, as such things will benefit more 
 than just mistral, but openstack as a whole :-)

OK.

 From: Kirill Izotov enyk...@stackstorm.com
 The whole idea of sub-flows within the scope of direct conditional 
 transitions is a bit unclear to me (and probably us all) at the moment, 
 though I'm trying to rely on them only as a means to lesser the complexity.

Yes, eventually it’s for reducing complexity. I would just add that it opens 
wide range of opportunities like:
ability to combine multiple physically independent workflows
reusability (using one workflow as a part of another)
isolation (different namespaces for data flow contexts etc)

Renat Akhmerov
@ Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [mistral] Tracking PoC readiness etherpad

Team,

Following up on the yesterday’s community meeting I created an etherpad and 
captured all the formal steps we need to make in order to consider Mistral PoC 
ready [0]. Please take a look and comment and/or add other items that you think 
I missed.

My expectation is to get all the items going before “Release under ‘poc’ tag in 
git knocked out this week. The rest is on the next week.

[0] https://etherpad.openstack.org/p/mistral-poc-readiness

Renat Akhmerov
@ Mirantis Inc.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

2014-04-15 Thread Joshua Harlow

I think we agree on the lazy execution model. At least at a high-level, I'd 
rather not agree on the API's that are exactly exposed until there is an 
implementation since I've found that agreeing to any type of API's before there 
is the needed groundwork to make it happen is pretty useless and a waste of 
energy on everyones part.

Decider sounds like it could work also as a name, although it seems from 
dataflow like work its called a switch or gate, either or I guess.

As far as the micro-language:

So there are typically 2 types of DSL's that occur, internal and external.

An internal DSL is like http://martinfowler.com/bliki/InternalDslStyle.html, 
taskflow is already a micro-DSL internal to python (mistral is an external 
DSL[1]). To me there is a drawback of becoming to much of a DSL (internal or 
external) in that it requires a lot of new learning (imho internal DSLs are 
easier to pick-up since they take advantage of the surrounding languages 
capabilities, in this case python). So that’s what I just want to keep in our 
minds that we need to make it simple *enough*, or we will die a nasty death of 
complexity :-P

[1] http://martinfowler.com/bliki/DomainSpecificLanguage.html

From: Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Tuesday, April 15, 2014 at 12:19 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

Some notes:

  *   Even though we use YAQL now our design is flexible enough to plug other 
ELs in.
  *   If it tells you something in Amazon SWF a component that makes a decision 
about a further route is called Decider.
  *   This discussion about conditionals is surely important but it doesn’t 
matter too much if we don’t agree on that lazy execution model.

Of course I'm trying to make the above not be its own micro-language as much as 
possible (a switch object starts to act like one, sadly).

Why do you think it’s going to be a micro-language?

[1] http://www.cs.cmu.edu/~aldrich/papers/onward2009-concurrency.pdf
[2] 
http://www.cs.ucf.edu/~dcm/Teaching/COT4810-Spring2011/Literature/DataFlowProgrammingLanguages.pdf

Cool, thanks!

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary

2014-04-15 Thread Joshua Harlow

Well Ivan afaik is thinking through it, but being a community project its not 
exactly easy to put estimations or timelines on things (I don't control Ivan, 
or others).

Likely faster if u guys want to get involved.

-Josh

From: Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Tuesday, April 15, 2014 at 12:35 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration 
summary


On 15 Apr 2014, at 11:13, Joshua Harlow 
harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote:

Sure, its not the fully complete lazy_engine, but piece by piece we can get 
there.

Did you make any estimations when it could happen? :)

Of course code/contributions are welcome, as such things will benefit more than 
just mistral, but openstack as a whole :-)

OK.

From: Kirill Izotov enyk...@stackstorm.commailto:enyk...@stackstorm.com
The whole idea of sub-flows within the scope of direct conditional transitions 
is a bit unclear to me (and probably us all) at the moment, though I'm trying 
to rely on them only as a means to lesser the complexity.

Yes, eventually it’s for reducing complexity. I would just add that it opens 
wide range of opportunities like:

  *   ability to combine multiple physically independent workflows
  *   reusability (using one workflow as a part of another)
  *   isolation (different namespaces for data flow contexts etc)

Renat Akhmerov
@ Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

On 16 Apr 2014, at 00:18, Joshua Harlow harlo...@yahoo-inc.com wrote:

 Decider sounds like it could work also as a name, although it seems from 
 dataflow like work its called a switch or gate, either or I guess.

That’s fine. It doesn’t matter too much to me personally.

 As far as the micro-language:
 
 So there are typically 2 types of DSL's that occur, internal and external.
 
 An internal DSL is like http://martinfowler.com/bliki/InternalDslStyle.html, 
 taskflow is already a micro-DSL internal to python (mistral is an external 
 DSL[1]). To me there is a drawback of becoming to much of a DSL (internal or 
 external) in that it requires a lot of new learning (imho internal DSLs are 
 easier to pick-up since they take advantage of the surrounding languages 
 capabilities, in this case python). So that’s what I just want to keep in our 
 minds that we need to make it simple *enough*, or we will die a nasty death 
 of complexity :-P
 
 [1] http://martinfowler.com/bliki/DomainSpecificLanguage.html

Ok, got it. Thanks. I’m just still not sure why you emphasize on that 
micro-language thing. IMO terms like that can scary people :) In fact, this 
‘switch’ (or decider, or whatever) is just an additional API which can be used 
to alter flow behavior. 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

2014-04-15 Thread Joshua Harlow

I have stricken the word micro and language from my vocabulary. Begone evil 
demons!! Haha :)

Sent from my really tiny device...

On Apr 15, 2014, at 10:35 PM, Renat Akhmerov 
rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote:

On 16 Apr 2014, at 00:18, Joshua Harlow 
harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote:

Decider sounds like it could work also as a name, although it seems from 
dataflow like work its called a switch or gate, either or I guess.

That’s fine. It doesn’t matter too much to me personally.

As far as the micro-language:

So there are typically 2 types of DSL's that occur, internal and external.

An internal DSL is like http://martinfowler.com/bliki/InternalDslStyle.html, 
taskflow is already a micro-DSL internal to python (mistral is an external 
DSL[1]). To me there is a drawback of becoming to much of a DSL (internal or 
external) in that it requires a lot of new learning (imho internal DSLs are 
easier to pick-up since they take advantage of the surrounding languages 
capabilities, in this case python). So that’s what I just want to keep in our 
minds that we need to make it simple *enough*, or we will die a nasty death of 
complexity :-P

[1] http://martinfowler.com/bliki/DomainSpecificLanguage.html

Ok, got it. Thanks. I’m just still not sure why you emphasize on that 
micro-language thing. IMO terms like that can scary people :) In fact, this 
‘switch’ (or decider, or whatever) is just an additional API which can be used 
to alter flow behavior.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [mistral] Community meeting reminder - 04/14/2014

2014-04-14 Thread Renat Akhmerov

Hi,

This a reminder about the community meeting in IRC (#openstack-meeting) that 
we’ll have today at 16.00 UTC.

Agenda:
Review action items
Current status (quickly by team members)
POC demo scenario readiness and ways to improve it
TaskFlow integration status
Open discussion

It could also be seen at https://wiki.openstack.org/wiki/Meetings/MistralAgenda 
as well as the links to the previous meeting minutes and logs.

See you there!

Renat Akhmerov
@ Mirantis Inc.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

2014-04-14 Thread Renat Akhmerov

The more the better :)

If seriously, this one is less detailed and rather focuses on high-level things 
so that everyone can have a high-level understanding of what’s been going on on 
Mistral/TaskFlow integration. In other words, this email raises question “What” 
and doesn’t raise “Why?” while [0] goes deeper into details.

[0] http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html

Renat Akhmerov
@ Mirantis Inc.



On 12 Apr 2014, at 00:59, Joshua Harlow harlo...@yahoo-inc.com wrote:

 I'm confused, why is this 2 emails??
 
 http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html
 
 Seems better to just have 1 chain, not 2.
 
 From: Dmitri Zimine d...@stackstorm.com
 Reply-To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Date: Friday, April 11, 2014 at 9:55 AM
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
 
 We prototyped Mistral / TaskFlow integration and have a follow-up 
 discussions. 
 
 SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow 
 library, with some required modifications to function resliently as a 
 service, and for smooth integration. However, the TaskFlow flow controls are 
 insufficient for Mistral use cases. 
 
 Details discussed on other thirds. 
 The prototype scope - [0]; code and discussion - [1] and techical highlights 
 - [2].
 
 DETAILS: 
 
 1) Embedding TaskFlow inside Mistral:
 * Required: make the engine lazy [3], [4].This is required to support 
 long-running delegates and not loose tasks when engine manager process 
 restarts.
 
 * Persistence: need clarity how to replace or mix-in TaskFlow persistence 
 with Mistral persistence. Renat is taking a look.
 
 * Declaring Flows in YAML DSL: done for simplest flow. Need to prototype for 
 data flow. Rich flow controls are missing in TaskFlow for a representative 
 prototype.
 
 * ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both 
 Mistral and TaskFlow implementations work. But we shall resolve the overlap. 
 
 * Ignored for now - unlikely any risks: Keystone integration, Mistral event 
 scheduler, Mistral declarative services and action definition.
 
 2) TaskFlow library features
 * Must: flow control - conditional transitions, references, expression 
 evaluation, to express real-life workflows [5]. The required flow control 
 primitives are 1) repeater 2) flow in flow 3) direct transition 4) 
 conditional transition 5) multiple data. TaskFlow has 1) and 2), need to add 
 3/4/5. 
 
 * Other details and smaller requests are in the discussion [1]
 
 3) Next Steps proposed:
 * Mistal team: summarize the requirements discussed and agreed on [2] and [3]
 * Mistral team: code sample (tests?) on how Mistral would like to consume 
 TaskFlow lazy engine 
 * Taskflow team: Provide a design for alternative TaskExecutor approach 
 (prototypes, boxes, arrows, crayons :)) 
 * Decide on lazy engine
 * Move the discussion on other elements on integration.
 
 References:
 [0] The scope of the prototype: 
 https://etherpad.openstack.org/p/mistral-taskflow-prototype
 [1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1
 [2] Techical summary 
 http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html
 [3] Email discussion on TaskFlow lazy eninge 
 http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html
 [4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/
 [5] Use cases https://github.com/dzimine/mistral-workflows/tree/add-usecases
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [mistral] Community meeting minutes - 04/14/2014

2014-04-14 Thread Renat Akhmerov

Thanks for joining today’s community meeting.

Meeting minutes: 
http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-04-14-15.59.html
Full log: 
http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-04-14-15.59.log.html

Please join us next monday on Apr 21st.

Renat Akhmerov
@ Mirantis Inc.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

Gotcha, thanks :)

From: Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Monday, April 14, 2014 at 5:11 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

The more the better :)

If seriously, this one is less detailed and rather focuses on high-level things 
so that everyone can have a high-level understanding of what’s been going on on 
Mistral/TaskFlow integration. In other words, this email raises question “What” 
and doesn’t raise “Why?” while [0] goes deeper into details.

[0] http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html

Renat Akhmerov
@ Mirantis Inc.

On 12 Apr 2014, at 00:59, Joshua Harlow 
harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote:

I'm confused, why is this 2 emails??

http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html

Seems better to just have 1 chain, not 2.

From: Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, April 11, 2014 at 9:55 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

We prototyped Mistral / TaskFlow integration and have a follow-up discussions.

SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, 
with some required modifications to function resliently as a service, and for 
smooth integration. However, the TaskFlow flow controls are insufficient for 
Mistral use cases.

Details discussed on other thirds.
The prototype scope - 
[0https://etherpad.openstack.org/p/mistral-taskflow-prototype]; code and 
discussion - [1https://github.com/enykeev/mistral/pull/1] and techical 
highlights - 
[2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html].

DETAILS:

1) Embedding TaskFlow inside Mistral:
* Required: make the engine lazy 
[3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html], 
[4http://paste.openstack.org/show/75389/].This is required to support 
long-running delegates and not loose tasks when engine manager process restarts.

* Persistence: need clarity how to replace or mix-in TaskFlow persistence with 
Mistral persistence. Renat is taking a look.

* Declaring Flows in YAML DSL: done for simplest flow. Need to prototype for 
data flow. Rich flow controls are missing in TaskFlow for a representative 
prototype.

* ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both Mistral 
and TaskFlow implementations work. But we shall resolve the overlap.

* Ignored for now - unlikely any risks: Keystone integration, Mistral event 
scheduler, Mistral declarative services and action definition.

2) TaskFlow library features
* Must: flow control - conditional transitions, references, expression 
evaluation, to express real-life workflows 
[5https://github.com/dzimine/mistral-workflows/tree/add-usecases]. The 
required flow control primitives are 1) repeater 2) flow in flow 3) direct 
transition 4) conditional transition 5) multiple data. TaskFlow has 1) and 2), 
need to add 3/4/5.

* Other details and smaller requests are in the discussion 
[1https://github.com/enykeev/mistral/pull/1]

3) Next Steps proposed:
* Mistal team: summarize the requirements discussed and agreed on 
[2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html] 
and 
[3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html]
* Mistral team: code sample (tests?) on how Mistral would like to consume 
TaskFlow lazy engine
* Taskflow team: Provide a design for alternative TaskExecutor approach 
(prototypes, boxes, arrows, crayons :))
* Decide on lazy engine
* Move the discussion on other elements on integration.

References:
[0] The scope of the prototype: 
https://etherpad.openstack.org/p/mistral-taskflow-prototype
[1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1
[2] Techical summary 
http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html
[3] Email discussion on TaskFlow lazy eninge 
http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html
[4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/
[5] Use cases https://github.com/dzimine/mistral-workflows/tree/add-usecases
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

Can we describe exactly what references, direct transition, expression 
evaluation are doing in #2.

Expression evaluation especially seems to be an odd one, what's wrong with 
pythons expression evaluation? I can't quite see why that would/should exist in 
taskflow.

I can see it being implemented in mistral, where mistral converts whatever DSL 
it wants into taskflow primitives and then taskflow runs the code; this 
decoupling ensures that taskflow does not force a DSL on people that want to 
use taskflow as a python library (this kind of restriction imho isn't 
acceptable for a library to do,  and limits taskflows own usage and 
integration).

Thanks,

Josh

From: Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, April 11, 2014 at 9:55 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

We prototyped Mistral / TaskFlow integration and have a follow-up discussions.

SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, 
with some required modifications to function resliently as a service, and for 
smooth integration. However, the TaskFlow flow controls are insufficient for 
Mistral use cases.

Details discussed on other thirds.
The prototype scope - 
[0https://etherpad.openstack.org/p/mistral-taskflow-prototype]; code and 
discussion - [1https://github.com/enykeev/mistral/pull/1] and techical 
highlights - 
[2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html].

DETAILS:

1) Embedding TaskFlow inside Mistral:
* Required: make the engine lazy 
[3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html], 
[4http://paste.openstack.org/show/75389/].This is required to support 
long-running delegates and not loose tasks when engine manager process restarts.

* Persistence: need clarity how to replace or mix-in TaskFlow persistence with 
Mistral persistence. Renat is taking a look.

* Declaring Flows in YAML DSL: done for simplest flow. Need to prototype for 
data flow. Rich flow controls are missing in TaskFlow for a representative 
prototype.

* ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both Mistral 
and TaskFlow implementations work. But we shall resolve the overlap.

* Ignored for now - unlikely any risks: Keystone integration, Mistral event 
scheduler, Mistral declarative services and action definition.

2) TaskFlow library features
* Must: flow control - conditional transitions, references, expression 
evaluation, to express real-life workflows 
[5https://github.com/dzimine/mistral-workflows/tree/add-usecases]. The 
required flow control primitives are 1) repeater 2) flow in flow 3) direct 
transition 4) conditional transition 5) multiple data. TaskFlow has 1) and 2), 
need to add 3/4/5.

* Other details and smaller requests are in the discussion 
[1https://github.com/enykeev/mistral/pull/1]

3) Next Steps proposed:
* Mistal team: summarize the requirements discussed and agreed on 
[2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html] 
and 
[3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html]
* Mistral team: code sample (tests?) on how Mistral would like to consume 
TaskFlow lazy engine
* Taskflow team: Provide a design for alternative TaskExecutor approach 
(prototypes, boxes, arrows, crayons :))
* Decide on lazy engine
* Move the discussion on other elements on integration.

References:
[0] The scope of the prototype: 
https://etherpad.openstack.org/p/mistral-taskflow-prototype
[1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1
[2] Techical summary 
http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html
[3] Email discussion on TaskFlow lazy eninge 
http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html
[4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/
[5] Use cases https://github.com/dzimine/mistral-workflows/tree/add-usecases
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

2014-04-14 Thread Kirill Izotov

They are all parts of conditional transitions: every task should have a number 
of possible transitions; each transition consist of a reference to the task we 
want to transit to and the condition that should evaluate to true for 
transition to start.  

At that point, I'd say that it perfectly fine for TaskFlow to evaluate python 
conditions rather than implementing YAQL, though there should be a place for us 
to pass the condition evaluation logic we are using.  

--  
Kirill Izotov


вторник, 15 апреля 2014 г. в 8:02, Joshua Harlow написал:

 Can we describe exactly what references, direct transition, expression 
 evaluation are doing in #2.
  
 Expression evaluation especially seems to be an odd one, what's wrong with 
 pythons expression evaluation? I can't quite see why that would/should exist 
 in taskflow.  
  
 I can see it being implemented in mistral, where mistral converts whatever 
 DSL it wants into taskflow primitives and then taskflow runs the code; this 
 decoupling ensures that taskflow does not force a DSL on people that want to 
 use taskflow as a python library (this kind of restriction imho isn't 
 acceptable for a library to do,  and limits taskflows own usage and 
 integration).  
  
 Thanks,  
  
 Josh  
  
 From: Dmitri Zimine d...@stackstorm.com (mailto:d...@stackstorm.com)
 Reply-To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org)
 Date: Friday, April 11, 2014 at 9:55 AM
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org)
 Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
  
  We prototyped Mistral / TaskFlow integration and have a follow-up 
  discussions.   
   
  SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow 
  library, with some required modifications to function resliently as a 
  service, and for smooth integration. However, the TaskFlow flow controls 
  are insufficient for Mistral use cases.   
   
  Details discussed on other thirds.   
  The prototype scope - [0 
  (https://etherpad.openstack.org/p/mistral-taskflow-prototype)]; code and 
  discussion - [1 (https://github.com/enykeev/mistral/pull/1)] and techical 
  highlights - [2 
  (http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html)].
   
  DETAILS:   
   
  1) Embedding TaskFlow inside Mistral:  
  * Required: make the engine lazy [3 
  (http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html)],
   [4 (http://paste.openstack.org/show/75389/)].This is required to support 
  long-running delegates and not loose tasks when engine manager process 
  restarts.
   
  * Persistence: need clarity how to replace or mix-in TaskFlow persistence 
  with Mistral persistence. Renat is taking a look.  
   
  * Declaring Flows in YAML DSL: done for simplest flow. Need to prototype 
  for data flow. Rich flow controls are missing in TaskFlow for a 
  representative prototype.  
   
  * ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both 
  Mistral and TaskFlow implementations work. But we shall resolve the 
  overlap.   
   
  * Ignored for now - unlikely any risks: Keystone integration, Mistral event 
  scheduler, Mistral declarative services and action definition.  
   
  2) TaskFlow library features  
  * Must: flow control - conditional transitions, references, expression 
  evaluation, to express real-life workflows [5 
  (https://github.com/dzimine/mistral-workflows/tree/add-usecases)]. The 
  required flow control primitives are 1) repeater 2) flow in flow 3) direct 
  transition 4) conditional transition 5) multiple data. TaskFlow has 1) and 
  2), need to add 3/4/5.  
   
  * Other details and smaller requests are in the discussion [1 
  (https://github.com/enykeev/mistral/pull/1)]  
   
  3) Next Steps proposed:  
  * Mistal team: summarize the requirements discussed and agreed on [2 
  (http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html)]
   and [3 
  (http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html)]
  * Mistral team: code sample (tests?) on how Mistral would like to consume 
  TaskFlow lazy engine  
  * Taskflow team: Provide a design for alternative TaskExecutor approach 
  (prototypes, boxes, arrows, crayons :))  
  * Decide on lazy engine
  * Move the discussion on other elements on integration.
   
  References:  
  [0] The scope of the prototype: 
  https://etherpad.openstack.org/p/mistral-taskflow-prototype
  [1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1
  [2] Techical summary 
  http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html
  [3] Email discussion on TaskFlow lazy eninge 
  http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html
  [4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/
  [5

Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary

2014-04-14 Thread Kirill Izotov

Thank for pointing that out, Joshua.

I had a look on [1] and it seems to me that it might actually do the trick to 
some degree, though I'm afraid this is still not what we are looking for. While 
Mistral is asynchronous and event-driven, this particular design is not and 
would still force us to store the engine in memory and therefore limit our 
means of scalability. The lazy engine (or better controller) I have proposed is 
asynchronous at its core and would fit the needs for both of us (since it's 
much easier to make sync from async, rather than backwards).

Regarding the retries, while it might work with the current flow design, I 
doubt it would work with conditional transitions. The attempt to build a 
repeater by incapsulating the tasks into sub-flow will basically means that 
every transition they produce will be in that flow and you can't leave it until 
they are all finished. The whole idea of sub-flows within the scope of direct 
conditional transitions is a bit unclear to me (and probably us all) at the 
moment, though I'm trying to rely on them only as a means to lesser the 
complexity.

[1] https://review.openstack.org/#/c/86470  

--  
Kirill Izotov


пятница, 11 апреля 2014 г. в 23:47, Joshua Harlow написал:

 Thanks for the write-up krill.
  
 Also some adjustments,  
  
 Both points are good, and putting some of this on @ 
 https://etherpad.openstack.org/p/taskflow-mistral-details so that we can have 
 it actively noted (feel free to adjust it).  
  
 I think ivan is working on some docs/code/… for the lazy engine idea, so 
 hopefully we can get back soon with that. Lets see what comes out of that 
 effort and iterate on that.  
  
 For (2), our are mostly correct about unconditional execution although [1] 
 does now change this, and there are a few active reviews that are being 
 worked [3] on to fit this mistral use-case better. I believe [2] can help 
 move in this direction, ivans ideas I think will also push it a little 
 farther to. Of course lets work together to make sure they fit the best so 
 that taskflow  mistral  openstack can be the best it can be (pigeons not 
 included).  
  
 Can we also make sure the small issues are noted somewhere (maybe in the 
 above etherpad??). Thanks!  
  
 [1] https://wiki.openstack.org/wiki/TaskFlow#Retries  
 [2] https://review.openstack.org/#/c/86470
 [3] 
 https://review.openstack.org/#/q/status:open+project:openstack/taskflow,n,z  
  
 From: Kirill Izotov enyk...@stackstorm.com (mailto:enyk...@stackstorm.com)
 Reply-To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org)
 Date: Thursday, April 10, 2014 at 9:20 PM
 To: OpenStack-dev@lists.openstack.org 
 (mailto:OpenStack-dev@lists.openstack.org) 
 OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org)
 Subject: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration 
 summary
  
  Hi everyone,  
   
  This is a summary to the prototype integration we did not too long ago: 
  http://github.com/enykeev/mistral/pull/1. Hope it would shed some light on 
  the aspects of the integration we are struggling with.  
   
  There is a possibility to build Mistral on top of TaskFlow as a library, 
  but in order to meet the requirements dictated by Mistral users and use 
  cases, both Mistral and TaskFlow should change.  
   
  There are two main sides of the story. One is engine. The other is flow 
  control capabilities.   
   
  1) THE ENGINE   
  The current TaskFlow implementation of engine doesn't fit Mistral needs 
  because it is synchronous, it blocks the thread, it requires us to store 
  the reference to the particular engine to be able to get its status and 
  suspend the execution and it lacks long-running task compatibility. To fix 
  this problem in a solid and maintainable way, we need to split the engine 
  into its synchronous and asynchronous counterparts.
   
  Lazy engine should be async and atomic, it should not have its own state, 
  instead it should rely on some kind of global state (db or in-memory, 
  depending on a type of application). It should have at least two methods: 
  run and task_complete. Run method should calculate the first batch of tasks 
  and schedule them for executing (either put them in queue or spawn the 
  threads). Task_complete should mark a certain task to be completed and then 
  schedule the next batch of tasks that became available due to resolution of 
  this one.  
   
  The desired use of lazy engine in Mistral is illustrated here: 
  https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign#Big_Picture.
   It should support long running tasks and survive engine process restart 
  without loosing the state of the running actions. So it must be passive 
  (lazy) and persistent.   
   
  On Mistral side we are using Lazy engine by patching async.run directly to 
  the API (or engine queue) and async.task_complete

Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

Thanks, that helps explain those.

Let's see where the conditional evaluation logic goes. Likely it won't be 
python conditions directly.

I think though we should be able to work with other condition logic. I started 
https://review.openstack.org/#/c/87417/ today, hopefully can flush it out more 
in the days to come.

The differences I can see so far are around how a taskflow engine activates 
this conditional (and what results when the switch chooses a path). In taskflow 
the whole workflow is analyzed before execution (and translated into a directed 
graph) as to what is provided and what is required by each task in the 
workflow. Conditionals change this since its not known ahead of time which path 
will be selected. For now (in the above review) I am making it so that each 
path that could be switched to will have to have the same requirements and the 
same outputs (so that the analysis logic still works correctly). I'm thinking 
that a switch 'task' will return which choice it made, then this will affect 
the further path that will be followed (basically all other path choices will 
be 'abandoned'). This fits pretty well I think into how typically this is done 
in dataflow-like way and won't affect the state-transitions or ability to 
resume and such (btw found some neat papers at [1],[2] that show some past 
history that I didn't know about).

Of course I'm trying to make the above not be its own micro-language as much as 
possible (a switch object starts to act like one, sadly).

Comments welcome. Code welcome even more :-P

[1] http://www.cs.cmu.edu/~aldrich/papers/onward2009-concurrency.pdf
[2] 
http://www.cs.ucf.edu/~dcm/Teaching/COT4810-Spring2011/Literature/DataFlowProgrammingLanguages.pdf

From: Kirill Izotov enyk...@stackstorm.commailto:enyk...@stackstorm.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Monday, April 14, 2014 at 8:31 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

They are all parts of conditional transitions: every task should have a number 
of possible transitions; each transition consist of a reference to the task we 
want to transit to and the condition that should evaluate to true for 
transition to start.

At that point, I'd say that it perfectly fine for TaskFlow to evaluate python 
conditions rather than implementing YAQL, though there should be a place for us 
to pass the condition evaluation logic we are using.

--
Kirill Izotov


вторник, 15 апреля 2014 г. в 8:02, Joshua Harlow написал:

Can we describe exactly what references, direct transition, expression 
evaluation are doing in #2.

Expression evaluation especially seems to be an odd one, what's wrong with 
pythons expression evaluation? I can't quite see why that would/should exist in 
taskflow.

I can see it being implemented in mistral, where mistral converts whatever DSL 
it wants into taskflow primitives and then taskflow runs the code; this 
decoupling ensures that taskflow does not force a DSL on people that want to 
use taskflow as a python library (this kind of restriction imho isn't 
acceptable for a library to do,  and limits taskflows own usage and 
integration).

Thanks,

Josh

From: Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, April 11, 2014 at 9:55 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

We prototyped Mistral / TaskFlow integration and have a follow-up discussions.

SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, 
with some required modifications to function resliently as a service, and for 
smooth integration. However, the TaskFlow flow controls are insufficient for 
Mistral use cases.

Details discussed on other thirds.
The prototype scope - 
[0https://etherpad.openstack.org/p/mistral-taskflow-prototype]; code and 
discussion - [1https://github.com/enykeev/mistral/pull/1] and techical 
highlights - 
[2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html].

DETAILS:

1) Embedding TaskFlow inside Mistral:
* Required: make the engine lazy 
[3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html], 
[4http://paste.openstack.org/show/75389/].This is required to support 
long-running delegates and not loose tasks when engine manager process restarts.

* Persistence: need clarity how to replace or mix-in TaskFlow persistence with 
Mistral persistence. Renat is taking a look.

* Declaring Flows in YAML DSL: done

Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary

Sure, its not the fully complete lazy_engine, but piece by piece we can get 
there.

Of course code/contributions are welcome, as such things will benefit more than 
just mistral, but openstack as a whole :-)

-Josh

From: Kirill Izotov enyk...@stackstorm.commailto:enyk...@stackstorm.com
Date: Monday, April 14, 2014 at 9:02 PM
To: Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com
Cc: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration 
summary

Thank for pointing that out, Joshua.

I had a look on [1] and it seems to me that it might actually do the trick to 
some degree, though I'm afraid this is still not what we are looking for. While 
Mistral is asynchronous and event-driven, this particular design is not and 
would still force us to store the engine in memory and therefore limit our 
means of scalability. The lazy engine (or better controller) I have proposed is 
asynchronous at its core and would fit the needs for both of us (since it's 
much easier to make sync from async, rather than backwards).

Regarding the retries, while it might work with the current flow design, I 
doubt it would work with conditional transitions. The attempt to build a 
repeater by incapsulating the tasks into sub-flow will basically means that 
every transition they produce will be in that flow and you can't leave it until 
they are all finished. The whole idea of sub-flows within the scope of direct 
conditional transitions is a bit unclear to me (and probably us all) at the 
moment, though I'm trying to rely on them only as a means to lesser the 
complexity.

[1] https://review.openstack.org/#/c/86470

--
Kirill Izotov

пятница, 11 апреля 2014 г. в 23:47, Joshua Harlow написал:

Thanks for the write-up krill.

Also some adjustments,

Both points are good, and putting some of this on @ 
https://etherpad.openstack.org/p/taskflow-mistral-details so that we can have 
it actively noted (feel free to adjust it).

I think ivan is working on some docs/code/… for the lazy engine idea, so 
hopefully we can get back soon with that. Lets see what comes out of that 
effort and iterate on that.

For (2), our are mostly correct about unconditional execution although [1] does 
now change this, and there are a few active reviews that are being worked [3] 
on to fit this mistral use-case better. I believe [2] can help move in this 
direction, ivans ideas I think will also push it a little farther to. Of course 
lets work together to make sure they fit the best so that taskflow  mistral  
openstack can be the best it can be (pigeons not included).

Can we also make sure the small issues are noted somewhere (maybe in the above 
etherpad??). Thanks!

[1] https://wiki.openstack.org/wiki/TaskFlow#Retries
[2] https://review.openstack.org/#/c/86470
[3] https://review.openstack.org/#/q/status:open+project:openstack/taskflow,n,z

From: Kirill Izotov enyk...@stackstorm.commailto:enyk...@stackstorm.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Thursday, April 10, 2014 at 9:20 PM
To: 
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org 
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
Subject: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration 
summary

Hi everyone,

This is a summary to the prototype integration we did not too long ago: 
http://github.com/enykeev/mistral/pull/1. Hope it would shed some light on the 
aspects of the integration we are struggling with.

There is a possibility to build Mistral on top of TaskFlow as a library, but in 
order to meet the requirements dictated by Mistral users and use cases, both 
Mistral and TaskFlow should change.

There are two main sides of the story. One is engine. The other is flow control 
capabilities.

1) THE ENGINE
The current TaskFlow implementation of engine doesn't fit Mistral needs because 
it is synchronous, it blocks the thread, it requires us to store the reference 
to the particular engine to be able to get its status and suspend the execution 
and it lacks long-running task compatibility. To fix this problem in a solid 
and maintainable way, we need to split the engine into its synchronous and 
asynchronous counterparts.

Lazy engine should be async and atomic, it should not have its own state, 
instead it should rely on some kind of global state (db or in-memory, depending 
on a type of application). It should have at least two methods: run and 
task_complete. Run method should calculate the first batch of tasks and 
schedule them for executing (either put them in queue or spawn the threads). 
Task_complete should mark a certain task to be completed and then schedule the 
next batch of tasks that became available due to resolution of this one

Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary

2014-04-11 Thread Joshua Harlow

Thanks for the write-up krill.

Also some adjustments,

Both points are good, and putting some of this on @ 
https://etherpad.openstack.org/p/taskflow-mistral-details so that we can have 
it actively noted (feel free to adjust it).

I think ivan is working on some docs/code/… for the lazy engine idea, so 
hopefully we can get back soon with that. Lets see what comes out of that 
effort and iterate on that.

For (2), our are mostly correct about unconditional execution although [1] does 
now change this, and there are a few active reviews that are being worked [3] 
on to fit this mistral use-case better. I believe [2] can help move in this 
direction, ivans ideas I think will also push it a little farther to. Of course 
lets work together to make sure they fit the best so that taskflow  mistral  
openstack can be the best it can be (pigeons not included).

Can we also make sure the small issues are noted somewhere (maybe in the above 
etherpad??). Thanks!

[1] https://wiki.openstack.org/wiki/TaskFlow#Retries
[2] https://review.openstack.org/#/c/86470
[3] https://review.openstack.org/#/q/status:open+project:openstack/taskflow,n,z

From: Kirill Izotov enyk...@stackstorm.commailto:enyk...@stackstorm.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Thursday, April 10, 2014 at 9:20 PM
To: 
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org 
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
Subject: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration 
summary

Hi everyone,

This is a summary to the prototype integration we did not too long ago: 
http://github.com/enykeev/mistral/pull/1. Hope it would shed some light on the 
aspects of the integration we are struggling with.

There is a possibility to build Mistral on top of TaskFlow as a library, but in 
order to meet the requirements dictated by Mistral users and use cases, both 
Mistral and TaskFlow should change.

There are two main sides of the story. One is engine. The other is flow control 
capabilities.

1) THE ENGINE
The current TaskFlow implementation of engine doesn't fit Mistral needs because 
it is synchronous, it blocks the thread, it requires us to store the reference 
to the particular engine to be able to get its status and suspend the execution 
and it lacks long-running task compatibility. To fix this problem in a solid 
and maintainable way, we need to split the engine into its synchronous and 
asynchronous counterparts.

Lazy engine should be async and atomic, it should not have its own state, 
instead it should rely on some kind of global state (db or in-memory, depending 
on a type of application). It should have at least two methods: run and 
task_complete. Run method should calculate the first batch of tasks and 
schedule them for executing (either put them in queue or spawn the threads). 
Task_complete should mark a certain task to be completed and then schedule the 
next batch of tasks that became available due to resolution of this one.

The desired use of lazy engine in Mistral is illustrated here: 
https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign#Big_Picture. 
It should support long running tasks and survive engine process restart without 
loosing the state of the running actions. So it must be passive (lazy) and 
persistent.

On Mistral side we are using Lazy engine by patching async.run directly to the 
API (or engine queue) and async.task_complete to the worker queue result 
channel (and the API for long running tasks). We are still sharing the same 
graph_analyzer, but instead of relying on loop and Futures, we are handling the 
execution ourselves in a scalable and robust way.

Then, on top of it you can build a sync engine by introducing Futures. You are 
using async.run() to schedule tasks by transforming them to Futures and then 
starting a loop, checking Futures for completion and sending their results to 
async.task_complete() which would produce even more Futures to check over. Just 
the same way TaskFlow do it right now.

The reason I'm proposing to extract Futures from async engine is because they 
won't work if we have multiple engine processes that should handle the task 
results concurrently (and without that there will be no scalability).

2) THE FLOW CONTROL CAPABILITIES

Since we treat TaskFlow as a library we expect them to provide us with a number 
of primitives to build our workflow with them. Most important of them to us for 
the moment are Direct Transitions, and Conditional Transitions.

The current implementation of flow transitions in TaskFlow are built on top of 
data flow dependencies where each task provides some data to the flow and 
requires some data to be present prior being executed. In other words, you are 
starting to build your flow tree from the last task through the first one by 
adding their requirements

[openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

2014-04-11 Thread Dmitri Zimine

We prototyped Mistral / TaskFlow integration and have a follow-up discussions. 

SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, 
with some required modifications to function resliently as a service, and for 
smooth integration. However, the TaskFlow flow controls are insufficient for 
Mistral use cases. 

Details discussed on other thirds. 
The prototype scope - [0]; code and discussion - [1] and techical highlights - 
[2].

DETAILS: 

1) Embedding TaskFlow inside Mistral:
* Required: make the engine lazy [3], [4].This is required to support 
long-running delegates and not loose tasks when engine manager process restarts.

* Persistence: need clarity how to replace or mix-in TaskFlow persistence with 
Mistral persistence. Renat is taking a look.

* Declaring Flows in YAML DSL: done for simplest flow. Need to prototype for 
data flow. Rich flow controls are missing in TaskFlow for a representative 
prototype.

* ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both Mistral 
and TaskFlow implementations work. But we shall resolve the overlap. 

* Ignored for now - unlikely any risks: Keystone integration, Mistral event 
scheduler, Mistral declarative services and action definition.

2) TaskFlow library features
* Must: flow control - conditional transitions, references, expression 
evaluation, to express real-life workflows [5]. The required flow control 
primitives are 1) repeater 2) flow in flow 3) direct transition 4) conditional 
transition 5) multiple data. TaskFlow has 1) and 2), need to add 3/4/5. 

* Other details and smaller requests are in the discussion [1]

3) Next Steps proposed:
* Mistal team: summarize the requirements discussed and agreed on [2] and [3]
* Mistral team: code sample (tests?) on how Mistral would like to consume 
TaskFlow lazy engine 
* Taskflow team: Provide a design for alternative TaskExecutor approach 
(prototypes, boxes, arrows, crayons :)) 
* Decide on lazy engine
* Move the discussion on other elements on integration.

References:
[0] The scope of the prototype: 
https://etherpad.openstack.org/p/mistral-taskflow-prototype
[1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1
[2] Techical summary 
http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html
[3] Email discussion on TaskFlow lazy eninge 
http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html
[4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/
[5] Use cases https://github.com/dzimine/mistral-workflows/tree/add-usecases___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

2014-04-11 Thread Joshua Harlow

I'm confused, why is this 2 emails??

http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html

Seems better to just have 1 chain, not 2.

From: Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, April 11, 2014 at 9:55 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

We prototyped Mistral / TaskFlow integration and have a follow-up discussions.

SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, 
with some required modifications to function resliently as a service, and for 
smooth integration. However, the TaskFlow flow controls are insufficient for 
Mistral use cases.

Details discussed on other thirds.
The prototype scope - 
[0https://etherpad.openstack.org/p/mistral-taskflow-prototype]; code and 
discussion - [1https://github.com/enykeev/mistral/pull/1] and techical 
highlights - 
[2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html].

DETAILS:

1) Embedding TaskFlow inside Mistral:
* Required: make the engine lazy 
[3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html], 
[4http://paste.openstack.org/show/75389/].This is required to support 
long-running delegates and not loose tasks when engine manager process restarts.

* Persistence: need clarity how to replace or mix-in TaskFlow persistence with 
Mistral persistence. Renat is taking a look.

* Declaring Flows in YAML DSL: done for simplest flow. Need to prototype for 
data flow. Rich flow controls are missing in TaskFlow for a representative 
prototype.

* ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both Mistral 
and TaskFlow implementations work. But we shall resolve the overlap.

* Ignored for now - unlikely any risks: Keystone integration, Mistral event 
scheduler, Mistral declarative services and action definition.

2) TaskFlow library features
* Must: flow control - conditional transitions, references, expression 
evaluation, to express real-life workflows 
[5https://github.com/dzimine/mistral-workflows/tree/add-usecases]. The 
required flow control primitives are 1) repeater 2) flow in flow 3) direct 
transition 4) conditional transition 5) multiple data. TaskFlow has 1) and 2), 
need to add 3/4/5.

* Other details and smaller requests are in the discussion 
[1https://github.com/enykeev/mistral/pull/1]

3) Next Steps proposed:
* Mistal team: summarize the requirements discussed and agreed on 
[2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html] 
and 
[3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html]
* Mistral team: code sample (tests?) on how Mistral would like to consume 
TaskFlow lazy engine
* Taskflow team: Provide a design for alternative TaskExecutor approach 
(prototypes, boxes, arrows, crayons :))
* Decide on lazy engine
* Move the discussion on other elements on integration.

References:
[0] The scope of the prototype: 
https://etherpad.openstack.org/p/mistral-taskflow-prototype
[1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1
[2] Techical summary 
http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html
[3] Email discussion on TaskFlow lazy eninge 
http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html
[4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/
[5] Use cases https://github.com/dzimine/mistral-workflows/tree/add-usecases
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [mistral] Formalizing mistral task policy.

2014-04-10 Thread Manas Kelshikar

Hi everyone,

Some offline conversations and thoughts yielded a notion of providing a
formal policy/transition backing to on-error, on-success and on-finish. I
have a few thoughts in and etherpad (see
https://etherpad.openstack.org/p/mistral_task_policy).

The headliner : on-error, on-success and on-finish should be more complex
than defining what task to call next.

Look forward to the comments.

Thanks,
Manas
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary

2014-04-10 Thread Kirill Izotov

Hi everyone,

This is a summary to the prototype integration we did not too long ago: 
http://github.com/enykeev/mistral/pull/1. Hope it would shed some light on the 
aspects of the integration we are struggling with.

There is a possibility to build Mistral on top of TaskFlow as a library, but in 
order to meet the requirements dictated by Mistral users and use cases, both 
Mistral and TaskFlow should change.

There are two main sides of the story. One is engine. The other is flow control 
capabilities. 

1) THE ENGINE 
The current TaskFlow implementation of engine doesn't fit Mistral needs because 
it is synchronous, it blocks the thread, it requires us to store the reference 
to the particular engine to be able to get its status and suspend the execution 
and it lacks long-running task compatibility. To fix this problem in a solid 
and maintainable way, we need to split the engine into its synchronous and 
asynchronous counterparts.

Lazy engine should be async and atomic, it should not have its own state, 
instead it should rely on some kind of global state (db or in-memory, depending 
on a type of application). It should have at least two methods: run and 
task_complete. Run method should calculate the first batch of tasks and 
schedule them for executing (either put them in queue or spawn the threads). 
Task_complete should mark a certain task to be completed and then schedule the 
next batch of tasks that became available due to resolution of this one.

The desired use of lazy engine in Mistral is illustrated here: 
https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign#Big_Picture. 
It should support long running tasks and survive engine process restart without 
loosing the state of the running actions. So it must be passive (lazy) and 
persistent. 

On Mistral side we are using Lazy engine by patching async.run directly to the 
API (or engine queue) and async.task_complete to the worker queue result 
channel (and the API for long running tasks). We are still sharing the same 
graph_analyzer, but instead of relying on loop and Futures, we are handling the 
execution ourselves in a scalable and robust way.

Then, on top of it you can build a sync engine by introducing Futures. You are 
using async.run() to schedule tasks by transforming them to Futures and then 
starting a loop, checking Futures for completion and sending their results to 
async.task_complete() which would produce even more Futures to check over. Just 
the same way TaskFlow do it right now.

The reason I'm proposing to extract Futures from async engine is because they 
won't work if we have multiple engine processes that should handle the task 
results concurrently (and without that there will be no scalability).

2) THE FLOW CONTROL CAPABILITIES

Since we treat TaskFlow as a library we expect them to provide us with a number 
of primitives to build our workflow with them. Most important of them to us for 
the moment are Direct Transitions, and Conditional Transitions. 

The current implementation of flow transitions in TaskFlow are built on top of 
data flow dependencies where each task provides some data to the flow and 
requires some data to be present prior being executed. In other words, you are 
starting to build your flow tree from the last task through the first one by 
adding their requirements to the tree. All the tasks of successfully finished 
flow should be successfully finished too. If one of the tasks finishes with 
error, the whole flow will be reverted back to its initial state 
unconditionally.

At the same time, Mistral use cases require direct control on the order of the 
task execution, with top-to-bottom scheme where the next task will be 
determined based on the results of the execution of the current one. This way 
to successfully finish a flow you don't have to execute all tasks in it. 
Besides, the error in execution of a particular task may cause execution of 
another one. The workflow examples (in pseudo DSL) are here: 
https://github.com/dzimine/mistral-workflows/tree/add-usecases

There is also a handful of small issues, but these two differences cover most 
basic parts of TaskFlow thus block us from integration and require substantial 
changes in TaskFlow engine design. Inability to make such changes will directly 
result in Mistral not being able to meet its requirements and thus rendering 
the whole project useless for its users.

-- 
Kirill Izotov

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [mistral] Community meeting reminder - 04/07/2014

2014-04-07 Thread Renat Akhmerov

Hi,

This is a reminder that we have another community meeting today at 
#openstack-meeting at 16.00 UTC.

Agenda:
Review action items
Current status (quickly by team members)
POC demo scenario readiness and ways to improve it
TaskFlow integration status
Open discussion

It can also be found at https://wiki.openstack.org/wiki/Meetings/MistralAgenda 
as well as the links to the previous meeting minutes and logs.

Renat Akhmerov
@ Mirantis Inc.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [mistral] Community meeting minutes - 04/07/2014

2014-04-07 Thread Renat Akhmerov

Thanks for joining today’s meeting!

As usually,

Minutes: 
http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-04-07-16.00.html
Full log: 
http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-04-07-16.00.log.html

Let’s meet next time on April 14th.

Renat Akhmerov
@ Mirantis Inc.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks

2014-04-04 Thread Renat Akhmerov


On 04 Apr 2014, at 07:33, Kirill Izotov enyk...@stackstorm.com wrote:

 Then, we can make task executor interface public and allow clients to
 provide their own task executors. It will be possible then for Mistral
 to implement its own task executor, or several, and share the
 executors between all the engine instances.
 I'm afraid that if we start to tear apart the TaskFlow engine, it would 
 quickly become a mess to support. Besides, the amount of things left to 
 integrate after we throw out engine might be so low it proof the whole 
 process of integration to be just nominal and we are back to square one. Any 
 way, task execution is the part that least bothers me, both graph action and 
 the engine itself is where the pain will be.

Would love to see something additional (boxedarrows) explaining this approach. 
Sorry, I’m hardly following the idea.

 That is part of our public API, it is stable and good enough. Basically,
 I don't think this API needs any major change.
 
 But whatever should and will be done about it, I daresay all that work
 can be done without affecting API more then I described above.
 
 I completely agree that we should not change the public API of the sync 
 engine, especially the one in helpers. What we need is, on the contrary, a 
 low level construct that would do the number of things i stated previously, 
 but will be a part of public API of TaskFlow so we can be sure it would work 
 exactly the same way it worked yesterday.

I’m 99.9% sure we’ll have to change API because all we’ve been discussing 
so far made me think this is a key point going implicitly through all our 
discussions: without have a public method like “task_done” we won’t build truly 
passive/async execution model. And it doesn’t matter wether it uses futures, 
callbacks or whatever else inside.

And again, just want to repeat. If we will be able to deal with all the 
challenges that passive/async execution model exposes then other models can be 
built trivially on top of it.

@Ivan,

Thanks for joining the conversation. Looks like we really need your active 
participation for you’re the one who knows all the TF internals and concepts 
very well. As for what you wrote about futures and callbacks, it would be 
helpful to see some illustration of your idea.

Renat Akhmerov
@ Mirantis Inc.___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Next crack at real workflows

2014-04-04 Thread Renat Akhmerov

Dmitri, nice work, will research them carefully early next week. I would ask 
other folks to do the same (especially Nikolay).

Renat Akhmerov
@ Mirantis Inc.

On 03 Apr 2014, at 06:22, Dmitri Zimine d...@stackstorm.com wrote:

 Two more workflows drafted - cloud cron, and lifecycle, version 1. 
 
 The mindset questions are: 
 1) is DSL syntax expressive, and capable and convenient to handle real use 
 cases?
 2) most importantly: what are the implied workflow capabilities which make it 
 all work? 
  
 * Take a look here: 
 https://github.com/dzimine/mistral-workflows/tree/add-usecases
 
 * Leave your comments - generally, or  line-by-line, in the pull request  
 https://github.com/dzimine/mistral-workflows/pull/1/files
 
 * Fork, do your own modifications and do another pull request. 
 
 * Or just reply with your comments  in email (lazy option :)) 
 
 NOTE: please keep this thread for specific comments on DSL and workflow 
 capabilities, create another thread if changing topic. Thanks! 
 
 DZ 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks

2014-04-03 Thread Renat Akhmerov


On 03 Apr 2014, at 12:49, Kirill Izotov enyk...@stackstorm.com wrote:

 Actually, the idea to make the concept more expandable is exactly my point =) 
 Mistral's Workbook is roughly the same as TaskFlow Flow and have nothing to 
 do with TaskFlow LogBook. The only major difference in case of Workbook is 
 that we have to store it after it was created using API or DSL. I very much 
 like the idea to inherit form basic Flow and add serialization and additional 
 metadata on top of it, but the concept of retrying within a Workbook\Flow is 
 not exactly what we have in mind. Instead of repeating the Flow, we are 
 repeating the Task (which has the same differences between Mistral and 
 TaskFlow as Workbook has) and to use the trick you have proposed we would 
 have to translate the Task into a Flow. How much sense does it make in 
 respect to expandability and generality?

Just one clarification. Workbook is a wider abstraction. Along with workflow 
itself it includes action definitions, triggers (even though it’s still 
controversial even within the team itself). Potentially it may include 
something else. So having a workbook in TaskFlow with strictly defined 
behaviour is not a good decision.

 Ya, I think u are hitting this issue, if a worker acks a message saying its 
 'working on it' then the worker dies, this is where a message queue ack 
 model won't really help in solving, the engine will believe that the worker 
 is working on it (since hey the worker acked it) but the engine will really 
 have no idea that the worker is actually dead. Of course timeouts can be 
 used to get around this but imho that’s really un-elegant when something 
 like zookeeper (or similar) can be used to be notified of this worker death 
 immediately; which means no timeouts are needed at all, and this separate 
 process or engine can be notified of this death (and resolve it 
 immediately). I don't believe this kind of 'liveness' connection is possible 
 with rabbit (we also must think of other things besides rabbit, like qpid 
 and such) but I might be wrong, I thought once a worker acks a message then 
 whoever receives that ack will believe the work has started and will be 
 finished someday in the future (aka, no connection that the work is actually 
 in progress). Anyways, we can discuss this more since I think its a common 
 confusion point :-)
 As far as i understand, what Renat is proposing here is not to acknowledge 
 the message until it was successfully executed. I'm not sure how exactly 
 RabbitMQ will react in that case, though you are right in the idea that we 
 must investigate the other solutions more carefully.

Joshua, I think your understanding of how it works is not exactly correct (or 
my understanding actually :)). Let’s talk in IRC, we’ll explain it.

I’d suggest we talk in IRC on Thursday at 8.30 pm (Friday 10.30 in our 
timezone).

Renat Akhmerov
@ Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral][Taskflow] meet on IRC to talk over TaskFlow/Mistral integration

2014-04-03 Thread Dmitri Zimine

IRC to discuss http://tinyurl.com/k3s2gmy

Joshua, 2000 UTC doesn't quite work for Renat and Kirill (3 am their time). 

The overlap is: 
PST (UTC-7) UTC NOVT (UTC+7)

04pm (16:00)11pm (23:00)6am (06:00)
10pm (22:00)05am (05:00)12pm (12:00)

Kirill's pref is 3am UTC, early is ok, if needed. 
@Joshua can you do 3 am UTC (8 pm local?)
@Renat? 

Can we pencil 3:00 UTC on #openstack-mistral and adjust for Renat if needed?

DZ 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks

2014-04-03 Thread Ivan Melnikov

I'm trying to catch up this rather long and interesting discussion,
sorry for somewhat late reply.

I can see aspects of 'lazy model' support in TaskFlow:
- how tasks are executed and reverted
- how flows are run
- how engine works internally

Let me address those aspects separately.

== Executing and reverting tasks ==

I think that should be done via different interface then running a flow
(or scheduling it to run), as it is completely different thing. In
current TaskFlow this interface is called task executor:
https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/executor.py#L57

That is actually how our WorkerBasedEngine was implemented: it's the
same engine with special task executor that schedules tasks on worker
instead of running task code locally.

Task executors are not aware of flows by design, all they do is
executing and reverting tasks. That means that task executors can be
easily shared between engines if that's wanted.

Current TaskExecutorBase interface uses futures (PEP 3148-like). When I
proposed it, futures looked like good tool for the task at hand (see
e.g. async task etherpad
https://etherpad.openstack.org/p/async-taskflow-tasks)

Now it may be time to reconsider that: having one future object per
running task may become a scalability issue. It may be worth to use
callbacks instead. It should not be too hard to refactor current engine
for that. Also, as TaskExecutorBase is an internal API, there should not
be any compatibility issues.

Then, we can make task executor interface public and allow clients to
provide their own task executors. It will be possible then for Mistral
to implement its own task executor, or several, and share the
executors between all the engine instances.

You can call it a plan;)

== Running flows ==

To run the flow TaskFlow client uses engine interface; also, there are
few of helper functions provided for convenience:

http://docs.openstack.org/developer/taskflow/engines.html#module-taskflow.engines.base
http://docs.openstack.org/developer/taskflow/engines.html#creating-engines

That is part of our public API, it is stable and good enough. Basically,
I don't think this API needs any major change.

Maybe it worth to add function or method to schedule running flow
without actually waiting for flow completion (at least, it was on my top
secret TODO list for quite a long time).

== Engine internals ==

Each engine eats resources, like thread it runs on; using these
resources to run one flow only is somewhat wasteful. Some work is
already planned to address this situation (see e.g.
https://blueprints.launchpad.net/taskflow/+spec/share-engine-thread).
Also, it might be good idea to implement different 'type' of engine to
support 'lazy' model, as Joshua suggests.

But whatever should and will be done about it, I daresay all that work
can be done without affecting API more then I described above.

--
WBR,
Ivan A. Melnikov

... tasks must flow ...

On 02.04.2014 01:51, Dmitri Zimine wrote:
Even more responses inline :)
[...]

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks

2014-04-03 Thread Joshua Harlow

Thank Ivan,

This does seem like a possible way forward also.

I'd be interesting to see what/how callbacks would work vs. futures and
what extension point we could provide to mistral for task execution (maybe
there task executor would complete by doing a call to some service, not
amqp for example?).

Maybe some example/POC code would help all :-)

-Josh

-Original Message-
From: Ivan Melnikov imelni...@griddynamics.com
Date: Thursday, April 3, 2014 at 12:04 PM
To: OpenStack Development Mailing List (not for usage questions)
openstack-dev@lists.openstack.org, Joshua Harlow harlo...@yahoo-inc.com
Subject: Re: [openstack-dev] [Mistral] How Mistral handling long running
delegate tasks


I'm trying to catch up this rather long and interesting discussion,
sorry for somewhat late reply.

I can see aspects of 'lazy model' support in TaskFlow:
- how tasks are executed and reverted
- how flows are run
- how engine works internally

Let me address those aspects separately.

== Executing and reverting tasks ==

I think that should be done via different interface then running a flow
(or scheduling it to run), as it is completely different thing. In
current TaskFlow this interface is called task executor:
https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_
engine/executor.py#L57

That is actually how our WorkerBasedEngine was implemented: it's the
same engine with special task executor that schedules tasks on worker
instead of running task code locally.

Task executors are not aware of flows by design, all they do is
executing and reverting tasks. That means that task executors can be
easily shared between engines if that's wanted.

Current TaskExecutorBase interface uses futures (PEP 3148-like). When I
proposed it, futures looked like good tool for the task at hand (see
e.g. async task etherpad
https://etherpad.openstack.org/p/async-taskflow-tasks)

Now it may be time to reconsider that: having one future object per
running task may become a scalability issue. It may be worth to use
callbacks instead. It should not be too hard to refactor current engine
for that. Also, as TaskExecutorBase is an internal API, there should not
be any compatibility issues.

Then, we can make task executor interface public and allow clients to
provide their own task executors. It will be possible then for Mistral
to implement its own task executor, or several, and share the
executors between all the engine instances.

You can call it a plan;)

== Running flows ==

To run the flow TaskFlow client uses engine interface; also, there are
few of helper functions provided for convenience:

http://docs.openstack.org/developer/taskflow/engines.html#module-taskflow.
engines.base
http://docs.openstack.org/developer/taskflow/engines.html#creating-engines

That is part of our public API, it is stable and good enough. Basically,
I don't think this API needs any major change.

Maybe it worth to add function or method to schedule running flow
without actually waiting for flow completion (at least, it was on my top
secret TODO list for quite a long time).

== Engine internals ==

Each engine eats resources, like thread it runs on; using these
resources to run one flow only is somewhat wasteful. Some work is
already planned to address this situation (see e.g.
https://blueprints.launchpad.net/taskflow/+spec/share-engine-thread).
Also, it might be good idea to implement different 'type' of engine to
support 'lazy' model, as Joshua suggests.

But whatever should and will be done about it, I daresay all that work
can be done without affecting API more then I described above.

-- 
WBR,
Ivan A. Melnikov

... tasks must flow ...


On 02.04.2014 01:51, Dmitri Zimine wrote:
 Even more responses inline :)
[...]


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks

2014-04-03 Thread Kirill Izotov

 Then, we can make task executor interface public and allow clients to
 provide their own task executors. It will be possible then for Mistral
 to implement its own task executor, or several, and share the
 executors between all the engine instances.

I'm afraid that if we start to tear apart the TaskFlow engine, it would quickly 
become a mess to support. Besides, the amount of things left to integrate after 
we throw out engine might be so low it proof the whole process of integration 
to be just nominal and we are back to square one. Any way, task execution is 
the part that least bothers me, both graph action and the engine itself is 
where the pain will be.

 That is part of our public API, it is stable and good enough. Basically,
 I don't think this API needs any major change.


 But whatever should and will be done about it, I daresay all that work
 can be done without affecting API more then I described above.



I completely agree that we should not change the public API of the sync engine, 
especially the one in helpers. What we need is, on the contrary, a low level 
construct that would do the number of things i stated previously, but will be a 
part of public API of TaskFlow so we can be sure it would work exactly the same 
way it worked yesterday.

--  
Kirill Izotov


пятница, 4 апреля 2014 г. в 2:04, Ivan Melnikov написал:

  
 I'm trying to catch up this rather long and interesting discussion,
 sorry for somewhat late reply.
  
 I can see aspects of 'lazy model' support in TaskFlow:
 - how tasks are executed and reverted
 - how flows are run
 - how engine works internally
  
 Let me address those aspects separately.
  
 == Executing and reverting tasks ==
  
 I think that should be done via different interface then running a flow
 (or scheduling it to run), as it is completely different thing. In
 current TaskFlow this interface is called task executor:
 https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/executor.py#L57
  
 That is actually how our WorkerBasedEngine was implemented: it's the
 same engine with special task executor that schedules tasks on worker
 instead of running task code locally.
  
 Task executors are not aware of flows by design, all they do is
 executing and reverting tasks. That means that task executors can be
 easily shared between engines if that's wanted.
  
 Current TaskExecutorBase interface uses futures (PEP 3148-like). When I
 proposed it, futures looked like good tool for the task at hand (see
 e.g. async task etherpad
 https://etherpad.openstack.org/p/async-taskflow-tasks)
  
 Now it may be time to reconsider that: having one future object per
 running task may become a scalability issue. It may be worth to use
 callbacks instead. It should not be too hard to refactor current engine
 for that. Also, as TaskExecutorBase is an internal API, there should not
 be any compatibility issues.
  
 Then, we can make task executor interface public and allow clients to
 provide their own task executors. It will be possible then for Mistral
 to implement its own task executor, or several, and share the
 executors between all the engine instances.
  
 You can call it a plan;)
  
 == Running flows ==
  
 To run the flow TaskFlow client uses engine interface; also, there are
 few of helper functions provided for convenience:
  
 http://docs.openstack.org/developer/taskflow/engines.html#module-taskflow.engines.base
 http://docs.openstack.org/developer/taskflow/engines.html#creating-engines
  
 That is part of our public API, it is stable and good enough. Basically,
 I don't think this API needs any major change.
  
 Maybe it worth to add function or method to schedule running flow
 without actually waiting for flow completion (at least, it was on my top
 secret TODO list for quite a long time).
  
 == Engine internals ==
  
 Each engine eats resources, like thread it runs on; using these
 resources to run one flow only is somewhat wasteful. Some work is
 already planned to address this situation (see e.g.
 https://blueprints.launchpad.net/taskflow/+spec/share-engine-thread).
 Also, it might be good idea to implement different 'type' of engine to
 support 'lazy' model, as Joshua suggests.
  
 But whatever should and will be done about it, I daresay all that work
 can be done without affecting API more then I described above.
  
 --  
 WBR,
 Ivan A. Melnikov
  
 ... tasks must flow ...
  
  
 On 02.04.2014 01:51, Dmitri Zimine wrote:
  Even more responses inline :)
  
 [...]
  
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org)
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
  
  


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks

2014-04-02 Thread Renat Akhmerov

On 02 Apr 2014, at 13:38, Kirill Izotov enyk...@stackstorm.com wrote:

 I agree that we probably need new engine for that kind of changes and, as 
 Renat already said in another thread, lazy model seems to be more basic and 
 it would be easier to build sync engine on top of that rather than other way 
 around. Yes, it will entail a lot of changes in engines that are currently 
 here, but it seems like the only way to get something that would fit us both.
 
 Since it seems like we are getting some kind of agreement here, we should 
 probably start shifting to the plane where we discuss the design of the new 
 engine, rather than the need of one. The idea has been spread over to many 
 places, so i'll try to gather it back. 
 
 Lazy engine should be async and atomic, it should not have its own state, 
 instead it should rely on on some kind of global state (db or in-memory, 
 depending on a type of application). I should have at least two methods: run 
 and task_complete. Run method should calculate the first batch of tasks and 
 schedule them to execution (either put them in queue or spawn the threads... 
 or send a pidgin, i insist =). Task_complete should mark a certain task to be 
 completed and then schedule the next batch of tasks that became available due 
 to resolution of this one.
 
 Then, on top of it you can build sync engine by introducing Futures. You are 
 using async.run() to schedule the tasks by transforming them to Futures and 
 then starting a loop, checking Futures for completion and sending their 
 results to async.task_complete() which would produce even more Futures to 
 check over. Just the same way TaskFlow do it right now.
 
 On the Mistral side we are using Lazy engine by patching async.run to the API 
 (or engine queue) and async.task_complete to the worker queue result channel 
 (and the API for long running tasks). We still sharing the same 
 graph_analyzer, but instead of relying on loop and Futures, we are handling 
 execution in a scalable and robust way.
 
 The reason i'm proposing to extract Futures from async engine is because they 
 won't work if we have multiple engines that should handle the task results 
 concurrently (and without that there will be no scalability).

I agree with all the major points here. It sounds like a plan.

 What i see is bothering Joshua is that worker that handles long running task 
 may just die in a process and there is no way for us to know for sure is it 
 still working or not. It is more important for sync engine because without 
 that it would just hang forever (though it is not clear do sync engine needs 
 long running tasks at all?). In case of sync engine, the role of watch-dog 
 may be performed by the loop, while in case of Mistral it might be a separate 
 process (though i bet there should be something for resending the message in 
 the RabbitMQ itself). The common solution is not necessary here.


As for not loosing messages (resending them to the queue) that’s why I 
mentioned message acknowledgement ([0]). Rabbit does it automatically if a 
message that has been pulled out of queue should be acknowledged and the 
corresponding rabbit client connection dies before the client acknowledges the 
message (at this point there’s a guarantee it’s been processed). It works this 
way now in Mistral. A slight problem with the current Mistral implementation is 
that engine works as follows when starting a workflow:

1.
Start DB Transaction
Make all smart logic and DB operations (updating persistent state of 
tasks and execution)
Commit DB Transaction
2. Submit task messages to MQ.

So there’s a window between steps 1 and 2 when if the system crashes DB state 
won’t be consistent with the state of the MQ. From a worker perspective looks 
like this problem doesn’t exist (at least we thought through it and didn’t find 
any synchronisation windows). Initially step 2 was a part of step 1 (a part of 
DB TX) and it would solve the problem but it’s generally a bad pattern to have 
interprocess communication inside DB TX so we got rid of it.

So this seems to me the only problem with the current architecture, we left it 
as it was and were going to get back to it after POC. I assume this might be 
describing the reason why Josh keeps talking about ‘jobboard’ thing, but I just 
don’t have enough details at this point to say if that’s what we need or not.

One solution I see here is just to consider this situation exceptional from 
workflow execution model perspective and apply policies to tasks that are 
persisted in DB but not submitted to MQ (what you guys discussed about retries 
and all that stuff..). So our watch-dog process could just resubmit tasks that 
have been hanging in DB longer than a configured period with the state IDLE. 
Something like that.

[0] http://www.rabbitmq.com/tutorials/tutorial-two-python.html

Renat Akhmerov
@ Mirantis Inc.

___
OpenStack-dev mailing list

[openstack-dev] [Mistral] Next crack at real workflows

2014-04-02 Thread Dmitri Zimine

Two more workflows drafted - cloud cron, and lifecycle, version 1. 

The mindset questions are: 
1) is DSL syntax expressive, and capable and convenient to handle real use 
cases?
2) most importantly: what are the implied workflow capabilities which make it 
all work? 
 
* Take a look here: 
https://github.com/dzimine/mistral-workflows/tree/add-usecases

* Leave your comments - generally, or  line-by-line, in the pull request  
https://github.com/dzimine/mistral-workflows/pull/1/files

* Fork, do your own modifications and do another pull request. 

* Or just reply with your comments  in email (lazy option :)) 

NOTE: please keep this thread for specific comments on DSL and workflow 
capabilities, create another thread if changing topic. Thanks! 

DZ ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks

2014-04-01 Thread Dmitri Zimine

Even more responses inline :)
On Mar 31, 2014, at 8:44 PM, Joshua Harlow harlo...@yahoo-inc.com wrote:

 More responses inline :)

 From: Dmitri Zimine d...@stackstorm.com
 Date: Monday, March 31, 2014 at 5:59 PM
 To: Joshua Harlow harlo...@yahoo-inc.com
 Cc: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [Mistral] How Mistral handling long running 
 delegate tasks

 Inline...
 On Mar 27, 2014, at 5:10 PM, Joshua Harlow harlo...@yahoo-inc.com wrote:

 Thanks for the description!

 The steps here seem very much like what a taskflow engine does (which is 
 good).

 To connect this to how I think could work in taskflow.
 Someone creates tasks/flows describing the work to-be-done (converting a 
 DSL - taskflow tasks/flows/retry[1] objects…)
 On execute(workflow) engine creates a new workflow execution, computes the 
 first batch of tasks, creates executor for those tasks (remote, local…) and 
 executes those tasks.
 Waits for response back from futures returned from executor.
 Receives futures responses (or receives new response DELAY, for example), 
 or exceptions…
 Continues sending out batches of tasks that can be still be executing (aka 
 tasks that don't have dependency on output of delayed tasks).
 If any delayed tasks after repeating #2-5 as many times as it can, the 
 engine will shut itself down (see http://tinyurl.com/l3x3rrb).
 Why would engine treat long running tasks differently? The model Mistral 
 tried out is the engine sends the batch of tasks and goes asleep; the 
 'worker/executor' is calling engine back when the task(s) complete. Can it 
 be applied 

 Not all ways of using taskflow I think want to go 'asleep', some active 
 conductors I would think actually stay 'awake', that’s why I was thinking 
 this would be a comprimise, the usage of 'DELAY' (via exception or return) 
 would allow a task to say it is going to be finished sometime in the future 
 (btw started this @ https://review.openstack.org/#/c/84277/ --- WIP, likely 
 won't go into taskflow 0.2 but seems like a reasonable start on something 
 like this proposal). To me offering just the going asleep way means that all 
 users of taskflow need to have some type of entrypoint (rest, mq, other…) 
 that restarts the engine on result finishing. I'd rather not force that model 
 onto all users (since that seems inappropriate).
I understand and agree with not enforcing this model on all users. On a flip 
side, TaskFlow enforces the current model on Mistral :) It calls for supporting 
more models (to your earlier point of 3rd mode of operations). The BP proposal 
is a step in the direction but still a compromise. How can we support both 
'active' and 'passive' models, sharing the same 'library' - of all things 
responsible for control flow and data flow?

Here is a strawman (DISCLAIMER: this is sharing ideas and brainstorming: I 
don't know enough of TaksFlow to suggest): 
1) break the run loop: add engine.run_async() method which instead of looping 
the tasks till the flow completes (http://tinyurl.com/nou2em9), only schedules 
the first batch and returns; 
https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L61-L68

2) introduce engine.task_complete(…) method that marks the current task 
complete, computes the next tasks ready for execution, schedules them and 
returns; loosely, doing this part: 
https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L75-L85

3) make executor signal engine back by calling engine.task_complete(…): 
executor.execute_task() submits the task to the worker queue and calls 
engine.task_complete(status, results, etc.) instead of waiting for task 
completion and returning the results. 

And looks like it all can be done as a third lazy_engine, without messing the 
current two engines? 

 On delay task finishing some API/webhook/other (the mechanism imho 
 shouldn't be tied to webhooks, at least not in taskflow, but should be left 
 up to the user of taskflow to decide how to accomplish this) will be/must 
 be responsible for resuming the engine and setting the result for the 
 previous delayed task.
 Oh no, webhook is the way to expose it to 3rd party system. From the library 
 standpoint it's just an API call. 

 One can do it even now by getting the appropriate Flow_details, 
 instantiating and engine (flow, flow_details) and running it to continue 
 from where it left out. Is it how you mean it? But I keep on dreaming of a 
 passive version of TaskFlow engine which treats all tasks the same and 
 exposes one extra method - handle_tasks. 

 I was thinking that, although I'm unsure on the one extra method idea, can u 
 explain more :) What would handle_tasks do? Restart/resume the engine 
 (basically the work u stated 'getting the appropriate Flow_details, 
 instantiating and engine (flow, flow_details) and running

Re: [openstack-dev] [Mistral][TaskFlow] Long running actions

2014-04-01 Thread Dmitri Zimine


On Apr 1, 2014, at 3:43 AM, Renat Akhmerov rakhme...@mirantis.com wrote:
 On 25 Mar 2014, at 01:51, Joshua Harlow harlo...@yahoo-inc.com wrote:
 
 The first execution model I would call the local execution model, this model 
 involves forming tasks and flows and then executing them inside an 
 application, that application is running for the duration of the workflow 
 (although if it crashes it can re-establish the task and flows that it was 
 doing and attempt to resume them). This could also be what openstack 
 projects would call the 'conductor' approach where nova, ironic, trove have 
 a conductor which manages these long-running actions (the conductor is 
 alive/running throughout the duration of these workflows, although it may be 
 restarted while running). The restarting + resuming part is something that 
 openstack hasn't handled so gracefully currently, typically requiring either 
 some type of cleanup at restart (or by operations), with taskflow using this 
 model the resumption part makes it possible to resume from the last saved 
 state (this connects into the persistence model that taskflow uses, the 
 state transitions, how execution occurrs itself...). 
 
 The second execution model is an extension of the first, whereby there is 
 still a type of 'conductor' that is managing the life-time of the workflow, 
 but instead of locally executing tasks in the conductor itself tasks are now 
 executed on remote-workers (see http://tinyurl.com/lf3yqe4
 ). The engine currently still is 'alive' for the life-time of the execution, 
 although the work that it is doing is relatively minimal (since its not 
 actually executing any task code, but proxying those requests to others 
 works). The engine while running does the conducting of the remote-workers 
 (saving persistence details, doing state-transtions, getting results, 
 sending requests to workers…).
 
 These two execution models are special cases of what you call “lazy execution 
 model” (or passive as we call it). To illustrate this idea we can take a look 
 at the first sequence diagram at [0], we basically will see the following 
 interaction:
 
 1) engine --(task)-- queue --(task)-- worker
 2) execute task
 3) worker --(result)-- queue --(result)-- engine
 
 This is how TaskFlow worker based model works.
 
 If we loosen the requirement in 3) and assume that not only worker can send a 
 task result back to engine we’ll get our passive model. Instead of worker it 
 can be anything else (some external system) that knows how to make this call. 
 A particular way is not too important, it can be a direct message or it can 
 be hidden behind an API method. In Mistral it’s now a REST API method however 
 we’re about to decouple engine from REST API so that engine is a standalone 
 process and listens to a queue. So worker-based model is basically the same 
 with the only strict requirement that only worker sends a result back.
 
 In order to implement local execution model on top of “lazy execution model” 
 we just need to abstract a transport (queue) so that we can use an in-process 
 transport. That’s it. It’s what Mistral already has implemented. Again, we 
 see that “lazy execution model” is more universal.
 
 IMO this “lazy execution model” should be the main execution model that 
 TaskFlow supports, others can be easily implemented on top of it. But the 
 opposite assertion is wrong. IMO this is the most important obstacle in all 
 our discussions, the reason why we don’t always understand each other well 
 enough. I know it may be a lot of work to shift a paradigm in TaskFlow team 
 but if we did that we would get enough freedom for using TaskFlow in lots of 
 cases.
 
 Let me know what you think. I might have missed something.

DZ: Interesting idea! So that other models of execution are based on lazy 
execution model? TaskFlow implements this, we can use it, and for other clients 
more convenient higher level execution models are provided? Interesting. Makes 
sense.
@Joshua? @Kirill? Others? 

 
 === HA ===
 
 So this is an interesting question, and to me is strongly connected to how 
 your engines are executing (and the persistence and state-transitions that 
 they go through while running). Without persistence of state and transitions 
 there is no good way (a bad way of course can be created, by just redoing 
 all the work, but that's not always feasible or the best option) to 
 accomplish resuming in a sane manner and there is also imho no way to 
 accomplish any type of automated HA of workflows. 
 
 Sure, no questions here.
 
 Let me describe:
 
 When you save the states of a workflow and any intermediate results of a 
 workflow to some database (for example) and the engine (see above models) 
 which is being used (for example the conductor type from above) the 
 application containing that engine may be prone to crashes (or just being 
 powered off due to software upgrades...). Since taskflows key primitives 
 were made to allow for

Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks

Cool, my thoughts added ;)

From: Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com
Date: Tuesday, April 1, 2014 at 2:51 PM
To: Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com
Cc: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral] How Mistral handling long running 
delegate tasks

Even more responses inline :)
On Mar 31, 2014, at 8:44 PM, Joshua Harlow 
harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote:

More responses inline :)

From: Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com
Date: Monday, March 31, 2014 at 5:59 PM
To: Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com
Cc: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral] How Mistral handling long running 
delegate tasks

Inline...
On Mar 27, 2014, at 5:10 PM, Joshua Harlow 
harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote:

Thanks for the description!

The steps here seem very much like what a taskflow engine does (which is good).

To connect this to how I think could work in taskflow.

  1.  Someone creates tasks/flows describing the work to-be-done (converting a 
DSL - taskflow tasks/flows/retry[1] objects…)
  2.  On execute(workflow) engine creates a new workflow execution, computes 
the first batch of tasks, creates executor for those tasks (remote, local…) and 
executes those tasks.
  3.  Waits for response back from 
futureshttp://docs.python.org/dev/library/concurrent.futures.html returned 
from executor.
  4.  Receives futures responses (or receives new response DELAY, for example), 
or exceptions…
  5.  Continues sending out batches of tasks that can be still be executing 
(aka tasks that don't have dependency on output of delayed tasks).
  6.  If any delayed tasks after repeating #2-5 as many times as it can, the 
engine will shut itself down (see http://tinyurl.com/l3x3rrb).

Why would engine treat long running tasks differently? The model Mistral tried 
out is the engine sends the batch of tasks and goes asleep; the 
'worker/executor' is calling engine back when the task(s) complete. Can it be 
applied

Not all ways of using taskflow I think want to go 'asleep', some active 
conductors I would think actually stay 'awake', that’s why I was thinking this 
would be a comprimise, the usage of 'DELAY' (via exception or return) would 
allow a task to say it is going to be finished sometime in the future (btw 
started this @ https://review.openstack.org/#/c/84277/ --- WIP, likely won't go 
into taskflow 0.2 but seems like a reasonable start on something like this 
proposal). To me offering just the going asleep way means that all users of 
taskflow need to have some type of entrypoint (rest, mq, other…) that restarts 
the engine on result finishing. I'd rather not force that model onto all users 
(since that seems inappropriate).
I understand and agree with not enforcing this model on all users. On a flip 
side, TaskFlow enforces the current model on Mistral :) It calls for supporting 
more models (to your earlier point of 3rd mode of operations). The BP proposal 
is a step in the direction but still a compromise. How can we support both 
'active' and 'passive' models, sharing the same 'library' - of all things 
responsible for control flow and data flow?

To me this is what engines 'types' are for, to support different execution 
models while using as many of the same components as possible (making usage as 
seamless as can be).

Here is a strawman (DISCLAIMER: this is sharing ideas and brainstorming: I 
don't know enough of TaksFlow to suggest):
1) break the run loop: add engine.run_async() method which instead of looping 
the tasks till the flow completes (http://tinyurl.com/nou2em9), only schedules 
the first batch and returns;
https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L61-L68

Seems like a possiblity, although I'd like to make this still a new engine type 
that does this (or maybe the existing engine types wrap this change to retain 
the usage that exists). This just seems like an extension of DELAY thought, 
where at each iteration the engine would return DELAYed, then the caller (the 
one executing) would become aware that this has happened and decide how to 
handle this accordingly. Exposing 'run_async' doesn't still feel right, maybe 
its just wording, but it seems like an 'async/lazy' engine just still has a 
run() method that returns DELAYed more often than an engine that loops and 
never returns DELAYed (or only returns DELAYed if a task causes this to 
happen). To me they are the same run() method, just different implementations 
of run(). Exposing the semantics of running by naming run_async exposes the 
underlying execution abstraction, which imho isn't

Re: [openstack-dev] [Mistral][TaskFlow] Long running actions

Inline responses.

From: Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Tuesday, April 1, 2014 at 3:43 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral][TaskFlow] Long running actions

On 25 Mar 2014, at 01:51, Joshua Harlow 
harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote:

The first execution model I would call the local execution model, this model 
involves forming tasks and flows and then executing them inside an application, 
that application is running for the duration of the workflow (although if it 
crashes it can re-establish the task and flows that it was doing and attempt to 
resume them). This could also be what openstack projects would call the 
'conductor' approach where nova, ironic, trove have a conductor which manages 
these long-running actions (the conductor is alive/running throughout the 
duration of these workflows, although it may be restarted while running). The 
restarting + resuming part is something that openstack hasn't handled so 
gracefully currently, typically requiring either some type of cleanup at 
restart (or by operations), with taskflow using this model the resumption part 
makes it possible to resume from the last saved state (this connects into the 
persistence model that taskflow uses, the state transitions, how execution 
occurrs itself...).

The second execution model is an extension of the first, whereby there is still 
a type of 'conductor' that is managing the life-time of the workflow, but 
instead of locally executing tasks in the conductor itself tasks are now 
executed on remote-workers (see http://tinyurl.com/lf3yqe4
). The engine currently still is 'alive' for the life-time of the execution, 
although the work that it is doing is relatively minimal (since its not 
actually executing any task code, but proxying those requests to others works). 
The engine while running does the conducting of the remote-workers (saving 
persistence details, doing state-transtions, getting results, sending requests 
to workers…).

These two execution models are special cases of what you call “lazy execution 
model” (or passive as we call it). To illustrate this idea we can take a look 
at the first sequence diagram at [0], we basically will see the following 
interaction:

1) engine --(task)-- queue --(task)-- worker
2) execute task
3) worker --(result)-- queue --(result)-- engine

This is how TaskFlow worker based model works.

If we loosen the requirement in 3) and assume that not only worker can send a 
task result back to engine we’ll get our passive model. Instead of worker it 
can be anything else (some external system) that knows how to make this call. A 
particular way is not too important, it can be a direct message or it can be 
hidden behind an API method. In Mistral it’s now a REST API method however 
we’re about to decouple engine from REST API so that engine is a standalone 
process and listens to a queue. So worker-based model is basically the same 
with the only strict requirement that only worker sends a result back.

In order to implement local execution model on top of “lazy execution model” we 
just need to abstract a transport (queue) so that we can use an in-process 
transport. That’s it. It’s what Mistral already has implemented. Again, we see 
that “lazy execution model” is more universal.

IMO this “lazy execution model” should be the main execution model that 
TaskFlow supports, others can be easily implemented on top of it. But the 
opposite assertion is wrong. IMO this is the most important obstacle in all our 
discussions, the reason why we don’t always understand each other well enough. 
I know it may be a lot of work to shift a paradigm in TaskFlow team but if we 
did that we would get enough freedom for using TaskFlow in lots of cases.

Everything is a lot of work ;) That’s just how it goes. I think we work through 
it then it's all fine in the end.

I think some of this is being resolved/discussed @ http://tinyurl.com/k3s2gmy

Let me know what you think. I might have missed something.

=== HA ===

So this is an interesting question, and to me is strongly connected to how your 
engines are executing (and the persistence and state-transitions that they go 
through while running). Without persistence of state and transitions there is 
no good way (a bad way of course can be created, by just redoing all the work, 
but that's not always feasible or the best option) to accomplish resuming in a 
sane manner and there is also imho no way to accomplish any type of automated 
HA of workflows.

Sure, no questions here.

Let me describe:

When you save the states of a workflow and any intermediate results of a 
workflow to some database

Re: [openstack-dev] [Mistral][TaskFlow] Long running actions

More inline.

From: Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Tuesday, April 1, 2014 at 2:59 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral][TaskFlow] Long running actions

On Apr 1, 2014, at 3:43 AM, Renat Akhmerov 
rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote:
On 25 Mar 2014, at 01:51, Joshua Harlow 
harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote:

The first execution model I would call the local execution model, this model 
involves forming tasks and flows and then executing them inside an application, 
that application is running for the duration of the workflow (although if it 
crashes it can re-establish the task and flows that it was doing and attempt to 
resume them). This could also be what openstack projects would call the 
'conductor' approach where nova, ironic, trove have a conductor which manages 
these long-running actions (the conductor is alive/running throughout the 
duration of these workflows, although it may be restarted while running). The 
restarting + resuming part is something that openstack hasn't handled so 
gracefully currently, typically requiring either some type of cleanup at 
restart (or by operations), with taskflow using this model the resumption part 
makes it possible to resume from the last saved state (this connects into the 
persistence model that taskflow uses, the state transitions, how execution 
occurrs itself...).

The second execution model is an extension of the first, whereby there is still 
a type of 'conductor' that is managing the life-time of the workflow, but 
instead of locally executing tasks in the conductor itself tasks are now 
executed on remote-workers (see http://tinyurl.com/lf3yqe4
). The engine currently still is 'alive' for the life-time of the execution, 
although the work that it is doing is relatively minimal (since its not 
actually executing any task code, but proxying those requests to others works). 
The engine while running does the conducting of the remote-workers (saving 
persistence details, doing state-transtions, getting results, sending requests 
to workers…).

These two execution models are special cases of what you call “lazy execution 
model” (or passive as we call it). To illustrate this idea we can take a look 
at the first sequence diagram at [0], we basically will see the following 
interaction:

1) engine --(task)-- queue --(task)-- worker
2) execute task
3) worker --(result)-- queue --(result)-- engine

This is how TaskFlow worker based model works.

If we loosen the requirement in 3) and assume that not only worker can send a 
task result back to engine we’ll get our passive model. Instead of worker it 
can be anything else (some external system) that knows how to make this call. A 
particular way is not too important, it can be a direct message or it can be 
hidden behind an API method. In Mistral it’s now a REST API method however 
we’re about to decouple engine from REST API so that engine is a standalone 
process and listens to a queue. So worker-based model is basically the same 
with the only strict requirement that only worker sends a result back.

In order to implement local execution model on top of “lazy execution model” we 
just need to abstract a transport (queue) so that we can use an in-process 
transport. That’s it. It’s what Mistral already has implemented. Again, we see 
that “lazy execution model” is more universal.

IMO this “lazy execution model” should be the main execution model that 
TaskFlow supports, others can be easily implemented on top of it. But the 
opposite assertion is wrong. IMO this is the most important obstacle in all our 
discussions, the reason why we don’t always understand each other well enough. 
I know it may be a lot of work to shift a paradigm in TaskFlow team but if we 
did that we would get enough freedom for using TaskFlow in lots of cases.

Let me know what you think. I might have missed something.

DZ: Interesting idea! So that other models of execution are based on lazy 
execution model? TaskFlow implements this, we can use it, and for other clients 
more convenient higher level execution models are provided? Interesting. Makes 
sense.
@Joshua? @Kirill? Others?

I think this is likely possible, which is simiar to whats in 
http://tinyurl.com/k3s2gmy, engine types can be built from each other (and if 
we wanted to alter the structure that exists in taskflow) then sure. But see 
that message for more of my concerns around exposing that engine API to library 
users (I think it could have its usage in mistral to expose this, but I'm not 
sure its useful for elsewhere, and once its public engine API, its public for a 
very long time).

=== HA ===

So

Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks

2014-04-01 Thread Renat Akhmerov

On 02 Apr 2014, at 05:45, Joshua Harlow harlo...@yahoo-inc.com wrote:

 Possibly, although to me this is still exposing the internal of engines to 
 users who shouldn't care (or only care if they are specifying an engine type 
 that gives them access to these details). Allowing public access to these 
 API's worries me in that they are now the API (which goes back to having an 
 engine type that exposes these, if that’s desired, and if we are willing to 
 accept the consequences of exposing them).

Given all we discussed by now calling it “internal of engines” is not correct 
anymore. If for some cases we know that only workers will be calling this API 
method and we need to protect the workflow execution from occasional calls from 
3rd parties I believe there’s a million ways how to solve this. The simplest 
thing that comes to my mind is just passing a generated token to confirm 
authority to perform this operation.

 Who responds to the timeout though? Isn't that process the watchdog then? 
 Likely the triggering of a timeout causes something to react (in both 
 cases).
 The workflow engine IS this watch-dog (and in Mistral, engine is a single 
 manager for all flow executions, in the prototype we call it engine_manager 
 and I hate this name :)) Engine live in process or as a separate process. 
 And it is passive - executed in a thread of a callee. E.g., in process is 
 either called on messaging event handler thread, or by web method thread.
 
 
 Right, which in mistral is a web-server right (aka, wherever mistral is 
 setup) since the tasks finish by calling a rest-endpoint (or something 
 sitting on MQ?)?

Not exactly right. Currently it’s a web server but we’re about to decouple 
engine and API server. Most of the work is done. Engine is supposed to listen 
to a queue and there may be any number of engines since they are stateless and 
hence what’s behind a web server can be scaled as needed. And actually a web 
server tier can be scaled easily too (assuming we have a loadbalancer in place).

 That may be good enough: when DSL is translated to flow, and the task 
 demands repetition with timeout, it's ok to do this trick under the hood 
 when compiling a flow. 
 flow.add(LinearFlow(subblahblah, 
 retry=XYZ).add(OneTask().add(Timeout(timeout))
 
 
 Yup, in a way most languages compilers do all these types of tricks under the 
 hood (in much much more complicated manners); as long as we retain 'user 
 intention' (aka don't mess how the code executes) we should be able to do any 
 tricks we want (in fact most compliers do many many tricks). To me the same 
 kind of tricks start to become possible after we get the basics right (can't 
 do optimizations, aka -O2, if u don't have basics in the first place).

I’d be careful about this assumption that we can convert DSL to flow, right now 
it’s impossible since we need to add more control flow primitives in TaskFlow. 
But that’s what Kirill described in the prototype description.

Renat Akhmerov
@ Mirantis Inc.___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][TaskFlow] Long running actions

2014-04-01 Thread Renat Akhmerov

On 02 Apr 2014, at 06:00, Joshua Harlow harlo...@yahoo-inc.com wrote:

 More inline.

 From: Dmitri Zimine d...@stackstorm.com
 Reply-To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Date: Tuesday, April 1, 2014 at 2:59 PM
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [Mistral][TaskFlow] Long running actions

 On Apr 1, 2014, at 3:43 AM, Renat Akhmerov rakhme...@mirantis.com wrote:
 On 25 Mar 2014, at 01:51, Joshua Harlow harlo...@yahoo-inc.com wrote:

 The first execution model I would call the local execution model, this 
 model involves forming tasks and flows and then executing them inside an 
 application, that application is running for the duration of the workflow 
 (although if it crashes it can re-establish the task and flows that it was 
 doing and attempt to resume them). This could also be what openstack 
 projects would call the 'conductor' approach where nova, ironic, trove 
 have a conductor which manages these long-running actions (the conductor 
 is alive/running throughout the duration of these workflows, although it 
 may be restarted while running). The restarting + resuming part is 
 something that openstack hasn't handled so gracefully currently, typically 
 requiring either some type of cleanup at restart (or by operations), with 
 taskflow using this model the resumption part makes it possible to resume 
 from the last saved state (this connects into the persistence model that 
 taskflow uses, the state transitions, how execution occurrs itself...). 

 The second execution model is an extension of the first, whereby there is 
 still a type of 'conductor' that is managing the life-time of the 
 workflow, but instead of locally executing tasks in the conductor itself 
 tasks are now executed on remote-workers (see http://tinyurl.com/lf3yqe4
 ). The engine currently still is 'alive' for the life-time of the 
 execution, although the work that it is doing is relatively minimal (since 
 its not actually executing any task code, but proxying those requests to 
 others works). The engine while running does the conducting of the 
 remote-workers (saving persistence details, doing state-transtions, 
 getting results, sending requests to workers…).

 These two execution models are special cases of what you call “lazy 
 execution model” (or passive as we call it). To illustrate this idea we can 
 take a look at the first sequence diagram at [0], we basically will see the 
 following interaction:

 1) engine --(task)-- queue --(task)-- worker
 2) execute task
 3) worker --(result)-- queue --(result)-- engine

 This is how TaskFlow worker based model works.

 If we loosen the requirement in 3) and assume that not only worker can send 
 a task result back to engine we’ll get our passive model. Instead of worker 
 it can be anything else (some external system) that knows how to make this 
 call. A particular way is not too important, it can be a direct message or 
 it can be hidden behind an API method. In Mistral it’s now a REST API 
 method however we’re about to decouple engine from REST API so that engine 
 is a standalone process and listens to a queue. So worker-based model is 
 basically the same with the only strict requirement that only worker sends 
 a result back.

 In order to implement local execution model on top of “lazy execution 
 model” we just need to abstract a transport (queue) so that we can use an 
 in-process transport. That’s it. It’s what Mistral already has implemented. 
 Again, we see that “lazy execution model” is more universal.

 IMO this “lazy execution model” should be the main execution model that 
 TaskFlow supports, others can be easily implemented on top of it. But the 
 opposite assertion is wrong. IMO this is the most important obstacle in all 
 our discussions, the reason why we don’t always understand each other well 
 enough. I know it may be a lot of work to shift a paradigm in TaskFlow team 
 but if we did that we would get enough freedom for using TaskFlow in lots 
 of cases.

 Let me know what you think. I might have missed something.

 DZ: Interesting idea! So that other models of execution are based on lazy 
 execution model? TaskFlow implements this, we can use it, and for other 
 clients more convenient higher level execution models are provided? 
 Interesting. Makes sense.
 @Joshua? @Kirill? Others? 

 I think this is likely possible, which is simiar to whats in 
 http://tinyurl.com/k3s2gmy, engine types can be built from each other (and if 
 we wanted to alter the structure that exists in taskflow) then sure. But see 
 that message for more of my concerns around exposing that engine API to 
 library users (I think it could have its usage in mistral to expose this, but 
 I'm not sure its useful for elsewhere, and once its public engine API, its 
 public for a very long time).

What are we

Re: [openstack-dev] [Mistral] Review reminder

2014-04-01 Thread Renat Akhmerov

LGTM. If others agree we can fix the jenkins job.

Renat Akhmerov
@ Mirantis Inc.



On 02 Apr 2014, at 07:51, Dmitri Zimine d...@stackstorm.com wrote:

 We agreed to review this and if/when OK, fix the murano-cli. Can you guys 
 please take a look? 
 https://review.openstack.org/#/c/81941/
 
 DZ
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][TaskFlow] Long running actions