Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
Ok, no problem :) Renat Akhmerov @ Mirantis Inc. On 16 Apr 2014, at 12:43, Joshua Harlow harlo...@yahoo-inc.com wrote: I have stricken the word micro and language from my vocabulary. Begone evil demons!! Haha :) Sent from my really tiny device... On Apr 15, 2014, at 10:35 PM, Renat Akhmerov rakhme...@mirantis.com wrote: On 16 Apr 2014, at 00:18, Joshua Harlow harlo...@yahoo-inc.com wrote: Decider sounds like it could work also as a name, although it seems from dataflow like work its called a switch or gate, either or I guess. That’s fine. It doesn’t matter too much to me personally. As far as the micro-language: So there are typically 2 types of DSL's that occur, internal and external. An internal DSL is like http://martinfowler.com/bliki/InternalDslStyle.html, taskflow is already a micro-DSL internal to python (mistral is an external DSL[1]). To me there is a drawback of becoming to much of a DSL (internal or external) in that it requires a lot of new learning (imho internal DSLs are easier to pick-up since they take advantage of the surrounding languages capabilities, in this case python). So that’s what I just want to keep in our minds that we need to make it simple *enough*, or we will die a nasty death of complexity :-P [1] http://martinfowler.com/bliki/DomainSpecificLanguage.html Ok, got it. Thanks. I’m just still not sure why you emphasize on that micro-language thing. IMO terms like that can scary people :) In fact, this ‘switch’ (or decider, or whatever) is just an additional API which can be used to alter flow behavior. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
Some notes: Even though we use YAQL now our design is flexible enough to plug other ELs in. If it tells you something in Amazon SWF a component that makes a decision about a further route is called Decider. This discussion about conditionals is surely important but it doesn’t matter too much if we don’t agree on that lazy execution model. Of course I'm trying to make the above not be its own micro-language as much as possible (a switch object starts to act like one, sadly). Why do you think it’s going to be a micro-language? [1] http://www.cs.cmu.edu/~aldrich/papers/onward2009-concurrency.pdf [2] http://www.cs.ucf.edu/~dcm/Teaching/COT4810-Spring2011/Literature/DataFlowProgrammingLanguages.pdf Cool, thanks! ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
I think we agree on the lazy execution model. At least at a high-level, I'd rather not agree on the API's that are exactly exposed until there is an implementation since I've found that agreeing to any type of API's before there is the needed groundwork to make it happen is pretty useless and a waste of energy on everyones part. Decider sounds like it could work also as a name, although it seems from dataflow like work its called a switch or gate, either or I guess. As far as the micro-language: So there are typically 2 types of DSL's that occur, internal and external. An internal DSL is like http://martinfowler.com/bliki/InternalDslStyle.html, taskflow is already a micro-DSL internal to python (mistral is an external DSL[1]). To me there is a drawback of becoming to much of a DSL (internal or external) in that it requires a lot of new learning (imho internal DSLs are easier to pick-up since they take advantage of the surrounding languages capabilities, in this case python). So that’s what I just want to keep in our minds that we need to make it simple *enough*, or we will die a nasty death of complexity :-P [1] http://martinfowler.com/bliki/DomainSpecificLanguage.html From: Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Tuesday, April 15, 2014 at 12:19 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary Some notes: * Even though we use YAQL now our design is flexible enough to plug other ELs in. * If it tells you something in Amazon SWF a component that makes a decision about a further route is called Decider. * This discussion about conditionals is surely important but it doesn’t matter too much if we don’t agree on that lazy execution model. Of course I'm trying to make the above not be its own micro-language as much as possible (a switch object starts to act like one, sadly). Why do you think it’s going to be a micro-language? [1] http://www.cs.cmu.edu/~aldrich/papers/onward2009-concurrency.pdf [2] http://www.cs.ucf.edu/~dcm/Teaching/COT4810-Spring2011/Literature/DataFlowProgrammingLanguages.pdf Cool, thanks! ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
On 16 Apr 2014, at 00:18, Joshua Harlow harlo...@yahoo-inc.com wrote: Decider sounds like it could work also as a name, although it seems from dataflow like work its called a switch or gate, either or I guess. That’s fine. It doesn’t matter too much to me personally. As far as the micro-language: So there are typically 2 types of DSL's that occur, internal and external. An internal DSL is like http://martinfowler.com/bliki/InternalDslStyle.html, taskflow is already a micro-DSL internal to python (mistral is an external DSL[1]). To me there is a drawback of becoming to much of a DSL (internal or external) in that it requires a lot of new learning (imho internal DSLs are easier to pick-up since they take advantage of the surrounding languages capabilities, in this case python). So that’s what I just want to keep in our minds that we need to make it simple *enough*, or we will die a nasty death of complexity :-P [1] http://martinfowler.com/bliki/DomainSpecificLanguage.html Ok, got it. Thanks. I’m just still not sure why you emphasize on that micro-language thing. IMO terms like that can scary people :) In fact, this ‘switch’ (or decider, or whatever) is just an additional API which can be used to alter flow behavior. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
I have stricken the word micro and language from my vocabulary. Begone evil demons!! Haha :) Sent from my really tiny device... On Apr 15, 2014, at 10:35 PM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: On 16 Apr 2014, at 00:18, Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote: Decider sounds like it could work also as a name, although it seems from dataflow like work its called a switch or gate, either or I guess. That’s fine. It doesn’t matter too much to me personally. As far as the micro-language: So there are typically 2 types of DSL's that occur, internal and external. An internal DSL is like http://martinfowler.com/bliki/InternalDslStyle.html, taskflow is already a micro-DSL internal to python (mistral is an external DSL[1]). To me there is a drawback of becoming to much of a DSL (internal or external) in that it requires a lot of new learning (imho internal DSLs are easier to pick-up since they take advantage of the surrounding languages capabilities, in this case python). So that’s what I just want to keep in our minds that we need to make it simple *enough*, or we will die a nasty death of complexity :-P [1] http://martinfowler.com/bliki/DomainSpecificLanguage.html Ok, got it. Thanks. I’m just still not sure why you emphasize on that micro-language thing. IMO terms like that can scary people :) In fact, this ‘switch’ (or decider, or whatever) is just an additional API which can be used to alter flow behavior. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
The more the better :) If seriously, this one is less detailed and rather focuses on high-level things so that everyone can have a high-level understanding of what’s been going on on Mistral/TaskFlow integration. In other words, this email raises question “What” and doesn’t raise “Why?” while [0] goes deeper into details. [0] http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html Renat Akhmerov @ Mirantis Inc. On 12 Apr 2014, at 00:59, Joshua Harlow harlo...@yahoo-inc.com wrote: I'm confused, why is this 2 emails?? http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html Seems better to just have 1 chain, not 2. From: Dmitri Zimine d...@stackstorm.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Friday, April 11, 2014 at 9:55 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary We prototyped Mistral / TaskFlow integration and have a follow-up discussions. SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, with some required modifications to function resliently as a service, and for smooth integration. However, the TaskFlow flow controls are insufficient for Mistral use cases. Details discussed on other thirds. The prototype scope - [0]; code and discussion - [1] and techical highlights - [2]. DETAILS: 1) Embedding TaskFlow inside Mistral: * Required: make the engine lazy [3], [4].This is required to support long-running delegates and not loose tasks when engine manager process restarts. * Persistence: need clarity how to replace or mix-in TaskFlow persistence with Mistral persistence. Renat is taking a look. * Declaring Flows in YAML DSL: done for simplest flow. Need to prototype for data flow. Rich flow controls are missing in TaskFlow for a representative prototype. * ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both Mistral and TaskFlow implementations work. But we shall resolve the overlap. * Ignored for now - unlikely any risks: Keystone integration, Mistral event scheduler, Mistral declarative services and action definition. 2) TaskFlow library features * Must: flow control - conditional transitions, references, expression evaluation, to express real-life workflows [5]. The required flow control primitives are 1) repeater 2) flow in flow 3) direct transition 4) conditional transition 5) multiple data. TaskFlow has 1) and 2), need to add 3/4/5. * Other details and smaller requests are in the discussion [1] 3) Next Steps proposed: * Mistal team: summarize the requirements discussed and agreed on [2] and [3] * Mistral team: code sample (tests?) on how Mistral would like to consume TaskFlow lazy engine * Taskflow team: Provide a design for alternative TaskExecutor approach (prototypes, boxes, arrows, crayons :)) * Decide on lazy engine * Move the discussion on other elements on integration. References: [0] The scope of the prototype: https://etherpad.openstack.org/p/mistral-taskflow-prototype [1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1 [2] Techical summary http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html [3] Email discussion on TaskFlow lazy eninge http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html [4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/ [5] Use cases https://github.com/dzimine/mistral-workflows/tree/add-usecases ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
Gotcha, thanks :) From: Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, April 14, 2014 at 5:11 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary The more the better :) If seriously, this one is less detailed and rather focuses on high-level things so that everyone can have a high-level understanding of what’s been going on on Mistral/TaskFlow integration. In other words, this email raises question “What” and doesn’t raise “Why?” while [0] goes deeper into details. [0] http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html Renat Akhmerov @ Mirantis Inc. On 12 Apr 2014, at 00:59, Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote: I'm confused, why is this 2 emails?? http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html Seems better to just have 1 chain, not 2. From: Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Friday, April 11, 2014 at 9:55 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary We prototyped Mistral / TaskFlow integration and have a follow-up discussions. SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, with some required modifications to function resliently as a service, and for smooth integration. However, the TaskFlow flow controls are insufficient for Mistral use cases. Details discussed on other thirds. The prototype scope - [0https://etherpad.openstack.org/p/mistral-taskflow-prototype]; code and discussion - [1https://github.com/enykeev/mistral/pull/1] and techical highlights - [2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html]. DETAILS: 1) Embedding TaskFlow inside Mistral: * Required: make the engine lazy [3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html], [4http://paste.openstack.org/show/75389/].This is required to support long-running delegates and not loose tasks when engine manager process restarts. * Persistence: need clarity how to replace or mix-in TaskFlow persistence with Mistral persistence. Renat is taking a look. * Declaring Flows in YAML DSL: done for simplest flow. Need to prototype for data flow. Rich flow controls are missing in TaskFlow for a representative prototype. * ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both Mistral and TaskFlow implementations work. But we shall resolve the overlap. * Ignored for now - unlikely any risks: Keystone integration, Mistral event scheduler, Mistral declarative services and action definition. 2) TaskFlow library features * Must: flow control - conditional transitions, references, expression evaluation, to express real-life workflows [5https://github.com/dzimine/mistral-workflows/tree/add-usecases]. The required flow control primitives are 1) repeater 2) flow in flow 3) direct transition 4) conditional transition 5) multiple data. TaskFlow has 1) and 2), need to add 3/4/5. * Other details and smaller requests are in the discussion [1https://github.com/enykeev/mistral/pull/1] 3) Next Steps proposed: * Mistal team: summarize the requirements discussed and agreed on [2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html] and [3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html] * Mistral team: code sample (tests?) on how Mistral would like to consume TaskFlow lazy engine * Taskflow team: Provide a design for alternative TaskExecutor approach (prototypes, boxes, arrows, crayons :)) * Decide on lazy engine * Move the discussion on other elements on integration. References: [0] The scope of the prototype: https://etherpad.openstack.org/p/mistral-taskflow-prototype [1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1 [2] Techical summary http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html [3] Email discussion on TaskFlow lazy eninge http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html [4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/ [5] Use cases https://github.com/dzimine/mistral-workflows/tree/add-usecases ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
Can we describe exactly what references, direct transition, expression evaluation are doing in #2. Expression evaluation especially seems to be an odd one, what's wrong with pythons expression evaluation? I can't quite see why that would/should exist in taskflow. I can see it being implemented in mistral, where mistral converts whatever DSL it wants into taskflow primitives and then taskflow runs the code; this decoupling ensures that taskflow does not force a DSL on people that want to use taskflow as a python library (this kind of restriction imho isn't acceptable for a library to do, and limits taskflows own usage and integration). Thanks, Josh From: Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Friday, April 11, 2014 at 9:55 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary We prototyped Mistral / TaskFlow integration and have a follow-up discussions. SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, with some required modifications to function resliently as a service, and for smooth integration. However, the TaskFlow flow controls are insufficient for Mistral use cases. Details discussed on other thirds. The prototype scope - [0https://etherpad.openstack.org/p/mistral-taskflow-prototype]; code and discussion - [1https://github.com/enykeev/mistral/pull/1] and techical highlights - [2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html]. DETAILS: 1) Embedding TaskFlow inside Mistral: * Required: make the engine lazy [3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html], [4http://paste.openstack.org/show/75389/].This is required to support long-running delegates and not loose tasks when engine manager process restarts. * Persistence: need clarity how to replace or mix-in TaskFlow persistence with Mistral persistence. Renat is taking a look. * Declaring Flows in YAML DSL: done for simplest flow. Need to prototype for data flow. Rich flow controls are missing in TaskFlow for a representative prototype. * ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both Mistral and TaskFlow implementations work. But we shall resolve the overlap. * Ignored for now - unlikely any risks: Keystone integration, Mistral event scheduler, Mistral declarative services and action definition. 2) TaskFlow library features * Must: flow control - conditional transitions, references, expression evaluation, to express real-life workflows [5https://github.com/dzimine/mistral-workflows/tree/add-usecases]. The required flow control primitives are 1) repeater 2) flow in flow 3) direct transition 4) conditional transition 5) multiple data. TaskFlow has 1) and 2), need to add 3/4/5. * Other details and smaller requests are in the discussion [1https://github.com/enykeev/mistral/pull/1] 3) Next Steps proposed: * Mistal team: summarize the requirements discussed and agreed on [2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html] and [3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html] * Mistral team: code sample (tests?) on how Mistral would like to consume TaskFlow lazy engine * Taskflow team: Provide a design for alternative TaskExecutor approach (prototypes, boxes, arrows, crayons :)) * Decide on lazy engine * Move the discussion on other elements on integration. References: [0] The scope of the prototype: https://etherpad.openstack.org/p/mistral-taskflow-prototype [1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1 [2] Techical summary http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html [3] Email discussion on TaskFlow lazy eninge http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html [4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/ [5] Use cases https://github.com/dzimine/mistral-workflows/tree/add-usecases ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
They are all parts of conditional transitions: every task should have a number of possible transitions; each transition consist of a reference to the task we want to transit to and the condition that should evaluate to true for transition to start. At that point, I'd say that it perfectly fine for TaskFlow to evaluate python conditions rather than implementing YAQL, though there should be a place for us to pass the condition evaluation logic we are using. -- Kirill Izotov вторник, 15 апреля 2014 г. в 8:02, Joshua Harlow написал: Can we describe exactly what references, direct transition, expression evaluation are doing in #2. Expression evaluation especially seems to be an odd one, what's wrong with pythons expression evaluation? I can't quite see why that would/should exist in taskflow. I can see it being implemented in mistral, where mistral converts whatever DSL it wants into taskflow primitives and then taskflow runs the code; this decoupling ensures that taskflow does not force a DSL on people that want to use taskflow as a python library (this kind of restriction imho isn't acceptable for a library to do, and limits taskflows own usage and integration). Thanks, Josh From: Dmitri Zimine d...@stackstorm.com (mailto:d...@stackstorm.com) Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org) Date: Friday, April 11, 2014 at 9:55 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org) Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary We prototyped Mistral / TaskFlow integration and have a follow-up discussions. SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, with some required modifications to function resliently as a service, and for smooth integration. However, the TaskFlow flow controls are insufficient for Mistral use cases. Details discussed on other thirds. The prototype scope - [0 (https://etherpad.openstack.org/p/mistral-taskflow-prototype)]; code and discussion - [1 (https://github.com/enykeev/mistral/pull/1)] and techical highlights - [2 (http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html)]. DETAILS: 1) Embedding TaskFlow inside Mistral: * Required: make the engine lazy [3 (http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html)], [4 (http://paste.openstack.org/show/75389/)].This is required to support long-running delegates and not loose tasks when engine manager process restarts. * Persistence: need clarity how to replace or mix-in TaskFlow persistence with Mistral persistence. Renat is taking a look. * Declaring Flows in YAML DSL: done for simplest flow. Need to prototype for data flow. Rich flow controls are missing in TaskFlow for a representative prototype. * ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both Mistral and TaskFlow implementations work. But we shall resolve the overlap. * Ignored for now - unlikely any risks: Keystone integration, Mistral event scheduler, Mistral declarative services and action definition. 2) TaskFlow library features * Must: flow control - conditional transitions, references, expression evaluation, to express real-life workflows [5 (https://github.com/dzimine/mistral-workflows/tree/add-usecases)]. The required flow control primitives are 1) repeater 2) flow in flow 3) direct transition 4) conditional transition 5) multiple data. TaskFlow has 1) and 2), need to add 3/4/5. * Other details and smaller requests are in the discussion [1 (https://github.com/enykeev/mistral/pull/1)] 3) Next Steps proposed: * Mistal team: summarize the requirements discussed and agreed on [2 (http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html)] and [3 (http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html)] * Mistral team: code sample (tests?) on how Mistral would like to consume TaskFlow lazy engine * Taskflow team: Provide a design for alternative TaskExecutor approach (prototypes, boxes, arrows, crayons :)) * Decide on lazy engine * Move the discussion on other elements on integration. References: [0] The scope of the prototype: https://etherpad.openstack.org/p/mistral-taskflow-prototype [1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1 [2] Techical summary http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html [3] Email discussion on TaskFlow lazy eninge http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html [4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/
Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
Thanks, that helps explain those. Let's see where the conditional evaluation logic goes. Likely it won't be python conditions directly. I think though we should be able to work with other condition logic. I started https://review.openstack.org/#/c/87417/ today, hopefully can flush it out more in the days to come. The differences I can see so far are around how a taskflow engine activates this conditional (and what results when the switch chooses a path). In taskflow the whole workflow is analyzed before execution (and translated into a directed graph) as to what is provided and what is required by each task in the workflow. Conditionals change this since its not known ahead of time which path will be selected. For now (in the above review) I am making it so that each path that could be switched to will have to have the same requirements and the same outputs (so that the analysis logic still works correctly). I'm thinking that a switch 'task' will return which choice it made, then this will affect the further path that will be followed (basically all other path choices will be 'abandoned'). This fits pretty well I think into how typically this is done in dataflow-like way and won't affect the state-transitions or ability to resume and such (btw found some neat papers at [1],[2] that show some past history that I didn't know about). Of course I'm trying to make the above not be its own micro-language as much as possible (a switch object starts to act like one, sadly). Comments welcome. Code welcome even more :-P [1] http://www.cs.cmu.edu/~aldrich/papers/onward2009-concurrency.pdf [2] http://www.cs.ucf.edu/~dcm/Teaching/COT4810-Spring2011/Literature/DataFlowProgrammingLanguages.pdf From: Kirill Izotov enyk...@stackstorm.commailto:enyk...@stackstorm.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, April 14, 2014 at 8:31 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary They are all parts of conditional transitions: every task should have a number of possible transitions; each transition consist of a reference to the task we want to transit to and the condition that should evaluate to true for transition to start. At that point, I'd say that it perfectly fine for TaskFlow to evaluate python conditions rather than implementing YAQL, though there should be a place for us to pass the condition evaluation logic we are using. -- Kirill Izotov вторник, 15 апреля 2014 г. в 8:02, Joshua Harlow написал: Can we describe exactly what references, direct transition, expression evaluation are doing in #2. Expression evaluation especially seems to be an odd one, what's wrong with pythons expression evaluation? I can't quite see why that would/should exist in taskflow. I can see it being implemented in mistral, where mistral converts whatever DSL it wants into taskflow primitives and then taskflow runs the code; this decoupling ensures that taskflow does not force a DSL on people that want to use taskflow as a python library (this kind of restriction imho isn't acceptable for a library to do, and limits taskflows own usage and integration). Thanks, Josh From: Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Friday, April 11, 2014 at 9:55 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary We prototyped Mistral / TaskFlow integration and have a follow-up discussions. SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, with some required modifications to function resliently as a service, and for smooth integration. However, the TaskFlow flow controls are insufficient for Mistral use cases. Details discussed on other thirds. The prototype scope - [0https://etherpad.openstack.org/p/mistral-taskflow-prototype]; code and discussion - [1https://github.com/enykeev/mistral/pull/1] and techical highlights - [2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html]. DETAILS: 1) Embedding TaskFlow inside Mistral: * Required: make the engine lazy [3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html], [4http://paste.openstack.org/show/75389/].This is required to support long-running delegates and not loose tasks when engine manager process restarts. * Persistence: need clarity how to replace or mix-in TaskFlow persistence with Mistral persistence. Renat is taking a look. * Declaring Flows in YAML DSL: done
Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
I'm confused, why is this 2 emails?? http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html Seems better to just have 1 chain, not 2. From: Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Friday, April 11, 2014 at 9:55 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary We prototyped Mistral / TaskFlow integration and have a follow-up discussions. SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, with some required modifications to function resliently as a service, and for smooth integration. However, the TaskFlow flow controls are insufficient for Mistral use cases. Details discussed on other thirds. The prototype scope - [0https://etherpad.openstack.org/p/mistral-taskflow-prototype]; code and discussion - [1https://github.com/enykeev/mistral/pull/1] and techical highlights - [2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html]. DETAILS: 1) Embedding TaskFlow inside Mistral: * Required: make the engine lazy [3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html], [4http://paste.openstack.org/show/75389/].This is required to support long-running delegates and not loose tasks when engine manager process restarts. * Persistence: need clarity how to replace or mix-in TaskFlow persistence with Mistral persistence. Renat is taking a look. * Declaring Flows in YAML DSL: done for simplest flow. Need to prototype for data flow. Rich flow controls are missing in TaskFlow for a representative prototype. * ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both Mistral and TaskFlow implementations work. But we shall resolve the overlap. * Ignored for now - unlikely any risks: Keystone integration, Mistral event scheduler, Mistral declarative services and action definition. 2) TaskFlow library features * Must: flow control - conditional transitions, references, expression evaluation, to express real-life workflows [5https://github.com/dzimine/mistral-workflows/tree/add-usecases]. The required flow control primitives are 1) repeater 2) flow in flow 3) direct transition 4) conditional transition 5) multiple data. TaskFlow has 1) and 2), need to add 3/4/5. * Other details and smaller requests are in the discussion [1https://github.com/enykeev/mistral/pull/1] 3) Next Steps proposed: * Mistal team: summarize the requirements discussed and agreed on [2http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html] and [3http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html] * Mistral team: code sample (tests?) on how Mistral would like to consume TaskFlow lazy engine * Taskflow team: Provide a design for alternative TaskExecutor approach (prototypes, boxes, arrows, crayons :)) * Decide on lazy engine * Move the discussion on other elements on integration. References: [0] The scope of the prototype: https://etherpad.openstack.org/p/mistral-taskflow-prototype [1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1 [2] Techical summary http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html [3] Email discussion on TaskFlow lazy eninge http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html [4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/ [5] Use cases https://github.com/dzimine/mistral-workflows/tree/add-usecases ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
